Now, the CephBlockPool CRD is managed with the controller-runtime.
So the watcher is outside of the main controller reconciliation loop of
CephCluster which brings numerous benefit such as:
* having its own reconciliation loop
* won't block anything from the main CephCluster controller loop
* fast than waiting for CephCluster loop to completion
Partially close: https://github.com/rook/rook/issues/1981
Signed-off-by: Sébastien Han <seb@redhat.com>
As of Octopus, Ceph will prevent you from creating a pool with a
replica size of 1. Allowing such pool could lead to data loss, so enable
the new option: requireSafeReplicaSize: false if you are **ABSOLUTELY**
certain that is what you want.
Closes: https://github.com/rook/rook/issues/4889
Signed-off-by: Sébastien Han <seb@redhat.com>
When running old client version, we must force the compatibility
features so that the orchestration does not fail in a loop.
Closes: https://github.com/rook/rook/issues/4842
Signed-off-by: Sébastien Han <seb@redhat.com>
Enabling pg_autoscalar module fails most probably becasue `ceph mgr module enable <>` is happening very quickly after ceph mgr starts.
This PR adds a retry (5 times) with a sleep interval of 5 seconds (for each interval) to handle failure while enabling mgr modules.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
When using the "errors" package, using `%+v` (extended format),
each Frame of the error's StackTrace will be printed in detail.
Let's only print `%v` to print the error.
If the error has a Cause it will be printed recursively.
Basically `%+v` has been replaced with `%v` for all `error` type
interfaces, whether the logger is Info, Warning or Error.
Signed-off-by: Sébastien Han <seb@redhat.com>
It's not 100% safe since before Nautilus osd safe-to-destroy command
didn't have any json output. So added a version check.
Signed-off-by: rohan47 <rohgupta@redhat.com>
ceph client CRD refactoring
Add info about client crd to documentation
Add tests for client controller CRD
Signed-off-by: Mateusz Los <los.mateusz@gmail.com>
The topology of the cluster should be based on the node labels rather
than a setting in the cluster CR. This allows a much richer and more
dynamic topology to be configured. The location will now be ignored
if specified in the cluster CR.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
In Octopus, this commit
https://github.com/ceph/ceph/pull/30859/commits/c09b82a80a392ccd0da7677c7b424ce5cd3fa5d6
moved the quorum_status command to a socket call. Thus, the `ceph
quorum_status` command does not exist anymore, to use it we need to do
`ceph daemon mon.$ID quorum_status`. However, this command only works
locally so it won't work if we run it from the operator.
The `ceph mon_status` command is a general command that run from
anywhere and returns the mon map, let's use this one on the operator
code.
This is not a breaking change.
Signed-off-by: Sébastien Han <seb@redhat.com>
The command `fs new` already enable the application pool settings for
the cephfs pools so we don't need to run that command again.
Signed-off-by: Sébastien Han <seb@redhat.com>
Checking for clean PGs is a recurring event when reconciling nodes
for upgrades and fencing. No need to log every request.
Signed-off-by: travisn <tnielsen@redhat.com>
We now differentiate the cases where:
* we only consume the external cluster
* we consume the external cluster as well as creating stateless
resources in Kubernetes (bootstrap mds,rgw, nfs)
This is mostly controlled via the image property spec. If not defined,
not extra CRs won't be able to be created.
Now the external cluster feature supports Ceph cluster as of Luminous 12.2.
Signed-off-by: Sébastien Han <seb@redhat.com>
A new CRD property `PreservePoolsOnDelete` has been added to Filesystem(fs) and
Object Store(os) resources in order to increase protection against data loss.
If it is set to `true`, associated pools won't be deleted when the main
resource(fs/os) is deleted. Creating again the deleted fs/os with the same name
will reuse the preserved pools.
Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
currently we are not checking the pool is empty or
not before deleting, This PR adds a check to check
if any images/snapshots using the pool, if yes it will
not delete the pool
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
Several commands were not using the cli finalizer function which builds
up the right argument to run a ceph command. This was working on the
operator since it used to have an /etc/ceph/ceph.conf file so we could
ran the command without specifying the ceph conf and key paths.
However, we recently removed that and it silently broke this "already"
broken code :).
Signed-off-by: Sébastien Han <seb@redhat.com>
Maintenance is only defined when managedDisruptionBudgets are enabled.
Details can be found in the design doc for manged poddisriptionbudgets.
Signed-off-by: Rohan CJ <rohantmp@gmail.com>
Allow users to specify overrides in the CephCluster CRD. The override
ConfigMap still exists for emergency situations and is mounted into
daemon pods directly instead of being merged into a Rook-created config
file.
Rook Ceph no longer uses a config file for managing daemons with the
exception of the OSDs which still generate a config in an init
container.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
With this change, the admin can specify a single mon and the operator
will add any other mons to both ClusterInfo and the configmap of mon
endpoints.
It will periodically check for changes too and update both ClusterInfo
and the configmap accordingly.
Signed-off-by: Sébastien Han <seb@redhat.com>
GetMonStatus is called by checkHealth in health.go and is already
printing GetMonStatus's response so there is no need to print it twice.
Signed-off-by: Sébastien Han <seb@redhat.com>
The daemon name always needs to have the prefix rook-ceph-mds-
removed from the deployment name in order to translate to the
daemon name. The daemon name is not dependent on the namespace.
Signed-off-by: travisn <tnielsen@redhat.com>
With the 1.1 release we should not be creating bluestore OSDs
with rook's legacy partitioning scheme. All new bluestore OSDs
should be created with ceph-volume. Therefore, the miminum
version allowed is a version of mimic supporting ceph-volume.
This commit also adds consistent version checking between
the Add and Update methods for the operator handling the
CR events.
Signed-off-by: travisn <tnielsen@redhat.com>
This is not changing the behavior of the code but making the
flow of logic in the createReplicationCrushRule() function consistent
by removing an else branch. This was just a reflex when I was reading
the code.
Signed-off-by: Michael Adam <obnox@redhat.com>
We do not determine the daemon type and name by string parsing anymore
but passing that info directly in the update call, it's more reliable.
Also removing a bunch of unused functions.
Signed-off-by: Sébastien Han <seb@redhat.com>
Do not consider clusterInfo Ceph version since it's the one picked by
the spec image.
The CephVersion provided here needs to be the current running version
of Ceph for when it is later passed to OkToStop and checked with
IsAtLeast. If you are upgrading from a pre 14.2.1 version to a
version 14.2.1 or newer, IsAtLeast will incorrectly return true and
then fail trying to run mon ok-to-stop on a mon pod running a pre 14.2.1 version.
Signed-off-by: Sébastien Han <seb@redhat.com>
When a cluster is updated with a different image version, this triggers
a serialized restart of all the pods. Prior to this commit, no safety
check were performed and rook was hoping for the best outcome.
Now before doing restarting a daemon we check it can be restarted. Once
it's restarted we also check we can pursue with the rest of the
platform. For instance, with monitors we check that they are in quorum,
for OSD we check that PGs are clean and for MDS we make sure they are
all active.
Fixes: https://github.com/rook/rook/issues/2889
Signed-off-by: Sébastien Han <seb@redhat.com>
Example of failure scenarios:
- requested PVC 500M = 500,000,000 bytes
- PV is created as:
(500M / 2**20)Mi = "476"Mi
But 476Mi = 476 * 2**20 = 499122176 < 500M
From the k8s documentation:
Claims will remain unbound indefinitely if a matching volume does not
exist. Claims will be bound as matching volumes become available. For
example, a cluster provisioned with many 50Gi PVs would not match a
PVC requesting 100Gi. The PVC can be bound when a 100Gi PV is added
to the cluster.
which is inline with the behavior observed in which a 500M PVC is never
bound. However, the documentation also states:
If a PV was dynamically provisioned for a new PVC, the loop will
always bind that PV to the PVC. Otherwise, the user will always get
at least what they asked for
which suggests that an _explicit_ binding can be created, and perhaps
something has changed with respect to this binding that has caused the
failures in newer versions of k8s.
fixes: #3391fixes: #2922
Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
This commit does:
* remove dead code SetCrushTunables()
* stop creating initial crushmap
There is no need to create an initial crushmap, since Ceph natively does
that for us.
We don't delete the existing configmap containing the initial crush map
of already deployed cluster, perhaps people are using for whatever
reason. It does not harm to keep it around.
Closes: https://github.com/rook/rook/issues/3138
Signed-off-by: Sébastien Han <seb@redhat.com>
This requirement is long gone and was reported in 2017. Clients should
have been updated by then, so there is no reason to set the CRUSH
tunable to such an old client. Actually, we should let Ceph run its own
tunable. They can always be changed later. Also, this was really
restrictive and applied on every orchestration and thus would override
any other config done by an administrator. Even worse, this will trigger
data movement back and forth...
Closes: https://github.com/rook/rook/issues/3138
Signed-off-by: Sébastien Han <seb@redhat.com>
- When an osd is marked out, and it is safe to be destroyed then delete the osds deployment.
- Removed osdGracePeriod
- Updated unit tests to test osd marked out action
Signed-off-by: rohan47 <rohgupta@redhat.com>
When OSDs are running on Nautilus we always disable old osd features and
aplpy the onces for Nautilus as described in the upgrade doc.
During an upgrade or the next time an orchestration will be called the
command will be applied. The command is idempotent so we can run it each
time.
This can be backported for 1.0.3
Closes: https://github.com/rook/rook/issues/2960
Signed-off-by: Sébastien Han <seb@redhat.com>
Sébastien Han
Santosh Pillai
rohan47
Mateusz Los
Mateusz Gozdek
Travis Nielsen
Juan Miguel Olmo Martínez
Madhu Rajanna
Rohan CJ
Maksim Nabokikh
Blaine Gardner
Michael Adam
Noah Watkins