In Octopus, this commit
https://github.com/ceph/ceph/pull/30859/commits/c09b82a80a392ccd0da7677c7b424ce5cd3fa5d6
moved the quorum_status command to a socket call. Thus, the `ceph
quorum_status` command does not exist anymore, to use it we need to do
`ceph daemon mon.$ID quorum_status`. However, this command only works
locally so it won't work if we run it from the operator.
The `ceph mon_status` command is a general command that run from
anywhere and returns the mon map, let's use this one on the operator
code.
This is not a breaking change.
Signed-off-by: Sébastien Han <seb@redhat.com>
The command `fs new` already enable the application pool settings for
the cephfs pools so we don't need to run that command again.
Signed-off-by: Sébastien Han <seb@redhat.com>
Checking for clean PGs is a recurring event when reconciling nodes
for upgrades and fencing. No need to log every request.
Signed-off-by: travisn <tnielsen@redhat.com>
We now differentiate the cases where:
* we only consume the external cluster
* we consume the external cluster as well as creating stateless
resources in Kubernetes (bootstrap mds,rgw, nfs)
This is mostly controlled via the image property spec. If not defined,
not extra CRs won't be able to be created.
Now the external cluster feature supports Ceph cluster as of Luminous 12.2.
Signed-off-by: Sébastien Han <seb@redhat.com>
A new CRD property `PreservePoolsOnDelete` has been added to Filesystem(fs) and
Object Store(os) resources in order to increase protection against data loss.
If it is set to `true`, associated pools won't be deleted when the main
resource(fs/os) is deleted. Creating again the deleted fs/os with the same name
will reuse the preserved pools.
Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
currently we are not checking the pool is empty or
not before deleting, This PR adds a check to check
if any images/snapshots using the pool, if yes it will
not delete the pool
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
Several commands were not using the cli finalizer function which builds
up the right argument to run a ceph command. This was working on the
operator since it used to have an /etc/ceph/ceph.conf file so we could
ran the command without specifying the ceph conf and key paths.
However, we recently removed that and it silently broke this "already"
broken code :).
Signed-off-by: Sébastien Han <seb@redhat.com>
Maintenance is only defined when managedDisruptionBudgets are enabled.
Details can be found in the design doc for manged poddisriptionbudgets.
Signed-off-by: Rohan CJ <rohantmp@gmail.com>
Allow users to specify overrides in the CephCluster CRD. The override
ConfigMap still exists for emergency situations and is mounted into
daemon pods directly instead of being merged into a Rook-created config
file.
Rook Ceph no longer uses a config file for managing daemons with the
exception of the OSDs which still generate a config in an init
container.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
With this change, the admin can specify a single mon and the operator
will add any other mons to both ClusterInfo and the configmap of mon
endpoints.
It will periodically check for changes too and update both ClusterInfo
and the configmap accordingly.
Signed-off-by: Sébastien Han <seb@redhat.com>
GetMonStatus is called by checkHealth in health.go and is already
printing GetMonStatus's response so there is no need to print it twice.
Signed-off-by: Sébastien Han <seb@redhat.com>
The daemon name always needs to have the prefix rook-ceph-mds-
removed from the deployment name in order to translate to the
daemon name. The daemon name is not dependent on the namespace.
Signed-off-by: travisn <tnielsen@redhat.com>
With the 1.1 release we should not be creating bluestore OSDs
with rook's legacy partitioning scheme. All new bluestore OSDs
should be created with ceph-volume. Therefore, the miminum
version allowed is a version of mimic supporting ceph-volume.
This commit also adds consistent version checking between
the Add and Update methods for the operator handling the
CR events.
Signed-off-by: travisn <tnielsen@redhat.com>
This is not changing the behavior of the code but making the
flow of logic in the createReplicationCrushRule() function consistent
by removing an else branch. This was just a reflex when I was reading
the code.
Signed-off-by: Michael Adam <obnox@redhat.com>
We do not determine the daemon type and name by string parsing anymore
but passing that info directly in the update call, it's more reliable.
Also removing a bunch of unused functions.
Signed-off-by: Sébastien Han <seb@redhat.com>
Do not consider clusterInfo Ceph version since it's the one picked by
the spec image.
The CephVersion provided here needs to be the current running version
of Ceph for when it is later passed to OkToStop and checked with
IsAtLeast. If you are upgrading from a pre 14.2.1 version to a
version 14.2.1 or newer, IsAtLeast will incorrectly return true and
then fail trying to run mon ok-to-stop on a mon pod running a pre 14.2.1 version.
Signed-off-by: Sébastien Han <seb@redhat.com>
When a cluster is updated with a different image version, this triggers
a serialized restart of all the pods. Prior to this commit, no safety
check were performed and rook was hoping for the best outcome.
Now before doing restarting a daemon we check it can be restarted. Once
it's restarted we also check we can pursue with the rest of the
platform. For instance, with monitors we check that they are in quorum,
for OSD we check that PGs are clean and for MDS we make sure they are
all active.
Fixes: https://github.com/rook/rook/issues/2889
Signed-off-by: Sébastien Han <seb@redhat.com>
Example of failure scenarios:
- requested PVC 500M = 500,000,000 bytes
- PV is created as:
(500M / 2**20)Mi = "476"Mi
But 476Mi = 476 * 2**20 = 499122176 < 500M
From the k8s documentation:
Claims will remain unbound indefinitely if a matching volume does not
exist. Claims will be bound as matching volumes become available. For
example, a cluster provisioned with many 50Gi PVs would not match a
PVC requesting 100Gi. The PVC can be bound when a 100Gi PV is added
to the cluster.
which is inline with the behavior observed in which a 500M PVC is never
bound. However, the documentation also states:
If a PV was dynamically provisioned for a new PVC, the loop will
always bind that PV to the PVC. Otherwise, the user will always get
at least what they asked for
which suggests that an _explicit_ binding can be created, and perhaps
something has changed with respect to this binding that has caused the
failures in newer versions of k8s.
fixes: #3391fixes: #2922
Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
This commit does:
* remove dead code SetCrushTunables()
* stop creating initial crushmap
There is no need to create an initial crushmap, since Ceph natively does
that for us.
We don't delete the existing configmap containing the initial crush map
of already deployed cluster, perhaps people are using for whatever
reason. It does not harm to keep it around.
Closes: https://github.com/rook/rook/issues/3138
Signed-off-by: Sébastien Han <seb@redhat.com>
This requirement is long gone and was reported in 2017. Clients should
have been updated by then, so there is no reason to set the CRUSH
tunable to such an old client. Actually, we should let Ceph run its own
tunable. They can always be changed later. Also, this was really
restrictive and applied on every orchestration and thus would override
any other config done by an administrator. Even worse, this will trigger
data movement back and forth...
Closes: https://github.com/rook/rook/issues/3138
Signed-off-by: Sébastien Han <seb@redhat.com>
- When an osd is marked out, and it is safe to be destroyed then delete the osds deployment.
- Removed osdGracePeriod
- Updated unit tests to test osd marked out action
Signed-off-by: rohan47 <rohgupta@redhat.com>
When OSDs are running on Nautilus we always disable old osd features and
aplpy the onces for Nautilus as described in the upgrade doc.
During an upgrade or the next time an orchestration will be called the
command will be applied. The command is idempotent so we can run it each
time.
This can be backported for 1.0.3
Closes: https://github.com/rook/rook/issues/2960
Signed-off-by: Sébastien Han <seb@redhat.com>
- Updated code to use deviceClass property when a pool is created for both "replicated" and "erasure code".
- Updated "ceph crush rule create-..." command to use "create-replicated" instead of "create-simple"
- Updated unit tests
- Updated (ceph-pool-crd.md) documentation to reflect the changes.
- Updated pending release notes.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
this adds a structure to hold various settings related to executing ceph
cli commands, and introduces a common interface for configuring and
running such commands.
Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
This reverts commit b7a564e929, reversing
changes made to 6996571b5c.
Reverting: Linking ceph noout to a node taint.
The taint is applied for reasons other than maintenance as well.
This also needs a way to respect user-set noout.
Need to revisit this with a better design.
Signed-off-by: Rohan CJ <rohantmp@gmail.com>
This commits allows the upgrade from Mimic to Nautilus to work by:
* removing the v2 brackets on the operator ceph config file generation,
we stick with v1 and can revert this back once we deprecate mimic there
is no rush since mons keep on listening to v1.
* enable messengers 2 when the cluster runs on Nautilus
Fixes: https://github.com/rook/rook/issues/2973
Signed-off-by: Sébastien Han <seb@redhat.com>
A ceph filesystem was brought down by marking the filesystem
as down and individually failing the MDSes. From nautilus onwards,
this could be more simply and efficiently done by using
`fs fail` command. See,
https://github.com/ceph/ceph/commit/4c49f165ec1Resolves: #2841
Signed-off-by: Ramana Raja <rraja@redhat.com>
The CephCluster resource will now expose the health of the ceph cluster
so admins can query the status with k8s api or kubectl instead of
needing to run the rook toolbox. The operator will query the
ceph status periodically and update the status in the custom resource.
Signed-off-by: travisn <tnielsen@redhat.com>
Update integration test suite fails on an error with getting the keyring
for the mgr. The mgr's cephx capabilities have changed and need to be
updated before get, create, or get-or-create can be called. Since other
daemons' caps may change in the future, make an attempt to update the
user's caps on any key generation. This makes the key generation more
declarative from within the operator code for individual Ceph daemons.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
Nautilus has an issue targeting Luminous when running the rbd info command.
Rook already has the info it needs for the call and can return it
after the rbd create command succeeds.
Signed-off-by: travisn <tnielsen@redhat.com>
The basic idea here is to replace the release name in the ClusterInfo
object with the fine grained version information queried at runtime.
This patch also removes the Name field from the cluster spec, which was
also runtime determined and was redundant with ClusterInfo relase name.
Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
From ceph mimic (13.x) onwards, `mds deactivate` command is
deprecated. To scale down the number of active MDSes of a filesystem,
it's sufficient to set the desired number of active MDSes using
'max_mds' setting, and is no longer required to individually
deactivate the unwanted MDSes.
Resolves: #2725
Signed-off-by: Ramana Raja <rraja@redhat.com>
The MDS mds_standby_for_*, mon_force_standby_active, and mds_standby_replay
configuration options have been obsoleted. Instead, the operator may now set
the new "allow_standby_replay" flag on the CephFS file system. This setting
causes standbys to become standby-replay for any available rank in the file
system.
Closes: https://github.com/rook/rook/issues/2768
Signed-off-by: Sébastien Han <seb@redhat.com>
Sébastien Han
Travis Nielsen
Juan Miguel Olmo Martínez
Madhu Rajanna
Rohan CJ
Maksim Nabokikh
Blaine Gardner
Michael Adam
Noah Watkins
rohan47
Santosh Pillai
Maximilian Mack
Ramana Raja