The node labels were already supported for zones and regions to add to
the CRUSH map. Now all layers of the CRUSH map will be supported
with the new labels such as topology.rook.io/rack.
The labels will be detected at the osd startup time
similar to the zone and region labels already being detected.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
when running OSD on PVC, orchestration should not be started again if there is any change in the device config
map. Added a check to skip orchestration on device CM update when StorageClassDeviceSets are available,
that is, when running osd on pvc.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
If the cluster has previously been initialized and the operator
is restarted, the operator needs to perform the first orchestration
to confirm basic cluster health, then start watching for CR events.
If there was a single mon pod in a failed or pending pod state, the
orchestration was blocked even if just one mon was in this state.
The scenario is when a node is lost and a mon and the operator happen
to be running on that node. The operator will restart, but the mon
will not restart because it is tied to the node that died (unless running
on a PVC). The mon would remain in a pending state while attempting
to schedule on the node that is down indefinitely and the operator would
be blocked indefinitely. Now we will just log the error and attempt
to continue.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Some testing on OpenShift has shown that enabling Hostnetworking causes
issues with ceph-volume creating OSD.
This is a pure Red Hat OpenShift downstream issue.
This is meant to be temporary while we are waiting for the Selinux team
to get back to us.
Signed-off-by: Sébastien Han <seb@redhat.com>
Generating keyrings after deployments created a race condition
resulting in intermittent OSD pod failure on the first attempt.
This change creates keyrings prior to creation of OSD deployments
and updates the owner after the creation of a deployment.
Partially fixes: rook#4089
Signed-off-by: egafford <egafford@redhat.com>
In Octopus, this commit
https://github.com/ceph/ceph/pull/30859/commits/c09b82a80a392ccd0da7677c7b424ce5cd3fa5d6
moved the quorum_status command to a socket call. Thus, the `ceph
quorum_status` command does not exist anymore, to use it we need to do
`ceph daemon mon.$ID quorum_status`. However, this command only works
locally so it won't work if we run it from the operator.
The `ceph mon_status` command is a general command that run from
anywhere and returns the mon map, let's use this one on the operator
code.
This is not a breaking change.
Signed-off-by: Sébastien Han <seb@redhat.com>
Even legacy osds were still running with this flag. In my previous patch
I was trying to minimize the impact of the change and was afraid the
config file was holding important information but that appears not to be
true.
The only option in the config could have the keyring path, but it's
passed in the CLI startup line.
Closes: https://github.com/rook/rook/issues/4063
Signed-off-by: Sébastien Han <seb@redhat.com>
The mgr modules take some time to run and have also been known
to hang. While the hang issues should have been resolved separately,
an improvement is still to allow the configuration to happen
in parallel for unrelated modules. All of the module configuration
will be completed before continuing with osd configuration
in order to keep the logging simpler.
Signed-off-by: travisn <tnielsen@redhat.com>
OSD on PVC does not upgrade when the user upgrades the ceph version on cluster-on-pvc yaml.
The solution incudes:
- upgrade osd prepare and daemon pods on upgrade
- skip c-v prepare if filesystem is already present on the pvc device.
- skip lvm release in case of upgrade.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
OSDs can be removed automatically with the current mechanism if a new
setting removeOSDsIfOutAndSafeToRemove is set to true. The default for
all new or upgraded clusters should be false.
Signed-off-by: rohan47 <rohgupta@redhat.com>
If mon deployments need to be created, the operator should
go ahead and create all of them instead of waiting for quorum
in between checking each mon. In particular, if a cluster is
being restored in a disaster recovery situation, none of the mons
deployments will exist. All of them should be created before
checking for quorum.
Signed-off-by: travisn <tnielsen@redhat.com>
Requests to enable mgr modules or call mgr modules should
timeout rather than hang. For example, if creating a self
signed cert request hangs, the mgr should continue with other
actions in the orchestration
Signed-off-by: travisn <tnielsen@redhat.com>
The osd-prepare jobs should not inherit osd resources since this might
cause scheduling issues.
Instead, we can now use the new CRD property `prepareosd` in the resource section.
Refer to the documentation for more details.
Closes: https://github.com/rook/rook/issues/2502
Signed-off-by: Sébastien Han <seb@redhat.com>
- OSD on PVC provisioning fails if k8s version is less than 1.13.0
- updated document to reflect minimum k8s version required.
- Added check to skip OSD on pvc provisoning if the minimum k8s version requirement is not met.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
The current implementation was overriding any previous annotations set on
the object meta.
Moving the logic to its own method as well as adding unit tests.
Closes: https://github.com/rook/rook/issues/4106
Signed-off-by: Sébastien Han <seb@redhat.com>
Following the Ceph OSD rework, my test environment regularly failed to
start OSDs, as they would start before the postStart lifecyle hook which
was chown'ing the log and data directories could finish. Pods failed to
start for an arbitrarily long time before starting successfully. This
was a manifestation of a race condition identified in Ceph monitors in
https://github.com/rook/rook/pull/3594#discussion_r312279176
Create a Ceph operator-level method to create an init container that
will chown the necessary directories for all Ceph daemons (any daemon
which runs as the ceph:ceph user-and-group). Also add this container to
expectations for Ceph pod templates in unit tests.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
Just as with the mon, mgr, mds, rgw, and rbd-mirror daemons, do not
generate a ceph.conf in an init container for directory-based OSDs.
Instead use the mon config database and the commandline to supply all
the needed arguments for running these OSDs.
Also generate a keyring secret to mount to OSD pods. Use this
secret for directory-based OSDs for now with the intention to
use this for other OSDs in the future.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
As with the rest of the Ceph daemons, use the new config patters for
dir-based OSDs. Do not generate a config to use in an init container;
instead use CLI flags and the mon config database.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
This is just one small unit of work that still results in usable
dir-based OSDs. The goal is to build on this to run dir-based OSDs using
the new config patterns used by the other Ceph daemons. This also
removes an unnecessary mount of the dataDirHostPath in directory-based
OSD containers.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
The cluster finalizer in some scenarios was not being removed
during cluster removal. Specifically, if the cluster CR had been
modified, the operator would always fail to remove the finalizer.
This could occur when the CR status is updated around the same time
that the cluster is deleted. Therefore, we need to remove the finalizer
from a freshly retrieved instance of the cluster.
Signed-off-by: travisn <tnielsen@redhat.com>
We now differentiate the cases where:
* we only consume the external cluster
* we consume the external cluster as well as creating stateless
resources in Kubernetes (bootstrap mds,rgw, nfs)
This is mostly controlled via the image property spec. If not defined,
not extra CRs won't be able to be created.
Now the external cluster feature supports Ceph cluster as of Luminous 12.2.
Signed-off-by: Sébastien Han <seb@redhat.com>
Update pkg/operator/ceph/cluster/osd/spec.go
Rook v1.1.2 uses a different command to start an OSD, and seems to
have problems when --osd-memory-target contains a number with a
decimal point.
Co-Authored-By: Sébastien Han <seb@redhat.com>
Signed-off-by: Erik Knudtson <eknudtson@discogsinc.com>
When osd nodes do not finish the prepare jobs, the user remove the
nodes manully and it causes nodes always is
'OrchestrationStatusStarting' status.
Try to remove them after completeProvisionTimeout.
Signed-off-by: James Lu <jamesluhz@gmail.com>
Add a finalizer to the CephCluster to check
the PV created by CSI so that we can block the
cluster deletion till the PVC are deleted.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
We don't need to force any Ceph config file for ceph-volume based OSDs
since the 'ceph-volume activate' command mount everything at start time
in the right place.
So now, OSD will keep running normally and they will also read back
normally the default location of the ceph configuration:
/etc/ceph/ceph.conf which will allow OSDs to read any configuration
overrides.
We cannot remove the generation of that file in /var/lib/rook/osd* since
we cannot tell at the stage that triggers the generation if we run
ceph-volume or not. So let's keep it around for some time.
If you are wondering how the OSD connects to the mons this is done via
the CEPH_ARGS env variable which was introduced in
https://github.com/rook/rook/pull/3894.
Closes: https://github.com/rook/rook/issues/3926
Signed-off-by: Sébastien Han <seb@redhat.com>
A job is started to detect the Ceph version. This job now allows setting
the node affinity and tolerations with the same setting that is
specified for the mon placement. No new placement spec is required, but
we will simply use the same spec that the mons use.
Signed-off-by: travisn <tnielsen@redhat.com>
Previously the debug flag was turned on for the 'activate' call only,
now 'prepare' will also take advantage of it.
Signed-off-by: Sébastien Han <seb@redhat.com>
When the flex driver is disabled, the check for
volume attachments will fail and the finalizer
is never removed. To avoid this, just log the
failure to list volumes and remove the
finalizer anyway.
Resolves#3912
Signed-off-by: Kristoffer Grönlund <kgronlund@suse.com>
Encrypted OSDs are failing because ceph-volume is unable to determine
how to contact the monitors. Setting the CEPH_CONF env variable for OSD
pods fixes this issue. (#3846)
Signed-off-by: Michael Vollman <michael.b.vollman@gmail.com>
When running osds on pvc, the lvm is still holding the device when the cluster is deleted. As a result the PVs go into `failed` state
when the cluster is deleted. On deactivating the device, the 'failed' pvs get deleted as well. Solution involves:
- Running `lvchange -an <vg group name>` to deactivate the device.
- This command runs after 'osd prepare' completes.
- This command also runs after the 'osd daemon pod' is deleted or if the ceph-osd process terminates for some reasons.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
The node selector for running OSDs on PVCs was using the node name
rather than the node's hostname. In clusters where the node name
is different from the hostname this would cause the osd daemons
to be in pending state indefinitely. Now the node selector will
use the hostname as expected.
Signed-off-by: travisn <tnielsen@redhat.com>
On rare occasion, the ceph checks might not be 100% correct and the user
might decide to force an upgrade anyway.
Correctly, if we are in an upgrade, we perform `ok-to-stop` check for
each daemon, if one of them fails, we retry and eventually fail.
They are corner-cases where users would like to force the upgrade anyway.
Enhance the new CR property: "skipUpgradeChecks: true".
Closes: https://github.com/rook/rook/issues/3872
Signed-off-by: Sébastien Han <seb@redhat.com>
Travis Nielsen
Santosh Pillai
Sébastien Han
egafford
rohan47
Mateusz Los
Blaine Gardner
Erik KnudtsonandSébastien Han
James Lu
Madhu Rajanna
Kristoffer Grönlund
Michael Vollman