When the OSD activate init container cannot match its configured
device, it falls back to a bare 'ceph-volume raw list', which
enumerates every block device on the node and opens each one to read
bluestore labels. Opening a krbd device whose backing cluster is
unhealthy can block in uninterruptible sleep; when that device's I/O
depends on the OSD being activated, activation deadlocks.
Build the fallback scan list explicitly with lsblk and exclude the
network-backed device types that can hang when their backing storage
is unavailable (rbd, nbd, drbd) along with zram (volatile RAM). Rook
never provisions OSDs on any of these, so the fallback can never
legitimately need to find an OSD on them. loop devices are kept,
since OSDs on loop devices are supported for CI and local testing.
Fail fast if the filtered list is empty rather than degenerating back
to a full scan.
Note this hardens only the rename-fallback path. The deadlock observed
in CI (kernel hung tasks under 'ceph-volume raw activate' opening a
stalled /dev/rbd0; actions runs 27325963851, 27333888237, and
27378983111, which reproduced it on this PR's own head) ultimately
originates inside ceph-volume: 'raw activate' runs an unconditional
LUKS2/TPM2 discovery sweep over all block devices even when given an
explicit --device. That requires a ceph-volume fix; this change makes
the Rook-controlled scan safe regardless of ceph version.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
when using encryptedDevice:true with host based (non pvc) osds, resizing
the underlying disk and restarting the OSD didn't expand the OSD because
the LUKS encryption layer was not resized.
This PR adds resize capabilities inside the activate container. Full
device stack must be resized: PV → LV → LUKS
Steps:
1. Run pvresize and lvextend to grow the LVM layers.
2. Retrieve the LUKS key from the Ceph config-key store using the
lockbox crednetials already available in the activate container
3. Pass the key explicitly to cryptsetup resize via --key-file.
Signed-off-by: Santosh <sapillai@redhat.com>
Fixes#16834 and #17384. Per-OSD CR deviceClass was ignored in
osd prepare job when reporting the osd config back to the operator,
and in the operator's reconcile override.
Fixed by resolving the per-device class from the CR devices list
in GetCephVolumeRawOSDs and in makeDeployment.
Signed-off-by: Artem Torubarov <artem.torubarov@sap.com>
Detect OSD encryption from the dmcrypt block path in `getOSDInfo` as a
fallback when the `encrypted` label is missing on pre-label deployments.
Include `osd.Encrypted` in the label check in `makeDeployment` so the
label reflects the actual OSD state on upgrade.
Fixes a bug where upgrading to a Rook version with the `encrypted` label
would set the label to `false` on already-encrypted OSDs, because the
label didn't exist yet and `getOSDInfo` defaulted to `false`.
Signed-off-by: Santosh <sapillai@redhat.com>
When an OSD is purged and reprovisioned, it comes back without the
correct CRUSH device class. NVMe devices configured with
deviceClass: nvme in the CR end up as ssd.
Root cause: the operator sets ROOK_OSD_CRUSH_DEVICE_CLASS on prepare
jobs but not on OSD deployments. During re-discovery
(GetCephVolumeRawOSDs), the daemon reads this env var, finds it
absent, and falls back to auto-detected deviceType which returns ssd
for NVMe.
Fix: add crushDeviceClassEnvVar(osd.DeviceClass) to the OSD
deployment spec in spec.go, alongside the existing
dataDeviceClassEnvVar. Existing deployments pick up the new env var
on operator reconciliation without manual intervention.
Fixes#17227
Signed-off-by: David Orman <ormandj@corenode.com>
Update `lockbox.keyring` rotation for encrypted OSDs to ensure the key
is not lost if the init container fails but continues assuming
(reasonably) that the key is likely still relevant.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
Previously, if the OSD cephx-keyring-update init container couldn't
contact the mons, the output redirect would overwrite the on-disk
keyring file with no text, removing the on-disk keyring.
Modify the init container to only overwrite the keyring file when the
ceph command to get the latest key suceeds, to avoid removing the
on-disk key.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
This fixes OSD info being incomplete when a sidecar container is first
in the .spec.Template.Spec.Containers list.
I added a small helper function to find a container named "osd" instead
of always assuming the first one.
Signed-off-by: Jan Breitkopf <jan.breitkopf@prorocketeers.com>
When writing to the operator log, where possible
the namespace of the resource on which the operator
is working will be logged as a standard prefix
to more easily troubleshoot what resource is being reconciled
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The OSDs may require different resources based on their device
class. The overall osd resources were only being used, rather
than properly looking up the specific device class resources.
Now the expected resources will take affect per device class.
Co-authored-by: Matthias Osthues <osthues.matthias@gmail.com>
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
When OSDs start, init containers attempt to get the latest CephX key for
the OSD to allow OSDs to potentially continue working after key rotation.
However, in Ceph corner cases, mons could be unavailable while OSDs
could still be viable with existing on-disk keys. Since key rotation is
a rare event, it's likely that the on-disk keys are still good.
The worst case scenario is that key updates were needed but didn't
succeed in init containers. In this case, instead of the init container
failing, the OSD startup will fail instead. Thus, this should introduce
no extra risk but merely shift the failure to a different container.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
ceph-volume creates osd-lockbox keys for the encryted osds. This PR
rotates those keys along with the OSD keys.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
Implement automated CephX key rotation for OSDs.
OSDs have marshalled CephX statuses applied as an annotation on the OSD
pod spec, which differs from other Rook Ceph daemons for a two reasons:
A large cluster could have hundreds or thousands of OSDs, so it is more
critical that Rook avoid rotating OSD keys more than once in the event a
CephCluster reconcile restarts partway through OSD creation/updation.
OSDs do not have their CephX keys stored in K8s Secrets, whose resource
version is used for enforcing the desired restart behavior other daemon
pods like RGWs.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
Create a new OSD pod init container that loads the latest CephX key for
the OSD. The init container must be run after OSD activation, due to
keyfile reset behavior of ceph-bluestore-tool, used during activation.
The primary purpose of this work is to allow OSDs to start and run
correctly after key rotation, whether done manually by users or
automatically by future Rook work.
This work makes redundant prior code changes done in #14826. The CI test
added in the prior work is kept to ensure no regression.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
If a locally mounted volume is blocked on reads because this OSD is
down, lvs will deadlock trying to read from it. To avoid this, tell LVM
to filter out RBD volumes from its scanning process.
Signed-off-by: Steven Fackler <sfackler@gmail.com>
Some ceph.conf settings need to be unique per node
depending on the hardware. The ceph.conf overrides
can now be customized per-node by creating a node-specific
configmap that will be loaded for all OSDs and OSD prepare
jobs on that node, instead of the default ceph.conf
that is loaded from the rook-config-override configmap.
The node-specific configmaps must have the label
node.config.rook.io/osd, and be named as:
rook-config-override-<hostname>
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
If host path keyring file exists and created during osd-prepare but not
imported to ceph auth then rook-ceph-osd activate init container will
create new keyring in ceph auth by ceph auth get-or-create command. To
avoid keyrings difference in host path and in ceph auth, consider
importing key from host path keyring file if no keyring found in ceph
auth.
Resolves: #14825
Signed-off-by: Peter Razumovsky <prazumovsky@mirantis.com>
Resolves CIS benchmark 5.2.8 rule by adding capabilities with
explicitly defined empty "add" list (where it needed) and with
NET_RAW capability in "drop" list.
5.2.8 Minimize the admission of containers with added capabilities
Containers must drop the `NET_RAW` capability and are not permitted
to add back any capabilities.
Signed-off-by: Peter Razumovsky <prazumovsky@mirantis.com>
The /run/udev directory is now mounted in the OSD init container. This change
is necessary because `ceph-volume activate` needs access to /run/udev [1] in order
to properly handle devices. Without this mount, `ceph-volume activate` could
fail to discover devices during OSD initialization.
[1] https://github.com/ceph/ceph/commit/c2e8c295dc46f1bd74861feb163d9d82f8bb5509
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>
Resolves CIS benchmark rule 5.7.3, Pods part. SecurityContext
should be explicitly defined in pod level of Pod spec section.
It is sufficient to specify empty securityContext to satisfy
CIS 5.7.3 rule.
5.7.3 Apply Security Context to Your Pods and Containers
When designing your containers and pods, make sure
that you configure the security context for your pods,
containers, and volumes.
Signed-off-by: Peter Razumovsky <prazumovsky@mirantis.com>
Given that Ceph Quincy (v17) is past end of life,
remove Quincy from the supported Ceph versions,
examples, and documentation.
Supported versions now include only Reef and Squid.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
This adds an operator config setting ROOK_REVISION_HISTORY_LIMIT
defaulting to kubernetes'value for RevisionHistoryLimit.
If configured, the provided value will be used as RevisionHistoryLimit
for all Deployments rook creates.
Fixes: #12722
Signed-off-by: Michael Adam <obnox@samba.org>
Normally the device class of an OSD is determined at provisioning
and is not updated thereafter. In some scenarios the admin may
want to force update the device class to a new value. The device
classes can be updated by first setting allowDeviceClassUpdate
in the storage spec of the cephcluster, then updating the
device class specified on the deviceSets or other OSDs.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Block devices can move between reboots. In corner cases, an OSDs block
device might move to a lower-indexed device while the previous device
does not exist. For example, an OSD on /dev/sde might move to /dev/sdd
on reboot if the original /dev/sdd died. There would be no /dev/sde
after that.
Users report that NVMe drives commonly change names, even when there are
no disk failures.
For these cases, ensure the activate script properly handles cases where
the previous disk is not present on the node and where the OSD is still
available on a different disk.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
LVM-based OSDs on PVCs are legacy and are not well tested.
Therefore, we add status to the cephcluster CR that at least
gives a clue that the OSDs should be replaced.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
OSDs on LVM-mode PVCs are failing to come up and crashing
in the expand-bluefs init container. To avoid the crash
and allow the OSDs to start, a workaround was found to
simply remove that init container. Now we disable the
OSD resize for this case to avoid others hitting this
during upgrade as well.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The key file deletion process is in the shell script commonly used by
all of encryption-open, encryption-open-metadata, and
encryption-open-wal init containers. The key file is deleted at the
encryption-open init container and encryption-open-metadata and
encryption-open-wal init containers are failed to open the key file.
The key file is in the /etc/ceph folder. Unless that folder is shared,
the key file anyway won't be available in the other init containers
even if it is not deleted by these init containers. And it will
naturally anyway be deleted after the init containers are
completed. So The key file deletion process in shell scripts is
unnecessary.
Fixes: #13737
Signed-off-by: Yuma Ogami <yuma-ogami@cybozu.co.jp>
When OSDs flap, ceph stops the OSD daemon if its marked down greater than
5 times in 600 seconds. But OSD pod restarts and marks the OSD `up` again.
This causes the PGs mapped to these OSDs to peer. While the PGs are peering,
IO to these PGs are blocked.
So we need to ensure that if ceph is marking OSD `down` due to flapping, OSD pod
should not restart to mark the OSDs `up` again.
This PR adds a sleep to the OSD pod if the container returned with a 0 exit code
Default behavior is to sleep for 6 hrs. But user can configure it from the
ceph cluster spec.
Signed-off-by: sp98 <sapillai@redhat.com>
Change how Rook detects network CIDRs for Multus networks. The IPAM
configuration is only defined as an arbitrary string JSON blob with a
"type" field and nothing more. Rook's detection of CIDRs for whereabouts
had already grown out of date since the initial implementation.
Additionally, Rook did not support DHCP IPAM, which is a reasonable
choice for users. And more, Rook did not support CNI plugin chaining,
which further complicates NADs. Based on the CNI spec, network chaning
can result in any changes to network CIDRs from the first-given plugin.
All these problems make it more and more difficult for Rook to support
Multus by inspecting the NAD itself to predict network CIDRs. Instead,
it is better for Rook to treat the CNI process as a black box. To
preserve legacy functionality of auto-detecting networks and to make
that as robust as possible, change to a canary-style architecture like
that used for Ceph mons, from which Rook will detect the network CIDRs
if possible.
Also allow users to specify overrides for CIDR ranges. This allows Rook
to still support esoteric and unexpected NAD or network configurations
where a CIDR range is not detectable or where the range detected would
be incomplete. Because it may be impossible for Rook to understand the
network CIDRs wholistically while residing only on a portion of the
network, this feature should have been present from Multus's inception.
Improving CIDR auto-detection and allowing users to specify overrides
for auto-detected CIDRs rounds out Rook's Multus support for CephCluster
(core/RADOS) installations. No further architectural changes should be
needed for CephClusters as regards application of public/cluster network
CIDRs for Multus networks.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
We need to restart all the ceph daemons whenever
cephCluster network settings are modified like
requiremsgr2, encryption and compression. This
required for Ceph to consider the new settings
it require new ceph daemons all over.
Signed-off-by: subhamkrai <srai@redhat.com>
Signed-off-by: avanthakkar <avanjohn@gmail.com>
Volume "ceph-daemons-sock-dir" is coming empty in case if dataDirHostPath,
which is the case for osd onPVC. Fix the volume creation by using the
ceph cluster spec dataDirHostPath, which allows to run socket commands
on osd containers.
The osd needs to bind to an ip that will allow both internal
and external access for rbd mirroring scenarios.
THe operator will create a service for each osd and
instruction the osd to bind to the pod ip while
publishing the clusterIP of the service as the
public IP.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
(cherry picked from commit 14e7225e0a18aff85ede1d476fcaddbbfbcbb732)
If a kernel device name change happens and a block device file
in the OSD directory becomes dangling link, this OSD fails
to start continuously. This problem can be resolved by confirming
the validity of the device file and recreating it if necessary.
The original problem happened in host-based clusters. A similar
problem exists in PVC-based clusters. In this case, if a PV,
corresponds to an existing OSD, and points to a missing block
device file, the OSD pod fails to consume this PV. Although this
behavior is undesirable, I don't think this problem should be
handled in Rook. If doing so, we must re-create the existing PV.
Rook shouldn't do such work.
Closes: https://github.com/rook/rook/issues/10860
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>