From da07f0c476e779e92614dfdae0ccd46e57db05a4 Mon Sep 17 00:00:00 2001 From: Joshua Hoblitt Date: Thu, 11 Jun 2026 14:27:52 -0700 Subject: [PATCH] osd: do not scan rbd devices in raw activate fallback When the OSD activate init container cannot match its configured device, it falls back to a bare 'ceph-volume raw list', which enumerates every block device on the node and opens each one to read bluestore labels. Opening a krbd device whose backing cluster is unhealthy can block in uninterruptible sleep; when that device's I/O depends on the OSD being activated, activation deadlocks. Build the fallback scan list explicitly with lsblk and exclude the network-backed device types that can hang when their backing storage is unavailable (rbd, nbd, drbd) along with zram (volatile RAM). Rook never provisions OSDs on any of these, so the fallback can never legitimately need to find an OSD on them. loop devices are kept, since OSDs on loop devices are supported for CI and local testing. Fail fast if the filtered list is empty rather than degenerating back to a full scan. Note this hardens only the rename-fallback path. The deadlock observed in CI (kernel hung tasks under 'ceph-volume raw activate' opening a stalled /dev/rbd0; actions runs 27325963851, 27333888237, and 27378983111, which reproduced it on this PR's own head) ultimately originates inside ceph-volume: 'raw activate' runs an unconditional LUKS2/TPM2 discovery sweep over all block devices even when given an explicit --device. That requires a ceph-volume fix; this change makes the Rook-controlled scan safe regardless of ceph version. Signed-off-by: Joshua Hoblitt --- pkg/operator/ceph/cluster/osd/spec.go | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/pkg/operator/ceph/cluster/osd/spec.go b/pkg/operator/ceph/cluster/osd/spec.go index f0cf86739..41d90b747 100644 --- a/pkg/operator/ceph/cluster/osd/spec.go +++ b/pkg/operator/ceph/cluster/osd/spec.go @@ -205,7 +205,18 @@ sys.exit('no disk found with OSD ID $OSD_ID') cat "$OSD_LIST" if ! find_device < "$OSD_LIST"; then - ceph-volume raw list > "$OSD_LIST" + # The disk may have been renamed, so scan disks to find the right one. Build the + # scan list explicitly instead of letting a bare 'ceph-volume raw list' scan every + # block device: opening an rbd device can block in uninterruptible sleep while this + # OSD is down (the same deadlock the lvm filter above prevents for lvm-mode OSDs). + # Skip rbd, nbd, and drbd (network-backed devices that can hang when their backing + # storage is unavailable) and zram (volatile RAM); Rook never provisions OSDs on any + # of these. loop devices are intentionally kept, since OSDs on loop devices are + # supported for CI and local testing. + SCAN_DEVICES="$(lsblk --noheadings --paths --list --output NAME,TYPE | awk '$2 == "disk" || $2 == "part" {print $1}' | grep -vE '^/dev/(rbd|nbd|zram|drbd)' || true)" + [[ -z "$SCAN_DEVICES" ]] && { echo "no devices to scan for OSD $OSD_ID" ; exit 1 ; } + # shellcheck disable=SC2086 # word splitting of the device list is intended + ceph-volume raw list $SCAN_DEVICES > "$OSD_LIST" cat "$OSD_LIST" DEVICE="$(find_device < "$OSD_LIST")"