feedc0de/my-rook-config
1
0
Fork 0
forked from rook/rook
Code Pull Requests Activity

osd: do not scan rbd devices in raw activate fallback

Browse Source 
When the OSD activate init container cannot match its configured
device, it falls back to a bare 'ceph-volume raw list', which
enumerates every block device on the node and opens each one to read
bluestore labels. Opening a krbd device whose backing cluster is
unhealthy can block in uninterruptible sleep; when that device's I/O
depends on the OSD being activated, activation deadlocks.

Build the fallback scan list explicitly with lsblk and exclude the
network-backed device types that can hang when their backing storage
is unavailable (rbd, nbd, drbd) along with zram (volatile RAM). Rook
never provisions OSDs on any of these, so the fallback can never
legitimately need to find an OSD on them. loop devices are kept,
since OSDs on loop devices are supported for CI and local testing.
Fail fast if the filtered list is empty rather than degenerating back
to a full scan.

Note this hardens only the rename-fallback path. The deadlock observed
in CI (kernel hung tasks under 'ceph-volume raw activate' opening a
stalled /dev/rbd0; actions runs 27325963851, 27333888237, and
27378983111, which reproduced it on this PR's own head) ultimately
originates inside ceph-volume: 'raw activate' runs an unconditional
LUKS2/TPM2 discovery sweep over all block devices even when given an
explicit --device. That requires a ceph-volume fix; this change makes
the Rook-controlled scan safe regardless of ceph version.

Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
This commit is contained in:
Joshua Hoblitt
2026-06-22 10:00:58 -07:00
parent 0f5d8ba37f
commit da07f0c476
1 changed files with 12 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
pkg/operator/ceph/cluster/osd/spec.go
+12 -1
View File
@@ -205,7 +205,18 @@ sys.exit('no disk found with OSD ID $OSD_ID')
cat "$OSD_LIST"
if ! find_device < "$OSD_LIST"; then
ceph-volume raw list > "$OSD_LIST"
# The disk may have been renamed, so scan disks to find the right one. Build the
# scan list explicitly instead of letting a bare 'ceph-volume raw list' scan every
# block device: opening an rbd device can block in uninterruptible sleep while this
# OSD is down (the same deadlock the lvm filter above prevents for lvm-mode OSDs).
# Skip rbd, nbd, and drbd (network-backed devices that can hang when their backing
# storage is unavailable) and zram (volatile RAM); Rook never provisions OSDs on any
# of these. loop devices are intentionally kept, since OSDs on loop devices are
# supported for CI and local testing.
SCAN_DEVICES="$(lsblk --noheadings --paths --list --output NAME,TYPE | awk '$2 == "disk" || $2 == "part" {print $1}' | grep -vE '^/dev/(rbd|nbd|zram|drbd)' || true)"
[[ -z "$SCAN_DEVICES" ]] && { echo "no devices to scan for OSD $OSD_ID" ; exit 1 ; }
# shellcheck disable=SC2086 # word splitting of the device list is intended
ceph-volume raw list $SCAN_DEVICES > "$OSD_LIST"
cat "$OSD_LIST"
DEVICE="$(find_device < "$OSD_LIST")"