Commit Graph
683 Commits
Author SHA1 Message Date
Artem Torubarov c9b1bfa805 osd: use a separate annotation to fence osd replacement instead of existing label
osd-replacement was relying on exising ceph.rook.io/do-not-reconcile
label for fencing osd destroy process owned by osd health goroutine from
controller. However, this label can be used but other components like
rook krew maintenance plugin and cannot be owned by osd-replacement
process. Added a separate osd.rook.io/replace-in-progress annotation for
that purpose.

Signed-off-by: Artem Torubarov <artem.torubarov@sap.com>
2026-07-29 11:03:29 +02:00
Artem Torubarov a0d215b428 osd: implement osd replacement health goroutine
A state machine for osd destroy phase of osd-replacement flow.
The state machine is implemented in OSD health goroutine, where each
step is handled on a separate tick. State machine is responsible
for draining, destroying OSD, and reserving its CRUSH position,
downscalling its deployment, and adding "readyForSwap" annotation.
Destroy phase does not zap the device on purpose. On disk signature
is kept to avoid automatic reprovisioning of destroyed OSD and to
detect physical swap.

Signed-off-by: Artem Torubarov <artem.torubarov@sap.com>
2026-07-27 18:42:07 +02:00
Artem Torubarov 514bd21552 osd: implement osd replacement controller
adds support of osd-replacement flow to cluster controller:
- predicate for osd deployment annotation
- validation of osd replacement annotation
- osd deployment recreation logic for replaced OSD

Signed-off-by: Artem Torubarov <artem.torubarov@sap.com>
2026-07-27 18:42:07 +02:00
Artem Torubarov c4727a0b21 osd: implement osd replacement crypto close job
defines a new entry point "close-encrypted-devices" in osd job.
The job is used by osd-replacement flow to close dm-crypt mappings
on host for destroyed encrypted OSDs. The job is owned by OSD
health goroutine implementing destroy phase of osd-replacement.

Signed-off-by: Artem Torubarov <artem.torubarov@sap.com>
2026-07-27 18:42:07 +02:00
Anas Khan 439b8c84d9 osd: skip upgrades for OSDs pending migration when the last migration is incomplete
startOSDMigration guards against upgrading OSDs while an OSD migration is
still in progress by returning the set of OSDs pending migration, which the
caller then removes from the update queue.

When the last migrated OSD's deployment has not been recreated yet,
isLastOSDMigrationComplete returns (false, nil). startOSDMigration then ran
return nil, errors.Wrapf(err, ...) with err already nil, and
errors.Wrapf(nil, ...) returns nil, so it returned (nil, nil), the same
value it uses to signal that no migration was requested. The caller
therefore skipped removing the pending OSDs from the update queue and could
upgrade and restart them mid-migration, which the pending-migration guard
was meant to prevent.

Return the freshly computed migration config in this case so the caller
still prunes the pending OSDs, without starting a new migration and without
aborting the reconcile. Aborting here would wedge the cluster because the
code that recreates the interrupted OSD runs later in Start, so an early
error return would leave that OSD deleted and block every later reconcile.
Letting the reconcile continue recreates the interrupted OSD and lets the
migration resume on a subsequent reconcile.

Also construct a non-nil error for the sibling unhealthy-PGs guard, which
wrapped a nil err in the same way.

Add a regression test asserting that startOSDMigration returns the pending
migration set, and does not delete another OSD deployment, while a previous
migration is still in progress.

Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
2026-07-20 17:51:30 -07:00
Artem Torubarov 75fe8db848 osd: rename internal OSD migration field to migrateOSD
Signed-off-by: Artem Torubarov <artem.torubarov@sap.com>
2026-07-10 11:45:47 +02:00
blackflytech c65663c529 core: use slices.Backward to simplify the code
Signed-off-by: blackflytech <blackflytech@outlook.com>
2026-07-01 00:37:50 +08:00
Joshua Hoblitt e5c6d75a73 core: narrow bare //nolint directives to specific linters
The //nolint directives at these sites omit the linter name, so each
suppresses every linter on its line rather than the one check it needs.
That hides any unrelated errcheck/gosec/govet finding later introduced
on the same line. Name the specific linter for each:

- staticcheck for the two operator sites: SA4004 (the intentional
  single-iteration loop in the OSD PVC host lookup) and SA1019 (the
  deliberate read of the deprecated S3.Enabled field in the RGW
  API-enable builder).
- errcheck for the rbd-mirror deferred token-file cleanup and the
  test-framework logging helpers (WriteString / writeHeader).

No behavior change.

Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
2026-06-29 16:32:59 -07:00
Joshua Hoblitt 49612461a4 docs: fix function comments to match their declaration names
Several godoc comments led with a stale or incorrect identifier, left
over from renames, exported/unexported changes, copy-paste between
sibling declarations, or plain typos. As a result the documented name no
longer matched the function, method, type, or var it describes. Correct
each leading word to the name of the declaration it documents.

Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
2026-06-26 12:52:39 -07:00
daixihegu 063dcad054 core: use slices.Contains to simplify code
Signed-off-by: daixihegu <daixihegu@163.com>
2026-06-25 10:17:53 +08:00
Satoru Takeuchi a8398f9996 Merge pull request #17724 from jhoblitt/osd-activate-skip-rbd-scan
osd: do not scan rbd devices in raw activate fallback
2026-06-23 07:44:02 +09:00
Joshua Hoblitt da07f0c476 osd: do not scan rbd devices in raw activate fallback
When the OSD activate init container cannot match its configured
device, it falls back to a bare 'ceph-volume raw list', which
enumerates every block device on the node and opens each one to read
bluestore labels. Opening a krbd device whose backing cluster is
unhealthy can block in uninterruptible sleep; when that device's I/O
depends on the OSD being activated, activation deadlocks.

Build the fallback scan list explicitly with lsblk and exclude the
network-backed device types that can hang when their backing storage
is unavailable (rbd, nbd, drbd) along with zram (volatile RAM). Rook
never provisions OSDs on any of these, so the fallback can never
legitimately need to find an OSD on them. loop devices are kept,
since OSDs on loop devices are supported for CI and local testing.
Fail fast if the filtered list is empty rather than degenerating back
to a full scan.

Note this hardens only the rename-fallback path. The deadlock observed
in CI (kernel hung tasks under 'ceph-volume raw activate' opening a
stalled /dev/rbd0; actions runs 27325963851, 27333888237, and
27378983111, which reproduced it on this PR's own head) ultimately
originates inside ceph-volume: 'raw activate' runs an unconditional
LUKS2/TPM2 discovery sweep over all block devices even when given an
explicit --device. That requires a ceph-volume fix; this change makes
the Rook-controlled scan safe regardless of ceph version.

Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
2026-06-22 10:00:58 -07:00
Travis Nielsen 63eed4ed47 Merge pull request #17279 from sophotechlabs/test/osd-config-store
test: add unit tests for OSD store config parsing
2026-06-19 08:12:00 -06:00
Joshua Hoblitt 41c81a281c Merge pull request #17684 from jhoblitt/maint-remove-dead-osd-updatequeue-clear
osd: remove unused updateQueue.Clear method
2026-06-10 14:20:04 -07:00
Travis Nielsen a9235090f4 Merge pull request #17649 from yuxiang-he/node-label-device-class
osd: Support using node labels for OSD device class assignment
2026-06-08 16:07:07 -06:00
Joshua Hoblitt 50c95d343e osd: remove unused updateQueue.Clear method
The Clear method on updateQueue had no callers anywhere in the repo
(verified with grep, deadcode, and staticcheck). The queue is emptied via
Remove/Pop in practice. Remove the dead method.

Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
2026-06-08 15:01:15 -07:00
Yuxiang He d9357dbfdc osd: support using node labels for OSD device class assignment
Add support for the osd.rook.io/device-class node label to assign
CRUSH device classes to all OSDs on a node.
The operator reads the label during OSD provisioning and
deployment reconciliation. If CR-level deviceClass conflicts
with the node label, the operator will skip reconciling the
node and logs an error.

Resolves: #17131
Signed-off-by: Yuxiang He <yuxiang@dropbox.com>
2026-06-08 13:21:05 -07:00
Sunnatillo f6fa25e844 osd: set require-osd-release from the health monitor
applyUpgradeOSDFunctionality() may run before all OSDs have registered
their new version, leaving require-osd-release unset and the cluster in
HEALTH_WARN.

Check for a single converged OSD version on each health-monitor tick and
set require-osd-release when it changes. The release name is cached to
avoid redundant ceph commands

Signed-off-by: Sunnatillo <sunnat.samadov@est.tech>
2026-06-04 16:54:36 +03:00
Artem Muterko 05da0d1294 test: add unit tests for OSD store config parsing
Signed-off-by: Artem Muterko <artem@sopho.tech>
2026-06-04 15:16:03 +02:00
Santosh 3e49c9f4cf osd: add cryptsetup resize for encrypted host based osds
when using encryptedDevice:true with host based (non pvc) osds, resizing
the underlying disk and restarting the OSD didn't expand the OSD because
the LUKS encryption layer was not resized.

This PR adds resize capabilities inside the activate container. Full
device stack must be resized: PV → LV → LUKS

Steps:
1. Run pvresize and lvextend to grow the LVM layers.
2. Retrieve the LUKS key from the Ceph config-key store using the
lockbox crednetials already available in the activate container
3. Pass the key explicitly to cryptsetup resize via --key-file.

Signed-off-by: Santosh <sapillai@redhat.com>
2026-05-29 12:09:17 +05:30
Blaine Gardner 645aa741eb Merge pull request #17530 from rook/maint-rm-unused-consts-opus-4.7
core: rm unused consts
2026-05-20 09:01:05 -06:00
Travis Nielsen 67395ad5e4 Merge pull request #17545 from sp98/fix-TestAddRemoveNode-unit-test
osd: fix flaky TestAddRemoveNode failure
2026-05-15 13:39:11 -06:00
Sunnatillo 0bd299aa60 build: fix gosec and go vet lint errors for Go 1.26
Signed-off-by: Sunnatillo <sunnat.samadov@est.tech>
2026-05-15 21:45:06 +03:00
Santosh 6781d2e58d osd: fix flaky TestAddRemoveNode failure
User RaceFreeFakeWatcher isntead of FakeWatcher in TestAddRemoveNode
function. The test runs c.Start() in a goroutine whch stops the watcher
when done. If mockNodeOrchestrationCompeltion sends a watch ever after,
that, FakeWatcher panics with "send on a closed channel".
RaceFreeFakeWatcher safely ignores the sends on a stopped channel.

Signed-off-by: Santosh <sapillai@redhat.com>
2026-05-15 17:44:03 +05:30
Travis Nielsen 3775bf206a Merge pull request #17407 from cobaltcore-dev/osd-deviceclass-minimal
osd: honor per-device deviceClass in raw-mode prepare and reconcile
2026-05-14 12:04:16 -06:00
Joshua Hoblitt 9eec9c730e core: rm unused consts
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
2026-05-14 09:34:52 -07:00
Artem Torubarov b41d63788d osd: honor per-device deviceClass in raw-mode prepare and reconcile
Fixes #16834 and #17384. Per-OSD CR deviceClass was ignored in
osd prepare job when reporting the osd config back to the operator,
and in the operator's reconcile override.
Fixed by resolving the per-device class from the CR devices list
in GetCephVolumeRawOSDs and in makeDeployment.

Signed-off-by: Artem Torubarov <artem.torubarov@sap.com>
2026-05-11 15:37:16 +02:00
Santosh fc9e3b54f3 osd: detect encryption label from dmcrypt path
Detect OSD encryption from the dmcrypt block path in `getOSDInfo` as a
fallback when the `encrypted` label is missing on pre-label deployments.

Include `osd.Encrypted` in the label check in `makeDeployment` so the
label reflects the actual OSD state on upgrade.

Fixes a bug where upgrading to a Rook version with the `encrypted` label
would set the label to `false` on already-encrypted OSDs, because the
label didn't exist yet and `getOSDInfo` defaulted to `false`.

Signed-off-by: Santosh <sapillai@redhat.com>
2026-04-29 19:18:05 +05:30
Santosh Pillai 59ce48ae88 Merge pull request #17228 from ormandj/fix/osd-device-class-reprovision
osd: fix CRUSH device class not applied during OSD re-discovery
2026-04-21 17:50:39 +05:30
Travis Nielsen a80ca49855 osd: add logging when detecting osd versions
When osd versions are being detected, log more info
about the versions to understand better if the
require-osd-version release is being set as
expected.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2026-04-07 13:13:54 -06:00
David Orman 1357d71177 osd: fix CRUSH device class not applied during OSD re-discovery
When an OSD is purged and reprovisioned, it comes back without the
correct CRUSH device class. NVMe devices configured with
deviceClass: nvme in the CR end up as ssd.

Root cause: the operator sets ROOK_OSD_CRUSH_DEVICE_CLASS on prepare
jobs but not on OSD deployments. During re-discovery
(GetCephVolumeRawOSDs), the daemon reads this env var, finds it
absent, and falls back to auto-detected deviceType which returns ssd
for NVMe.

Fix: add crushDeviceClassEnvVar(osd.DeviceClass) to the OSD
deployment spec in spec.go, alongside the existing
dataDeviceClassEnvVar. Existing deployments pick up the new env var
on operator reconciliation without manual intervention.

Fixes #17227
Signed-off-by: David Orman <ormandj@corenode.com>
2026-03-27 15:42:55 -05:00
Travis Nielsen 96f4320a34 test: increase osd reconcile wait timeout
The osd unit tests periodically fail with a timing issue
when canceling an orchestration. Attempt to stabilize
the test with a longer timeout before canceling the
test.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2026-03-04 15:43:32 -07:00
Blaine Gardner 72e03bed49 Merge pull request #17112 from BlaineEXE/update-osd-lockbox-key-rotation-init
osd: update lockbox key rotation for encrypted OSDs
2026-02-26 08:41:20 -07:00
Satoru Takeuchi dac71d9edd osd: set device-type label on update
We should set OSDInfo.DeviceType if "device-type" label is
set on the corresponding Deployment. Otherwise, Rook
deletes this label on Update.

Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
2026-02-25 08:29:55 +00:00
Blaine Gardner 931006fb91 osd: update lockbox key rotation for encrypted OSDs
Update `lockbox.keyring` rotation for encrypted OSDs to ensure the key
is not lost if the init container fails but continues assuming
(reasonably) that the key is likely still relevant.

Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
2026-02-24 13:52:22 -07:00
Blaine Gardner 50ff29140a Merge pull request #17052 from BlaineEXE/osd-cephx-key-init-fix
osd: in cephx key init, don't overwrite key on failure
2026-02-09 12:21:54 -07:00
Blaine Gardner d69ffdfd2c osd: in cephx key init, don't overwrite key on failure
Previously, if the OSD cephx-keyring-update init container couldn't
contact the mons, the output redirect would overwrite the on-disk
keyring file with no text, removing the on-disk keyring.

Modify the init container to only overwrite the keyring file when the
ceph command to get the latest key suceeds, to avoid removing the
on-disk key.

Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
2026-02-09 10:06:27 -07:00
Jan Breitkopf 1ff6f70311 operator: fix OSD info
This fixes OSD info being incomplete when a sidecar container is first
in the .spec.Template.Spec.Containers list.
I added a small helper function to find a container named "osd" instead
of always assuming the first one.

Signed-off-by: Jan Breitkopf <jan.breitkopf@prorocketeers.com>
2026-01-27 11:20:41 +01:00
dependabot[bot] 3c51e808f2 build(deps): bump the k8s-dependencies group with 6 updates
Bumps the k8s-dependencies group with 6 updates:

| Package | From | To |
| --- | --- | --- |
| [k8s.io/api](https://github.com/kubernetes/api) | `0.34.2` | `0.34.3` |
| [k8s.io/apiextensions-apiserver](https://github.com/kubernetes/apiextensions-apiserver) | `0.34.2` | `0.34.3` |
| [k8s.io/apimachinery](https://github.com/kubernetes/apimachinery) | `0.34.2` | `0.34.3` |
| [k8s.io/cli-runtime](https://github.com/kubernetes/cli-runtime) | `0.34.2` | `0.34.3` |
| [k8s.io/client-go](https://github.com/kubernetes/client-go) | `0.34.2` | `0.34.3` |
| [k8s.io/cloud-provider](https://github.com/kubernetes/cloud-provider) | `0.34.2` | `0.34.3` |

Updates `k8s.io/api` from 0.34.2 to 0.34.3
- [Commits](https://github.com/kubernetes/api/compare/v0.34.2...v0.34.3)

Updates `k8s.io/apiextensions-apiserver` from 0.34.2 to 0.34.3
- [Release notes](https://github.com/kubernetes/apiextensions-apiserver/releases)
- [Commits](https://github.com/kubernetes/apiextensions-apiserver/compare/v0.34.2...v0.34.3)

Updates `k8s.io/apimachinery` from 0.34.2 to 0.34.3
- [Commits](https://github.com/kubernetes/apimachinery/compare/v0.34.2...v0.34.3)

Updates `k8s.io/cli-runtime` from 0.34.2 to 0.34.3
- [Commits](https://github.com/kubernetes/cli-runtime/compare/v0.34.2...v0.34.3)

Updates `k8s.io/client-go` from 0.34.2 to 0.34.3
- [Changelog](https://github.com/kubernetes/client-go/blob/master/CHANGELOG.md)
- [Commits](https://github.com/kubernetes/client-go/compare/v0.34.2...v0.34.3)

Updates `k8s.io/cloud-provider` from 0.34.2 to 0.34.3
- [Commits](https://github.com/kubernetes/cloud-provider/compare/v0.34.2...v0.34.3)

---
updated-dependencies:
- dependency-name: k8s.io/api
  dependency-version: 0.34.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: k8s-dependencies
- dependency-name: k8s.io/apiextensions-apiserver
  dependency-version: 0.34.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: k8s-dependencies
- dependency-name: k8s.io/apimachinery
  dependency-version: 0.34.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: k8s-dependencies
- dependency-name: k8s.io/cli-runtime
  dependency-version: 0.34.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: k8s-dependencies
- dependency-name: k8s.io/client-go
  dependency-version: 0.34.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: k8s-dependencies
- dependency-name: k8s.io/cloud-provider
  dependency-version: 0.34.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: k8s-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-01-07 07:59:21 -07:00
subhamkrai 304cdc8c83 core: remove support for reef v18 in Rook v1.19
For Rook v1.19,removing support for Ceph Reef. It is at end of life.
Users on Reef can continue to use Rook v1.18.x or older.
The latest two Ceph versions Squid and Tentacle are supported.

Signed-off-by: subhamkrai <srai@redhat.com>
2025-12-04 13:11:35 +05:30
Travis Nielsen 93adc5dd6b core: include namespace more consistently in logging
When writing to the operator log, where possible
the namespace of the resource on which the operator
is working will be logged as a standard prefix
to more easily troubleshoot what resource is being reconciled

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2025-11-20 15:49:43 -07:00
Travis Nielsen c7dc2439f8 Merge pull request #16505 from elias-dbx/cleanup_preprovision_jobs
osd: cleanup orphaned osd provision jobs
2025-11-13 09:44:59 -07:00
Elias Carter 82a7edc268 osd: cleanup orphaned osd provision jobs
In a running cluster, I noticed that a node which was provisioning was removed from k8s
and the following log was emitted by the rook operator:
```
op-osd: cleaning up dangling OSD prepare status configmap "rook-ceph-osd-xxx-status"
```

But the osd provisioning job was still trying to run on the removed node:
```
rook-ceph-osd-prepare-xxx         Running    0/1           4d22h      4d22h
```

This PR implements a mechanism to remove any osd provision jobs which are bound to a
node which no longer exists within kubernetes. I added a test for this cleanup as
there wasn't a pre-existing test.

Signed-off-by: Elias Carter <elias@dropbox.com>
2025-11-12 12:35:49 -08:00
Travis Nielsen 23fd211544 Merge pull request #16675 from parth-gr/deviceclass-prepare
osd: add device class label to the osd prepare pods
2025-11-12 11:16:08 -07:00
subham rai ca440a6d1b Merge pull request #16679 from huyejia/master
osd: replace Split in loops with more efficient SplitSeq
2025-11-12 15:37:26 +05:30
huyejia a6239dedfe osd: replace Split in loops with more efficient SplitSeq
Signed-off-by: huyejia <huyejia@outlook.jp>
2025-11-12 16:52:12 +08:00
parth-gr 4fcdfaad1f osd: add device class label to the osd prepare pods
currently the device class label was added to the osd pod,
we need the device class label on prepare pod,
so to spread across prpeare pods evenly among the faiure domains
per device class

Signed-off-by: parth-gr <partharora1010@gmail.com>
2025-11-10 17:10:05 +05:30
Travis Nielsen d34eeee786 Merge pull request #16669 from YiteGu/use-sync-map-for-monitoringRoutines
core: use sync.Map for cluster.monitoringRoutines
2025-11-06 11:18:35 -07:00
Yite Gu bf6c295d02 core: use sync.Map for cluster.monitoringRoutines
`cluster.monitoringRoutines` stores three types of
daemon health: "mon", "osd", and "status". These are
used by three goroutines respectively, resulting in:
fatal error: concurrent map reads and map writes.

Signed-off-by: Yite Gu <guyite@bytedance.com>
2025-11-06 19:36:40 +08:00
Joshua Hoblitt 43f26534ed osd: add CephCluster.spec.storage.OSDMaxUpdatesInParallel
Configures the maximum number of OSDs to update in parallel.

Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
2025-11-03 12:10:28 -07:00