Commit Graph
14185 Commits
Author SHA1 Message Date
Artem Torubarov a0d215b428 osd: implement osd replacement health goroutine
A state machine for osd destroy phase of osd-replacement flow.
The state machine is implemented in OSD health goroutine, where each
step is handled on a separate tick. State machine is responsible
for draining, destroying OSD, and reserving its CRUSH position,
downscalling its deployment, and adding "readyForSwap" annotation.
Destroy phase does not zap the device on purpose. On disk signature
is kept to avoid automatic reprovisioning of destroyed OSD and to
detect physical swap.

Signed-off-by: Artem Torubarov <artem.torubarov@sap.com>
2026-07-27 18:42:07 +02:00
Artem Torubarov 514bd21552 osd: implement osd replacement controller
adds support of osd-replacement flow to cluster controller:
- predicate for osd deployment annotation
- validation of osd replacement annotation
- osd deployment recreation logic for replaced OSD

Signed-off-by: Artem Torubarov <artem.torubarov@sap.com>
2026-07-27 18:42:07 +02:00
Artem Torubarov c4727a0b21 osd: implement osd replacement crypto close job
defines a new entry point "close-encrypted-devices" in osd job.
The job is used by osd-replacement flow to close dm-crypt mappings
on host for destroyed encrypted OSDs. The job is owned by OSD
health goroutine implementing destroy phase of osd-replacement.

Signed-off-by: Artem Torubarov <artem.torubarov@sap.com>
2026-07-27 18:42:07 +02:00
Artem Torubarov 887d533001 osd: implement osd replacement prepare-job
contains osd prepare job edits to support osd replacement flow:
- removes destroyed OSD IDs from node status CM to stop cluster
  controller from reprovisioning destroyed OSD
- adds provisioning logic for OSD replacement to match available
  blank devices to destroyed OSD slots

Signed-off-by: Artem Torubarov <artem.torubarov@sap.com>
2026-07-27 18:42:07 +02:00
dependabot[bot] 9a7005a870 build(deps): bump go.yaml.in/yaml/v3 from 3.0.4 to 3.0.5
Bumps [go.yaml.in/yaml/v3](https://github.com/yaml/go-yaml) from 3.0.4 to 3.0.5.
- [Commits](https://github.com/yaml/go-yaml/compare/v3.0.4...v3.0.5)

---
updated-dependencies:
- dependency-name: go.yaml.in/yaml/v3
  dependency-version: 3.0.5
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-07-27 12:50:37 +00:00
dependabot[bot] 03d5028ef9 build(deps): bump the github-dependencies group with 6 updates
Bumps the github-dependencies group with 6 updates:

| Package | From | To |
| --- | --- | --- |
| [github.com/aws/aws-sdk-go-v2](https://github.com/aws/aws-sdk-go-v2) | `1.42.1` | `1.43.0` |
| [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) | `1.32.30` | `1.32.31` |
| [github.com/aws/aws-sdk-go-v2/credentials](https://github.com/aws/aws-sdk-go-v2) | `1.19.29` | `1.19.30` |
| [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2) | `1.105.2` | `1.106.0` |
| [github.com/aws/aws-sdk-go-v2/service/sns](https://github.com/aws/aws-sdk-go-v2) | `1.41.1` | `1.42.0` |
| [github.com/go-logr/logr](https://github.com/go-logr/logr) | `1.4.3` | `1.4.4` |


Updates `github.com/aws/aws-sdk-go-v2` from 1.42.1 to 1.43.0
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/v1.42.1...v1.43.0)

Updates `github.com/aws/aws-sdk-go-v2/config` from 1.32.30 to 1.32.31
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/config/v1.32.30...config/v1.32.31)

Updates `github.com/aws/aws-sdk-go-v2/credentials` from 1.19.29 to 1.19.30
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/credentials/v1.19.29...credentials/v1.19.30)

Updates `github.com/aws/aws-sdk-go-v2/service/s3` from 1.105.2 to 1.106.0
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/s3/v1.105.2...service/s3/v1.106.0)

Updates `github.com/aws/aws-sdk-go-v2/service/sns` from 1.41.1 to 1.42.0
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/v1.41.1...v1.42.0)

Updates `github.com/go-logr/logr` from 1.4.3 to 1.4.4
- [Release notes](https://github.com/go-logr/logr/releases)
- [Changelog](https://github.com/go-logr/logr/blob/master/CHANGELOG.md)
- [Commits](https://github.com/go-logr/logr/compare/v1.4.3...v1.4.4)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2
  dependency-version: 1.43.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: github-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/config
  dependency-version: 1.32.31
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: github-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/credentials
  dependency-version: 1.19.30
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: github-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/service/s3
  dependency-version: 1.106.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: github-dependencies
- dependency-name: github.com/aws/aws-sdk-go-v2/service/sns
  dependency-version: 1.42.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: github-dependencies
- dependency-name: github.com/go-logr/logr
  dependency-version: 1.4.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: github-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-07-27 12:50:27 +00:00
dependabot[bot] cfe5743b3b build(deps): bump github/codeql-action/upload-sarif
Bumps [github/codeql-action/upload-sarif](https://github.com/github/codeql-action) from 4.37.1 to 4.37.3.
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/7188fc363630916deb702c7fdcf4e481b751f97a...e4fba868fa4b1b91e1fdab776edc8cfbe6e9fb81)

---
updated-dependencies:
- dependency-name: github/codeql-action/upload-sarif
  dependency-version: 4.37.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-07-27 12:43:27 +00:00
dependabot[bot] a493aa983e build(deps): bump ossf/scorecard-action from 2.4.3 to 2.4.4
Bumps [ossf/scorecard-action](https://github.com/ossf/scorecard-action) from 2.4.3 to 2.4.4.
- [Release notes](https://github.com/ossf/scorecard-action/releases)
- [Changelog](https://github.com/ossf/scorecard-action/blob/main/RELEASE.md)
- [Commits](https://github.com/ossf/scorecard-action/compare/4eaacf0543bb3f2c246792bd56e8cdeffafb205a...2d1146689b8cda280b9bc96326124645441f03bc)

---
updated-dependencies:
- dependency-name: ossf/scorecard-action
  dependency-version: 2.4.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-07-27 12:43:19 +00:00
dependabot[bot] e85c7144a4 build(deps): bump aws-actions/configure-aws-credentials
Bumps [aws-actions/configure-aws-credentials](https://github.com/aws-actions/configure-aws-credentials) from 6.2.2 to 6.2.3.
- [Release notes](https://github.com/aws-actions/configure-aws-credentials/releases)
- [Changelog](https://github.com/aws-actions/configure-aws-credentials/blob/main/CHANGELOG.md)
- [Commits](https://github.com/aws-actions/configure-aws-credentials/compare/517a711dbcd0e402f90c77e7e2f81e849156e31d...e6de054238d6b7531b4efff3b6587d9aade6a06c)

---
updated-dependencies:
- dependency-name: aws-actions/configure-aws-credentials
  dependency-version: 6.2.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-07-27 12:43:09 +00:00
dependabot[bot] 013cca80bc build(deps): bump docker/login-action from 4.4.0 to 4.5.1
Bumps [docker/login-action](https://github.com/docker/login-action) from 4.4.0 to 4.5.1.
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](https://github.com/docker/login-action/compare/af1e73f918a031802d376d3c8bbc3fe56130a9b0...abd2ef45e78c5afb21d64d4ca52ee8550d9572c7)

---
updated-dependencies:
- dependency-name: docker/login-action
  dependency-version: 4.5.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-07-27 12:42:44 +00:00
raaizikandBlaine Gardner a400ae6fd5 doc: document CephNFS custom port and exposure examples
Document spec.server.port for host-network port conflicts, and describe
how user-managed LoadBalancer and NodePort Services must align port and
targetPort with the CR.

Signed-off-by: raaizik <raaizik@yahoo.com>
Co-Authored-By: Blaine Gardner <b.blaine.gardner@gmail.com>
2026-07-27 13:54:57 +03:00
raaizik b97296c47c nfs: add unit tests for custom NFS port
Add coverage for GetPort defaults and custom values, Service and probe
port wiring, Ganesha NFS_Port config, and create-or-update when
spec.server.port changes on an existing Service.

Signed-off-by: raaizik <raaizik@yahoo.com>
2026-07-27 13:54:57 +03:00
raaizik 5e36fa4bc3 nfs: support custom NFS server port
When NFS runs with host networking, port 2049 may already be in use.
Add CephNFS spec.server.port (default 2049), wire it through Ganesha
config, the operator Service, and probes, and regenerate CRDs.

Signed-off-by: raaizik <raaizik@yahoo.com>
2026-07-27 13:54:57 +03:00
Joshua Hoblitt faf2bffe5a helm: add toolbox repository and tag settings
The operator image in the rook-ceph chart and the Ceph image in the
rook-ceph-cluster chart are both configured as a repository and tag
pair. The toolbox was the exception, taking a single toolbox.image
string.

Add toolbox.repository and toolbox.tag, so the toolbox image can be
configured the same way as the others. Either setting may be applied on
its own; whichever is left unset falls back to cephImage.repository or
cephImage.tag.

The new settings take precedence over toolbox.image, which continues to
apply when neither is set, so existing values files are unaffected.

Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
2026-07-24 15:16:35 -07:00
Travis Nielsen 5b06d241fb Merge pull request #18041 from OdedViner/set_pool_application_nvmeof
nvmeof: set nvmeof-meta application tag on .nvmeof pool
2026-07-24 10:01:17 -06:00
Oded Viner f095a530c8 nvmeof: set nvmeof-meta application tag on .nvmeof pool
the .nvmeof pool was incorrectly tagged with the rbd
application and initialized as an rbd pool. add .nvmeof
to the built-in pool switch so it gets the nvmeof-meta
application tag instead, and skip rbd pool init.

Signed-off-by: Oded Viner <oviner@redhat.com>
2026-07-24 18:11:37 +03:00
Joshua Hoblitt 6fec883e62 Merge pull request #18013 from jhoblitt/ss-17966
osd: skip upgrades for OSDs pending migration when the last migration is incomplete
2026-07-23 07:52:49 -07:00
Travis Nielsen 7dbfbbc2f9 Merge pull request #18035 from pasanchamikara/ci-canary-arm64-add-v19-v20
Ci canary arm64 add v19 v20
2026-07-23 08:19:24 -06:00
Pasan Chamikara b94e535215 ci: add ceph v19 and v20 images to canary-arm64 test matrix
Signed-off-by: Pasan Chamikara <pasan.chamikara@owasp.org>
2026-07-23 18:28:00 +05:30
Praveen MandClaude Opus 4.6 b2f2288032 csi: add v1 VolumeGroupSnapshot examples for RBD and NFS
Add VolumeGroupSnapshotClass and VolumeGroupSnapshot example manifests
for RBD and NFS, similar to the existing CephFS examples.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Praveen M <m.praveen@ibm.com>
2026-07-23 16:33:08 +05:30
Joshua Hoblitt 6bf24ac507 test: extract the zone.json pool canary into its own package
The canary that checks every RGW zone.json *_pool field is covered by
Rook's zonePoolNSSuffix map was inline in runObjectE2ETest, running
against the legacy per-pass store before the shared store existed.

Move it into tests/integration/object/zonepools as a standalone
shared-store consumer. It now validates the shared store's zone, which
carries the real shared-pool placements the canary is meant to guard,
and runs first among the shared-store packages so it still sees a fresh
zone.

Add a Sharedstore.Installer() accessor so the packaged canary can run
radosgw-admin inside the cluster while keeping the uniform
(t, k8sh, store) package entry signature.

Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
2026-07-22 16:04:42 -07:00
Joshua Hoblitt 012d69a076 Merge pull request #17894 from jhoblitt/convert-bucket-lifecycle-to-sharedstore
test: convert the object bucket lifecycle suite to the shared store
2026-07-22 12:55:11 -07:00
Travis Nielsen 9a52f468fc Merge pull request #17991 from chimanjain/migrate-yaml
build(deps): Migrate YAML library from gopkg.in/yaml.v2 to go.yaml.in/yaml/v3
2026-07-22 09:07:04 -06:00
Travis Nielsen 955f7c5428 Merge pull request #17785 from beyondcloud-co/fix/17761-duplicate-mgr-ips-in-endpointslice
fix(ceph): deduplicate external MGR endpoints in EndpointSlice
2026-07-22 09:05:31 -06:00
Chiman Jain 2034be1f37 build: migrate gopkg.in/yaml to go.yaml.in/yaml
Signed-off-by: Chiman Jain <chimanjain15@gmail.com>
2026-07-22 20:05:39 +05:30
Parth Arora 1bb5bdfb55 Merge pull request #17971 from parth-gr/watch-rns-svg
external: add watcher to the rns controler for watching csi and clien…
2026-07-22 19:57:09 +05:30
parth-gr 7f602f12b0 external: add watcher to the rns controler for watching csi and client profile
if the csi secret is updated or a client profile is updated
we need a re-reconile of rns
So added the watcher for it

Signed-off-by: parth-gr <partharora1010@gmail.com>
2026-07-22 19:40:12 +05:30
Parth Arora 65af8e4f1b Merge pull request #17960 from parth-gr/external-update
external: import script should create mon secret before rns
2026-07-22 13:37:57 +05:30
Travis Nielsen c74766a86a Merge pull request #18024 from travisn/go-mod-text
core: Update the go text module
2026-07-21 15:43:08 -06:00
Travis Nielsen 8c397f6381 core: update the go text module
The go text package v0.37 has a vulnerability, thus update
to the latest version of the package.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2026-07-21 15:18:00 -06:00
Anas Khan 0bf6f84ff5 object: return an error when the multisite zone is not found
retrieveMultisiteZone is meant to gate the object-store reconcile on the
backing Ceph zone existing: it runs "radosgw-admin zone get" and, when
that fails, returns a non-nil error so the caller requeues. That gate is
dead code. The ENOENT check declares an inner err from exec.ExtractExitCode
that shadows the outer command error, and ExtractExitCode returns a nil
error for the ordinary exit failures radosgw-admin produces. Both the
ENOENT branch and the else branch then wrap that shadowed nil, and
errors.Wrapf(nil, ...) is nil, so the function returns
(waitForRequeueIfObjectStoreNotReady, nil). The caller only propagates the
requeue when the error is non-nil, so the requeue is dropped and reconcile
runs on.

The result is that a failed "zone get" no longer backs off. Reconcile
proceeds to stand up the object store anyway -- the RGW service, the
admin-ops endpoint, the deployment, and the pools radosgw scaffolds as it
comes up -- for a multisite store whose backing zone does not exist. This
is not the "normal multisite bootstrap" transient the original wording
suggested. getMultisiteResourceNames runs immediately before this and
already requeues until the CephObjectZone CR reports Ready, and the zone
controller marks it Ready only after it has created the Ceph zone, so a
healthy bootstrap never reaches this gate with a missing zone. That
CR-Ready gate, not this one, is what actually blocks bootstrap.

Where the dead gate does bite is the cases the CR status cannot cover:

- the Ceph zone deleted or renamed out of band while the CR still reads Ready
- a zone controller that reports Ready without leaving a usable zone behind
- any non-ENOENT "zone get" failure, e.g. a permission or connectivity
  error, which the else branch swallows the same way

The check was correct when fc579f4520 introduced it in 2020 with
exec.ExitStatus, which returns (code, ok) and leaves err unshadowed.
bd58790c31 ("ceph: proxy ceph commands when multus is configured", 2021)
swapped it to exec.ExtractExitCode as an unrelated drive-by, inverting the
contract and killing the gate. The sibling realm, zonegroup, and zone
controllers were left untouched and still use exec.ExitStatus today.

Restore exec.ExitStatus so the outer error is no longer shadowed, both
branches wrap the real command error, and the caller requeues until the
zone exists -- matching the sibling controllers.

Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
2026-07-21 12:40:25 -07:00
Travis Nielsen c4a8bfe955 Merge pull request #17989 from jhoblitt/helm-mk-dep-archive-race
build: exclude helm dep archives from chart package prerequisites
2026-07-21 12:45:28 -06:00
Travis Nielsen ce05eff9e5 Merge pull request #17947 from OdedViner/fix_nvmeof_ci
nvmeof: use .nvmeof pool for gateway and remove pool field from CRD
2026-07-21 12:12:09 -06:00
Travis Nielsen 51ca1a46f5 Merge pull request #17974 from pasanchamikara/ci-canary-arm64-simplify
ci: Simplify canary-arm64 image selection to a matrix variable
2026-07-21 11:33:44 -06:00
Travis Nielsen aa113bcec9 Merge pull request #18014 from subhamkrai/add-ceph-v21-daily-test
ci: adding ceph v21 to the daily ci test image list
2026-07-21 11:28:00 -06:00
Oded Viner ec502c3f80 nvmeof: use .nvmeof pool for gateway and remove pool field from CRD
Changes:
- Use a dedicated .nvmeof pool (via CephBlockPool CR named
  builtin-nvmeof) for the NVMe-oF gateway internal state.
- Use a separate nvmeof pool for the StorageClass data.
- Remove the pool field from the CephNVMeOFGateway CRD.
  The gateway now always uses the .nvmeof pool, hardcoded
  in the operator.
- Add .nvmeof to the allowed CephBlockPool name overrides.
- Create a production example nvmeof.yaml (instances: 2,
  replicas: 3) and a CI-only nvmeof-test.yaml (instances: 1,
  replicas: 1).
- Update documentation and CI test script accordingly.

Signed-off-by: Oded Viner <oviner@redhat.com>
2026-07-21 20:19:44 +03:00
Joshua Hoblitt 9312bbb350 Merge pull request #17980 from anxkhn/patch-10
core: parse subvolumegroup bytes_quota reported as "infinite"
2026-07-21 09:26:28 -07:00
Joshua Hoblitt a9bd575002 Merge pull request #17968 from anxkhn/patch-21
mon: fix data race on failedMonSchedule during scheduling
2026-07-21 09:24:24 -07:00
Anas Khan 7630d21e85 core: fix data race reading exec output on timeout
executeCommandWithTimeout wires a single bytes.Buffer to both cmd.Stdout
and cmd.Stderr and joins the command in a goroutine via cmd.Wait(). On
the timeout kill path it called cmd.Process.Kill() and then read that
buffer without waiting for cmd.Wait() to return. Kill() only signals the
process; it does not wait for os/exec's output-copier goroutines (joined
only by cmd.Wait) to finish, so reading the buffer there races with those
writers, which the race detector flags.

Only read the buffer once the copier goroutines have drained, signaled by
cmd.Wait() on the done channel. Because a killed process can leave an
orphaned descendant holding the output pipe open -- a D-state
cryptsetup/dmsetup child, or the downstream of a "sh -c '... | head'"
pipeline -- cmd.Wait() may never return, so bound the drain with a short
grace period and give up on the captured output rather than blocking the
caller forever. This keeps the bounded return the timeout path is meant
to guarantee. On a failed Kill() the process may still be writing, so
return without reading the buffer.

Add a regression test that drives the kill path with a child that ignores
SIGINT while writing to stdout; it fails under -race before this change.

Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
2026-07-21 09:01:12 -07:00
Joshua Hoblitt 0e21738451 Merge pull request #18011 from jhoblitt/ss-17967
osd: archive crash on OSD removal
2026-07-21 08:39:35 -07:00
subhamkrai 9caf0ee5c9 ci: adding ceph v21 to the daily ci test image list
ceph version v21 was release couple of days back, this
commit add the version to daily ci test list.

Also, updated the go.work

Signed-off-by: subhamkrai <srai@redhat.com>
2026-07-21 20:44:38 +05:30
parth-gr ce3a2c77ae external: update the csi secrets ordering
create the csi secret before creating the mon cm
as the cluster controller will look for the csi
secrets for creating client profile

Signed-off-by: parth-gr <partharora1010@gmail.com>
2026-07-21 19:52:38 +05:30
parth-gr 56153b344f external: import script should create mon secret before rns
the order of creating the svg and rns is important,
the svg and rns controller is dependent on the ceph health,
so first we should create the mon secret to get the cephcluster `health ok`

Signed-off-by: parth-gr <partharora1010@gmail.com>
2026-07-21 19:52:38 +05:30
Parth Arora d721d3c3a2 Merge pull request #18015 from OdedViner/fix_go_work
ci: update go.work to require go 1.26.4
2026-07-21 19:51:45 +05:30
Oded Viner 38fd1a2ca9 ci: update go.work to require go 1.26.4
The go.work file specified go 1.26.0, but both go.mod files
require go 1.26.4. This mismatch causes `go list` to fail
silently, breaking code generation and other tooling.

Signed-off-by: Oded Viner <oviner@redhat.com>
2026-07-21 16:11:54 +03:00
subham rai f5052c522c Merge pull request #18001 from rook/dependabot/go_modules/github-dependencies-6d2e00e983
build(deps): bump the github-dependencies group across 1 directory with 10 updates
2026-07-21 15:24:02 +05:30
Pasan 18657e9775 ci: broaden canary-arm64 matrix to cover main and release branches
Add the arm64 devel/release images built by ceph-ci (main-arm64,
squid-release-arm64, tentacle-arm64, umbrella-release-arm64) to the
canary-arm64 matrix, so the job exercises multiple Ceph versions on
arm64, matching the ceph-suite-tests/canary-tests coverage pattern.

Resolves: #17625

Signed-off-by: Pasan <pasan.chamikara@owasp.org>
2026-07-21 10:59:18 +05:30
Joshua Hoblitt fc4db1a914 Merge pull request #18009 from jhoblitt/ss-17981
object: return an error when reconciled rgw user has no keys
2026-07-20 20:42:21 -07:00
Anas Khan 439b8c84d9 osd: skip upgrades for OSDs pending migration when the last migration is incomplete
startOSDMigration guards against upgrading OSDs while an OSD migration is
still in progress by returning the set of OSDs pending migration, which the
caller then removes from the update queue.

When the last migrated OSD's deployment has not been recreated yet,
isLastOSDMigrationComplete returns (false, nil). startOSDMigration then ran
return nil, errors.Wrapf(err, ...) with err already nil, and
errors.Wrapf(nil, ...) returns nil, so it returned (nil, nil), the same
value it uses to signal that no migration was requested. The caller
therefore skipped removing the pending OSDs from the update queue and could
upgrade and restart them mid-migration, which the pending-migration guard
was meant to prevent.

Return the freshly computed migration config in this case so the caller
still prunes the pending OSDs, without starting a new migration and without
aborting the reconcile. Aborting here would wedge the cluster because the
code that recreates the interrupted OSD runs later in Start, so an early
error return would leave that OSD deleted and block every later reconcile.
Letting the reconcile continue recreates the interrupted OSD and lets the
migration resume on a subsequent reconcile.

Also construct a non-nil error for the sibling unhealthy-PGs guard, which
wrapped a nil err in the same way.

Add a regression test asserting that startOSDMigration returns the pending
migration set, and does not delete another OSD deployment, while a previous
migration is still in progress.

Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
2026-07-20 17:51:30 -07:00
Anas Khan 1630ad1049 osd: archive crash on OSD removal
archiveCrash is meant to silence the RECENT_CRASH health warning for a
removed OSD by archiving its crash, but the early-return guard was
inverted. GetCrashList unmarshals `ceph crash ls` into a non-nil slice on
every success, so `if crash != nil` was always true. The function always
logged "no ceph crash to silence" and the archive loop was unreachable, so
OSD removal never archived the crash and the warning lingered. Guard on
`len(crash) == 0` so only a genuinely empty list short-circuits.

Reviving the loop surfaced two latent bugs it had always masked. On a
non-empty list with no entry for the removed OSD the crash id stayed empty
and the code ran `ceph crash archive ""`, which the mgr rejects with EINVAL
and logs as a spurious error during routine OSD removal, so skip the
archive when no crash matches. The loop also broke on the first match, but
`ceph crash ls` is oldest-first and includes already-archived entries, so
an OSD with several crashes had only its oldest one archived while
RECENT_CRASH persisted; archive every matching entry instead of just the
first.

Extend TestArchiveCrash to cover all four cases: a single matching crash, a
list with several matching crashes that are all archived, an empty list,
and a non-empty list with no matching entry where nothing is archived.

Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
2026-07-20 17:45:02 -07:00