Commit Graph
420 Commits
Author SHA1 Message Date
Anas Khan 0bbd5de84c mon: fix data race on failedMonSchedule during scheduling
assignMons schedules each mon in its own goroutine and shares a
failedMonSchedule bool to record whether any of them failed. The
resultLock mutex already guards the shared c.mapping.Schedule map
write, but the three failedMonSchedule = true assignments were done
without holding the lock. When two or more mons fail to schedule at
the same time (for example when waitForMonitorScheduling errors, the
node choice is nil, or getNodeInfoFromNode fails) their goroutines
write the flag concurrently, which is a write-write data race.

Guard the failedMonSchedule writes with the existing resultLock, the
same lock the goroutines already use for the map update. The post-Wait
read is left as is since the WaitGroup orders it after every write.

Add a regression test that fails scheduling for multiple mons at once
and confirms assignMons returns an error; run it with -race to catch
the race.

Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>
2026-07-14 18:03:13 +05:30
Parth Arora 6ee3198716 Merge pull request #17689 from subhamkrai/update-tnf-script-hostname
mon: update how hostname was set for drbd
2026-07-09 21:07:54 +05:30
subhamkrai c49ef5f079 mon: update how hostname was set for drbd
currently, drbd hostname was ready using command
but now the change we use kubernetes built-in api
to read hostname from `spec.nodeName`.`

Signed-off-by: subhamkrai <srai@redhat.com>
2026-07-07 08:45:28 +05:30
Joshua Hoblitt 49612461a4 docs: fix function comments to match their declaration names
Several godoc comments led with a stale or incorrect identifier, left
over from renames, exported/unexported changes, copy-paste between
sibling declarations, or plain typos. As a result the documented name no
longer matched the function, method, type, or var it describes. Correct
each leading word to the name of the declaration it documents.

Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
2026-06-26 12:52:39 -07:00
pengqima 240c431f42 docs: fix function comment to match actual function name
Signed-off-by: pengqima <pengqima@outlook.com>
2026-06-26 23:43:22 +08:00
daixihegu 063dcad054 core: use slices.Contains to simplify code
Signed-off-by: daixihegu <daixihegu@163.com>
2026-06-25 10:17:53 +08:00
Parth Arora 451c3f3d1b Merge pull request #17805 from parth-gr/drbd-mons
core: reduce the floating mon reschdule time
2026-06-23 20:55:48 +05:30
parth-gr 9e0b6e5bbe core: reduce the floating mon reschdule time
currently k8s add the default eviction time as 300 sec
setting it to 5 sec for the quick re-schedule

Signed-off-by: parth-gr <partharora1010@gmail.com>
2026-06-23 20:15:56 +05:30
subhamkrai ad26ebdab0 build: update ceph version to latest v20.2.2
updating the ceph version from v20.2.1 to v20.2.2

Signed-off-by: subhamkrai <srai@redhat.com>
2026-06-17 22:00:11 +05:30
Joshua Hoblitt 8dd1778655 Merge pull request #17561 from jhoblitt/maint-ref-all-containers-by-name-instead-of-index
core: reference all containers in a spec by name instead of index
2026-05-26 12:10:09 -07:00
Joshua Hoblitt d059c351fc core: reference all containers in a spec by name instead of index
This changeset excludes converting _test.go code as there may be a
legitimate reasons (E.g. convenience) for unit test to be sensitive to
ordering.

Related to:
- https://github.com/rook/rook/issues/15291
- https://github.com/rook/rook/pull/16969

Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
2026-05-26 10:09:23 -07:00
Blaine Gardner 645aa741eb Merge pull request #17530 from rook/maint-rm-unused-consts-opus-4.7
core: rm unused consts
2026-05-20 09:01:05 -06:00
Travis Nielsen 52b959e22e Merge pull request #17421 from subhamkrai/add-floating-mon-container-resources
mon: add resource request for mon drbd containers
2026-05-14 11:15:25 -06:00
Joshua Hoblitt 9eec9c730e core: rm unused consts
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
2026-05-14 09:34:52 -07:00
Prabhala Tara Aasrita aeae61e5be mon: check node existence on every health check iteration
Previously, shouldFailoverMonImmediately() was called only when a mon first went out of quorum. If the node gets deleted after the timeout starts, it does not get detected until the timeout expires. Moved the call outside the timeout initialization block so it runs on every health check iteration and detects node deletion immediately.

Fixes #15953

Signed-off-by: Prabhala Tara Aasrita <taraasrita@ibm.com>
2026-05-06 20:04:05 +05:30
subhamkrai 4eefad42e8 csi: move csi management to admin
Going forward, admin will manage the csi operator
CR's and rook will only manage Ceph Connection cr
and client Profile cr.

The old csi driver is completely removed from Rook
and can no longer be used starting in Rook v1.20.

The upgrade guide will contain the needed transition steps
for managing the csi operator settings.

Signed-off-by: subhamkrai <srai@redhat.com>
2026-04-29 14:24:26 -06:00
subhamkrai d88e287726 mon: add resource request for mon drbd containers
adding missing resource request for mon drbd containers
log-collector and shut-down-app and update the unit test
to validate.

Signed-off-by: subhamkrai <srai@redhat.com>
2026-04-28 19:13:44 +05:30
Travis Nielsen 0160ba5bd0 Merge pull request #17288 from lumir-sliva/relax-stretch-crush-weight-check
mon: relax balanced CRUSH weight check for stretch mode
2026-04-23 12:05:12 -06:00
Lumir Sliva 27f95f2feb mon: relax balanced CRUSH weight check for stretch mode
Ceph relaxed the stretch mode CRUSH weight check in ceph/ceph#66580 to
allow weights within 10 percent of each other. This change is available
in Ceph Umbrella and was backported to tentacle v20.2.2 via ceph/ceph
number 67790. The squid backport is still pending.

Signed-off-by: Lumir Sliva <61183145+lumir-sliva@users.noreply.github.com>
2026-04-22 19:14:48 +02:00
Travis Nielsen 80b50969b2 mon: prevent mon drains more reliably when mons are down
The mon PDB was previously only being adjusted
during mon failover, where the maxunavailable was set to 0
before failover, then reset to 1 after failover. This allowed
for a race condition where one mon had been drained and is
not fully back in quorum before the next mon is allowed to drain.
This is now prevented by disallowing drains until a mon
is fully back in quorum. Anytime a mon falls out of quorum,
the mon pdb will be adjusted to prevent further drains
from causing a mon quorum outage.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2026-04-17 10:35:00 -06:00
subhamkrai 833be6bc76 mon: add floating mon api changes and implementation
for tnf, adding initial api changes and implementation for floating mon.
The current pr follow embeded yaml approach for creating floating mon
deploymen instead of configmap approach.

Signed-off-by: subhamkrai <srai@redhat.com>
2026-03-30 15:45:16 +05:30
Travis Nielsen d315d0ce46 mon: change do not reconcile to be more granular
The do-not-reconcile label on the mon deployments was
causing the mon reconcile to be skipped for all mons,
even if the label was only on a single mon. Now, the
skipping of reconcile will only be scoped to a single
mon.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2026-01-20 14:47:19 -07:00
Travis Nielsen 93adc5dd6b core: include namespace more consistently in logging
When writing to the operator log, where possible
the namespace of the resource on which the operator
is working will be logged as a standard prefix
to more easily troubleshoot what resource is being reconciled

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2025-11-20 15:49:43 -07:00
Yite Gu bf6c295d02 core: use sync.Map for cluster.monitoringRoutines
`cluster.monitoringRoutines` stores three types of
daemon health: "mon", "osd", and "status". These are
used by three goroutines respectively, resulting in:
fatal error: concurrent map reads and map writes.

Signed-off-by: Yite Gu <guyite@bytedance.com>
2025-11-06 19:36:40 +08:00
Santosh Pillai 3137a857df mon: wait for the canary pods to be terminated
Wait for the mon-canary pods to be terminated before starting the actual
mons. When topology spread constraints are added to evenly spread the
mons, then pod-anti affinity does not allow mons to be created on the
same node where other mons are running, this also includes the
mon-canary pods. We delete the mon-canary pods and then start the actual
mon pods. Sometimes the termination of the canary pods might take more
time, so schedular won't schedular mon pods on that node.

Signed-off-by: Santosh Pillai <sapillai@redhat.com>
2025-10-23 21:11:13 +05:30
Travis Nielsen 48b70500fa mon: trap the sigterm in the mon canary
The mon canary should quickly shut down after the pod is
deleted, since it has no purpose to continue running.
Since the sleep does not catch the SIGTERM as pid 1,
use a bash trap to exit quickly.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2025-10-22 15:37:13 -06:00
parth-gr 3ba911e79e external: update rook ceph mon endpoint cm to use v2
if v2 is enabled update the rook ceph mon endpoint cm
with v2 ports

Signed-off-by: parth-gr <partharora1010@gmail.com>
2025-10-21 18:31:55 +05:30
Santosh Pillai d399615bdc mon: fix mon health nil pointer exception
shouldFailoverMonImmediately has a nil pointer exception with mons on
PVC. Rook-ceph-mon-endpoint does not store node information of the mons
on PVC. So `mapping.Hostname` throws nill pointer

Signed-off-by: Santosh Pillai <sapillai@redhat.com>
2025-09-15 18:13:46 +05:30
Blaine Gardner 7c33186e72 core: admin cephx key rotation
Implement CephX key rotation for Rook's client.admin user.

Admin user rotation is risky, so this has been tested extensively both
in unit tests as well as by manually injecting failures during runtime.
In testing, all failures were able to be recovered by the recovery
routine.

A mutex is also added to help ensure that two simultaneous admin key
rotation processes cannot be running simultaneously for any given
namespace. The mutex is tested in unit tests, and it was verified during
runtime via  manual testing.

Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
2025-08-27 14:47:45 -06:00
Blaine Gardner a952b7511a core: allow cephx rotation for ceph 19.2.3 except mons
Allow Rook to rotate CephX keys for Ceph v19.2.3 and up with the
exception of Ceph mons. While v19.2.3 has the `ceph auth rotate` command
that Rook needs to perform key rotation, there is still ongoing work in
Ceph to change how the `mon.` key is tracked internally. Rook's mon key
rotation requires those currently-unreleased Ceph changes.

Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
2025-08-18 16:50:11 -06:00
Blaine Gardner 2fd9c9a88c core: finalize CephCluster CephX status handling
Cleanup up overall usage of CephX statuses in CephCluster.

Convert all CephX statuses to non-pointer types. There is no specific
need for pointer types, and keeping these as non-pointers means there is
not a risk of nil pointer exceptions in all the various Rook controllers
for upgraded clusters.

Move initialization of CephCluster CephX status items to
`preMonStartupActions()`. This helps resolve 2 issues:

1. 'Uninitialized' state was being set for external CephClusters where
   CephX statuses are not relevant.
2. 'Uninitialized' state was being set for upgraded clusters whose key
   status cannot be determined.

Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
2025-08-18 14:53:08 -06:00
Travis Nielsen 36a47ac01a Merge pull request #16176 from sp98/mon-key-rotation
mon: rotate cephx key
2025-08-15 11:06:31 -06:00
Santosh Pillai fe42fe102f mon: rotate keys
rotate mon daemon keys and update the cephx status for mons in the
cephcluster resource.

Signed-off-by: Santosh Pillai <sapillai@redhat.com>
2025-08-14 11:16:39 +05:30
Blaine Gardner 59ca78229d Merge pull request #16075 from sp98/mirror-key-rotation
rbd mirror peer key rotation
2025-08-11 10:54:20 -06:00
Santosh Pillai ee0b137d75 core: rotate rbd mirror peer key
rotate the client.rbd-mirror-peer key based on the cephxconfig.
Also update the cephxStatus in the cephcluster as well as the
mirrored cephblockpool.

Signed-off-by: Santosh Pillai <sapillai@redhat.com>
2025-08-11 21:53:14 +05:30
Madhu Rajanna 1ff3f07ac5 csi: use cluster namespace for cephConn
CephConnection CR is created with the
kubernetes namespace name as its the
unique name in the entire cluster.
This PR removes unwanted input
parameters and adjust the namespace
name for the cephConnection CR.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2025-08-11 07:38:27 +02:00
Travis Nielsen 8b4d3b19cf mon: failover mon immediately if node not exist
During mon failover, if a mon is assigned to a node
but the node does not exist, the mon should be failed
over immediately instead of waiting for a 10 or 20
minute timeout. If the node does not exist, the mon pod
will be in a pending state until the failover is finally
triggered, and no point in waiting that long if the
node is gone.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2025-08-07 10:06:57 -06:00
Travis Nielsen aa61ddb57e mon: failover with host network must use different node
When a mon is failed over, the old mon is left in mon quorum
until the new quorum is confirmed in quorum. When host networking
is enabled, this may result in two mons being in quorum with the
same ip, if the new mon is scheduled on the same node as the
down mon. This results in daemon crashes in the cluster since
they are confused by multiple of the same ip in the mon map.

The solution is to avoid scaling down the bad mon when host networking
is enabled. This will ensure the new mon is not scheduled on the
same node as the previous mon. With host networking, either another
node is necessary or the original mon must come back online to
restore quorum.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2025-07-10 10:13:17 -06:00
Patryk Rostkowski 7f3e6bd7fa mon: allow running mon pods as root
This change addresses a permission issue where mon pods crashloop
on some Kubernetes setups with SELinux enabled, even when
ROOK_HOSTPATH_REQUIRES_PRIVILEGED is set.

To resolve this, a new function makeMonSecurityContext() was introduced
to explicitly set runAsUser: 0 at the pod level for mon pods
when the environment variable ROOK_CEPH_MON_RUN_AS_ROOT is set to true.

Additionally, the function previously named PodSecurityContext() was renamed
to DefaultContainerSecurityContext() to avoid confusion between
container-level and pod-level security context configuration. All container
SecurityContext usages across Ceph daemons were updated to reflect this change.

This ensures the root user configuration is applied
automatically and consistently in environments where it is required
for mon pod startup, while preserving clear separation of container
and pod-level security logic.

Signed-off-by: Patryk Rostkowski <patrostkowski@gmail.com>
2025-06-16 18:29:31 +02:00
Blaine Gardner 029c345372 Merge pull request #15813 from BlaineEXE/auth-rotate
object: automate RGW cephx key rotation
2025-06-10 12:51:50 -06:00
Blaine Gardner 5f50e85a25 object: implement cephx key rotation for rgw keys
This is the first implementation of CephX key rotation in Rook.
Adds key rotation API from design PR 15915.

Also add helper methods for rotating keys, determining when keys
need to be rotated, and for generating CephX key statuses. These will be
reusable for other Rook reconciles beyond object/RGW.

Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
2025-06-10 09:06:58 -06:00
Carlos Barria c63ebbe0e3 core: fix golangci-lint check ST1019
This change cleans up and standardizes import statements across the Ceph operator code. It removes redundant or duplicate imports and reorganizes alias names for improved clarity and consistency. Additionally, the ST1019 exception was removed from .golangci.yaml now that the code complies with the rule.

Signed-off-by: Carlos Barria <cbarria@yahoo.com>
2025-05-27 17:38:22 -04:00
parth-gr 94450c0db7 osd: osd should watch for pvc resize
With a osd resize operation, currently pvc and pv may
takes more time then expected.
And so the osd dont get resized and need manual re-start
as currently we don't have a watch on osd's
controller that watches pv and get it resized.

add a wait time till pvc is not resized

Signed-off-by: parth-gr <partharora1010@gmail.com>
2025-05-15 12:42:04 +05:30
Artem Torubarov c1fd2f2ee8 rgw: use pod name in ops log filename
Signed-off-by: Artem Torubarov <artem.torubarov@sap.com>
2025-04-29 09:43:35 +02:00
Santosh Pillai 75c6bb8cd2 mon: always delete mon canary pods
In case of multicluster service, we need to always delete the mon-canary
pods even if there is any failure while exporting the service, else
multiple mon canary pods will get created.

Signed-off-by: Santosh Pillai <sapillai@redhat.com>
2025-04-16 10:49:39 +05:30
Joshua Hoblitt 9f1ed201db core: typed watch handlers and predicates
All existing controller runtime watches are converted to use "typed"
handlers and predicates instead of operating on `client.Object`.  The
intent is to be bug for bug equivalent with the existing logic while
replacing run time type assertions and switch statements with compile
time type constraints and type casts. In several cases, functions using
assertions were split up such that each function only handles a single
Kind at a time. It is hoped that this will improve readability and
maintainability while facilitating future refactoring such as migrating
some watches to using IndexFields.

Of particular note is that the massive switch statement in
`WatchControllerPredicate()`  from
`pkg/operator/ceph/controller/predicate.go` has been replaced with
generics, reflection, and splitting the obc logic into its own predicate
function. There are still many helper functions operating on
`client.Object`. These were not updated unless required by the compiler
in order to limit the size of this change. The type safety of these
funcs should be improved as followup work.

 It is strongly suggested that going forward, handlers and predicates
 only handle a single Kind (generic or not) and that switches / type
 assertions are heavily discouraged or forbidden. This PR removed all
 but a single switch statement in a predicate, which should be addressed
 in future work.

Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
2025-04-04 09:40:56 -07:00
Patryk Rostkowski 08199b27c9 mon: manage endpoints containing set of mon ip addresses
The change creates and manages an Endpoints resource
with the current Ceph monitor (mon) IPs, allowing
clients to resolve mon IPs via DNS without relying
on the rook-ceph-mon-endpoints ConfigMap.

Signed-off-by: Patryk Rostkowski <patrostkowski@gmail.com>
2025-03-31 18:05:37 +02:00
Joshua Hoblitt 3cb343f62a core: run gofumpt on all files
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
2025-03-26 10:41:48 -07:00
Artem Torubarov 0b2e830111 mon: support external mons in local rook cluster
Implements #14733. Allows to set IDs of external mons to
Cluster CRD. Rook will not remove external mons from quorum
and will add external mon addresses to mon endpoints.
Use-case for external mon is to maintain quorum for 2-AZ
k8s cluster in case of zone outage.

Signed-off-by: Artem Torubarov <artem.torubarov@clyso.com>
2025-03-03 14:49:39 +01:00
Travis Nielsen 8429affc79 mgr: wait for builtin mgr pool to exist before enabling stretch
There is a race condition in ceph v19 at startup of a
stretch cluster where if the .mgr pool is created by
the mgr pod after stretch mode is entered, the
devicehealth mgr module will crash and cause a
ceph HEALTH_ERR. The error then causes rook reconcile
of all CRs to be skipped because Rook is assuming
admin intervention is needed to continue. The fix
is to wait until the .mgr pool is created by the mgr
pod before entering stretch mode.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2025-02-27 08:10:52 -07:00