Commit Graph
220 Commits
Author SHA1 Message Date
Anas Khan 0bbd5de84c mon: fix data race on failedMonSchedule during scheduling
assignMons schedules each mon in its own goroutine and shares a
failedMonSchedule bool to record whether any of them failed. The
resultLock mutex already guards the shared c.mapping.Schedule map
write, but the three failedMonSchedule = true assignments were done
without holding the lock. When two or more mons fail to schedule at
the same time (for example when waitForMonitorScheduling errors, the
node choice is nil, or getNodeInfoFromNode fails) their goroutines
write the flag concurrently, which is a write-write data race.

Guard the failedMonSchedule writes with the existing resultLock, the
same lock the goroutines already use for the map update. The post-Wait
read is left as is since the WaitGroup orders it after every write.

Add a regression test that fails scheduling for multiple mons at once
and confirms assignMons returns an error; run it with -race to catch
the race.

Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>
2026-07-14 18:03:13 +05:30
Joshua Hoblitt 49612461a4 docs: fix function comments to match their declaration names
Several godoc comments led with a stale or incorrect identifier, left
over from renames, exported/unexported changes, copy-paste between
sibling declarations, or plain typos. As a result the documented name no
longer matched the function, method, type, or var it describes. Correct
each leading word to the name of the declaration it documents.

Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
2026-06-26 12:52:39 -07:00
pengqima 240c431f42 docs: fix function comment to match actual function name
Signed-off-by: pengqima <pengqima@outlook.com>
2026-06-26 23:43:22 +08:00
daixihegu 063dcad054 core: use slices.Contains to simplify code
Signed-off-by: daixihegu <daixihegu@163.com>
2026-06-25 10:17:53 +08:00
Joshua Hoblitt 8dd1778655 Merge pull request #17561 from jhoblitt/maint-ref-all-containers-by-name-instead-of-index
core: reference all containers in a spec by name instead of index
2026-05-26 12:10:09 -07:00
Joshua Hoblitt d059c351fc core: reference all containers in a spec by name instead of index
This changeset excludes converting _test.go code as there may be a
legitimate reasons (E.g. convenience) for unit test to be sensitive to
ordering.

Related to:
- https://github.com/rook/rook/issues/15291
- https://github.com/rook/rook/pull/16969

Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
2026-05-26 10:09:23 -07:00
Joshua Hoblitt 9eec9c730e core: rm unused consts
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
2026-05-14 09:34:52 -07:00
subhamkrai 4eefad42e8 csi: move csi management to admin
Going forward, admin will manage the csi operator
CR's and rook will only manage Ceph Connection cr
and client Profile cr.

The old csi driver is completely removed from Rook
and can no longer be used starting in Rook v1.20.

The upgrade guide will contain the needed transition steps
for managing the csi operator settings.

Signed-off-by: subhamkrai <srai@redhat.com>
2026-04-29 14:24:26 -06:00
Travis Nielsen 0160ba5bd0 Merge pull request #17288 from lumir-sliva/relax-stretch-crush-weight-check
mon: relax balanced CRUSH weight check for stretch mode
2026-04-23 12:05:12 -06:00
Lumir Sliva 27f95f2feb mon: relax balanced CRUSH weight check for stretch mode
Ceph relaxed the stretch mode CRUSH weight check in ceph/ceph#66580 to
allow weights within 10 percent of each other. This change is available
in Ceph Umbrella and was backported to tentacle v20.2.2 via ceph/ceph
number 67790. The squid backport is still pending.

Signed-off-by: Lumir Sliva <61183145+lumir-sliva@users.noreply.github.com>
2026-04-22 19:14:48 +02:00
Travis Nielsen 80b50969b2 mon: prevent mon drains more reliably when mons are down
The mon PDB was previously only being adjusted
during mon failover, where the maxunavailable was set to 0
before failover, then reset to 1 after failover. This allowed
for a race condition where one mon had been drained and is
not fully back in quorum before the next mon is allowed to drain.
This is now prevented by disallowing drains until a mon
is fully back in quorum. Anytime a mon falls out of quorum,
the mon pdb will be adjusted to prevent further drains
from causing a mon quorum outage.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2026-04-17 10:35:00 -06:00
subhamkrai 833be6bc76 mon: add floating mon api changes and implementation
for tnf, adding initial api changes and implementation for floating mon.
The current pr follow embeded yaml approach for creating floating mon
deploymen instead of configmap approach.

Signed-off-by: subhamkrai <srai@redhat.com>
2026-03-30 15:45:16 +05:30
Travis Nielsen d315d0ce46 mon: change do not reconcile to be more granular
The do-not-reconcile label on the mon deployments was
causing the mon reconcile to be skipped for all mons,
even if the label was only on a single mon. Now, the
skipping of reconcile will only be scoped to a single
mon.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2026-01-20 14:47:19 -07:00
Travis Nielsen 93adc5dd6b core: include namespace more consistently in logging
When writing to the operator log, where possible
the namespace of the resource on which the operator
is working will be logged as a standard prefix
to more easily troubleshoot what resource is being reconciled

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2025-11-20 15:49:43 -07:00
Santosh Pillai 3137a857df mon: wait for the canary pods to be terminated
Wait for the mon-canary pods to be terminated before starting the actual
mons. When topology spread constraints are added to evenly spread the
mons, then pod-anti affinity does not allow mons to be created on the
same node where other mons are running, this also includes the
mon-canary pods. We delete the mon-canary pods and then start the actual
mon pods. Sometimes the termination of the canary pods might take more
time, so schedular won't schedular mon pods on that node.

Signed-off-by: Santosh Pillai <sapillai@redhat.com>
2025-10-23 21:11:13 +05:30
Travis Nielsen 48b70500fa mon: trap the sigterm in the mon canary
The mon canary should quickly shut down after the pod is
deleted, since it has no purpose to continue running.
Since the sleep does not catch the SIGTERM as pid 1,
use a bash trap to exit quickly.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2025-10-22 15:37:13 -06:00
Blaine Gardner a952b7511a core: allow cephx rotation for ceph 19.2.3 except mons
Allow Rook to rotate CephX keys for Ceph v19.2.3 and up with the
exception of Ceph mons. While v19.2.3 has the `ceph auth rotate` command
that Rook needs to perform key rotation, there is still ongoing work in
Ceph to change how the `mon.` key is tracked internally. Rook's mon key
rotation requires those currently-unreleased Ceph changes.

Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
2025-08-18 16:50:11 -06:00
Blaine Gardner 2fd9c9a88c core: finalize CephCluster CephX status handling
Cleanup up overall usage of CephX statuses in CephCluster.

Convert all CephX statuses to non-pointer types. There is no specific
need for pointer types, and keeping these as non-pointers means there is
not a risk of nil pointer exceptions in all the various Rook controllers
for upgraded clusters.

Move initialization of CephCluster CephX status items to
`preMonStartupActions()`. This helps resolve 2 issues:

1. 'Uninitialized' state was being set for external CephClusters where
   CephX statuses are not relevant.
2. 'Uninitialized' state was being set for upgraded clusters whose key
   status cannot be determined.

Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
2025-08-18 14:53:08 -06:00
Travis Nielsen 36a47ac01a Merge pull request #16176 from sp98/mon-key-rotation
mon: rotate cephx key
2025-08-15 11:06:31 -06:00
Santosh Pillai fe42fe102f mon: rotate keys
rotate mon daemon keys and update the cephx status for mons in the
cephcluster resource.

Signed-off-by: Santosh Pillai <sapillai@redhat.com>
2025-08-14 11:16:39 +05:30
Madhu Rajanna 1ff3f07ac5 csi: use cluster namespace for cephConn
CephConnection CR is created with the
kubernetes namespace name as its the
unique name in the entire cluster.
This PR removes unwanted input
parameters and adjust the namespace
name for the cephConnection CR.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2025-08-11 07:38:27 +02:00
Travis Nielsen aa61ddb57e mon: failover with host network must use different node
When a mon is failed over, the old mon is left in mon quorum
until the new quorum is confirmed in quorum. When host networking
is enabled, this may result in two mons being in quorum with the
same ip, if the new mon is scheduled on the same node as the
down mon. This results in daemon crashes in the cluster since
they are confused by multiple of the same ip in the mon map.

The solution is to avoid scaling down the bad mon when host networking
is enabled. This will ensure the new mon is not scheduled on the
same node as the previous mon. With host networking, either another
node is necessary or the original mon must come back online to
restore quorum.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2025-07-10 10:13:17 -06:00
Blaine Gardner 029c345372 Merge pull request #15813 from BlaineEXE/auth-rotate
object: automate RGW cephx key rotation
2025-06-10 12:51:50 -06:00
Blaine Gardner 5f50e85a25 object: implement cephx key rotation for rgw keys
This is the first implementation of CephX key rotation in Rook.
Adds key rotation API from design PR 15915.

Also add helper methods for rotating keys, determining when keys
need to be rotated, and for generating CephX key statuses. These will be
reusable for other Rook reconciles beyond object/RGW.

Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
2025-06-10 09:06:58 -06:00
Carlos Barria c63ebbe0e3 core: fix golangci-lint check ST1019
This change cleans up and standardizes import statements across the Ceph operator code. It removes redundant or duplicate imports and reorganizes alias names for improved clarity and consistency. Additionally, the ST1019 exception was removed from .golangci.yaml now that the code complies with the rule.

Signed-off-by: Carlos Barria <cbarria@yahoo.com>
2025-05-27 17:38:22 -04:00
parth-gr 94450c0db7 osd: osd should watch for pvc resize
With a osd resize operation, currently pvc and pv may
takes more time then expected.
And so the osd dont get resized and need manual re-start
as currently we don't have a watch on osd's
controller that watches pv and get it resized.

add a wait time till pvc is not resized

Signed-off-by: parth-gr <partharora1010@gmail.com>
2025-05-15 12:42:04 +05:30
Patryk Rostkowski 08199b27c9 mon: manage endpoints containing set of mon ip addresses
The change creates and manages an Endpoints resource
with the current Ceph monitor (mon) IPs, allowing
clients to resolve mon IPs via DNS without relying
on the rook-ceph-mon-endpoints ConfigMap.

Signed-off-by: Patryk Rostkowski <patrostkowski@gmail.com>
2025-03-31 18:05:37 +02:00
Joshua Hoblitt 3cb343f62a core: run gofumpt on all files
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
2025-03-26 10:41:48 -07:00
Artem Torubarov 0b2e830111 mon: support external mons in local rook cluster
Implements #14733. Allows to set IDs of external mons to
Cluster CRD. Rook will not remove external mons from quorum
and will add external mon addresses to mon endpoints.
Use-case for external mon is to maintain quorum for 2-AZ
k8s cluster in case of zone outage.

Signed-off-by: Artem Torubarov <artem.torubarov@clyso.com>
2025-03-03 14:49:39 +01:00
Travis Nielsen 8429affc79 mgr: wait for builtin mgr pool to exist before enabling stretch
There is a race condition in ceph v19 at startup of a
stretch cluster where if the .mgr pool is created by
the mgr pod after stretch mode is entered, the
devicehealth mgr module will crash and cause a
ceph HEALTH_ERR. The error then causes rook reconcile
of all CRs to be skipped because Rook is assuming
admin intervention is needed to continue. The fix
is to wait until the .mgr pool is created by the mgr
pod before entering stretch mode.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2025-02-27 08:10:52 -07:00
Artem Torubarov 25ee6b4f59 operator: custom hostname topology label
Signed-off-by: Artem Torubarov <artem.torubarov@clyso.com>
2025-01-24 15:59:07 +01:00
Travis Nielsen e2cadab6fe mon: do not remove extra mon in middle of failover
If the mon failover is in progress, ensure the removal
of an extra mon deployment is skipped since that code
path only has one mon in the list for the mon that was
just newly started. The extra mon was erroneously removing
a random mon in that case, followed immediately by the mon
failover completing and removing the expected failed mon,
and potentially causing mon quroum loss.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2024-10-04 14:29:57 -06:00
Travis Nielsen 8987d26a6d mon: remove extra mon from quorum before taking down pod
When removing a mon from quorum, there is a race condition that
can result in mon quorum going being lost at least temporarily.
The mon pod was being deleted first, and then the mon removed
from quorum. If any other mon went down between the time the
pod of the bad mon was deleted and when the mon was removed from
quorum, there may not be sufficient quorum to complete the
action of removing the mon from quorum and the operator would
be stuck.

For example, there could be 4 mons temporarily due to timing
of upgrading K8s nodes where mons may be taken down for some
number of minutes. Say a new mon is started while the down
mon also comes back up. Now the operator sees it can remove
the 4th mon from quorum, so it starts to remove it. Now say
another mon goes down on another node that is being updated
or otherwise drained. Since the 4th mon pod was deleted
and another mon is down, there are only two mons remaining
in quorum, but 3 mons are required in quorum when there
are 4 mons. Therefore, the quorum is stuck until the
third mon comes back up.

The solution is to first remove the extra mon from quorum
before taking down the mon pod.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2024-09-04 13:06:22 -06:00
subhamkrai 667e044013 csi: add new CSI-operator config CR
adding changes to create CSI-operator config CR based on
every radosnamesapce and subvolumegroup.

Signed-off-by: subhamkrai <srai@redhat.com>
2024-08-08 11:21:49 +05:30
subhamkrai 4b0b3a55d9 csi: add code for new CSI operator CR cephcluster
adding code changes,rbac changes required for create the new
Ceph-CSI operator CR named cephCluster in api group 'csi.ceph.io'.

Signed-off-by: subhamkrai <srai@redhat.com>
2024-08-08 11:21:45 +05:30
Satoru Takeuchi f776c0b35e core: remove pacific specific code
Since pacific is no longer supported, we can remove pacific specific
code.

Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
2024-05-15 02:26:33 +00:00
Blaine Gardner 4ff881240f csi: fix missing namespace bug in csi cluster config map
Someone testing the Multus holder pod removal feature encountered an
issue where the migration process failed to lead to a system state where
PVCs could be created successfully.

The root cause was found to be a ceph csi config map wherein the primary
CephCluster entry was lacking a value for the "namespace" field.

I observed this once in my development on the holder pod removal
feature, but I was unable to reproduce and assumed it was my own error.
Since this has been seen in a user environment, it must be that the
error is a race condition, and I am unable to determine the exact source
of the bug.

I do not believe this bug would be present if the code that updates the
CSI configmap were properly idempotent, but it has many conditions based
on prior states, and I was unable to determine how to resolve this
underlying impelementation pattern issue.

Instead, I opted to separate the `clusterKey` parameter into two clear
parts:
1. `clusterID` for when `clusterKey` is used as an analogue for
   `clusterID`
2. `clusterNamespace` for when `clusterKey` is used as an analogue for
   `clusterNamespace`

I added unit tests to ensure that SaveClusterConfig() will be able to
detect when the namespace is currently missing, and using the new
`clusterNamespace` field, it should always know what value to use as
input when correcting the bug in already-installed systems.

I also verified that this update works when the function simultaneously
removes netNamespaceFilePath entries, and that those entries are removed
properly.

Finally, manual testing also verifies the change.

Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
2024-05-02 13:58:00 -06:00
Redouane KachachandBlaine Gardner bd7477ec61 manifest: reduce CRD size by removing some descriptions
Reduce CRD size by removing some descriptions for things that
(a) are long, (b) are repeated, and (c) are commong kubernetes types.

Remove the `Status` field from VolumeClaimTemplates (a.k.a. PVCs) in
types.go to remove all unnecessary status-related fields.

Use `yq` to go through generated `crds.yaml` to strip out descriptions
for well-known k8s types that are long and repeated where we can't
remove them from types.go. This is mostly placement descriptions and
volumeSource descriptions.

The overall `crds.yaml` file is reduced to less than 60% of its original
size.

Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
Signed-off-by: Redouane Kachach <rkachach@redhat.com>
Co-authored-by: Blaine Gardner <blaine.gardner@ibm.com>
2024-02-20 10:08:44 +01:00
Praveen M dfd3d625a8 csi: use ceph-csi exposed csi config map API
This commit uses an csi config map API exposed by Ceph-CSI.
This will reduce the efforts of keeping the configuration in
sync with Ceph-CSI.

Signed-off-by: Praveen M <m.praveen@ibm.com>
2024-01-12 15:04:20 +05:30
sp98 42ea2897cf mon: allow changing hostNetwork settings
Allow changing spec.network.hostNetwork settings on a running cluster
by failing over the mons

Signed-off-by: sp98 <sapillai@redhat.com>
2023-12-18 10:30:48 +05:30
sp98 4b6fd89365 mon: fix mon failover on path change
This PR fails over mon when the mon path is changed from hostPath to PVC or vice versa

Signed-off-by: sp98 <sapillai@redhat.com>
2023-12-12 10:32:05 +05:30
gauravsitlani 3d0049c547 core: operator to skip reconcile of mgr, rgw, mds and rbd-mirror daemons in debug
During certain maintenance tasks the admin will own running
operations on the ceph mgr, rgw, mds and rbd-mirror daemons
and the operator should not interfere with those operations.

Co-authored-by: gauravsitlani <gaurav.sitlani@live.com>
Signed-off-by: subhamkrai <srai@redhat.com>
2023-11-21 20:14:16 +05:30
sp98 09356636fa mon: failover mon from hostpath to pv
failover mons from  hostpath to pv and vice versa

Signed-off-by: sp98 <sapillai@redhat.com>
2023-10-30 11:46:00 +05:30
Blaine Gardner 3c43268d0a multus: detect network CIDRs via canary
Change how Rook detects network CIDRs for Multus networks. The IPAM
configuration is only defined as an arbitrary string JSON blob with a
"type" field and nothing more. Rook's detection of CIDRs for whereabouts
had already grown out of date since the initial implementation.
Additionally, Rook did not support DHCP IPAM, which is a reasonable
choice for users. And more, Rook did not support CNI plugin chaining,
which further complicates NADs. Based on the CNI spec, network chaning
can result in any changes to network CIDRs from the first-given plugin.

All these problems make it more and more difficult for Rook to support
Multus by inspecting the NAD itself to predict network CIDRs. Instead,
it is better for Rook to treat the CNI process as a black box. To
preserve legacy functionality of auto-detecting networks and to make
that as robust as possible, change to a canary-style architecture like
that used for Ceph mons, from which Rook will detect the network CIDRs
if possible.

Also allow users to specify overrides for CIDR ranges. This allows Rook
to still support esoteric and unexpected NAD or network configurations
where a CIDR range is not detectable or where the range detected would
be incomplete. Because it may be impossible for Rook to understand the
network CIDRs wholistically while residing only on a portion of the
network, this feature should have been present from Multus's inception.

Improving CIDR auto-detection and allowing users to specify overrides
for auto-detected CIDRs rounds out Rook's Multus support for CephCluster
(core/RADOS) installations. No further architectural changes should be
needed for CephClusters as regards application of public/cluster network
CIDRs for Multus networks.

Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
2023-09-07 10:12:55 -06:00
Travis Nielsen afaadd78c4 Merge pull request #12384 from ideepika/feature/ksd-174
mon: add mon storage class support
2023-08-17 11:52:20 -06:00
Deepika Upadhyay 427c53786b docs: amend for clarity on failureDomainLabel and VolumeClaimTemplate
Signed-off-by: Deepika Upadhyay <deepika@koor.tech>
2023-08-09 21:21:40 +05:30
Deepika Upadhyay b67bf2d6f0 mon: update mon deployment
* fixes mon zonal deployment of 2 per zone to 1 if zones field is
  specified.
* adds zones field doc to CRDs Documentation.
* add code for picking up failureDomainLabel if specified for zones
  field in mon deployment

Signed-off-by: Deepika Upadhyay <deepika@koor.tech>
2023-08-03 17:01:51 +05:30
Deepika Upadhyay 0d86588620 mon: fix some of the zonal storageclass support failures
Signed-off-by: Deepika Upadhyay <deepika@koor.tech>
2023-08-03 16:41:24 +05:30
Deepika Upadhyay 764f6597c0 mon: add failure domain label for zonal topology
Signed-off-by: Deepika Upadhyay <deepika@koor.tech>
2023-08-03 16:41:24 +05:30
Deepika Upadhyay 2231d73259 mon: add mon storage class support
if zones field is specified, deploy the mon based on the zone specified,
this will help with higher mon availability.

See also: https://github.com/rook/rook/pull/11407/

Signed-off-by: Deepika Upadhyay <deepika@koor.tech>
2023-08-03 16:41:23 +05:30