Clean up stale CRUSH rules after the Ceph mgr starts so rules left behind by pool failure-domain or device-class changes do not accumulate indefinitely.
The cleanup is guarded by a package-level RWMutex. Pool create and update paths hold the read lock while creating and assigning CRUSH rules, while cluster-wide cleanup holds the write lock before listing pools and deleting unused rules. This keeps pool reconciles parallel with each other while preventing cleanup from deleting a rule that another reconcile has just created but not yet attached to a pool.
Keep direct pool-delete cleanup for the pool's current CRUSH rule, make the cluster-wide cleanup best-effort across all unused rules, and add an operator-level ROOK_DELETE_UNUSED_CRUSH_RULES setting for clusters that need to leave unused custom rules in place.
Document the operator and Helm settings, regenerate the Helm chart docs, and add a pending release note for the default cleanup behavior.
Signed-off-by: Asish Kumar <officialasishkumar@gmail.com>
Ceph relaxed the stretch mode CRUSH weight check in ceph/ceph#66580 to
allow weights within 10 percent of each other. This change is available
in Ceph Umbrella and was backported to tentacle v20.2.2 via ceph/ceph
number 67790. The squid backport is still pending.
Signed-off-by: Lumir Sliva <61183145+lumir-sliva@users.noreply.github.com>
The floating mon in a TNF cluster is designed to share a node with a pinned mon. Fixed the pre-cluster start evaluation logic to allow the same when allowMultiplePerNode is set to false.
Signed-off-by: Prabhala Tara Aasrita <taraasrita@ibm.com>
The mon PDB was previously only being adjusted
during mon failover, where the maxunavailable was set to 0
before failover, then reset to 1 after failover. This allowed
for a race condition where one mon had been drained and is
not fully back in quorum before the next mon is allowed to drain.
This is now prevented by disallowing drains until a mon
is fully back in quorum. Anytime a mon falls out of quorum,
the mon pdb will be adjusted to prevent further drains
from causing a mon quorum outage.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Ceph incorrectly raises the MON_NETSPLIT health warning in stretch cluster configurations. Added a call to MuteHealthWarning() in postMonStartupActions() which runs `ceph health mute MON_NETSPLIT --sticky` to suppress the warning during reconcile. All errors are logged as warnings and the reconcile proceeds.
Signed-off-by: Prabhala Tara Aasrita <taraasrita@ibm.com>
When osd versions are being detected, log more info
about the versions to understand better if the
require-osd-version release is being set as
expected.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
for tnf, adding initial api changes and implementation for floating mon.
The current pr follow embeded yaml approach for creating floating mon
deploymen instead of configmap approach.
Signed-off-by: subhamkrai <srai@redhat.com>
When an OSD is purged and reprovisioned, it comes back without the
correct CRUSH device class. NVMe devices configured with
deviceClass: nvme in the CR end up as ssd.
Root cause: the operator sets ROOK_OSD_CRUSH_DEVICE_CLASS on prepare
jobs but not on OSD deployments. During re-discovery
(GetCephVolumeRawOSDs), the daemon reads this env var, finds it
absent, and falls back to auto-detected deviceType which returns ssd
for NVMe.
Fix: add crushDeviceClassEnvVar(osd.DeviceClass) to the OSD
deployment spec in spec.go, alongside the existing
dataDeviceClassEnvVar. Existing deployments pick up the new env var
on operator reconciliation without manual intervention.
Fixes#17227
Signed-off-by: David Orman <ormandj@corenode.com>
Clean up stale exporter deployments whose target node no longer exists.
This handles the case where the operator was down when a node was deleted,
so the deletion event was missed and the stale deployment was never removed.
Signed-off-by: Muhammad Adil Ghaffar <muhammad.adil.ghaffar@est.tech>
When reconciling the crash collectors and exporters, if one cluster
has disabled them, they will be deleted, then the reconcile
will complete, thus skipping reconcile of the crash collectors
and exporters for the remaining clusters.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Remove the exporter deployment when needed as best effort
in case the removal fails, there is no need to block other
reconcile.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Rook no longer supports earlier than Ceph v18, so remove the
obsolete version checking in the ceph exporter reconcile.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The osd unit tests periodically fail with a timing issue
when canceling an orchestration. Attempt to stabilize
the test with a longer timeout before canceling the
test.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
the ceph exporter port was hardcoded to 9926, preventing
co-location of exporters from different ceph clusters on
the same node. add a port field to cephexporterspec so
users can override it via spec.monitoring.exporter.port.
Signed-off-by: Oded Viner <oviner@redhat.com>
We should set OSDInfo.DeviceType if "device-type" label is
set on the corresponding Deployment. Otherwise, Rook
deletes this label on Update.
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
Update `lockbox.keyring` rotation for encrypted OSDs to ensure the key
is not lost if the init container fails but continues assuming
(reasonably) that the key is likely still relevant.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
Previously, if the OSD cephx-keyring-update init container couldn't
contact the mons, the output redirect would overwrite the on-disk
keyring file with no text, removing the on-disk keyring.
Modify the init container to only overwrite the keyring file when the
ceph command to get the latest key suceeds, to avoid removing the
on-disk key.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
with controller runtime latest version vO.23.0, golangci lint
is complaining about deprecated api deprecated.
This commit updates the api and adds the requried rbacs
Signed-off-by: subhamkrai <srai@redhat.com>
similar to other ceph pods like osd,mon,mgr,mirror
pod this commit add log collector container to the
main exporter pod to rotate the logs of exporter pod.
Signed-off-by: subhamkrai <srai@redhat.com>
This fixes OSD info being incomplete when a sidecar container is first
in the .spec.Template.Spec.Containers list.
I added a small helper function to find a container named "osd" instead
of always assuming the first one.
Signed-off-by: Jan Breitkopf <jan.breitkopf@prorocketeers.com>
since node fencing is disabled in rook for sometime
and this feature is implemented in csi. So, this
commits remove obsolete code related to node loss
Signed-off-by: subhamkrai <srai@redhat.com>
The do-not-reconcile label on the mon deployments was
causing the mon reconcile to be skipped for all mons,
even if the label was only on a single mon. Now, the
skipping of reconcile will only be scoped to a single
mon.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
if admin secret is created stop creating csi secret from
the rook operator
the csi creds and secret will still be created by the python and
the import script
Signed-off-by: parth-gr <partharora1010@gmail.com>
add support for ceph.rook.io/do-not-reconcile on cephcluster crs
skip reconcile early and emit a normal reconcile-skipped event
avoid reloadmanager on updates for labeled cephclusters
add unit test for skip behavior
Signed-off-by: Oded Viner <oviner@redhat.com>
The udev is required for cleanup in both host-based clusters
and pvc-based clusters since the tentacle release.
The cleanup jobs were failing in pvc-based clusters
with tentacle where we skipped adding the udev mount.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
For all of the controllers besides the cluster controller,
the logging now includes the namespaced name of the resource
that is being reconciled. This will help with log troubleshooting
to help analyze logs consistently for the resource being
reconciled.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
For Rook v1.19,removing support for Ceph Reef. It is at end of life.
Users on Reef can continue to use Rook v1.18.x or older.
The latest two Ceph versions Squid and Tentacle are supported.
Signed-off-by: subhamkrai <srai@redhat.com>
Add a resource watch to reconcile clusters when any referenced Secrets
in cephConfigFromSecret are updated. This matches the behaviour of
reconciling clusters when cephConfig changes. Without this change, if a
Secret that is referenced by cephConfigFromSecret is updated, then the
cluster is not immediately updated, so the cluster state will not match
the configuration until something else triggers a reconcile.
Resolves#16371
Signed-off-by: Martin Rowe <martin.p.rowe@gmail.com>
The logger names are seen on each line of logging. As
long as they are unique, they really don't need to be
long and descriptive, so let's make them a bit more
concise, and independent from controller names.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
When writing to the operator log, where possible
the namespace of the resource on which the operator
is working will be logged as a standard prefix
to more easily troubleshoot what resource is being reconciled
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Use a sync map for the clusterMap in the cluster controller
and fix other issues to allow for concurrent cluster
reconcile.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The cephcluster controller allows setting concurrent reconciles
with a setting in the operator configmap.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
In a running cluster, I noticed that a node which was provisioning was removed from k8s
and the following log was emitted by the rook operator:
```
op-osd: cleaning up dangling OSD prepare status configmap "rook-ceph-osd-xxx-status"
```
But the osd provisioning job was still trying to run on the removed node:
```
rook-ceph-osd-prepare-xxx Running 0/1 4d22h 4d22h
```
This PR implements a mechanism to remove any osd provision jobs which are bound to a
node which no longer exists within kubernetes. I added a test for this cleanup as
there wasn't a pre-existing test.
Signed-off-by: Elias Carter <elias@dropbox.com>
Asish Kumar
Travis Nielsen
Lumir Sliva
Parth Arora
Prabhala Tara Aasrita
Santosh Pillai
subhamkrai
David Orman
Muhammad Adil Ghaffar
Blaine Gardner
Oded Viner
Satoru Takeuchi
Jan Breitkopf
dependabot[bot]
Martin Rowe
Elias Carter