Clean up stale CRUSH rules after the Ceph mgr starts so rules left behind by pool failure-domain or device-class changes do not accumulate indefinitely.
The cleanup is guarded by a package-level RWMutex. Pool create and update paths hold the read lock while creating and assigning CRUSH rules, while cluster-wide cleanup holds the write lock before listing pools and deleting unused rules. This keeps pool reconciles parallel with each other while preventing cleanup from deleting a rule that another reconcile has just created but not yet attached to a pool.
Keep direct pool-delete cleanup for the pool's current CRUSH rule, make the cluster-wide cleanup best-effort across all unused rules, and add an operator-level ROOK_DELETE_UNUSED_CRUSH_RULES setting for clusters that need to leave unused custom rules in place.
Document the operator and Helm settings, regenerate the Helm chart docs, and add a pending release note for the default cleanup behavior.
Signed-off-by: Asish Kumar <officialasishkumar@gmail.com>
add unit and integration tests to verify that erasure
coded cephblockpool status transitions to ready and
does not remain stuck in progressing.
Signed-off-by: Oded Viner <oviner@redhat.com>
deleting an erasure coded CephBlockPool left the ec profile
orphaned. recreating with different settings failed because
the old profile could not be overridden.
- delete ec profile during CephBlockPool deletion
- disable --format json on ec profile set command
- fix ec profile name mismatch in CephObjectStore deletion
Signed-off-by: Oded Viner <oviner@redhat.com>
ec pools skip the mirroring configuration path where
the status is set to ready, leaving them stuck in
progressing state forever. add status update to ready
in the else branch when mirroring is skipped.
Signed-off-by: Oded Viner <oviner@redhat.com>
the ec pool doesnt support mirroring from ceph end
so skip if the mirroring spec is set on the ec pools
Signed-off-by: parth-gr <partharora1010@gmail.com>
with controller runtime latest version vO.23.0, golangci lint
is complaining about deprecated api deprecated.
This commit updates the api and adds the requried rbacs
Signed-off-by: subhamkrai <srai@redhat.com>
Read affinity in Ceph v20.2.0 can cause corruption.
Disable this feature for this version of Ceph to
avoid hitting any corruption in this case. When the next
patch release is out for tentacle, this issue will be
resolved and when upgrading to it, read affinity would
automatically be re-enabled if needed.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
For all of the controllers besides the cluster controller,
the logging now includes the namespaced name of the resource
that is being reconciled. This will help with log troubleshooting
to help analyze logs consistently for the resource being
reconciled.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
if the pool is just created and getting the status
updated rn controller can still go and proceed to
check the pool status
If the pool status is empty we should retry instead of
proceeding further
Signed-off-by: parth-gr <partharora1010@gmail.com>
currently, in RNS crd while updating status we don't
have retryOnConflicts, similar to other CRD due to
this RNS status was not updated succesfully and to
fix this we have restart the rook operator pod.
Signed-off-by: subhamkrai <srai@redhat.com>
Cleanup up overall usage of CephX statuses in CephCluster.
Convert all CephX statuses to non-pointer types. There is no specific
need for pointer types, and keeping these as non-pointers means there is
not a risk of nil pointer exceptions in all the various Rook controllers
for upgraded clusters.
Move initialization of CephCluster CephX status items to
`preMonStartupActions()`. This helps resolve 2 issues:
1. 'Uninitialized' state was being set for external CephClusters where
CephX statuses are not relevant.
2. 'Uninitialized' state was being set for upgraded clusters whose key
status cannot be determined.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
When removeing CephBlockPool check first that related ceph pool is exist in cluster,
since it may be manually removed by mistake.
Related-Issue: #16328
Signed-off-by: Denis Egorenko <degorenko@mirantis.com>
if the blockpool mirroring is disabled and status check
is not removed, currently rn cr still watches for the status
Add a check if the pool or rados mirroring is disabled then dont
watch for status
Signed-off-by: parth-gr <partharora1010@gmail.com>
rotate the client.rbd-mirror-peer key based on the cephxconfig.
Also update the cephxStatus in the cephcluster as well as the
mirrored cephblockpool.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
CephConnection CR is created with the
kubernetes namespace name as its the
unique name in the entire cluster.
This PR removes unwanted input
parameters and adjust the namespace
name for the cephConnection CR.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
add RecoverAndLogException() helper to log panics with stack trace.
added defer call in all rook controller Reconcile() methods for
better error visibility in operator logs
Signed-off-by: Oded Viner <oviner@redhat.com>
Providing an option for user to specify the clusterID
they want to use in the storageclass while creating
the radosnamespace.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
The rados namespace status was not reporting if the namespace
was empty. Now the condition will be properly updated if
it is empty.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
there can be a scenario where a new RNS CR is created with
the same spec.blockPoolName and spec.name, in that case we want to delete
the old RNS CR without checking dependent child resources.
Signed-off-by: subhamkrai <srai@redhat.com>
When a rados namespace is deleted, it cannot be purged
until its images and snapshots are deleted. Now a condition
is being added to the rados namespace CR status so the user
does not need to read the operator log to discover the cause.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
When the blockpool is deleted and there are images or
snapshots in the pool, the deletion is blocked by the
finalizer until they are all removed. The condition added
to the cephblockpool status only indicated if there were
radosnamespace dependencies. A new condition is now added
to describe if there are images or snapshots.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
In some cases, for example, kube-api nods, CephBlockPool's status
could be failed to update. In that case, rook-operator does not
tries to update status again. This fix requests for requeue if pool's
status update is failed.
Signed-off-by: Peter Razumovsky <prazumovsky@mirantis.com>
When finalizers are added to the CRs, a follow-up reconcile
will be triggered due to the increased generation on the CR.
Therefore, abort the initial reconcile when adding the finalizer,
and allow the follow-up reconcile to complete the configuration.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
All existing controller runtime watches are converted to use "typed"
handlers and predicates instead of operating on `client.Object`. The
intent is to be bug for bug equivalent with the existing logic while
replacing run time type assertions and switch statements with compile
time type constraints and type casts. In several cases, functions using
assertions were split up such that each function only handles a single
Kind at a time. It is hoped that this will improve readability and
maintainability while facilitating future refactoring such as migrating
some watches to using IndexFields.
Of particular note is that the massive switch statement in
`WatchControllerPredicate()` from
`pkg/operator/ceph/controller/predicate.go` has been replaced with
generics, reflection, and splitting the obc logic into its own predicate
function. There are still many helper functions operating on
`client.Object`. These were not updated unless required by the compiler
in order to limit the size of this change. The type safety of these
funcs should be improved as followup work.
It is strongly suggested that going forward, handlers and predicates
only handle a single Kind (generic or not) and that switches / type
assertions are heavily discouraged or forbidden. This PR removed all
but a single switch statement in a predicate, which should be addressed
in future work.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
this commit add support for mirroring b/w implicit rados namespace and
defined rados namespace. To support this, rook will treat `<implicit>`
name on cephblockpool as `""` empty string. This will allow mirroring
b/w implicit and defined radosnamespace.
Signed-off-by: subhamkrai <srai@redhat.com>
today, csi operator cephConnection and client profile
CR's where not getting created in external mode. So, let's
start creating those.
Signed-off-by: subhamkrai <srai@redhat.com>
Implements #14733. Allows to set IDs of external mons to
Cluster CRD. Rook will not remove external mons from quorum
and will add external mon addresses to mon endpoints.
Use-case for external mon is to maintain quorum for 2-AZ
k8s cluster in case of zone outage.
Signed-off-by: Artem Torubarov <artem.torubarov@clyso.com>
The ci was using a pretty old version og golangci-lint.
This updates to the latest version.
Additionally, it silences some
gosec integer conversion overflow false positves
and fixes some real errors of this category
and string format errors found by golangci-lint, while at it.
Co-authored-by: Blaine Gardner <b.blaine.gardner@gmail.com>
Co-authored-by: Travis Nielsen <tnielsen@redhat.com>
Signed-off-by: Michael Adam <obnox@samba.org>
in external mode deployments ceph iamge is not provided
and job keeps on failing to get the cephversion
fix by checking if ceph iamge is available
Signed-off-by: parth-gr <partharora1010@gmail.com>
enable and disable the status of rados namespace
mirroring by looking at statusCheck spec of blockpool
Signed-off-by: parth-gr <partharora1010@gmail.com>
Asish Kumar
Travis Nielsen
Oded Viner
parth-gr
subhamkrai
Jongwoo Han
cuiweixie
Blaine Gardner
Denis Egorenko
Santosh Pillai
Madhu Rajanna
Peter Razumovsky
Joshua Hoblitt
subham rai
Artem Torubarov
