rotate cephx keys based on keyRotationPolicy
and update cephxStatus for the ceph exporter daemon pods.
Signed-off-by: parth-gr <partharora1010@gmail.com>
(cherry picked from commit 9ec7a767c3)
Implement automated CephX key rotation for OSDs.
OSDs have marshalled CephX statuses applied as an annotation on the OSD
pod spec, which differs from other Rook Ceph daemons for a two reasons:
A large cluster could have hundreds or thousands of OSDs, so it is more
critical that Rook avoid rotating OSD keys more than once in the event a
CephCluster reconcile restarts partway through OSD creation/updation.
OSDs do not have their CephX keys stored in K8s Secrets, whose resource
version is used for enforcing the desired restart behavior other daemon
pods like RGWs.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
(cherry picked from commit 4189a196c1)
rotate cephx keys based on keyRotationPolicy
and update cephxStatus for the crashcollector
daemon pods.
Signed-off-by: parth-gr <partharora1010@gmail.com>
add RecoverAndLogException() helper to log panics with stack trace.
added defer call in all rook controller Reconcile() methods for
better error visibility in operator logs
Signed-off-by: Oded Viner <oviner@redhat.com>
This PR implements the CSI cephx key rotation feature,
which follows overlapping rotation for non-daemon keys.
Refer to the cephx rotation API from design PR 15915 for details.
Signed-off-by: subhamkrai <srai@redhat.com>
When a mon is failed over, the old mon is left in mon quorum
until the new quorum is confirmed in quorum. When host networking
is enabled, this may result in two mons being in quorum with the
same ip, if the new mon is scheduled on the same node as the
down mon. This results in daemon crashes in the cluster since
they are confused by multiple of the same ip in the mon map.
The solution is to avoid scaling down the bad mon when host networking
is enabled. This will ensure the new mon is not scheduled on the
same node as the previous mon. With host networking, either another
node is necessary or the original mon must come back online to
restore quorum.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Create a new OSD pod init container that loads the latest CephX key for
the OSD. The init container must be run after OSD activation, due to
keyfile reset behavior of ceph-bluestore-tool, used during activation.
The primary purpose of this work is to allow OSDs to start and run
correctly after key rotation, whether done manually by users or
automatically by future Rook work.
This work makes redundant prior code changes done in #14826. The CI test
added in the prior work is kept to ensure no regression.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
This change addresses a permission issue where mon pods crashloop
on some Kubernetes setups with SELinux enabled, even when
ROOK_HOSTPATH_REQUIRES_PRIVILEGED is set.
To resolve this, a new function makeMonSecurityContext() was introduced
to explicitly set runAsUser: 0 at the pod level for mon pods
when the environment variable ROOK_CEPH_MON_RUN_AS_ROOT is set to true.
Additionally, the function previously named PodSecurityContext() was renamed
to DefaultContainerSecurityContext() to avoid confusion between
container-level and pod-level security context configuration. All container
SecurityContext usages across Ceph daemons were updated to reflect this change.
This ensures the root user configuration is applied
automatically and consistently in environments where it is required
for mon pod startup, while preserving clear separation of container
and pod-level security logic.
Signed-off-by: Patryk Rostkowski <patrostkowski@gmail.com>
This is the first implementation of CephX key rotation in Rook.
Adds key rotation API from design PR 15915.
Also add helper methods for rotating keys, determining when keys
need to be rotated, and for generating CephX key statuses. These will be
reusable for other Rook reconciles beyond object/RGW.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
Allow force creation of OSD on a device previously configured in a different ceph cluster.
The OSD prepare pod job will zap the OSD disk using ceph-volume lvm zap
if that disk has metadata from a different ceph cluster.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
this commit remove the code which sets the dmcrypt encryption
key from secret as environment variable in the prepare pod job
spec. Now, the env will be set under function `setKEKinEnv`, which
tries to reads secret from k8s first method `c.IsK8s()`under
`GetSecret` and then check for KMS to ready and apply env directly.
Signed-off-by: subhamkrai <srai@redhat.com>
update comment to correctly state that reconcilecephcluster
manages the cephcluster object, not cephfilesystem.
Signed-off-by: Oded Viner <oviner@redhat.com>
This change cleans up and standardizes import statements across the Ceph operator code. It removes redundant or duplicate imports and reorganizes alias names for improved clarity and consistency. Additionally, the ST1019 exception was removed from .golangci.yaml now that the code complies with the rule.
Signed-off-by: Carlos Barria <cbarria@yahoo.com>
Refactored multiple conditional if-else blocks into switch statements to improve readability and maintainability. Also removed outdated staticcheck rule comments (QF1002, QF1003) from .golangci.yaml.
Signed-off-by: Carlos Barria <cbarria@yahoo.com>
Labels cannot be longer than 64 characters, and if the image name
is too long we log a verbose message about the label that we can
just log now as debug level.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
When monitoring is enabled, if the service monitor fails
creation, the reconcile will fail and rook will never
create the OSDs. A number of users have been blocked during
install due to this issue. Let's allow reconcile of the
full cluster even in this error case. After they install
prometheus, they would then need to trigger a reconcile again
perhaps by restarting the rook operator to enable
the service monitor.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
With a osd resize operation, currently pvc and pv may
takes more time then expected.
And so the osd dont get resized and need manual re-start
as currently we don't have a watch on osd's
controller that watches pv and get it resized.
add a wait time till pvc is not resized
Signed-off-by: parth-gr <partharora1010@gmail.com>
Named containerPort is a useful feature when generating dynamic scrape
targets with prometheus.
This makes it clear that the containerPort is intended for metrics
scraping.
The name matches that which is on the mgr deployment.
Signed-off-by: Justin Cichra <jcichra@cloudflare.com>
Adds .spec.cephConfigFromSecret to CephCluster for loading
Ceph config parameters from a Kubernetes Secret.
Signed-off-by: Patryk Rostkowski <patrostkowski@gmail.com>
In case of multicluster service, we need to always delete the mon-canary
pods even if there is any failure while exporting the service, else
multiple mon canary pods will get created.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
When finalizers are added to the CRs, a follow-up reconcile
will be triggered due to the increased generation on the CR.
Therefore, abort the initial reconcile when adding the finalizer,
and allow the follow-up reconcile to complete the configuration.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
All existing controller runtime watches are converted to use "typed"
handlers and predicates instead of operating on `client.Object`. The
intent is to be bug for bug equivalent with the existing logic while
replacing run time type assertions and switch statements with compile
time type constraints and type casts. In several cases, functions using
assertions were split up such that each function only handles a single
Kind at a time. It is hoped that this will improve readability and
maintainability while facilitating future refactoring such as migrating
some watches to using IndexFields.
Of particular note is that the massive switch statement in
`WatchControllerPredicate()` from
`pkg/operator/ceph/controller/predicate.go` has been replaced with
generics, reflection, and splitting the obc logic into its own predicate
function. There are still many helper functions operating on
`client.Object`. These were not updated unless required by the compiler
in order to limit the size of this change. The type safety of these
funcs should be improved as followup work.
It is strongly suggested that going forward, handlers and predicates
only handle a single Kind (generic or not) and that switches / type
assertions are heavily discouraged or forbidden. This PR removed all
but a single switch statement in a predicate, which should be addressed
in future work.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
Fixes an ordering issue where spec.cephConfig settings failed to apply
because the manager wasn't yet running. Moved the config logic from
postMonStartupActions() to a new postMgrStartupActions() to ensure
the mgr is ready before applying config, allowing successful reconciliation.
Signed-off-by: Patryk Rostkowski <patrostkowski@gmail.com>
The change creates and manages an Endpoints resource
with the current Ceph monitor (mon) IPs, allowing
clients to resolve mon IPs via DNS without relying
on the rook-ceph-mon-endpoints ConfigMap.
Signed-off-by: Patryk Rostkowski <patrostkowski@gmail.com>
One ore more OSDs can be down and PGs can still be active+clean (data
was rebalanced to other available OSDs). This PR sets
maxunavailable=1+downOSDs to allow one healthy to be drained.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
mergify[bot]
parth-gr
Blaine Gardner
Santosh Pillai
chentao.2022
Travis Nielsen
Oded Viner
subhamkrai
raaizik
Muhammad Adil Ghaffar
Kyori Sakao
Patryk Rostkowski
Carlos Barria
subham rai
Justin Cichra
Artem Torubarov
Joshua Hoblitt