Wait for the mon-canary pods to be terminated before starting the actual
mons. When topology spread constraints are added to evenly spread the
mons, then pod-anti affinity does not allow mons to be created on the
same node where other mons are running, this also includes the
mon-canary pods. We delete the mon-canary pods and then start the actual
mon pods. Sometimes the termination of the canary pods might take more
time, so schedular won't schedular mon pods on that node.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
(cherry picked from commit 3137a857df)
The OSDs may require different resources based on their device
class. The overall osd resources were only being used, rather
than properly looking up the specific device class resources.
Now the expected resources will take affect per device class.
Co-authored-by: Matthias Osthues <osthues.matthias@gmail.com>
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
(cherry picked from commit d3c0bacd88)
The mon canary should quickly shut down after the pod is
deleted, since it has no purpose to continue running.
Since the sleep does not catch the SIGTERM as pid 1,
use a bash trap to exit quickly.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
(cherry picked from commit 48b70500fa)
if there a bit long to update osds in parallel,
we need to timeout and abort the synchronous loop
and start the cluster reconcile again from the beginning
Signed-off-by: parth-gr <partharora1010@gmail.com>
(cherry picked from commit 4377823675)
When OSDs start, init containers attempt to get the latest CephX key for
the OSD to allow OSDs to potentially continue working after key rotation.
However, in Ceph corner cases, mons could be unavailable while OSDs
could still be viable with existing on-disk keys. Since key rotation is
a rare event, it's likely that the on-disk keys are still good.
The worst case scenario is that key updates were needed but didn't
succeed in init containers. In this case, instead of the init container
failing, the OSD startup will fail instead. Thus, this should introduce
no extra risk but merely shift the failure to a different container.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
(cherry picked from commit d9d31fe527)
external mode was dependednt on the ceph status to
update the cluster connected condition,
BUt as now the status check will only update the ceph status,
so we will explicitly update the ceph status condition in the
end of the reconcile
Signed-off-by: parth-gr <partharora1010@gmail.com>
(cherry picked from commit 3f5ebe2f4f)
currently the health checkup was reporting the
condition and reason to be always connected,
But which might not be the case,
So instead of that dont report the condition and
its phase during the health checkups
Signed-off-by: parth-gr <partharora1010@gmail.com>
(cherry picked from commit 454c221c53)
If the CephCluster spec is updated before the prevalidation check, then
Cluster.context still refers to the cancelled context of the operator.
This PR just updates the cluster context to use the latest operator
context.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
(cherry picked from commit 1ff7604f82)
shouldFailoverMonImmediately has a nil pointer exception with mons on
PVC. Rook-ceph-mon-endpoint does not store node information of the mons
on PVC. So `mapping.Hostname` throws nill pointer
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
(cherry picked from commit d399615bdc)
when the osd is created there is no need for
osd bootstrap keyring, so remove the keyring
and keyring file,
New keyring will be created if
new osd need to be provisoned
Signed-off-by: parth-gr <partharora1010@gmail.com>
(cherry picked from commit 794a232835)
Delete the client.boostrap.* keys. These keys are not created by rook so
it can't rotate them. This PR just delete these keys in the
postMonStartUpAction.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
(cherry picked from commit 4cdba0df54)
ceph-volume creates osd-lockbox keys for the encryted osds. This PR
rotates those keys along with the OSD keys.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
(cherry picked from commit b4e1c1720d)
this commit add watcher for cm `rook-ceph-operator-config`
to look for updated value of `ROOK_USE_CSI_OPERATOR` before
adding this watcher we had to restart rook operator pod
so that it reconciles and create cephConnection CR.
Signed-off-by: subhamkrai <srai@redhat.com>
(cherry picked from commit dd6693ff97)
If the network is encrypted with msgr2, cephfs requires
the mount options to include ms_mode=secure. Now this
will be set by default if the mount options are not
already set in the cephcluster CR or the operator env
var CSI_CEPHFS_KERNEL_MOUNT_OPTIONS.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
(cherry picked from commit 6309cd367e)
Implement CephX key rotation for Rook's client.admin user.
Admin user rotation is risky, so this has been tested extensively both
in unit tests as well as by manually injecting failures during runtime.
In testing, all failures were able to be recovered by the recovery
routine.
A mutex is also added to help ensure that two simultaneous admin key
rotation processes cannot be running simultaneously for any given
namespace. The mutex is tested in unit tests, and it was verified during
runtime via manual testing.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
(cherry picked from commit 7c33186e72)
Allow Rook to rotate CephX keys for Ceph v19.2.3 and up with the
exception of Ceph mons. While v19.2.3 has the `ceph auth rotate` command
that Rook needs to perform key rotation, there is still ongoing work in
Ceph to change how the `mon.` key is tracked internally. Rook's mon key
rotation requires those currently-unreleased Ceph changes.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
(cherry picked from commit a952b7511a)
Cleanup up overall usage of CephX statuses in CephCluster.
Convert all CephX statuses to non-pointer types. There is no specific
need for pointer types, and keeping these as non-pointers means there is
not a risk of nil pointer exceptions in all the various Rook controllers
for upgraded clusters.
Move initialization of CephCluster CephX status items to
`preMonStartupActions()`. This helps resolve 2 issues:
1. 'Uninitialized' state was being set for external CephClusters where
CephX statuses are not relevant.
2. 'Uninitialized' state was being set for upgraded clusters whose key
status cannot be determined.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
(cherry picked from commit 2fd9c9a88c)
rotate mon daemon keys and update the cephx status for mons in the
cephcluster resource.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
(cherry picked from commit fe42fe102f)
Mount /run/udev inside cluster cleanup jobs and disable udev sync
with DM_DISABLE_UDEV.
Related-Issue: rook#16292
Signed-off-by: Denis Egorenko <degorenko@mirantis.com>
(cherry picked from commit 09622a8b8d)
rotate the client.rbd-mirror-peer key based on the cephxconfig.
Also update the cephxStatus in the cephcluster as well as the
mirrored cephblockpool.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
(cherry picked from commit ee0b137d75)
CephConnection CR is created with the
kubernetes namespace name as its the
unique name in the entire cluster.
This PR removes unwanted input
parameters and adjust the namespace
name for the cephConnection CR.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
(cherry picked from commit 1ff3f07ac5)
rotate RBD mirror daemon keys, add secret resource version to the mirror
deployment and update the mirror resource status after the key is
rotated
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
(cherry picked from commit d0fbf6c424)
During mon failover, if a mon is assigned to a node
but the node does not exist, the mon should be failed
over immediately instead of waiting for a 10 or 20
minute timeout. If the node does not exist, the mon pod
will be in a pending state until the failover is finally
triggered, and no point in waiting that long if the
node is gone.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
(cherry picked from commit 8b4d3b19cf)
add a validation check to fail osd deployment when topology labels
like rack or chassis are duplicated across parent domains
(e.g., zones). this prevents invalid crush map hierarchies
(dfbugs-2610).
Signed-off-by: Oded Viner <oviner@redhat.com>
(cherry picked from commit 973f8a3a35)
rotate cephx keys based on keyRotationPolicy
and update cephxStatus for the ceph exporter daemon pods.
Signed-off-by: parth-gr <partharora1010@gmail.com>
(cherry picked from commit 9ec7a767c3)
Implement automated CephX key rotation for OSDs.
OSDs have marshalled CephX statuses applied as an annotation on the OSD
pod spec, which differs from other Rook Ceph daemons for a two reasons:
A large cluster could have hundreds or thousands of OSDs, so it is more
critical that Rook avoid rotating OSD keys more than once in the event a
CephCluster reconcile restarts partway through OSD creation/updation.
OSDs do not have their CephX keys stored in K8s Secrets, whose resource
version is used for enforcing the desired restart behavior other daemon
pods like RGWs.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
(cherry picked from commit 4189a196c1)
rotate cephx keys based on keyRotationPolicy
and update cephxStatus for the crashcollector
daemon pods.
Signed-off-by: parth-gr <partharora1010@gmail.com>
add RecoverAndLogException() helper to log panics with stack trace.
added defer call in all rook controller Reconcile() methods for
better error visibility in operator logs
Signed-off-by: Oded Viner <oviner@redhat.com>
This PR implements the CSI cephx key rotation feature,
which follows overlapping rotation for non-daemon keys.
Refer to the cephx rotation API from design PR 15915 for details.
Signed-off-by: subhamkrai <srai@redhat.com>
When a mon is failed over, the old mon is left in mon quorum
until the new quorum is confirmed in quorum. When host networking
is enabled, this may result in two mons being in quorum with the
same ip, if the new mon is scheduled on the same node as the
down mon. This results in daemon crashes in the cluster since
they are confused by multiple of the same ip in the mon map.
The solution is to avoid scaling down the bad mon when host networking
is enabled. This will ensure the new mon is not scheduled on the
same node as the previous mon. With host networking, either another
node is necessary or the original mon must come back online to
restore quorum.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Santosh Pillai
Travis Nielsen
Blaine Gardner
parth-gr
Oded Viner
subhamkrai
Denis Egorenko
Madhu Rajanna
mergify[bot]
chentao.2022
raaizik
Muhammad Adil Ghaffar
Kyori Sakao