For all of the controllers besides the cluster controller,
the logging now includes the namespaced name of the resource
that is being reconciled. This will help with log troubleshooting
to help analyze logs consistently for the resource being
reconciled.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
For Rook v1.19,removing support for Ceph Reef. It is at end of life.
Users on Reef can continue to use Rook v1.18.x or older.
The latest two Ceph versions Squid and Tentacle are supported.
Signed-off-by: subhamkrai <srai@redhat.com>
Add a resource watch to reconcile clusters when any referenced Secrets
in cephConfigFromSecret are updated. This matches the behaviour of
reconciling clusters when cephConfig changes. Without this change, if a
Secret that is referenced by cephConfigFromSecret is updated, then the
cluster is not immediately updated, so the cluster state will not match
the configuration until something else triggers a reconcile.
Resolves#16371
Signed-off-by: Martin Rowe <martin.p.rowe@gmail.com>
The logger names are seen on each line of logging. As
long as they are unique, they really don't need to be
long and descriptive, so let's make them a bit more
concise, and independent from controller names.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
When writing to the operator log, where possible
the namespace of the resource on which the operator
is working will be logged as a standard prefix
to more easily troubleshoot what resource is being reconciled
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Use a sync map for the clusterMap in the cluster controller
and fix other issues to allow for concurrent cluster
reconcile.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The cephcluster controller allows setting concurrent reconciles
with a setting in the operator configmap.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
In a running cluster, I noticed that a node which was provisioning was removed from k8s
and the following log was emitted by the rook operator:
```
op-osd: cleaning up dangling OSD prepare status configmap "rook-ceph-osd-xxx-status"
```
But the osd provisioning job was still trying to run on the removed node:
```
rook-ceph-osd-prepare-xxx Running 0/1 4d22h 4d22h
```
This PR implements a mechanism to remove any osd provision jobs which are bound to a
node which no longer exists within kubernetes. I added a test for this cleanup as
there wasn't a pre-existing test.
Signed-off-by: Elias Carter <elias@dropbox.com>
currently the device class label was added to the osd pod,
we need the device class label on prepare pod,
so to spread across prpeare pods evenly among the faiure domains
per device class
Signed-off-by: parth-gr <partharora1010@gmail.com>
`cluster.monitoringRoutines` stores three types of
daemon health: "mon", "osd", and "status". These are
used by three goroutines respectively, resulting in:
fatal error: concurrent map reads and map writes.
Signed-off-by: Yite Gu <guyite@bytedance.com>
Add optinal hostnetwork field for Ceph Mgr to control whether the
mgr pods to run in host network. If unset, this defaults to
cluster level Network.HostNetwork value. This particulary useful
when user wants to restrict ceph mgr pod from exposing metrics
port in all host interfaces.
Signed-off-by: Sunnatillo <sunnat.samadov@est.tech>
Wait for the mon-canary pods to be terminated before starting the actual
mons. When topology spread constraints are added to evenly spread the
mons, then pod-anti affinity does not allow mons to be created on the
same node where other mons are running, this also includes the
mon-canary pods. We delete the mon-canary pods and then start the actual
mon pods. Sometimes the termination of the canary pods might take more
time, so schedular won't schedular mon pods on that node.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
The mon canary should quickly shut down after the pod is
deleted, since it has no purpose to continue running.
Since the sleep does not catch the SIGTERM as pid 1,
use a bash trap to exit quickly.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The OSDs may require different resources based on their device
class. The overall osd resources were only being used, rather
than properly looking up the specific device class resources.
Now the expected resources will take affect per device class.
Co-authored-by: Matthias Osthues <osthues.matthias@gmail.com>
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
When OSDs start, init containers attempt to get the latest CephX key for
the OSD to allow OSDs to potentially continue working after key rotation.
However, in Ceph corner cases, mons could be unavailable while OSDs
could still be viable with existing on-disk keys. Since key rotation is
a rare event, it's likely that the on-disk keys are still good.
The worst case scenario is that key updates were needed but didn't
succeed in init containers. In this case, instead of the init container
failing, the OSD startup will fail instead. Thus, this should introduce
no extra risk but merely shift the failure to a different container.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
external mode was dependednt on the ceph status to
update the cluster connected condition,
BUt as now the status check will only update the ceph status,
so we will explicitly update the ceph status condition in the
end of the reconcile
Signed-off-by: parth-gr <partharora1010@gmail.com>
currently the health checkup was reporting the
condition and reason to be always connected,
But which might not be the case,
So instead of that dont report the condition and
its phase during the health checkups
Signed-off-by: parth-gr <partharora1010@gmail.com>
if there a bit long to update osds in parallel,
we need to timeout and abort the synchronous loop
and start the cluster reconcile again from the beginning
Signed-off-by: parth-gr <partharora1010@gmail.com>
we are now adding tentacle to the supported versions and add the
upgrade test from squid to tentacle. Also, clean some unused code.
Signed-off-by: subhamkrai <srai@redhat.com>
The dependency blocking unit test was failing with the latest
library updates due to change in behavior that the garbage
collector is not actually deleting the cluster CR after
the dependencies are cleaned up. Instead, check that the
condition indicates the resource has no dependencies remaining.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
If the CephCluster spec is updated before the prevalidation check, then
Cluster.context still refers to the cancelled context of the operator.
This PR just updates the cluster context to use the latest operator
context.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
shouldFailoverMonImmediately has a nil pointer exception with mons on
PVC. Rook-ceph-mon-endpoint does not store node information of the mons
on PVC. So `mapping.Hostname` throws nill pointer
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
when the osd is created there is no need for
osd bootstrap keyring, so remove the keyring
and keyring file,
New keyring will be created if
new osd need to be provisoned
Signed-off-by: parth-gr <partharora1010@gmail.com>
Delete the client.boostrap.* keys. These keys are not created by rook so
it can't rotate them. This PR just delete these keys in the
postMonStartUpAction.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
ceph-volume creates osd-lockbox keys for the encryted osds. This PR
rotates those keys along with the OSD keys.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
this commit add watcher for cm `rook-ceph-operator-config`
to look for updated value of `ROOK_USE_CSI_OPERATOR` before
adding this watcher we had to restart rook operator pod
so that it reconciles and create cephConnection CR.
Signed-off-by: subhamkrai <srai@redhat.com>
Travis Nielsen
subhamkrai
Martin Rowe
Elias Carter
subham rai
huyejia
parth-gr
Yite Gu
Sunnatillo
Joshua Hoblitt
Santosh Pillai
Blaine Gardner
Oded Viner