This fixes OSD info being incomplete when a sidecar container is first
in the .spec.Template.Spec.Containers list.
I added a small helper function to find a container named "osd" instead
of always assuming the first one.
Signed-off-by: Jan Breitkopf <jan.breitkopf@prorocketeers.com>
When writing to the operator log, where possible
the namespace of the resource on which the operator
is working will be logged as a standard prefix
to more easily troubleshoot what resource is being reconciled
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
In a running cluster, I noticed that a node which was provisioning was removed from k8s
and the following log was emitted by the rook operator:
```
op-osd: cleaning up dangling OSD prepare status configmap "rook-ceph-osd-xxx-status"
```
But the osd provisioning job was still trying to run on the removed node:
```
rook-ceph-osd-prepare-xxx Running 0/1 4d22h 4d22h
```
This PR implements a mechanism to remove any osd provision jobs which are bound to a
node which no longer exists within kubernetes. I added a test for this cleanup as
there wasn't a pre-existing test.
Signed-off-by: Elias Carter <elias@dropbox.com>
when the osd is created there is no need for
osd bootstrap keyring, so remove the keyring
and keyring file,
New keyring will be created if
new osd need to be provisoned
Signed-off-by: parth-gr <partharora1010@gmail.com>
Cleanup up overall usage of CephX statuses in CephCluster.
Convert all CephX statuses to non-pointer types. There is no specific
need for pointer types, and keeping these as non-pointers means there is
not a risk of nil pointer exceptions in all the various Rook controllers
for upgraded clusters.
Move initialization of CephCluster CephX status items to
`preMonStartupActions()`. This helps resolve 2 issues:
1. 'Uninitialized' state was being set for external CephClusters where
CephX statuses are not relevant.
2. 'Uninitialized' state was being set for upgraded clusters whose key
status cannot be determined.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
add a validation check to fail osd deployment when topology labels
like rack or chassis are duplicated across parent domains
(e.g., zones). this prevents invalid crush map hierarchies
(dfbugs-2610).
Signed-off-by: Oded Viner <oviner@redhat.com>
Implement automated CephX key rotation for OSDs.
OSDs have marshalled CephX statuses applied as an annotation on the OSD
pod spec, which differs from other Rook Ceph daemons for a two reasons:
A large cluster could have hundreds or thousands of OSDs, so it is more
critical that Rook avoid rotating OSD keys more than once in the event a
CephCluster reconcile restarts partway through OSD creation/updation.
OSDs do not have their CephX keys stored in K8s Secrets, whose resource
version is used for enforcing the desired restart behavior other daemon
pods like RGWs.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
One ore more OSDs can be down and PGs can still be active+clean (data
was rebalanced to other available OSDs). This PR sets
maxunavailable=1+downOSDs to allow one healthy to be drained.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
Some ceph.conf settings need to be unique per node
depending on the hardware. The ceph.conf overrides
can now be customized per-node by creating a node-specific
configmap that will be loaded for all OSDs and OSD prepare
jobs on that node, instead of the default ceph.conf
that is loaded from the rook-config-override configmap.
The node-specific configmaps must have the label
node.config.rook.io/osd, and be named as:
rook-config-override-<hostname>
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
GetOSDResources method was populating
wrong device class to get the osd
resources
This fix will fix the deviceClass label
for the GetOSDResources function
Signed-off-by: parth-gr <partharora1010@gmail.com>
currently device class and device type
labels were clubbed with eachother
create a seperate label, as these two labels
can be seperate
Signed-off-by: parth-gr <partharora1010@gmail.com>
MustDelete DeleteOption was set to `true` when deleting
osd-repalce-config configMap. This cm won't existing after
all the OSDs get migrated. So deletion of the non-existent
cm will result in error. This PR removes `mustDelete: true` from
the DeletionOption when deleting osd-replace-config configmap
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
Normally the device class of an OSD is determined at provisioning
and is not updated thereafter. In some scenarios the admin may
want to force update the device class to a new value. The device
classes can be updated by first setting allowDeviceClassUpdate
in the storage spec of the cephcluster, then updating the
device class specified on the deviceSets or other OSDs.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Currently we allow upgrade of other OSDs while migration
of OSD, due to change in backing store, is pending. This
will not work if the updated OSD ceph version does not support
the currently applied backing store. So rook will skip any
OSD upgrade if the OSD migration is pending.
Signed-off-by: sp98 <sapillai@redhat.com>
LVM-based OSDs on PVCs are legacy and are not well tested.
Therefore, we add status to the cephcluster CR that at least
gives a clue that the OSDs should be replaced.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
If device set names are duplicated, the OSDs will be very confused
when multiple OSDs attempt to be configured with the same
device set names. Updating the device set names does not help
get out of the error situation since the OSDs will also be
confused about which device set they belong to. The reconcile
must fail completely if the device set names are duplicated
to avoid this error scenario.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
If migration of the OSDs is enabled, operator should wait
for the pgs to be clean after the all the OSDs have reconciled.
This will help add the exponential backoff for the operator
restarts when OSD migration is needed but PGs are still rebalancing the
data after last OSD migration.
Signed-off-by: sp98 <sapillai@redhat.com>
This patch adds `pgHealthyRegex` field to DisruptionManagementSpec.
`pgHealthyRegex` is a regular expression that is used to determine which
PG states should be considered healthy. The default value of
`pgHealthyRegex` is:
^(active\+clean|active\+clean\+scrubbing|active\+clean\+scrubbing\+deep)$
which is effectively the same as before.
Signed-off-by: Ryotaro Banno <ryotaro.banno@gmail.com>
During certain maintenance tasks the admin will own running
operations on the ceph mgr, rgw, mds and rbd-mirror daemons
and the operator should not interfere with those operations.
Co-authored-by: gauravsitlani <gaurav.sitlani@live.com>
Signed-off-by: subhamkrai <srai@redhat.com>
When OSDs flap, ceph stops the OSD daemon if its marked down greater than
5 times in 600 seconds. But OSD pod restarts and marks the OSD `up` again.
This causes the PGs mapped to these OSDs to peer. While the PGs are peering,
IO to these PGs are blocked.
So we need to ensure that if ceph is marking OSD `down` due to flapping, OSD pod
should not restart to mark the OSDs `up` again.
This PR adds a sleep to the OSD pod if the container returned with a 0 exit code
Default behavior is to sleep for 6 hrs. But user can configure it from the
ceph cluster spec.
Signed-off-by: sp98 <sapillai@redhat.com>
If osd store is updated in the ceph cluster, then
delete OSDs one by one, cleanup disks and provision a new OSD on
the same disk
Signed-off-by: sp98 <sapillai@redhat.com>
This commit adds code to supply variables required
to enable read affinity for rbd volumes.
It leverages `read_from_replica=localize,crush_locations=key:value`
krbd map option from ceph to serve reads from the most local OSD.
refer:
https://docs.ceph.com/en/latest/man/8/rbd/#kernel-rbd-krbd-options
Signed-off-by: Rakshith R <rar@redhat.com>
The tool gofmt in go 1.19 requires certain formatting
in the comments section for better rendering.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
During certain maintenance tasks the admin will own running
operations on the ceph mons and osds, and the operator should
not interfere with those operations. If the operator sees
any mon in debug mode, every reconcile and mon health check
will be skipped. Thus, mons will not be updated while any
one of them is in maintenance. During OSD reconcile, individual
OSD deployment updates will only be skipped for OSDs that are
actively being debugged.
The debug mode for osd and mon deployments is signaled by
creating the ceph.rook.io/do-not-reconcile label.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
With octopus coming to end of life, we remove support from
Rook for deploying Ceph Octopus and assume a min version of
Pacific v16. Any checks for octopus or earlier are removed
from the reconciles since they are obsolete.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Once we are done cleaning up the content of the encrypted osd, let's
close the main LUKS device.
This avoids having dm devices on the system after the cleanup.
Closes: https://github.com/rook/rook/issues/10181
Signed-off-by: Sébastien Han <seb@redhat.com>
This removes double package imports. Example:
```
"github.com/rook/rook/pkg/apis/ceph.rook.io/v1"
cephv1 "github.com/rook/rook/pkg/apis/ceph.rook.io/v1"
```
Only one is now being used as shown in go-staticcheck ST1019
Signed-off-by: Alexander Trost <galexrt@googlemail.com>
After all OSDs are updated to each major release of Ceph, the
require-osd-release version is set to that release so all the latest
functionality for osds can be enabled by Ceph. In Quincy there is
a warning if all osds are on Quincy and this flag is not set, so
this will resolve the health warning.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
This commit adds context parameter to k8sutil node functions. By this,
we can handle cancellation during API call of node resource.
Signed-off-by: Yuichiro Ueno <y1r.ueno@gmail.com>
This commit is a large refactor on how the operator starts, stops and
how it starts various sub-components such as the ceph-csi driver. It
also refines the way we cancel orchestrations. We don't use breakpoints
anymore but send our self a SIGUP to reload our controller runtime
manager.
The reload will happen under different circonstances like:
* a new adminission controller secret is created/deleted/changed
* a CephCluster CR is edited
As mentioned earlier, the csi driver now has its own controller, just
like flex. It reacts to change in the operator config map for particular
ROOK_CSI_ fields.
A second new controller for the operator's general config has been
created, it manages:
* the logging level
* the ceph CLI command timeout
* the discovery daemon
The operator reacts much more rapidly to cancellation events by stopping
the manager's context and reloading it.
Signed-off-by: Sébastien Han <seb@redhat.com>