When `managePodBudgets` is enabled, the Rook operator sets a blocking
PDB by considering the failure domains of the OSDs. This functionality
is implemented by `reconcilePDBsForOSDs`, and it sets the PDB only after
unhealthy PGs appear. However, there are no unhealthy PGs when an OSD
with no PGs becomes down. In this case, the PDB is never enabled.
This PR makes the operator configure the blocking PDB without waiting
for the unhealthy PGs to appear. This PR solves the above problem
because the blocking PDB is always enabled when a down OSD is detected.
Signed-off-by: Ryotaro Banno <ryotaro.banno@gmail.com>
This patch adds `pgHealthyRegex` field to DisruptionManagementSpec.
`pgHealthyRegex` is a regular expression that is used to determine which
PG states should be considered healthy. The default value of
`pgHealthyRegex` is:
^(active\+clean|active\+clean\+scrubbing|active\+clean\+scrubbing\+deep)$
which is effectively the same as before.
Signed-off-by: Ryotaro Banno <ryotaro.banno@gmail.com>
During certain maintenance tasks the admin will own running
operations on the ceph mons and osds, and the operator should
not interfere with those operations. If the operator sees
any mon in debug mode, every reconcile and mon health check
will be skipped. Thus, mons will not be updated while any
one of them is in maintenance. During OSD reconcile, individual
OSD deployment updates will only be skipped for OSDs that are
actively being debugged.
The debug mode for osd and mon deployments is signaled by
creating the ceph.rook.io/do-not-reconcile label.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Some context canceled were missed and not reported correctly. We had
multiple reconcilers running in //. Since they all run in goroutine then
will continue to run until they timeout (if timeout is available,
otherwise would run forever until the op is restarted).
Signed-off-by: Sébastien Han <seb@redhat.com>
The controller for disruption budgets is verbose at cluster creation
when the cluster is not yet created, which generates a lot of noise
in the log that makes the cluster look unhealthy. This is an expected
condition, so we can change this logging to debug level.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Rook checks for down OSDs by checking the `ReadyReplicas` count
in the OSD deployement. When an OSD pod goes into CBLO due to
disk failure, there is a delay before this `ReadyReplicas` count
becomes 0. The deplay is very small but may result in rook missing
OSD down event. As a result no blocking PDBs will be created and
only default PDB with `AllowedDisruptions` count as 0 is available.
This PR tries to solve this. The OSD pdb reconciler will be
reconciled again if `AllowedDisruptions` count in the main
PDB is 0.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
This commit update the PodDisruptionBudget policy to use version v1
Updated to policy/v1 as policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+
Closes: https://github.com/rook/rook/issues/7917
Signed-off-by: parth-gr <paarora@redhat.com>
This PR has following changes around OSD pdbs
- Detect node drains more reliably. When OSD is down and OSD pod is not scheduled to any node or if the scheduled node is not ready, then assume that node is draining.
- Don't set no-out flag on the failure domain if the OSD is down due to reasons other than node drain (say disk failure)
- update unit tests
- update design doc to reflect above changes
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
If the ceph cli outputs an error with "error calling conf_read_file"
this means that the operator has not written its ceph configuration
file. Thus ceph cli commands will fail, so we can just ignore that since
the operator will soon write this file in its initialization sequence.
Signed-off-by: Sébastien Han <seb@redhat.com>
Currently the reconcile of the clusterDisruption controller was triggered mainly due to the
ceph status update in the cephCluster CR. This PR adds reconciles the cluster when:
1. Reconcile when the cluster is created. (This will trigger the first reconcile)
2. Reconcile only when the clusterSpec is updated. (This will avoid triggers when cluster status is updated)
3. Reconcile for events on cephblockpool, cephfilesystem and cephObjectStore.
4. Reconcile for events on Main PDB and when `DisruptionsAllowed` is 0. (that is, when one of the OSD goes down).
5. Reconcile after 30 seconds when there is an active drain going on, that is, pdbStateMap has `draining-failure-domain` as not empty.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
-creates a single PDB (max-unavailable=1) for all OSDs. This PDB allows one OSD to go down at a given time.
-When a drain is detected, blocking PDBs (max-unavailable=0) will be created for each failure domain that is not being drained and the main PDB (max-unavilable=1) will be deleted. This will allow all the OSDs in the currently drained failure domain to be removed while blocking the deletion of OSDs in other failure domains.
-Once the PGs are healthy again, the blocking PDBs will be deleted and the main PDB will be restored.
-Add PG healthcheck timeout
-Delete any legacy node drain pods and blocking OSD PDBs
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
this commit handles all the gosec g601
error code (i.e Implicit memory aliasing
of items from a range statement).
Signed-off-by: subhamkrai <subhamkumarrai03@gmail.com>
To provide more context for executing commands in a ceph cluster,
the full clusterInfo is now passed to the ceph execution commands.
All information about the cluster will now be available throughout
all the areas of the operator. The namespace, ceph credentials,
mon endpoints, and other info is a core part of that cluster info.
Arguments passed through the controllers are also simplified for
mons, mgr, osds, and other daemons where the parameters had
become too complex.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
This commit is to handle all those unhandled errors which raises the gosec warning.
Fixed G104: Unhandled Errors are handled now
Signed-off-by: Nizamudeen <nia@redhat.com>
When using the "errors" package, using `%+v` (extended format),
each Frame of the error's StackTrace will be printed in detail.
Let's only print `%v` to print the error.
If the error has a Cause it will be printed recursively.
Basically `%+v` has been replaced with `%v` for all `error` type
interfaces, whether the logger is Info, Warning or Error.
Signed-off-by: Sébastien Han <seb@redhat.com>
Earlier, simultaneous drains could allow the drain detection to switch
disabled PDBs before the Ceph health had fully reflected the effects.
Signed-off-by: Rohan CJ <rohantmp@gmail.com>
- Checking whether drain was incorrectly checked precense in map instead
of string length. The actual zero value was an empty string.
- Unsetting drain did not check if the cluster was clean.
Signed-off-by: Rohan CJ <rohantmp@gmail.com>
The clusterdisruption starts up before the Ceph cluster is created and
during the initilization, it'll display invalid error messages such as:
2019-10-14T04:22:21.243679454Z 2019-10-14 04:22:21.243668 E | clusterdisruption-controller: could not check cluster health: failed to get status: exit status 1
2019-10-14T04:22:22.524222494Z 2019-10-14 04:22:22.524179 I | exec: Running command: ceph status --connect-timeout=15 --cluster=openshift-storage --conf=/var/lib/rook/openshift-storage/openshift-storage.config --keyring=/var/lib/rook/openshift-storage/client.admin.keyring --format json --out-file /tmp/605071670
2019-10-14T04:22:22.617370007Z 2019-10-14 04:22:22.617321 I | exec: Error initializing cluster client: ObjectNotFound('error calling conf_read_file',)
That error message is invalid since the cluster is not up and running
YET.
So let's simply log a message.
Signed-off-by: Sébastien Han <seb@redhat.com>
piao
Ryotaro Banno
parth-grandBlaine Gardner
Travis Nielsen
Sébastien Han
Divyansh Kamboj
Santosh Pillai
Lars Lehtonen
Satoru Takeuchi
subhamkrai
Nizamudeen
Rohan CJ