The logger names are seen on each line of logging. As
long as they are unique, they really don't need to be
long and descriptive, so let's make them a bit more
concise, and independent from controller names.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
If the drained node is added back and one or more others nodes are down
update the blocking pdbs, that is, add blocking pdb to node that was
just added back.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
Currently, when there are down OSDs but all PGs are clean we add the
number of downed OSDs ot the main PDB maxUnavailable. For example if
OSDs 1,3,5 are down and all PGs are clean we create the following PDB:
```yaml
maxUnavailable: 4
selector:
matchLabels:
app: rook-ceph-osd
```
This is potentially dangerous as there can be race conditions which can
break the cluster, for example:
1. A new node is added to the cluster with 4 OSDs
2. Those OSDs are down as they have not started up and joined the cluster yet
3. All PGs are healthy as the new OSDs haven't started yet and they are new
4. The rook operator sets maxUnavailable to 5 on the OSD PDB
5. The new OSDs move from down to up as they start
6. Something in the cluster starts a disruption (eviction request)
7. Any 5 OSDs can be disrupted as maxUnavilable is still 5 and the downed OSDs are now up
8. If the 5 disrupted OSDs contain all the replicas of a PG it will cause the PG to go down
To better protect from this scenario, we can instead exclude the downed
OSDs from the main PDB instead of adding to maxUnavailable.
```yaml
maxUnavailable: 1
selector:
matchExpressions:
- key: app
operator: In
values: ["rook-ceph-osd"]
- key: osd
operator: NotIn
values: ["1", "3", "5"]
```
This allows drains to still process when there are downed OSDs and PGs
are clean without allowing the non-downed OSDs to be disrupted.
Signed-off-by: Elias Carter <elias@dropbox.com>
add RecoverAndLogException() helper to log panics with stack trace.
added defer call in all rook controller Reconcile() methods for
better error visibility in operator logs
Signed-off-by: Oded Viner <oviner@redhat.com>
If the node for an OSD does not exist within kubernetes, the PDB
reconciliation logic will fail to make progress. This can be problematic
as new OSDs added to the cluster may not be protected by OSD PDBs after
they are added since the PDB reconcilation logic is failing.
The solution is to treat non existing OSD nodes as drained which is
logically consistent with treating unschedulable OSD nodes as drained as
you cannot schedule a pod onto a node which does not exist.
Fixes: #16086
Signed-off-by: Elias Carter <elias@dropbox.com>
the maxunavailable count was getting reset back to 1 after 60 seconds.
This PR prevents resetting of the maxUnavailable OSD count.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
All existing controller runtime watches are converted to use "typed"
handlers and predicates instead of operating on `client.Object`. The
intent is to be bug for bug equivalent with the existing logic while
replacing run time type assertions and switch statements with compile
time type constraints and type casts. In several cases, functions using
assertions were split up such that each function only handles a single
Kind at a time. It is hoped that this will improve readability and
maintainability while facilitating future refactoring such as migrating
some watches to using IndexFields.
Of particular note is that the massive switch statement in
`WatchControllerPredicate()` from
`pkg/operator/ceph/controller/predicate.go` has been replaced with
generics, reflection, and splitting the obc logic into its own predicate
function. There are still many helper functions operating on
`client.Object`. These were not updated unless required by the compiler
in order to limit the size of this change. The type safety of these
funcs should be improved as followup work.
It is strongly suggested that going forward, handlers and predicates
only handle a single Kind (generic or not) and that switches / type
assertions are heavily discouraged or forbidden. This PR removed all
but a single switch statement in a predicate, which should be addressed
in future work.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
One ore more OSDs can be down and PGs can still be active+clean (data
was rebalanced to other available OSDs). This PR sets
maxunavailable=1+downOSDs to allow one healthy to be drained.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
The ci was using a pretty old version og golangci-lint.
This updates to the latest version.
Additionally, it silences some
gosec integer conversion overflow false positves
and fixes some real errors of this category
and string format errors found by golangci-lint, while at it.
Co-authored-by: Blaine Gardner <b.blaine.gardner@gmail.com>
Co-authored-by: Travis Nielsen <tnielsen@redhat.com>
Signed-off-by: Michael Adam <obnox@samba.org>
This commit adds optional support to specify the MDS metadata pool name. It defaults to `<fsName>-metadata` as current implementation expects but allows customization if needed e.g. if exisiting naming conventions used `<fsName>_metadata`. Additionally an "preservePoolNames" boolean has been added to indicate that no generated pool names should be used.
Signed-off-by: Tobias Wolf <wolf@b1-systems.de>
Given that Ceph Quincy (v17) is past end of life,
remove Quincy from the supported Ceph versions,
examples, and documentation.
Supported versions now include only Reef and Squid.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
When `managePodBudgets` is enabled, the Rook operator sets a blocking
PDB by considering the failure domains of the OSDs. This functionality
is implemented by `reconcilePDBsForOSDs`, and it sets the PDB only after
unhealthy PGs appear. However, there are no unhealthy PGs when an OSD
with no PGs becomes down. In this case, the PDB is never enabled.
This PR makes the operator configure the blocking PDB without waiting
for the unhealthy PGs to appear. This PR solves the above problem
because the blocking PDB is always enabled when a down OSD is detected.
Signed-off-by: Ryotaro Banno <ryotaro.banno@gmail.com>
This patch adds `pgHealthyRegex` field to DisruptionManagementSpec.
`pgHealthyRegex` is a regular expression that is used to determine which
PG states should be considered healthy. The default value of
`pgHealthyRegex` is:
^(active\+clean|active\+clean\+scrubbing|active\+clean\+scrubbing\+deep)$
which is effectively the same as before.
Signed-off-by: Ryotaro Banno <ryotaro.banno@gmail.com>
This commit adds code to supply variables required
to enable read affinity for rbd volumes.
It leverages `read_from_replica=localize,crush_locations=key:value`
krbd map option from ceph to serve reads from the most local OSD.
refer:
https://docs.ceph.com/en/latest/man/8/rbd/#kernel-rbd-krbd-options
Signed-off-by: Rakshith R <rar@redhat.com>
During certain maintenance tasks the admin will own running
operations on the ceph mons and osds, and the operator should
not interfere with those operations. If the operator sees
any mon in debug mode, every reconcile and mon health check
will be skipped. Thus, mons will not be updated while any
one of them is in maintenance. During OSD reconcile, individual
OSD deployment updates will only be skipped for OSDs that are
actively being debugged.
The debug mode for osd and mon deployments is signaled by
creating the ceph.rook.io/do-not-reconcile label.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Some context canceled were missed and not reported correctly. We had
multiple reconcilers running in //. Since they all run in goroutine then
will continue to run until they timeout (if timeout is available,
otherwise would run forever until the op is restarted).
Signed-off-by: Sébastien Han <seb@redhat.com>
The controller for disruption budgets is verbose at cluster creation
when the cluster is not yet created, which generates a lot of noise
in the log that makes the cluster look unhealthy. This is an expected
condition, so we can change this logging to debug level.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The built-in pools device_health_metrics and .nfs created by ceph
need to be configured for replicas, failure domain, etc.
To support this, we allow the pool to be created as a CR.
Since K8s does not support underscores in the resource names
the operator must translate this special pool name into
the name expected by ceph.
This also sets the basis for allowing filesystem data
pools to specify the desired pool name instead of requiring
a generated name.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
This commit is a large refactor on how the operator starts, stops and
how it starts various sub-components such as the ceph-csi driver. It
also refines the way we cancel orchestrations. We don't use breakpoints
anymore but send our self a SIGUP to reload our controller runtime
manager.
The reload will happen under different circonstances like:
* a new adminission controller secret is created/deleted/changed
* a CephCluster CR is edited
As mentioned earlier, the csi driver now has its own controller, just
like flex. It reacts to change in the operator config map for particular
ROOK_CSI_ fields.
A second new controller for the operator's general config has been
created, it manages:
* the logging level
* the ceph CLI command timeout
* the discovery daemon
The operator reacts much more rapidly to cancellation events by stopping
the manager's context and reloading it.
Signed-off-by: Sébastien Han <seb@redhat.com>
Rook checks for down OSDs by checking the `ReadyReplicas` count
in the OSD deployement. When an OSD pod goes into CBLO due to
disk failure, there is a delay before this `ReadyReplicas` count
becomes 0. The deplay is very small but may result in rook missing
OSD down event. As a result no blocking PDBs will be created and
only default PDB with `AllowedDisruptions` count as 0 is available.
This PR tries to solve this. The OSD pdb reconciler will be
reconciled again if `AllowedDisruptions` count in the main
PDB is 0.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
Both `ExecuteCommandWithOutputFileTimeout()` and
`ExecuteCommandWithOutputFile()` generate unnecessary system calls by
creating/reading/removing files where the stream output of the command
can simply be used. So sticking with `ExecuteCommandWithOutput()` and
`ExecuteCommandWithCombinedOutput()` for reading outputs is sufficient.
Closes: https://github.com/rook/rook/issues/8343
Signed-off-by: Sébastien Han <seb@redhat.com>
This commit update the PodDisruptionBudget policy to use version v1
Updated to policy/v1 as policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+
Closes: https://github.com/rook/rook/issues/7917
Signed-off-by: parth-gr <paarora@redhat.com>
It's better to create PDB of RGW if ObjectStore's `instances` field is 2.
In the current implementation, at least three desired RGW instances are needed
to create this. If `instances` is 1, we can still safely skip the creation
of PDB because it's useless.
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
This PR has following changes around OSD pdbs
- Detect node drains more reliably. When OSD is down and OSD pod is not scheduled to any node or if the scheduled node is not ready, then assume that node is draining.
- Don't set no-out flag on the failure domain if the OSD is down due to reasons other than node drain (say disk failure)
- update unit tests
- update design doc to reflect above changes
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
- Update mon PDB to use maxUnavailable=1 instead of minAvailable. The maxUnavailable will always be 1 irrespective of the number of mons in the cluster
- Move mon PDB reconcile logic from Disruption Controller to Cluster Controller
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
This PR makes create and delete events on pdb a no-op. Update event only triggers reconcile if its the main OSD pdb and allowed disruptions is 0.
This prevents controller to reconcile on pdb events namespaces outside rook ceph.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
If the ceph cli outputs an error with "error calling conf_read_file"
this means that the operator has not written its ceph configuration
file. Thus ceph cli commands will fail, so we can just ignore that since
the operator will soon write this file in its initialization sequence.
Signed-off-by: Sébastien Han <seb@redhat.com>
If there is a single mon, we don't expect the PDBs to be created
and neither do we need to log that it is an error condition.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Currently the reconcile of the clusterDisruption controller was triggered mainly due to the
ceph status update in the cephCluster CR. This PR adds reconciles the cluster when:
1. Reconcile when the cluster is created. (This will trigger the first reconcile)
2. Reconcile only when the clusterSpec is updated. (This will avoid triggers when cluster status is updated)
3. Reconcile for events on cephblockpool, cephfilesystem and cephObjectStore.
4. Reconcile for events on Main PDB and when `DisruptionsAllowed` is 0. (that is, when one of the OSD goes down).
5. Reconcile after 30 seconds when there is an active drain going on, that is, pdbStateMap has `draining-failure-domain` as not empty.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
-creates a single PDB (max-unavailable=1) for all OSDs. This PDB allows one OSD to go down at a given time.
-When a drain is detected, blocking PDBs (max-unavailable=0) will be created for each failure domain that is not being drained and the main PDB (max-unavilable=1) will be deleted. This will allow all the OSDs in the currently drained failure domain to be removed while blocking the deletion of OSDs in other failure domains.
-Once the PGs are healthy again, the blocking PDBs will be deleted and the main PDB will be restored.
-Add PG healthcheck timeout
-Delete any legacy node drain pods and blocking OSD PDBs
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
Travis Nielsen
Santosh Pillai
Elias Carter
Oded Viner
Joshua Hoblitt
Tobias Wolf
subhamkrai
findmyhappy
Liang Zheng
piao
Ryotaro Banno
parth-grandBlaine Gardner
Rakshith R
Sébastien Han
Josh Soref
Divyansh Kamboj
Denis Egorenko
Satoru Takeuchi
Lars Lehtonen