Commit Graph
24 Commits
Author SHA1 Message Date
Rohan CJ e408c5fd7f Ceph: Add delay between drain switches
Earlier, simultaneous drains could allow the drain detection to switch
disabled PDBs before the Ceph health had fully reflected the effects.

Signed-off-by: Rohan CJ <rohantmp@gmail.com>
2019-11-21 07:22:22 +05:30
Travis Nielsen 93543a2565 Merge pull request #4321 from leseb/reorganize-design-dir
design: add backends to their respective directories
2019-11-15 11:01:15 -07:00
Sébastien Han 91b76cc24c doc: update design links
Since we re-organized the design directory we need to update the links
in various files.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-11-15 11:29:46 +01:00
Rohan CJ e8198d44d5 Ceph: Fix bug: clusterdisruption noout timestamp was being reset periodically
The timestamp was being periodically cleared and reset, so the noout never expired.

Signed-off-by: Rohan CJ <rohantmp@gmail.com>
2019-11-14 15:29:12 +05:30
Rohan CJ e51e9f3ec9 Ceph: Ensure draining state is set and checked correctly.
- Checking whether drain was incorrectly checked precense in map instead
  of string length. The actual zero value was an empty string.
- Unsetting drain did not check if the cluster was clean.

Signed-off-by: Rohan CJ <rohantmp@gmail.com>
2019-11-14 13:20:55 +05:30
Rohan CJ 1ba2633c27 Ceph: Clean up canaries on nodes that no longer have OSDs or when the node is deleted.
This prevents long-lasting false positives that prevent the ordered progression
of drains.

Signed-off-by: Rohan CJ <rohantmp@gmail.com>
2019-11-13 16:04:45 +05:30
Rohan CJ 9ec12a587b Ceph: Fix node name checking in the clusterdisruption controller.
Signed-off-by: Rohan CJ <rohantmp@gmail.com>
2019-11-13 16:04:24 +05:30
Rohan CJ 4d7c6df034 Ceph: Fix topologyAware in clusterdisruption package does not respect the rook topolgy prefix.
Allow the clusterdisruption package to recognize node topologies other than zone.

Signed-off-by: Rohan CJ <rohantmp@gmail.com>
2019-11-11 09:38:33 +05:30
Sébastien Han 7dac59f68e ceph: use %q to print
Since I was parsing this file with the previous commits let's add some
cosmetic changes.

%q is more readable since it doubles quotes the variable.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-11-08 16:04:57 +01:00
Sébastien Han f45bdb119d ceph: print the full err fields
Let's print the whole struct and its fields.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-11-08 16:03:34 +01:00
Sébastien Han d8b5d52438 ceph: hide wrong error
WriteConnectionConfig() is called after the controller starts and thus
the 'osd find' command fails to complete because there is no ceph.conf
or key on the operator.
Let's skip this error print if that's the case since it's misleading and
polluting the logs.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-11-08 16:01:02 +01:00
Sébastien Han de172b96e8 ceph: add missing errors
The current code wasn't returning the full error properly.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-11-08 16:00:54 +01:00
Sébastien Han 386ec12032 ceph: fix error message
Re-phrase error message and use %q to put the var into quotes.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-11-08 15:59:08 +01:00
Travis Nielsen 831c15086a ceph: support all layers of CRUSH map with node labels
The node labels were already supported for zones and regions to add to
the CRUSH map. Now all layers of the CRUSH map will be supported
with the new labels such as topology.rook.io/rack.
The labels will be detected at the osd startup time
similar to the zone and region labels already being detected.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2019-10-31 14:47:18 -06:00
Sébastien Han 05140d7a9e ceph: hide wrong error for clusterdisruption controller
The clusterdisruption starts up before the Ceph cluster is created and
during the initilization, it'll display invalid error messages such as:

2019-10-14T04:22:21.243679454Z 2019-10-14 04:22:21.243668 E | clusterdisruption-controller: could not check cluster health: failed to get status: exit status 1
2019-10-14T04:22:22.524222494Z 2019-10-14 04:22:22.524179 I | exec: Running command: ceph status --connect-timeout=15 --cluster=openshift-storage --conf=/var/lib/rook/openshift-storage/openshift-storage.config --keyring=/var/lib/rook/openshift-storage/client.admin.keyring --format json --out-file /tmp/605071670
2019-10-14T04:22:22.617370007Z 2019-10-14 04:22:22.617321 I | exec: Error initializing cluster client: ObjectNotFound('error calling conf_read_file',)

That error message is invalid since the cluster is not up and running
YET.
So let's simply log a message.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-10-16 11:00:14 +02:00
Rohan CJ 0d9442982b ceph: Fix PDB for MDS is not created if minAvailable <= 1.
It should be minAvailable < 1.

Signed-off-by: Rohan CJ <rohantmp@gmail.com>
2019-10-10 21:34:20 +05:30
Rohan CJ 3c5856fc2a Ceph: Take activeStandby into account for CephFileSystem disruption budget.
When activeStandby is set on the CephFilesystem, the effective
activeCount will be incremented.

Signed-off-by: Rohan CJ <rohantmp@gmail.com>
2019-10-10 21:31:58 +05:30
Sébastien Han 233df963e7 ceph: fix various spelling issues
Correct spelling.

Signed-off-by: Sébastien Han <seb@redhat.com>
2019-10-03 11:42:40 +02:00
Travis Nielsen 6e4b40b842 Merge pull request #3786 from rohantmp/drainFix
Fix: only schedule node drain canaries on nodes with osds
2019-09-09 22:53:15 +03:00
travisnandAshish Ranjan 638b591535 ceph: Adds support for MDB controller and machineLabel controller
Add support for MachineDisruptionBudget controller and MachineLabel controller required for fencing
in OpenShift. This will ensure that machines are only fenced and OSDs are only stopped
when Ceph is in a healthy state.

Co-authored-by: Ashish Ranjan <aranjan@redhat.com>
Signed-off-by: travisn <tnielsen@redhat.com>
2019-09-06 16:32:06 -06:00
Rohan CJ f8b7537830 Fix topologyAware on PVC-based OSDs
Signed-off-by: Rohan CJ <rohantmp@gmail.com>
2019-09-06 23:17:01 +05:30
Rohan CJ 7d3f313cab clusterdisruption: Bypass crush location cache if active failureDomain is not populated
Signed-off-by: Rohan CJ <rohantmp@gmail.com>
2019-09-06 09:43:48 +05:30
travisn a9deac7d19 ceph: clean up verbose logging
The log messages for orchestrations being skipped due to nodes still
being schedulable or unschedulable is filling the logs and not necessary.
The log message for reconciling is now debug, and also fixed another
formatting log message.

Signed-off-by: travisn <tnielsen@redhat.com>
2019-08-30 15:01:35 -06:00
Rohan CJ 4072d14557 Add tests for the clusterdisruption controller and update the docs and
pending releas notes for managed PDBs.

Signed-off-by: Rohan CJ <rohantmp@gmail.com>
2019-08-30 14:04:23 +05:30