forked from rook/rook
Elias Carter
60a15306e9
Currently, when there are down OSDs but all PGs are clean we add the
number of downed OSDs ot the main PDB maxUnavailable. For example if
OSDs 1,3,5 are down and all PGs are clean we create the following PDB:
```yaml
maxUnavailable: 4
selector:
matchLabels:
app: rook-ceph-osd
```
This is potentially dangerous as there can be race conditions which can
break the cluster, for example:
1. A new node is added to the cluster with 4 OSDs
2. Those OSDs are down as they have not started up and joined the cluster yet
3. All PGs are healthy as the new OSDs haven't started yet and they are new
4. The rook operator sets maxUnavailable to 5 on the OSD PDB
5. The new OSDs move from down to up as they start
6. Something in the cluster starts a disruption (eviction request)
7. Any 5 OSDs can be disrupted as maxUnavilable is still 5 and the downed OSDs are now up
8. If the 5 disrupted OSDs contain all the replicas of a PG it will cause the PG to go down
To better protect from this scenario, we can instead exclude the downed
OSDs from the main PDB instead of adding to maxUnavailable.
```yaml
maxUnavailable: 1
selector:
matchExpressions:
- key: app
operator: In
values: ["rook-ceph-osd"]
- key: osd
operator: NotIn
values: ["1", "3", "5"]
```
This allows drains to still process when there are downed OSDs and PGs
are clean without allowing the non-downed OSDs to be disrupted.
Signed-off-by: Elias Carter <elias@dropbox.com>
Rook Feature Proposal
Please follow the design section guideline.