docs: add design to update ceph osds in parallel
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
This commit is contained in:
2 files changed
+172
-3
No files matched your search
@@ -808,7 +808,7 @@ spec:
|
||||
|
||||
The topology of the cluster is important in production environments where you want your data spread across failure domains. The topology
|
||||
can be controlled by adding labels to the nodes. When the labels are found on a node at first OSD deployment, Rook will add them to
|
||||
the desired level in the [CRUSH map](http://docs.ceph.com/docs/master/rados/operations/crush-map/).
|
||||
the desired level in the [CRUSH map](https://docs.ceph.com/en/latest/rados/operations/crush-map/).
|
||||
|
||||
The complete list of labels in hierarchy order from highest to lowest is:
|
||||
|
||||
@@ -828,7 +828,7 @@ For example, if the following labels were added to a node:
|
||||
|
||||
```console
|
||||
kubectl label node mynode topology.kubernetes.io/zone=zone1
|
||||
kubectl label node mynode topology.rook.io/rack=rack1
|
||||
kubectl label node mynode topology.rook.io/rack=zone1-rack1
|
||||
```
|
||||
|
||||
> For versions previous to K8s 1.17, use the topology key: failure-domain.beta.kubernetes.io/zone or region
|
||||
@@ -840,7 +840,7 @@ These labels would result in the following hierarchy for OSDs on that node (this
|
||||
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
|
||||
-1 0.01358 root default
|
||||
-5 0.01358 zone zone1
|
||||
-4 0.01358 rack rack1
|
||||
-4 0.01358 rack zone1-rack1
|
||||
-3 0.01358 host mynode
|
||||
0 hdd 0.00679 osd.0 up 1.00000 1.00000
|
||||
1 hdd 0.00679 osd.1 up 1.00000 1.00000
|
||||
|
||||
@@ -0,0 +1,169 @@
|
||||
# Updating OSDs in parallel
|
||||
**Targeted for v1.6**
|
||||
|
||||
## Background
|
||||
In clusters with large numbers of OSDs, it can take a very long time to update all of the OSDs. This
|
||||
occurs on updates of Rook and Ceph both for major as well as the most minor updates. To better
|
||||
support large clusters, Rook should be able to update (and upgrade) multiple OSDs in parallel.
|
||||
|
||||
|
||||
## High-level requirements
|
||||
|
||||
### Ability to set a maximumum number of OSDs to be updated simultaneously
|
||||
In the worst (but unlikely) case, all OSDs which are updated for a given parallel update operation
|
||||
might fail to come back online after they are updated. Users may wish to limit the number of OSDs
|
||||
updated in parallel in order to avoid too many OSDs failing in this way.
|
||||
|
||||
### Cluster growth takes precedence over updates
|
||||
Adding new OSDs to a cluster should occur as quickly as possible. This allows users to make use of
|
||||
newly added storage as quickly as possible, which they may need for critical applications using the
|
||||
underlying Ceph storage. In some degraded cases, adding new storage may be necessary in order to
|
||||
allow currently-running Ceph OSDs to be updated without experiencing storage cluster downtime.
|
||||
|
||||
This does not necessarily mean that adding new OSDs needs to happen before updates.
|
||||
|
||||
This prioritization might delay updates significantly since adding OSDs not only adds capacity to
|
||||
the Ceph cluster but also necessitates data rebalancing. Rebalancing generates data movement which
|
||||
needs to settle for updates to be able to proceed.
|
||||
|
||||
### OSD updates should not starve other resources of updates
|
||||
For Ceph cluster with huge numbers of OSDs, Rook's process to update OSDs should not starve other
|
||||
resources out of the opportunity to get configuration updates.
|
||||
|
||||
|
||||
## Technical implementation details
|
||||
|
||||
### Changes to Ceph
|
||||
The Ceph manager (mgr) will add functionality to allow querying the maximum number of OSDs that are
|
||||
okay to stop safely. The command will take an initial OSD ID to include in the results. It should
|
||||
return error if the initial OSD cannot be stopped safely. Otherwise it returns a list of 1 or more
|
||||
OSDs that can be stopped safely in parallel. It should take a `--max=<int>` parameter that limits
|
||||
the number of OSDs returned.
|
||||
|
||||
It will look similar to this on the command line `ceph osd ok-to-stop $id --max $int`.
|
||||
|
||||
The command will have an internal algorithm that follows the flow below:
|
||||
1. Query `ok-to-stop` for the "seed" OSD ID. This represents the CRUSH hierarchy bucket at the "osd"
|
||||
(or "device") level.
|
||||
2. If the previous operation reports that it safe to update, batch query `ok-to-stop` for all OSDs
|
||||
that fall under the CRUSH bucket one level up from the current level.
|
||||
3. Repeat step 3 moving up the CRUSH hierarchy until one of the following two conditions:
|
||||
1. The number of OSDs in the batch query is greater than or equal to the `max` parameter, OR
|
||||
2. It is no longer `ok-to-stop` all OSDs in the CRUSH bucket.
|
||||
4. Update OSD Deployments in parallel for the last CRUSH bucket where it was `ok-to-stop` the OSDs.
|
||||
- If there are more OSDs in the CRUSH bucket than allowed by the user that are okay to stop,
|
||||
return only the `max` number of OSD IDs from the CRUSH bucket.
|
||||
|
||||
The pull request for this feature in the Ceph project can be found at
|
||||
https://github.com/ceph/ceph/pull/39455.
|
||||
|
||||
### Rook Operator Workflow
|
||||
1. Build an "existence list" of OSDs which already have Deployments created for them.
|
||||
1. Build an "update queue" of OSD Deployments which need updated.
|
||||
1. Start OSD prepare Jobs as needed for OSDs on PVC and OSDs on nodes.
|
||||
1. Note which prepare Jobs are started
|
||||
1. Provision Loop
|
||||
1. If all prepare Jobs have been completed and the update queue is empty, stop Provision Loop.
|
||||
1. If there is a `CephCluster` update/delete, stop Provision Loop with a special error.
|
||||
1. Create OSDs: if a prepare Job has completed, read the results.
|
||||
1. If any OSDs reported by prepare Job do not exist in the "existence list", create them.
|
||||
1. Mark the prepare Job as completed.
|
||||
1. Restart Provision Loop.
|
||||
1. Update OSDs: if the update queue is not empty, update a batch of OSD Deployments.
|
||||
1. Query `ceph osd ok-to-stop <osd-id> --max=<int>` for each OSD in the update queue until
|
||||
a list of OSD IDs is returned.
|
||||
- If no OSDs in the update queue are okay to stop, Restart Provision Loop.
|
||||
1. Update all of the OSD Deployments in parallel.
|
||||
1. Record any failures.
|
||||
1. Remove all OSDs from the batch from the update queue (even failures).
|
||||
1. Restart Provision Loop.
|
||||
1. If there are any recorded errors/failures, return with an error. Otherwise return success.
|
||||
|
||||
### How to make sure Ceph cluster updates are not starved by OSD updates
|
||||
Because [cluster growth takes precedence over updates](#cluster-growth-takes-precedence-over-updates),
|
||||
it could take a long time for all OSDs in a cluster to be updated. In order for Rook to have
|
||||
opportunity to reconcile other components of a Ceph cluster's `CephCluster` resource, Rook should
|
||||
ensure that the OSD update reconciliation does not create a scenario where the `CephCluster` cannot
|
||||
be modified in other ways.
|
||||
|
||||
https://github.com/rook/rook/pull/6693 introduced a means of interrupting the current OSD
|
||||
orchestration to handle newer `CephCluster` resource changes. This functionality should remain so
|
||||
that user changes to the `CephCluster` can begin reconciliation quickly. The Rook Operator should
|
||||
stop OSD orchestration on any updates to the `CephCluster` spec and be able to resume OSD
|
||||
orchestration with the next reconcile.
|
||||
|
||||
### How to build the existence list
|
||||
List all OSD Deployments belonging to the Rook cluster. Build a list of OSD IDs matching the OSD
|
||||
Deployments. Record this in a data structure that allows O(1) lookup.
|
||||
|
||||
### How to build the update queue
|
||||
List all OSD Deployments belonging to the Rook cluster to use as the update queue. Ignore OSD
|
||||
Deployments which are already updated. OSD Deployments which need updated will have a
|
||||
`rook-version` label that does not match the current version of the Rook operator and a
|
||||
`ceph-version` label that does not match the current Ceph version being deployed.
|
||||
|
||||
Go channels are likely not the best option for the update queue since it may be necessary to select
|
||||
individual items from the queue at any position and not merely grab the top/bottom item from the
|
||||
queue.
|
||||
|
||||
The minimal information each item in the queue needs is only the OSD ID. The OSD Deployment managed
|
||||
by Rook can easily be inferred from the OSD ID.
|
||||
|
||||
### User configuration
|
||||
|
||||
#### Modifications to `CephCluster` CRD
|
||||
Establish a new `updatePolicy` section in the `CephCluster` `spec`. In this section, users can
|
||||
set options for how OSDs should be updated in parallel. Additionally, we can move some existing
|
||||
one-off configs related to updates to this section for better coherence. This also allows for
|
||||
a natural location where future update options can be added.
|
||||
|
||||
```yaml
|
||||
apiVersion: ceph.rook.io/v1
|
||||
kind: CephCluster
|
||||
# ...
|
||||
spec:
|
||||
# ...
|
||||
# Move these to the new updatePolicy but keep them here for backwards compatibility.
|
||||
# These can be marked deprecated, but do not remove them until CephCluster CRD v2.
|
||||
skipUpgradeChecks:
|
||||
continueUpgradeAfterChecksEvenIfNotHealthy:
|
||||
removeOSDsIfOutAndSafeToDestroy:
|
||||
|
||||
# Specify policies related to updating the Ceph cluster and its components. This applies to
|
||||
# minor updates as well as upgrades.
|
||||
updatePolicy:
|
||||
# skipUpgradeChecks is merely relocated from spec
|
||||
skipUpgradeChecks: <bool, default=false>
|
||||
|
||||
# continueUpgradeAfterChecksEvenIfNotHealthy is merely relocated from spec
|
||||
continueUpgradeAfterChecksEvenIfNotHealthy: <bool, default=false, relocated from spec>
|
||||
|
||||
# allow for future additions to updatePolicy like healthErrorsToIgnore
|
||||
|
||||
# Update policy for OSDs.
|
||||
osds:
|
||||
# removeIfOutAndSafeToDestroy is merely relocated from spec (removeOSDsIfOutAndSafeToRemove)
|
||||
removeIfOutAndSafeToDestroy: <bool, default=false>
|
||||
|
||||
# Max number of OSDs in the cluster to update at once. Rook will try to update this many OSDs
|
||||
# at once if it is safe to do so. It will update fewer OSDs at once if it would be unsafe to
|
||||
# update maxInParallelPerCluster at once. This can be a discrete number or a percentage of
|
||||
# total OSDs in the Ceph cluster.
|
||||
# Rook defaults to updating 15% of OSDs in the cluster simultaneously if this value is unset.
|
||||
# Inspired by Kubernetes apps/v1 RollingUpdateDeployment.MaxUnavailable.
|
||||
# Note: I think we can hide the information about CRUSH from the user since it is not
|
||||
# necessary for them to understand that complexity.
|
||||
maxInParallelPerCluster: <k8s.io/apimachinery/pkg/util/intstr.intOrString, default=15%>
|
||||
```
|
||||
|
||||
Default `maxInParallelPerCluster`: Ceph defaults to keeping 3 replicas of an item or 2+1 erasure
|
||||
coding. It should be impossible to update more than one-third (33.3%) of a default Ceph cluster at
|
||||
any given time. It should be safe and fairly easy to update slightly less than half of one-third at
|
||||
once, which rounds down to 16%. 15% is a more round number, so that is chosen instead.
|
||||
|
||||
|
||||
## Future considerations
|
||||
Some users may wish to update OSDs in a particular failure domain or zone completely before moving
|
||||
onto updates in another zone to minimize risk from updates to a single failure domain. This is out
|
||||
of scope for this initial design, but we should consider how to allow space to more easily implement
|
||||
this change when it is needed.
|
||||
Reference in new issue
Block a user