docs: disaster recovery guide for deleted cephcluster

If the CephCluster CR or other CRs are mistakenly deleted,
the cluster can remain intact with steps to remove the owner
references to the cluster and re-creating the CRs.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
This commit is contained in:
Travis Nielsen committed 2021-06-03 12:00:59 -06:00
1 parent b059cb9c75
commit abdaab1a75
2 files changed
+98 -3

No files matched your search

+2 -1
View File
@@ -11,6 +11,7 @@ storage cluster.
* [Prerequisites](#prerequisites)
* [Using alternate namespaces](#using-alternate-namespaces)
* [Deploying a second cluster](#deploying-a-second-cluster)
* [Use custom Ceph user and secret for mounting](#use-custom-ceph-user-and-secret-for-mounting)
* [Log Collection](#log-collection)
* [OSD Information](#osd-information)
@@ -371,7 +372,7 @@ we would see the following defaults after creating a cluster:
kubectl -n rook-ceph get ConfigMap rook-config-override -o yaml
```
```console
```yaml
kind: ConfigMap
apiVersion: v1
metadata:
+96 -2
View File
@@ -6,6 +6,12 @@ indent: true
# Disaster Recovery
Under extenuating circumstances, steps may be necessary to recover the cluster health. There are several types of recovery addressed in this document:
* [Restoring Mon Quorum](#restoring-mon-quorum)
* [Restoring CRDs After Deletion](#restoring-crds-after-deletion)
* [Adopt an existing Rook Ceph cluster into a new Kubernetes cluster](#adopt-an-existing-rook-ceph-cluster-into-a-new-kubernetes-cluster)
* [Backing up and restoring a cluster based on PVCs into a new Kubernetes cluster](#backing-up-and-restoring-a-cluster-based-on-pvcs-into-a-new-kubernetes-cluster)
## Restoring Mon Quorum
Under extenuating circumstances, the mons may lose quorum. If the mons cannot form quorum again,
@@ -231,9 +237,97 @@ kubectl -n rook-ceph scale deployment rook-ceph-operator --replicas=1
The operator will automatically add more mons to increase the quorum size again, depending on the `mon.count`.
# Adopt an existing Rook Ceph cluster into a new Kubernetes cluster
## Restoring CRDs After Deletion
## Situations this section can help resolve
When the Rook CRDs are deleted, the Rook operator will respond to the deletion event to attempt to clean up the cluster resources.
If any data appears present in the cluster, Rook will refuse to allow the resources to be deleted since the operator will
refuse to remove the finalizer on the CRs until the underlying data is deleted. For more details, see the
[dependency design doc](https://github.com/rook/rook/blob/master/design/ceph/resource-dependencies.md).
While it is good that the CRs will not be deleted and the underlying Ceph data and daemons continue to be
available, the CRs will be stuck indefinitely in a `terminating` state in which the operator will not
continue to reconcile. Upgrades will be blocked, further updates to the CRs are prevented, and so on.
Since Kubernetes does not allow undeleting resources, the following procedure will allow you to restore
the CRs to their prior state without even necessarily suffering cluster downtime.
1. Scale down the operator
```console
kubectl -n rook-ceph scale --replicas=0 deploy/rook-ceph-operator
```
2. Backup all Rook CRs and critical metadata
```console
# Store the CephCluster CR settings. Also, save other Rook CRs that are in terminating state.
kubectl -n rook-ceph get cephcluster rook-ceph -o yaml > cluster.yaml
# Backup critical secrets and configmaps in case something goes wrong later in the procedure
kubectl -n rook-ceph get secret -o yaml > secrets.yaml
kubectl -n rook-ceph get configmap -o yaml > configmaps.yaml
```
3. Remove the owner references from all critical Rook resources that were referencing the CephCluster CR.
The critical resources include:
- Secrets: `rook-ceph-admin-keyring`, `rook-ceph-config`, `rook-ceph-mon`, `rook-ceph-mons-keyring`
- ConfigMap: `rook-ceph-mon-endpoints`
- Services: `rook-ceph-mon-*`, `rook-ceph-mgr-*`
- Deployments: `rook-ceph-mon-*`, `rook-ceph-osd-*`, `rook-ceph-mgr-*`
- PVCs (if applicable): `rook-ceph-mon-*` and the OSD PVCs (named `<deviceset>-*`, for example `set1-data-*`)
For example, remove this entire block from each resource:
```yaml
ownerReferences:
- apiVersion: ceph.rook.io/v1
blockOwnerDeletion: true
controller: true
kind: CephCluster
name: rook-ceph
uid: <uid>
```
4. **After confirming all critical resources have had the owner reference to the CephCluster CR removed**, now
we allow the cluster CR to be deleted. Remove the finalizer by editing the CephCluster CR.
```console
kubectl -n rook-ceph edit cephcluster
```
For example, remove the following from the CR metadata:
```yaml
finalizers:
- cephcluster.ceph.rook.io
```
After the finalizer is removed, the CR will be immediately deleted. If all owner references were properly removed,
all ceph daemons will continue running and there will be no downtime.
5. Create the CephCluster CR with the same settings as previously
```shell
# Use the same cluster settings as exported above in step 2.
kubectl create -f cluster.yaml
```
6. If there are other CRs in terminating state such as CephBlockPools, CephObjectStores, or CephFilesystems,
follow the above steps as well for those CRs:
- Backup the CR
- Remove the finalizer and confirm the CR is deleted (the underlying Ceph resources will be preserved)
- Create the CR again
7. Scale up the operator
```shell
kubectl -n rook-ceph --replicas=1 deploy/rook-ceph-operator
```
Watch the operator log to confirm that the reconcile completes successfully.
## Adopt an existing Rook Ceph cluster into a new Kubernetes cluster
Situations this section can help resolve:
1. The Kubernetes environment underlying a running Rook Ceph cluster failed catastrophically, requiring a new Kubernetes environment in which the user wishes to recover the previous Rook Ceph cluster.
2. The user wishes to migrate their existing Rook Ceph cluster to a new Kubernetes environment, and downtime can be tolerated.