Merge pull request #3555 from feisuzhu/master

doc: Add disaster recovery doc
2019-08-19 18:37:27 +02:00
parent f09441f1e4 b651239d3f
commit 7e1939cc08
1 changed files with 102 additions and 0 deletions
@@ -106,3 +106,105 @@ kubectl create -f operator.yaml
 ```

 The operator will automatically add more mons to increase the quorum size again, depending on the `monCount`.
+
+# Adopt an existing Rook Ceph cluster into a new Kubernetes cluster
+
+## Situations this section can help resolve
+
+1. The Kubernetes environment underlying a running Rook Ceph cluster failed catastrophically, requiring a new Kubernetes environment in which the user wishes to recover the previous Rook Ceph cluster.
+2. The user wishes to migrate their existing Rook Ceph cluster to a new Kubernetes environment, and downtime can be tolerated.
+
+### Prerequisites
+
+1. A working Kubernetes cluster to which we will migrate the previous Rook Ceph cluster.
+2. At least one Ceph mon db is in quorum, and sufficient number of Ceph OSD is `up` and `in` before disaster.
+3. The previous Rook Ceph cluster is not running.
+
+### Overview for Steps below
+1. Start a new and clean Rook Ceph cluster, with old `CephCluster` `CephBlockPool` `CephFilesystem` `CephNFS` `CephObjectStore`.
+2. Shut the new cluster down when it has been created successfully.
+3. Replace ceph-mon data with that of the old cluster.
+4. Replace `fsid` in `secrets/rook-ceph-mon` with that of the old one.
+5. Fix monmap in ceph-mon db.
+6. Fix ceph mon auth key.
+7. Disable auth.
+8. Start the new cluster, watch it resurrect.
+9. Fix admin auth key, and enable auth.
+10. Restart cluster for the final time.
+
+### Steps
+
+Assuming `dataHostPathData` is `/var/lib/rook`, and the `CephCluster` trying to adopt is named `rook-ceph`.
+
+1. Make sure the old Kubernetes cluster is completely torn down and the new Kubernetes cluster is up and running without Rook Ceph.
+1. Backup `/var/lib/rook` in all the Rook Ceph nodes to a different directory. Backups will be used later.
+1. Pick a `/var/lib/rook/rook-ceph/rook-ceph.config` from any previous Rook Ceph node and save the old cluster `fsid` from its content.
+1. Remove `/var/lib/rook` from all the Rook Ceph nodes.
+1. Add identical `CephCluster` descriptor to the new Kubernetes cluster, especially identical `spec.storage.config` and `spec.storage.nodes`, except `mon.count`, which should be set to `1`.
+1. Add identical `CephFilesystem` `CephBlockPool` `CephNFS` `CephObjectStore` descriptors (if any) to the new Kubernetes cluster.
+1. Install Rook Ceph in the new Kubernetes cluster.
+1. Watch the operator logs with `kubectl -n rook-ceph logs -f rook-ceph-operator-xxxxxxx`, and wait until the orchestration has settled.
+1. **STATE:** Now the cluster will have `rook-ceph-mon-a`, `rook-ceph-mgr-a`, and all the auxiliary pods up and running, and zero (hopefully) `rook-ceph-osd-ID-xxxxxx` running. `ceph -s` output should report 1 mon, 1 mgr running, and all of the OSDs down, all PGs are in `unknown` state. Rook should not start any OSD daemon since all devices belongs to the old cluster (which have a different `fsid`). 
+1. Run `kubectl -n rook-ceph exec -it rook-ceph-mon-a-xxxxxxxx bash` to enter the `rook-ceph-mon-a` pod,
+
+    ```shell
+    mon-a# cat /etc/ceph/keyring-store/keyring  # save this keyring content for later use
+    mon-a# exit
+    ```
+
+1. Stop the Rook operator by running `kubectl -n rook-ceph edit deploy/rook-ceph-operator` and set `replicas` to `0`.
+1. Stop cluster daemons by running `kubectl -n rook-ceph delete deploy/X` where X is every deployment in namespace `rook-ceph`, except `rook-ceph-operator` and `rook-ceph-tools`.
+1. Save the `rook-ceph-mon-a` address with `kubectl -n rook-ceph get cm/rook-ceph-mon-endpoints -o yaml` in the new Kubernetes cluster for later use.
+
+1. SSH to the host where `rook-ceph-mon-a` in the new Kubernetes cluster resides.
+    1. Remove `/var/lib/rook/mon-a`
+    2. Pick a healthy `rook-ceph-mon-ID` directory (`/var/lib/rook/mon-ID`) in the previous backup, copy to `/var/lib/rook/mon-a`. `ID` is any healthy mon node ID of the old cluster.
+    3. Replace `/var/lib/rook/mon-a/keyring` with the saved keyring, preserving only the `[mon.]` section, remove `[client.admin]` section.
+    4. Run `docker run -it --rm -v /var/lib/rook:/var/lib/rook ceph/ceph:v14.2.1-20190430 bash`. The Docker image tag should match the Ceph version used in the Rook cluster.
+
+        ```shell
+        container# cd /var/lib/rook
+        container# ceph-mon --extract-monmap monmap --mon-data ./mon-a/data  # Extract monmap from old ceph-mon db and save as monmap
+        container# monmaptool --print monmap  # Print the monmap content, which reflects the old cluster ceph-mon configuration.
+        container# monmaptool --rm a monmap  # Delete `a` from monmap.
+        container# monmaptool --rm b monmap  # Repeat, and delete `b` from monmap.
+        container# monmaptool --rm c monmap  # Repeat this pattern until all the old ceph-mons are removed
+        container# monmaptool --rm d monmap
+        container# monmaptool --rm e monmap
+        container# monmaptool --add a 10.77.2.216:6789 monmap   # Replace it with the rook-ceph-mon-a address you got from previous command.
+        container# ceph-mon --inject-monmap monmap --mon-data ./mon-a/data  # Replace monmap in ceph-mon db with our modified version.
+        container# rm monmap
+        container# exit
+        ```
+
+1. Tell Rook to run as old cluster by running `kubectl -n rook-ceph edit secret/rook-ceph-mon` and changing `fsid` to the original `fsid`.
+1. Disable authentication by running `kubectl -n rook-ceph edit cm/rook-config-override` and adding content below:
+
+    ```
+    data:
+    config: |
+        [global]
+        auth cluster required = none
+        auth service required = none
+        auth client required = none
+        auth supported = none
+    ```
+
+1. Bring the Rook Ceph operator back online by running `kubectl -n rook-ceph edit deploy/rook-ceph-operator` and set `replicas` to `1`.
+1. Watch the operator logs with `kubectl -n rook-ceph logs -f rook-ceph-operator-xxxxxxx`, and wait until the orchestration has settled.
+1. **STATE:** Now the new cluster should be up and running with authentication disabled. `ceph -s` should report 1 mon & 1 mgr & all of the OSDs up and running, and all PGs in either `active` or `degraded` state.
+1. Run `kubectl -n rook-ceph exec -it rook-ceph-tools-XXXXXXX bash` to enter tools pod:
+
+    ```shell
+    tools# vi key
+    [paste keyring content saved before, preserving only `[client admin]` section]
+    tools# ceph auth import -i key
+    tools# rm key
+    ```
+
+1. Re-enable authentication by running `kubectl -n rook-ceph edit cm/rook-config-override` and removing auth configuration added in previous steps.
+1. Stop the Rook operator by running `kubectl -n rook-ceph edit deploy/rook-ceph-operator` and set `replicas` to `0`.
+1. Shut down entire new cluster by running `kubectl -n rook-ceph delete deploy/X` where X is every deployment in namespace `rook-ceph`, except `rook-ceph-operator` and `rook-ceph-tools`, again. This time OSD daemons are present and should be removed too.
+1. Bring the Rook Ceph operator back online by running `kubectl -n rook-ceph edit deploy/rook-ceph-operator` and set `replicas` to `1`.
+1. Watch the operator logs with `kubectl -n rook-ceph logs -f rook-ceph-operator-xxxxxxx`, and wait until the orchestration has settled.
+1. **STATE:** Now the new cluster should be up and running with authentication enabled. `ceph -s` output should not change much comparing to previous steps.