Merge pull request #6592 from sp98/update-cleanupPolicy-design

ceph: update cleanupPolicy design doc

design/ceph/ceph-cluster-cleanup.md

@@ -6,9 +6,14 @@ As a rook user, I want to clean up data on the hosts after I intentionally unins
 
 ## Background
 
-### Host-based cluster deletion
+### Cluster deletion
+If the user deletes a rook-ceph cluster and wants to start a new cluster on the same hosts, then following manual steps should be performed:
+- Delete the dataDirHostPath on each host. Otherwise, stale keys and other configs will remain from the previous cluster and the new mons will fail to start.
+- Clean the OSD disks from the previous cluster before starting a new one.
 
-If the user deletes a host-based cluster (running on raw devices and not on PVs) and starts a new cluster on the same hosts, the path used by dataDirHostPath must be deleted. Otherwise, stale keys and other config will remain from the previous cluster and the new mons will fail to start. As of now, the user has to manually delete the dataDirHostPath on each host. This implementation aims to automate this deletion.
+Read more about the manual clean up steps [here](https://github.com/rook/rook/blob/master/Documentation/ceph-teardown.md#delete-the-data-on-hosts)
+
+This implementation aims to automate both of these manual steps.
 
 ## Design Flow
 
@@ -19,7 +24,7 @@ If the user deletes a host-based cluster (running on raw devices and not on PVs)
 
 ### How to add user confirmation
 
-- If the user really wants to clean up the data on the cluster, then user should update the ceph cluster CRD with cleanupPolicy configuration like below :
+- If the user really wants to clean up the data on the cluster, then update the ceph cluster CRD with cleanupPolicy configuration like below :
 
 ```yaml
 apiVersion: ceph.rook.io/v1
@@ -38,22 +43,33 @@ spec:
     useAllNodes: true
     useAllDevices: true
   cleanupPolicy:
-    deleteDataDirOnHosts: yes-really-destroy-data
+    confirmation: yes-really-destroy-data
+    sanitizeDisks:
+      method: quick
+      dataSource: zero
+      iteration: 1
+    allowUninstallWithVolumes: false
 ```
 
-- Adding `cleanupPolicy` would cause the operator to refuse running an orchestration
+- Updating the cluster `cleanupPolicy` with `confirmation: yes-really-destroy-data` would cause the operator to refuse running any further orchestration.
 
 ### How the Operator cleans up the cluster
 
 - Operator starts the clean up flow only when deletionTimeStamp is present on the ceph Cluster.
-- Operator checks for user confirmation (for example `deleteDataDirOnHosts: yes-really-destroy-data`) on the ceph cluster before starting the clean up.
+- Operator checks for user confirmation (that is, `confirmation: yes-really-destroy-data`) on the ceph cluster before starting the clean up.
 - Identify the nodes where ceph daemons are running.
 - Wait till all the ceph daemons are destroyed on each node. This is important because deleting the data (say dataDirHostPath) before the daemons would cause the daemons to panic.
 - Create a batch job that runs on each of the above nodes.
 - The job performs the following action on each node based on the user confirmation:
-  - cleanup the cluster namespace on the dataDirHostPath
-  - Delete all the ceph monitor directories on the dataDirHostPath. For example mon-a, mon-b, etc.
-  - Clean up the devices on each node.
+  - cleanup the cluster namespace on the dataDirHostPath. For example `/var/lib/rook/rook-ceph`
+  - Delete all the ceph monitor directories on the dataDirHostPath. For example `/var/lib/rook/mon-a`, `/var/lib/rook/mon-b`, etc.
+  - Sanitize the local disks used by OSDs on each node.
+- Local disk sanitization can be further configured by the admin with following options:
+  - `method`: use `complete` to sanitize the entire disk and `quick` (default) to sanitize only ceph's metadata.
+  - `dataSource`: indicate where to get random bytes from to write on the disk. Possible choices are `zero` (default) or `random`.
+  Using random sources will consume entropy from the system and will take much more time then the zero source.
+  - `iteration`: overwrite N times instead of the default (1). Takes an integer value.
+- If `allowUninstallWithVolumes` is `false` (default), then operator would wait for the PVCs to be deleted before finally deleting the cluster.
 
 #### Cleanup Job Spec:
 
@@ -61,7 +77,12 @@ spec:
 apiVersion: batch/v1
 kind: Job
 metadata:
-  name: rook-ceph-cleanup-<node-name>
+  name: cluster-cleanup-job-<node-name>
+  namespace: <namespace>
+  labels:
+    app: rook-ceph-cleanup
+    rook-ceph-cleanup: "true"
+    rook_cluster: <namespace>
 spec:
   template:
     spec:
@@ -74,15 +95,34 @@ spec:
           # if ROOK_DATA_DIR_HOST_PATH is available, then delete the dataDirHostPath
           - name: ROOK_DATA_DIR_HOST_PATH
             value: <dataDirHostPath>
+          - name: ROOK_NAMESPACE_DIR
+            value: <namespace>
+          - name: ROOK_MON_SECRET
+            value: <dataDirHostPath>
+          - name: ROOK_CLUSTER_FSID
+            value: <dataDirHostPath>
+          - name: ROOK_LOG_LEVEL
+            value: <dataDirHostPath>
+          - name: ROOK_SANITIZE_METHOD
+            value: <method>
+          - name: ROOK_SANITIZE_DATA_SOURCE
+            value: <dataSource>
+            - name: ROOK_SANITIZE_ITERATION
+            value: <iteration>
           args: []string{"ceph", "clean"}
           volumeMounts:
             - name: cleanup-volume
               # data dir host path that needs to be cleaned up.
               mountPath: <dataDirHostPath>
+            - name: devices
+              mountPath: /dev
       volume:
         - name: cleanup-volume
           hostPath:
             #directory location on the host
             path: <dataDirHostPath>
-      restartPolicy: Never
+        - name: devices
+          hostpath:
+            path: /dev
+      restartPolicy: OnFailure
 ```