docs: add doc to recover from pod from lost node

This commit adds the doc which has the manual
steps to recover from the specific scenario
like
`on the node lost, the new pod can't mount the
same volume`.

Closes: https://github.com/rook/rook/issues/1507
Signed-off-by: subhamkrai <srai@redhat.com>
This commit is contained in:
subhamkrai committed 2021-10-05 21:13:13 +05:30
1 parent c084ee715f
commit 7587704743
1 file changed
+54
+54
View File
@@ -441,3 +441,57 @@ $ rbd ls --id=csi-rbd-node -m=10.111.136.166:6789 --key=AQDpIQhg+v83EhAAgLboWIbl
```
Where `-m` is one of the mon endpoints and the `--key` is the key used by the CSI driver for accessing the Ceph cluster.
## Node Loss
When a node is lost, you will see application pods on the node stuck in the `Terminating` state while another pod is rescheduled and is in the `ContainerCreating` state.
To allow the application pod to start on another node, force delete the pod.
### Force deleting the pod
To force delete the pod stuck in the `Terminating` state:
```console
$ kubectl -n rook-ceph delete pod my-app-69cd495f9b-nl6hf --grace-period 0 --force
```
After the force delete, wait for a timeout of about 8-10 minutes. If the pod still not in the running state, continue with the next section to blocklist the node.
### Blocklisting a node
To shorten the timeout, you can mark the node as "blocklisted" from the [Rook toolbox](ceph-toolbox.md) so Rook can safely failover the pod sooner.
If the Ceph version is at least Pacific(v16.2.x), run the following command:
```console
$ ceph osd blocklist add <NODE_IP> # get the node IP you want to blocklist
blocklisting <NODE_IP>
```
If the Ceph version is Octopus(v15.2.x) or older, run the following command:
```console
$ ceph osd blacklist add <NODE_IP> # get the node IP you want to blacklist
blacklisting <NODE_IP>
```
After running the above command within a few minutes the pod will be running.
### Removing a node blocklist
After you are absolutely sure the node is permanently offline and that the node no longer needs to be blocklisted, remove the node from the blocklist.
If the Ceph version is at least Pacific(v16.2.x), run:
```console
$ ceph osd blocklist rm <NODE_IP>
un-blocklisting <NODE_IP>
```
If the Ceph version is Octopus(v15.2.x) or older, run:
```console
$ ceph osd blacklist rm <NODE_IP> # get the node IP you want to blacklist
un-blacklisting <NODE_IP>
```