Blaine Gardner
c9d99e01a0
mkdocs uses a markdown renderer that is hardcoded to 4 spaces per tab
for detecting indentation levels, including ordered- and
unordered-lists. Since we cannot easily change the renderer, begin using
a markdown linter in CI that will fail if official docs do not adhere to
the spacing rules.
As a starting point, the markdownlint config does not begin with the
default set of checks, which might overwhelm attempts to fix them.
Instead, focus on list-tab-spacing rules and a few other highly useful
checks.
markdownlint also has some gaps in its abilities that allow common Rook
doc issues to pass acceptance. However, it allows creating custom
linting plugins. Create 2 such linting plugins to check 2 things:
- all doc lines (except code blocks) must be aligned to a 4-space
boundary, without exception. This ensures that markdown will render
correctly with mkdocs. This unfortunately makes it possible to create
lists that are internally aligned strangely.
- admonitions must all follow the same format of
```
!!! header
body
```
For the strange lists, this is allowed and renders correctly, but it
looks strange:
```md
- first bullet
- second bullet
still second bullet
- third bullet
has a paragraph
of text inside
- last bullet
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
105 lines
5.6 KiB
Markdown
105 lines
5.6 KiB
Markdown
---
|
||
title: RBD Asynchronous DR Failover and Failback
|
||
---
|
||
|
||
## Planned Migration and Disaster Recovery
|
||
|
||
Rook comes with the volume replication support, which allows users to perform disaster recovery and planned migration of clusters.
|
||
|
||
The following document will help to track the procedure for failover and failback in case of a Disaster recovery or Planned migration use cases.
|
||
|
||
!!! note
|
||
The document assumes that RBD Mirroring is set up between the peer clusters.
|
||
For information on rbd mirroring and how to set it up using rook, please refer to
|
||
the [rbd-mirroring guide](rbd-mirroring.md).
|
||
|
||
## Planned Migration
|
||
|
||
!!! info
|
||
Use cases: Datacenter maintenance, technology refresh, disaster avoidance, etc.
|
||
|
||
### Relocation
|
||
|
||
The Relocation operation is the process of switching production to a
|
||
backup facility(normally your recovery site) or vice versa. For relocation,
|
||
access to the image on the primary site should be stopped.
|
||
The image should now be made *primary* on the secondary cluster so that
|
||
the access can be resumed there.
|
||
|
||
!!! note
|
||
:memo: Periodic or one-time backup of
|
||
the application should be available for restore on the secondary site (cluster-2).
|
||
|
||
Follow the below steps for planned migration of workload from the primary
|
||
cluster to the secondary cluster:
|
||
|
||
* Scale down all the application pods which are using the
|
||
mirrored PVC on the Primary Cluster.
|
||
* [Take a backup](rbd-mirroring.md#backup-&-restore) of PVC and PV object from the primary cluster.
|
||
This can be done using some backup tools like
|
||
[velero](https://velero.io/docs/main/).
|
||
* [Update VolumeReplication CR](rbd-mirroring.md#create-a-volumereplication-cr) to set `replicationState` to `secondary` at the Primary Site.
|
||
When the operator sees this change, it will pass the information down to the
|
||
driver via GRPC request to mark the dataSource as `secondary`.
|
||
* If you are manually recreating the PVC and PV on the secondary cluster,
|
||
remove the `claimRef` section in the PV objects. (See [this](rbd-mirroring.md#restore-the-backup-on-cluster-2) for details)
|
||
* Recreate the storageclass, PVC, and PV objects on the secondary site.
|
||
* As you are creating the static binding between PVC and PV, a new PV won’t
|
||
be created here, the PVC will get bind to the existing PV.
|
||
* [Create the VolumeReplicationClass](rbd-mirroring.md#create-a-volume-replication-class-cr) on the secondary site.
|
||
* [Create VolumeReplications](rbd-mirroring.md#create-a-volumereplication-cr) for all the PVC’s for which mirroring is enabled
|
||
* `replicationState` should be `primary` for all the PVC’s on the secondary site.
|
||
* [Check VolumeReplication CR status](rbd-mirroring.md#checking-replication-status) to verify if the image is marked `primary` on the secondary site.
|
||
* Once the Image is marked as `primary`, the PVC is now ready
|
||
to be used. Now, we can scale up the applications to use the PVC.
|
||
|
||
!!! warning
|
||
:memo: In Async Disaster recovery use case, we don't get
|
||
the complete data.
|
||
We will only get the crash-consistent data based on the snapshot interval time.
|
||
|
||
## Disaster Recovery
|
||
|
||
!!! info
|
||
Use cases: Natural disasters, Power failures, System failures, and crashes, etc.
|
||
|
||
!!! note
|
||
To effectively resume operations after a failover/relocation,
|
||
backup of the kubernetes artifacts like deployment, PVC, PV, etc need to be created beforehand by the admin; so that the application can be restored on the peer cluster. For more information, see [backup and restore](rbd-mirroring.md#backup-&-restore).
|
||
|
||
### Failover (abrupt shutdown)
|
||
|
||
In case of Disaster recovery, create VolumeReplication CR at the Secondary Site.
|
||
Since the connection to the Primary Site is lost, the operator automatically
|
||
sends a GRPC request down to the driver to forcefully mark the dataSource as `primary`
|
||
on the Secondary Site.
|
||
|
||
* If you are manually creating the PVC and PV on the secondary cluster, remove
|
||
the claimRef section in the PV objects. (See [this](rbd-mirroring.md#restore-the-backup-on-cluster-2) for details)
|
||
* Create the storageclass, PVC, and PV objects on the secondary site.
|
||
* As you are creating the static binding between PVC and PV, a new PV won’t be
|
||
created here, the PVC will get bind to the existing PV.
|
||
* [Create the VolumeReplicationClass](rbd-mirroring.md#create-a-volume-replication-class-cr) and [VolumeReplication CR](rbd-mirroring.md#create-a-volumereplication-cr) on the secondary site.
|
||
* [Check VolumeReplication CR status](rbd-mirroring.md#checking-replication-status) to verify if the image is marked `primary` on the secondary site.
|
||
* Once the Image is marked as `primary`, the PVC is now ready to be used. Now,
|
||
we can scale up the applications to use the PVC.
|
||
|
||
### Failback (post-disaster recovery)
|
||
|
||
Once the failed cluster is recovered on the primary site and you want to failback
|
||
from secondary site, follow the below steps:
|
||
|
||
* Scale down the running applications (if any) on the primary site.
|
||
Ensure that all persistent volumes in use by the workload are no
|
||
longer in use on the primary cluster.
|
||
* [Update VolumeReplication CR](rbd-mirroring.md#create-a-volumereplication-cr) replicationState
|
||
from `primary` to `secondary` on the primary site.
|
||
* Scale down the applications on the secondary site.
|
||
* [Update VolumeReplication CR](rbd-mirroring.md#create-a-volumereplication-cr) replicationState state from `primary` to
|
||
`secondary` in secondary site.
|
||
* On the primary site, [verify the VolumeReplication status](rbd-mirroring.md#checking-replication-status) is marked as
|
||
volume ready to use.
|
||
* Once the volume is marked to ready to use, change the replicationState state
|
||
from `secondary` to `primary` in primary site.
|
||
* Scale up the applications again on the primary site.
|