Merge pull request #6978 from travisn/mon-health-doc

ceph: Add mon design to the official docs
This commit is contained in:
Travis Nielsen authored and GitHub committed 2021-01-15 10:30:17 -07:00
commit a01e294aba
3 files changed
+55 -20

No files matched your search

+1 -1
View File
@@ -195,7 +195,7 @@ If this value is empty, each pod will get an ephemeral directory to store their
* If you have multiple Rook Ceph clusters in the same Kubernetes cluster, choose the same namespace to set `rulesNamespace` for all the clusters (ideally, namespace with prometheus deployed). Otherwise, you will get duplicate alerts with duplicate alert definitions.
* `network`: For the network settings for the cluster, refer to the [network configuration settings](#network-configuration-settings)
* `mon`: contains mon related options [mon settings](#mon-settings)
For more details on the mons and when to choose a number other than `3`, see the [mon health design doc](https://github.com/rook/rook/blob/master/design/ceph/mon-health.md).
For more details on the mons and when to choose a number other than `3`, see the [mon health doc](ceph-mon-health.md).
* `mgr`: manager top level section
* `modules`: is the list of Ceph manager modules to enable
* `crashCollector`: The settings for crash collector daemon(s).
@@ -1,3 +1,9 @@
---
title: Monitor Health
weight: 11130
indent: true
---
# Monitor Health
Failure in a distributed system is to be expected. Ceph was designed from the ground up to deal with the failures of a distributed system.
@@ -9,6 +15,7 @@ The Ceph monitors (mons) are the brains of the distributed cluster. They control
to store and retrieve your data as well as keep it safe. If the monitors are not in a healthy state you will risk losing all the data in your system.
## Monitor Identity
Each monitor in a Ceph cluster has a static identity. Every component in the cluster is aware of the identity, and that identity
must be immutable. The identity of a mon is its IP address.
@@ -17,16 +24,19 @@ To have an immutable IP address in Kubernetes, Rook creates a K8s service for ea
When a monitor pod starts, it will bind to its podIP and it will expect communication to be via its service IP address.
## Monitor Quorum
Multiple mons work together to provide redundancy by each keeping a copy of the metadata. A variation of the distributed algorithm Paxos
is used to establish consensus about the state of the cluster. Paxos requires a super-majority of mons to be running in order to establish
quorum and perform operations in the cluster. If the majority of mons are not running, quorum is lost and nothing can be done in the cluster.
### How many mons?
Most commonly a cluster will have three mons. This would mean that one mon could go down and allow the cluster to remain healthy.
You would still have 2/3 mons running to give you consensus in the cluster for any operation.
You will always want an odd number of mons. Fifty percent of mons will not be sufficient to maintain quorum. If you had two mons and one
of them went down, you would have 1/2 of quorum. Since that is not a super-majority, the cluster would have to wait until the second mon is up again.
Therefore, Rook prohibits an even number of mons.
The number of mons to create in a cluster depends on your tolerance for losing a node. If you have 1 mon zero nodes can be lost
to maintain quorum. With 3 mons one node can be lost, and with 5 mons two nodes can be lost. Because the Rook operator will automatically
@@ -34,46 +44,71 @@ start a new new monitor if one dies, you typically only need three mons. The mor
a change to the cluster, which could become a performance issue in a large cluster.
## Mitigating Monitor Failure
Whatever the reason that a mon may fail (power failure, software crash, software hang, etc), there are several layers of mitigation in place
to help recover the mon. It is always better to bring an existing mon back up than to failover to bring up a new mon.
The Rook operator creates a mon with a ReplicaSet to ensure that the mon pod will always be restarted if it fails. If a mon pod stops
The Rook operator creates a mon with a Deployment to ensure that the mon pod will always be restarted if it fails. If a mon pod stops
for any reason, Kubernetes will automatically start the pod up again.
In order for a mon to support a pod/node restart, you will need to set the `dataDirHostPath` so the mon metadata will be persisted to disk
in this folder. This will allow the mon to start back up with its existing metadata and continue where it left off even if the pod had
to be re-created. Without this host path, the mon cannot start again.
In order for a mon to support a pod/node restart, the mon metadata is persisted to disk, either under the `dataDirHostPath` specified
in the CephCluster CR, or in the volume defined by the `volumeClaimTemplate` in the CephCluster CR.
This will allow the mon to start back up with its existing metadata and continue where it left off even if the pod had
to be re-created. Without this persistence, the mon cannot restart.
## Failing over a Monitor
If the mitigating steps for mon failure don't bring a mon back up, the operator will make the decision to terminate the old monitor pod
and bring up a new monitor with a new identity. This is an operation that must be done while there is mon quorum.
The operator checks for mon health every 45 seconds. If a monitor is down, the operator will wait 5 minutes before failing over the bad mon.
If a mon is unhealthy and the K8s pod restart or liveness probe are not sufficient to bring a mon back up, the operator will make the decision
to terminate the unhealthy monitor deployment and bring up a new monitor with a new identity.
This is an operation that must be done while mon quorum is maintained by other mons in the cluster.
The operator checks for mon health every 45 seconds. If a monitor is down, the operator will wait 10 minutes before failing over the unhealthy mon.
These two intervals can be configured as parameters to the CephCluster CR (see below). If the intervals are too short, it could be unhealthy if the mons are failed over too aggressively. If the intervals are too long, the cluster could be at risk of losing quorum if a new monitor is not brought up before another mon fails.
```yaml
healthCheck:
daemonHealth:
mon:
interval: 10s
timeout: 45s
disabled: false
interval: 45s
timeout: 10m
```
If you want to force a mon to failover for testing or other purposes, you can scale down the mon deployment to 0, then wait
for the timeout. Note that the operator may scale up the mon again automatically if the operator is restarted or if a full
reconcile is triggered, such as when the CephCluster CR is updated.
### Example Failover
Rook will create mons with pod names such as mon0, mon1, and mon2. Let's say mon1 had an issue and the pod failed.
Rook will create mons with pod names such as mon-a, mon-b, and mon-c. Let's say mon-b had an issue and the pod failed.
```
$ kubectl -n rook-ceph get pod -l app=rook-ceph-mon
NAME READY STATUS RESTARTS AGE
rook-ceph-mon0-7976n 1/1 Running 0 9m
rook-ceph-mon1-m675r 1/1 Error 0 9m
rook-ceph-mon2-cjbgk 1/1 Running 0 8m
NAME READY STATUS RESTARTS AGE
rook-ceph-mon-a-74dc96545-ch5ns 1/1 Running 0 9m
rook-ceph-mon-b-6b9d895c4c-bcl2h 1/1 Error 2 9m
rook-ceph-mon-c-7d6df6d65c-5cjwl 1/1 Running 0 8m
```
After a failover, you will see the unhealthy mon removed and a new mon added such as mon0, mon2, and mon3. A fully healthy mon quorum is now running again.
After a failover, you will see the unhealthy mon removed and a new mon added such as mon-d. A fully healthy mon quorum is now running again.
```
$ kubectl -n rook-ceph get pod -l app=rook-ceph-mon
NAME READY STATUS RESTARTS AGE
rook-ceph-mon0-7976n 1/1 Running 0 9m
rook-ceph-mon2-cjbgk 1/1 Running 0 8m
rook-ceph-mon3-zjfb1 1/1 Running 0 1m
rook-ceph-mon-a-74dc96545-ch5ns 1/1 Running 0 19m
rook-ceph-mon-c-7d6df6d65c-5cjwl 1/1 Running 0 18m
rook-ceph-mon-d-9e7ea7e76d-4bhxm 1/1 Running 0 20s
```
From the toolbox we can verify the status of the health mon quorum:
```
[root@rook-ceph-tools-78cdfd976c-8p6dn /]# ceph -s
cluster:
id: 35179270-8a39-4e08-a352-a10c52bb04ff
health: HEALTH_OK
services:
mon: 3 daemons, quorum a,b,d (age 2m)
mgr: a(active, since 12m)
osd: 3 osds: 3 up (since 10m), 3 in (since 10m)
...
```
+1 -1
View File
@@ -18,7 +18,7 @@ automates configuration of storage components and monitors the cluster to ensure
and healthy.
The Rook operator is a simple container that has all that is needed to bootstrap
and monitor the storage cluster. The operator will start and monitor [Ceph monitor pods](https://github.com/rook/rook/blob/master/design/ceph/mon-health.md), the Ceph OSD daemons to provide RADOS storage, as well as start and manage other Ceph daemons. The operator manages CRDs for pools, object stores (S3/Swift), and filesystems by initializing the pods and other artifacts necessary to run the services.
and monitor the storage cluster. The operator will start and monitor [Ceph monitor pods](ceph-mon-health.md), the Ceph OSD daemons to provide RADOS storage, as well as start and manage other Ceph daemons. The operator manages CRDs for pools, object stores (S3/Swift), and filesystems by initializing the pods and other artifacts necessary to run the services.
The operator will monitor the storage daemons to ensure the cluster is healthy. Ceph mons will be started or failed over when necessary, and
other adjustments are made as the cluster grows or shrinks. The operator will also watch for desired state changes