diff --git a/Documentation/ceph-cluster-crd.md b/Documentation/ceph-cluster-crd.md index cc1f8a50b..364264a49 100755 --- a/Documentation/ceph-cluster-crd.md +++ b/Documentation/ceph-cluster-crd.md @@ -195,7 +195,7 @@ If this value is empty, each pod will get an ephemeral directory to store their * If you have multiple Rook Ceph clusters in the same Kubernetes cluster, choose the same namespace to set `rulesNamespace` for all the clusters (ideally, namespace with prometheus deployed). Otherwise, you will get duplicate alerts with duplicate alert definitions. * `network`: For the network settings for the cluster, refer to the [network configuration settings](#network-configuration-settings) * `mon`: contains mon related options [mon settings](#mon-settings) -For more details on the mons and when to choose a number other than `3`, see the [mon health design doc](https://github.com/rook/rook/blob/master/design/ceph/mon-health.md). +For more details on the mons and when to choose a number other than `3`, see the [mon health doc](ceph-mon-health.md). * `mgr`: manager top level section * `modules`: is the list of Ceph manager modules to enable * `crashCollector`: The settings for crash collector daemon(s). diff --git a/design/ceph/mon-health.md b/Documentation/ceph-mon-health.md similarity index 61% rename from design/ceph/mon-health.md rename to Documentation/ceph-mon-health.md index 3b4729a93..65972c01f 100644 --- a/design/ceph/mon-health.md +++ b/Documentation/ceph-mon-health.md @@ -1,3 +1,9 @@ +--- +title: Monitor Health +weight: 11130 +indent: true +--- + # Monitor Health Failure in a distributed system is to be expected. Ceph was designed from the ground up to deal with the failures of a distributed system. @@ -9,6 +15,7 @@ The Ceph monitors (mons) are the brains of the distributed cluster. They control to store and retrieve your data as well as keep it safe. If the monitors are not in a healthy state you will risk losing all the data in your system. ## Monitor Identity + Each monitor in a Ceph cluster has a static identity. Every component in the cluster is aware of the identity, and that identity must be immutable. The identity of a mon is its IP address. @@ -17,16 +24,19 @@ To have an immutable IP address in Kubernetes, Rook creates a K8s service for ea When a monitor pod starts, it will bind to its podIP and it will expect communication to be via its service IP address. ## Monitor Quorum + Multiple mons work together to provide redundancy by each keeping a copy of the metadata. A variation of the distributed algorithm Paxos is used to establish consensus about the state of the cluster. Paxos requires a super-majority of mons to be running in order to establish quorum and perform operations in the cluster. If the majority of mons are not running, quorum is lost and nothing can be done in the cluster. ### How many mons? + Most commonly a cluster will have three mons. This would mean that one mon could go down and allow the cluster to remain healthy. You would still have 2/3 mons running to give you consensus in the cluster for any operation. You will always want an odd number of mons. Fifty percent of mons will not be sufficient to maintain quorum. If you had two mons and one of them went down, you would have 1/2 of quorum. Since that is not a super-majority, the cluster would have to wait until the second mon is up again. +Therefore, Rook prohibits an even number of mons. The number of mons to create in a cluster depends on your tolerance for losing a node. If you have 1 mon zero nodes can be lost to maintain quorum. With 3 mons one node can be lost, and with 5 mons two nodes can be lost. Because the Rook operator will automatically @@ -34,46 +44,71 @@ start a new new monitor if one dies, you typically only need three mons. The mor a change to the cluster, which could become a performance issue in a large cluster. ## Mitigating Monitor Failure + Whatever the reason that a mon may fail (power failure, software crash, software hang, etc), there are several layers of mitigation in place to help recover the mon. It is always better to bring an existing mon back up than to failover to bring up a new mon. -The Rook operator creates a mon with a ReplicaSet to ensure that the mon pod will always be restarted if it fails. If a mon pod stops +The Rook operator creates a mon with a Deployment to ensure that the mon pod will always be restarted if it fails. If a mon pod stops for any reason, Kubernetes will automatically start the pod up again. -In order for a mon to support a pod/node restart, you will need to set the `dataDirHostPath` so the mon metadata will be persisted to disk -in this folder. This will allow the mon to start back up with its existing metadata and continue where it left off even if the pod had -to be re-created. Without this host path, the mon cannot start again. +In order for a mon to support a pod/node restart, the mon metadata is persisted to disk, either under the `dataDirHostPath` specified +in the CephCluster CR, or in the volume defined by the `volumeClaimTemplate` in the CephCluster CR. +This will allow the mon to start back up with its existing metadata and continue where it left off even if the pod had +to be re-created. Without this persistence, the mon cannot restart. ## Failing over a Monitor -If the mitigating steps for mon failure don't bring a mon back up, the operator will make the decision to terminate the old monitor pod -and bring up a new monitor with a new identity. This is an operation that must be done while there is mon quorum. -The operator checks for mon health every 45 seconds. If a monitor is down, the operator will wait 5 minutes before failing over the bad mon. +If a mon is unhealthy and the K8s pod restart or liveness probe are not sufficient to bring a mon back up, the operator will make the decision +to terminate the unhealthy monitor deployment and bring up a new monitor with a new identity. +This is an operation that must be done while mon quorum is maintained by other mons in the cluster. + +The operator checks for mon health every 45 seconds. If a monitor is down, the operator will wait 10 minutes before failing over the unhealthy mon. These two intervals can be configured as parameters to the CephCluster CR (see below). If the intervals are too short, it could be unhealthy if the mons are failed over too aggressively. If the intervals are too long, the cluster could be at risk of losing quorum if a new monitor is not brought up before another mon fails. ```yaml healthCheck: daemonHealth: mon: - interval: 10s - timeout: 45s + disabled: false + interval: 45s + timeout: 10m ``` +If you want to force a mon to failover for testing or other purposes, you can scale down the mon deployment to 0, then wait +for the timeout. Note that the operator may scale up the mon again automatically if the operator is restarted or if a full +reconcile is triggered, such as when the CephCluster CR is updated. + ### Example Failover -Rook will create mons with pod names such as mon0, mon1, and mon2. Let's say mon1 had an issue and the pod failed. + +Rook will create mons with pod names such as mon-a, mon-b, and mon-c. Let's say mon-b had an issue and the pod failed. ``` $ kubectl -n rook-ceph get pod -l app=rook-ceph-mon -NAME READY STATUS RESTARTS AGE -rook-ceph-mon0-7976n 1/1 Running 0 9m -rook-ceph-mon1-m675r 1/1 Error 0 9m -rook-ceph-mon2-cjbgk 1/1 Running 0 8m +NAME READY STATUS RESTARTS AGE +rook-ceph-mon-a-74dc96545-ch5ns 1/1 Running 0 9m +rook-ceph-mon-b-6b9d895c4c-bcl2h 1/1 Error 2 9m +rook-ceph-mon-c-7d6df6d65c-5cjwl 1/1 Running 0 8m ``` -After a failover, you will see the unhealthy mon removed and a new mon added such as mon0, mon2, and mon3. A fully healthy mon quorum is now running again. +After a failover, you will see the unhealthy mon removed and a new mon added such as mon-d. A fully healthy mon quorum is now running again. ``` $ kubectl -n rook-ceph get pod -l app=rook-ceph-mon NAME READY STATUS RESTARTS AGE -rook-ceph-mon0-7976n 1/1 Running 0 9m -rook-ceph-mon2-cjbgk 1/1 Running 0 8m -rook-ceph-mon3-zjfb1 1/1 Running 0 1m +rook-ceph-mon-a-74dc96545-ch5ns 1/1 Running 0 19m +rook-ceph-mon-c-7d6df6d65c-5cjwl 1/1 Running 0 18m +rook-ceph-mon-d-9e7ea7e76d-4bhxm 1/1 Running 0 20s +``` + +From the toolbox we can verify the status of the health mon quorum: +``` +[root@rook-ceph-tools-78cdfd976c-8p6dn /]# ceph -s + cluster: + id: 35179270-8a39-4e08-a352-a10c52bb04ff + health: HEALTH_OK + + services: + mon: 3 daemons, quorum a,b,d (age 2m) + mgr: a(active, since 12m) + osd: 3 osds: 3 up (since 10m), 3 in (since 10m) + + ... ``` diff --git a/Documentation/ceph-storage.md b/Documentation/ceph-storage.md index 18ac84bea..28dc963f1 100644 --- a/Documentation/ceph-storage.md +++ b/Documentation/ceph-storage.md @@ -18,7 +18,7 @@ automates configuration of storage components and monitors the cluster to ensure and healthy. The Rook operator is a simple container that has all that is needed to bootstrap -and monitor the storage cluster. The operator will start and monitor [Ceph monitor pods](https://github.com/rook/rook/blob/master/design/ceph/mon-health.md), the Ceph OSD daemons to provide RADOS storage, as well as start and manage other Ceph daemons. The operator manages CRDs for pools, object stores (S3/Swift), and filesystems by initializing the pods and other artifacts necessary to run the services. +and monitor the storage cluster. The operator will start and monitor [Ceph monitor pods](ceph-mon-health.md), the Ceph OSD daemons to provide RADOS storage, as well as start and manage other Ceph daemons. The operator manages CRDs for pools, object stores (S3/Swift), and filesystems by initializing the pods and other artifacts necessary to run the services. The operator will monitor the storage daemons to ensure the cluster is healthy. Ceph mons will be started or failed over when necessary, and other adjustments are made as the cluster grows or shrinks. The operator will also watch for desired state changes