docs: update node loss handling instructions
Expand the Node Loss section in block-storage.md to accurately describe the automatic fencing flow, and add dedicated Network Fencing sections across the CSI documentation to provide configuration examples for both manifest-based and Helm-based deployments. * block-storage.md: introduce the Network Fencing prerequisite paragraph with links to the new example sections, replace the single-line "auto-fencing" description with three bullets describing the client blocklist flow, and update Node Recovery to a single sentence covering workload rescheduling and the 5-minute cool-down behavior that triggers automatic unblocklist. Add a Warning admonition instructing administrators not to remove the out-of-service taint before the node completes a full power cycle. * csi-configuration.md: add a Network Fencing section enumerating the three prerequisites (CSI-Addons controller, CSI-Addons sidecar, enableFencing flag) and showing the RBD Driver CR manifest with deployCsiAddons and enableFencing set together. * csi-drivers-chart.md: add a Network Fencing subsection under Custom settings showing the per-driver Helm values for deployCsiAddons and enableFencing, with a cross-reference to csi-configuration.md. * ceph-csi-drivers.md: replace the legacy CSI_ENABLE_CSIADDONS configmap patch in the Enable CSI-Addons Sidecar section with the modern deployCsiAddons approach on OperatorConfig or Driver, and add a Network Fencing overview section that links out for configuration details and the operational flow. Signed-off-by: N3rdBot <wujiafeng17@gmail.com>
This commit is contained in:
4 files changed
+68
-13
No files matched your search
@@ -41,6 +41,23 @@ operatorConfig:
|
||||
|
||||
See: [CSI-Addons sidecar](../Storage-Configuration/Ceph-CSI/csi-configuration.md#csi-addons-sidecar)
|
||||
|
||||
### Network Fencing
|
||||
|
||||
The example below is scoped to the RBD driver only.
|
||||
|
||||
```yaml
|
||||
drivers:
|
||||
rbd:
|
||||
deployCsiAddons: true
|
||||
enableFencing: true
|
||||
```
|
||||
|
||||
!!! important
|
||||
The Network Fencing feature requires the CSI-Addons controller to be deployed separately for auto-unfencing. See
|
||||
[Deploying the controller](../Storage-Configuration/Ceph-CSI/ceph-csi-drivers.md#deploying-the-controller).
|
||||
|
||||
See: [Network Fencing](../Storage-Configuration/Ceph-CSI/csi-configuration.md#network-fencing)
|
||||
|
||||
### Controller plugin replicas
|
||||
|
||||
```yaml
|
||||
|
||||
@@ -209,10 +209,13 @@ The erasure coded pool must be set as the `dataPool` parameter in
|
||||
|
||||
If a node goes down where a pod is running where a volume is mounted, the volume cannot automatically be mounted on another node. The node must be guaranteed to be offline before the volume can be mounted on another node.
|
||||
|
||||
Network fencing is designed to prevent a failed node from accessing the volume before it is remounted elsewhere. The feature is **not enabled by default**. For step-by-step configuration examples to enable automatic network fencing and unfencing, see:
|
||||
|
||||
* The [`Network Fencing`](../Ceph-CSI/csi-configuration.md#network-fencing) section in the CSI configuration documentation for manifest-based examples using the RBD `Driver` CR.
|
||||
* The [`Network Fencing`](../../Helm-Charts/csi-drivers-chart.md#network-fencing) section in the Ceph-CSI driver Helm chart documentation for Helm-based examples.
|
||||
|
||||
### Handling Node Loss
|
||||
|
||||
|
||||
When a node is confirmed to be down, add the following taints to the node:
|
||||
|
||||
```console
|
||||
@@ -220,11 +223,20 @@ kubectl taint nodes <node-name> node.kubernetes.io/out-of-service=nodeshutdown:N
|
||||
kubectl taint nodes <node-name> node.kubernetes.io/out-of-service=nodeshutdown:NoSchedule
|
||||
```
|
||||
|
||||
After the taint is added to the node, the CephCSI driver will automatically handle the network fencing to prevent connections to Ceph from the volume on that node.
|
||||
After the taint is added to the node and the fencing feature is properly configured:
|
||||
|
||||
* The CSI driver will retrieve the stored client address from the RBD image metadata
|
||||
* If the node has the `out-of-service` taint, the driver will add the client address to the Ceph blocklist to prevent connections to Ceph from the volume on that node
|
||||
* The blocklist entry has an extended duration (approximately 3 years) to ensure the protection persists until the node undergoes a complete power cycle.
|
||||
|
||||
!!! warning
|
||||
When a node becomes out of service, its mounts and device mappings will persist until the node undergoes a complete power lifecycle (includes shutdown and startup). To prevent data inconsistency or corruption, administrators **MUST NOT** remove the `node.kubernetes.io/out-of-service` taint until the node has successfully completed a full power cycle. Removing the taint prematurely may leave stale device states, active client sessions, or lingering mounts, which can lead to serious data integrity issues.
|
||||
|
||||
### Node Recovery
|
||||
|
||||
If the node comes back online, the CephCSI driver will automatically handle the network unfencing when the node taints are removed:
|
||||
If the node comes back online and the fencing feature is properly configured, workloads can be rescheduled onto the recovered node once the `out-of-service` taints are removed. If the 5-minute cool-down period has passed, the CSI-Addons controller will automatically remove the client address from the blocklist.
|
||||
|
||||
To manually remove the taints after the node has completed a full power cycle:
|
||||
|
||||
```console
|
||||
kubectl taint nodes <node-name> node.kubernetes.io/out-of-service=nodeshutdown:NoExecute-
|
||||
|
||||
@@ -188,17 +188,9 @@ To use the features provided by the CSI-Addons, the `csi-addons`
|
||||
containers need to be deployed in the RBD provisioner and nodeplugin pods,
|
||||
which are not enabled by default.
|
||||
|
||||
Execute the following to enable the CSI-Addons sidecars:
|
||||
To enable the CSI-Addons sidecars, please set `deployCsiAddons` to `true` on `OperatorConfig` or `Driver`. After applying the setting, a new sidecar container named `csi-addons` will start automatically in the RBD CSI provisioner and nodeplugin pods.
|
||||
|
||||
* Update the `rook-ceph-operator-config` configmap and patch the
|
||||
following configuration:
|
||||
|
||||
```console
|
||||
kubectl patch cm rook-ceph-operator-config -nrook-ceph -p $'data:\n "CSI_ENABLE_CSIADDONS": "true"'
|
||||
```
|
||||
|
||||
* After enabling `CSI_ENABLE_CSIADDONS` in the configmap, a new sidecar container named `csi-addons`
|
||||
will start automatically in the RBD CSI provisioner and nodeplugin pods.
|
||||
For details and examples, see [CSI-Addons sidecar](csi-configuration.md#csi-addons-sidecar).
|
||||
|
||||
### CSI-Addons Operations
|
||||
|
||||
@@ -222,6 +214,18 @@ CSI-Addons supports the following operations:
|
||||
* [Annotating Namespace](https://github.com/csi-addons/kubernetes-csi-addons/blob/v0.14.0/docs/encryptionkeyrotation.md#annotating-namespace)
|
||||
* [Annotating StorageClass](https://github.com/csi-addons/kubernetes-csi-addons/blob/v0.14.0/docs/encryptionkeyrotation.md#annotating-storageclass)
|
||||
|
||||
## Network Fencing
|
||||
|
||||
Network fencing automatically blocklists a Ceph client when its node has the
|
||||
`node.kubernetes.io/out-of-service` taint, and removes the blocklist entry once
|
||||
the taint is cleared and the cool-down period has passed. This protects
|
||||
against data corruption when a volume is rescheduled from a node that is
|
||||
assumed to be down but may still have active mounts.
|
||||
|
||||
Currently, network fencing feature is supported in CephFS and RBD.
|
||||
|
||||
For RBD, see the [Network Fencing](csi-configuration.md#network-fencing) section for configuration examples and the [Node Loss](../Block-Storage-RBD/block-storage.md#node-loss) section for the complete operational flow.
|
||||
|
||||
## Enable RBD and CephFS Encryption Support
|
||||
|
||||
Ceph-CSI supports encrypting PersistentVolumeClaims (PVCs) for both RBD and CephFS.
|
||||
|
||||
@@ -52,6 +52,28 @@ spec:
|
||||
deployCsiAddons: true
|
||||
```
|
||||
|
||||
## Network Fencing
|
||||
|
||||
Network fencing requires:
|
||||
|
||||
1. Enable the [CSI-Addons controller](ceph-csi-drivers.md#csi-addons-controller).
|
||||
2. Enable the CSI-Addons sidecar by setting `deployCsiAddons: true` on the RBD `Driver`.
|
||||
3. Set `enableFencing` to `true` on the RBD `Driver`.
|
||||
|
||||
Here is an example of RBD `Driver`:
|
||||
|
||||
```yaml
|
||||
# Driver CR (manifest)
|
||||
apiVersion: csi.ceph.io/v1
|
||||
kind: Driver
|
||||
metadata:
|
||||
name: rook-ceph.rbd.csi.ceph.com
|
||||
namespace: rook-ceph
|
||||
spec:
|
||||
deployCsiAddons: true
|
||||
enableFencing: true
|
||||
```
|
||||
|
||||
## Custom container images
|
||||
|
||||
Previously: `CSI_*_IMAGE` keys in `rook-ceph-operator-config`. Now use the `ImageSet` ConfigMap referenced by `OperatorConfig`.
|
||||
|
||||
Reference in new issue
Block a user