From bc8d40095db61a82812ae96a4fc0ec86ed75dbf6 Mon Sep 17 00:00:00 2001 From: N3rdBot Date: Wed, 15 Jul 2026 16:29:37 +0800 Subject: [PATCH] docs: update node loss handling instructions Expand the Node Loss section in block-storage.md to accurately describe the automatic fencing flow, and add dedicated Network Fencing sections across the CSI documentation to provide configuration examples for both manifest-based and Helm-based deployments. * block-storage.md: introduce the Network Fencing prerequisite paragraph with links to the new example sections, replace the single-line "auto-fencing" description with three bullets describing the client blocklist flow, and update Node Recovery to a single sentence covering workload rescheduling and the 5-minute cool-down behavior that triggers automatic unblocklist. Add a Warning admonition instructing administrators not to remove the out-of-service taint before the node completes a full power cycle. * csi-configuration.md: add a Network Fencing section enumerating the three prerequisites (CSI-Addons controller, CSI-Addons sidecar, enableFencing flag) and showing the RBD Driver CR manifest with deployCsiAddons and enableFencing set together. * csi-drivers-chart.md: add a Network Fencing subsection under Custom settings showing the per-driver Helm values for deployCsiAddons and enableFencing, with a cross-reference to csi-configuration.md. * ceph-csi-drivers.md: replace the legacy CSI_ENABLE_CSIADDONS configmap patch in the Enable CSI-Addons Sidecar section with the modern deployCsiAddons approach on OperatorConfig or Driver, and add a Network Fencing overview section that links out for configuration details and the operational flow. Signed-off-by: N3rdBot --- .../Helm-Charts/csi-drivers-chart.md | 17 +++++++++++++ .../Block-Storage-RBD/block-storage.md | 18 +++++++++++--- .../Ceph-CSI/ceph-csi-drivers.md | 24 +++++++++++-------- .../Ceph-CSI/csi-configuration.md | 22 +++++++++++++++++ 4 files changed, 68 insertions(+), 13 deletions(-) diff --git a/Documentation/Helm-Charts/csi-drivers-chart.md b/Documentation/Helm-Charts/csi-drivers-chart.md index 75370e2f7..64d4f2f44 100644 --- a/Documentation/Helm-Charts/csi-drivers-chart.md +++ b/Documentation/Helm-Charts/csi-drivers-chart.md @@ -41,6 +41,23 @@ operatorConfig: See: [CSI-Addons sidecar](../Storage-Configuration/Ceph-CSI/csi-configuration.md#csi-addons-sidecar) +### Network Fencing + +The example below is scoped to the RBD driver only. + +```yaml +drivers: + rbd: + deployCsiAddons: true + enableFencing: true +``` + +!!! important + The Network Fencing feature requires the CSI-Addons controller to be deployed separately for auto-unfencing. See + [Deploying the controller](../Storage-Configuration/Ceph-CSI/ceph-csi-drivers.md#deploying-the-controller). + +See: [Network Fencing](../Storage-Configuration/Ceph-CSI/csi-configuration.md#network-fencing) + ### Controller plugin replicas ```yaml diff --git a/Documentation/Storage-Configuration/Block-Storage-RBD/block-storage.md b/Documentation/Storage-Configuration/Block-Storage-RBD/block-storage.md index f07fd90d9..1281f71ce 100644 --- a/Documentation/Storage-Configuration/Block-Storage-RBD/block-storage.md +++ b/Documentation/Storage-Configuration/Block-Storage-RBD/block-storage.md @@ -209,10 +209,13 @@ The erasure coded pool must be set as the `dataPool` parameter in If a node goes down where a pod is running where a volume is mounted, the volume cannot automatically be mounted on another node. The node must be guaranteed to be offline before the volume can be mounted on another node. +Network fencing is designed to prevent a failed node from accessing the volume before it is remounted elsewhere. The feature is **not enabled by default**. For step-by-step configuration examples to enable automatic network fencing and unfencing, see: + +* The [`Network Fencing`](../Ceph-CSI/csi-configuration.md#network-fencing) section in the CSI configuration documentation for manifest-based examples using the RBD `Driver` CR. +* The [`Network Fencing`](../../Helm-Charts/csi-drivers-chart.md#network-fencing) section in the Ceph-CSI driver Helm chart documentation for Helm-based examples. ### Handling Node Loss - When a node is confirmed to be down, add the following taints to the node: ```console @@ -220,11 +223,20 @@ kubectl taint nodes node.kubernetes.io/out-of-service=nodeshutdown:N kubectl taint nodes node.kubernetes.io/out-of-service=nodeshutdown:NoSchedule ``` -After the taint is added to the node, the CephCSI driver will automatically handle the network fencing to prevent connections to Ceph from the volume on that node. +After the taint is added to the node and the fencing feature is properly configured: + +* The CSI driver will retrieve the stored client address from the RBD image metadata +* If the node has the `out-of-service` taint, the driver will add the client address to the Ceph blocklist to prevent connections to Ceph from the volume on that node +* The blocklist entry has an extended duration (approximately 3 years) to ensure the protection persists until the node undergoes a complete power cycle. + +!!! warning + When a node becomes out of service, its mounts and device mappings will persist until the node undergoes a complete power lifecycle (includes shutdown and startup). To prevent data inconsistency or corruption, administrators **MUST NOT** remove the `node.kubernetes.io/out-of-service` taint until the node has successfully completed a full power cycle. Removing the taint prematurely may leave stale device states, active client sessions, or lingering mounts, which can lead to serious data integrity issues. ### Node Recovery -If the node comes back online, the CephCSI driver will automatically handle the network unfencing when the node taints are removed: +If the node comes back online and the fencing feature is properly configured, workloads can be rescheduled onto the recovered node once the `out-of-service` taints are removed. If the 5-minute cool-down period has passed, the CSI-Addons controller will automatically remove the client address from the blocklist. + +To manually remove the taints after the node has completed a full power cycle: ```console kubectl taint nodes node.kubernetes.io/out-of-service=nodeshutdown:NoExecute- diff --git a/Documentation/Storage-Configuration/Ceph-CSI/ceph-csi-drivers.md b/Documentation/Storage-Configuration/Ceph-CSI/ceph-csi-drivers.md index 2cd80098a..9e0256c38 100644 --- a/Documentation/Storage-Configuration/Ceph-CSI/ceph-csi-drivers.md +++ b/Documentation/Storage-Configuration/Ceph-CSI/ceph-csi-drivers.md @@ -188,17 +188,9 @@ To use the features provided by the CSI-Addons, the `csi-addons` containers need to be deployed in the RBD provisioner and nodeplugin pods, which are not enabled by default. -Execute the following to enable the CSI-Addons sidecars: +To enable the CSI-Addons sidecars, please set `deployCsiAddons` to `true` on `OperatorConfig` or `Driver`. After applying the setting, a new sidecar container named `csi-addons` will start automatically in the RBD CSI provisioner and nodeplugin pods. -* Update the `rook-ceph-operator-config` configmap and patch the - following configuration: - - ```console - kubectl patch cm rook-ceph-operator-config -nrook-ceph -p $'data:\n "CSI_ENABLE_CSIADDONS": "true"' - ``` - -* After enabling `CSI_ENABLE_CSIADDONS` in the configmap, a new sidecar container named `csi-addons` - will start automatically in the RBD CSI provisioner and nodeplugin pods. +For details and examples, see [CSI-Addons sidecar](csi-configuration.md#csi-addons-sidecar). ### CSI-Addons Operations @@ -222,6 +214,18 @@ CSI-Addons supports the following operations: * [Annotating Namespace](https://github.com/csi-addons/kubernetes-csi-addons/blob/v0.14.0/docs/encryptionkeyrotation.md#annotating-namespace) * [Annotating StorageClass](https://github.com/csi-addons/kubernetes-csi-addons/blob/v0.14.0/docs/encryptionkeyrotation.md#annotating-storageclass) +## Network Fencing + +Network fencing automatically blocklists a Ceph client when its node has the +`node.kubernetes.io/out-of-service` taint, and removes the blocklist entry once +the taint is cleared and the cool-down period has passed. This protects +against data corruption when a volume is rescheduled from a node that is +assumed to be down but may still have active mounts. + +Currently, network fencing feature is supported in CephFS and RBD. + +For RBD, see the [Network Fencing](csi-configuration.md#network-fencing) section for configuration examples and the [Node Loss](../Block-Storage-RBD/block-storage.md#node-loss) section for the complete operational flow. + ## Enable RBD and CephFS Encryption Support Ceph-CSI supports encrypting PersistentVolumeClaims (PVCs) for both RBD and CephFS. diff --git a/Documentation/Storage-Configuration/Ceph-CSI/csi-configuration.md b/Documentation/Storage-Configuration/Ceph-CSI/csi-configuration.md index 8e5bd9350..90fb68a31 100644 --- a/Documentation/Storage-Configuration/Ceph-CSI/csi-configuration.md +++ b/Documentation/Storage-Configuration/Ceph-CSI/csi-configuration.md @@ -52,6 +52,28 @@ spec: deployCsiAddons: true ``` +## Network Fencing + +Network fencing requires: + +1. Enable the [CSI-Addons controller](ceph-csi-drivers.md#csi-addons-controller). +2. Enable the CSI-Addons sidecar by setting `deployCsiAddons: true` on the RBD `Driver`. +3. Set `enableFencing` to `true` on the RBD `Driver`. + +Here is an example of RBD `Driver`: + +```yaml +# Driver CR (manifest) +apiVersion: csi.ceph.io/v1 +kind: Driver +metadata: + name: rook-ceph.rbd.csi.ceph.com + namespace: rook-ceph +spec: + deployCsiAddons: true + enableFencing: true +``` + ## Custom container images Previously: `CSI_*_IMAGE` keys in `rook-ceph-operator-config`. Now use the `ImageSet` ConfigMap referenced by `OperatorConfig`.