forked from rook/rook
N3rdBot
bc8d40095d
Expand the Node Loss section in block-storage.md to accurately describe the automatic fencing flow, and add dedicated Network Fencing sections across the CSI documentation to provide configuration examples for both manifest-based and Helm-based deployments. * block-storage.md: introduce the Network Fencing prerequisite paragraph with links to the new example sections, replace the single-line "auto-fencing" description with three bullets describing the client blocklist flow, and update Node Recovery to a single sentence covering workload rescheduling and the 5-minute cool-down behavior that triggers automatic unblocklist. Add a Warning admonition instructing administrators not to remove the out-of-service taint before the node completes a full power cycle. * csi-configuration.md: add a Network Fencing section enumerating the three prerequisites (CSI-Addons controller, CSI-Addons sidecar, enableFencing flag) and showing the RBD Driver CR manifest with deployCsiAddons and enableFencing set together. * csi-drivers-chart.md: add a Network Fencing subsection under Custom settings showing the per-driver Helm values for deployCsiAddons and enableFencing, with a cross-reference to csi-configuration.md. * ceph-csi-drivers.md: replace the legacy CSI_ENABLE_CSIADDONS configmap patch in the Enable CSI-Addons Sidecar section with the modern deployCsiAddons approach on OperatorConfig or Driver, and add a Network Fencing overview section that links out for configuration details and the operational flow. Signed-off-by: N3rdBot <wujiafeng17@gmail.com>
245 lines
12 KiB
Markdown
245 lines
12 KiB
Markdown
---
|
|
title: Block Storage Overview
|
|
---
|
|
|
|
Block storage allows a single pod to mount storage. This guide shows how to create a simple, multi-tier web application on Kubernetes using persistent volumes enabled by Rook.
|
|
|
|
## Prerequisites
|
|
|
|
This guide assumes a Rook cluster as explained in the [Quickstart](../../Getting-Started/quickstart.md).
|
|
|
|
## Provision Storage
|
|
|
|
Before Rook can provision storage, a [`StorageClass`](https://kubernetes.io/docs/concepts/storage/storage-classes) and [`CephBlockPool` CR](../../CRDs/Block-Storage/ceph-block-pool-crd.md) need to be created. This will allow Kubernetes to interoperate with Rook when provisioning persistent volumes.
|
|
|
|
!!! note
|
|
This sample requires *at least 1 OSD per node*, with each OSD located on *3 different nodes*.
|
|
|
|
Each OSD must be located on a different node, because the [`failureDomain`](../../CRDs/Block-Storage/ceph-block-pool-crd.md#spec) is set to `host` and the `replicated.size` is set to `3`.
|
|
|
|
Save this `StorageClass` definition as `storageclass.yaml`:
|
|
|
|
```yaml
|
|
apiVersion: ceph.rook.io/v1
|
|
kind: CephBlockPool
|
|
metadata:
|
|
name: replicapool
|
|
namespace: rook-ceph
|
|
spec:
|
|
failureDomain: host
|
|
replicated:
|
|
size: 3
|
|
---
|
|
apiVersion: storage.k8s.io/v1
|
|
kind: StorageClass
|
|
metadata:
|
|
name: rook-ceph-block
|
|
# Change "rook-ceph" provisioner prefix to match the operator namespace if needed
|
|
provisioner: rook-ceph.rbd.csi.ceph.com
|
|
parameters:
|
|
# clusterID is the namespace where the rook cluster is running
|
|
clusterID: rook-ceph
|
|
# Ceph pool into which the RBD image shall be created
|
|
pool: replicapool
|
|
|
|
# (optional) mapOptions is a comma-separated list of map options.
|
|
# For krbd options refer
|
|
# https://docs.ceph.com/docs/master/man/8/rbd/#kernel-rbd-krbd-options
|
|
# For nbd options refer
|
|
# https://docs.ceph.com/docs/master/man/8/rbd-nbd/#options
|
|
# mapOptions: lock_on_read,queue_depth=1024
|
|
|
|
# (optional) unmapOptions is a comma-separated list of unmap options.
|
|
# For krbd options refer
|
|
# https://docs.ceph.com/docs/master/man/8/rbd/#kernel-rbd-krbd-options
|
|
# For nbd options refer
|
|
# https://docs.ceph.com/docs/master/man/8/rbd-nbd/#options
|
|
# unmapOptions: force
|
|
|
|
# RBD image format. Defaults to "2".
|
|
imageFormat: "2"
|
|
|
|
# RBD image features
|
|
# Available for imageFormat: "2". Older releases of CSI RBD
|
|
# support only the `layering` feature. The Linux kernel (KRBD) supports the
|
|
# full complement of features as of 5.4
|
|
# `layering` alone corresponds to Ceph's bitfield value of "2" ;
|
|
# `layering` + `fast-diff` + `object-map` + `deep-flatten` + `exclusive-lock` together
|
|
# correspond to Ceph's OR'd bitfield value of "63". Here we use
|
|
# a symbolic, comma-separated format:
|
|
# For 5.4 or later kernels:
|
|
#imageFeatures: layering,fast-diff,object-map,deep-flatten,exclusive-lock
|
|
# For 5.3 or earlier kernels:
|
|
imageFeatures: layering
|
|
|
|
# The secrets contain Ceph admin credentials.
|
|
csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
|
|
csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph
|
|
csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner
|
|
csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph
|
|
csi.storage.k8s.io/controller-publish-secret-name: rook-csi-rbd-provisioner
|
|
csi.storage.k8s.io/controller-publish-secret-namespace: rook-ceph # namespace:cluster
|
|
csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
|
|
csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph
|
|
# Needed for VolumeAttributesClass (dynamic QoS modification on existing volumes).
|
|
csi.storage.k8s.io/controller-modify-secret-name: rook-csi-rbd-provisioner
|
|
csi.storage.k8s.io/controller-modify-secret-namespace: rook-ceph # namespace:cluster
|
|
# Needed for krbd QoS enforcement.
|
|
csi.storage.k8s.io/node-publish-secret-name: rook-csi-rbd-node
|
|
csi.storage.k8s.io/node-publish-secret-namespace: rook-ceph # namespace:cluster
|
|
|
|
# Specify the filesystem type of the volume. If not specified, csi-provisioner
|
|
# will set default as `ext4`. Note that `xfs` is not recommended due to potential deadlock
|
|
# in hyperconverged settings where the volume is mounted on the same node as the osds.
|
|
csi.storage.k8s.io/fstype: ext4
|
|
|
|
# Delete the rbd volume when a PVC is deleted
|
|
reclaimPolicy: Delete
|
|
|
|
# Optional, if you want to add dynamic resize for PVC.
|
|
# For now only ext3, ext4, xfs resize support provided, like in Kubernetes itself.
|
|
allowVolumeExpansion: true
|
|
```
|
|
|
|
If you've deployed the Rook operator in a namespace other than `rook-ceph`,
|
|
change the prefix in the provisioner to match the namespace you used. For
|
|
example, if the Rook operator is running in the namespace `my-namespace` the
|
|
provisioner value should be `my-namespace.rbd.csi.ceph.com`.
|
|
|
|
Create the storage class.
|
|
|
|
```console
|
|
kubectl create -f deploy/examples/csi/rbd/storageclass.yaml
|
|
```
|
|
|
|
!!! note
|
|
As [specified by Kubernetes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#retain), when using the `Retain` reclaim policy, any Ceph RBD image that is backed by a `PersistentVolume` will continue to exist even after the `PersistentVolume` has been deleted. These Ceph RBD images will need to be cleaned up manually using `rbd rm`.
|
|
|
|
## Consume the storage: Wordpress sample
|
|
|
|
We create a sample app to consume the block storage provisioned by Rook with the classic wordpress and mysql apps.
|
|
Both of these apps will make use of block volumes provisioned by Rook.
|
|
|
|
Start mysql and wordpress from the `deploy/examples` folder:
|
|
|
|
```console
|
|
kubectl create -f mysql.yaml
|
|
kubectl create -f wordpress.yaml
|
|
```
|
|
|
|
Both of these apps create a block volume and mount it to their respective pod. You can see the Kubernetes volume claims by running the following:
|
|
|
|
```console
|
|
kubectl get pvc
|
|
```
|
|
|
|
!!! example "Example Output: `kubectl get pvc`"
|
|
```console
|
|
NAME STATUS VOLUME CAPACITY ACCESSMODES AGE
|
|
mysql-pv-claim Bound pvc-95402dbc-efc0-11e6-bc9a-0cc47a3459ee 20Gi RWO 1m
|
|
wp-pv-claim Bound pvc-39e43169-efc1-11e6-bc9a-0cc47a3459ee 20Gi RWO 1m
|
|
```
|
|
|
|
Once the wordpress and mysql pods are in the `Running` state, get the cluster IP of the wordpress app and enter it in your browser:
|
|
|
|
```console
|
|
kubectl get svc wordpress
|
|
```
|
|
|
|
!!! example "Example Output: `kubectl get svc wordpress`"
|
|
```console
|
|
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
|
|
wordpress 10.3.0.155 <pending> 80:30841/TCP 2m
|
|
```
|
|
|
|
You should see the wordpress app running.
|
|
|
|
If you are using Minikube, the Wordpress URL can be retrieved with this one-line command:
|
|
|
|
```console
|
|
echo http://$(minikube ip):$(kubectl get service wordpress -o jsonpath='{.spec.ports[0].nodePort}')
|
|
```
|
|
|
|
!!! note
|
|
When running in a vagrant environment, there will be no external IP address to reach wordpress with. You will only be able to reach wordpress via the `CLUSTER-IP` from inside the Kubernetes cluster.
|
|
|
|
## RBD QoS (Quality of Service)
|
|
|
|
To set IOPS and bandwidth limits on RBD volumes, see the [RBD QoS documentation](rbd-qos.md).
|
|
QoS limits can be applied dynamically via VolumeAttributesClass.
|
|
|
|
## Consume the storage: Toolbox
|
|
|
|
With the pool that was created above, we can also create a block image and mount it directly in a pod. See the [Direct Block Tools](../../Troubleshooting/direct-tools.md#block-storage-tools) topic for more details.
|
|
|
|
## Teardown
|
|
|
|
To clean up all the artifacts created by the block demo:
|
|
|
|
```console
|
|
kubectl delete -f wordpress.yaml
|
|
kubectl delete -f mysql.yaml
|
|
kubectl delete -n rook-ceph cephblockpools.ceph.rook.io replicapool
|
|
kubectl delete storageclass rook-ceph-block
|
|
```
|
|
|
|
## Advanced Example: Erasure Coded Block Storage
|
|
|
|
If you want to use erasure coded pool with RBD, your OSDs must use `bluestore` as their `storeType`.
|
|
Additionally the nodes that are going to mount the erasure coded RBD block storage must have Linux kernel >= `4.11`.
|
|
|
|
!!! attention
|
|
This example requires *at least 3 bluestore OSDs*, with each OSD located on a *different node*.
|
|
|
|
The OSDs must be located on different nodes, because the [`failureDomain`](../../CRDs/Block-Storage/ceph-block-pool-crd.md#spec) is set to `host` and the `erasureCoded` chunk settings require at least 3 different OSDs (2 `dataChunks` + 1 `codingChunks`).
|
|
|
|
To be able to use an erasure coded pool you need to create two pools (as seen below in the definitions): one erasure coded and one replicated.
|
|
|
|
!!! attention
|
|
This example requires *at least 3 bluestore OSDs*, with each OSD located on a *different node*.
|
|
|
|
The OSDs must be located on different nodes, because the [`failureDomain`](../../CRDs/Block-Storage/ceph-block-pool-crd.md#spec) is set to `host` and the `erasureCoded` chunk settings require at least 3 different OSDs (2 `dataChunks` + 1 `codingChunks`).
|
|
|
|
### Erasure Coded CSI Driver
|
|
|
|
The erasure coded pool must be set as the `dataPool` parameter in
|
|
[`storageclass-ec.yaml`](https://github.com/rook/rook/blob/master/deploy/examples/csi/rbd/storageclass-ec.yaml) It is used for the data of the RBD images.
|
|
|
|
## Node Loss
|
|
|
|
If a node goes down where a pod is running where a volume is mounted, the volume cannot automatically be mounted on another node. The node must be guaranteed to be offline before the volume can be mounted on another node.
|
|
|
|
Network fencing is designed to prevent a failed node from accessing the volume before it is remounted elsewhere. The feature is **not enabled by default**. For step-by-step configuration examples to enable automatic network fencing and unfencing, see:
|
|
|
|
* The [`Network Fencing`](../Ceph-CSI/csi-configuration.md#network-fencing) section in the CSI configuration documentation for manifest-based examples using the RBD `Driver` CR.
|
|
* The [`Network Fencing`](../../Helm-Charts/csi-drivers-chart.md#network-fencing) section in the Ceph-CSI driver Helm chart documentation for Helm-based examples.
|
|
|
|
### Handling Node Loss
|
|
|
|
When a node is confirmed to be down, add the following taints to the node:
|
|
|
|
```console
|
|
kubectl taint nodes <node-name> node.kubernetes.io/out-of-service=nodeshutdown:NoExecute
|
|
kubectl taint nodes <node-name> node.kubernetes.io/out-of-service=nodeshutdown:NoSchedule
|
|
```
|
|
|
|
After the taint is added to the node and the fencing feature is properly configured:
|
|
|
|
* The CSI driver will retrieve the stored client address from the RBD image metadata
|
|
* If the node has the `out-of-service` taint, the driver will add the client address to the Ceph blocklist to prevent connections to Ceph from the volume on that node
|
|
* The blocklist entry has an extended duration (approximately 3 years) to ensure the protection persists until the node undergoes a complete power cycle.
|
|
|
|
!!! warning
|
|
When a node becomes out of service, its mounts and device mappings will persist until the node undergoes a complete power lifecycle (includes shutdown and startup). To prevent data inconsistency or corruption, administrators **MUST NOT** remove the `node.kubernetes.io/out-of-service` taint until the node has successfully completed a full power cycle. Removing the taint prematurely may leave stale device states, active client sessions, or lingering mounts, which can lead to serious data integrity issues.
|
|
|
|
### Node Recovery
|
|
|
|
If the node comes back online and the fencing feature is properly configured, workloads can be rescheduled onto the recovered node once the `out-of-service` taints are removed. If the 5-minute cool-down period has passed, the CSI-Addons controller will automatically remove the client address from the blocklist.
|
|
|
|
To manually remove the taints after the node has completed a full power cycle:
|
|
|
|
```console
|
|
kubectl taint nodes <node-name> node.kubernetes.io/out-of-service=nodeshutdown:NoExecute-
|
|
kubectl taint nodes <node-name> node.kubernetes.io/out-of-service=nodeshutdown:NoSchedule-
|
|
```
|