my-rook-config/Documentation/Storage-Configuration/Block-Storage-RBD/rbd-qos.md

title

title
RBD QoS

RBD Quality of Service (QoS) allows administrators to set IOPS and bandwidth limits on RBD block volumes. This prevents noisy-neighbor issues in multi-tenant clusters by ensuring that no single workload can saturate the storage cluster's I/O capacity.

QoS is configured through a VolumeAttributesClass resource, which allows dynamic modification of QoS limits on existing volumes without recreation using the ControllerModifyVolume CSI operation.

Supported Mounters

krbd (cgroup v2)

The default krbd (kernel RBD) mounter enforces QoS via the Linux cgroup v2 io.max controller at the container level.

!!! note Kubernetes >= v1.34 is required (VolumeAttributesClass is GA since v1.34). cgroup v2 must be enabled on the nodes (default on most modern Linux distributions) and Linux kernel >= 5.8 is required.

QoS Parameters Reference

krbd (cgroup v2) Parameters

The following parameters can be set on a VolumeAttributesClass resource. All parameters are optional — set only the limits you need.

Parameter	Description	Example
maxReadIops	Maximum read IOPS limit	"1000"
maxWriteIops	Maximum write IOPS limit	"2000"
maxReadBps	Maximum read bandwidth (bytes/sec)	"104857600" (100 MiB/s)
maxWriteBps	Maximum write bandwidth (bytes/sec)	"209715200" (200 MiB/s)

Fresh Cluster Setup

For new Rook Ceph deployments, follow these steps to enable QoS.

1. Ensure the StorageClass has the required secret references for modify and publish operations. The default storageclass.yaml already includes these:

   parameters:
     # ... existing parameters ...
     # Required for VolumeAttributesClass (dynamic QoS modification)
     csi.storage.k8s.io/controller-modify-secret-name: rook-csi-rbd-provisioner
     csi.storage.k8s.io/controller-modify-secret-namespace: rook-ceph # namespace:cluster
     # Required for krbd QoS (cgroup v2)
     csi.storage.k8s.io/node-publish-secret-name: rook-csi-rbd-node
     csi.storage.k8s.io/node-publish-secret-namespace: rook-ceph # namespace:cluster
   

2. Create the VolumeAttributesClass:

   kubectl create -f deploy/examples/csi/rbd/volumeattributesclass-cgroup.yaml
   

   See the volumeattributesclass-cgroup.yaml for the full example with all available parameters.

3. Create a PVC with the VolumeAttributesClass:

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: rbd-pvc-qos
   spec:
     accessModes:
       - ReadWriteOnce
     volumeAttributesClassName: rook-ceph-rbd-cgroup-qos
     resources:
       requests:
         storage: 10Gi
     storageClassName: rook-ceph-block
   

4. To change QoS on an existing volume, update the PVC's volumeAttributesClassName:

   kubectl patch pvc rbd-pvc-qos -p '{"spec":{"volumeAttributesClassName":"rook-ceph-rbd-cgroup-qos"}}'
   

   !!! note A single pod can have multiple volumes with different QoS limits by using different VolumeAttributesClasses for each PVC.

Upgrading Existing Clusters

VolumeAttributesClass allows applying QoS to existing volumes without migration. Follow these steps to enable QoS on upgraded clusters.

1. Update the existing StorageClass to add the required secret references. Since StorageClass parameters are immutable, you must delete and recreate it:

   !!! warning Deleting a StorageClass does not affect existing PVs or PVCs. However, new PVC creation will fail until the StorageClass is recreated.

   kubectl get storageclass rook-ceph-block -o yaml > storageclass-backup.yaml
   kubectl delete storageclass rook-ceph-block
   

   Edit the storageclass-backup.yaml to add the modify-secret and publish-secret parameters as described in Step 1 of the Fresh Cluster Setup, then recreate:

   kubectl create -f storageclass-backup.yaml
   

2. Create the VolumeAttributesClass:

   kubectl create -f deploy/examples/csi/rbd/volumeattributesclass-cgroup.yaml
   

3. Apply QoS to existing PVCs by patching them with the VolumeAttributesClass:

   kubectl patch pvc <pvc-name> -p '{"spec":{"volumeAttributesClassName":"rook-ceph-rbd-cgroup-qos"}}'
   

4. Verify QoS is applied by checking the PVC status:

   kubectl get pvc <pvc-name> -o jsonpath='{.status.currentVolumeAttributesClassName}'
   

   The output should match the VolumeAttributesClass name you applied.

Troubleshooting

QoS Limits Not Being Enforced

1. Verify VolumeAttributesClass: Check that the VolumeAttributesClass exists and the PVC references it:

   kubectl get volumeattributesclass
   kubectl get pvc <pvc-name> -o jsonpath='{.spec.volumeAttributesClassName}'
   

2. Check CSI driver logs: Look for QoS-related errors in the CSI node plugin:

   kubectl logs -n rook-ceph -l app=csi-rbdplugin -c csi-rbdplugin --tail=100 | grep -i qos
   

cgroup v2 QoS Not Working

1. Verify cgroup v2: Check that nodes use cgroup v2:

   kubectl debug node/<node-name> -it --image=busybox -- cat /host/sys/fs/cgroup/cgroup.controllers
   

   If the file exists and lists io, cgroup v2 is active with the io controller enabled.

2. Check Kubernetes version: VolumeAttributesClass requires Kubernetes >= v1.34. Verify:

   kubectl version --short