From b856338c5e9fd2709d52ca88b684d57a6eb99b87 Mon Sep 17 00:00:00 2001 From: Travis Nielsen Date: Wed, 6 Aug 2025 15:57:11 -0600 Subject: [PATCH] docs: csi operator deployment documentation With the csi operator becoming the default for deployment, the docs are now updated with the details in the quickstart guide and upgrade guide to help with the transition. Signed-off-by: Travis Nielsen --- Documentation/Getting-Started/quickstart.md | 44 ++++++++++----------- Documentation/Upgrade/rook-upgrade.md | 26 ++++++++---- 2 files changed, 39 insertions(+), 31 deletions(-) diff --git a/Documentation/Getting-Started/quickstart.md b/Documentation/Getting-Started/quickstart.md index 77cb04414..b8e158347 100644 --- a/Documentation/Getting-Started/quickstart.md +++ b/Documentation/Getting-Started/quickstart.md @@ -38,7 +38,7 @@ A simple Rook cluster is created for Kubernetes with the following `kubectl` com ```console $ git clone --single-branch --branch master https://github.com/rook/rook.git cd rook/deploy/examples -kubectl create -f crds.yaml -f common.yaml -f operator.yaml +kubectl create -f crds.yaml -f common.yaml -f csi-operator.yaml -f operator.yaml kubectl create -f cluster.yaml ``` @@ -59,7 +59,7 @@ The first step is to deploy the Rook operator. ```console cd deploy/examples -kubectl create -f crds.yaml -f common.yaml -f operator.yaml +kubectl create -f crds.yaml -f common.yaml -f csi-operator.yaml -f operator.yaml # verify the rook-ceph-operator is in the `Running` state before proceeding kubectl -n rook-ceph get pod @@ -68,8 +68,6 @@ kubectl -n rook-ceph get pod Before starting the operator in production, consider these settings: 1. Some Rook features are disabled by default. See the [operator.yaml](https://github.com/rook/rook/blob/master/deploy/examples/operator.yaml) for these and other advanced settings. - 1. Device discovery: Rook will watch for new devices to configure if the `ROOK_ENABLE_DISCOVERY_DAEMON` setting is enabled, commonly used in bare metal clusters. - 2. Node affinity and tolerations: The CSI driver by default will run on any node in the cluster. To restrict the CSI driver affinity, several settings are available. 2. If deploying Rook into a namespace other than the default `rook-ceph`, see the topic on [using an alternative namespace](../Storage-Configuration/Advanced/ceph-configuration.md#using-alternate-namespaces). @@ -111,25 +109,25 @@ For the default `cluster.yaml` above, one OSD will be created for each available ```console $ kubectl -n rook-ceph get pod NAME READY STATUS RESTARTS AGE -csi-cephfsplugin-provisioner-d77bb49c6-n5tgs 5/5 Running 0 140s -csi-cephfsplugin-provisioner-d77bb49c6-v9rvn 5/5 Running 0 140s -csi-cephfsplugin-rthrp 3/3 Running 0 140s -csi-rbdplugin-hbsm7 3/3 Running 0 140s -csi-rbdplugin-provisioner-5b5cd64fd-nvk6c 6/6 Running 0 140s -csi-rbdplugin-provisioner-5b5cd64fd-q7bxl 6/6 Running 0 140s -rook-ceph-crashcollector-minikube-5b57b7c5d4-hfldl 1/1 Running 0 105s -rook-ceph-mgr-a-64cd7cdf54-j8b5p 2/2 Running 0 77s -rook-ceph-mgr-b-657d54fc89-2xxw7 2/2 Running 0 56s -rook-ceph-mon-a-694bb7987d-fp9w7 1/1 Running 0 105s -rook-ceph-mon-b-856fdd5cb9-5h2qk 1/1 Running 0 94s -rook-ceph-mon-c-57545897fc-j576h 1/1 Running 0 85s -rook-ceph-operator-85f5b946bd-s8grz 1/1 Running 0 92m -rook-ceph-osd-0-6bb747b6c5-lnvb6 1/1 Running 0 23s -rook-ceph-osd-1-7f67f9646d-44p7v 1/1 Running 0 24s -rook-ceph-osd-2-6cd4b776ff-v4d68 1/1 Running 0 25s -rook-ceph-osd-prepare-node1-vx2rz 0/2 Completed 0 60s -rook-ceph-osd-prepare-node2-ab3fd 0/2 Completed 0 60s -rook-ceph-osd-prepare-node3-w4xyz 0/2 Completed 0 60s +ceph-csi-controller-manager-7794476bc9-h9xk4 1/1 Running 0 110s +rook-ceph-crashcollector-minikube-5b57b7c5d4-hfldl 1/1 Running 0 105s +rook-ceph-exporter-minikube-7b5fd5bbdd-h9c4b 1/1 Running 0 105s +rook-ceph-mgr-a-64cd7cdf54-j8b5p 2/2 Running 0 77s +rook-ceph-mgr-b-657d54fc89-2xxw7 2/2 Running 0 56s +rook-ceph-mon-a-694bb7987d-fp9w7 1/1 Running 0 105s +rook-ceph-mon-b-856fdd5cb9-5h2qk 1/1 Running 0 94s +rook-ceph-mon-c-57545897fc-j576h 1/1 Running 0 85s +rook-ceph-operator-85f5b946bd-s8grz 1/1 Running 0 92m +rook-ceph-osd-0-6bb747b6c5-lnvb6 1/1 Running 0 23s +rook-ceph-osd-1-7f67f9646d-44p7v 1/1 Running 0 24s +rook-ceph-osd-2-6cd4b776ff-v4d68 1/1 Running 0 25s +rook-ceph-osd-prepare-node1-vx2rz 0/2 Completed 0 60s +rook-ceph-osd-prepare-node2-ab3fd 0/2 Completed 0 60s +rook-ceph-osd-prepare-node3-w4xyz 0/2 Completed 0 60s +rook-ceph.cephfs.csi.ceph.com-ctrlplugin-7b75d5b98c-wj444 5/5 Running 0 100s +rook-ceph.cephfs.csi.ceph.com-nodeplugin-prx2j 3/3 Running 0 100s +rook-ceph.rbd.csi.ceph.com-ctrlplugin-75b99498fb-7mq7f 6/6 Running 0 100s +rook-ceph.rbd.csi.ceph.com-nodeplugin-cfcxr 3/3 Running 0 100s ``` To verify that the cluster is in a healthy state, connect to the [Rook toolbox](../Troubleshooting/ceph-toolbox.md) and run the diff --git a/Documentation/Upgrade/rook-upgrade.md b/Documentation/Upgrade/rook-upgrade.md index f0431115a..086d222a7 100644 --- a/Documentation/Upgrade/rook-upgrade.md +++ b/Documentation/Upgrade/rook-upgrade.md @@ -56,6 +56,19 @@ those releases. * The minimum supported Kubernetes version is v1.29. +* The **CSI operator** is required for RBD, CephFS, and NFS volumes. Previously, the CSI driver was automatically configured + by Rook. Now the CSI operator has been factored out of Rook to run independently to manage the Ceph-CSI + driver. During the upgrade and throughout the v1.18.x releases, Rook will automatically convert any Rook CSI settings + to the new CSI operator CRs. This transition is expected to be completely transparent. + Starting in v1.19, the Rook settings related to CSI will be deprecated and there will be a guide on how to configure the new CSI operator CRs directly. + * The CSI operator is created directly via `csi-operator.yaml`, or by the helm setting `csi.rookUseCsiOperator` + in the `rook-ceph` chart. + * The new CSI operator CRs are `cephconnections.csi.ceph.io`, `drivers.csi.ceph.io`, + `operatorconfigs.csi.ceph.io`, `clientprofiles.csi.ceph.io`, and `clientprofilemappings.csi.ceph.io`. + * The CSI operator has been in experimental mode since Rook v1.15 and we have confidence in its stability. That said, if + there is any blocking issue with the csi operator, the previous CSI driver can be re-enabled by setting + `ROOK_USE_CSI_OPERATOR: false` in operator.yaml or applying the helm setting `csi.rookUseCsiOperator: false`. + * Rook now validates node topology during CephCluster creation to prevent misconfigured CRUSH hierarchies. For example, if [topology labels](../CRDs/Cluster/ceph-cluster-crd.md#osd-topology) like `topology.rook.io/rack` are duplicated across zones, cluster creation will fail. The check applies only to new clusters. Existing clusters will only log a warning in the operator log and continue. ## Considerations @@ -87,7 +100,7 @@ section for instructions on how to change the default namespaces in `common.yaml Then, apply the latest changes from v1.18, and update the Rook Operator image. ```console -kubectl apply -f common.yaml -f crds.yaml +kubectl apply -f common.yaml -f crds.yaml -f csi-operator.yaml kubectl -n rook-ceph set image deploy/rook-ceph-operator rook-ceph-operator=rook/ceph:v1.18.1 ``` @@ -103,7 +116,7 @@ Also update optional resources like Prometheus monitoring noted more fully in th If Rook is installed via the Helm chart, Helm will handle some details of the upgrade itself. The upgrade steps in this guide will clarify what Helm handles automatically. -The `rook-ceph` helm chart upgrade performs the Rook upgrade. +The `rook-ceph` helm chart upgrade performs the Rook operator and CSI operator upgrades. The `rook-ceph-cluster` helm chart upgrade performs a [Ceph upgrade](./ceph-upgrade.md) if the Ceph image is updated. The `rook-ceph` chart should be upgraded before `rook-ceph-cluster`, so the latest operator has the opportunity to update custom resources as necessary. @@ -168,7 +181,7 @@ sed -i.bak \ **Apply the resources.** ```console -kubectl apply -f common.yaml -f crds.yaml +kubectl apply -f common.yaml -f crds.yaml -f csi-operator.yaml ``` #### **Prometheus Updates** @@ -192,7 +205,7 @@ When the operator is updated, it will proceed to update all of the Ceph daemons. kubectl -n $ROOK_OPERATOR_NAMESPACE set image deploy/rook-ceph-operator rook-ceph-operator=rook/ceph:master ``` -### **3. Update Ceph CSI** +### **3. Update Ceph CSI Custom Images ** !!! hint This is automatically updated if custom CSI image versions are not set. @@ -200,9 +213,6 @@ kubectl -n $ROOK_OPERATOR_NAMESPACE set image deploy/rook-ceph-operator rook-cep Update to the latest Ceph-CSI drivers if custom CSI images are specified. See the [CSI Custom Images](../Storage-Configuration/Ceph-CSI/custom-images.md) documentation. -!!! note - If using snapshots, refer to the [Upgrade Snapshot API guide](../Storage-Configuration/Ceph-CSI/ceph-csi-snapshot.md#upgrade-snapshot-api). - ### **4. Wait for the upgrade to complete** Watch now in amazement as the Ceph mons, mgrs, OSDs, rbd-mirrors, MDSes and RGWs are terminated and @@ -216,7 +226,7 @@ watch --exec kubectl -n $ROOK_CLUSTER_NAMESPACE get deployments -l rook_cluster= ``` As an example, this cluster is midway through updating the OSDs. When all deployments report `1/1/1` -availability and `rook-version=v1.17.0`, the Ceph cluster's core components are fully updated. +availability and `rook-version=v1.18.0`, the Ceph cluster's core components are fully updated. ```console Every 2.0s: kubectl -n rook-ceph get deployment -o j...