From 901143349bf382674022a1beb2d00c8c84aeab84 Mon Sep 17 00:00:00 2001 From: Madhu Rajanna Date: Wed, 27 Feb 2019 10:50:44 +0530 Subject: [PATCH] Add cephfs csi doc updated ceph-csi-drivers.md document to test cephfs csi driver. Signed-off-by: Madhu Rajanna --- Documentation/ceph-csi-drivers.md | 236 ++++++++++++++++++++++++------ 1 file changed, 193 insertions(+), 43 deletions(-) diff --git a/Documentation/ceph-csi-drivers.md b/Documentation/ceph-csi-drivers.md index 7ee19e94b..9b61d5015 100644 --- a/Documentation/ceph-csi-drivers.md +++ b/Documentation/ceph-csi-drivers.md @@ -6,18 +6,22 @@ indent: true # Running Ceph CSI drivers with Rook -Here is a guide on how to use Rook to deploy ceph-csi drivers on a Kubernetes cluster. +Here is a guide on how to use Rook to deploy ceph-csi drivers on a Kubernetes +cluster. -- [Enable CSI drivers](#enable-csi-drivers) -- [Test the CSI driver](#test-the-csi-driver) -- [Snapshots](#snapshots) -- [Cleanup](#cleanup) +- [Enable CSI drivers](#csi-drivers-enablement) +- [Test RBD CSI driver](#Test-RBD-CSI-Driver) +- [Test CephFS CSI driver](#Test-CephFs-CSI-Driver) ## Prerequisites 1. a Kubernetes v1.13+ is needed in order to support CSI Spec 1.0. -2. `--allow-privileged` flag set to true in [kubelet](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/) and your [API server](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/) -3. An up and running Rook instance (see [Rook - Ceph quickstart guide](https://github.com/rook/rook/blob/master/Documentation/ceph-quickstart.md)) +2. `--allow-privileged` flag set to true in + [kubelet](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/) + and your [API + server](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/) +3. An up and running Rook instance (see [Rook - Ceph quickstart + guide](https://github.com/rook/rook/blob/master/Documentation/ceph-quickstart.md)) ## CSI Drivers Enablement @@ -27,7 +31,8 @@ Here is a guide on how to use Rook to deploy ceph-csi drivers on a Kubernetes cl #check if rook-ceph-system namespace is already created, if not create it kubectl get ns kubectl create namespace rook-ceph-system -# create rbac. Since rook operator is not permitted to create rbac rules, these rules have to be created outside of operator +# create rbac. Since rook operator is not permitted to create rbac rules, +these rules have to be created outside of operator kubectl apply -f cluster/examples/kubernetes/ceph/csi/rbac/rbd/ kubectl apply -f cluster/examples/kubernetes/ceph/csi/rbac/cephfs/ ``` @@ -83,30 +88,37 @@ statefulset.apps/csi-rbdplugin-attacher 1 1 1d statefulset.apps/csi-rbdplugin-provisioner 1 1 1d ``` -## Test the CSI driver - Once the plugin is successfully deployed, test it by running the following example. -### Create the StorageClass +# Test RBD CSI Driver -This [storageclass](../cluster/examples/kubernetes/ceph/csi/example/rbd/storageclass.yaml) expect a pool named `rbd` in your Ceph cluster. You can create this pool using [rook pool CRD](https://github.com/rook/rook/blob/master/Documentation/ceph-pool-crd.md). +## Create RBD StorageClass + +This +[storageclass](../cluster/examples/kubernetes/ceph/csi/example/rbd/storageclass.yaml) +expects a pool named `rbd` in your Ceph cluster. You can create this pool using +[rook pool +CRD](https://github.com/rook/rook/blob/master/Documentation/ceph-pool-crd.md). Please update `monitors` to reflect the Ceph monitors. -### Create the Secret +## Create RBD Secret -Create a Secret that matches that specified in the [storageclass](../cluster/examples/kubernetes/ceph/csi/example/rbd/storageclass.yaml). +Create a Secret that matches `adminid` or `userid` specified in the +[storageclass](../cluster/examples/kubernetes/ceph/csi/example/rbd/storageclass.yaml). Find a Ceph mon pod (in the following example, the pod is `rook-ceph-mon-a-6c4f9f6b6-rzp6r`) and create a Ceph user for that pool called `kubernetes`: ```bash -kubectl exec -ti -n rook-ceph rook-ceph-mon-a-6c4f9f6b6-rzp6r -- bash -c "ceph -c /var/lib/rook/rook-ceph/rook-ceph.config auth get-or-create-key client.kub2 mon \"allow profile rbd\" osd \"profile rbd pool=rbd\"" +kubectl exec -ti -n rook-ceph rook-ceph-mon-a-6c4f9f6b6-rzp6r -- bash -c "ceph -c /var/lib/rook/rook-ceph/rook-ceph.config auth get-or-create-key client.kubernetes mon \"allow profile rbd\" osd \"profile rbd pool=rbd\"" ``` Then create a Secret using admin and `kubernetes` keyrings: +## rbd-secret.yaml + ```yaml apiVersion: v1 kind: Secret @@ -123,11 +135,13 @@ data: #monitors: BASE64-ENCODED-Comma-Delimited-Mons ``` -Here, you need your Ceph admin/user password encoded in base64. Run `ceph auth ls` in one of your Ceph pod, encode the key of your admin/user and replace `BASE64-ENCODED-PASSWORD` by your encoded key. +Here, you need your Ceph admin/user password encoded in base64. Run `ceph auth +ls` in one of your Ceph pods, encode the key of your admin/user and replace +`BASE64-ENCODED-PASSWORD` by your encoded key. -### Create the PersistentVolumeClaim +## Create RBD PersistentVolumeClaim -#### pvc.yaml +### rbd-pvc.yaml ```yaml apiVersion: v1 @@ -143,9 +157,10 @@ spec: storageClassName: csi-rbd ``` -Make sure your `storageClassName` is the name of the StorageClass previously defined in storageclass.yaml +Make sure your `storageClassName` is the name of the StorageClass previously +defined in storageclass.yaml -### Verify the PVC has successfully been created +## Verify RBD PVC has successfully been created ```yaml # kubectl get pvc @@ -153,11 +168,12 @@ NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS rbd-pvc Bound pvc-c20495c0d5de11e8 1Gi RWO csi-rbd 21s ``` -If your PVC status isn't `Bound`, check the csi-rbdplugin logs to see what's preventing the PVC from being up and bound. +If your PVC status isn't `Bound`, check the csi-rbdplugin logs to see what's +preventing the PVC from being up and bound. -### Create the demo Pod +## Create RBD demo Pod -#### pod.yaml +### rbd-pod.yaml ```yaml apiVersion: v1 @@ -178,41 +194,63 @@ spec: readOnly: false ``` -When running `rbd list block --pool [yourpool]` in one of your Ceph pod you should see the created PVC: +When running `rbd list block --pool [yourpool]` in one of your Ceph pods you +should see the created PVC: ```bash # rbd list block --pool rbd pvc-c20495c0d5de11e8 ``` -# Additional features +## Additional features -## Snapshots +### RBD Snapshots -This example is based on [kubernetes-csi/external-snapshotter](https://github.com/kubernetes-csi/external-snapshotter), with a few tweaks to make it work along ceph-csi RBD plugin. This is a basic example of the kubernetes snapshot feature. For more information and functionalities please refer to the [volume snapshot documentation](https://kubernetes.io/docs/concepts/storage/volume-snapshots/). +This example is based on +[kubernetes-csi/external-snapshotter](https://github.com/kubernetes-csi/external-snapshotter), +with a few tweaks to make it work along with the ceph-csi RBD plugin. This is a basic +example of the kubernetes snapshot feature. For more information and +functionalities please refer to the [volume snapshot +documentation](https://kubernetes.io/docs/concepts/storage/volume-snapshots/). -Since this feature is still in [alpha stage](https://kubernetes.io/blog/2018/10/09/introducing-volume-snapshot-alpha-for-kubernetes/) (k8s 1.12+), make sure to enable `VolumeSnapshotDataSource` feature gate in your Kubernetes cluster. +Since this feature is still in [alpha +stage](https://kubernetes.io/blog/2018/10/09/introducing-volume-snapshot-alpha-for-kubernetes/) +(k8s 1.12+), make sure to enable `VolumeSnapshotDataSource` feature gate in +your Kubernetes cluster. -### Enable csi-snapshotter +### Enable RBD csi-snapshotter -First, create RBAC rules to authorize the snapshotter to access the needed resources +First, create RBAC rules to authorize the snapshotter to access the needed +resources ```console kubectl create -f https://raw.githubusercontent.com/ceph/ceph-csi/master/examples/rbd/csi-snapshotter-rbac.yaml ``` -Then, deploy the csi-snapshotter service. This file is based on [setup-csi-snapshotter.yaml](https://github.com/kubernetes-csi/external-snapshotter/blob/master/deploy/kubernetes/setup-csi-snapshotter.yaml) without the csi-provisioner and hostpath-plugin containers that are given as an example. The `volumes` part has been modified to match the ceph-csi plugin socket path. +Then, deploy the csi-snapshotter service. This file is based on +[setup-csi-snapshotter.yaml](https://github.com/kubernetes-csi/external-snapshotter/blob/master/deploy/kubernetes/setup-csi-snapshotter.yaml) +without the csi-provisioner and hostpath-plugin containers that are given as an +example. The `volumes` part has been modified to match the ceph-csi plugin +socket path. -If you followed this guide without changing anything, this file should be left as is. If you made modifications like changing the socket-dir in the plugin deployment, you must edit this file to match your configuration. +If you followed this guide without changing anything, this file should be left +as is. If you made modifications like changing the socket-dir in the plugin +deployment, you must edit this file to match your configuration. ```console kubectl create -f https://raw.githubusercontent.com/ceph/ceph-csi/master/examples/rbd/csi-snapshotter.yaml ``` -### Test csi-snapshotter +### Test RBD csi-snapshotter -Next you need to create the SnapshotClass. The purpose of a SnapshotClass is defined in [the kubernetes documentation](https://kubernetes.io/docs/concepts/storage/volume-snapshot-classes/). In short, as the documentation describes it: -> Just like StorageClass provides a way for administrators to describe the “classes” of storage they offer when provisioning a volume, VolumeSnapshotClass provides a way to describe the “classes” of storage when provisioning a volume snapshot. +Next you need to create the SnapshotClass. The purpose of a SnapshotClass is +defined in [the kubernetes +documentation](https://kubernetes.io/docs/concepts/storage/volume-snapshot-classes/). +In short, as the documentation describes it: +> Just like StorageClass provides a way for administrators to describe the +> “classes” of storage they offer when provisioning a volume, +> VolumeSnapshotClass provides a way to describe the “classes” of storage when +> provisioning a volume snapshot. You must download this file and modify it to match your Ceph cluster. @@ -220,19 +258,24 @@ You must download this file and modify it to match your Ceph cluster. wget https://raw.githubusercontent.com/ceph/ceph-csi/master/examples/rbd/snapshotclass.yaml ``` -The `csiSnapshotterSecretName` parameter should reference the name of the secret created for the ceph-csi plugin you deployed. The monitors are a comma separated list of your Ceph monitors, same as in the StorageClass of the plugin you chosen. When this is done, run: +The `csiSnapshotterSecretName` parameter should reference the name of the +secret created for the ceph-csi plugin you deployed. The monitors are a comma +separated list of your Ceph monitors, same as in the StorageClass of the plugin +you chosen. When this is done, run: ```console kubectl create -f snapshotclass.yaml ``` -Finally, create the VolumeSnapshot resource. its `snapshotClassName` should be the name of the VolumeSnapshotClass previously created. The source name should be the name of the PVC you created earlier. +Finally, create the VolumeSnapshot resource. its `snapshotClassName` should be +the name of the VolumeSnapshotClass previously created. The source name should +be the name of the PVC you created earlier. ```console kubectl create -f https://raw.githubusercontent.com/ceph/ceph-csi/master/examples/rbd/snapshot.yaml ``` -### Verify the Snapshot has successfully been created +### Verify RBD Snapshot has successfully been created ```bash # kubectl get volumesnapshotclass @@ -252,15 +295,15 @@ SNAPID NAME 4 csi-rbd-pvc-c20495c0d5de11e8-snap-4c0b455b-d5fe-11e8-bebb-525400123456 1024 MB Mon Oct 22 13:28:03 2018 ``` -## Cleanup +## RBD resource Cleanup To clean your cluster of the resources created by this example, run the following: ```console -kubectl delete -f pod.yaml -kubectl delete -f pvc.yaml -kubectl delete -f secret.yaml -kubectl delete -f storageclass.yaml +kubectl delete -f rbd-pod.yaml +kubectl delete -f rbd-pvc.yaml +kubectl delete -f rbd-secret.yaml +kubectl delete -f rbd-storageclass.yaml ``` If you tested snapshots too: @@ -271,3 +314,110 @@ kubectl delete -f snapshotclass.yaml kubectl delete -f https://raw.githubusercontent.com/ceph/ceph-csi/master/examples/rbd/csi-snapshotter.yaml kubectl delete -f https://raw.githubusercontent.com/ceph/ceph-csi/master/examples/rbd/csi-snapshotter-rbac.yaml ``` + +# Test CephFS CSI Driver + +## Create CephFS StorageClass + +This +[storageclass](../cluster/examples/kubernetes/ceph/csi/example/cephfs/storageclass.yaml) +expect a pool named `cephfs_data` in your Ceph cluster. You can create this +pool using [rook file-system +CRD](https://github.com/rook/rook/blob/master/Documentation/ceph-filesystem-crd.md). + +Please update `monitors` to reflect the Ceph monitors. + +## Create CephFS Secret + +Create a Secret that matches `provisionVolume` type specified in the [storageclass](../cluster/examples/kubernetes/ceph/csi/example/cephfs/storageclass.yaml). + +### cephfs-secret.yaml + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: csi-cephfs-secret + namespace: default +data: + # Required if provisionVolume is set to false + userID: BASE64-ENCODED-USER + userKey: BASE64-ENCODED-PASSWORD + + # Required if provisionVolume is set to true + adminID: BASE64-ENCODED-USER + adminKey: BASE64-ENCODED-PASSWORD +``` + +Here, you need your Ceph admin/user ID and password encoded in base64. Encode +admin/user ID in base64 format and replace `BASE64-ENCODED-USER` and Run `ceph +auth ls` in one of your Ceph pods, encode the key of your admin/user and replace +`BASE64-ENCODED-PASSWORD` by your encoded key. + +## Create CephFS PersistentVolumeClaim + +### cephfs-pvc.yaml + +```yaml +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: cephfs-pvc +spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + storageClassName: csi-cephfs +``` + +Make sure your `storageClassName` is the name of the StorageClass previously +defined in storageclass.yaml + +## Verify CephFS PVC has successfully been created + +```bash +# kubectl get pvc +NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE +cephfs-pvc Bound pvc-6bc76846-3a4a-11e9-971d-525400c2d871 1Gi RWO csi-cephfs 25s + +``` + +If your PVC status isn't `Bound`, check the csi-cephfsplugin logs to see what's +preventing the PVC from being up and bound. + +## Create CephFS demo Pod + +### cephfs-pod.yaml + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: csicephfs-demo-pod +spec: + containers: + - name: web-server + image: nginx + volumeMounts: + - name: mypvc + mountPath: /var/lib/www/html + volumes: + - name: mypvc + persistentVolumeClaim: + claimName: cephfs-pvc + readOnly: false +``` + +## CephFS resource Cleanup + +To clean your cluster of the resources created by this example, run the +following: + +```console +kubectl delete -f cephfs-pod.yaml +kubectl delete -f cephfs-pvc.yaml +kubectl delete -f cephfs-secret.yaml +kubectl delete -f cephfs-storageclass.yaml +```