Document ceph-csi drivers implementation
Signed-off-by: mickymiek <meunie_m@etna-alternance.net>
This commit is contained in:
1 parent
9ba4fcc450
commit
5b8edd2907
1 file changed
+279
@@ -0,0 +1,279 @@
|
||||
---
|
||||
title: Ceph CSI
|
||||
weight: 32
|
||||
indent: true
|
||||
---
|
||||
|
||||
# Running ceph CSI drivers with Rook
|
||||
|
||||
Here is a guide on how to implement ceph-csi drivers on a Kubernetes cluster with Rook. The way to implement it should change relatively soon as there is an effort to manage deployments of the ceph-csi plugin within Rook directly ([#1385](https://github.com/rook/rook/issues/1385), [#2059](https://github.com/rook/rook/pull/2059)).
|
||||
|
||||
- [Enable CSI drivers](#enable-csi-drivers)
|
||||
- [Test the CSI driver](#test-the-csi-driver)
|
||||
- [Snapshots](#snapshots)
|
||||
- [Cleanup](#cleanup)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
1. A Kubernetes v1.12+ cluster with at least one node
|
||||
2. `--allow-privileged` flag set to true in [kubelet](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/) and your [API server](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/)
|
||||
3. An up and running Rook instance (see [Rook - ceph quickstart guide](https://github.com/rook/rook/blob/master/Documentation/ceph-quickstart.md))
|
||||
|
||||
## Enable CSI drivers
|
||||
|
||||
As described by the ceph-csi [Cephfs](https://github.com/ceph/ceph-csi/blob/master/docs/deploy-cephfs.md) and [RBD](https://github.com/ceph/ceph-csi/blob/master/docs/deploy-rbd.md) plugins deployment guides, the plugins yaml manifests are located in ceph-csi repository in the [deploy folder](https://github.com/ceph/ceph-csi/tree/master/deploy), and should be deployed as follow:
|
||||
|
||||
### Deploy RBACs for node plugins, csi-attacher and csi-provisioner (common to Cephfs and RBD):
|
||||
```
|
||||
kubectl create -f https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-attacher-rbac.yaml
|
||||
kubectl create -f https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-provisioner-rbac.yaml
|
||||
kubectl create -f https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-nodeplugin-rbac.yaml
|
||||
```
|
||||
Those manifests deploy service accounts, cluster roles and cluster role bindings.
|
||||
### Deploy csi-attacher and csi-provisioner containers:
|
||||
Deploys stateful sets for external-attacher and external-provisioner sidecar containers.
|
||||
##### Cephfs
|
||||
|
||||
```
|
||||
kubectl create -f https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/cephfs/kubernetes/csi-cephfsplugin-attacher.yaml
|
||||
kubectl create -f https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/cephfs/kubernetes/csi-cephfsplugin-provisioner.yaml
|
||||
```
|
||||
##### RBD
|
||||
```
|
||||
kubectl create -f https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-rbdplugin-attacher.yaml
|
||||
kubectl create -f https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-rbdplugin-provisioner.yaml
|
||||
```
|
||||
### Deploy the CSI driver:
|
||||
Deploys a daemon set with two containers: CSI driver-registrar and the driver.
|
||||
##### Cephfs
|
||||
```
|
||||
kubectl create -f https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/cephfs/kubernetes/csi-cephfsplugin.yaml
|
||||
```
|
||||
##### RBD
|
||||
```
|
||||
kubectl create -f https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-rbdplugin.yaml
|
||||
```
|
||||
|
||||
### Verify the plugin, its attacher and provisioner are running:
|
||||
You should see an output similar to this when you run `kubectl get all`.
|
||||
```
|
||||
# kubectl get all
|
||||
|
||||
NAMESPACE NAME READY STATUS RESTARTS AGE
|
||||
default pod/csi-rbdplugin-975c4 2/2 Running 0 132m
|
||||
default pod/csi-rbdplugin-attacher-0 1/1 Running 0 132m
|
||||
default pod/csi-rbdplugin-provisioner-0 1/1 Running 0 132m
|
||||
|
||||
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
|
||||
default service/csi-rbdplugin-attacher ClusterIP 10.107.65.227 <none> 12345/TCP 132m
|
||||
default service/csi-rbdplugin-provisioner ClusterIP 10.108.120.159 <none> 12345/TCP 132m
|
||||
|
||||
NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
|
||||
default daemonset.apps/csi-rbdplugin 1 1 1 1 1 <none> 132m
|
||||
|
||||
NAMESPACE NAME DESIRED CURRENT AGE
|
||||
default statefulset.apps/csi-rbdplugin-attacher 1 1 132m
|
||||
default statefulset.apps/csi-rbdplugin-provisioner 1 1 132m
|
||||
|
||||
```
|
||||
|
||||
## Test the CSI driver
|
||||
|
||||
Once the plugin is successfully deployed, test it by running the following example
|
||||
|
||||
This example is based on the ceph-csi [examples](https://github.com/ceph/ceph-csi/tree/master/examples) directory. It will describe how to test the RBD csi-driver, but it is very similar to run the cephfs one.
|
||||
|
||||
### Create the StorageClass:
|
||||
This storageclass expect a pool named `rbd` in your ceph cluster. You can create this pool using [rook pool CRD](https://github.com/rook/rook/blob/master/Documentation/ceph-pool-crd.md)
|
||||
##### storageclass.yaml
|
||||
```
|
||||
apiVersion: storage.k8s.io/v1
|
||||
kind: StorageClass
|
||||
metadata:
|
||||
name: csi-rbd
|
||||
provisioner: csi-rbdplugin
|
||||
parameters:
|
||||
# Comma separated list of Ceph monitors
|
||||
# if using FQDN, make sure csi plugin's dns policy is appropriate.
|
||||
monitors: mon1:port,mon2:port,...
|
||||
|
||||
# if "monitors" parameter is not set, driver to get monitors from same
|
||||
# secret as admin/user credentials. "monValueFromSecret" provides the
|
||||
# key in the secret whose value is the mons
|
||||
#monValueFromSecret: "monitors"
|
||||
|
||||
|
||||
# Ceph pool into which the RBD image shall be created
|
||||
pool: rbd
|
||||
|
||||
# RBD image format. Defaults to "2".
|
||||
imageFormat: "2"
|
||||
|
||||
# RBD image features. Available for imageFormat: "2". CSI RBD currently supports only `layering` feature.
|
||||
imageFeatures: layering
|
||||
|
||||
# The secrets have to contain Ceph admin credentials.
|
||||
csiProvisionerSecretName: csi-rbd-secret
|
||||
csiProvisionerSecretNamespace: default
|
||||
csiNodePublishSecretName: csi-rbd-secret
|
||||
csiNodePublishSecretNamespace: default
|
||||
|
||||
# Ceph users for operating RBD
|
||||
adminid: admin
|
||||
userid: kubernetes
|
||||
# uncomment the following to use rbd-nbd as mounter on supported nodes
|
||||
#mounter: rbd-nbd
|
||||
reclaimPolicy: Delete
|
||||
```
|
||||
### Create the Secret:
|
||||
##### secret.yaml
|
||||
```
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: csi-rbd-secret
|
||||
namespace: default
|
||||
data:
|
||||
# Key value corresponds to a user name defined in ceph cluster
|
||||
admin: BASE64-ENCODED-PASSWORD
|
||||
# Key value corresponds to a user name defined in ceph cluster
|
||||
kubernetes: BASE64-ENCODED-PASSWORD
|
||||
# if monValueFromSecret is set to "monitors", uncomment the
|
||||
# following and set the mon there
|
||||
#monitors: BASE64-ENCODED-Comma-Delimited-Mons
|
||||
```
|
||||
Here, you need your ceph admin/user password encoded in base64. Run `ceph auth ls` in one of your ceph pod, encode the key of your admin/user and replace `BASE64-ENCODED-PASSWORD` by your encoded key.
|
||||
### Create the PersistentVolumeClaim:
|
||||
##### pvc.yaml
|
||||
```
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: rbd-pvc
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
resources:
|
||||
requests:
|
||||
storage: 1Gi
|
||||
storageClassName: csi-rbd
|
||||
```
|
||||
Make sure your `storageClassName` is the name of the StorageClass previously defined in storageclass.yaml
|
||||
|
||||
### Verify the PVC has successfully been created:
|
||||
```
|
||||
# kubectl get pvc
|
||||
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
|
||||
rbd-pvc Bound pvc-c20495c0d5de11e8 1Gi RWO csi-rbd 21s
|
||||
```
|
||||
If your PVC status isn't `Bound`, check the csi-rbdplugin logs to see what's preventing the PVC from being up and bound.
|
||||
### Create the demo Pod:
|
||||
##### pod.yaml
|
||||
```
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: csirbd-demo-pod
|
||||
spec:
|
||||
containers:
|
||||
- name: web-server
|
||||
image: nginx
|
||||
volumeMounts:
|
||||
- name: mypvc
|
||||
mountPath: /var/lib/www/html
|
||||
volumes:
|
||||
- name: mypvc
|
||||
persistentVolumeClaim:
|
||||
claimName: rbd-pvc
|
||||
readOnly: false
|
||||
```
|
||||
|
||||
When running `rbd list block --pool [yourpool]` in one of your ceph pod you should see the created PVC:
|
||||
```
|
||||
# rbd list block --pool rbd
|
||||
pvc-c20495c0d5de11e8
|
||||
```
|
||||
|
||||
# Additional features
|
||||
|
||||
### Snapshots
|
||||
This example is based on [kubernetes-csi/external-snapshotter](https://github.com/kubernetes-csi/external-snapshotter), with a few tweaks to make it work along ceph-csi RBD plugin. This is a basic example of the kubernetes snapshot feature. For more information and functionalities please refer to the [volume snapshot documentation](https://kubernetes.io/docs/concepts/storage/volume-snapshots/).
|
||||
|
||||
Since this feature is still in [alpha stage](https://kubernetes.io/blog/2018/10/09/introducing-volume-snapshot-alpha-for-kubernetes/) (k8s 1.12+), make sure to enable `VolumeSnapshotDataSource` feature gate in your Kubernetes cluster.
|
||||
|
||||
### Enable csi-snapshotter
|
||||
First, create RBAC rules to authorize the snapshotter to access the needed resources
|
||||
```
|
||||
# kubectl create -f https://raw.githubusercontent.com/ceph/ceph-csi/master/examples/rbd/csi-snapshotter-rbac.yaml
|
||||
```
|
||||
Then, deploy the csi-snapshotter service. This file is based on [setup-csi-snapshotter.yaml](https://github.com/kubernetes-csi/external-snapshotter/blob/master/deploy/kubernetes/setup-csi-snapshotter.yaml) without the csi-provisioner and hostpath-plugin containers that are given as an example. The `volumes` part has been modified to match the ceph-csi plugin socket path.
|
||||
|
||||
If you followed this guide without changing anything, this file should be left as is. If you made modifications like changing the socket-dir in the plugin deployment, you must edit this file to match your configuration.
|
||||
```
|
||||
# kubectl create -f https://raw.githubusercontent.com/ceph/ceph-csi/master/examples/rbd/csi-snapshotter.yaml
|
||||
```
|
||||
### Test csi-snapshotter
|
||||
Next you need to create the SnapshotClass. The purpose of a SnapshotClass is defined in [the kubernetes documentation](https://kubernetes.io/docs/concepts/storage/volume-snapshot-classes/). In short, as the documentation describes it:
|
||||
> Just like StorageClass provides a way for administrators to describe the “classes” of storage they offer when provisioning a volume, VolumeSnapshotClass provides a way to describe the “classes” of storage when provisioning a volume snapshot.
|
||||
|
||||
You must download this file and modify it to match your Ceph cluster.
|
||||
```
|
||||
# wget https://raw.githubusercontent.com/ceph/ceph-csi/master/examples/rbd/snapshotclass.yaml
|
||||
```
|
||||
The `csiSnapshotterSecretName` parameter should reference the name of the secret created for the ceph-csi plugin you deployed. The monitors are a comma separated list of your ceph monitors, same as in the StorageClass of the plugin you chosen. When this is done, run:
|
||||
```
|
||||
# kubectl create -f snapshotclass.yaml
|
||||
```
|
||||
|
||||
Finally, create the VolumeSnapshot resource. its `snapshotClassName` should be the name of the VolumeSnapshotClass previously created. The source name should be the name of the PVC you created earlier.
|
||||
```
|
||||
# kubectl create -f https://raw.githubusercontent.com/ceph/ceph-csi/master/examples/rbd/snapshot.yaml
|
||||
```
|
||||
### Verify the Snapshot has successfully been created:
|
||||
```
|
||||
# kubectl get volumesnapshotclass
|
||||
NAME AGE
|
||||
csi-rbdplugin-snapclass 4s
|
||||
# kubectl get volumesnapshot
|
||||
NAME AGE
|
||||
rbd-pvc-snapshot 6s
|
||||
```
|
||||
In one of your ceph pod, run `rbd snap ls [name-of-your-pvc]`.
|
||||
The output should be similar to this:
|
||||
```
|
||||
# rbd snap ls pvc-c20495c0d5de11e8
|
||||
SNAPID NAME SIZE TIMESTAMP
|
||||
4 csi-rbd-pvc-c20495c0d5de11e8-snap-4c0b455b-d5fe-11e8-bebb-525400123456 1024 MB Mon Oct 22 13:28:03 2018
|
||||
|
||||
```
|
||||
|
||||
## Cleanup
|
||||
|
||||
To clean your cluster of the resources created by this example, run the following:
|
||||
|
||||
```
|
||||
# kubectl delete -f pod.yaml
|
||||
# kubectl delete -f pvc.yaml
|
||||
# kubectl delete -f secret.yaml
|
||||
# kubectl delete -f storageclass.yaml
|
||||
|
||||
# kubectl delete -f https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-rbdplugin.yaml
|
||||
# Or https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/cephfs/kubernetes/csi-cephfsplugin.yaml if you deployed cephfs plugin
|
||||
|
||||
# kubectl delete -f https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-rbdplugin-provisioner.yaml
|
||||
# Or https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/cephfs/kubernetes/csi-cephfsplugin-provisioner.yaml
|
||||
|
||||
# kubectl delete -f https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-rbdplugin-attacher.yaml
|
||||
# Or https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/cephfs/kubernetes/csi-cephfsplugin-attacher.yaml
|
||||
|
||||
# kubectl delete -f https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-nodeplugin-rbac.yaml
|
||||
# kubectl delete -f https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-provisioner-rbac.yaml
|
||||
# kubectl delete -f https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-attacher-rbac.yaml
|
||||
```
|
||||
If you tested snapshots too:
|
||||
```
|
||||
# kubectl delete -f https://raw.githubusercontent.com/ceph/ceph-csi/master/examples/rbd/snapshot.yaml
|
||||
# kubectl delete -f snapshotclass.yaml
|
||||
# kubectl delete -f https://raw.githubusercontent.com/ceph/ceph-csi/master/examples/rbd/csi-snapshotter.yaml
|
||||
# kubectl delete -f https://raw.githubusercontent.com/ceph/ceph-csi/master/examples/rbd/csi-snapshotter-rbac.yaml
|
||||
```
|
||||
Reference in new issue
Block a user