From 855e18fe40db4a771d5d97e58f336e4988f49afc Mon Sep 17 00:00:00 2001 From: John Mulligan Date: Mon, 18 Mar 2019 16:50:16 -0400 Subject: [PATCH 1/4] design: Improve and update intro paragraphs * Minor grammar and working improvements * Update to current state of CSI in k8s, ceph-csi Signed-off-by: John Mulligan --- design/ceph-csi-driver.md | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/design/ceph-csi-driver.md b/design/ceph-csi-driver.md index ad50fd2cf..00b54b400 100644 --- a/design/ceph-csi-driver.md +++ b/design/ceph-csi-driver.md @@ -3,10 +3,17 @@ ## Background -Container Storage Interface (CSI) is a set of gRPC specifications for container orchestrators to manage storage drivers. CSI spec abstracts -common storage features such as create/delete volumes, publish/unpublish volumes, stage/unstage volumes, and more. It is currently at 0.3.0 release. +Container Storage Interface (CSI) is a set of specifications for container +orchestration frameworks to manage storage. The CSI spec abstracts common +storage features such as create/delete volumes, publish/unpublish volumes, +stage/unstage volumes, and more. It is currently at the 1.0 release. -Kubernetes started to CSI driver alpha support in [1.9](https://kubernetes.io/blog/2018/01/introducing-container-storage-interface/), beta support in [1.10](https://kubernetes.io/blog/2018/04/10/container-storage-interface-beta/). +Kubernetes started to support CSI with alpha support in +[1.9](https://kubernetes.io/blog/2018/01/introducing-container-storage-interface/), +beta support in +[1.10](https://kubernetes.io/blog/2018/04/10/container-storage-interface-beta/), +and CSI 1.0 in [Kubernetes +1.13](https://kubernetes.io/blog/2018/12/03/kubernetes-1-13-release-announcement/). It is projected that CSI will be the only supported persistent storage driver in the near feature. In-tree drivers such as Ceph RBD and CephFS will be replaced with their respective CSI drivers. @@ -14,7 +21,9 @@ in the near feature. In-tree drivers such as Ceph RBD and CephFS will be replace ## Ceph CSI Drivers Status There have been active Ceph CSI drivers developments since Kubernetes 1.9. -Both Ceph RBD and CephFS drivers can be found at [ceph/ceph-csi](https://github.com/ceph/ceph-csi). Currently the drivers are up to CSI v0.3.0 spec. +Both Ceph RBD and CephFS drivers can be found at +[ceph/ceph-csi](https://github.com/ceph/ceph-csi). Currently ceph-csi +supports both the CSI v0.3.0 spec and CSI v1.0 spec. * RBD driver. Currently, rbd CSI driver supports both krbd and rbd-nbd. There is a consideration to support other forms of TCMU based drivers. * CephFS driver. Both Kernel CephFS and Ceph FUSE are supported. When `ceph-fuse` is installed on the CSI plugin container, it can be used to mount CephFS shares. From 2b97bdfc1671e0f936df82e4b907817a4ee3cf54 Mon Sep 17 00:00:00 2001 From: John Mulligan Date: Mon, 18 Mar 2019 17:04:19 -0400 Subject: [PATCH 2/4] design: simplify "Coexist with flex driver" section The "Coexist with flex driver" section was overly technical about details in the implementation of CSI vs. flex. Remove the detail and replace it with more goal oriented paragraphs. Signed-off-by: John Mulligan --- design/ceph-csi-driver.md | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/design/ceph-csi-driver.md b/design/ceph-csi-driver.md index 00b54b400..3e7431603 100644 --- a/design/ceph-csi-driver.md +++ b/design/ceph-csi-driver.md @@ -50,11 +50,18 @@ Rook Ceph Agent supports Ceph mon failover by storing Ceph cluster instead of Ce ### Coexist with flex driver -Rook's local Ceph Node Agent is modeled after flexvolume drivers. There are some overlapping with Kubernetes CSI model: +No changes to the current method of deploying Storage Classes with the flex +driver should be required. Eventually, the flex driver approach will be +deprecated and CSI will become the default method of working with Storage +Classes. At or around the time of flex driver deprecation Rook should provide a +method to upgrade/convert existing flex driver based provisioning with CSI +based provisioning. + +The behavior of the CSI integration must be complementary to the approach +currently taken by the flex volume driver. When CSI is managed through Rook it +should work with the existing Rook CRDs and aim to minimize the required +configuration parameters. -- VolumeAttachment API objects are currently in Kubernetes storage API group. The VolumeAttachment watchers and finalizers can be replaced with CSI attacher. -- There is external CSI volume provisioner that calls out the drivers' create/delete volume. Rook's own provisioner needs to be integrated/replaced with the external provisioner -- The flexvolume drivers can be replaced by CSI drivers when running on Kubernetes 1.10+. ### Work with out-of-band CSI drivers deployment From 58750699945872ce473b72ef1138401cb4e23309 Mon Sep 17 00:00:00 2001 From: John Mulligan Date: Mon, 18 Mar 2019 17:42:27 -0400 Subject: [PATCH 3/4] design: rewrite the rook/ceph-csi itegration section Re-write the Rook/ceph-csi (short term) integration section to conform to my goals and understanding of the integration needs. Signed-off-by: John Mulligan --- design/ceph-csi-driver.md | 54 ++++++++++++++++++++++++++++++++++++--- 1 file changed, 51 insertions(+), 3 deletions(-) diff --git a/design/ceph-csi-driver.md b/design/ceph-csi-driver.md index 3e7431603..1d407a7b5 100644 --- a/design/ceph-csi-driver.md +++ b/design/ceph-csi-driver.md @@ -65,10 +65,58 @@ configuration parameters. ### Work with out-of-band CSI drivers deployment -If CSI drivers are already successfully deployed, Rook should help monitor the Storage Classes used by the CSI drivers, ensuring mon addresses, Pools, Filesystems referenced by the Storage Classes up-to-date. +Rook generally has more information about the state of the cluster than the +static settings in the Storage Class and is more up-to-date than the system's +administrators. When so configured, Rook could additionally manage the +configuration of CSI not directly managed by Rook. + ### Rook initiated CSI drivers deployment -If Rook is instructed to deploy CSI drivers and other external controllers, Rook Operator should create all the driver and daemon deployments as well. +With the addition of CSI 1.0 support in Kubernetes 1.13 Rook should become a +fully-featured method of deploying Ceph-CSI aiming to minimize extra steps +needed to use CSI targeting the Rook managed Ceph cluster(s). Initially, this +would be an opt-in feature that requires Kubernetes 1.13. Supporting CSI +versions earlier than 1.0 will be a non-goal. -The probe based CSI driver discovery feature is currently in alpha in Kubernetes 1.11. If the feature is not turned on, CSI driver registratar will not be able to annotate the node, the external attacher will thus fail to attach the volume. The feature is expected to be in beta in Kubernetes 1.12. Before then, Rook Operator has to annotate the node if necessary. +Opting in to CSI should be simple and require very few changes from the +currently documented approach for deploying Rook. Configuring Rook to use CSI +should require changing only the default mechanism for interacting with the +storage classes. The standard deployment should include the needed RBAC files +for managing storage with Ceph-CSI. Rook should package and/or source all other +needed configuration files/templates. All other configuration must be defaulted +to reasonable values and only require changing if the user requires it. + +The following are plans to converge the user experience when choosing to use +CSI rather than the flex volume method: + +#### Ceph-CSI Requirements + +To manage CSI with Rook the following requirements are placed on the +ceph-csi project: +* Configuration parameters currently set in the Storage Class will be + configurable via secrets: + * Mons (done) + * Admin Id + * User Id +* The key "blockpool" will serve as an alias to "pool" + +To additionally minimize the required parameters in a Storage Class +it may require changes to create CSI instance secrets; secrets +that are associated with CSI outside of the storage class (see https://github.com/ceph/ceph-csi/pull/224). +If this change is made nearly no parameters will be directly required in the +storage class. + +#### Rook Requirements + +To manage CSI with Rook the following requirements are place on Rook: +* Rook deployments must include all the needed RBAC rules to set up CSI +* Rook deploys all additional CSI components required to provision + and mount a volume +* Rook must be able to dynamically update the secrets used to configure + Ceph-CSI, including but not limited to the mons list. +* Users should not be required to deploy Rook differently when using + CSI versus flex except minimal steps to opt in to CSI +* When provisioning Ceph-CSI Rook must uniquely identify the + driver/provisioner name so that multiple CSI drivers or multiple Rook + instances within a (Kubernetes) cluster will not collide From 3c22d31e17b204346ea13143eeb146d5b3946af6 Mon Sep 17 00:00:00 2001 From: John Mulligan Date: Mon, 18 Mar 2019 17:53:57 -0400 Subject: [PATCH 4/4] design: future directions for rook + ceph-csi Signed-off-by: John Mulligan --- design/ceph-csi-driver.md | 33 ++++++++++++++++++++++++++++----- 1 file changed, 28 insertions(+), 5 deletions(-) diff --git a/design/ceph-csi-driver.md b/design/ceph-csi-driver.md index 1d407a7b5..018986527 100644 --- a/design/ceph-csi-driver.md +++ b/design/ceph-csi-driver.md @@ -42,6 +42,9 @@ For example, deploying a CephFS CSI driver consists of the following steps: ## Integration Plan +The aim is to support CSI 1.0 in Rook as a beta with Rook release 1.0. In Rook +1.1 CSI support will be considered stable. + ### How can Rook improve CSI drivers reliability? Rook can ensure resources used by CSI drivers and their associated Storage Classes are created, protected, and updated. Specifically, when a CSI based Storage Class is created, the referenced Pools and Filesystems should exist by the time a PVC uses this Storage Class. Otherwise Rook should resolve missing resources to avoid PVC creation failure. Rook should prevent Pools or Filesystems that are still being used by PVCs from accidental removal. Similarly, when the resources, especially mon addresses, are updated, Rook should try to update the Storage Class as well. @@ -101,11 +104,11 @@ ceph-csi project: * User Id * The key "blockpool" will serve as an alias to "pool" -To additionally minimize the required parameters in a Storage Class -it may require changes to create CSI instance secrets; secrets -that are associated with CSI outside of the storage class (see https://github.com/ceph/ceph-csi/pull/224). -If this change is made nearly no parameters will be directly required in the -storage class. +To additionally minimize the required parameters in a Storage Class it may +require changes to create CSI instance secrets; secrets that are associated +with CSI outside of the storage class (see [ceph-csi +PR#244](https://github.com/ceph/ceph-csi/pull/224)). If this change is made +nearly no parameters will be directly required in the storage class. #### Rook Requirements @@ -120,3 +123,23 @@ To manage CSI with Rook the following requirements are place on Rook: * When provisioning Ceph-CSI Rook must uniquely identify the driver/provisioner name so that multiple CSI drivers or multiple Rook instances within a (Kubernetes) cluster will not collide + + +### Future points of integration + +While not immediately required this section outlines a few improvements +that could be made to improve the Rook and CSI integration: + +#### Extend CephBlockPool and CephFilesystem CRDs to provision Storage Classes + +Extend CephBlockPool and CephFilesystem CRDs to automatically provision Storage +Classes when so configured. Instead of requiring an administrator to create a +CRD and a Storage Class, add metatdata to the CRD such that Rook will +automatically create storage classes based on that additional metadata. + +#### Select Flex Provisioning or CSI based on CephCluster CRD + +Currently the code requires changing numerous parameters to enable CSI. This +document aims to change that to a single parameter. In the future it may be +desirable to make this more of a "runtime" parameter that could be managed in +the cluster CRD.