feedc0de/my-rook-config
1
0 Fork 0
forked from rook/rook
Code Pull Requests Activity

Merge pull request #2857 from phlogistonjohn/csi-one-oh

Browse Source 
Ceph-CSI 1.0 support and integration doc update
This commit is contained in:
Travis Nielsen authored and GitHub committed 2019-04-05 10:24:49 -06:00
2 parents ef336964cd 3c22d31e17
commit 7bac02b06a
1 file changed
+98 -11
Download Patch File Download Diff File Expand all files Collapse all files
design/ceph-csi-driver.md
+98 -11
View File
@@ -3,10 +3,17 @@
## Background
Container Storage Interface (CSI) is a set of gRPC specifications for container orchestrators to manage storage drivers. CSI spec abstracts
common storage features such as create/delete volumes, publish/unpublish volumes, stage/unstage volumes, and more. It is currently at 0.3.0 release.
Container Storage Interface (CSI) is a set of specifications for container
orchestration frameworks to manage storage. The CSI spec abstracts common
storage features such as create/delete volumes, publish/unpublish volumes,
stage/unstage volumes, and more. It is currently at the 1.0 release.
Kubernetes started to CSI driver alpha support in [1.9](https://kubernetes.io/blog/2018/01/introducing-container-storage-interface/), beta support in [1.10](https://kubernetes.io/blog/2018/04/10/container-storage-interface-beta/).
Kubernetes started to support CSI with alpha support in
[1.9](https://kubernetes.io/blog/2018/01/introducing-container-storage-interface/),
beta support in
[1.10](https://kubernetes.io/blog/2018/04/10/container-storage-interface-beta/),
and CSI 1.0 in [Kubernetes
1.13](https://kubernetes.io/blog/2018/12/03/kubernetes-1-13-release-announcement/).
It is projected that CSI will be the only supported persistent storage driver
in the near feature. In-tree drivers such as Ceph RBD and CephFS will be replaced with their respective CSI drivers.
@@ -14,7 +21,9 @@ in the near feature. In-tree drivers such as Ceph RBD and CephFS will be replace
## Ceph CSI Drivers Status
There have been active Ceph CSI drivers developments since Kubernetes 1.9.
Both Ceph RBD and CephFS drivers can be found at [ceph/ceph-csi](https://github.com/ceph/ceph-csi). Currently the drivers are up to CSI v0.3.0 spec.
Both Ceph RBD and CephFS drivers can be found at
[ceph/ceph-csi](https://github.com/ceph/ceph-csi). Currently ceph-csi
supports both the CSI v0.3.0 spec and CSI v1.0 spec.
* RBD driver. Currently, rbd CSI driver supports both krbd and rbd-nbd. There is a consideration to support other forms of TCMU based drivers.
* CephFS driver. Both Kernel CephFS and Ceph FUSE are supported. When `ceph-fuse` is installed on the CSI plugin container, it can be used to mount CephFS shares.
@@ -33,6 +42,9 @@ For example, deploying a CephFS CSI driver consists of the following steps:
## Integration Plan
The aim is to support CSI 1.0 in Rook as a beta with Rook release 1.0. In Rook
1.1 CSI support will be considered stable.
### How can Rook improve CSI drivers reliability?
Rook can ensure resources used by CSI drivers and their associated Storage Classes are created, protected, and updated. Specifically, when a CSI based Storage Class is created, the referenced Pools and Filesystems should exist by the time a PVC uses this Storage Class. Otherwise Rook should resolve missing resources to avoid PVC creation failure. Rook should prevent Pools or Filesystems that are still being used by PVCs from accidental removal. Similarly, when the resources, especially mon addresses, are updated, Rook should try to update the Storage Class as well.
@@ -41,18 +53,93 @@ Rook Ceph Agent supports Ceph mon failover by storing Ceph cluster instead of Ce
### Coexist with flex driver
Rook's local Ceph Node Agent is modeled after flexvolume drivers. There are some overlapping with Kubernetes CSI model:
No changes to the current method of deploying Storage Classes with the flex
driver should be required. Eventually, the flex driver approach will be
deprecated and CSI will become the default method of working with Storage
Classes. At or around the time of flex driver deprecation Rook should provide a
method to upgrade/convert existing flex driver based provisioning with CSI
based provisioning.
The behavior of the CSI integration must be complementary to the approach
currently taken by the flex volume driver. When CSI is managed through Rook it
should work with the existing Rook CRDs and aim to minimize the required
configuration parameters.
- VolumeAttachment API objects are currently in Kubernetes storage API group. The VolumeAttachment watchers and finalizers can be replaced with CSI attacher.
- There is external CSI volume provisioner that calls out the drivers' create/delete volume. Rook's own provisioner needs to be integrated/replaced with the external provisioner
- The flexvolume drivers can be replaced by CSI drivers when running on Kubernetes 1.10+.
### Work with out-of-band CSI drivers deployment
If CSI drivers are already successfully deployed, Rook should help monitor the Storage Classes used by the CSI drivers, ensuring mon addresses, Pools, Filesystems referenced by the Storage Classes up-to-date.
Rook generally has more information about the state of the cluster than the
static settings in the Storage Class and is more up-to-date than the system's
administrators. When so configured, Rook could additionally manage the
configuration of CSI not directly managed by Rook.
### Rook initiated CSI drivers deployment
If Rook is instructed to deploy CSI drivers and other external controllers, Rook Operator should create all the driver and daemon deployments as well.
With the addition of CSI 1.0 support in Kubernetes 1.13 Rook should become a
fully-featured method of deploying Ceph-CSI aiming to minimize extra steps
needed to use CSI targeting the Rook managed Ceph cluster(s). Initially, this
would be an opt-in feature that requires Kubernetes 1.13. Supporting CSI
versions earlier than 1.0 will be a non-goal.
The probe based CSI driver discovery feature is currently in alpha in Kubernetes 1.11. If the feature is not turned on, CSI driver registratar will not be able to annotate the node, the external attacher will thus fail to attach the volume. The feature is expected to be in beta in Kubernetes 1.12. Before then, Rook Operator has to annotate the node if necessary.
Opting in to CSI should be simple and require very few changes from the
currently documented approach for deploying Rook. Configuring Rook to use CSI
should require changing only the default mechanism for interacting with the
storage classes. The standard deployment should include the needed RBAC files
for managing storage with Ceph-CSI. Rook should package and/or source all other
needed configuration files/templates. All other configuration must be defaulted
to reasonable values and only require changing if the user requires it.
The following are plans to converge the user experience when choosing to use
CSI rather than the flex volume method:
#### Ceph-CSI Requirements
To manage CSI with Rook the following requirements are placed on the
ceph-csi project:
* Configuration parameters currently set in the Storage Class will be
configurable via secrets:
* Mons (done)
* Admin Id
* User Id
* The key "blockpool" will serve as an alias to "pool"
To additionally minimize the required parameters in a Storage Class it may
require changes to create CSI instance secrets; secrets that are associated
with CSI outside of the storage class (see [ceph-csi
PR#244](https://github.com/ceph/ceph-csi/pull/224)). If this change is made
nearly no parameters will be directly required in the storage class.
#### Rook Requirements
To manage CSI with Rook the following requirements are place on Rook:
* Rook deployments must include all the needed RBAC rules to set up CSI
* Rook deploys all additional CSI components required to provision
and mount a volume
* Rook must be able to dynamically update the secrets used to configure
Ceph-CSI, including but not limited to the mons list.
* Users should not be required to deploy Rook differently when using
CSI versus flex except minimal steps to opt in to CSI
* When provisioning Ceph-CSI Rook must uniquely identify the
driver/provisioner name so that multiple CSI drivers or multiple Rook
instances within a (Kubernetes) cluster will not collide
### Future points of integration
While not immediately required this section outlines a few improvements
that could be made to improve the Rook and CSI integration:
#### Extend CephBlockPool and CephFilesystem CRDs to provision Storage Classes
Extend CephBlockPool and CephFilesystem CRDs to automatically provision Storage
Classes when so configured. Instead of requiring an administrator to create a
CRD and a Storage Class, add metatdata to the CRD such that Rook will
automatically create storage classes based on that additional metadata.
#### Select Flex Provisioning or CSI based on CephCluster CRD
Currently the code requires changing numerous parameters to enable CSI. This
document aims to change that to a single parameter. In the future it may be
desirable to make this more of a "runtime" parameter that could be managed in
the cluster CRD.