feedc0de/my-rook-config
1
0
Fork 0
forked from rook/rook
Code Pull Requests Activity

Merge pull request #12977 from anthonyeleven/improve-csi-addons

Browse Source 
12976 : Improve Documentation/Storage-Configuration/Ceph-CSI/ceph-csi…
This commit is contained in:
Travis Nielsen
2023-12-04 10:10:52 -07:00
committed by GitHub
parent 3097455d78 59d0240676
commit 22b95dbbfd
5 changed files with 74 additions and 74 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/canary-integration-test.yml
+1 -1
View File
@@ -1303,7 +1303,7 @@ jobs:
# echo "snapshots: $snaps"
# if [ $num_snaps_target = $snaps ]
# then echo "Snapshots have synced."
# else echo "Snaps have not synced. NEEDS INVESTIGATION"
# else echo "Snapshots have not synced. NEEDS INVESTIGATION"
# fi
- name: collect common logs
Documentation/Storage-Configuration/Ceph-CSI/ceph-csi-drivers.md
+67 -67
View File
@@ -2,17 +2,17 @@
title: Ceph CSI Drivers
---
There are three CSI drivers integrated with Rook that will enable different scenarios:
There are three CSI drivers integrated with Rook that are used in different scenarios:
* RBD: This block storage driver is optimized for RWO pod access where only one pod may access the
storage. [More information](../Block-Storage-RBD/block-storage.md).
* CephFS: This file storage driver allows for RWX with one or more pods accessing the same storage.
[More information](../Shared-Filesystem-CephFS/filesystem-storage.md).
* NFS (experimental): This file storage driver allows creating NFS exports that can be mounted to
pods, or the exports can be mounted directly via an NFS client from inside or outside the
* NFS (experimental): This file storage driver allows creating NFS exports that can be mounted on
pods, or directly via an NFS client from inside or outside the
Kubernetes cluster. [More information](../NFS/nfs-csi-driver.md)
The Ceph Filesystem (CephFS) and RADOS Block Device (RBD) drivers are enabled automatically with
The Ceph Filesystem (CephFS) and RADOS Block Device (RBD) drivers are enabled automatically by
the Rook operator. The NFS driver is disabled by default. All drivers will be started in the same
namespace as the operator when the first CephCluster CR is created.
@@ -23,22 +23,28 @@ for more information.
## Static Provisioning
Both drivers also support the creation of static PV and static PVC from existing RBD image/CephFS volume. Refer to [static PVC](https://github.com/ceph/ceph-csi/blob/devel/docs/static-pvc.md) for more information.
The RBD and CephFS drivers support the creation of static PVs and static PVCs from
an existing RBD image or CephFS volume/subvolume. Refer
to the [static PVC](https://github.com/ceph/ceph-csi/blob/devel/docs/static-pvc.md) documentation for more information.
## Configure CSI Drivers in non-default namespace
If you've deployed the Rook operator in a namespace other than "rook-ceph",
If you've deployed the Rook operator in a namespace other than `rook-ceph`,
change the prefix in the provisioner to match the namespace you used. For
example, if the Rook operator is running in the namespace "my-namespace" the
provisioner value should be "my-namespace.rbd.csi.ceph.com". The same provisioner
name needs to be set in both the storageclass and snapshotclass.
example, if the Rook operator is running in the namespace `my-namespace` the
provisioner value should be `my-namespace.rbd.csi.ceph.com`. The same provisioner
name must be set in both the storageclass and snapshotclass.
## Liveness Sidecar
All CSI pods are deployed with a sidecar container that provides a prometheus metric for tracking if the CSI plugin is alive and running.
These metrics are meant to be collected by prometheus but can be accesses through a GET request to a specific node ip.
for example `curl -X get http://[pod ip]:[liveness-port][liveness-path] 2>/dev/null | grep csi`
the expected output should be
All CSI pods are deployed with a sidecar container that provides a Prometheus
metric for tracking whether the CSI plugin is alive and running.
These metrics are meant to be scraped (collected) by Prometheus but can also be
accessed through a GET request to a specific node as follows:
`curl -X get http://[pod ip]:[liveness-port][liveness-path] 2>/dev/null | grep csi`
For example:
```console
$ curl -X GET http://10.109.65.142:9080/metrics 2>/dev/null | grep csi
@@ -47,47 +53,42 @@ $ curl -X GET http://10.109.65.142:9080/metrics 2>/dev/null | grep csi
csi_liveness 1
```
Check the [monitoring doc](../Monitoring/ceph-monitoring.md) to see how to integrate CSI
liveness metrics into ceph monitoring.
Check the [monitoring documentation](../Monitoring/ceph-monitoring.md) to see how to integrate CSI
liveness and GRPC metrics into Ceph monitoring.
## Dynamically Expand Volume
### Prerequisite
* For filesystem resize to be supported for your Kubernetes cluster, the
kubernetes version running in your cluster should be >= v1.15 and for block
volume resize support the Kubernetes version should be >= v1.16. Also,
`ExpandCSIVolumes` feature gate has to be enabled for the volume resize
functionality to work.
### Prerequisites
To expand the PVC the controlling StorageClass must have `allowVolumeExpansion`
set to `true`. `csi.storage.k8s.io/controller-expand-secret-name` and
`csi.storage.k8s.io/controller-expand-secret-namespace` values set in
storageclass. Now expand the PVC by editing the PVC
`csi.storage.k8s.io/controller-expand-secret-namespace` values set in the
storageclass. Now expand the PVC by editing the PVC's
`pvc.spec.resource.requests.storage` to a higher values than the current size.
Once PVC is expanded on backend and same is reflected size is reflected on
application mountpoint, the status capacity `pvc.status.capacity.storage` of
PVC will be updated to new size.
Once the PVC is expanded on the back end and the new size is reflected on
the application mountpoint, the status capacity `pvc.status.capacity.storage` of
the PVC will be updated to the new size.
## RBD Mirroring
To support RBD Mirroring, the [CSI-Addons sidecar](https://github.com/csi-addons/kubernetes-csi-addons#readme) will be started in the RBD provisioner pod.
The CSI-Addons supports the VolumeReplication operation. The volume replication controller provides common and reusable APIs for storage disaster recovery. It is based on [csi-addons/spec](https://github.com/csi-addons/spec) specification and can be used by any storage provider.
It follows the controller pattern and provides extended APIs for storage disaster recovery. The extended APIs are provided via Custom Resource Definitions (CRDs).
### Prerequisites
Kubernetes version 1.21 or greater is required.
To support RBD Mirroring,
the [CSI-Addons sidecar](https://github.com/csi-addons/kubernetes-csi-addons#readme)
will be started in the RBD provisioner pod. CSI-Addons support the `VolumeReplication`
operation. The volume replication controller provides common and reusable APIs for
storage disaster recovery. It is based on the [csi-addons](https://github.com/csi-addons/spec) specification.
It follows the controller pattern and provides extended APIs for storage disaster recovery.
The extended APIs are provided via Custom Resource Definitions (CRDs).
### Enable CSIAddons Sidecar
To enable the CSIAddons sidecar and deploy the controller, Please follow the steps [below](#csi-addons-controller)
To enable the CSIAddons sidecar and deploy the controller, follow the
steps [below](#csi-addons-controller)
## Ephemeral volume support
The generic ephemeral volume feature adds support for specifying PVCs in the
`volumes` field to indicate a user would like to create a Volume as part of the pod spec.
This feature requires the GenericEphemeralVolume feature gate to be enabled.
`volumes` field to create a Volume as part of the pod spec.
This feature requires the `GenericEphemeralVolume` feature gate to be enabled.
For example:
@@ -107,20 +108,23 @@ apiVersion: v1
storage: 1Gi
```
A volume claim template is defined inside the pod spec which refers to a volume
provisioned and used by the pod with its lifecycle. The volumes are provisioned
when pod get spawned and destroyed at time of pod delete.
A volume claim template is defined inside the pod spec, and defines a volume to be
provisioned and used by the pod within its lifecycle. Volumes are provisioned
when a pod is spawned and destroyed when the pod is deleted.
Refer to the [ephemeral-doc](https://kubernetes.io/docs/concepts/storage/ephemeral-volumes/#generic-ephemeral-volumes) for more info.
See example manifests for an [RBD ephemeral volume](https://github.com/rook/rook/tree/master/deploy/examples/csi/rbd/pod-ephemeral.yaml)
and a [CephFS ephemeral volume](https://github.com/rook/rook/tree/master/deploy/examples/csi/cephfs/pod-ephemeral.yaml).
Refer to [ephemeral-doc](https://kubernetes.io/docs/concepts/storage/ephemeral-volumes/#generic-ephemeral-volumes) for more info.
Also, See the example manifests for an [RBD ephemeral volume](https://github.com/rook/rook/tree/master/deploy/examples/csi/rbd/pod-ephemeral.yaml) and a [CephFS ephemeral volume](https://github.com/rook/rook/tree/master/deploy/examples/csi/cephfs/pod-ephemeral.yaml).
## CSI-Addons Controller
The CSI-Addons Controller handles the requests from users to initiate an operation. Users create a CR that the controller inspects, and forwards a request to one or more CSI-Addons side-cars for execution.
The CSI-Addons Controller handles requests from users. Users create a CR
that the controller inspects and forwards to one or more CSI-Addons sidecars for execution.
### Deploying the controller
Users can deploy the controller by running the following commands:
Deploy the controller by running the following commands:
```console
kubectl create -f https://raw.githubusercontent.com/csi-addons/kubernetes-csi-addons/v0.7.0/deploy/controller/crds.yaml
@@ -128,7 +132,7 @@ kubectl create -f https://raw.githubusercontent.com/csi-addons/kubernetes-csi-ad
kubectl create -f https://raw.githubusercontent.com/csi-addons/kubernetes-csi-addons/v0.7.0/deploy/controller/setup-controller.yaml
```
This creates the required crds and configure permissions.
This creates the required CRDs and configures permissions.
### Enable the CSI-Addons Sidecar
@@ -136,23 +140,19 @@ To use the features provided by the CSI-Addons, the `csi-addons`
containers need to be deployed in the RBD provisioner and nodeplugin pods,
which are not enabled by default.
Execute the following command in the cluster to enable the CSI-Addons
sidecar:
Execute the following to enable the CSI-Addons sidecars:
* Update the `rook-ceph-operator-config` configmap and patch the
following configurations
following configuration:
```console
kubectl patch cm rook-ceph-operator-config -nrook-ceph -p $'data:\n "CSI_ENABLE_CSIADDONS": "true"'
```
* After enabling `CSI_ENABLE_CSIADDONS` in the configmap, a new sidecar container with name `csi-addons`
should now start automatically in the RBD CSI provisioner and nodeplugin pods.
* After enabling `CSI_ENABLE_CSIADDONS` in the configmap, a new sidecar container named `csi-addons`
will start automatically in the RBD CSI provisioner and nodeplugin pods.
!!! note
Make sure the version of ceph-csi used is `v3.5.0+`.
### CSI-ADDONS Operation
### CSI-Addons Operations
CSI-Addons supports the following operations:
@@ -169,20 +169,20 @@ CSI-Addons supports the following operations:
## Enable RBD Encryption Support
Ceph-CSI supports encrypting individual RBD PersistentVolumeClaim with LUKS encryption. More details can be found
Ceph-CSI supports encrypting individual RBD PersistentVolumeClaims with LUKS. More details can be found
[here](https://github.com/ceph/ceph-csi/blob/v3.6.0/docs/deploy-rbd.md#encryption-for-rbd-volumes)
with full list of supported encryption configurations. A sample configmap can be found
including a full list of supported encryption configurations. A sample configmap can be found
[here](https://github.com/ceph/ceph-csi/blob/v3.6.0/examples/kms/vault/kms-config.yaml).
!!! note
Rook also supports OSD encryption (see `encryptedDevice` option [here](../../CRDs/Cluster/ceph-cluster-crd.md#osd-configuration-settings)).
Rook also supports OSD-level encryption (see `encryptedDevice` option [here](../../CRDs/Cluster/ceph-cluster-crd.md#osd-configuration-settings)).
Using both RBD PVC encryption and OSD encryption together will lead to double encryption and may reduce read/write performance.
Using both RBD PVC encryption and OSD encryption at the same time will lead to double encryption and may reduce read/write performance.
Unlike OSD encryption, existing ceph clusters can also enable Ceph-CSI RBD PVC encryption support and multiple kinds of encryption
KMS can be used on the same ceph cluster using different storageclasses.
Existing Ceph clusters can also enable Ceph-CSI RBD PVC encryption support and multiple kinds of encryption
KMS can be used on the same Ceph cluster using different storageclasses.
Following steps demonstrate how to enable support for encryption:
The following steps demonstrate how to enable support for encryption:
* Create the `rook-ceph-csi-kms-config` configmap with required encryption configuration in
the same namespace where the Rook operator is deployed. An example is shown below:
@@ -210,8 +210,8 @@ data:
kubectl patch cm rook-ceph-operator-config -nrook-ceph -p $'data:\n "CSI_ENABLE_ENCRYPTION": "true"'
```
* Create necessary resources (secrets, configmaps etc) as required by the encryption type.
In this case, create `storage-encryption-secret` secret in the namespace of pvc as shown:
* Create the resources (secrets, configmaps etc) required by the encryption type.
In this case, create `storage-encryption-secret` secret in the namespace of the PVC as follows:
```yaml
apiVersion: v1
@@ -242,11 +242,11 @@ parameters:
## Enable Read affinity for RBD volumes
Ceph CSI supports mapping RBD volumes with krbd options to allow
serving reads from an OSD in proximity to the client, according to
Ceph CSI supports mapping RBD volumes with KRBD options to allow
serving reads from an OSD closest to the client, according to
OSD locations defined in the CRUSH map and topology labels on nodes.
Refer to the [krbd-options](https://docs.ceph.com/en/latest/man/8/rbd/#kernel-rbd-krbd-options)
Refer to the [krbd-options document](https://docs.ceph.com/en/latest/man/8/rbd/#kernel-rbd-krbd-options)
for more details.
Execute the following step to enable read affinity for a specific ceph cluster:
@@ -270,4 +270,4 @@ Ceph CSI will extract the CRUSH location from the topology labels found on the n
and pass it though krbd options during mapping RBD volumes.
!!! note
This requires kernel version 5.8 or higher.
This requires Linux kernel version 5.8 or higher.
Documentation/Storage-Configuration/ceph-teardown.md
+2 -2
View File
@@ -112,8 +112,8 @@ blkdiscard $DISK
partprobe $DISK
```
Ceph can leave LVM and device mapper data that can lock the disks, preventing the disks from being
used again. These steps can help to free up old Ceph disks for reuse. Note that this only needs to
Ceph can leave LVM and device mapper data on storage drives, preventing them from being
redeployed. These steps can clean former Ceph drives for reuse. Note that this only needs to
be run once on each node. If you have **only one** Rook cluster and **all** Ceph disks are
being wiped, run the following command.
deploy/examples/create-external-cluster-resources.py
+1 -1
View File
@@ -557,7 +557,7 @@ class RadosJSON:
)
def _invalid_endpoint(self, endpoint_str):
# separating port, by getting last split of `:` delimiter
# extract the port by getting the last split on `:` delimiter
try:
endpoint_str_ip, port = endpoint_str.rsplit(":", 1)
except ValueError:
tests/framework/clients/block.go
+3 -3
View File
@@ -48,9 +48,9 @@ func CreateBlockOperation(k8shelp *utils.K8sHelper, manifests installer.CephMani
// BlockCreate Function to create a Block using Rook
// Input parameters -
// manifest - pod definition that creates a pvc in k8s - yaml should describe name and size of pvc being created
// size - not user for k8s implementation since its described on the pvc yaml definition
// Output - k8s create pvc operation output and/or error
// manifest - pod definition that creates a PVC in k8s - YAML should describe the name and size of the PVC being created
// size - not user for k8s implementation since it is described in the PVC YAML definition
// Output - k8s create PVC operation output and/or error
func (b *BlockOperation) Create(manifest string, size int) (string, error) {
args := []string{"apply", "-f", "-"}
result, err := b.k8sClient.KubectlWithStdin(manifest, args...)