diff --git a/.github/workflows/docs-check.yml b/.github/workflows/docs-check.yml index 1f64e2b77..b83601c4e 100644 --- a/.github/workflows/docs-check.yml +++ b/.github/workflows/docs-check.yml @@ -33,6 +33,15 @@ jobs: with: python-version: 3.9 + - name: Run helm-docs + run: | + make helm-docs + DIFF=$(git diff) + if [ ! -z "$DIFF" ]; then + echo "Please run 'make helm-docs' locally, commit the updated docs, and push the change. See https://rook.io/docs/rook/latest/Contributing/documentation/#running-helm-docs" + fi + git diff --exit-code + - name: Install mkdocs and dependencies run: cd build/release/ && make deps.docs diff --git a/.vscode/settings.json b/.vscode/settings.json index 8dab85e93..677a1a977 100644 --- a/.vscode/settings.json +++ b/.vscode/settings.json @@ -11,4 +11,5 @@ "python.formatting.provider": "black", "python.linting.pylintEnabled": true, "python.linting.enabled": true, + "markdown.extension.tableFormatter.enabled": false, } diff --git a/Documentation/Contributing/documentation.md b/Documentation/Contributing/documentation.md index ecaaafbce..086ad74e3 100644 --- a/Documentation/Contributing/documentation.md +++ b/Documentation/Contributing/documentation.md @@ -34,3 +34,10 @@ When previewing, now you can navigate your browser to [http://127.0.0.1:8000/](h ``` Please make sure that your Python binary path is included in your `PATH`. + +## Running helm-docs +[helm-docs](https://github.com/norwoodj/helm-docs) is a tool that generates the documentation for a helm chart automatically. If there are changes in the helm chart, you need to run helm-docs manually, and check in the resulting autogenerated md files at the path `/Documentation/Helm-Charts` + +```console +make helm-docs +``` diff --git a/Documentation/Helm-Charts/_templates.gotmpl b/Documentation/Helm-Charts/_templates.gotmpl new file mode 100644 index 000000000..b175944a1 --- /dev/null +++ b/Documentation/Helm-Charts/_templates.gotmpl @@ -0,0 +1,14 @@ +{{ define "chart.valuesTable" -}} +| Parameter | Description | Default | +|-----------|-------------|---------| + {{- range .Values }} +| `{{ .Key }}` | {{ if .Description }}{{ .Description }}{{ else }}{{ .AutoDescription }}{{ end }} | {{ if .Default }}{{ .Default }}{{ else }}{{ .AutoDefault }}{{ end }} | + {{- end }} +{{- end -}} + +{{ define "generatedDocsWarning" -}} + +{{- end -}} diff --git a/Documentation/Helm-Charts/ceph-cluster-chart.gotmpl.md b/Documentation/Helm-Charts/ceph-cluster-chart.gotmpl.md new file mode 100644 index 000000000..8a719c877 --- /dev/null +++ b/Documentation/Helm-Charts/ceph-cluster-chart.gotmpl.md @@ -0,0 +1,157 @@ +--- +title: Ceph Cluster Helm Chart +--- +{{ template "generatedDocsWarning" . }} + +Creates Rook resources to configure a [Ceph](https://ceph.io/) cluster using the [Helm](https://helm.sh) package manager. +This chart is a simple packaging of templates that will optionally create Rook resources such as: + +* CephCluster, CephFilesystem, and CephObjectStore CRs +* Storage classes to expose Ceph RBD volumes, CephFS volumes, and RGW buckets +* Ingress for external access to the dashboard +* Toolbox + +## Prerequisites + +* Kubernetes 1.19+ +* Helm 3.x +* Install the [Rook Operator chart](operator-chart.md) + +## Installing + +The `helm install` command deploys rook on the Kubernetes cluster in the default configuration. +The [configuration](#configuration) section lists the parameters that can be configured during installation. It is +recommended that the rook operator be installed into the `rook-ceph` namespace. The clusters can be installed +into the same namespace as the operator or a separate namespace. + +Rook currently publishes builds of this chart to the `release` and `master` channels. + +**Before installing, review the values.yaml to confirm if the default settings need to be updated.** + +* If the operator was installed in a namespace other than `rook-ceph`, the namespace + must be set in the `operatorNamespace` variable. +* Set the desired settings in the `cephClusterSpec`. The [defaults](https://github.com/rook/rook/tree/master/deploy/charts/rook-ceph-cluster/values.yaml) + are only an example and not likely to apply to your cluster. +* The `monitoring` section should be removed from the `cephClusterSpec`, as it is specified separately in the helm settings. +* The default values for `cephBlockPools`, `cephFileSystems`, and `CephObjectStores` will create one of each, and their corresponding storage classes. +* All Ceph components now have default values for the pod resources. The resources may need to be adjusted in production clusters depending on the load. The resources can also be disabled if Ceph should not be limited (e.g. test clusters). + +### **Release** + +The release channel is the most recent release of Rook that is considered stable for the community. + +The example install assumes you have first installed the [Rook Operator Helm Chart](operator-chart.md) +and created your customized values-override.yaml. + +```console +helm repo add rook-release https://charts.rook.io/release +helm install --create-namespace --namespace rook-ceph rook-ceph-cluster \ + --set operatorNamespace=rook-ceph rook-release/rook-ceph-cluster -f values-override.yaml +``` + +## Configuration + +The following table lists the configurable parameters of the rook-operator chart and their default values. + +{{ template "chart.valuesTable" . }} + +### **Ceph Cluster Spec** + +The `CephCluster` CRD takes its spec from `cephClusterSpec.*`. This is not an exhaustive list of parameters. +For the full list, see the [Cluster CRD](../CRDs/Cluster/ceph-cluster-crd.md) topic. + +### **Ceph Block Pools** + +The `cephBlockPools` array in the values file will define a list of CephBlockPool as described in the table below. + +| Parameter | Description | Default | +| --------- | ----------- | ------- | +| `name` | The name of the CephBlockPool | `ceph-blockpool` | +| `spec` | The CephBlockPool spec, see the [CephBlockPool](../CRDs/Block-Storage/ceph-block-pool-crd.md#spec) documentation. | `{}` | +| `storageClass.enabled` | Whether a storage class is deployed alongside the CephBlockPool | `true` | +| `storageClass.isDefault` | Whether the storage class will be the default storage class for PVCs. See [PersistentVolumeClaim documentation](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims) for details. | `true` | +| `storageClass.name` | The name of the storage class | `ceph-block` | +| `storageClass.parameters` | See [Block Storage](../Storage-Configuration/Block-Storage-RBD/block-storage.md) documentation or the helm values.yaml for suitable values | see values.yaml | +| `storageClass.reclaimPolicy` | The default [Reclaim Policy](https://kubernetes.io/docs/concepts/storage/storage-classes/#reclaim-policy) to apply to PVCs created with this storage class. | `Delete` | +| `storageClass.allowVolumeExpansion` | Whether [volume expansion](https://kubernetes.io/docs/concepts/storage/storage-classes/#allow-volume-expansion) is allowed by default. | `true` | +| `storageClass.mountOptions` | Specifies the mount options for storageClass | `[]` | + +### **Ceph File Systems** + +The `cephFileSystems` array in the values file will define a list of CephFileSystem as described in the table below. + +| Parameter | Description | Default | +| --------- | ----------- | ------- | +| `name` | The name of the CephFileSystem | `ceph-filesystem` | +| `spec` | The CephFileSystem spec, see the [CephFilesystem CRD](../CRDs/Shared-Filesystem/ceph-filesystem-crd.md) documentation. | see values.yaml | +| `storageClass.enabled` | Whether a storage class is deployed alongside the CephFileSystem | `true` | +| `storageClass.name` | The name of the storage class | `ceph-filesystem` | +| `storageClass.pool` | The name of [Data Pool](../CRDs/Shared-Filesystem/ceph-filesystem-crd.md#pools), without the filesystem name prefix | `data0` | +| `storageClass.parameters` | See [Shared Filesystem](../Storage-Configuration/Shared-Filesystem-CephFS/filesystem-storage.md) documentation or the helm values.yaml for suitable values | see values.yaml | +| `storageClass.reclaimPolicy` | The default [Reclaim Policy](https://kubernetes.io/docs/concepts/storage/storage-classes/#reclaim-policy) to apply to PVCs created with this storage class. | `Delete` | +| `storageClass.mountOptions` | Specifies the mount options for storageClass | `[]` | + +### **Ceph Object Stores** + +The `cephObjectStores` array in the values file will define a list of CephObjectStore as described in the table below. + +| Parameter | Description | Default | +| --------- | ----------- | ------- | +| `name` | The name of the CephObjectStore | `ceph-objectstore` | +| `spec` | The CephObjectStore spec, see the [CephObjectStore CRD](../CRDs/Object-Storage/ceph-object-store-crd.md) documentation. | see values.yaml | +| `storageClass.enabled` | Whether a storage class is deployed alongside the CephObjectStore | `true` | +| `storageClass.name` | The name of the storage class | `ceph-bucket` | +| `storageClass.parameters` | See [Object Store storage class](../Storage-Configuration/Object-Storage-RGW/ceph-object-bucket-claim.md) documentation or the helm values.yaml for suitable values | see values.yaml | +| `storageClass.reclaimPolicy` | The default [Reclaim Policy](https://kubernetes.io/docs/concepts/storage/storage-classes/#reclaim-policy) to apply to PVCs created with this storage class. | `Delete` | + +### **Existing Clusters** + +If you have an existing CephCluster CR that was created without the helm chart and you want the helm +chart to start managing the cluster: + +1. Extract the `spec` section of your existing CephCluster CR and copy to the `cephClusterSpec` + section in `values-override.yaml`. + +2. Add the following annotations and label to your existing CephCluster CR: + +```yaml + annotations: + meta.helm.sh/release-name: rook-ceph-cluster + meta.helm.sh/release-namespace: rook-ceph + labels: + app.kubernetes.io/managed-by: Helm +``` + +1. Run the `helm install` command in the [Installing section](#release) to create the chart. + +2. In the future when updates to the cluster are needed, ensure the values-override.yaml always + contains the desired CephCluster spec. + +### **Development Build** + +To deploy from a local build from your development environment: + +```console +cd deploy/charts/rook-ceph-cluster +helm install --create-namespace --namespace rook-ceph rook-ceph-cluster -f values-override.yaml . +``` + +## Uninstalling the Chart + +To see the currently installed Rook chart: + +```console +helm ls --namespace rook-ceph +``` + +To uninstall/delete the `rook-ceph-cluster` chart: + +```console +helm delete --namespace rook-ceph rook-ceph-cluster +``` + +The command removes all the Kubernetes components associated with the chart and deletes the release. Removing the cluster +chart does not remove the Rook operator. In addition, all data on hosts in the Rook data directory +(`/var/lib/rook` by default) and on OSD raw devices is kept. To reuse disks, you will have to wipe them before recreating the cluster. + +See the [teardown documentation](../Storage-Configuration/ceph-teardown.md) for more information. diff --git a/Documentation/Helm-Charts/ceph-cluster-chart.md b/Documentation/Helm-Charts/ceph-cluster-chart.md index 048e07675..dc69cf4fd 100644 --- a/Documentation/Helm-Charts/ceph-cluster-chart.md +++ b/Documentation/Helm-Charts/ceph-cluster-chart.md @@ -1,6 +1,10 @@ --- title: Ceph Cluster Helm Chart --- + Creates Rook resources to configure a [Ceph](https://ceph.io/) cluster using the [Helm](https://helm.sh) package manager. This chart is a simple packaging of templates that will optionally create Rook resources such as: @@ -50,26 +54,33 @@ helm install --create-namespace --namespace rook-ceph rook-ceph-cluster \ ## Configuration -The following tables lists the configurable parameters of the rook-operator chart and their default values. +The following table lists the configurable parameters of the rook-operator chart and their default values. -| Parameter | Description | Default | -| --------------------------------------- | ------------------------------------------------------------------------------------------ | --------------- | -| `operatorNamespace` | Namespace of the Rook Operator | `rook-ceph` | -| `kubeVersion` | Optional override of the target kubernetes version | `` | -| `configOverride` | Cluster ceph.conf override | `` | -| `toolbox.enabled` | Enable Ceph debugging pod deployment. See [toolbox](../Troubleshooting/ceph-toolbox.md) | `false` | -| `toolbox.tolerations` | Toolbox tolerations | `[]` | -| `toolbox.affinity` | Toolbox affinity | `{}` | -| `toolbox.resources` | Toolbox resources | see values.yaml | -| `monitoring.enabled` | Enable Prometheus integration, will also create necessary RBAC rules | `false` | -| `monitoring.createPrometheusRules` | Whether to create the Prometheus rules for Ceph alerts | `false` | -| `monitoring.prometheusRule.labels` | Labels applied to PrometheusRule | `false` | -| `monitoring.prometheusRule.annotations` | Annotations applied to PrometheusRule | `false` | -| `cephClusterSpec.*` | Cluster configuration, see below | See below | -| `ingress.dashboard` | Enable an ingress for the ceph-dashboard | `{}` | -| `cephBlockPools.[*]` | A list of CephBlockPool configurations to deploy | See below | -| `cephFileSystems.[*]` | A list of CephFileSystem configurations to deploy | See below | -| `cephObjectStores.[*]` | A list of CephObjectStore configurations to deploy | See below | +| Parameter | Description | Default | +|-----------|-------------|---------| +| `cephBlockPools` | A list of CephBlockPool configurations to deploy | See [below](#ceph-block-pools) | +| `cephBlockPoolsVolumeSnapshotClass` | Settings for the block pool snapshot class | See [RBD Snapshots](../Storage-Configuration/Ceph-CSI/ceph-csi-snapshot.md#rbd-snapshots) | +| `cephClusterSpec` | Cluster configuration. | See [below](#ceph-cluster-spec) | +| `cephFileSystemVolumeSnapshotClass` | Settings for the filesystem snapshot class | See [CephFS Snapshots](../Storage-Configuration/Ceph-CSI/ceph-csi-snapshot.md#cephfs-snapshots) | +| `cephFileSystems` | A list of CephFileSystem configurations to deploy | See [below](#ceph-file-systems) | +| `cephObjectStores` | A list of CephObjectStore configurations to deploy | See [below](#ceph-object-stores) | +| `clusterName` | The metadata.name of the CephCluster CR | The same as the namespace | +| `configOverride` | Cluster ceph.conf override | `nil` | +| `ingress.dashboard` | Enable an ingress for the ceph-dashboard | `{}` | +| `kubeVersion` | Optional override of the target kubernetes version | `nil` | +| `monitoring.createPrometheusRules` | Whether to create the Prometheus rules for Ceph alerts | `false` | +| `monitoring.enabled` | Enable Prometheus integration, will also create necessary RBAC rules to allow Operator to create ServiceMonitors. Monitoring requires Prometheus to be pre-installed | `false` | +| `monitoring.prometheusRule.annotations` | Annotations applied to PrometheusRule | `{}` | +| `monitoring.prometheusRule.labels` | Labels applied to PrometheusRule | `{}` | +| `monitoring.rulesNamespaceOverride` | The namespace in which to create the prometheus rules, if different from the rook cluster namespace If you have multiple rook-ceph clusters in the same k8s cluster, choose the same namespace (ideally, namespace with prometheus deployed) to set rulesNamespace for all the clusters. Otherwise, you will get duplicate alerts with multiple alert definitions. | `nil` | +| `operatorNamespace` | Namespace of the main rook operator | `"rook-ceph"` | +| `pspEnable` | Create & use PSP resources. Set this to the same value as the rook-ceph chart. | `false` | +| `toolbox.affinity` | Toolbox affinity | `{}` | +| `toolbox.enabled` | Enable Ceph debugging pod deployment. See [toolbox](../Troubleshooting/ceph-toolbox.md) | `false` | +| `toolbox.image` | Toolbox image | `"rook/ceph:VERSION"` | +| `toolbox.priorityClassName` | Set the priority class for the toolbox if desired | `nil` | +| `toolbox.resources` | Toolbox resources | `{"limits":{"cpu":"500m","memory":"1Gi"},"requests":{"cpu":"100m","memory":"128Mi"}}` | +| `toolbox.tolerations` | Toolbox tolerations | `[]` | ### **Ceph Cluster Spec** @@ -80,45 +91,45 @@ For the full list, see the [Cluster CRD](../CRDs/Cluster/ceph-cluster-crd.md) to The `cephBlockPools` array in the values file will define a list of CephBlockPool as described in the table below. -| Parameter | Description | Default | -| ----------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------- | -| `name` | The name of the CephBlockPool | `ceph-blockpool` | -| `spec` | The CephBlockPool spec, see the [CephBlockPool](../CRDs/Block-Storage/ceph-block-pool-crd.md#spec) documentation. | `{}` | -| `storageClass.enabled` | Whether a storage class is deployed alongside the CephBlockPool | `true` | -| `storageClass.isDefault` | Whether the storage class will be the default storage class for PVCs. See [PersistentVolumeClaim documentation](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims) for details. | `true` | -| `storageClass.name` | The name of the storage class | `ceph-block` | -| `storageClass.parameters` | See [Block Storage](../Storage-Configuration/Block-Storage-RBD/block-storage.md) documentation or the helm values.yaml for suitable values | see values.yaml | -| `storageClass.reclaimPolicy` | The default [Reclaim Policy](https://kubernetes.io/docs/concepts/storage/storage-classes/#reclaim-policy) to apply to PVCs created with this storage class. | `Delete` | -| `storageClass.allowVolumeExpansion` | Whether [volume expansion](https://kubernetes.io/docs/concepts/storage/storage-classes/#allow-volume-expansion) is allowed by default. | `true` | -| `storageClass.mountOptions` | Specifies the mount options for storageClass | `[]` | +| Parameter | Description | Default | +| --------- | ----------- | ------- | +| `name` | The name of the CephBlockPool | `ceph-blockpool` | +| `spec` | The CephBlockPool spec, see the [CephBlockPool](../CRDs/Block-Storage/ceph-block-pool-crd.md#spec) documentation. | `{}` | +| `storageClass.enabled` | Whether a storage class is deployed alongside the CephBlockPool | `true` | +| `storageClass.isDefault` | Whether the storage class will be the default storage class for PVCs. See [PersistentVolumeClaim documentation](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims) for details. | `true` | +| `storageClass.name` | The name of the storage class | `ceph-block` | +| `storageClass.parameters` | See [Block Storage](../Storage-Configuration/Block-Storage-RBD/block-storage.md) documentation or the helm values.yaml for suitable values | see values.yaml | +| `storageClass.reclaimPolicy` | The default [Reclaim Policy](https://kubernetes.io/docs/concepts/storage/storage-classes/#reclaim-policy) to apply to PVCs created with this storage class. | `Delete` | +| `storageClass.allowVolumeExpansion` | Whether [volume expansion](https://kubernetes.io/docs/concepts/storage/storage-classes/#allow-volume-expansion) is allowed by default. | `true` | +| `storageClass.mountOptions` | Specifies the mount options for storageClass | `[]` | ### **Ceph File Systems** The `cephFileSystems` array in the values file will define a list of CephFileSystem as described in the table below. -| Parameter | Description | Default | -| ---------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------- | -| `name` | The name of the CephFileSystem | `ceph-filesystem` | -| `spec` | The CephFileSystem spec, see the [CephFilesystem CRD](../CRDs/Shared-Filesystem/ceph-filesystem-crd.md) documentation. | see values.yaml | -| `storageClass.enabled` | Whether a storage class is deployed alongside the CephFileSystem | `true` | -| `storageClass.name` | The name of the storage class | `ceph-filesystem` | -| `storageClass.pool` | The name of [Data Pool](../CRDs/Shared-Filesystem/ceph-filesystem-crd.md#pools), without the filesystem name prefix | `data0` | -| `storageClass.parameters` | See [Shared Filesystem](../Storage-Configuration/Shared-Filesystem-CephFS/filesystem-storage.md) documentation or the helm values.yaml for suitable values | see values.yaml | -| `storageClass.reclaimPolicy` | The default [Reclaim Policy](https://kubernetes.io/docs/concepts/storage/storage-classes/#reclaim-policy) to apply to PVCs created with this storage class. | `Delete` | -| `storageClass.mountOptions` | Specifies the mount options for storageClass | `[]` | +| Parameter | Description | Default | +| --------- | ----------- | ------- | +| `name` | The name of the CephFileSystem | `ceph-filesystem` | +| `spec` | The CephFileSystem spec, see the [CephFilesystem CRD](../CRDs/Shared-Filesystem/ceph-filesystem-crd.md) documentation. | see values.yaml | +| `storageClass.enabled` | Whether a storage class is deployed alongside the CephFileSystem | `true` | +| `storageClass.name` | The name of the storage class | `ceph-filesystem` | +| `storageClass.pool` | The name of [Data Pool](../CRDs/Shared-Filesystem/ceph-filesystem-crd.md#pools), without the filesystem name prefix | `data0` | +| `storageClass.parameters` | See [Shared Filesystem](../Storage-Configuration/Shared-Filesystem-CephFS/filesystem-storage.md) documentation or the helm values.yaml for suitable values | see values.yaml | +| `storageClass.reclaimPolicy` | The default [Reclaim Policy](https://kubernetes.io/docs/concepts/storage/storage-classes/#reclaim-policy) to apply to PVCs created with this storage class. | `Delete` | +| `storageClass.mountOptions` | Specifies the mount options for storageClass | `[]` | ### **Ceph Object Stores** The `cephObjectStores` array in the values file will define a list of CephObjectStore as described in the table below. -| Parameter | Description | Default | -| ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------ | -| `name` | The name of the CephObjectStore | `ceph-objectstore` | -| `spec` | The CephObjectStore spec, see the [CephObjectStore CRD](../CRDs/Object-Storage/ceph-object-store-crd.md) documentation. | see values.yaml | -| `storageClass.enabled` | Whether a storage class is deployed alongside the CephObjectStore | `true` | -| `storageClass.name` | The name of the storage class | `ceph-bucket` | -| `storageClass.parameters` | See [Object Store storage class](../Storage-Configuration/Object-Storage-RGW/ceph-object-bucket-claim.md) documentation or the helm values.yaml for suitable values | see values.yaml | -| `storageClass.reclaimPolicy` | The default [Reclaim Policy](https://kubernetes.io/docs/concepts/storage/storage-classes/#reclaim-policy) to apply to PVCs created with this storage class. | `Delete` | +| Parameter | Description | Default | +| --------- | ----------- | ------- | +| `name` | The name of the CephObjectStore | `ceph-objectstore` | +| `spec` | The CephObjectStore spec, see the [CephObjectStore CRD](../CRDs/Object-Storage/ceph-object-store-crd.md) documentation. | see values.yaml | +| `storageClass.enabled` | Whether a storage class is deployed alongside the CephObjectStore | `true` | +| `storageClass.name` | The name of the storage class | `ceph-bucket` | +| `storageClass.parameters` | See [Object Store storage class](../Storage-Configuration/Object-Storage-RGW/ceph-object-bucket-claim.md) documentation or the helm values.yaml for suitable values | see values.yaml | +| `storageClass.reclaimPolicy` | The default [Reclaim Policy](https://kubernetes.io/docs/concepts/storage/storage-classes/#reclaim-policy) to apply to PVCs created with this storage class. | `Delete` | ### **Existing Clusters** diff --git a/Documentation/Helm-Charts/operator-chart.gotmpl.md b/Documentation/Helm-Charts/operator-chart.gotmpl.md new file mode 100644 index 000000000..1a7d177f2 --- /dev/null +++ b/Documentation/Helm-Charts/operator-chart.gotmpl.md @@ -0,0 +1,78 @@ +--- +title: Ceph Operator Helm Chart +--- +{{ template "generatedDocsWarning" . }} + +Installs [rook](https://github.com/rook/rook) to create, configure, and manage Ceph clusters on Kubernetes. + +## Introduction + +This chart bootstraps a [rook-ceph-operator](https://github.com/rook/rook) deployment on a [Kubernetes](http://kubernetes.io) cluster using the [Helm](https://helm.sh) package manager. + +## Prerequisites + +* Kubernetes 1.19+ +* Helm 3.x + +See the [Helm support matrix](https://helm.sh/docs/topics/version_skew/) for more details. + +## Installing + +The Ceph Operator helm chart will install the basic components necessary to create a storage platform for your Kubernetes cluster. + +1. Install the Helm chart +1. [Create a Rook cluster](../Getting-Started/quickstart.md#create-a-ceph-cluster). + +The `helm install` command deploys rook on the Kubernetes cluster in the default configuration. The [configuration](#configuration) section lists the parameters that can be configured during installation. It is recommended that the rook operator be installed into the `rook-ceph` namespace (you will install your clusters into separate namespaces). + +Rook currently publishes builds of the Ceph operator to the `release` and `master` channels. + +### **Release** + +The release channel is the most recent release of Rook that is considered stable for the community. + +```console +helm repo add rook-release https://charts.rook.io/release +helm install --create-namespace --namespace rook-ceph rook-ceph rook-release/rook-ceph -f values.yaml +``` + +For example settings, see the next section or [values.yaml](https://github.com/rook/rook/tree/master/deploy/charts/rook-ceph/values.yaml) + +## Configuration + +The following table lists the configurable parameters of the rook-operator chart and their default values. + +{{ template "chart.valuesTable" . }} + +[^1]: `nodeAffinity` and `*NodeAffinity` options should have the format `"role=storage,rook; storage=ceph"` or `storage=;role=rook-example` or `storage=;` (_checks only for presence of key_) + +### **Development Build** + +To deploy from a local build from your development environment: + +1. Build the Rook docker image: `make` +1. Copy the image to your K8s cluster, such as with the `docker save` then the `docker load` commands +1. Install the helm chart: + +```console +cd deploy/charts/rook-ceph +helm install --create-namespace --namespace rook-ceph rook-ceph . +``` + +## Uninstalling the Chart + +To see the currently installed Rook chart: + +```console +helm ls --namespace rook-ceph +``` + +To uninstall/delete the `rook-ceph` deployment: + +```console +helm delete --namespace rook-ceph rook-ceph +``` + +The command removes all the Kubernetes components associated with the chart and deletes the release. + +After uninstalling you may want to clean up the CRDs as described on the [teardown documentation](../Storage-Configuration/ceph-teardown.md#removing-the-cluster-crd-finalizer). diff --git a/Documentation/Helm-Charts/operator-chart.md b/Documentation/Helm-Charts/operator-chart.md index 133e01445..5cb3a4533 100644 --- a/Documentation/Helm-Charts/operator-chart.md +++ b/Documentation/Helm-Charts/operator-chart.md @@ -1,6 +1,10 @@ --- title: Ceph Operator Helm Chart --- + Installs [rook](https://github.com/rook/rook) to create, configure, and manage Ceph clusters on Kubernetes. @@ -39,107 +43,105 @@ For example settings, see the next section or [values.yaml](https://github.com/r ## Configuration -The following tables lists the configurable parameters of the rook-operator chart and their default values. +The following table lists the configurable parameters of the rook-operator chart and their default values. -| Parameter | Description | Default | -| ------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------- | -| `image.repository` | Image | `rook/ceph` | -| `image.tag` | Image tag | `master` | -| `image.pullPolicy` | Image pull policy | `IfNotPresent` | -| `crds.enabled` | If true, the helm chart will create the Rook CRDs. Do NOT change to `false` in a running cluster or CRs will be deleted! | `true` | -| `rbacEnable` | If true, create & use RBAC resources | `true` | -| `pspEnable` | If true, create & use PSP resources | `false` | -| `resources` | Pod resource requests & limits | `{}` | -| `annotations` | Pod annotations | `{}` | -| `logLevel` | Global log level | `INFO` | -| `nodeSelector` | Kubernetes `nodeSelector` to add to the Deployment. | `` | -| `tolerations` | List of Kubernetes `tolerations` to add to the Deployment. | `[]` | -| `unreachableNodeTolerationSeconds` | Delay to use for the node.kubernetes.io/unreachable pod failure toleration to override the Kubernetes default of 5 minutes | `5s` | -| `currentNamespaceOnly` | Whether the operator should watch cluster CRD in its own namespace or not | `false` | -| `hostpathRequiresPrivileged` | Runs Ceph Pods as privileged to be able to write to `hostPath`s in OpenShift with SELinux restrictions. | `false` | -| `allowLoopDevices` | If true, loop devices are allowed to be used for osds in test clusters | `false` | -| `discover.priorityClassName` | The priority class name to add to the discover pods | `` | -| `discover.toleration` | Toleration for the discover pods | `` | -| `discover.tolerationKey` | The specific key of the taint to tolerate | `` | -| `discover.tolerations` | Array of tolerations in YAML format which will be added to discover deployment | `` | -| `discover.nodeAffinity` | The node labels for affinity of `discover-agent` [^1] | `` | -| `discover.podLabels` | Labels to add to the discover pods. | `` | -| `discover.resources` | Add resources to discover daemon pods. | `` | -| `csi.enableRbdDriver` | Enable Ceph CSI RBD driver. | `true` | -| `csi.enableCephfsDriver` | Enable Ceph CSI CephFS driver. | `true` | -| `csi.enableCephfsSnapshotter` | Enable Snapshotter in CephFS provisioner pod. | `true` | -| `csi.enableNFSSnapshotter` | Enable Snapshotter in NFS provisioner pod. | `true` | -| `csi.enableRBDSnapshotter` | Enable Snapshotter in RBD provisioner pod. | `true` | -| `csi.pluginPriorityClassName` | PriorityClassName to be set on csi driver plugin pods. | `` | -| `csi.provisionerPriorityClassName` | PriorityClassName to be set on csi driver provisioner pods. | `` | -| `csi.enableOMAPGenerator` | EnableOMAP generator deploys omap sidecar in CSI provisioner pod, to enable it set it to true | `false` | -| `csi.rbdFSGroupPolicy` | Policy for modifying a volume's ownership or permissions when the RBD PVC is being mounted | `ReadWriteOnceWithFSType` | -| `csi.cephFSFSGroupPolicy` | Policy for modifying a volume's ownership or permissions when the CephFS PVC is being mounted | `ReadWriteOnceWithFSType` | -| `csi.nfsFSGroupPolicy` | Policy for modifying a volume's ownership or permissions when the NFS PVC is being mounted | `ReadWriteOnceWithFSType` | -| `csi.enableMetadata` | Set volume metadata on the CephFS subvolumes and RBD images. | `false` | -| `csi.clusterName` | Cluster name identifier to set as metadata on the CephFS subvolumes and RBD images. | `` | -| `csi.logLevel` | Set logging level for cephCSI containers maintained by the cephCSI. Supported values from 0 to 5. 0 for general useful logs, 5 for trace level verbosity. | `0` | -| `csi.sidecarLogLevel` | Set logging level for cephCSI sidecar containers. Supported values from 0 to 5. 0 for general useful logs (the default), 5 for trace level verbosity. | `0` | -| `csi.grpcTimeoutInSeconds` | Set GRPC timeout for csi containers. | `150` | -| `csi.provisionerReplicas` | Set replicas for csi provisioner deployment. | `2` | -| `csi.enableGrpcMetrics` | Enable Ceph CSI GRPC Metrics. | `false` | -| `csi.enableCSIHostNetwork` | Enable Host Networking for Ceph CSI nodeplugins. | `false` | -| `csi.enablePluginSelinuxHostMount` | Enable Host mount for /etc/selinux directory for Ceph CSI nodeplugins. | `false` | -| `csi.enableCSIEncryption` | Enable Ceph CSI PVC encryption support. | `false` | -| `csi.provisionerTolerations` | Array of tolerations in YAML format which will be added to CSI provisioner deployment. | `` | -| `csi.provisionerNodeAffinity` | The node labels for affinity of the CSI provisioner deployment [^1] | `` | -| `csi.pluginTolerations` | Array of tolerations in YAML format which will be added to CephCSI plugin DaemonSet | `` | -| `csi.pluginNodeAffinity` | The node labels for affinity of the CephCSI plugin DaemonSet [^1] | `` | -| `csi.rbdProvisionerTolerations` | Array of tolerations in YAML format which will be added to CephCSI RBD provisioner deployment. | `` | -| `csi.rbdProvisionerNodeAffinity` | The node labels for affinity of the CephCSI RBD provisioner deployment [^1] | `` | -| `csi.rbdPluginTolerations` | Array of tolerations in YAML format which will be added to CephCSI RBD plugin DaemonSet | `` | -| `csi.rbdPluginNodeAffinity` | The node labels for affinity of the CephCSI RBD plugin DaemonSet [^1] | `` | -| `csi.cephFSProvisionerTolerations` | Array of tolerations in YAML format which will be added to CephCSI CephFS provisioner deployment. | `` | -| `csi.cephFSProvisionerNodeAffinity` | The node labels for affinity of the CephCSI CephFS provisioner deployment [^1] | `` | -| `csi.cephFSPluginTolerations` | Array of tolerations in YAML format which will be added to CephCSI CephFS plugin DaemonSet | `` | -| `csi.cephFSPluginNodeAffinity` | The node labels for affinity of the CephCSI CephFS plugin DaemonSet [^1] | `` | -| `csi.nfsProvisionerTolerations` | Array of tolerations in YAML format which will be added to CephCSI NFS provisioner deployment. | `` | -| `csi.nfsProvisionerNodeAffinity` | The node labels for affinity of the CephCSI NFS provisioner deployment [^1] | `` | -| `csi.nfsPluginTolerations` | Array of tolerations in YAML format which will be added to CephCSI NFS plugin DaemonSet | `` | -| `csi.nfsPluginNodeAffinity` | The node labels for affinity of the CephCSI NFS plugin DaemonSet [^1] | `` | -| `csi.csiRBDProvisionerResource` | CEPH CSI RBD provisioner resource requirement list. | `` | -| `csi.csiRBDPluginResource` | CEPH CSI RBD plugin resource requirement list. | `` | -| `csi.csiCephFSProvisionerResource` | CEPH CSI CephFS provisioner resource requirement list. | `` | -| `csi.csiCephFSPluginResource` | CEPH CSI CephFS plugin resource requirement list. | `` | -| `csi.csiNFSProvisionerResource` | CEPH CSI NFS provisioner resource requirement list. | `` | -| `csi.csiNFSPluginResource` | CEPH CSI NFS plugin resource requirement list. | `` | -| `csi.cephfsGrpcMetricsPort` | CSI CephFS driver GRPC metrics port. | `9091` | -| `csi.cephfsLivenessMetricsPort` | CSI CephFS driver metrics port. | `9081` | -| `csi.rbdGrpcMetricsPort` | Ceph CSI RBD driver GRPC metrics port. | `9090` | -| `csi.csiAddonsPort` | CSI Addons server port. | `9070` | -| `csi.rbdLivenessMetricsPort` | Ceph CSI RBD driver metrics port. | `8080` | -| `csi.enableLiveness` | Enable Ceph CSI Liveness sidecar deployment. | `false` | -| `csi.forceCephFSKernelClient` | Enable Ceph Kernel clients on kernel < 4.17 which support quotas for Cephfs. | `true` | -| `csi.kubeletDirPath` | Kubelet root directory path (if the Kubelet uses a different path for the `--root-dir` flag) | `/var/lib/kubelet` | -| `csi.cephcsi.image` | Ceph CSI image. | `quay.io/cephcsi/cephcsi:v3.7.2` | -| `csi.rbdPluginUpdateStrategy` | CSI Rbd plugin daemonset update strategy, supported values are OnDelete and RollingUpdate. | `RollingUpdate` | -| `csi.rbdPluginUpdateStrategyMaxUnavailable` | A maxUnavailable parameter of CSI RBD plugin daemonset update strategy. | `1` | -| `csi.cephFSPluginUpdateStrategy` | CSI CephFS plugin daemonset update strategy, supported values are OnDelete and RollingUpdate. | `RollingUpdate` | -| `csi.nfsPluginUpdateStrategy` | CSI NFS plugin daemonset update strategy, supported values are OnDelete and RollingUpdate. | `RollingUpdate` | -| `csi.registrar.image` | Kubernetes CSI registrar image. | `registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1` | -| `csi.resizer.image` | Kubernetes CSI resizer image. | `registry.k8s.io/sig-storage/csi-resizer:v1.6.0` | -| `csi.provisioner.image` | Kubernetes CSI provisioner image. | `registry.k8s.io/sig-storage/csi-provisioner:v3.3.0` | -| `csi.snapshotter.image` | Kubernetes CSI snapshotter image. | `registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0` | -| `csi.attacher.image` | Kubernetes CSI Attacher image. | `registry.k8s.io/sig-storage/csi-attacher:v4.0.0` | -| `csi.cephfsPodLabels` | Labels to add to the CSI CephFS Pods. | `` | -| `csi.rbdPodLabels` | Labels to add to the CSI RBD Pods. | `` | -| `csi.csiAddons.enabled` | Enable CSIAddons | `false` | -| `csi.csiAddons.image` | CSIAddons Sidecar image. | `quay.io/csiaddons/k8s-sidecar:v0.5.0` | -| `csi.topology.enabled` | Enable topology based provisioning. | `false` | -| `csi.topology.domainLabels` | domainLabels define which node labels to use as domains for CSI nodeplugins to advertise their domains. | `` | -| `csi.nfs.enabled` | Enable nfs driver. | `false` | -| `admissionController.tolerations` | Array of tolerations in YAML format which will be added to admission controller deployment. | `` | -| `admissionController.nodeAffinity` | The node labels for affinity of the admission controller deployment [^1] | `` | -| `monitoring.enabled` | Create necessary RBAC rules for Rook to integrate with Prometheus monitoring in the operator namespace. Requires Prometheus to be pre-installed. | `false` | -| `csi.csiRBDPluginVolume` | The volume of the CephCSI RBD plugin DaemonSet | `` | -| `csi.csiRBDPluginVolumeMount` | The volume mounts of the CephCSI RBD plugin DaemonSet | `` | -| `csi.csiCephFSPluginVolume` | The volume of the CephCSI CephFS plugin DaemonSet | `` | -| `csi.csiCephFSPluginVolumeMount` | The volume mounts of the CephCSI CephFS plugin DaemonSet | `` | +| Parameter | Description | Default | +|-----------|-------------|---------| +| `admissionController` | Set tolerations and nodeAffinity [^1] for admission controller pod. The admission controller would be best to start on the same nodes as other ceph daemons. | `nil` | +| `allowLoopDevices` | If true, loop devices are allowed to be used for osds in test clusters | `false` | +| `annotations` | Pod annotations | `{}` | +| `cephCommandsTimeoutSeconds` | The timeout for ceph commands in seconds | `"15"` | +| `crds.enabled` | Whether the helm chart should create and update the CRDs. If false, the CRDs must be managed independently with deploy/examples/crds.yaml. **WARNING** Only set during first deployment. If later disabled the cluster may be DESTROYED. If the CRDs are deleted in this case, see [the disaster recovery guide](https://rook.io/docs/rook/latest/Troubleshooting/disaster-recovery/#restoring-crds-after-deletion) to restore them. | `true` | +| `csi.allowUnsupportedVersion` | Allow starting an unsupported ceph-csi image | `false` | +| `csi.attacher.image` | Kubernetes CSI Attacher image | `registry.k8s.io/sig-storage/csi-attacher:v4.0.0` | +| `csi.cephFSFSGroupPolicy` | Policy for modifying a volume's ownership or permissions when the CephFS PVC is being mounted. supported values are documented at https://kubernetes-csi.github.io/docs/support-fsgroup.html | `"File"` | +| `csi.cephFSPluginUpdateStrategy` | CSI CephFS plugin daemonset update strategy, supported values are OnDelete and RollingUpdate | `RollingUpdate` | +| `csi.cephcsi.image` | Ceph CSI image | `quay.io/cephcsi/cephcsi:v3.7.2` | +| `csi.cephfsGrpcMetricsPort` | CSI CephFS driver GRPC metrics port | `9091` | +| `csi.cephfsLivenessMetricsPort` | CSI CephFS driver metrics port | `9081` | +| `csi.cephfsPodLabels` | Labels to add to the CSI CephFS Deployments and DaemonSets Pods | `nil` | +| `csi.clusterName` | Cluster name identifier to set as metadata on the CephFS subvolume and RBD images. This will be useful in cases like for example, when two container orchestrator clusters (Kubernetes/OCP) are using a single ceph cluster | `nil` | +| `csi.csiAddons.enabled` | Enable CSIAddons | `false` | +| `csi.csiAddons.image` | CSIAddons Sidecar image | `"quay.io/csiaddons/k8s-sidecar:v0.5.0"` | +| `csi.csiAddonsPort` | CSI Addons server port | `9070` | +| `csi.csiCephFSPluginResource` | CEPH CSI CephFS plugin resource requirement list | see values.yaml | +| `csi.csiCephFSPluginVolume` | The volume of the CephCSI CephFS plugin DaemonSet | `nil` | +| `csi.csiCephFSPluginVolumeMount` | The volume mounts of the CephCSI CephFS plugin DaemonSet | `nil` | +| `csi.csiCephFSProvisionerResource` | CEPH CSI CephFS provisioner resource requirement list | see values.yaml | +| `csi.csiNFSPluginResource` | CEPH CSI NFS plugin resource requirement list | see values.yaml | +| `csi.csiNFSProvisionerResource` | CEPH CSI NFS provisioner resource requirement list | see values.yaml | +| `csi.csiRBDPluginResource` | CEPH CSI RBD plugin resource requirement list | see values.yaml | +| `csi.csiRBDPluginVolume` | The volume of the CephCSI RBD plugin DaemonSet | `nil` | +| `csi.csiRBDPluginVolumeMount` | The volume mounts of the CephCSI RBD plugin DaemonSet | `nil` | +| `csi.csiRBDProvisionerResource` | CEPH CSI RBD provisioner resource requirement list csi-omap-generator resources will be applied only if `enableOMAPGenerator` is set to `true` | see values.yaml | +| `csi.enableCSIEncryption` | Enable Ceph CSI PVC encryption support | `false` | +| `csi.enableCSIHostNetwork` | Enable host networking for CSI CephFS and RBD nodeplugins. This may be necessary in some network configurations where the SDN does not provide access to an external cluster or there is significant drop in read/write performance | `true` | +| `csi.enableCephfsDriver` | Enable Ceph CSI CephFS driver | `true` | +| `csi.enableCephfsSnapshotter` | Enable Snapshotter in CephFS provisioner pod | `true` | +| `csi.enableGrpcMetrics` | Enable Ceph CSI GRPC Metrics | `false` | +| `csi.enableLiveness` | Enable Ceph CSI Liveness sidecar deployment | `false` | +| `csi.enableMetadata` | Enable adding volume metadata on the CephFS subvolumes and RBD images. Not all users might be interested in getting volume/snapshot details as metadata on CephFS subvolume and RBD images. Hence enable metadata is false by default | `false` | +| `csi.enableNFSSnapshotter` | Enable Snapshotter in NFS provisioner pod | `true` | +| `csi.enableOMAPGenerator` | OMAP generator generates the omap mapping between the PV name and the RBD image which helps CSI to identify the rbd images for CSI operations. `CSI_ENABLE_OMAP_GENERATOR` needs to be enabled when we are using rbd mirroring feature. By default OMAP generator is disabled and when enabled, it will be deployed as a sidecar with CSI provisioner pod, to enable set it to true. | `false` | +| `csi.enablePluginSelinuxHostMount` | Enable Host mount for `/etc/selinux` directory for Ceph CSI nodeplugins | `false` | +| `csi.enableRBDSnapshotter` | Enable Snapshotter in RBD provisioner pod | `true` | +| `csi.enableRbdDriver` | Enable Ceph CSI RBD driver | `true` | +| `csi.forceCephFSKernelClient` | Enable Ceph Kernel clients on kernel < 4.17. If your kernel does not support quotas for CephFS you may want to disable this setting. However, this will cause an issue during upgrades with the FUSE client. See the [upgrade guide](https://rook.io/docs/rook/v1.2/ceph-upgrade.html) | `true` | +| `csi.grpcTimeoutInSeconds` | Set GRPC timeout for csi containers (in seconds). It should be >= 120. If this value is not set or is invalid, it defaults to 150 | `150` | +| `csi.imagePullPolicy` | Image pull policy | `"IfNotPresent"` | +| `csi.kubeletDirPath` | Kubelet root directory path (if the Kubelet uses a different path for the `--root-dir` flag) | `/var/lib/kubelet` | +| `csi.logLevel` | Set logging level for cephCSI containers maintained by the cephCSI. Supported values from 0 to 5. 0 for general useful logs, 5 for trace level verbosity. | `0` | +| `csi.nfs.enabled` | Enable the nfs csi driver | `false` | +| `csi.nfsFSGroupPolicy` | Policy for modifying a volume's ownership or permissions when the NFS PVC is being mounted. supported values are documented at https://kubernetes-csi.github.io/docs/support-fsgroup.html | `"File"` | +| `csi.nfsPluginUpdateStrategy` | CSI NFS plugin daemonset update strategy, supported values are OnDelete and RollingUpdate | `RollingUpdate` | +| `csi.nfsPodLabels` | Labels to add to the CSI NFS Deployments and DaemonSets Pods | `nil` | +| `csi.pluginNodeAffinity` | The node labels for affinity of the CephCSI RBD plugin DaemonSet [^1] | `nil` | +| `csi.pluginPriorityClassName` | PriorityClassName to be set on csi driver plugin pods | `"system-node-critical"` | +| `csi.pluginTolerations` | Array of tolerations in YAML format which will be added to CephCSI plugin DaemonSet | `nil` | +| `csi.provisioner.image` | Kubernetes CSI provisioner image | `registry.k8s.io/sig-storage/csi-provisioner:v3.3.0` | +| `csi.provisionerNodeAffinity` | The node labels for affinity of the CSI provisioner deployment [^1] | `nil` | +| `csi.provisionerPriorityClassName` | PriorityClassName to be set on csi driver provisioner pods | `"system-cluster-critical"` | +| `csi.provisionerReplicas` | Set replicas for csi provisioner deployment | `2` | +| `csi.provisionerTolerations` | Array of tolerations in YAML format which will be added to CSI provisioner deployment | `nil` | +| `csi.rbdFSGroupPolicy` | Policy for modifying a volume's ownership or permissions when the RBD PVC is being mounted. supported values are documented at https://kubernetes-csi.github.io/docs/support-fsgroup.html | `"File"` | +| `csi.rbdGrpcMetricsPort` | Ceph CSI RBD driver GRPC metrics port | `9090` | +| `csi.rbdLivenessMetricsPort` | Ceph CSI RBD driver metrics port | `8080` | +| `csi.rbdPluginUpdateStrategy` | CSI RBD plugin daemonset update strategy, supported values are OnDelete and RollingUpdate | `RollingUpdate` | +| `csi.rbdPluginUpdateStrategyMaxUnavailable` | A maxUnavailable parameter of CSI RBD plugin daemonset update strategy. | `1` | +| `csi.rbdPodLabels` | Labels to add to the CSI RBD Deployments and DaemonSets Pods | `nil` | +| `csi.registrar.image` | Kubernetes CSI registrar image | `registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1` | +| `csi.resizer.image` | Kubernetes CSI resizer image | `registry.k8s.io/sig-storage/csi-resizer:v1.6.0` | +| `csi.sidecarLogLevel` | Set logging level for Kubernetes-csi sidecar containers. Supported values from 0 to 5. 0 for general useful logs (the default), 5 for trace level verbosity. | `0` | +| `csi.snapshotter.image` | Kubernetes CSI snapshotter image | `registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0` | +| `csi.topology.domainLabels` | domainLabels define which node labels to use as domains for CSI nodeplugins to advertise their domains | `nil` | +| `csi.topology.enabled` | Enable topology based provisioning | `false` | +| `currentNamespaceOnly` | Whether the operator should watch cluster CRD in its own namespace or not | `false` | +| `disableAdmissionController` | Disable adnissions controller | `false` | +| `disableDeviceHotplug` | Disable automatic orchestration when new devices are discovered. | `false` | +| `discover.nodeAffinity` | The node labels for affinity of `discover-agent` [^1] | `nil` | +| `discover.podLabels` | Labels to add to the discover pods | `nil` | +| `discover.resources` | Add resources to discover daemon pods | `nil` | +| `discover.toleration` | Toleration for the discover pods. Options: `NoSchedule`, `PreferNoSchedule` or `NoExecute` | `nil` | +| `discover.tolerationKey` | The specific key of the taint to tolerate | `nil` | +| `discover.tolerations` | Array of tolerations in YAML format which will be added to discover deployment | `nil` | +| `discoverDaemonUdev` | Blacklist certain disks according to the regex provided. | `nil` | +| `enableDiscoveryDaemon` | Enable discovery daemon | `false` | +| `enableOBCWatchOperatorNamespace` | Whether the OBC provisioner should watch on the operator namespace or not, if not the namespace of the cluster will be used | `true` | +| `hostpathRequiresPrivileged` | Runs Ceph Pods as privileged to be able to write to `hostPaths` in OpenShift with SELinux restrictions. | `false` | +| `image.pullPolicy` | Image pull policy | `"IfNotPresent"` | +| `image.repository` | Image | `"rook/ceph"` | +| `image.tag` | Image tag | `master` | +| `imagePullSecrets` | imagePullSecrets option allow to pull docker images from private docker registry. Option will be passed to all service accounts. | `nil` | +| `logLevel` | Global log level for the operator. Options: `ERROR`, `WARNING`, `INFO`, `DEBUG` | `"INFO"` | +| `monitoring.enabled` | Enable monitoring. Requires Prometheus to be pre-installed. Enabling will also create RBAC rules to allow Operator to create ServiceMonitors | `false` | +| `nodeSelector` | Kubernetes [`nodeSelector`](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector) to add to the Deployment. | `{}` | +| `priorityClassName` | Set the priority class for the rook operator deployment if desired | `nil` | +| `pspEnable` | If true, create & use PSP resources | `false` | +| `rbacEnable` | If true, create & use RBAC resources | `true` | +| `resources` | Pod resource requests & limits | `{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"128Mi"}}` | +| `tolerations` | List of Kubernetes [`tolerations`](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) to add to the Deployment. | `[]` | +| `unreachableNodeTolerationSeconds` | Delay to use for the `node.kubernetes.io/unreachable` pod failure toleration to override the Kubernetes default of 5 minutes | `5` | +| `useOperatorHostNetwork` | if true, run rook operator on the host network | `nil` | [^1]: `nodeAffinity` and `*NodeAffinity` options should have the format `"role=storage,rook; storage=ceph"` or `storage=;role=rook-example` or `storage=;` (_checks only for presence of key_) diff --git a/Makefile b/Makefile index ed128c371..3f81bfcbe 100644 --- a/Makefile +++ b/Makefile @@ -196,6 +196,16 @@ gen-rbac: $(HELM) $(YQ) ## Generate RBAC from Helm charts HELM=$(HELM) ./build/rbac/gen-nfs-rbac.sh HELM=$(HELM) ./build/rbac/gen-psp.sh +helm-docs: $(HELM_DOCS) ## Use helm-docs to generate documentation from helm charts + $(HELM_DOCS) -c deploy/charts/rook-ceph \ + -o ../../../Documentation/Helm-Charts/operator-chart.md \ + -t ../../../Documentation/Helm-Charts/operator-chart.gotmpl.md \ + -t ../../../Documentation/Helm-Charts/_templates.gotmpl + $(HELM_DOCS) -c deploy/charts/rook-ceph-cluster \ + -o ../../../Documentation/Helm-Charts/ceph-cluster-chart.md \ + -t ../../../Documentation/Helm-Charts/ceph-cluster-chart.gotmpl.md \ + -t ../../../Documentation/Helm-Charts/_templates.gotmpl + docs-preview: ## Preview the documentation through mkdocs mkdocs serve diff --git a/build/makelib/helm.mk b/build/makelib/helm.mk index e73848512..74fab6d74 100644 --- a/build/makelib/helm.mk +++ b/build/makelib/helm.mk @@ -52,3 +52,18 @@ $(HELM_INDEX): $(HELM) $(HELM_OUTPUT_DIR) @$(HELM) repo index $(HELM_OUTPUT_DIR) helm.build: $(HELM_INDEX) + +# ==================================================================================== +# Makefile helper functions for helm-docs: https://github.com/norwoodj/helm-docs +# + +HELM_DOCS_VERSION := v1.11.0 +HELM_DOCS := $(TOOLS_HOST_DIR)/helm-docs-$(HELM_DOCS_VERSION) +HELM_DOCS_REPO := github.com/norwoodj/helm-docs/cmd/helm-docs + +$(HELM_DOCS): ## Installs helm-docs + @echo === installing helm-docs + @mkdir -p $(TOOLS_HOST_DIR)/tmp + @GOBIN=$(TOOLS_HOST_DIR)/tmp GO111MODULE=on go install $(HELM_DOCS_REPO)@$(HELM_DOCS_VERSION) + @mv $(TOOLS_HOST_DIR)/tmp/helm-docs $(HELM_DOCS) + @rm -fr $(TOOLS_HOST_DIR)/tmp diff --git a/deploy/charts/rook-ceph-cluster/values.yaml b/deploy/charts/rook-ceph-cluster/values.yaml index adc30925f..15dce6133 100644 --- a/deploy/charts/rook-ceph-cluster/values.yaml +++ b/deploy/charts/rook-ceph-cluster/values.yaml @@ -2,16 +2,18 @@ # This is a YAML-formatted file. # Declare variables to be passed into your templates. -# Namespace of the main rook operator +# -- Namespace of the main rook operator operatorNamespace: rook-ceph -# The metadata.name of the CephCluster CR. The default name is the same as the namespace. -# clusterName: rook-ceph +# -- The metadata.name of the CephCluster CR +# @default -- The same as the namespace +clusterName: -# Ability to override the kubernetes version used in rendering the helm chart -# kubeVersion: 1.21 +# -- Optional override of the target kubernetes version +kubeVersion: -# Ability to override ceph.conf +# -- Cluster ceph.conf override +configOverride: # configOverride: | # [global] # mon_allow_pool_delete = true @@ -20,10 +22,15 @@ operatorNamespace: rook-ceph # Installs a debugging toolbox deployment toolbox: + # -- Enable Ceph debugging pod deployment. See [toolbox](../Troubleshooting/ceph-toolbox.md) enabled: false + # -- Toolbox image image: rook/ceph:VERSION + # -- Toolbox tolerations tolerations: [] + # -- Toolbox affinity affinity: {} + # -- Toolbox resources resources: limits: cpu: "500m" @@ -31,16 +38,16 @@ toolbox: requests: cpu: "100m" memory: "128Mi" - # Set the priority class for the toolbox if desired - # priorityClassName: class + # -- Set the priority class for the toolbox if desired + priorityClassName: -# monitoring requires Prometheus to be pre-installed monitoring: - # enabling will also create RBAC rules to allow Operator to create ServiceMonitors + # -- Enable Prometheus integration, will also create necessary RBAC rules to allow Operator to create ServiceMonitors. + # Monitoring requires Prometheus to be pre-installed enabled: false - # whether to create the prometheus rules + # -- Whether to create the Prometheus rules for Ceph alerts createPrometheusRules: false - # the namespace in which to create the prometheus rules, if different from the rook cluster namespace + # -- The namespace in which to create the prometheus rules, if different from the rook cluster namespace # If you have multiple rook-ceph clusters in the same k8s cluster, choose the same namespace (ideally, namespace with prometheus # deployed) to set rulesNamespace for all the clusters. Otherwise, you will get duplicate alerts with multiple alert definitions. rulesNamespaceOverride: @@ -49,10 +56,12 @@ monitoring: # externalMgrPrometheusPort: # allow adding custom labels and annotations to the prometheus rule prometheusRule: + # -- Labels applied to PrometheusRule labels: {} + # -- Annotations applied to PrometheusRule annotations: {} -# If true, create & use PSP resources. Set this to the same value as the rook-ceph chart. +# -- Create & use PSP resources. Set this to the same value as the rook-ceph chart. pspEnable: false # imagePullSecrets option allow to pull docker images from private docker registry. Option will be passed to all service accounts. @@ -60,8 +69,10 @@ pspEnable: false # - name: my-registry-secret # All values below are taken from the CephCluster CRD -# More information can be found at [Ceph Cluster CRD](/Documentation/CRDs/ceph-cluster-crd.md) +# -- Cluster configuration. +# @default -- See [below](#ceph-cluster-spec) cephClusterSpec: + # For more details, check https://rook.io/docs/rook/v1.10/CRDs/Cluster/ceph-cluster-crd/ cephVersion: # The container image used to launch the Ceph daemon pods (mon, mgr, osd, mds, rgw). # v16 is Pacific, v17 is Quincy. @@ -83,7 +94,7 @@ cephClusterSpec: # Whether or not upgrade should continue even if a check fails # This means Ceph's status could be degraded and we don't recommend upgrading but you might decide otherwise # Use at your OWN risk - # To understand Rook's upgrade process of Ceph, read https://rook.io/docs/rook/latest/ceph-upgrade.html#ceph-version-upgrades + # To understand Rook's upgrade process of Ceph, read https://rook.io/docs/rook/v1.10/Upgrade/ceph-upgrade/ skipUpgradeChecks: false # Whether or not continue if PGs are not clean during an upgrade @@ -368,6 +379,7 @@ cephClusterSpec: disabled: false ingress: + # -- Enable an ingress for the ceph-dashboard dashboard: {} # annotations: @@ -389,6 +401,8 @@ ingress: ## to set the ingress class # ingressClassName: nginx +# -- A list of CephBlockPool configurations to deploy +# @default -- See [below](#ceph-block-pools) cephBlockPools: - name: ceph-blockpool # see https://github.com/rook/rook/blob/master/Documentation/CRDs/Block-Storage/ceph-block-pool-crd.md#spec for available configuration @@ -443,6 +457,8 @@ cephBlockPools: # in hyperconverged settings where the volume is mounted on the same node as the osds. csi.storage.k8s.io/fstype: ext4 +# -- A list of CephFileSystem configurations to deploy +# @default -- See [below](#ceph-file-systems) cephFileSystems: - name: ceph-filesystem # see https://github.com/rook/rook/blob/master/Documentation/CRDs/Shared-Filesystem/ceph-filesystem-crd.md#filesystem-settings for available configuration @@ -490,6 +506,8 @@ cephFileSystems: # in hyperconverged settings where the volume is mounted on the same node as the osds. csi.storage.k8s.io/fstype: ext4 +# -- Settings for the filesystem snapshot class +# @default -- See [CephFS Snapshots](../Storage-Configuration/Ceph-CSI/ceph-csi-snapshot.md#cephfs-snapshots) cephFileSystemVolumeSnapshotClass: enabled: false name: ceph-filesystem @@ -497,9 +515,11 @@ cephFileSystemVolumeSnapshotClass: deletionPolicy: Delete annotations: {} labels: {} - # see https://rook.io/docs/rook/latest/ceph-csi-snapshot.html#cephfs-snapshots for available configuration + # see https://rook.io/docs/rook/v1.10/Storage-Configuration/Ceph-CSI/ceph-csi-snapshot/#cephfs-snapshots for available configuration parameters: {} +# -- Settings for the block pool snapshot class +# @default -- See [RBD Snapshots](../Storage-Configuration/Ceph-CSI/ceph-csi-snapshot.md#rbd-snapshots) cephBlockPoolsVolumeSnapshotClass: enabled: false name: ceph-block @@ -507,9 +527,11 @@ cephBlockPoolsVolumeSnapshotClass: deletionPolicy: Delete annotations: {} labels: {} - # see https://rook.io/docs/rook/latest/ceph-csi-snapshot.html#rbd-snapshots for available configuration + # see https://rook.io/docs/rook/v1.10/Storage-Configuration/Ceph-CSI/ceph-csi-snapshot/#rbd-snapshots for available configuration parameters: {} +# -- A list of CephObjectStore configurations to deploy +# @default -- See [below](#ceph-object-stores) cephObjectStores: - name: ceph-objectstore # see https://github.com/rook/rook/blob/master/Documentation/CRDs/Object-Storage/ceph-object-store-crd.md#object-store-settings for available configuration diff --git a/deploy/charts/rook-ceph/values.yaml b/deploy/charts/rook-ceph/values.yaml index 1539564ba..4d1eb6526 100644 --- a/deploy/charts/rook-ceph/values.yaml +++ b/deploy/charts/rook-ceph/values.yaml @@ -3,18 +3,24 @@ # Declare variables to be passed into your templates. image: + # -- Image repository: rook/ceph + # -- Image tag + # @default -- `master` tag: VERSION + # -- Image pull policy pullPolicy: IfNotPresent crds: - # Whether the helm chart should create and update the CRDs. If false, the CRDs must be + # -- Whether the helm chart should create and update the CRDs. If false, the CRDs must be # managed independently with deploy/examples/crds.yaml. # **WARNING** Only set during first deployment. If later disabled the cluster may be DESTROYED. - # If the CRDs are deleted in this case, see the disaster recovery guide to restore them. - # https://rook.io/docs/rook/latest/Troubleshooting/disaster-recovery/#restoring-crds-after-deletion + # If the CRDs are deleted in this case, see + # [the disaster recovery guide](https://rook.io/docs/rook/latest/Troubleshooting/disaster-recovery/#restoring-crds-after-deletion) + # to restore them. enabled: true +# -- Pod resource requests & limits resources: limits: cpu: 500m @@ -23,123 +29,135 @@ resources: cpu: 100m memory: 128Mi +# -- Kubernetes [`nodeSelector`](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector) to add to the Deployment. nodeSelector: {} # Constraint rook-ceph-operator Deployment to nodes with label `disktype: ssd`. # For more info, see https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector # disktype: ssd -# Tolerations for the rook-ceph-operator to allow it to run on nodes with particular taints +# -- List of Kubernetes [`tolerations`](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) to add to the Deployment. tolerations: [] -# Delay to use in node.kubernetes.io/unreachable toleration +# -- Delay to use for the `node.kubernetes.io/unreachable` pod failure toleration to override +# the Kubernetes default of 5 minutes unreachableNodeTolerationSeconds: 5 -# Whether rook watches its current namespace for CRDs or the entire cluster, defaults to false +# -- Whether the operator should watch cluster CRD in its own namespace or not currentNamespaceOnly: false -## Annotations to be added to pod +# -- Pod annotations annotations: {} -## The logging level for the operator: ERROR | WARNING | INFO | DEBUG +# -- Global log level for the operator. +# Options: `ERROR`, `WARNING`, `INFO`, `DEBUG` logLevel: INFO -## If true, create & use RBAC resources -## +# -- If true, create & use RBAC resources rbacEnable: true -## If true, create & use PSP resources -## +# -- If true, create & use PSP resources pspEnable: false -# Set the priority class for the rook operator deployment if desired -# priorityClassName: class +# -- Set the priority class for the rook operator deployment if desired +priorityClassName: -# If true, loop devices are allowed to be used for osds in test clusters +# -- If true, loop devices are allowed to be used for osds in test clusters allowLoopDevices: false -## Settings for whether to disable the drivers or other daemons if they are not -## needed +# Settings for whether to disable the drivers or other daemons if they are not +# needed csi: + # -- Enable Ceph CSI RBD driver enableRbdDriver: true + # -- Enable Ceph CSI CephFS driver enableCephfsDriver: true + # -- Enable Ceph CSI GRPC Metrics enableGrpcMetrics: false - # Set to true to enable host networking for CSI CephFS and RBD nodeplugins. This may be necessary + # -- Enable host networking for CSI CephFS and RBD nodeplugins. This may be necessary # in some network configurations where the SDN does not provide access to an external cluster or - # there is significant drop in read/write performance. + # there is significant drop in read/write performance enableCSIHostNetwork: true - # set to false to disable deployment of snapshotter container in CephFS provisioner pod. + # -- Enable Snapshotter in CephFS provisioner pod enableCephfsSnapshotter: true - # set to false to disable deployment of snapshotter container in NFS provisioner pod. + # -- Enable Snapshotter in NFS provisioner pod enableNFSSnapshotter: true - # set to false to disable deployment of snapshotter container in RBD provisioner pod. + # -- Enable Snapshotter in RBD provisioner pod enableRBDSnapshotter: true - # set to false if the selinux is not enabled or unavailable in cluster nodes. + # -- Enable Host mount for `/etc/selinux` directory for Ceph CSI nodeplugins enablePluginSelinuxHostMount: false - # set to true to enable Ceph CSI pvc encryption support. + # -- Enable Ceph CSI PVC encryption support enableCSIEncryption: false - # (Optional) set user created priorityclassName for csi plugin pods. + # -- PriorityClassName to be set on csi driver plugin pods pluginPriorityClassName: system-node-critical - # (Optional) set user created priorityclassName for csi provisioner pods. + # -- PriorityClassName to be set on csi driver provisioner pods provisionerPriorityClassName: system-cluster-critical - # (Optional) policy for modifying a volume's ownership or permissions when the RBD PVC is being mounted. + # -- Policy for modifying a volume's ownership or permissions when the RBD PVC is being mounted. # supported values are documented at https://kubernetes-csi.github.io/docs/support-fsgroup.html rbdFSGroupPolicy: "File" - # (Optional) policy for modifying a volume's ownership or permissions when the CephFS PVC is being mounted. + # -- Policy for modifying a volume's ownership or permissions when the CephFS PVC is being mounted. # supported values are documented at https://kubernetes-csi.github.io/docs/support-fsgroup.html cephFSFSGroupPolicy: "File" - # (Optional) policy for modifying a volume's ownership or permissions when the NFS PVC is being mounted. + # -- Policy for modifying a volume's ownership or permissions when the NFS PVC is being mounted. # supported values are documented at https://kubernetes-csi.github.io/docs/support-fsgroup.html nfsFSGroupPolicy: "File" - # OMAP generator generates the omap mapping between the PV name and the RBD image + # -- OMAP generator generates the omap mapping between the PV name and the RBD image # which helps CSI to identify the rbd images for CSI operations. - # CSI_ENABLE_OMAP_GENERATOR need to be enabled when we are using rbd mirroring feature. - # By default OMAP generator is disabled and when enabled it will be deployed as a + # `CSI_ENABLE_OMAP_GENERATOR` needs to be enabled when we are using rbd mirroring feature. + # By default OMAP generator is disabled and when enabled, it will be deployed as a # sidecar with CSI provisioner pod, to enable set it to true. enableOMAPGenerator: false - # (Optional) set to true to enable adding volume metadata on the CephFS subvolumes and RBD images. + # -- Enable adding volume metadata on the CephFS subvolumes and RBD images. # Not all users might be interested in getting volume/snapshot details as metadata on CephFS subvolume and RBD images. - # Hence enable metadata is false by default. + # Hence enable metadata is false by default enableMetadata: false - # Set replicas for csi provisioner deployment. + # -- Set replicas for csi provisioner deployment provisionerReplicas: 2 - # (Optional) cluster name identifier to set as metadata on the CephFS subvolume and RBD images. This will be useful - # in cases like for example, when two container orchestrator clusters (Kubernetes/OCP) are using a single ceph cluster. - # clusterName: "my-prod-cluster" - # Set logging level for cephCSI containers maintained by the cephCSI. + # -- Cluster name identifier to set as metadata on the CephFS subvolume and RBD images. This will be useful + # in cases like for example, when two container orchestrator clusters (Kubernetes/OCP) are using a single ceph cluster + clusterName: + + # -- Set logging level for cephCSI containers maintained by the cephCSI. # Supported values from 0 to 5. 0 for general useful logs, 5 for trace level verbosity. - #logLevel: 0 - # Set logging level for Kubernetes-csi sidecar containers. + logLevel: 0 + + # -- Set logging level for Kubernetes-csi sidecar containers. # Supported values from 0 to 5. 0 for general useful logs (the default), 5 for trace level verbosity. - #sidecarLogLevel: 0 - # CSI RBD plugin daemonset update strategy, supported values are OnDelete and RollingUpdate. - # Default value is RollingUpdate. - #rbdPluginUpdateStrategy: OnDelete - # A maxUnavailable parameter of CSI RBD plugin daemonset update strategy. - # Default value is 1. - #rbdPluginUpdateStrategyMaxUnavailable: "1" - # CSI CephFS plugin daemonset update strategy, supported values are OnDelete and RollingUpdate. - # Default value is RollingUpdate. - #cephFSPluginUpdateStrategy: OnDelete - # CSI NFS plugin daemonset update strategy, supported values are OnDelete and RollingUpdate. - # Default value is RollingUpdate. - #nfsPluginUpdateStrategy: OnDelete - # The CSI GRPC timeout value (in seconds). It should be >= 120. If this variable is not set or is an invalid value, it's default to 150. + # @default -- `0` + sidecarLogLevel: + + # -- CSI RBD plugin daemonset update strategy, supported values are OnDelete and RollingUpdate + # @default -- `RollingUpdate` + rbdPluginUpdateStrategy: + + # -- A maxUnavailable parameter of CSI RBD plugin daemonset update strategy. + # @default -- `1` + rbdPluginUpdateStrategyMaxUnavailable: + + # -- CSI CephFS plugin daemonset update strategy, supported values are OnDelete and RollingUpdate + # @default -- `RollingUpdate` + cephFSPluginUpdateStrategy: + + # -- CSI NFS plugin daemonset update strategy, supported values are OnDelete and RollingUpdate + # @default -- `RollingUpdate` + nfsPluginUpdateStrategy: + + # -- Set GRPC timeout for csi containers (in seconds). It should be >= 120. If this value is not set or is invalid, it defaults to 150 grpcTimeoutInSeconds: 150 - # Allow starting unsupported ceph-csi image + # -- Allow starting an unsupported ceph-csi image allowUnsupportedVersion: false - # CephCSI RBD plugin Volumes - # csiRBDPluginVolume: + # -- The volume of the CephCSI RBD plugin DaemonSet + csiRBDPluginVolume: # - name: lib-modules # hostPath: # path: /run/booted-system/kernel-modules/lib/modules/ @@ -147,14 +165,14 @@ csi: # hostPath: # path: /nix - # CephCSI RBD plugin Volume mounts - # csiRBDPluginVolumeMount: + # -- The volume mounts of the CephCSI RBD plugin DaemonSet + csiRBDPluginVolumeMount: # - name: host-nix # mountPath: /nix # readOnly: true - # CephCSI CephFS plugin Volumes - # csiCephFSPluginVolume: + # -- The volume of the CephCSI CephFS plugin DaemonSet + csiCephFSPluginVolume: # - name: lib-modules # hostPath: # path: /run/booted-system/kernel-modules/lib/modules/ @@ -162,15 +180,15 @@ csi: # hostPath: # path: /nix - # CephCSI CephFS plugin Volume mounts - # csiCephFSPluginVolumeMount: + # -- The volume mounts of the CephCSI CephFS plugin DaemonSet + csiCephFSPluginVolumeMount: # - name: host-nix # mountPath: /nix # readOnly: true - # CEPH CSI RBD provisioner resource requirement list, Put here list of resource - # requests and limits you want to apply for provisioner pod - # csi-omap-generator resources will be applied only if enableOMAPGenerator is set to true + # -- CEPH CSI RBD provisioner resource requirement list + # csi-omap-generator resources will be applied only if `enableOMAPGenerator` is set to `true` + # @default -- see values.yaml csiRBDProvisionerResource: | - name : csi-provisioner resource: @@ -228,8 +246,9 @@ csi: limits: memory: 256Mi cpu: 100m - # CEPH CSI RBD plugin resource requirement list, Put here list of resource - # requests and limits you want to apply for plugin pod + + # -- CEPH CSI RBD plugin resource requirement list + # @default -- see values.yaml csiRBDPluginResource: | - name : driver-registrar resource: @@ -255,8 +274,9 @@ csi: limits: memory: 256Mi cpu: 100m - # CEPH CSI CephFS provisioner resource requirement list, Put here list of resource - # requests and limits you want to apply for provisioner pod + + # -- CEPH CSI CephFS provisioner resource requirement list + # @default -- see values.yaml csiCephFSProvisionerResource: | - name : csi-provisioner resource: @@ -306,8 +326,9 @@ csi: limits: memory: 256Mi cpu: 100m - # CEPH CSI CephFS plugin resource requirement list, Put here list of resource - # requests and limits you want to apply for plugin pod + + # -- CEPH CSI CephFS plugin resource requirement list + # @default -- see values.yaml csiCephFSPluginResource: | - name : driver-registrar resource: @@ -333,8 +354,9 @@ csi: limits: memory: 256Mi cpu: 100m - # CEPH CSI NFS provisioner resource requirement list, Put here list of resource - # requests and limits you want to apply for provisioner pod + + # -- CEPH CSI NFS provisioner resource requirement list + # @default -- see values.yaml csiNFSProvisionerResource: | - name : csi-provisioner resource: @@ -352,8 +374,9 @@ csi: limits: memory: 1Gi cpu: 500m - # CEPH CSI NFS plugin resource requirement list, Put here list of resource - # requests and limits you want to apply for plugin pod + + # -- CEPH CSI NFS plugin resource requirement list + # @default -- see values.yaml csiNFSPluginResource: | - name : driver-registrar resource: @@ -374,124 +397,193 @@ csi: # Set provisionerTolerations and provisionerNodeAffinity for provisioner pod. # The CSI provisioner would be best to start on the same nodes as other ceph daemons. - # provisionerTolerations: + + # -- Array of tolerations in YAML format which will be added to CSI provisioner deployment + provisionerTolerations: # - key: key # operator: Exists # effect: NoSchedule - # provisionerNodeAffinity: key1=value1,value2; key2=value3 + + # -- The node labels for affinity of the CSI provisioner deployment [^1] + provisionerNodeAffinity: #key1=value1,value2; key2=value3 # Set pluginTolerations and pluginNodeAffinity for plugin daemonset pods. # The CSI plugins need to be started on all the nodes where the clients need to mount the storage. - # pluginTolerations: + + # -- Array of tolerations in YAML format which will be added to CephCSI plugin DaemonSet + pluginTolerations: # - key: key # operator: Exists # effect: NoSchedule - # pluginNodeAffinity: key1=value1,value2; key2=value3 - # Set to true to enable Ceph CSI liveness container. + + # -- The node labels for affinity of the CephCSI RBD plugin DaemonSet [^1] + pluginNodeAffinity: # key1=value1,value2; key2=value3 + + # -- Enable Ceph CSI Liveness sidecar deployment enableLiveness: false - #cephfsGrpcMetricsPort: 9091 - #cephfsLivenessMetricsPort: 9081 - #rbdGrpcMetricsPort: 9090 - #csiAddonsPort: 9070 - # Enable Ceph Kernel clients on kernel < 4.17. If your kernel does not support quotas for CephFS + + # -- CSI CephFS driver GRPC metrics port + # @default -- `9091` + cephfsGrpcMetricsPort: + + # -- CSI CephFS driver metrics port + # @default -- `9081` + cephfsLivenessMetricsPort: + + # -- Ceph CSI RBD driver GRPC metrics port + # @default -- `9090` + rbdGrpcMetricsPort: + + # -- CSI Addons server port + # @default -- `9070` + csiAddonsPort: + + # -- Enable Ceph Kernel clients on kernel < 4.17. If your kernel does not support quotas for CephFS # you may want to disable this setting. However, this will cause an issue during upgrades - # with the FUSE client. See the upgrade guide: https://rook.io/docs/rook/v1.2/ceph-upgrade.html + # with the FUSE client. See the [upgrade guide](https://rook.io/docs/rook/v1.2/ceph-upgrade.html) forceCephFSKernelClient: true - #rbdLivenessMetricsPort: 9080 - #kubeletDirPath: /var/lib/kubelet - #cephcsi: - # image: quay.io/cephcsi/cephcsi:v3.7.2 - #registrar: - # image: registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1 - #provisioner: - # image: registry.k8s.io/sig-storage/csi-provisioner:v3.3.0 - #snapshotter: - # image: registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0 - #attacher: - # image: registry.k8s.io/sig-storage/csi-attacher:v4.0.0 - #resizer: - # image: registry.k8s.io/sig-storage/csi-resizer:v1.6.0 - #imagePullPolicy: IfNotPresent - # Labels to add to the CSI CephFS Deployments and DaemonSets Pods. - #cephfsPodLabels: "key1=value1,key2=value2" - # Labels to add to the CSI NFS Deployments and DaemonSets Pods. - #nfsPodLabels: "key1=value1,key2=value2" - # Labels to add to the CSI RBD Deployments and DaemonSets Pods. - #rbdPodLabels: "key1=value1,key2=value2" - # Enable the CSIAddons sidecar. + + # -- Ceph CSI RBD driver metrics port + # @default -- `8080` + rbdLivenessMetricsPort: + + # -- Kubelet root directory path (if the Kubelet uses a different path for the `--root-dir` flag) + # @default -- `/var/lib/kubelet` + kubeletDirPath: + + cephcsi: + # -- Ceph CSI image + # @default -- `quay.io/cephcsi/cephcsi:v3.7.2` + image: + + registrar: + # -- Kubernetes CSI registrar image + # @default -- `registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1` + image: + + provisioner: + # -- Kubernetes CSI provisioner image + # @default -- `registry.k8s.io/sig-storage/csi-provisioner:v3.3.0` + image: + + snapshotter: + # -- Kubernetes CSI snapshotter image + # @default -- `registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0` + image: + + attacher: + # -- Kubernetes CSI Attacher image + # @default -- `registry.k8s.io/sig-storage/csi-attacher:v4.0.0` + image: + + resizer: + # -- Kubernetes CSI resizer image + # @default -- `registry.k8s.io/sig-storage/csi-resizer:v1.6.0` + image: + + # -- Image pull policy + imagePullPolicy: IfNotPresent + + # -- Labels to add to the CSI CephFS Deployments and DaemonSets Pods + cephfsPodLabels: #"key1=value1,key2=value2" + + # -- Labels to add to the CSI NFS Deployments and DaemonSets Pods + nfsPodLabels: #"key1=value1,key2=value2" + + # -- Labels to add to the CSI RBD Deployments and DaemonSets Pods + rbdPodLabels: #"key1=value1,key2=value2" + csiAddons: + # -- Enable CSIAddons enabled: false - #image: "quay.io/csiaddons/k8s-sidecar:v0.5.0" - # Enable the nfs csi driver. + # -- CSIAddons Sidecar image + image: "quay.io/csiaddons/k8s-sidecar:v0.5.0" + nfs: + # -- Enable the nfs csi driver enabled: false - # Enable topology based provisioning. + topology: + # -- Enable topology based provisioning enabled: false - # domainLabels define which node labels to use as domains - # for CSI nodeplugins to advertise their domains # NOTE: the value here serves as an example and needs to be # updated with node labels that define domains of interest - # domainLabels: + # -- domainLabels define which node labels to use as domains + # for CSI nodeplugins to advertise their domains + domainLabels: # - kubernetes.io/hostname # - topology.kubernetes.io/zone # - topology.rook.io/rack + +# -- Enable discovery daemon enableDiscoveryDaemon: false + +# -- The timeout for ceph commands in seconds cephCommandsTimeoutSeconds: "15" -## if true, run rook operator on the host network -# useOperatorHostNetwork: true +# -- if true, run rook operator on the host network +useOperatorHostNetwork: ## Rook Discover configuration ## toleration: NoSchedule, PreferNoSchedule or NoExecute ## tolerationKey: Set this to the specific key of the taint to tolerate ## tolerations: Array of tolerations in YAML format which will be added to agent deployment ## nodeAffinity: Set to labels of the node to match -# discover: -# toleration: NoSchedule -# tolerationKey: key -# tolerations: -# - key: key -# operator: Exists -# effect: NoSchedule -# nodeAffinity: key1=value1,value2; key2=value3 -# podLabels: "key1=value1,key2=value2" -# resources: -# - limits: -# cpu: 500m -# memory: 512Mi -# - requests: -# cpu: 100m -# memory: 128Mi +discover: + # -- Toleration for the discover pods. + # Options: `NoSchedule`, `PreferNoSchedule` or `NoExecute` + toleration: + # -- The specific key of the taint to tolerate + tolerationKey: + # -- Array of tolerations in YAML format which will be added to discover deployment + tolerations: + # - key: key + # operator: Exists + # effect: NoSchedule + # -- The node labels for affinity of `discover-agent` [^1] + nodeAffinity: # key1=value1,value2; key2=value3 + # -- Labels to add to the discover pods + podLabels: # "key1=value1,key2=value2" + # -- Add resources to discover daemon pods + resources: + # - limits: + # cpu: 500m + # memory: 512Mi + # - requests: + # cpu: 100m + # memory: 128Mi + +# -- Disable adnissions controller disableAdmissionController: false -# Writing to the hostPath is required for the Ceph mon and osd pods. Given the restricted permissions in OpenShift with SELinux, -# the pod must be running privileged in order to write to the hostPath volume, this must be set to true then. +# -- Runs Ceph Pods as privileged to be able to write to `hostPaths` in OpenShift with SELinux restrictions. hostpathRequiresPrivileged: false -# Disable automatic orchestration when new devices are discovered. +# -- Disable automatic orchestration when new devices are discovered. disableDeviceHotplug: false -# Blacklist certain disks according to the regex provided. +# -- Blacklist certain disks according to the regex provided. discoverDaemonUdev: -# imagePullSecrets option allow to pull docker images from private docker registry. Option will be passed to all service accounts. -# imagePullSecrets: +# -- imagePullSecrets option allow to pull docker images from private docker registry. Option will be passed to all service accounts. +imagePullSecrets: # - name: my-registry-secret -# Whether the OBC provisioner should watch on the operator namespace or not, if not the namespace of the cluster will be used +# -- Whether the OBC provisioner should watch on the operator namespace or not, if not the namespace of the cluster will be used enableOBCWatchOperatorNamespace: true +# -- Set tolerations and nodeAffinity [^1] for admission controller pod. +# The admission controller would be best to start on the same nodes as other ceph daemons. admissionController: - # Set tolerations and nodeAffinity for admission controller pod. - # The admission controller would be best to start on the same nodes as other ceph daemons. # tolerations: # - key: key # operator: Exists # effect: NoSchedule # nodeAffinity: key1=value1,value2; key2=value3 +# [^1]: `nodeAffinity` and `*NodeAffinity` options should have the format `"role=storage,rook; storage=ceph"` or `storage=;role=rook-example` or `storage=;` (_checks only for presence of key_) + monitoring: - # requires Prometheus to be pre-installed - # enabling will also create RBAC rules to allow Operator to create ServiceMonitors + # -- Enable monitoring. Requires Prometheus to be pre-installed. + # Enabling will also create RBAC rules to allow Operator to create ServiceMonitors enabled: false diff --git a/mkdocs.yml b/mkdocs.yml index 79deac621..fe2cbdf71 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -59,6 +59,8 @@ plugins: - exclude: glob: - README.md + - "*.gotmpl" + - "*.gotmpl.md" - awesome-pages - macros: module_name: .docs/macros/includes/main @@ -77,7 +79,7 @@ plugins: css_dir: css # the directory to put the version selector's CSS javascript_dir: js # the directory to put the version selector's JS canonical_version: null # the version for ; `null` - # uses the version specified via `mike deploy` + # uses the version specified via `mike deploy` markdown_extensions: - admonition - attr_list @@ -111,7 +113,7 @@ extra: social: - icon: fontawesome/brands/slack link: https://slack.rook.io/ - - icon: fontawesome/brands/twitter + - icon: fontawesome/brands/twitter link: https://twitter.com/rook_io - icon: fontawesome/solid/envelopes-bulk link: 'https://groups.google.com/forum/#!forum/rook-dev'