From 5e86ef8b0c7e6f683f84bde0cf179d5686f2eadb Mon Sep 17 00:00:00 2001 From: Travis Nielsen Date: Wed, 2 Jun 2021 18:39:19 -0600 Subject: [PATCH] docs: clarify not to change crdsenabled in helm chart Changing the crds.enabled setting in a running cluster will cause the CRs to be deleted, which could be disastrous for a cluster. Add and clarify the warning in the docs to prevent it and link to recovery steps if it does happen. Signed-off-by: Travis Nielsen --- Documentation/ceph-disaster-recovery.md | 4 +- Documentation/helm-operator.md | 160 ++++++++++++------------ cluster/charts/rook-ceph/values.yaml | 4 +- 3 files changed, 85 insertions(+), 83 deletions(-) diff --git a/Documentation/ceph-disaster-recovery.md b/Documentation/ceph-disaster-recovery.md index e47d9fac0..0d230e986 100644 --- a/Documentation/ceph-disaster-recovery.md +++ b/Documentation/ceph-disaster-recovery.md @@ -245,8 +245,8 @@ refuse to remove the finalizer on the CRs until the underlying data is deleted. [dependency design doc](https://github.com/rook/rook/blob/master/design/ceph/resource-dependencies.md). While it is good that the CRs will not be deleted and the underlying Ceph data and daemons continue to be -available, the CRs will be stuck indefinitely in a `terminating` state in which the operator will not -continue to reconcile. Upgrades will be blocked, further updates to the CRs are prevented, and so on. +available, the CRs will be stuck indefinitely in a `Deleting` state in which the operator will not +continue to ensure cluster health. Upgrades will be blocked, further updates to the CRs are prevented, and so on. Since Kubernetes does not allow undeleting resources, the following procedure will allow you to restore the CRs to their prior state without even necessarily suffering cluster downtime. diff --git a/Documentation/helm-operator.md b/Documentation/helm-operator.md index e300cc0f6..7791704d4 100644 --- a/Documentation/helm-operator.md +++ b/Documentation/helm-operator.md @@ -77,86 +77,86 @@ After uninstalling you may want to clean up the CRDs as described on the [teardo The following tables lists the configurable parameters of the rook-operator chart and their default values. -| Parameter | Description | Default | -| ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------- | -| `image.repository` | Image | `rook/ceph` | -| `image.tag` | Image tag | `master` | -| `image.pullPolicy` | Image pull policy | `IfNotPresent` | -| `crds.enabled` | If true, the helm chart will create the Rook CRDs. | `true` | -| `rbacEnable` | If true, create & use RBAC resources | `true` | -| `pspEnable` | If true, create & use PSP resources | `true` | -| `resources` | Pod resource requests & limits | `{}` | -| `annotations` | Pod annotations | `{}` | -| `logLevel` | Global log level | `INFO` | -| `nodeSelector` | Kubernetes `nodeSelector` to add to the Deployment. | | -| `tolerations` | List of Kubernetes `tolerations` to add to the Deployment. | `[]` | -| `unreachableNodeTolerationSeconds` | Delay to use for the node.kubernetes.io/unreachable pod failure toleration to override the Kubernetes default of 5 minutes | `5s` | -| `currentNamespaceOnly` | Whether the operator should watch cluster CRD in its own namespace or not | `false` | -| `hostpathRequiresPrivileged` | Runs Ceph Pods as privileged to be able to write to `hostPath`s in OpenShift with SELinux restrictions. | `false` | -| `discover.priorityClassName` | The priority class name to add to the discover pods | | -| `discover.toleration` | Toleration for the discover pods | | -| `discover.tolerationKey` | The specific key of the taint to tolerate | | -| `discover.tolerations` | Array of tolerations in YAML format which will be added to discover deployment | | -| `discover.nodeAffinity` | The node labels for affinity of `discover-agent` (***) | | -| `discover.podLabels` | Labels to add to the discover pods. | | -| `csi.enableRbdDriver` | Enable Ceph CSI RBD driver. | `true` | -| `csi.enableCephfsDriver` | Enable Ceph CSI CephFS driver. | `true` | -| `csi.enableCephfsSnapshotter` | Enable Snapshotter in CephFS provisioner pod. | `true` | -| `csi.enableRBDSnapshotter` | Enable Snapshotter in RBD provisioner pod. | `true` | -| `csi.pluginPriorityClassName` | PriorityClassName to be set on csi driver plugin pods. | | -| `csi.provisionerPriorityClassName` | PriorityClassName to be set on csi driver provisioner pods. | | -| `csi.enableOMAPGenerator` | EnableOMAP generator deploys omap sidecar in CSI provisioner pod, to enable it set it to true | `false` | -| `csi.rbdFSGroupPolicy` | Policy for modifying a volume's ownership or permissions when the RBD PVC is being mounted | ReadWriteOnceWithFSType | -| `csi.cephFSFSGroupPolicy` | Policy for modifying a volume's ownership or permissions when the CephFS PVC is being mounted | ReadWriteOnceWithFSType | -| `csi.logLevel` | Set logging level for csi containers. Supported values from 0 to 5. 0 for general useful logs, 5 for trace level verbosity. | `0` | -| `csi.enableGrpcMetrics` | Enable Ceph CSI GRPC Metrics. | `false` | -| `csi.enableCSIHostNetwork` | Enable Host Networking for Ceph CSI nodeplugins. | `false` | -| `csi.provisionerTolerations` | Array of tolerations in YAML format which will be added to CSI provisioner deployment. | | -| `csi.provisionerNodeAffinity` | The node labels for affinity of the CSI provisioner deployment (***) | | -| `csi.pluginTolerations` | Array of tolerations in YAML format which will be added to CephCSI plugin DaemonSet | | -| `csi.pluginNodeAffinity` | The node labels for affinity of the CephCSI plugin DaemonSet (***) | | -| `csi.rbdProvisionerTolerations` | Array of tolerations in YAML format which will be added to CephCSI RBD provisioner deployment. | | -| `csi.rbdProvisionerNodeAffinity` | The node labels for affinity of the CephCSI RBD provisioner deployment (***) | | -| `csi.rbdPluginTolerations` | Array of tolerations in YAML format which will be added to CephCSI RBD plugin DaemonSet | | -| `csi.rbdPluginNodeAffinity` | The node labels for affinity of the CephCSI RBD plugin DaemonSet (***) | | -| `csi.cephFSProvisionerTolerations` | Array of tolerations in YAML format which will be added to CephCSI CephFS provisioner deployment. | | -| `csi.cephFSProvisionerNodeAffinity`| The node labels for affinity of the CephCSI CephFS provisioner deployment (***) | | -| `csi.cephFSPluginTolerations` | Array of tolerations in YAML format which will be added to CephCSI CephFS plugin DaemonSet | | -| `csi.cephFSPluginNodeAffinity` | The node labels for affinity of the CephCSI CephFS plugin DaemonSet (***) | | -| `csi.csiRBDProvisionerResource` | CEPH CSI RBD provisioner resource requirement list. | | -| `csi.csiRBDPluginResource` | CEPH CSI RBD plugin resource requirement list. | | -| `csi.csiCephFSProvisionerResource` | CEPH CSI CephFS provisioner resource requirement list. | | -| `csi.csiCephFSPluginResource` | CEPH CSI CephFS plugin resource requirement list. | | -| `csi.cephfsGrpcMetricsPort` | CSI CephFS driver GRPC metrics port. | `9091` | -| `csi.cephfsLivenessMetricsPort` | CSI CephFS driver metrics port. | `9081` | -| `csi.rbdGrpcMetricsPort` | Ceph CSI RBD driver GRPC metrics port. | `9090` | -| `csi.rbdLivenessMetricsPort` | Ceph CSI RBD driver metrics port. | `8080` | -| `csi.forceCephFSKernelClient` | Enable Ceph Kernel clients on kernel < 4.17 which support quotas for Cephfs. | `true` | -| `csi.kubeletDirPath` | Kubelet root directory path (if the Kubelet uses a different path for the `--root-dir` flag) | `/var/lib/kubelet` | -| `csi.cephcsi.image` | Ceph CSI image. | `quay.io/cephcsi/cephcsi:v3.3.1` | -| `csi.rbdPluginUpdateStrategy` | CSI Rbd plugin daemonset update strategy, supported values are OnDelete and RollingUpdate. | `OnDelete` | -| `csi.cephFSPluginUpdateStrategy` | CSI CephFS plugin daemonset update strategy, supported values are OnDelete and RollingUpdate. | `OnDelete` | -| `csi.registrar.image` | Kubernetes CSI registrar image. | `k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.0.1` | -| `csi.resizer.image` | Kubernetes CSI resizer image. | `k8s.gcr.io/sig-storage/csi-resizer:v1.0.0` | -| `csi.provisioner.image` | Kubernetes CSI provisioner image. | `k8s.gcr.io/sig-storage/csi-provisioner:v2.0.0` | -| `csi.snapshotter.image` | Kubernetes CSI snapshotter image. | `k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0` | -| `csi.attacher.image` | Kubernetes CSI Attacher image. | `k8s.gcr.io/sig-storage/csi-attacher:v3.0.0` | -| `csi.cephfsPodLabels` | Labels to add to the CSI CephFS Pods. | | -| `csi.rbdPodLabels` | Labels to add to the CSI RBD Pods. | | -| `csi.volumeReplication.enabled` | Enable Volume Replication. | `false` | -| `csi.volumeReplication.image` | Volume Replication Controller image. | `quay.io/csiaddons/volumereplication-operator:v0.1.0` | -| `agent.flexVolumeDirPath` | Path where the Rook agent discovers the flex volume plugins (*) | `/usr/libexec/kubernetes/kubelet-plugins/volume/exec/` | -| `agent.libModulesDirPath` | Path where the Rook agent should look for kernel modules (*) | `/lib/modules` | -| `agent.mounts` | Additional paths to be mounted in the agent container (**) | | -| `agent.mountSecurityMode` | Mount Security Mode for the agent. | `Any` | -| `agent.priorityClassName` | The priority class name to add to the agent pods | | -| `agent.toleration` | Toleration for the agent pods | | -| `agent.tolerationKey` | The specific key of the taint to tolerate | | -| `agent.tolerations` | Array of tolerations in YAML format which will be added to agent deployment | | -| `agent.nodeAffinity` | The node labels for affinity of `rook-agent` (***) | | -| `admissionController.tolerations` | Array of tolerations in YAML format which will be added to admission controller deployment. | | -| `admissionController.nodeAffinity` | The node labels for affinity of the admission controller deployment (***) | | -| `allowMultipleFilesystems` | **(experimental)** Allows multiple filesystems to be deployed to a Ceph cluster. Octopus (v15) or Nautilus (v14) | `false` | +| Parameter | Description | Default | +| ----------------------------------- | --------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------- | +| `image.repository` | Image | `rook/ceph` | +| `image.tag` | Image tag | `master` | +| `image.pullPolicy` | Image pull policy | `IfNotPresent` | +| `crds.enabled` | If true, the helm chart will create the Rook CRDs. Do NOT change to `false` in a running cluster or CRs will be deleted! | `true` | +| `rbacEnable` | If true, create & use RBAC resources | `true` | +| `pspEnable` | If true, create & use PSP resources | `true` | +| `resources` | Pod resource requests & limits | `{}` | +| `annotations` | Pod annotations | `{}` | +| `logLevel` | Global log level | `INFO` | +| `nodeSelector` | Kubernetes `nodeSelector` to add to the Deployment. | | +| `tolerations` | List of Kubernetes `tolerations` to add to the Deployment. | `[]` | +| `unreachableNodeTolerationSeconds` | Delay to use for the node.kubernetes.io/unreachable pod failure toleration to override the Kubernetes default of 5 minutes | `5s` | +| `currentNamespaceOnly` | Whether the operator should watch cluster CRD in its own namespace or not | `false` | +| `hostpathRequiresPrivileged` | Runs Ceph Pods as privileged to be able to write to `hostPath`s in OpenShift with SELinux restrictions. | `false` | +| `discover.priorityClassName` | The priority class name to add to the discover pods | | +| `discover.toleration` | Toleration for the discover pods | | +| `discover.tolerationKey` | The specific key of the taint to tolerate | | +| `discover.tolerations` | Array of tolerations in YAML format which will be added to discover deployment | | +| `discover.nodeAffinity` | The node labels for affinity of `discover-agent` (***) | | +| `discover.podLabels` | Labels to add to the discover pods. | | +| `csi.enableRbdDriver` | Enable Ceph CSI RBD driver. | `true` | +| `csi.enableCephfsDriver` | Enable Ceph CSI CephFS driver. | `true` | +| `csi.enableCephfsSnapshotter` | Enable Snapshotter in CephFS provisioner pod. | `true` | +| `csi.enableRBDSnapshotter` | Enable Snapshotter in RBD provisioner pod. | `true` | +| `csi.pluginPriorityClassName` | PriorityClassName to be set on csi driver plugin pods. | | +| `csi.provisionerPriorityClassName` | PriorityClassName to be set on csi driver provisioner pods. | | +| `csi.enableOMAPGenerator` | EnableOMAP generator deploys omap sidecar in CSI provisioner pod, to enable it set it to true | `false` | +| `csi.rbdFSGroupPolicy` | Policy for modifying a volume's ownership or permissions when the RBD PVC is being mounted | ReadWriteOnceWithFSType | +| `csi.cephFSFSGroupPolicy` | Policy for modifying a volume's ownership or permissions when the CephFS PVC is being mounted | ReadWriteOnceWithFSType | +| `csi.logLevel` | Set logging level for csi containers. Supported values from 0 to 5. 0 for general useful logs, 5 for trace level verbosity. | `0` | +| `csi.enableGrpcMetrics` | Enable Ceph CSI GRPC Metrics. | `false` | +| `csi.enableCSIHostNetwork` | Enable Host Networking for Ceph CSI nodeplugins. | `false` | +| `csi.provisionerTolerations` | Array of tolerations in YAML format which will be added to CSI provisioner deployment. | | +| `csi.provisionerNodeAffinity` | The node labels for affinity of the CSI provisioner deployment (***) | | +| `csi.pluginTolerations` | Array of tolerations in YAML format which will be added to CephCSI plugin DaemonSet | | +| `csi.pluginNodeAffinity` | The node labels for affinity of the CephCSI plugin DaemonSet (***) | | +| `csi.rbdProvisionerTolerations` | Array of tolerations in YAML format which will be added to CephCSI RBD provisioner deployment. | | +| `csi.rbdProvisionerNodeAffinity` | The node labels for affinity of the CephCSI RBD provisioner deployment (***) | | +| `csi.rbdPluginTolerations` | Array of tolerations in YAML format which will be added to CephCSI RBD plugin DaemonSet | | +| `csi.rbdPluginNodeAffinity` | The node labels for affinity of the CephCSI RBD plugin DaemonSet (***) | | +| `csi.cephFSProvisionerTolerations` | Array of tolerations in YAML format which will be added to CephCSI CephFS provisioner deployment. | | +| `csi.cephFSProvisionerNodeAffinity` | The node labels for affinity of the CephCSI CephFS provisioner deployment (***) | | +| `csi.cephFSPluginTolerations` | Array of tolerations in YAML format which will be added to CephCSI CephFS plugin DaemonSet | | +| `csi.cephFSPluginNodeAffinity` | The node labels for affinity of the CephCSI CephFS plugin DaemonSet (***) | | +| `csi.csiRBDProvisionerResource` | CEPH CSI RBD provisioner resource requirement list. | | +| `csi.csiRBDPluginResource` | CEPH CSI RBD plugin resource requirement list. | | +| `csi.csiCephFSProvisionerResource` | CEPH CSI CephFS provisioner resource requirement list. | | +| `csi.csiCephFSPluginResource` | CEPH CSI CephFS plugin resource requirement list. | | +| `csi.cephfsGrpcMetricsPort` | CSI CephFS driver GRPC metrics port. | `9091` | +| `csi.cephfsLivenessMetricsPort` | CSI CephFS driver metrics port. | `9081` | +| `csi.rbdGrpcMetricsPort` | Ceph CSI RBD driver GRPC metrics port. | `9090` | +| `csi.rbdLivenessMetricsPort` | Ceph CSI RBD driver metrics port. | `8080` | +| `csi.forceCephFSKernelClient` | Enable Ceph Kernel clients on kernel < 4.17 which support quotas for Cephfs. | `true` | +| `csi.kubeletDirPath` | Kubelet root directory path (if the Kubelet uses a different path for the `--root-dir` flag) | `/var/lib/kubelet` | +| `csi.cephcsi.image` | Ceph CSI image. | `quay.io/cephcsi/cephcsi:v3.3.1` | +| `csi.rbdPluginUpdateStrategy` | CSI Rbd plugin daemonset update strategy, supported values are OnDelete and RollingUpdate. | `OnDelete` | +| `csi.cephFSPluginUpdateStrategy` | CSI CephFS plugin daemonset update strategy, supported values are OnDelete and RollingUpdate. | `OnDelete` | +| `csi.registrar.image` | Kubernetes CSI registrar image. | `k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.0.1` | +| `csi.resizer.image` | Kubernetes CSI resizer image. | `k8s.gcr.io/sig-storage/csi-resizer:v1.0.0` | +| `csi.provisioner.image` | Kubernetes CSI provisioner image. | `k8s.gcr.io/sig-storage/csi-provisioner:v2.0.0` | +| `csi.snapshotter.image` | Kubernetes CSI snapshotter image. | `k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0` | +| `csi.attacher.image` | Kubernetes CSI Attacher image. | `k8s.gcr.io/sig-storage/csi-attacher:v3.0.0` | +| `csi.cephfsPodLabels` | Labels to add to the CSI CephFS Pods. | | +| `csi.rbdPodLabels` | Labels to add to the CSI RBD Pods. | | +| `csi.volumeReplication.enabled` | Enable Volume Replication. | `false` | +| `csi.volumeReplication.image` | Volume Replication Controller image. | `quay.io/csiaddons/volumereplication-operator:v0.1.0` | +| `agent.flexVolumeDirPath` | Path where the Rook agent discovers the flex volume plugins (*) | `/usr/libexec/kubernetes/kubelet-plugins/volume/exec/` | +| `agent.libModulesDirPath` | Path where the Rook agent should look for kernel modules (*) | `/lib/modules` | +| `agent.mounts` | Additional paths to be mounted in the agent container (**) | | +| `agent.mountSecurityMode` | Mount Security Mode for the agent. | `Any` | +| `agent.priorityClassName` | The priority class name to add to the agent pods | | +| `agent.toleration` | Toleration for the agent pods | | +| `agent.tolerationKey` | The specific key of the taint to tolerate | | +| `agent.tolerations` | Array of tolerations in YAML format which will be added to agent deployment | | +| `agent.nodeAffinity` | The node labels for affinity of `rook-agent` (***) | | +| `admissionController.tolerations` | Array of tolerations in YAML format which will be added to admission controller deployment. | | +| `admissionController.nodeAffinity` | The node labels for affinity of the admission controller deployment (***) | | +| `allowMultipleFilesystems` | **(experimental)** Allows multiple filesystems to be deployed to a Ceph cluster. Octopus (v15) or Nautilus (v14) | `false` | * For information on what to set `agent.flexVolumeDirPath` to, please refer to the [Rook flexvolume documentation](flexvolume.md) diff --git a/cluster/charts/rook-ceph/values.yaml b/cluster/charts/rook-ceph/values.yaml index a9f79d524..a435fb3a3 100644 --- a/cluster/charts/rook-ceph/values.yaml +++ b/cluster/charts/rook-ceph/values.yaml @@ -11,7 +11,9 @@ image: crds: # Whether the helm chart should create and update the CRDs. If false, the CRDs must be # managed independently with cluster/examples/kubernetes/ceph/crds.yaml. - # **WARNING** Only set during first deployment. If later disabled the cluster will be destroyed. + # **WARNING** Only set during first deployment. If later disabled the cluster may be DESTROYED. + # If the CRDs are deleted in this case, see the disaster recovery guide to restore them. + # https://rook.github.io/docs/rook/master/ceph-disaster-recovery.html#restoring-crds-after-deletion enabled: true resources: