diff --git a/Documentation/ceph-quickstart.md b/Documentation/ceph-quickstart.md index 9601a1748..b286fc699 100644 --- a/Documentation/ceph-quickstart.md +++ b/Documentation/ceph-quickstart.md @@ -78,13 +78,16 @@ kubectl create -f operator.yaml kubectl -n rook-ceph get pod ``` -**NOTE** - -If you are using kubernetes v1.15 or older you need to create CRDs found here `/cluster/examples/kubernetes/ceph` as apiextension v1beta1 version of CustomResourceDefinition -is deprecated in Kubernetes v1.16 and will no longer be supported from k8s v1.22. - You can also deploy the operator with the [Rook Helm Chart](helm-operator.md). +Before you start the operator in production, there are some settings that you may want to consider: +1. If you are using kubernetes v1.15 or older you need to create CRDs found here `/cluster/examples/kubernetes/ceph/pre-k8s-1.16/crd.yaml`. + The apiextension v1beta1 version of CustomResourceDefinition was deprecated in Kubernetes v1.16. +2. Consider if you want to enable certain Rook features that are disabled by default. See the [operator.yaml](https://github.com/rook/rook/blob/{{ branchName }}/cluster/examples/kubernetes/ceph/operator.yaml) for these and other advanced settings. + 1. Device discovery: Rook will watch for new devices to configure if the `ROOK_ENABLE_DISCOVERY_DAEMON` setting is enabled, commonly used in bare metal clusters. + 2. Flex driver: The flex driver is deprecated in favor of the CSI driver, but can still be enabled with the `ROOK_ENABLE_FLEX_DRIVER` setting. + 3. Node affinity and tolerations: The CSI driver by default will run on any node in the cluster. To configure the CSI driver affinity, several settings are available. + ## Create a Rook Ceph Cluster Now that the Rook operator is running we can create the Ceph cluster. For the cluster to survive reboots, @@ -99,7 +102,7 @@ kubectl create -f cluster.yaml Use `kubectl` to list pods in the `rook-ceph` namespace. You should be able to see the following pods once they are all running. The number of osd pods will depend on the number of nodes in the cluster and the number of devices configured. If you did not modify the `cluster.yaml` above, it is expected that one OSD will be created per node. -The CSI, `rook-ceph-agent`, and `rook-discover` pods are also optional depending on your settings. +The CSI, `rook-ceph-agent` (flex driver), and `rook-discover` pods are also optional depending on your settings. > If the `rook-ceph-mon`, `rook-ceph-mgr`, or `rook-ceph-osd` pods are not created, please refer to the > [Ceph common issues](ceph-common-issues.md) for more details and potential solutions. @@ -113,7 +116,6 @@ csi-cephfsplugin-rthrp 3/3 Running 0 csi-rbdplugin-hbsm7 3/3 Running 0 140s csi-rbdplugin-provisioner-5b5cd64fd-nvk6c 6/6 Running 0 140s csi-rbdplugin-provisioner-5b5cd64fd-q7bxl 6/6 Running 0 140s -rook-ceph-agent-4zkg8 1/1 Running 0 140s rook-ceph-crashcollector-minikube-5b57b7c5d4-hfldl 1/1 Running 0 105s rook-ceph-mgr-a-64cd7cdf54-j8b5p 1/1 Running 0 77s rook-ceph-mon-a-694bb7987d-fp9w7 1/1 Running 0 105s @@ -126,7 +128,6 @@ rook-ceph-osd-2-6cd4b776ff-v4d68 1/1 Running 0 rook-ceph-osd-prepare-node1-vx2rz 0/2 Completed 0 60s rook-ceph-osd-prepare-node2-ab3fd 0/2 Completed 0 60s rook-ceph-osd-prepare-node3-w4xyz 0/2 Completed 0 60s -rook-discover-dhkb8 1/1 Running 0 140s ``` To verify that the cluster is in a healthy state, connect to the [Rook toolbox](ceph-toolbox.md) and run the diff --git a/PendingReleaseNotes.md b/PendingReleaseNotes.md index c5dfcccb3..33c474f4f 100644 --- a/PendingReleaseNotes.md +++ b/PendingReleaseNotes.md @@ -13,6 +13,8 @@ v1.5... - Ceph mons require an odd number for a healthy quorum. An even number of mons is now disallowed. - Update deprecated CRD apiextensions.k8s.io/v1beta1 to v1 ([#6424](https://github.com/rook/rook/pull/6424)) - preservePoolsOnDelete is deprecated for CephFilesystem and is no longer allowed because of data-loss concerns. preserveFilesystemOnDelete acts as a replacement and preserves the filesystem when the CephFilesystem CRD is deleted (which implicitly includes all associated pools). See [#6495](https://github.com/rook/rook/pull/6495) for more details. +- The discovery daemon is disabled by default since the discovery is not necessary in most clusters. Enable the discovery daemon if + devices are being added to nodes and you want to automatically configure OSDs on new devices without restarting the operator. ## Features diff --git a/cluster/charts/rook-ceph/values.yaml b/cluster/charts/rook-ceph/values.yaml index 9125607c9..a57d4391e 100644 --- a/cluster/charts/rook-ceph/values.yaml +++ b/cluster/charts/rook-ceph/values.yaml @@ -252,7 +252,7 @@ csi: #image: k8s.gcr.io/sig-storage/csi-resizer:v1.0.0 enableFlexDriver: false -enableDiscoveryDaemon: true +enableDiscoveryDaemon: false ## if true, run rook operator on the host network # useOperatorHostNetwork: true diff --git a/cluster/examples/kubernetes/ceph/operator-openshift.yaml b/cluster/examples/kubernetes/ceph/operator-openshift.yaml index a1bc96ea0..58817e3bb 100644 --- a/cluster/examples/kubernetes/ceph/operator-openshift.yaml +++ b/cluster/examples/kubernetes/ceph/operator-openshift.yaml @@ -460,7 +460,7 @@ spec: # Whether to start the discovery daemon to watch for raw storage devices on nodes in the cluster. # This daemon does not need to run if you are only going to create your OSDs based on StorageClassDeviceSets with PVCs. - name: ROOK_ENABLE_DISCOVERY_DAEMON - value: "true" + value: "false" # Whether to start machineDisruptionBudget and machineLabel controller to watch for the osd pods and MDBs. - name: ROOK_ENABLE_MACHINE_DISRUPTION_BUDGET diff --git a/cluster/examples/kubernetes/ceph/operator.yaml b/cluster/examples/kubernetes/ceph/operator.yaml index 4d8a193b7..be0f2f172 100644 --- a/cluster/examples/kubernetes/ceph/operator.yaml +++ b/cluster/examples/kubernetes/ceph/operator.yaml @@ -411,7 +411,7 @@ spec: # Whether to start the discovery daemon to watch for raw storage devices on nodes in the cluster. # This daemon does not need to run if you are only going to create your OSDs based on StorageClassDeviceSets with PVCs. - name: ROOK_ENABLE_DISCOVERY_DAEMON - value: "true" + value: "false" # Time to wait until the node controller will move Rook pods to other # nodes after detecting an unreachable node. diff --git a/cmd/rook/ceph/operator.go b/cmd/rook/ceph/operator.go index 36c1e26a1..f3fe312f8 100644 --- a/cmd/rook/ceph/operator.go +++ b/cmd/rook/ceph/operator.go @@ -48,7 +48,7 @@ func init() { operatorCmd.Flags().DurationVar(&mon.MonOutTimeout, "mon-out-timeout", mon.MonOutTimeout, "mon out timeout (duration)") operatorCmd.Flags().BoolVar(&operator.EnableFlexDriver, "enable-flex-driver", true, "enable the rook flex driver") - operatorCmd.Flags().BoolVar(&operator.EnableDiscoveryDaemon, "enable-discovery-daemon", true, "enable the rook discovery daemon") + operatorCmd.Flags().BoolVar(&operator.EnableDiscoveryDaemon, "enable-discovery-daemon", false, "enable the rook discovery daemon") // csi deployment templates operatorCmd.Flags().StringVar(&csi.RBDPluginTemplatePath, "csi-rbd-plugin-template-path", csi.DefaultRBDPluginTemplatePath, "path to ceph-csi rbd plugin template") diff --git a/tests/framework/installer/ceph_manifests.go b/tests/framework/installer/ceph_manifests.go index b3db612d5..97beba52f 100644 --- a/tests/framework/installer/ceph_manifests.go +++ b/tests/framework/installer/ceph_manifests.go @@ -3076,6 +3076,8 @@ spec: value: "true" - name: ROOK_CURRENT_NAMESPACE_ONLY value: "false"` + openshiftEnv + ` + - name: ROOK_ENABLE_DISCOVERY_DAEMON + value: "true" volumeMounts: - mountPath: /var/lib/rook name: rook-config