From 9d41b380959bf2a6b35bf0a78a5da20dd1a5c338 Mon Sep 17 00:00:00 2001 From: Blaine Gardner Date: Tue, 31 Mar 2020 15:52:04 -0600 Subject: [PATCH 1/2] Ceph: upgrade docs: update upgrade docs for v1.3 Update both documents and `upgrade-from-v1.2*.yaml` files for Rook v1.3 release. Of note: - note when the upgrade to CSI driver v2.0 should be updated - note that CSI image envs can be removed to have Rook use defaults - update Ceph version upgrade section - update supported Ceph version info - update CRDs before upgrade b/c new 'status' fields - remove special CSI update info left from v1.2 Signed-off-by: Blaine Gardner (cherry picked from commit 2ccff83d942bfdc1842ed05e4cc1e59a841933b8) --- Documentation/ceph-upgrade.md | 220 ++++++------------ .../examples/kubernetes/ceph/operator.yaml | 26 ++- .../ceph/upgrade-from-v1.2-apply.yaml | 16 +- .../ceph/upgrade-from-v1.2-crds.yaml | 4 +- 4 files changed, 100 insertions(+), 166 deletions(-) diff --git a/Documentation/ceph-upgrade.md b/Documentation/ceph-upgrade.md index b5272dc25..0f136e93e 100644 --- a/Documentation/ceph-upgrade.md +++ b/Documentation/ceph-upgrade.md @@ -26,6 +26,7 @@ supported. For a guide to upgrade previous versions of Rook, please refer to the version of documentation for those releases. +* [Upgrade 1.1 to 1.2](https://rook.io/docs/rook/v1.2/ceph-upgrade.html) * [Upgrade 1.0 to 1.1](https://rook.io/docs/rook/v1.1/ceph-upgrade.html) * [Upgrade 0.9 to 1.0](https://rook.io/docs/rook/v1.0/ceph-upgrade.html) * [Upgrade 0.8 to 0.9](https://rook.io/docs/rook/v0.9/ceph-upgrade.html) @@ -56,42 +57,7 @@ released, the process of updating from v1.2.0 is as simple as running the follow kubectl -n rook-ceph set image deploy/rook-ceph-operator rook-ceph-operator=rook/ceph:v1.2.1 ``` -If you want to enable the CSI v2.0 driver, additional steps are needed in the -latest Rook patch release. There are some new features in 2.0 such as -supporting resizing volumes and support for a new csi attacher. If you don't -need these features now, you can skip this step for the patch upgrade. - -## Enabling the CSI v2.0 driver - -> If you are deploying the rook-operator with the helm chart you only need to specify the related images in your values.yaml. The RBAC settings will be created for you by the chart. - -To enable the CSI v2.0 driver, you will need to apply updated RBAC settings: - -```sh -kubectl apply -f upgrade-from-v1.2-apply.yaml -``` - -The following `env` variables will need to be configured in the Rook-Ceph -operator deployment. The suggested upstream images are included below, which -you should change to match where your images are located. - -```yaml - env: - - name: ROOK_CSI_CEPH_IMAGE - value: "quay.io/cephcsi/cephcsi:v2.0.0" - - name: ROOK_CSI_REGISTRAR_IMAGE - value: "quay.io/k8scsi/csi-node-driver-registrar:v1.2.0" - - name: ROOK_CSI_PROVISIONER_IMAGE - value: "quay.io/k8scsi/csi-provisioner:v1.4.0" - - name: ROOK_CSI_SNAPSHOTTER_IMAGE - value: "quay.io/k8scsi/csi-snapshotter:v1.2.2" - - name: ROOK_CSI_ATTACHER_IMAGE - value: "quay.io/k8scsi/csi-attacher:v2.1.0" - - name: ROOK_CSI_RESIZER_IMAGE - value: "quay.io/k8scsi/csi-resizer:v0.4.0" -``` - -## Upgrading from v1.1 to v1.2 +## Upgrading from v1.2 to v1.3 **Rook releases from master are expressly unsupported.** It is strongly recommended that you use [official releases](https://github.com/rook/rook/releases) of Rook. Unreleased versions from the @@ -114,9 +80,9 @@ created before v0.8 are likely to be: * Clusters created in v0.8 or v0.9: `rook-ceph-system` and `rook-ceph` * Clusters created in v1.0 or newer: only `rook-ceph` -With this guide, we do our best not to assume the namespaces in your cluster. -To make things as easy as possible, modify and use the below snippet to configure your environment. -We will use these environment variables throughout this document. +With this guide, we do our best not to assume the namespaces in your cluster. To make things as easy +as possible, modify and use the below snippet to configure your environment. We will use these +environment variables throughout this document. ```sh # Parameterize the environment @@ -124,11 +90,17 @@ export ROOK_SYSTEM_NAMESPACE="rook-ceph" export ROOK_NAMESPACE="rook-ceph" ``` +You must also perform a sed-replace on one of the upgrade manifests to set the namespace for your +unique cluster if you aren't using the `rook-ceph` namespace above. +```sh +sed -i.bak "s/namespace: rook-ceph/namespace: $ROOK_NAMESPACE/g" upgrade-from-v1.2-apply.yaml +``` + In order to successfully upgrade a Rook cluster, the following prerequisites must be met: -* The cluster should be in a healthy state with full functionality. - Review the [health verification section](#health-verification) in order to verify your cluster is - in a good starting state. +* The cluster should be in a healthy state with full functionality. Review the + [health verification section](#health-verification) in order to verify your cluster is in a good + starting state. * All pods consuming Rook storage should be created, running, and in a steady state. No Rook persistent volumes should be in the act of being created or deleted. @@ -246,9 +218,9 @@ Any pod that is using a Rook volume should also remain healthy: ## Rook Operator Upgrade Process -In the examples given in this guide, we will be upgrading a live Rook cluster running `v1.1.7` to -the version `v1.2.0`. This upgrade should work from any official patch release of Rook v1.1 to any -official patch release of v1.2. We will further assume that your previous cluster was created using +In the examples given in this guide, we will be upgrading a live Rook cluster running `v1.2.7` to +the version `v1.3.0`. This upgrade should work from any official patch release of Rook v1.2 to any +official patch release of v1.3. We will further assume that your previous cluster was created using an earlier version of this guide and manifests. If you have created custom manifests, these steps may not work as written. @@ -260,70 +232,44 @@ time without compatibility support and without prior notice. Let's get started! -## 1. Update the RBAC and CRDs +## 1. Update Ceph to Nautilus version 14.2.5 or higher -First update the Ceph Custom Resource Definitions and the privileges (RBAC) needed by the operator. -A new `CephClient` CRD is included in v1.2 and the CSI driver privileges changed slightly. +Rook v1.3 supports Ceph Nautilus v14.2.5 or newer and Ceph Octopus v15.2.0 or newer. These are the +only supported major versions of Ceph. Rook v1.2 does not support Ceph Octopus. Therefore, if your +Rook v1.2 cluster must be updated to Ceph Nautilus v14.2.5 or higher before upgrading the Rook +Operator to v1.3. If an upgrade is necessary, follow the Ceph upgrade steps from Rook's v1.2 release +[here](https://rook.io/docs/rook/v1.2/ceph-upgrade.html#ceph-version-upgrades). + + +## 2. Update the RBAC and CRDs + +First apply new resources. This includes slightly modified privileges (RBAC) needed by the Operator. +Also update Ceph Custom Resource Definitions (CRDs) at this time. Many CRDs have had a `status` item +added to them which must be present for Rook v1.3 to update CRD statuses. ```sh -kubectl apply -f upgrade-from-v1.1-apply.yaml +kubectl apply -f upgrade-from-v1.2-apply.yaml -f upgrade-from-v1.2-crds.yaml ``` -## 2. CSI upgrade pre-requisites +## 3. Update Ceph CSI version to v2.0 -In some scenarios there is an issue in the CSI driver that will cause application pods to be -disconnected from their mounts when the CSI driver is restarted. Since the upgrade would cause the CSI -driver to restart if it is updated, you need to be aware of whether this affects your applications. -This issue will happen when using the Ceph `fuse` client or `rbd-nbd`: -- CephFS: If you are provision volumes for CephFS and have a kernel less than version 4.17, -The CSI driver will fall back to use the FUSE client. -- RBD: If you have set the `mounter: rbd-nbd` option in the -[RBD storage class](https://github.com/rook/rook/blob/release-1.2/cluster/examples/kubernetes/ceph/csi/rbd/storageclass.yaml#L40), -the NBD mounter will have this issue. This setting is **not** enabled by default. +If you have specified custom CSI images in the Rook-Ceph Operator deployment, you should update your +the deployment to use the latest Ceph-CSI v2.0 and related drivers. If you have not specified custom +CSI images in the Operator deployment this step is unnecessary; Rook v1.3 will use the latest +drivers by default. -If you are affected by this issue, you will need to proceed carefully during the upgrade to restart -your application pods. The first recommended step is to modify the update strategy of the -CSI driver so that you can control when the CSI driver pods are restarted on each node. -As seen in the example below, set `CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY` and `CSI_RBD_PLUGIN_UPDATE_STRATEGY` -values to `OnDelete`. +See the section [CSI Updates](#csi-updates) for details about how to do this. -To avoid this issue in future upgrades, we recommend that you **do not** use the `fuse` client or `rbd-nbd`. -The `fuse` client can be avoided by setting the following environment variables in the operator now before this upgrade. -The side effect of this setting is that the PVC size will not be enforced if the CephFS quotas are not supported in your kernel version. -To enable this option and avoid this upgrade issue in the future, set `CSI_FORCE_CEPHFS_KERNEL_CLIENT` to `true`. - -```console -kubectl -n $ROOK_SYSTEM_NAMESPACE edit deploy rook-ceph-operator -``` -```yaml - # If you set this image at the same time as the env variables, you can skip step 3 of this guide and avoid a second operator restart - image: rook/ceph:v1.2.0 - env: - # Change the update strategy for the CephFS driver if your cluster is affected - - name: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY - value: "OnDelete" - # Change the update strategy for the RBD driver if your cluster is affected - - name: CSI_RBD_PLUGIN_UPDATE_STRATEGY - value: "OnDelete" - # To avoid this upgrade issue in the future for CephFS, force enable the kernel client - - name: CSI_FORCE_CEPHFS_KERNEL_CLIENT - value: "true" -``` - -After the operator and cluster are updated, we will continue with the CSI and application -pod restarts in Step 5. - -## 3. Update the Rook Operator +## 4. Update the Rook Operator The largest portion of the upgrade is triggered when the operator's image is updated to `v1.2.x`. When the operator is updated, it will proceed to update all of the Ceph daemons. -(If step 1 was completed, this change has already been applied.) ```sh kubectl -n $ROOK_SYSTEM_NAMESPACE set image deploy/rook-ceph-operator rook-ceph-operator=rook/ceph:v1.2.0 ``` -## 4. Wait for the upgrade to complete +## 5. Wait for the upgrade to complete Watch now in amazement as the Ceph mons, mgrs, OSDs, rbd-mirrors, MDSes and RGWs are terminated and replaced with updated versions in sequence. The cluster may be offline very briefly as mons update, @@ -353,7 +299,7 @@ rook-ceph-osd-1 req/upd/avl: 1/1/1 rook-version=v1.1.7 rook-ceph-osd-2 req/upd/avl: 1/1/1 rook-version=v1.1.7 ``` -The MDSes and RGWs are the last daemons to update. An easy check to see if the upgrade is totally +The MDS, NFS, and RGW daemons are the last to update. An easy check to see if the upgrade is totally finished is to check that there is only one `rook-version` reported across the cluster. It is safe to proceed with the next step before the MDSes and RGWs are finished updating. @@ -366,46 +312,17 @@ This cluster is finished: rook-version=v1.2.0 ``` -## 5. Verify the updated cluster +## 6. Verify the updated cluster At this point, your Rook operator should be running version `rook/ceph:v1.2.0`. Verify the Ceph cluster's health using the [health verification section](#health-verification). -## 6. CSI Manual Update (optional) - -If you determined in step 1 that you were affected by the CSI driver restart issue that disconnects -the application pods from their mounts, continue with this section. Otherwise, you can skip to step 7. - -Your cluster should now be in a state where Rook has upgraded everything except the CSI driver. -The CSI driver pods will not be updated until you delete them manually. This allows you to control -when your application pods will be affected by the CSI driver restart. - -For each node: -- Drain your application pods from the node -- Delete the CSI driver pods on the node - - The pods to delete will be named with a `csi-cephfsplugin` or `csi-rbdplugin` prefix and have a random suffix on each node. - However, no need to delete the provisioner pods: `csi-cephfsplugin-provisioner-*` or `csi-rbdplugin-provisioner-*` - - The pod deletion causes the pods to be restarted and updated automatically on the node - - -## 7. Update Rook-Ceph custom resource definitions - -> **IMPORTANT**: Do not perform this step until ALL existing Rook-Ceph clusters are updated! - -After all Rook-Ceph clusters have been updated following the steps above, update the Rook-Ceph -Custom Resource Definitions. This will help with creating or modifying Rook-Ceph -deployments in the future with the updated schema validation. - -```sh -kubectl apply -f upgrade-from-v1.2-crds.yaml -``` - ## Ceph Version Upgrades -Rook v1.2 supports Ceph Mimic v13.2.4 or newer and Ceph Nautilus v14.2.0 or newer. These are the only -supported major versions of Ceph. +Rook v1.3 supports Ceph Nautilus 14.2.5 or newer and Ceph Octopus v15.2.0 or newer. These are the +only supported major versions of Ceph. > **IMPORTANT: When an update is requested, the operator will check Ceph's status, if it is in `HEALTH_ERR` it will refuse to do the upgrade.** @@ -422,15 +339,15 @@ until all the daemons have been updated. Official Ceph container images can be found on [Docker Hub](https://hub.docker.com/r/ceph/ceph/tags/). These images are tagged in a few ways: -* The most explicit form of tags are full-ceph-version-and-build tags (e.g., `v14.2.5-20191210`). +* The most explicit form of tags are full-ceph-version-and-build tags (e.g., `v15.2.0-20200324`). These tags are recommended for production clusters, as there is no possibility for the cluster to be heterogeneous with respect to the version of Ceph running in containers. -* Ceph major version tags (e.g., `v14`) are useful for development and test clusters so that the +* Ceph major version tags (e.g., `v15`) are useful for development and test clusters so that the latest version of Ceph is always available. **Ceph containers other than the official images from the registry above will not be supported.** -### Example upgrade to Ceph Nautilus +### Example upgrade to Ceph Octopus #### 1. Update the main Ceph daemons @@ -438,7 +355,7 @@ The majority of the upgrade will be handled by the Rook operator. Begin the upgr Ceph image field in the cluster CRD (`spec:cephVersion:image`). ```sh -NEW_CEPH_IMAGE='ceph/ceph:v14.2.5-20191210' +NEW_CEPH_IMAGE='ceph/ceph:v15.2.0-20200324' CLUSTER_NAME="$ROOK_NAMESPACE" # change if your cluster name is not the Rook namespace kubectl -n $ROOK_NAMESPACE patch CephCluster $CLUSTER_NAME --type=merge -p "{\"spec\": {\"cephVersion\": {\"image\": \"$NEW_CEPH_IMAGE\"}}}" ``` @@ -457,14 +374,12 @@ Determining when the Ceph has fully updated is rather simple. ```console # kubectl -n $ROOK_NAMESPACE get deployment -l rook_cluster=$ROOK_NAMESPACE -o jsonpath='{range .items[*]}{"ceph-version="}{.metadata.labels.ceph-version}{"\n"}{end}' | sort | uniq This cluster is not yet finished: - ceph-version=13.2.6 - ceph-version=14.2.5-0 + ceph-version=14.2.7-0 + ceph-version=15.2.0-0 This cluster is finished: - ceph-version=14.2.5-0 + ceph-version=15.2.0-0 ``` -Note that Ceph version now includes an additional build tag, the `-0` suffix in the example above. - #### 3. Verify the updated cluster Verify the Ceph cluster's health using the [health verification section](#health-verification). @@ -492,15 +407,15 @@ kubectl -n $ROOK_NAMESPACE exec -it $TOOLS_POD -- ceph config set global mon_war ## CSI Updates -If you have a v1.1 cluster running with CSI drivers enabled and you have configured the Rook-Ceph -operator to use custom CSI images, the environment (`env`) variables need to be updated -periodically. If this is the case, it is easiest to `kubectl edit` the operator deployment and +If you have a cluster running with CSI drivers enabled and you have configured the Rook-Ceph +Operator to use custom CSI images, the environment (`env`) variables need to be updated +periodically. If this is the case, it is easiest to `kubectl edit` the Operator Deployment and modify everything needed at once. You can switch between using Rook-Ceph's default CSI images and custom CSI images as you wish. ### Use default images If you would like Rook to use the inbuilt default upstream images, then you may simply remove all -`env` variables with the `ROOK_CSI_` prefix from the Rook-Ceph operator deployment. +`env` variables matching `ROOK_CSI_*_IMAGE` from the Rook-Ceph operator deployment. ### Use custom images OR, if you would like to use images hosted in a different location like a local image registry, then @@ -510,16 +425,18 @@ located. ```yaml env: - - name: ROOK_CSI_CEPH_IMAGE - value: "quay.io/cephcsi/cephcsi:v1.2.2" - - name: ROOK_CSI_REGISTRAR_IMAGE - value: "quay.io/k8scsi/csi-node-driver-registrar:v1.2.0" - - name: ROOK_CSI_PROVISIONER_IMAGE - value: "quay.io/k8scsi/csi-provisioner:v1.4.0" - - name: ROOK_CSI_SNAPSHOTTER_IMAGE - value: "quay.io/k8scsi/csi-snapshotter:v1.2.2" - - name: ROOK_CSI_ATTACHER_IMAGE - value: "quay.io/k8scsi/csi-attacher:v1.2.0" + - name: ROOK_CSI_CEPH_IMAGE + value: "quay.io/cephcsi/cephcsi:v2.0.0" + - name: ROOK_CSI_REGISTRAR_IMAGE + value: "quay.io/k8scsi/csi-node-driver-registrar:v1.2.0" + - name: ROOK_CSI_PROVISIONER_IMAGE + value: "quay.io/k8scsi/csi-provisioner:v1.4.0" + - name: ROOK_CSI_SNAPSHOTTER_IMAGE + value: "quay.io/k8scsi/csi-snapshotter:v1.2.2" + - name: ROOK_CSI_ATTACHER_IMAGE + value: "quay.io/k8scsi/csi-attacher:v2.1.0" + - name: ROOK_CSI_RESIZER_IMAGE + value: "quay.io/k8scsi/csi-resizer:v0.4.0" ``` ### Verifying updates @@ -529,9 +446,10 @@ are updated. ```console # kubectl --namespace rook-ceph get pod -o jsonpath='{range .items[*]}{range .spec.containers[*]}{.image}{"\n"}' -l 'app in (csi-rbdplugin,csi-rbdplugin-provisioner,csi-cephfsplugin,csi-cephfsplugin-provisioner)' | sort | uniq -quay.io/cephcsi/cephcsi:v1.2.2 -quay.io/k8scsi/csi-attacher:v1.2.0 +quay.io/cephcsi/cephcsi:v2.0.0 +quay.io/k8scsi/csi-attacher:v2.1.0 quay.io/k8scsi/csi-node-driver-registrar:v1.2.0 quay.io/k8scsi/csi-provisioner:v1.4.0 +quay.io/k8scsi/csi-resizer:v0.4.0 quay.io/k8scsi/csi-snapshotter:v1.2.2 ``` diff --git a/cluster/examples/kubernetes/ceph/operator.yaml b/cluster/examples/kubernetes/ceph/operator.yaml index 669d92c25..3e9b3398e 100644 --- a/cluster/examples/kubernetes/ceph/operator.yaml +++ b/cluster/examples/kubernetes/ceph/operator.yaml @@ -8,12 +8,14 @@ # # Also see other operator sample files for variations of operator.yaml: # - operator-openshift.yaml: Common settings for running in OpenShift -################################################################################################################# -# Rook Ceph Operator Config -# Use this ConfigMap to override operator configurations -# Precedence will be given to this config in case -# Env Var also exists for the same -# +############################################################################################################### + +# Rook Ceph Operator Config ConfigMap +# Use this ConfigMap to override Rook-Ceph Operator configurations. +# NOTE! Precedence will be given to this config if the same Env Var config also exists in the +# Operator Deployment. +# To move a configuration(s) from the Operator Deployment to this ConfigMap, add the config +# here. It is recommended to then remove it from the Deployment to eliminate any future confusion. kind: ConfigMap apiVersion: v1 metadata: @@ -33,12 +35,12 @@ data: # Set logging level for csi containers. # Supported values from 0 to 5. 0 for general useful logs, 5 for trace level verbosity. # CSI_LOG_LEVEL: "0" - + # Enable Ceph Kernel clients on kernel < 4.17 which support quotas for Cephfs # If you disable the kernel client, your application may be disrupted during upgrade. # See the upgrade guide: https://rook.io/docs/rook/master/ceph-upgrade.html CSI_FORCE_CEPHFS_KERNEL_CLIENT: "true" - + # (Optional) Allow starting unsupported ceph-csi image ROOK_CSI_ALLOW_UNSUPPORTED_VERSION: "false" # The default version of CSI supported by Rook will be started. To change the version @@ -50,17 +52,17 @@ data: # ROOK_CSI_PROVISIONER_IMAGE: "quay.io/k8scsi/csi-provisioner:v1.4.0" # ROOK_CSI_SNAPSHOTTER_IMAGE: "quay.io/k8scsi/csi-snapshotter:v1.2.2" # ROOK_CSI_ATTACHER_IMAGE: "quay.io/k8scsi/csi-attacher:v2.1.0" - + # CSI CephFS plugin daemonset update strategy, supported values are OnDelete and RollingUpdate. # Default value is RollingUpdate. # CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY: "OnDelete" # CSI RBD plugin daemonset update strategy, supported values are OnDelete and RollingUpdate. # Default value is RollingUpdate. # CSI_RBD_PLUGIN_UPDATE_STRATEGY: "OnDelete" - + # kubelet directory path, if kubelet configured to use other than /var/lib/kubelet path. # ROOK_CSI_KUBELET_DIR_PATH: "/var/lib/kubelet" - + # (Optional) Ceph Provisioner NodeAffinity. # CSI_PROVISIONER_NODE_AFFINITY: "role=storage-node; storage=rook, ceph" # (Optional) CEPH CSI provisioner tolerations list. Put here list of taints you want to tolerate in YAML format. @@ -83,7 +85,7 @@ data: # - effect: NoExecute # key: node-role.kubernetes.io/etcd # operator: Exists - + # Configure CSI CSI Ceph FS grpc and liveness metrics port # CSI_CEPHFS_GRPC_METRICS_PORT: "9091" # CSI_CEPHFS_LIVENESS_METRICS_PORT: "9081" diff --git a/cluster/examples/kubernetes/ceph/upgrade-from-v1.2-apply.yaml b/cluster/examples/kubernetes/ceph/upgrade-from-v1.2-apply.yaml index a8163f44f..860aa315e 100644 --- a/cluster/examples/kubernetes/ceph/upgrade-from-v1.2-apply.yaml +++ b/cluster/examples/kubernetes/ceph/upgrade-from-v1.2-apply.yaml @@ -220,4 +220,18 @@ rules: resources: - csidrivers verbs: - - create \ No newline at end of file + - create +--- +# Rook Ceph Operator Config ConfigMap +# Use this ConfigMap to override Rook-Ceph Operator configurations. +# NOTE! Precedence will be given to this config if the same Env Var config also exists in the +# Operator Deployment. +# To move a configuration(s) from the Operator Deployment to this ConfigMap, add the config +# here. It is recommended to then remove it from the Deployment to eliminate any future confusion. +kind: ConfigMap +apiVersion: v1 +metadata: + name: rook-ceph-operator-config + # should be in the namespace of the operator + namespace: rook-ceph +data: diff --git a/cluster/examples/kubernetes/ceph/upgrade-from-v1.2-crds.yaml b/cluster/examples/kubernetes/ceph/upgrade-from-v1.2-crds.yaml index 50f62c2be..86e3bb271 100644 --- a/cluster/examples/kubernetes/ceph/upgrade-from-v1.2-crds.yaml +++ b/cluster/examples/kubernetes/ceph/upgrade-from-v1.2-crds.yaml @@ -212,7 +212,7 @@ spec: caps: type: object subresources: - status: {} + status: {} --- apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition @@ -522,4 +522,4 @@ spec: scope: Namespaced version: v1alpha2 subresources: - status: {} \ No newline at end of file + status: {} From f30c8066d199a5d18ce01a9c3ef8df34da63bb0e Mon Sep 17 00:00:00 2001 From: Travis Nielsen Date: Wed, 1 Apr 2020 15:21:03 -0600 Subject: [PATCH 2/2] docs: update docs for Rook v1.3 release Various updates are needed for the v1.3 release. - The pending release notes were missing some features - Added a section to the upgrade guide for breaking changes to OSDs - Clarifications around Ceph prereqs and min version - Other misc clarifications Signed-off-by: Travis Nielsen --- Documentation/ceph-prerequisites.md | 7 +++++ Documentation/ceph-quickstart.md | 6 ++-- Documentation/ceph-teardown.md | 3 ++ Documentation/ceph-upgrade.md | 46 +++++++++++++++++++++++++---- Documentation/helm-operator.md | 4 +-- PendingReleaseNotes.md | 17 +++++++++-- tests/README.md | 13 ++------ 7 files changed, 71 insertions(+), 25 deletions(-) diff --git a/Documentation/ceph-prerequisites.md b/Documentation/ceph-prerequisites.md index 4e674ec4c..5e9819aa6 100644 --- a/Documentation/ceph-prerequisites.md +++ b/Documentation/ceph-prerequisites.md @@ -6,6 +6,13 @@ indent: true # Ceph Prerequisites +To make sure you have a Kubernetes cluster that is ready for `Rook`, review the general [Rook Prerequisites](k8s-pre-reqs.md). + +In order to configure the Ceph storage cluster, at least one of these local storage options are required: +- Raw devices (no partitions or formatted filesystems) +- Raw partitions (no formatted filesystem) +- PVs available from a storage class in `block` mode + ## LVM package Ceph OSDs have a dependency on LVM in the following scenarios: diff --git a/Documentation/ceph-quickstart.md b/Documentation/ceph-quickstart.md index 95ed9abfd..5e6d24c37 100644 --- a/Documentation/ceph-quickstart.md +++ b/Documentation/ceph-quickstart.md @@ -14,15 +14,15 @@ from other pods running in your cluster. ## Minimum Version -Kubernetes **v1.10** or higher is supported by Rook. +Kubernetes **v1.11** or higher is supported by Rook. ## Prerequisites To make sure you have a Kubernetes cluster that is ready for `Rook`, you can [follow these instructions](k8s-pre-reqs.md). -In order to configure the storage cluster, at least one of these local storage options are required: +In order to configure the Ceph storage cluster, at least one of these local storage options are required: - Raw devices (no partitions or formatted filesystems) -- Raw partitions (no formatted filsystem) +- Raw partitions (no formatted filesystem) - PVs available from a storage class in `block` mode ## TL;DR diff --git a/Documentation/ceph-teardown.md b/Documentation/ceph-teardown.md index cd1315644..ccf9c5732 100644 --- a/Documentation/ceph-teardown.md +++ b/Documentation/ceph-teardown.md @@ -66,6 +66,8 @@ In the future this step will not be necessary when we build on the K8s local sto If you modified the demo settings, additional cleanup is up to you for devices, host paths, etc. +### Zapping Devices + Disks on nodes used by Rook for osds can be reset to a usable state with the following methods: ```console @@ -74,6 +76,7 @@ DISK="/dev/sdb" # Zap the disk to a fresh, usable state (zap-all is important, b/c MBR has to be clean) # You will have to run this step for all disks. sgdisk --zap-all $DISK +dd if=/dev/zero of="$DISK" bs=1M count=100 oflag=direct,dsync # These steps only have to be run once on each node # If rook sets up osds using ceph-volume, teardown leaves some devices mapped that lock the disks. diff --git a/Documentation/ceph-upgrade.md b/Documentation/ceph-upgrade.md index 0f136e93e..c5135213f 100644 --- a/Documentation/ceph-upgrade.md +++ b/Documentation/ceph-upgrade.md @@ -45,6 +45,38 @@ With this upgrade guide, there are a few notes to consider: both Rook operator updates and for Ceph version updates. * We recommend that you read this document in full before you undertake a Rook cluster upgrade. +## Before you Upgrade + +Rook v1.3 has several breaking changes that must be considered **before** upgrading to Rook v1.3. +1. Ceph Mimic is no longer supported. You must update to Ceph Nautilus v14.2.5 or newer **before** updating to Rook v1.3. See the + Rook v1.2 [Upgrade Guide for Ceph](https://rook.github.io/docs/rook/v1.2/ceph-upgrade.html#ceph-version-upgrades). +2. Directory-based OSDs are no longer supported. If you are specifying a + [`directory`](https://github.com/rook/rook/blob/release-1.2/cluster/examples/kubernetes/ceph/cluster-test.yaml#L49-L50) + in your CephCluster CR you will need to convert these OSDs to device-based OSDs. See the next section. +3. OSDs created **before Rook v0.9** are no longer supported during upgrade. To detect if you have these legacy + OSDs in your cluster, run `lsblk` on each host to see what partitions exist. + - If you see **three partitions** on a device running an OSD, this is a legacy OSD. See below for instructions to convert them. + - If you see a single partition or LV with the `ceph` prefix, the OSD is **not** a legacy OSD and you will not need to convert the OSDs. + +### Converting Legacy OSDs + +If you have legacy OSDs as described in the previous section, it is recommended to migrate the OSDs before the upgrade, +although the migration is not strictly required before upgrade. If you proceed with the upgrade to v1.3 the OSDs will +continue running. However, they will no longer be managed by Rook. The OSDs will never again be updated when you update +Ceph or any other settings in your CephCluster CR. + +The procedure to migrate the OSDs requires raw devices or partitions be available as mentioned in the [Ceph Prerequisites](ceph-prerequisites.md). +After you have available storage devices available in your cluster, do the following: +1. Configure OSDs on the new devices using the [Storage selection settings](https://rook.github.io/docs/rook/v1.3/ceph-cluster-crd.html#storage-selection-settings). + If you have specified `useAllDevices: true`, you may need to restart the operator in order to trigger creation of new OSDs on new devices. +2. Follow the [OSD Removal Guide](https://rook.github.io/docs/rook/v1.3/ceph-osd-mgmt.html#remove-an-osd) to remove the legacy OSDs. + - Before removing each OSD, ensure the PGs are all `active+clean` before continuing with the next OSD to ensure your data is safe. + +If you need to re-use your devices and add OSDs to the same devices, you'll need to zap a device before creating a new +OSD on the same device. For steps on zapping the device see the [Rook cleanup instructions](ceph-teardown.md#zapping-devices). + +> **Removing OSDs and zapping devices is destructive so proceed with extreme caution** + ## Upgrading the Rook-Ceph Operator ## Patch Release Upgrades @@ -232,13 +264,15 @@ time without compatibility support and without prior notice. Let's get started! +## 0. Migrate Legacy OSDs + +As described above in the [Before You Upgrade](#before-you-upgrade) section, you must migrate legacy OSDs +before upgrading to Rook v1.3. + ## 1. Update Ceph to Nautilus version 14.2.5 or higher -Rook v1.3 supports Ceph Nautilus v14.2.5 or newer and Ceph Octopus v15.2.0 or newer. These are the -only supported major versions of Ceph. Rook v1.2 does not support Ceph Octopus. Therefore, if your -Rook v1.2 cluster must be updated to Ceph Nautilus v14.2.5 or higher before upgrading the Rook -Operator to v1.3. If an upgrade is necessary, follow the Ceph upgrade steps from Rook's v1.2 release -[here](https://rook.io/docs/rook/v1.2/ceph-upgrade.html#ceph-version-upgrades). +As described above in the [Before You Upgrade](#before-you-upgrade) section, you must upgrade to +Nautilus version 14.2.5 or higher before upgrading to Rook v1.3. ## 2. Update the RBAC and CRDs @@ -255,7 +289,7 @@ kubectl apply -f upgrade-from-v1.2-apply.yaml -f upgrade-from-v1.2-crds.yaml If you have specified custom CSI images in the Rook-Ceph Operator deployment, you should update your the deployment to use the latest Ceph-CSI v2.0 and related drivers. If you have not specified custom -CSI images in the Operator deployment this step is unnecessary; Rook v1.3 will use the latest +CSI images in the Operator deployment this step is unnecessary since Rook v1.3 will use the latest drivers by default. See the section [CSI Updates](#csi-updates) for details about how to do this. diff --git a/Documentation/helm-operator.md b/Documentation/helm-operator.md index 8fec03379..d2af285f3 100644 --- a/Documentation/helm-operator.md +++ b/Documentation/helm-operator.md @@ -14,7 +14,7 @@ This chart bootstraps a [rook-ceph-operator](https://github.com/rook/rook) deplo ## Prerequisites -* Kubernetes 1.10+ +* Kubernetes 1.11+ ### RBAC @@ -130,7 +130,7 @@ The following tables lists the configurable parameters of the rook-operator char | `csi.rbdLivenessMetricsPort` | Ceph CSI RBD driver metrics port. | `8080` | | `csi.enableSnapshotter` | Enable deployment of snapshotter container in ceph-csi provisioner. | `true` | | `csi.forceCephFSKernelClient` | Enable Ceph Kernel clients on kernel < 4.17 which support quotas for Cephfs. | `true` | -| `csi.allowUnsupportedVersion` | Allow to run unsupported and untested CSI image. | `false` | +| `csi.allowUnsupportedVersion` | Allow to run unsupported and untested CSI image. | `false` | | `csi.kubeletDirPath` | Kubelet root directory path (if the Kubelet uses a different path for the `--root-dir` flag) | `/var/lib/kubelet` | | `csi.cephcsi.image` | Ceph CSI image. | `quay.io/cephcsi/cephcsi:v2.0.0` | | `csi.rbdPluginUpdateStrategy` | CSI Rbd plugin daemonset update strategy, supported values are OnDelete and RollingUpdate. | `OnDelete` | diff --git a/PendingReleaseNotes.md b/PendingReleaseNotes.md index a03266f31..ed014291f 100644 --- a/PendingReleaseNotes.md +++ b/PendingReleaseNotes.md @@ -4,14 +4,18 @@ ## Notable Features -- Added K8s 1.17 to the test matrix and removed K8s 1.12 from the test matrix. +- Added K8s 1.17 and K8s 1.18 to the test matrix and removed K8s 1.12 from the test matrix. +- Tests are still run against K8s 1.11 in the release branch as the minimum supported version. +- Rook is now built with go modules in golang 1.13 ### Ceph +- Ceph Octopus is now supported, with continued support for Nautilus. + - Ceph Nautilus 14.2.5 is the minimum supported version + - Support for Mimic was removed. Clusters must be upgraded to Nautilus before upgrading to v1.3. - OSD refactor: drop support for Rook legacy OSD, directory OSD and Filestore OSD. For more details refer to the [corresponding issue](https://github.com/rook/rook/issues/4724). - OSD on PVC now supports a metadata device, [refer to the cluster on PVC section](Documentation/ceph-cluster-crd.html#dedicated-metatada-device) or the [corresponding issue](https://github.com/rook/rook/issues/3852). - OSD on PVC now supports PVC expansion, if the size of the underlying block increases the Bluestore main block and the overall storage capacity will grow up. -- Ceph Nautilus 14.2.5 is the minimum supported version - OSD on PVC doesn't use LVM anymore to configure OSD, but solely relies on the entire block device, done [here](https://github.com/rook/rook/pull/4435). - Specific devices for OSDs can now be specified using the full udev path (e.g. /dev/disk/by-id/ata-ST4000DM004-XXXX) instead of the device name. - OSD on PVC CRUSH device storage class can now be changed by setting an annotation "crushDeviceClass" on the "data" volume template. See "cluster-on-pvc.yaml" for example. @@ -20,6 +24,10 @@ ConfigMap can be used in mix with the already existing Env Vars defined in operator deployment manifest. Precedence will be given to ConfigMap in case of conflicting configurations. - Rook Ceph cleanupPolicy will clean up the dataDirHostPath only after user confirmation. For more info about CleanUpPolicy [read the design](https://github.com/rook/rook/blob/master/design/ceph/ceph-cluster-cleanup.md) as well as the documentation [cleanupPolicy](Documentation/ceph-cluster-crd.md#cluster-settings) - Rook monitor, mds and osd now have liveness probe checks on their respective sockets +- Most of the operator's controller implementations are now based on the controller runtime. No impact to user clusters is expected. +- Object stores can be created without specifying pools in the CephObjectStore CR to allow for pool management outside of Rook such as in the dashboard. +- Operator logs are more concise with the Ceph commands only printing at debug level. +- Various improvements to the integration tests including tests for an external cluster and for a cluster running on PVCs. - Pools can now be configured to inline compress the data using the `compressionMode` parameter. Support added [here](https://github.com/rook/rook/pull/5124) - Ceph OSDs do not use the host PID, but the PID namespace of the pod (more security). The OSD does not see host running processes anymore. @@ -35,7 +43,10 @@ ConfigMap can be used in mix with the already existing Env Vars defined in opera ## Breaking Changes -### +### Ceph +- Support for Mimic was removed +- Support for filestore (directory-based) OSDs was removed. See the upgrade guide for migration instructions. +- Support for OSDs created in Rook v0.9 or earlier was removed (created with Rook partitions instead of ceph-volume). See the upgrade guide for migration instructions. ### Minio - The minio operator was removed from Rook diff --git a/tests/README.md b/tests/README.md index f0624226b..a71fadb75 100644 --- a/tests/README.md +++ b/tests/README.md @@ -100,15 +100,6 @@ Integration tests can be run using tests binary `_output/tests/${platform}/integ _output/tests/${platform}/integration -test.v ``` -### Test parameters -In addition to standard go tests parameters, the following custom parameters are available while running tests: - -| Parameter | Description | Possible values | Default | -| ------------- | -------------------------------------- | ---------------- | ---------- | -| rook_platform | platform Rook needs to be installed on | kubernetes | kubernetes | -| k8s_version | version of Kubernetes to be installed | v1.10+ | v1.13.1 | -| rook_image | Rook image name to be installed | valid image name | rook/ceph | - ### Running Tests with parameters. #### To run all integration tests run ``` @@ -117,10 +108,10 @@ go test -v -timeout 1800s github.com/rook/rook/tests/integration #### To run a specific suite (uses regex) ``` -go test -v -timeout 1800s -run SmokeSuite github.com/rook/rook/tests/integration +go test -v -timeout 1800s -run CephSmokeSuite github.com/rook/rook/tests/integration ``` #### To run specific tests inside a suite: ``` -go test -v -timeout 1800s -run SmokeSuite github.com/rook/rook/tests/integration -testify.m TestRookClusterInstallation_SmokeTest +go test -v -timeout 1800s -run CephSmokeSuite github.com/rook/rook/tests/integration -testify.m TestRookClusterInstallation_SmokeTest ```