From f30c8066d199a5d18ce01a9c3ef8df34da63bb0e Mon Sep 17 00:00:00 2001 From: Travis Nielsen Date: Wed, 1 Apr 2020 15:21:03 -0600 Subject: [PATCH] docs: update docs for Rook v1.3 release Various updates are needed for the v1.3 release. - The pending release notes were missing some features - Added a section to the upgrade guide for breaking changes to OSDs - Clarifications around Ceph prereqs and min version - Other misc clarifications Signed-off-by: Travis Nielsen --- Documentation/ceph-prerequisites.md | 7 +++++ Documentation/ceph-quickstart.md | 6 ++-- Documentation/ceph-teardown.md | 3 ++ Documentation/ceph-upgrade.md | 46 +++++++++++++++++++++++++---- Documentation/helm-operator.md | 4 +-- PendingReleaseNotes.md | 17 +++++++++-- tests/README.md | 13 ++------ 7 files changed, 71 insertions(+), 25 deletions(-) diff --git a/Documentation/ceph-prerequisites.md b/Documentation/ceph-prerequisites.md index 4e674ec4c..5e9819aa6 100644 --- a/Documentation/ceph-prerequisites.md +++ b/Documentation/ceph-prerequisites.md @@ -6,6 +6,13 @@ indent: true # Ceph Prerequisites +To make sure you have a Kubernetes cluster that is ready for `Rook`, review the general [Rook Prerequisites](k8s-pre-reqs.md). + +In order to configure the Ceph storage cluster, at least one of these local storage options are required: +- Raw devices (no partitions or formatted filesystems) +- Raw partitions (no formatted filesystem) +- PVs available from a storage class in `block` mode + ## LVM package Ceph OSDs have a dependency on LVM in the following scenarios: diff --git a/Documentation/ceph-quickstart.md b/Documentation/ceph-quickstart.md index 95ed9abfd..5e6d24c37 100644 --- a/Documentation/ceph-quickstart.md +++ b/Documentation/ceph-quickstart.md @@ -14,15 +14,15 @@ from other pods running in your cluster. ## Minimum Version -Kubernetes **v1.10** or higher is supported by Rook. +Kubernetes **v1.11** or higher is supported by Rook. ## Prerequisites To make sure you have a Kubernetes cluster that is ready for `Rook`, you can [follow these instructions](k8s-pre-reqs.md). -In order to configure the storage cluster, at least one of these local storage options are required: +In order to configure the Ceph storage cluster, at least one of these local storage options are required: - Raw devices (no partitions or formatted filesystems) -- Raw partitions (no formatted filsystem) +- Raw partitions (no formatted filesystem) - PVs available from a storage class in `block` mode ## TL;DR diff --git a/Documentation/ceph-teardown.md b/Documentation/ceph-teardown.md index cd1315644..ccf9c5732 100644 --- a/Documentation/ceph-teardown.md +++ b/Documentation/ceph-teardown.md @@ -66,6 +66,8 @@ In the future this step will not be necessary when we build on the K8s local sto If you modified the demo settings, additional cleanup is up to you for devices, host paths, etc. +### Zapping Devices + Disks on nodes used by Rook for osds can be reset to a usable state with the following methods: ```console @@ -74,6 +76,7 @@ DISK="/dev/sdb" # Zap the disk to a fresh, usable state (zap-all is important, b/c MBR has to be clean) # You will have to run this step for all disks. sgdisk --zap-all $DISK +dd if=/dev/zero of="$DISK" bs=1M count=100 oflag=direct,dsync # These steps only have to be run once on each node # If rook sets up osds using ceph-volume, teardown leaves some devices mapped that lock the disks. diff --git a/Documentation/ceph-upgrade.md b/Documentation/ceph-upgrade.md index 0f136e93e..c5135213f 100644 --- a/Documentation/ceph-upgrade.md +++ b/Documentation/ceph-upgrade.md @@ -45,6 +45,38 @@ With this upgrade guide, there are a few notes to consider: both Rook operator updates and for Ceph version updates. * We recommend that you read this document in full before you undertake a Rook cluster upgrade. +## Before you Upgrade + +Rook v1.3 has several breaking changes that must be considered **before** upgrading to Rook v1.3. +1. Ceph Mimic is no longer supported. You must update to Ceph Nautilus v14.2.5 or newer **before** updating to Rook v1.3. See the + Rook v1.2 [Upgrade Guide for Ceph](https://rook.github.io/docs/rook/v1.2/ceph-upgrade.html#ceph-version-upgrades). +2. Directory-based OSDs are no longer supported. If you are specifying a + [`directory`](https://github.com/rook/rook/blob/release-1.2/cluster/examples/kubernetes/ceph/cluster-test.yaml#L49-L50) + in your CephCluster CR you will need to convert these OSDs to device-based OSDs. See the next section. +3. OSDs created **before Rook v0.9** are no longer supported during upgrade. To detect if you have these legacy + OSDs in your cluster, run `lsblk` on each host to see what partitions exist. + - If you see **three partitions** on a device running an OSD, this is a legacy OSD. See below for instructions to convert them. + - If you see a single partition or LV with the `ceph` prefix, the OSD is **not** a legacy OSD and you will not need to convert the OSDs. + +### Converting Legacy OSDs + +If you have legacy OSDs as described in the previous section, it is recommended to migrate the OSDs before the upgrade, +although the migration is not strictly required before upgrade. If you proceed with the upgrade to v1.3 the OSDs will +continue running. However, they will no longer be managed by Rook. The OSDs will never again be updated when you update +Ceph or any other settings in your CephCluster CR. + +The procedure to migrate the OSDs requires raw devices or partitions be available as mentioned in the [Ceph Prerequisites](ceph-prerequisites.md). +After you have available storage devices available in your cluster, do the following: +1. Configure OSDs on the new devices using the [Storage selection settings](https://rook.github.io/docs/rook/v1.3/ceph-cluster-crd.html#storage-selection-settings). + If you have specified `useAllDevices: true`, you may need to restart the operator in order to trigger creation of new OSDs on new devices. +2. Follow the [OSD Removal Guide](https://rook.github.io/docs/rook/v1.3/ceph-osd-mgmt.html#remove-an-osd) to remove the legacy OSDs. + - Before removing each OSD, ensure the PGs are all `active+clean` before continuing with the next OSD to ensure your data is safe. + +If you need to re-use your devices and add OSDs to the same devices, you'll need to zap a device before creating a new +OSD on the same device. For steps on zapping the device see the [Rook cleanup instructions](ceph-teardown.md#zapping-devices). + +> **Removing OSDs and zapping devices is destructive so proceed with extreme caution** + ## Upgrading the Rook-Ceph Operator ## Patch Release Upgrades @@ -232,13 +264,15 @@ time without compatibility support and without prior notice. Let's get started! +## 0. Migrate Legacy OSDs + +As described above in the [Before You Upgrade](#before-you-upgrade) section, you must migrate legacy OSDs +before upgrading to Rook v1.3. + ## 1. Update Ceph to Nautilus version 14.2.5 or higher -Rook v1.3 supports Ceph Nautilus v14.2.5 or newer and Ceph Octopus v15.2.0 or newer. These are the -only supported major versions of Ceph. Rook v1.2 does not support Ceph Octopus. Therefore, if your -Rook v1.2 cluster must be updated to Ceph Nautilus v14.2.5 or higher before upgrading the Rook -Operator to v1.3. If an upgrade is necessary, follow the Ceph upgrade steps from Rook's v1.2 release -[here](https://rook.io/docs/rook/v1.2/ceph-upgrade.html#ceph-version-upgrades). +As described above in the [Before You Upgrade](#before-you-upgrade) section, you must upgrade to +Nautilus version 14.2.5 or higher before upgrading to Rook v1.3. ## 2. Update the RBAC and CRDs @@ -255,7 +289,7 @@ kubectl apply -f upgrade-from-v1.2-apply.yaml -f upgrade-from-v1.2-crds.yaml If you have specified custom CSI images in the Rook-Ceph Operator deployment, you should update your the deployment to use the latest Ceph-CSI v2.0 and related drivers. If you have not specified custom -CSI images in the Operator deployment this step is unnecessary; Rook v1.3 will use the latest +CSI images in the Operator deployment this step is unnecessary since Rook v1.3 will use the latest drivers by default. See the section [CSI Updates](#csi-updates) for details about how to do this. diff --git a/Documentation/helm-operator.md b/Documentation/helm-operator.md index 8fec03379..d2af285f3 100644 --- a/Documentation/helm-operator.md +++ b/Documentation/helm-operator.md @@ -14,7 +14,7 @@ This chart bootstraps a [rook-ceph-operator](https://github.com/rook/rook) deplo ## Prerequisites -* Kubernetes 1.10+ +* Kubernetes 1.11+ ### RBAC @@ -130,7 +130,7 @@ The following tables lists the configurable parameters of the rook-operator char | `csi.rbdLivenessMetricsPort` | Ceph CSI RBD driver metrics port. | `8080` | | `csi.enableSnapshotter` | Enable deployment of snapshotter container in ceph-csi provisioner. | `true` | | `csi.forceCephFSKernelClient` | Enable Ceph Kernel clients on kernel < 4.17 which support quotas for Cephfs. | `true` | -| `csi.allowUnsupportedVersion` | Allow to run unsupported and untested CSI image. | `false` | +| `csi.allowUnsupportedVersion` | Allow to run unsupported and untested CSI image. | `false` | | `csi.kubeletDirPath` | Kubelet root directory path (if the Kubelet uses a different path for the `--root-dir` flag) | `/var/lib/kubelet` | | `csi.cephcsi.image` | Ceph CSI image. | `quay.io/cephcsi/cephcsi:v2.0.0` | | `csi.rbdPluginUpdateStrategy` | CSI Rbd plugin daemonset update strategy, supported values are OnDelete and RollingUpdate. | `OnDelete` | diff --git a/PendingReleaseNotes.md b/PendingReleaseNotes.md index a03266f31..ed014291f 100644 --- a/PendingReleaseNotes.md +++ b/PendingReleaseNotes.md @@ -4,14 +4,18 @@ ## Notable Features -- Added K8s 1.17 to the test matrix and removed K8s 1.12 from the test matrix. +- Added K8s 1.17 and K8s 1.18 to the test matrix and removed K8s 1.12 from the test matrix. +- Tests are still run against K8s 1.11 in the release branch as the minimum supported version. +- Rook is now built with go modules in golang 1.13 ### Ceph +- Ceph Octopus is now supported, with continued support for Nautilus. + - Ceph Nautilus 14.2.5 is the minimum supported version + - Support for Mimic was removed. Clusters must be upgraded to Nautilus before upgrading to v1.3. - OSD refactor: drop support for Rook legacy OSD, directory OSD and Filestore OSD. For more details refer to the [corresponding issue](https://github.com/rook/rook/issues/4724). - OSD on PVC now supports a metadata device, [refer to the cluster on PVC section](Documentation/ceph-cluster-crd.html#dedicated-metatada-device) or the [corresponding issue](https://github.com/rook/rook/issues/3852). - OSD on PVC now supports PVC expansion, if the size of the underlying block increases the Bluestore main block and the overall storage capacity will grow up. -- Ceph Nautilus 14.2.5 is the minimum supported version - OSD on PVC doesn't use LVM anymore to configure OSD, but solely relies on the entire block device, done [here](https://github.com/rook/rook/pull/4435). - Specific devices for OSDs can now be specified using the full udev path (e.g. /dev/disk/by-id/ata-ST4000DM004-XXXX) instead of the device name. - OSD on PVC CRUSH device storage class can now be changed by setting an annotation "crushDeviceClass" on the "data" volume template. See "cluster-on-pvc.yaml" for example. @@ -20,6 +24,10 @@ ConfigMap can be used in mix with the already existing Env Vars defined in operator deployment manifest. Precedence will be given to ConfigMap in case of conflicting configurations. - Rook Ceph cleanupPolicy will clean up the dataDirHostPath only after user confirmation. For more info about CleanUpPolicy [read the design](https://github.com/rook/rook/blob/master/design/ceph/ceph-cluster-cleanup.md) as well as the documentation [cleanupPolicy](Documentation/ceph-cluster-crd.md#cluster-settings) - Rook monitor, mds and osd now have liveness probe checks on their respective sockets +- Most of the operator's controller implementations are now based on the controller runtime. No impact to user clusters is expected. +- Object stores can be created without specifying pools in the CephObjectStore CR to allow for pool management outside of Rook such as in the dashboard. +- Operator logs are more concise with the Ceph commands only printing at debug level. +- Various improvements to the integration tests including tests for an external cluster and for a cluster running on PVCs. - Pools can now be configured to inline compress the data using the `compressionMode` parameter. Support added [here](https://github.com/rook/rook/pull/5124) - Ceph OSDs do not use the host PID, but the PID namespace of the pod (more security). The OSD does not see host running processes anymore. @@ -35,7 +43,10 @@ ConfigMap can be used in mix with the already existing Env Vars defined in opera ## Breaking Changes -### +### Ceph +- Support for Mimic was removed +- Support for filestore (directory-based) OSDs was removed. See the upgrade guide for migration instructions. +- Support for OSDs created in Rook v0.9 or earlier was removed (created with Rook partitions instead of ceph-volume). See the upgrade guide for migration instructions. ### Minio - The minio operator was removed from Rook diff --git a/tests/README.md b/tests/README.md index f0624226b..a71fadb75 100644 --- a/tests/README.md +++ b/tests/README.md @@ -100,15 +100,6 @@ Integration tests can be run using tests binary `_output/tests/${platform}/integ _output/tests/${platform}/integration -test.v ``` -### Test parameters -In addition to standard go tests parameters, the following custom parameters are available while running tests: - -| Parameter | Description | Possible values | Default | -| ------------- | -------------------------------------- | ---------------- | ---------- | -| rook_platform | platform Rook needs to be installed on | kubernetes | kubernetes | -| k8s_version | version of Kubernetes to be installed | v1.10+ | v1.13.1 | -| rook_image | Rook image name to be installed | valid image name | rook/ceph | - ### Running Tests with parameters. #### To run all integration tests run ``` @@ -117,10 +108,10 @@ go test -v -timeout 1800s github.com/rook/rook/tests/integration #### To run a specific suite (uses regex) ``` -go test -v -timeout 1800s -run SmokeSuite github.com/rook/rook/tests/integration +go test -v -timeout 1800s -run CephSmokeSuite github.com/rook/rook/tests/integration ``` #### To run specific tests inside a suite: ``` -go test -v -timeout 1800s -run SmokeSuite github.com/rook/rook/tests/integration -testify.m TestRookClusterInstallation_SmokeTest +go test -v -timeout 1800s -run CephSmokeSuite github.com/rook/rook/tests/integration -testify.m TestRookClusterInstallation_SmokeTest ```