docs: update docs for Rook v1.3 release

Various updates are needed for the v1.3 release.
- The pending release notes were missing some features
- Added a section to the upgrade guide for breaking changes to OSDs
- Clarifications around Ceph prereqs and min version
- Other misc clarifications

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
This commit is contained in:
Travis Nielsen committed 2020-04-02 10:26:00 -06:00
1 parent 9d41b38095
commit f30c8066d1
7 files changed
+71 -25

No files matched your search

+7
View File
@@ -6,6 +6,13 @@ indent: true
# Ceph Prerequisites
To make sure you have a Kubernetes cluster that is ready for `Rook`, review the general [Rook Prerequisites](k8s-pre-reqs.md).
In order to configure the Ceph storage cluster, at least one of these local storage options are required:
- Raw devices (no partitions or formatted filesystems)
- Raw partitions (no formatted filesystem)
- PVs available from a storage class in `block` mode
## LVM package
Ceph OSDs have a dependency on LVM in the following scenarios:
+3 -3
View File
@@ -14,15 +14,15 @@ from other pods running in your cluster.
## Minimum Version
Kubernetes **v1.10** or higher is supported by Rook.
Kubernetes **v1.11** or higher is supported by Rook.
## Prerequisites
To make sure you have a Kubernetes cluster that is ready for `Rook`, you can [follow these instructions](k8s-pre-reqs.md).
In order to configure the storage cluster, at least one of these local storage options are required:
In order to configure the Ceph storage cluster, at least one of these local storage options are required:
- Raw devices (no partitions or formatted filesystems)
- Raw partitions (no formatted filsystem)
- Raw partitions (no formatted filesystem)
- PVs available from a storage class in `block` mode
## TL;DR
+3
View File
@@ -66,6 +66,8 @@ In the future this step will not be necessary when we build on the K8s local sto
If you modified the demo settings, additional cleanup is up to you for devices, host paths, etc.
### Zapping Devices
Disks on nodes used by Rook for osds can be reset to a usable state with the following methods:
```console
@@ -74,6 +76,7 @@ DISK="/dev/sdb"
# Zap the disk to a fresh, usable state (zap-all is important, b/c MBR has to be clean)
# You will have to run this step for all disks.
sgdisk --zap-all $DISK
dd if=/dev/zero of="$DISK" bs=1M count=100 oflag=direct,dsync
# These steps only have to be run once on each node
# If rook sets up osds using ceph-volume, teardown leaves some devices mapped that lock the disks.
+40 -6
View File
@@ -45,6 +45,38 @@ With this upgrade guide, there are a few notes to consider:
both Rook operator updates and for Ceph version updates.
* We recommend that you read this document in full before you undertake a Rook cluster upgrade.
## Before you Upgrade
Rook v1.3 has several breaking changes that must be considered **before** upgrading to Rook v1.3.
1. Ceph Mimic is no longer supported. You must update to Ceph Nautilus v14.2.5 or newer **before** updating to Rook v1.3. See the
Rook v1.2 [Upgrade Guide for Ceph](https://rook.github.io/docs/rook/v1.2/ceph-upgrade.html#ceph-version-upgrades).
2. Directory-based OSDs are no longer supported. If you are specifying a
[`directory`](https://github.com/rook/rook/blob/release-1.2/cluster/examples/kubernetes/ceph/cluster-test.yaml#L49-L50)
in your CephCluster CR you will need to convert these OSDs to device-based OSDs. See the next section.
3. OSDs created **before Rook v0.9** are no longer supported during upgrade. To detect if you have these legacy
OSDs in your cluster, run `lsblk` on each host to see what partitions exist.
- If you see **three partitions** on a device running an OSD, this is a legacy OSD. See below for instructions to convert them.
- If you see a single partition or LV with the `ceph` prefix, the OSD is **not** a legacy OSD and you will not need to convert the OSDs.
### Converting Legacy OSDs
If you have legacy OSDs as described in the previous section, it is recommended to migrate the OSDs before the upgrade,
although the migration is not strictly required before upgrade. If you proceed with the upgrade to v1.3 the OSDs will
continue running. However, they will no longer be managed by Rook. The OSDs will never again be updated when you update
Ceph or any other settings in your CephCluster CR.
The procedure to migrate the OSDs requires raw devices or partitions be available as mentioned in the [Ceph Prerequisites](ceph-prerequisites.md).
After you have available storage devices available in your cluster, do the following:
1. Configure OSDs on the new devices using the [Storage selection settings](https://rook.github.io/docs/rook/v1.3/ceph-cluster-crd.html#storage-selection-settings).
If you have specified `useAllDevices: true`, you may need to restart the operator in order to trigger creation of new OSDs on new devices.
2. Follow the [OSD Removal Guide](https://rook.github.io/docs/rook/v1.3/ceph-osd-mgmt.html#remove-an-osd) to remove the legacy OSDs.
- Before removing each OSD, ensure the PGs are all `active+clean` before continuing with the next OSD to ensure your data is safe.
If you need to re-use your devices and add OSDs to the same devices, you'll need to zap a device before creating a new
OSD on the same device. For steps on zapping the device see the [Rook cleanup instructions](ceph-teardown.md#zapping-devices).
> **Removing OSDs and zapping devices is destructive so proceed with extreme caution**
## Upgrading the Rook-Ceph Operator
## Patch Release Upgrades
@@ -232,13 +264,15 @@ time without compatibility support and without prior notice.
Let's get started!
## 0. Migrate Legacy OSDs
As described above in the [Before You Upgrade](#before-you-upgrade) section, you must migrate legacy OSDs
before upgrading to Rook v1.3.
## 1. Update Ceph to Nautilus version 14.2.5 or higher
Rook v1.3 supports Ceph Nautilus v14.2.5 or newer and Ceph Octopus v15.2.0 or newer. These are the
only supported major versions of Ceph. Rook v1.2 does not support Ceph Octopus. Therefore, if your
Rook v1.2 cluster must be updated to Ceph Nautilus v14.2.5 or higher before upgrading the Rook
Operator to v1.3. If an upgrade is necessary, follow the Ceph upgrade steps from Rook's v1.2 release
[here](https://rook.io/docs/rook/v1.2/ceph-upgrade.html#ceph-version-upgrades).
As described above in the [Before You Upgrade](#before-you-upgrade) section, you must upgrade to
Nautilus version 14.2.5 or higher before upgrading to Rook v1.3.
## 2. Update the RBAC and CRDs
@@ -255,7 +289,7 @@ kubectl apply -f upgrade-from-v1.2-apply.yaml -f upgrade-from-v1.2-crds.yaml
If you have specified custom CSI images in the Rook-Ceph Operator deployment, you should update your
the deployment to use the latest Ceph-CSI v2.0 and related drivers. If you have not specified custom
CSI images in the Operator deployment this step is unnecessary; Rook v1.3 will use the latest
CSI images in the Operator deployment this step is unnecessary since Rook v1.3 will use the latest
drivers by default.
See the section [CSI Updates](#csi-updates) for details about how to do this.
+2 -2
View File
@@ -14,7 +14,7 @@ This chart bootstraps a [rook-ceph-operator](https://github.com/rook/rook) deplo
## Prerequisites
* Kubernetes 1.10+
* Kubernetes 1.11+
### RBAC
@@ -130,7 +130,7 @@ The following tables lists the configurable parameters of the rook-operator char
| `csi.rbdLivenessMetricsPort` | Ceph CSI RBD driver metrics port. | `8080` |
| `csi.enableSnapshotter` | Enable deployment of snapshotter container in ceph-csi provisioner. | `true` |
| `csi.forceCephFSKernelClient` | Enable Ceph Kernel clients on kernel < 4.17 which support quotas for Cephfs. | `true` |
| `csi.allowUnsupportedVersion` | Allow to run unsupported and untested CSI image. | `false` |
| `csi.allowUnsupportedVersion` | Allow to run unsupported and untested CSI image. | `false` |
| `csi.kubeletDirPath` | Kubelet root directory path (if the Kubelet uses a different path for the `--root-dir` flag) | `/var/lib/kubelet` |
| `csi.cephcsi.image` | Ceph CSI image. | `quay.io/cephcsi/cephcsi:v2.0.0` |
| `csi.rbdPluginUpdateStrategy` | CSI Rbd plugin daemonset update strategy, supported values are OnDelete and RollingUpdate. | `OnDelete` |
+14 -3
View File
@@ -4,14 +4,18 @@
## Notable Features
- Added K8s 1.17 to the test matrix and removed K8s 1.12 from the test matrix.
- Added K8s 1.17 and K8s 1.18 to the test matrix and removed K8s 1.12 from the test matrix.
- Tests are still run against K8s 1.11 in the release branch as the minimum supported version.
- Rook is now built with go modules in golang 1.13
### Ceph
- Ceph Octopus is now supported, with continued support for Nautilus.
- Ceph Nautilus 14.2.5 is the minimum supported version
- Support for Mimic was removed. Clusters must be upgraded to Nautilus before upgrading to v1.3.
- OSD refactor: drop support for Rook legacy OSD, directory OSD and Filestore OSD. For more details refer to the [corresponding issue](https://github.com/rook/rook/issues/4724).
- OSD on PVC now supports a metadata device, [refer to the cluster on PVC section](Documentation/ceph-cluster-crd.html#dedicated-metatada-device) or the [corresponding issue](https://github.com/rook/rook/issues/3852).
- OSD on PVC now supports PVC expansion, if the size of the underlying block increases the Bluestore main block and the overall storage capacity will grow up.
- Ceph Nautilus 14.2.5 is the minimum supported version
- OSD on PVC doesn't use LVM anymore to configure OSD, but solely relies on the entire block device, done [here](https://github.com/rook/rook/pull/4435).
- Specific devices for OSDs can now be specified using the full udev path (e.g. /dev/disk/by-id/ata-ST4000DM004-XXXX) instead of the device name.
- OSD on PVC CRUSH device storage class can now be changed by setting an annotation "crushDeviceClass" on the "data" volume template. See "cluster-on-pvc.yaml" for example.
@@ -20,6 +24,10 @@
ConfigMap can be used in mix with the already existing Env Vars defined in operator deployment manifest. Precedence will be given to ConfigMap in case of conflicting configurations.
- Rook Ceph cleanupPolicy will clean up the dataDirHostPath only after user confirmation. For more info about CleanUpPolicy [read the design](https://github.com/rook/rook/blob/master/design/ceph/ceph-cluster-cleanup.md) as well as the documentation [cleanupPolicy](Documentation/ceph-cluster-crd.md#cluster-settings)
- Rook monitor, mds and osd now have liveness probe checks on their respective sockets
- Most of the operator's controller implementations are now based on the controller runtime. No impact to user clusters is expected.
- Object stores can be created without specifying pools in the CephObjectStore CR to allow for pool management outside of Rook such as in the dashboard.
- Operator logs are more concise with the Ceph commands only printing at debug level.
- Various improvements to the integration tests including tests for an external cluster and for a cluster running on PVCs.
- Pools can now be configured to inline compress the data using the `compressionMode` parameter. Support added [here](https://github.com/rook/rook/pull/5124)
- Ceph OSDs do not use the host PID, but the PID namespace of the pod (more security). The OSD does not see host running processes anymore.
@@ -35,7 +43,10 @@ ConfigMap can be used in mix with the already existing Env Vars defined in opera
## Breaking Changes
### <Storage Provider>
### Ceph
- Support for Mimic was removed
- Support for filestore (directory-based) OSDs was removed. See the upgrade guide for migration instructions.
- Support for OSDs created in Rook v0.9 or earlier was removed (created with Rook partitions instead of ceph-volume). See the upgrade guide for migration instructions.
### Minio
- The minio operator was removed from Rook
+2 -11
View File
@@ -100,15 +100,6 @@ Integration tests can be run using tests binary `_output/tests/${platform}/integ
_output/tests/${platform}/integration -test.v
```
### Test parameters
In addition to standard go tests parameters, the following custom parameters are available while running tests:
| Parameter | Description | Possible values | Default |
| ------------- | -------------------------------------- | ---------------- | ---------- |
| rook_platform | platform Rook needs to be installed on | kubernetes | kubernetes |
| k8s_version | version of Kubernetes to be installed | v1.10+ | v1.13.1 |
| rook_image | Rook image name to be installed | valid image name | rook/ceph |
### Running Tests with parameters.
#### To run all integration tests run
```
@@ -117,10 +108,10 @@ go test -v -timeout 1800s github.com/rook/rook/tests/integration
#### To run a specific suite (uses regex)
```
go test -v -timeout 1800s -run SmokeSuite github.com/rook/rook/tests/integration
go test -v -timeout 1800s -run CephSmokeSuite github.com/rook/rook/tests/integration
```
#### To run specific tests inside a suite:
```
go test -v -timeout 1800s -run SmokeSuite github.com/rook/rook/tests/integration -testify.m TestRookClusterInstallation_SmokeTest
go test -v -timeout 1800s -run CephSmokeSuite github.com/rook/rook/tests/integration -testify.m TestRookClusterInstallation_SmokeTest
```