docs: update docs for Rook v1.3 release
Various updates are needed for the v1.3 release. - The pending release notes were missing some features - Added a section to the upgrade guide for breaking changes to OSDs - Clarifications around Ceph prereqs and min version - Other misc clarifications Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
This commit is contained in:
7 files changed
+71
-25
No files matched your search
@@ -6,6 +6,13 @@ indent: true
|
||||
|
||||
# Ceph Prerequisites
|
||||
|
||||
To make sure you have a Kubernetes cluster that is ready for `Rook`, review the general [Rook Prerequisites](k8s-pre-reqs.md).
|
||||
|
||||
In order to configure the Ceph storage cluster, at least one of these local storage options are required:
|
||||
- Raw devices (no partitions or formatted filesystems)
|
||||
- Raw partitions (no formatted filesystem)
|
||||
- PVs available from a storage class in `block` mode
|
||||
|
||||
## LVM package
|
||||
|
||||
Ceph OSDs have a dependency on LVM in the following scenarios:
|
||||
|
||||
@@ -14,15 +14,15 @@ from other pods running in your cluster.
|
||||
|
||||
## Minimum Version
|
||||
|
||||
Kubernetes **v1.10** or higher is supported by Rook.
|
||||
Kubernetes **v1.11** or higher is supported by Rook.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
To make sure you have a Kubernetes cluster that is ready for `Rook`, you can [follow these instructions](k8s-pre-reqs.md).
|
||||
|
||||
In order to configure the storage cluster, at least one of these local storage options are required:
|
||||
In order to configure the Ceph storage cluster, at least one of these local storage options are required:
|
||||
- Raw devices (no partitions or formatted filesystems)
|
||||
- Raw partitions (no formatted filsystem)
|
||||
- Raw partitions (no formatted filesystem)
|
||||
- PVs available from a storage class in `block` mode
|
||||
|
||||
## TL;DR
|
||||
|
||||
@@ -66,6 +66,8 @@ In the future this step will not be necessary when we build on the K8s local sto
|
||||
|
||||
If you modified the demo settings, additional cleanup is up to you for devices, host paths, etc.
|
||||
|
||||
### Zapping Devices
|
||||
|
||||
Disks on nodes used by Rook for osds can be reset to a usable state with the following methods:
|
||||
|
||||
```console
|
||||
@@ -74,6 +76,7 @@ DISK="/dev/sdb"
|
||||
# Zap the disk to a fresh, usable state (zap-all is important, b/c MBR has to be clean)
|
||||
# You will have to run this step for all disks.
|
||||
sgdisk --zap-all $DISK
|
||||
dd if=/dev/zero of="$DISK" bs=1M count=100 oflag=direct,dsync
|
||||
|
||||
# These steps only have to be run once on each node
|
||||
# If rook sets up osds using ceph-volume, teardown leaves some devices mapped that lock the disks.
|
||||
|
||||
@@ -45,6 +45,38 @@ With this upgrade guide, there are a few notes to consider:
|
||||
both Rook operator updates and for Ceph version updates.
|
||||
* We recommend that you read this document in full before you undertake a Rook cluster upgrade.
|
||||
|
||||
## Before you Upgrade
|
||||
|
||||
Rook v1.3 has several breaking changes that must be considered **before** upgrading to Rook v1.3.
|
||||
1. Ceph Mimic is no longer supported. You must update to Ceph Nautilus v14.2.5 or newer **before** updating to Rook v1.3. See the
|
||||
Rook v1.2 [Upgrade Guide for Ceph](https://rook.github.io/docs/rook/v1.2/ceph-upgrade.html#ceph-version-upgrades).
|
||||
2. Directory-based OSDs are no longer supported. If you are specifying a
|
||||
[`directory`](https://github.com/rook/rook/blob/release-1.2/cluster/examples/kubernetes/ceph/cluster-test.yaml#L49-L50)
|
||||
in your CephCluster CR you will need to convert these OSDs to device-based OSDs. See the next section.
|
||||
3. OSDs created **before Rook v0.9** are no longer supported during upgrade. To detect if you have these legacy
|
||||
OSDs in your cluster, run `lsblk` on each host to see what partitions exist.
|
||||
- If you see **three partitions** on a device running an OSD, this is a legacy OSD. See below for instructions to convert them.
|
||||
- If you see a single partition or LV with the `ceph` prefix, the OSD is **not** a legacy OSD and you will not need to convert the OSDs.
|
||||
|
||||
### Converting Legacy OSDs
|
||||
|
||||
If you have legacy OSDs as described in the previous section, it is recommended to migrate the OSDs before the upgrade,
|
||||
although the migration is not strictly required before upgrade. If you proceed with the upgrade to v1.3 the OSDs will
|
||||
continue running. However, they will no longer be managed by Rook. The OSDs will never again be updated when you update
|
||||
Ceph or any other settings in your CephCluster CR.
|
||||
|
||||
The procedure to migrate the OSDs requires raw devices or partitions be available as mentioned in the [Ceph Prerequisites](ceph-prerequisites.md).
|
||||
After you have available storage devices available in your cluster, do the following:
|
||||
1. Configure OSDs on the new devices using the [Storage selection settings](https://rook.github.io/docs/rook/v1.3/ceph-cluster-crd.html#storage-selection-settings).
|
||||
If you have specified `useAllDevices: true`, you may need to restart the operator in order to trigger creation of new OSDs on new devices.
|
||||
2. Follow the [OSD Removal Guide](https://rook.github.io/docs/rook/v1.3/ceph-osd-mgmt.html#remove-an-osd) to remove the legacy OSDs.
|
||||
- Before removing each OSD, ensure the PGs are all `active+clean` before continuing with the next OSD to ensure your data is safe.
|
||||
|
||||
If you need to re-use your devices and add OSDs to the same devices, you'll need to zap a device before creating a new
|
||||
OSD on the same device. For steps on zapping the device see the [Rook cleanup instructions](ceph-teardown.md#zapping-devices).
|
||||
|
||||
> **Removing OSDs and zapping devices is destructive so proceed with extreme caution**
|
||||
|
||||
## Upgrading the Rook-Ceph Operator
|
||||
|
||||
## Patch Release Upgrades
|
||||
@@ -232,13 +264,15 @@ time without compatibility support and without prior notice.
|
||||
|
||||
Let's get started!
|
||||
|
||||
## 0. Migrate Legacy OSDs
|
||||
|
||||
As described above in the [Before You Upgrade](#before-you-upgrade) section, you must migrate legacy OSDs
|
||||
before upgrading to Rook v1.3.
|
||||
|
||||
## 1. Update Ceph to Nautilus version 14.2.5 or higher
|
||||
|
||||
Rook v1.3 supports Ceph Nautilus v14.2.5 or newer and Ceph Octopus v15.2.0 or newer. These are the
|
||||
only supported major versions of Ceph. Rook v1.2 does not support Ceph Octopus. Therefore, if your
|
||||
Rook v1.2 cluster must be updated to Ceph Nautilus v14.2.5 or higher before upgrading the Rook
|
||||
Operator to v1.3. If an upgrade is necessary, follow the Ceph upgrade steps from Rook's v1.2 release
|
||||
[here](https://rook.io/docs/rook/v1.2/ceph-upgrade.html#ceph-version-upgrades).
|
||||
As described above in the [Before You Upgrade](#before-you-upgrade) section, you must upgrade to
|
||||
Nautilus version 14.2.5 or higher before upgrading to Rook v1.3.
|
||||
|
||||
|
||||
## 2. Update the RBAC and CRDs
|
||||
@@ -255,7 +289,7 @@ kubectl apply -f upgrade-from-v1.2-apply.yaml -f upgrade-from-v1.2-crds.yaml
|
||||
|
||||
If you have specified custom CSI images in the Rook-Ceph Operator deployment, you should update your
|
||||
the deployment to use the latest Ceph-CSI v2.0 and related drivers. If you have not specified custom
|
||||
CSI images in the Operator deployment this step is unnecessary; Rook v1.3 will use the latest
|
||||
CSI images in the Operator deployment this step is unnecessary since Rook v1.3 will use the latest
|
||||
drivers by default.
|
||||
|
||||
See the section [CSI Updates](#csi-updates) for details about how to do this.
|
||||
|
||||
@@ -14,7 +14,7 @@ This chart bootstraps a [rook-ceph-operator](https://github.com/rook/rook) deplo
|
||||
|
||||
## Prerequisites
|
||||
|
||||
* Kubernetes 1.10+
|
||||
* Kubernetes 1.11+
|
||||
|
||||
### RBAC
|
||||
|
||||
@@ -130,7 +130,7 @@ The following tables lists the configurable parameters of the rook-operator char
|
||||
| `csi.rbdLivenessMetricsPort` | Ceph CSI RBD driver metrics port. | `8080` |
|
||||
| `csi.enableSnapshotter` | Enable deployment of snapshotter container in ceph-csi provisioner. | `true` |
|
||||
| `csi.forceCephFSKernelClient` | Enable Ceph Kernel clients on kernel < 4.17 which support quotas for Cephfs. | `true` |
|
||||
| `csi.allowUnsupportedVersion` | Allow to run unsupported and untested CSI image. | `false` |
|
||||
| `csi.allowUnsupportedVersion` | Allow to run unsupported and untested CSI image. | `false` |
|
||||
| `csi.kubeletDirPath` | Kubelet root directory path (if the Kubelet uses a different path for the `--root-dir` flag) | `/var/lib/kubelet` |
|
||||
| `csi.cephcsi.image` | Ceph CSI image. | `quay.io/cephcsi/cephcsi:v2.0.0` |
|
||||
| `csi.rbdPluginUpdateStrategy` | CSI Rbd plugin daemonset update strategy, supported values are OnDelete and RollingUpdate. | `OnDelete` |
|
||||
|
||||
+14
-3
@@ -4,14 +4,18 @@
|
||||
|
||||
## Notable Features
|
||||
|
||||
- Added K8s 1.17 to the test matrix and removed K8s 1.12 from the test matrix.
|
||||
- Added K8s 1.17 and K8s 1.18 to the test matrix and removed K8s 1.12 from the test matrix.
|
||||
- Tests are still run against K8s 1.11 in the release branch as the minimum supported version.
|
||||
- Rook is now built with go modules in golang 1.13
|
||||
|
||||
### Ceph
|
||||
|
||||
- Ceph Octopus is now supported, with continued support for Nautilus.
|
||||
- Ceph Nautilus 14.2.5 is the minimum supported version
|
||||
- Support for Mimic was removed. Clusters must be upgraded to Nautilus before upgrading to v1.3.
|
||||
- OSD refactor: drop support for Rook legacy OSD, directory OSD and Filestore OSD. For more details refer to the [corresponding issue](https://github.com/rook/rook/issues/4724).
|
||||
- OSD on PVC now supports a metadata device, [refer to the cluster on PVC section](Documentation/ceph-cluster-crd.html#dedicated-metatada-device) or the [corresponding issue](https://github.com/rook/rook/issues/3852).
|
||||
- OSD on PVC now supports PVC expansion, if the size of the underlying block increases the Bluestore main block and the overall storage capacity will grow up.
|
||||
- Ceph Nautilus 14.2.5 is the minimum supported version
|
||||
- OSD on PVC doesn't use LVM anymore to configure OSD, but solely relies on the entire block device, done [here](https://github.com/rook/rook/pull/4435).
|
||||
- Specific devices for OSDs can now be specified using the full udev path (e.g. /dev/disk/by-id/ata-ST4000DM004-XXXX) instead of the device name.
|
||||
- OSD on PVC CRUSH device storage class can now be changed by setting an annotation "crushDeviceClass" on the "data" volume template. See "cluster-on-pvc.yaml" for example.
|
||||
@@ -20,6 +24,10 @@
|
||||
ConfigMap can be used in mix with the already existing Env Vars defined in operator deployment manifest. Precedence will be given to ConfigMap in case of conflicting configurations.
|
||||
- Rook Ceph cleanupPolicy will clean up the dataDirHostPath only after user confirmation. For more info about CleanUpPolicy [read the design](https://github.com/rook/rook/blob/master/design/ceph/ceph-cluster-cleanup.md) as well as the documentation [cleanupPolicy](Documentation/ceph-cluster-crd.md#cluster-settings)
|
||||
- Rook monitor, mds and osd now have liveness probe checks on their respective sockets
|
||||
- Most of the operator's controller implementations are now based on the controller runtime. No impact to user clusters is expected.
|
||||
- Object stores can be created without specifying pools in the CephObjectStore CR to allow for pool management outside of Rook such as in the dashboard.
|
||||
- Operator logs are more concise with the Ceph commands only printing at debug level.
|
||||
- Various improvements to the integration tests including tests for an external cluster and for a cluster running on PVCs.
|
||||
- Pools can now be configured to inline compress the data using the `compressionMode` parameter. Support added [here](https://github.com/rook/rook/pull/5124)
|
||||
- Ceph OSDs do not use the host PID, but the PID namespace of the pod (more security). The OSD does not see host running processes anymore.
|
||||
|
||||
@@ -35,7 +43,10 @@ ConfigMap can be used in mix with the already existing Env Vars defined in opera
|
||||
|
||||
## Breaking Changes
|
||||
|
||||
### <Storage Provider>
|
||||
### Ceph
|
||||
- Support for Mimic was removed
|
||||
- Support for filestore (directory-based) OSDs was removed. See the upgrade guide for migration instructions.
|
||||
- Support for OSDs created in Rook v0.9 or earlier was removed (created with Rook partitions instead of ceph-volume). See the upgrade guide for migration instructions.
|
||||
|
||||
### Minio
|
||||
- The minio operator was removed from Rook
|
||||
|
||||
+2
-11
@@ -100,15 +100,6 @@ Integration tests can be run using tests binary `_output/tests/${platform}/integ
|
||||
_output/tests/${platform}/integration -test.v
|
||||
```
|
||||
|
||||
### Test parameters
|
||||
In addition to standard go tests parameters, the following custom parameters are available while running tests:
|
||||
|
||||
| Parameter | Description | Possible values | Default |
|
||||
| ------------- | -------------------------------------- | ---------------- | ---------- |
|
||||
| rook_platform | platform Rook needs to be installed on | kubernetes | kubernetes |
|
||||
| k8s_version | version of Kubernetes to be installed | v1.10+ | v1.13.1 |
|
||||
| rook_image | Rook image name to be installed | valid image name | rook/ceph |
|
||||
|
||||
### Running Tests with parameters.
|
||||
#### To run all integration tests run
|
||||
```
|
||||
@@ -117,10 +108,10 @@ go test -v -timeout 1800s github.com/rook/rook/tests/integration
|
||||
|
||||
#### To run a specific suite (uses regex)
|
||||
```
|
||||
go test -v -timeout 1800s -run SmokeSuite github.com/rook/rook/tests/integration
|
||||
go test -v -timeout 1800s -run CephSmokeSuite github.com/rook/rook/tests/integration
|
||||
```
|
||||
|
||||
#### To run specific tests inside a suite:
|
||||
```
|
||||
go test -v -timeout 1800s -run SmokeSuite github.com/rook/rook/tests/integration -testify.m TestRookClusterInstallation_SmokeTest
|
||||
go test -v -timeout 1800s -run CephSmokeSuite github.com/rook/rook/tests/integration -testify.m TestRookClusterInstallation_SmokeTest
|
||||
```
|
||||
Reference in new issue
Block a user