feedc0de/my-rook-config
1
0
Fork 0
forked from rook/rook
Code Pull Requests Activity

Merge pull request #3732 from SUSE/upgrade-docs

Browse Source 
Ceph: update upgrade docs for v1.1 release
This commit is contained in:
Blaine Gardner
2019-09-03 12:36:43 -06:00
committed by GitHub
parent 1dcaecab0f a4bb5e41af
commit 7eef87fbcb
4 changed files with 619 additions and 107 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Documentation/ceph-upgrade.md
+123 -105
View File
@@ -4,7 +4,7 @@ weight: 3800
indent: true
---
# Ceph Upgrades
# Rook-Ceph Upgrades
This guide will walk you through the steps to upgrade the software in a Rook-Ceph cluster from one
version to the next. This includes both the Rook-Ceph operator software itself as well as the Ceph
cluster software.
@@ -16,10 +16,10 @@ further feedback for improving automation and improving Rook.
We welcome feedback and opening issues!
## Supported Versions
Please refer to the upgrade guides from previous releases for supported upgrade paths.
Rook ugrades are only supported between official releases. Upgrades to and from `master` are not supported.
Rook upgrades are only supported between official releases. Upgrades to and from `master` are not
supported.
For a guide to upgrade previous versions of Rook, please refer to the version of documentation for
those releases.
@@ -29,7 +29,6 @@ those releases.
- [Upgrade 0.6 to 0.7](https://rook.io/docs/rook/v0.7/upgrade.html)
- [Upgrade 0.5 to 0.6](https://rook.io/docs/rook/v0.6/upgrade.html)
## Considerations
With this upgrade guide, there are a few notes to consider:
* **WARNING:** Upgrading a Rook cluster is not without risk. There may be unexpected issues or
@@ -44,23 +43,23 @@ With this upgrade guide, there are a few notes to consider:
## Patch Release Upgrades
Unless otherwise noted due to extenuating requirements, upgrades from one patch release of Rook to
another are as simple as updating the image of the Rook operator. For example, when Rook v1.0.1 is
released, the process of updating from v1.0.0 is as simple as running the following:
another are as simple as updating the image of the Rook operator. For example, when Rook v1.1.1 is
released, the process of updating from v1.1.0 is as simple as running the following:
```
kubectl -n rook-ceph set image deploy/rook-ceph-operator rook-ceph-operator=rook/ceph:v1.0.1
kubectl -n rook-ceph set image deploy/rook-ceph-operator rook-ceph-operator=rook/ceph:v1.1.1
```
## Upgrading from v1.0 to the latest
Rook upgrades to master builds are not supported. When v1.1 is released, the upgrade will be supported
from 1.0 to 1.1. Master builds should only be used for test environments.
## Upgrading from v1.0 to v1.1
**Rook releases from master are expressly unsupported.** It is strongly recommended that you use
[official releases](https://github.com/rook/rook/releases) of Rook, as unreleased versions from the
[official releases](https://github.com/rook/rook/releases) of Rook. Unreleased versions from the
master branch are subject to changes and incompatibilities that will not be supported in the
official releases. Builds from the master branch can have functionality changed and even removed at
any time without compatibility support and without prior notice.
**Users are required to upgrade Ceph to Mimic (v13.2.4 or newer) or Nautilus (v14.2.x) now.** Rook 1.0
was the last Rook release which will support Ceph's Luminous (v12.x.x) version. These are the only
supported major versions of Ceph.
## Prerequisites
We will do all our work in the Ceph example manifests directory.
```sh
@@ -80,13 +79,14 @@ export ROOK_NAMESPACE="rook-ceph"
```
In order to successfully upgrade a Rook cluster, the following prerequisites must be met:
* The cluster must be running Ceph Mimic (v13.2.3 or newer) or Nautilus (v14.2.x) before upgrading
to Rook 1.1; Ceph Luminous (v12.x.x) is no longer supported.
* The cluster should be in a healthy state with full functionality.
Review the [health verification section](#health-verification) in order to verify your cluster is
in a good starting state.
* All pods consuming Rook storage should be created, running, and in a steady state. No Rook
persistent volumes should be in the act of being created or deleted.
## Health Verification
Before we begin the upgrade process, let's first review some ways that you can verify the health of
your cluster, ensuring that the upgrade is going smoothly after each step. Most of the health
@@ -110,41 +110,50 @@ kubectl -n $ROOK_NAMESPACE get pods
The Rook toolbox contains the Ceph tools that can give you status details of the cluster with the
`ceph status` command. Let's look at an output sample and review some of the details:
```sh
> TOOLS_POD=$(kubectl -n $ROOK_NAMESPACE get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}')
> kubectl -n $ROOK_NAMESPACE exec -it $TOOLS_POD -- ceph status
cluster:
id: fe7ae378-dc77-46a1-801b-de05286aa78e
health: HEALTH_OK
services:
mon: 3 daemons, quorum rook-ceph-mon0,rook-ceph-mon1,rook-ceph-mon2
mgr: rook-ceph-mgr0(active)
osd: 1 osds: 1 up, 1 in
data:
pools: 1 pools, 100 pgs
objects: 0 objects, 0 bytes
usage: 2049 MB used, 15466 MB / 17516 MB avail
pgs: 100 active+clean
TOOLS_POD=$(kubectl -n $ROOK_NAMESPACE get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}')
kubectl -n $ROOK_NAMESPACE exec -it $TOOLS_POD -- ceph status
# cluster:
# id: a3f4d647-9538-4aff-9fd1-b845873c3fe9
# health: HEALTH_OK
#
# services:
# mon: 3 daemons, quorum b,c,a
# mgr: a(active)
# mds: myfs-1/1/1 up {0=myfs-a=up:active}, 1 up:standby-replay
# osd: 6 osds: 6 up, 6 in
# rgw: 1 daemon active
#
# data:
# pools: 9 pools, 900 pgs
# objects: 67 objects, 11 KiB
# usage: 6.1 GiB used, 54 GiB / 60 GiB avail
# pgs: 900 active+clean
#
# io:
# client: 7.4 KiB/s rd, 681 B/s wr, 11 op/s rd, 4 op/s wr
# recovery: 164 B/s, 1 objects/s
```
In the output above, note the following indications that the cluster is in a healthy state:
* Cluster health: The overall cluster status is `HEALTH_OK` and there are no warning or error status
messages displayed.
* Monitors (mon): All of the monitors are included in the `quorum` list.
* OSDs (osd): All OSDs are `up` and `in`.
* Manager (mgr): The Ceph manager is in the `active` state.
* OSDs (osd): All OSDs are `up` and `in`.
* Placement groups (pgs): All PGs are in the `active+clean` state.
* (If applicable) Ceph filesystem metadata server (mds): all MDSes are `active` for all filesystems
* (If applicable) Ceph object store RADOS gateways (rgw): all daemons are `active`
If your `ceph status` output has deviations from the general good health described above, there may
be an issue that needs to be investigated further. There are other commands you may run for more
details on the health of the system, such as `ceph osd status`.
details on the health of the system, such as `ceph osd status`. See the
[Ceph troubleshooting docs](https://docs.ceph.com/docs/master/rados/troubleshooting/) for help.
### Container Versions
The container version running in a specific pod in the Rook cluster can be verified in its pod spec
output. For example for the monitor pod `mon-b`, we can verify the container version it is running
with the below commands:
```bash
```sh
POD_NAME=$(kubectl -n $ROOK_NAMESPACE get pod -o custom-columns=name:.metadata.name --no-headers | grep rook-ceph-mon-b)
kubectl -n $ROOK_NAMESPACE get pod ${POD_NAME} -o jsonpath='{.spec.containers[0].image}'
```
@@ -153,19 +162,18 @@ kubectl -n $ROOK_NAMESPACE get pod ${POD_NAME} -o jsonpath='{.spec.containers[0]
The status and container versions for all Rook pods can be collected all at once with the following
commands:
```sh
kubectl -n $ROOK_SYSTEM_NAMESPACE get pod -o jsonpath='{range .items[*]}{.metadata.name}{"\n\t"}{.status.phase}{"\t"}{.spec.containers[0].image}{"\t"}{.spec.initContainers[0]}{"\n"}{end}' && \
kubectl -n $ROOK_NAMESPACE get pod -o jsonpath='{range .items[*]}{.metadata.name}{"\n\t"}{.status.phase}{"\t"}{.spec.containers[0].image}{"\t"}{.spec.initContainers[0].image}{"\n"}{end}'
kubectl -n $ROOK_SYSTEM_NAMESPACE get pod -o jsonpath='{range .items[*]}{.metadata.name}{"\n\t"}{.status.phase}{"\t\t"}{.spec.containers[0].image}{"\t"}{.spec.initContainers[0]}{"\n"}{end}' && \
kubectl -n $ROOK_NAMESPACE get pod -o jsonpath='{range .items[*]}{.metadata.name}{"\n\t"}{.status.phase}{"\t\t"}{.spec.containers[0].image}{"\t"}{.spec.initContainers[0].image}{"\n"}{end}'
```
Rook 1.0 also introduces the `rook-version` label to Ceph controller resources. For various
resource controllers, a summary of the resource controllers can be gained with the commands below.
These will report the requested, updated, and currently available replicas for various Rook-Ceph
resources in addition to the version of Rook for resources managed by the updated Rook-Ceph operator.
The `rook-version` label exists on Ceph controller resources. For various resource controllers, a
summary of the resource controllers can be gained with the commands below. These will report the
requested, updated, and currently available replicas for various Rook-Ceph resources in addition to
the version of Rook for resources managed by the updated Rook-Ceph operator. Note that the operator
and toolbox deployments do not have a `rook-version` label set.
```sh
kubectl -n $ROOK_NAMESPACE get deployments -o jsonpath='{range .items[*]}{.metadata.name}{" \treq/upd/avl: "}{.spec.replicas}{"/"}{.status.updatedReplicas}{"/"}{.status.readyReplicas}{" \trook-version="}{.metadata.labels.rook-version}{"\n"}{end}'
kubectl -n $ROOK_NAMESPACE get daemonsets -o jsonpath='{range .items[*]}{.metadata.name}{" \treq/upd/avl: "}{.status.desiredNumberScheduled}{"/"}{.status.updatedNumberScheduled}{"/"}{.status.numberAvailable}{" \trook-version="}{.metadata.labels.rook-version}{"\n"}{end}'
kubectl -n $ROOK_NAMESPACE get jobs -o jsonpath='{range .items[*]}{.metadata.name}{" \tsucceeded: "}{.status.succeeded}{" \trook-version="}{.metadata.labels.rook-version}{"\n"}{end}'
```
@@ -175,23 +183,18 @@ Any pod that is using a Rook volume should also remain healthy:
* There should be no errors in its logs
* The pod should still be able to read and write to the attached Rook volume.
## Rook Operator Upgrade Process
**This is only a placeholder guide for upgrading from v1.0 to v1.1. This is not final documentation
for the 1.1 release**
In the examples given in this guide, we will be upgrading a live Rook cluster running `v1.0.2` to
the version `v1.1.0`. This upgrade should work from any official patch release of Rook 1.0 to any official
patch release of 1.1. We will further assume that your previous cluster was created using an earlier
version of this guide and manifests. If you have created custom manifests, these steps may not work
as written.
In the examples given in this guide, we will be upgrading a live Rook cluster running `v1.0.6` to
the version `v1.1.0`. This upgrade should work from any official patch release of Rook 1.0 to any
official patch release of 1.1. We will further assume that your previous cluster was created using
an earlier version of this guide and manifests. If you have created custom manifests, these steps
may not work as written.
**Rook release from master are expressly unsupported.** It is strongly recommended that you use
[official releases](https://github.com/rook/rook/releases) of Rook, as unreleased versions from the
[official releases](https://github.com/rook/rook/releases) of Rook. Unreleased versions from the
master branch are subject to changes and incompatibilities that will not be supported in the
official releases. Builds from the master branch can have functionality changed and even removed at
any time without compatibility support and without prior notice.
official releases. Builds from the master branch can have functionality changed or removed at any
time without compatibility support and without prior notice.
Let's get started!
@@ -213,12 +216,13 @@ mv *.bak backups/
```
### 2. Update modified permissions
A few of permissions have been added or modified in v1.1. To make updating these resources easy,
special upgrade manifests have been created.
A few permissions have been added in v1.1. To make updating these resources easy, special upgrade
manifests have been created.
```sh
kubectl delete -f upgrade-from-v1.0-delete.yaml
kubectl create -f upgrade-from-v1.0-create.yaml
kubectl apply -f upgrade-from-v1.0-apply.yaml
```
Upgrade notes have been added to `upgrade-from-v1.0-create.yaml` identifying the changes made to 1.1
to aid users in any manifest-related auditing they wish to do.
@@ -228,40 +232,41 @@ The largest portion of the upgrade is triggered when the operator's image is upd
kubectl -n $ROOK_SYSTEM_NAMESPACE set image deploy/rook-ceph-operator rook-ceph-operator=rook/ceph:v1.1.0
```
Watch now in amazement as the Ceph mons, mgrs, OSDs, rbd-mirrors, mdses and rgws are terminated and
Watch now in amazement as the Ceph mons, mgrs, OSDs, rbd-mirrors, MDSes and RGWs are terminated and
replaced with updated versions in sequence. The cluster may be offline very briefly as mons update,
and the Ceph Filesystem may fall offline a few times while the mdses are upgrading. This is normal.
and the Ceph Filesystem may fall offline a few times while the MDSes are upgrading. This is normal.
Continue on to the next upgrade step while the update is commencing.
### 4. Wait for the upgrade to complete
Before moving on, the Ceph cluster's core (RADOS) components should be fully updated.
Before moving on, the Ceph cluster's core (RADOS) components (i.e., mons, mgrs, and OSDs) must be
fully updated.
```sh
watch --exec kubectl -n $ROOK_NAMESPACE get deployments -o jsonpath='{range .items[*]}{.metadata.name}{" \treq/upd/avl: "}{.spec.replicas}{"/"}{.status.updatedReplicas}{"/"}{.status.readyReplicas}{" \trook-version="}{.metadata.labels.rook-version}{"\n"}{end}'
watch --exec kubectl -n $ROOK_NAMESPACE get deployments -l rook_cluster=$ROOK_NAMESPACE -o jsonpath='{range .items[*]}{.metadata.name}{" \treq/upd/avl: "}{.spec.replicas}{"/"}{.status.updatedReplicas}{"/"}{.status.readyReplicas}{" \trook-version="}{.metadata.labels.rook-version}{"\n"}{end}'
```
As an example, this cluster is midway through updating the OSDs from 1.0 to 0.9. When all
As an example, this cluster is midway through updating the OSDs from 1.0 to 1.1. When all
deployments report `1/1/1` availability and `rook-version=v1.1.0`, the Ceph cluster's core
components are fully updated.
```sh
```
Every 2.0s: kubectl -n rook-ceph get deployment -o j...
rook-ceph-mgr-a req/upd/avl: 1/1/1 rook-version=v1.1.0
rook-ceph-mon-a req/upd/avl: 1/1/1 rook-version=v1.1.0
rook-ceph-mon-b req/upd/avl: 1/1/1 rook-version=v1.1.0
rook-ceph-mon-c req/upd/avl: 1/1/1 rook-version=v1.1.0
rook-ceph-osd-0 req/upd/avl: 1/1/1 rook-version=v1.1.0
rook-ceph-osd-1 req/upd/avl: 1// rook-version=v1.1.0
rook-ceph-osd-2 req/upd/avl: 1/1/1 rook-version=
rook-ceph-osd-0 req/upd/avl: 1// rook-version=v1.1.0
rook-ceph-osd-1 req/upd/avl: 1/1/1 rook-version=v1.0.6
rook-ceph-osd-2 req/upd/avl: 1/1/1 rook-version=v1.0.6
```
The mdses and rgws are the last daemons to update. An easy check to see if the upgrade is totally
The MDSes and RGWs are the last daemons to update. An easy check to see if the upgrade is totally
finished is to check that there is only one `rook-version` reported across the cluster. It is safe
to proceed with the next step before the mdses and rgws are finished updating.
to proceed with the next step before the MDSes and RGWs are finished updating.
```sh
(kubectl -n $ROOK_NAMESPACE get deployment -o jsonpath='{range .items[*]}{"rook-version="}{.metadata.labels.rook-version}{"\n"}{end}' && kubectl -n $ROOK_NAMESPACE get daemonset -o jsonpath='{range .items[*]}{"rook-version="}{.metadata.labels.rook-version}{"\n"}{end}') | uniq
kubectl -n $ROOK_NAMESPACE get deployment -l rook_cluster=$ROOK_NAMESPACE -o jsonpath='{range .items[*]}{"rook-version="}{.metadata.labels.rook-version}{"\n"}{end}' | sort | uniq
# This cluster is not yet finished:
# rook-version=
# rook-version=v1.0.6
# rook-version=v1.1.0
# This cluster is finished:
# rook-version=v1.1.0
@@ -319,21 +324,36 @@ spec:
# ....
```
# Ceph Version Upgrades
Rook 1.0 was the last Rook release which will support Ceph's Luminous (v12.x.x) version. Users are
required to upgrade to Mimic (v13.2.4 or newer) or Nautilus (v14.2.x) now. Rook 1.1 will only run with
Mimic or newer, though running with Octopus requires the `allowUnsupported: true` flag.
### 7. Update Rook-Ceph custom resource definitions
**IMPORTANT: Do not perform this step until ALL existing Rook-Ceph clusters are updated**
**IMPORTANT: This section only applies to clusters running Rook 1.0 or newer**
After all Rook-Ceph clusters have been updated following the steps above, update the Rook-Ceph
Custom Resource Definitions. This is important to get the latest schema validations that will
support running external Ceph clusters and will also help with creating or modifying Rook-Ceph
deployments in the future.
```sh
kubectl apply -f upgrade-from-v1.0-crds.yaml
```
# Ceph Version Upgrades
Rook 1.1 supports Ceph Mimic v13.2.4 or newer and Ceph Nautilus v14.2.0 or newer. These are the only
supported major versions of Ceph.
**IMPORTANT: When an update is requested, the operator will check Ceph's status, if it is in `HEALTH_ERR` it will refuse to do the upgrade.**
Rook is cautious when performing upgrades. When an upgrade is requested (the Ceph image has been updated in the CR), Rook will go through all the daemons one by one and will individually perform checks on them. It will make sure a particular daemon can be stopped before performing the upgrade, once the deployment has been updated, it checks if this is ok to continue. After each daemon is updated we wait for things to settle (monitors to be in a quorum, PGs to be clean for OSDs, up for MDSs, etc.), then only when the condition is met we move to the next daemon. We repeat this process until all the daemons have been updated.
Rook is cautious when performing upgrades. When an upgrade is requested (the Ceph image has been
updated in the CR), Rook will go through all the daemons one by one and will individually perform
checks on them. It will make sure a particular daemon can be stopped before performing the upgrade,
once the deployment has been updated, it checks if this is ok to continue. After each daemon is
updated we wait for things to settle (monitors to be in a quorum, PGs to be clean for OSDs, up for
MDSs, etc.), then only when the condition is met we move to the next daemon. We repeat this process
until all the daemons have been updated.
## Ceph images
Official Ceph container images can be found on [Docker Hub](https://hub.docker.com/r/ceph/ceph/tags/).
These images are tagged in a few ways:
* The most explicit form of tags are full-ceph-version-and-build tags (e.g., `v13.2.2-20181023`).
* The most explicit form of tags are full-ceph-version-and-build tags (e.g., `v13.2.6-20190830`).
These tags are recommended for production clusters, as there is no possibility for the cluster to
be heterogeneous with respect to the version of Ceph running in containers.
* Ceph major version tags (e.g., `v13`) are useful for development and test clusters so that the
@@ -347,47 +367,45 @@ These images are tagged in a few ways:
The majority of the upgrade will be handled by the Rook operator. Begin the upgrade by changing the
Ceph image field in the cluster CRD (`spec:cephVersion:image`).
```sh
NEW_CEPH_IMAGE='ceph/ceph:v14.2.2-20190826'
NEW_CEPH_IMAGE='ceph/ceph:v14.2.2-20190830'
CLUSTER_NAME="$ROOK_NAMESPACE" # change if your cluster name is not the Rook namespace
kubectl -n $ROOK_NAMESPACE patch CephCluster $CLUSTER_NAME --type=merge \
-p "{\"spec\": {\"cephVersion\": {\"image\": \"$NEW_CEPH_IMAGE\"}}}"
```
### 2. Wait for the daemon pod updates to complete
As with upgrading Rook, you must now wait for the upgrade to complete. Determining when the Ceph
version has fully updated is rather simple.
As with upgrading Rook, you must now wait for the upgrade to complete. Status can be determined in a
similar way to the Rook upgrade as well.
```sh
watch kubectl -n $ROOK_NAMESPACE describe pods | grep "Image:.*ceph/ceph" | sort | uniq
watch --exec kubectl -n $ROOK_NAMESPACE get deployments -l rook_cluster=$ROOK_NAMESPACE -o jsonpath='{range .items[*]}{.metadata.name}{" \treq/upd/avl: "}{.spec.replicas}{"/"}{.status.updatedReplicas}{"/"}{.status.readyReplicas}{" \tceph-version="}{.metadata.labels.ceph-version}{"\n"}{end}'
```
Determining when the Ceph has fully updated is rather simple.
```sh
kubectl -n $ROOK_NAMESPACE get deployment -l rook_cluster=$ROOK_NAMESPACE -o jsonpath='{range .items[*]}{"ceph-version="}{.metadata.labels.ceph-version}{"\n"}{end}' | sort | uniq
# This cluster is not yet finished:
# Image: ceph/ceph:v13.2.2-20181023
# Image: ceph/ceph:v14.2.2-20190826
# This cluster is also finished:
# Image: ceph/ceph:v14.2.2-20190826
# ceph-version=13.2.6
# ceph-version=14.2.2
# This cluster is finished:
# ceph-version=14.2.2
```
### 3. Enable Nautilus features
When upgrading to Nautilus (v14.x.x), a final command should be issued to Ceph to take advantage of
the latest Ceph features. This does not apply for upgrades to Mimic (v13.x.x)
```sh
TOOLS_POD=$(kubectl -n $ROOK_NAMESPACE get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}')
kubectl -n $ROOK_NAMESPACE exec -it $TOOLS_POD -- ceph osd require-osd-release nautilus
```
### 4. Verify the updated cluster
### 3. Verify the updated cluster
Verify the Ceph cluster's health using the [health verification section](#health-verification).
If you see a health warning about enabling msgr2, please see the section in the 1.0 guide on
[Updating the Mon Ports](https://rook.io/docs/rook/v1.0/ceph-upgrade.html#6-update-the-mon-ports).
```
[root@minikube /]# ceph -s
cluster:
id: b02807da-986a-40b0-ab7a-fa57582b1e4f
health: HEALTH_WARN
3 monitors have not enabled msgr2
If you see a health warning about enabling msgr2, please see the section in the Rook v1.0 guide on
[updating the mon ports](https://rook.io/docs/rook/v1.0/ceph-upgrade.html#6-update-the-mon-ports).
```sh
TOOLS_POD=$(kubectl -n $ROOK_NAMESPACE get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}')
kubectl -n $ROOK_NAMESPACE exec -it $TOOLS_POD -- ceph status
# cluster:
# id: b02807da-986a-40b0-ab7a-fa57582b1e4f
# health: HEALTH_WARN
# 3 monitors have not enabled msgr2
```
Alternatively, this warning can suppressed if a temporary workaround is needed.
```
ceph config set global mon_warn_on_msgr2_not_enabled false
```sh
TOOLS_POD=$(kubectl -n $ROOK_NAMESPACE get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}')
kubectl -n $ROOK_NAMESPACE exec -it $TOOLS_POD -- ceph config set global mon_warn_on_msgr2_not_enabled false
```
cluster/examples/kubernetes/ceph/upgrade-from-v1.0-crds.yaml
+405
View File
@@ -0,0 +1,405 @@
# Latest custom resource definitions with schema validations
# Do NOT apply this until after all Rook-Ceph clusters have been updated
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: cephclusters.ceph.rook.io
spec:
group: ceph.rook.io
names:
kind: CephCluster
listKind: CephClusterList
plural: cephclusters
singular: cephcluster
scope: Namespaced
version: v1
validation:
openAPIV3Schema:
properties:
spec:
properties:
annotations: {}
cephVersion:
properties:
allowUnsupported:
type: boolean
image:
type: string
dashboard:
properties:
enabled:
type: boolean
urlPrefix:
type: string
port:
type: integer
minimum: 0
maximum: 65535
ssl:
type: boolean
dataDirHostPath:
pattern: ^/(\S+)
type: string
mon:
properties:
allowMultiplePerNode:
type: boolean
count:
maximum: 9
minimum: 0
type: integer
network:
properties:
hostNetwork:
type: boolean
storage:
properties:
useAllNodes:
type: boolean
nodes:
items:
properties:
name:
type: string
config:
properties:
metadataDevice:
type: string
storeType:
type: string
pattern: ^(filestore|bluestore)$
databaseSizeMB:
type: string
walSizeMB:
type: string
journalSizeMB:
type: string
osdsPerDevice:
type: string
encryptedDevice:
type: string
pattern: ^(true|false)$
useAllDevices:
type: boolean
deviceFilter: {}
directories:
type: array
items:
properties:
path:
type: string
devices:
type: array
items:
properties:
name:
type: string
config: {}
location: {}
resources: {}
type: array
useAllDevices:
type: boolean
deviceFilter: {}
location: {}
directories:
type: array
items:
properties:
path:
type: string
config: {}
topologyAware:
type: boolean
monitoring:
properties:
enabled:
type: boolean
rulesNamespace:
type: string
rbdMirroring:
properties:
workers:
type: integer
placement: {}
resources: {}
configOverrides:
items:
properties:
who:
type: string
option:
type: string
value:
type: string
additionalPrinterColumns:
- name: DataDirHostPath
type: string
description: Directory used on the K8s nodes
JSONPath: .spec.dataDirHostPath
- name: MonCount
type: string
description: Number of MONs
JSONPath: .spec.mon.count
- name: Age
type: date
JSONPath: .metadata.creationTimestamp
- name: State
type: string
description: Current State
JSONPath: .status.state
- name: Health
type: string
description: Ceph Health
JSONPath: .status.ceph.health
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: cephfilesystems.ceph.rook.io
spec:
group: ceph.rook.io
names:
kind: CephFilesystem
listKind: CephFilesystemList
plural: cephfilesystems
singular: cephfilesystem
scope: Namespaced
version: v1
validation:
openAPIV3Schema:
properties:
spec:
properties:
metadataServer:
properties:
activeCount:
type: integer
activeStandby:
type: boolean
annotations: {}
placement: {}
resources: {}
metadataPool:
properties:
failureDomain:
type: string
replicated:
properties:
size:
type: integer
erasureCoded:
properties:
dataChunks:
type: integer
codingChunks:
type: integer
dataPools:
type: array
items:
properties:
failureDomain:
type: string
replicated:
properties:
size:
type: integer
erasureCoded:
properties:
dataChunks:
type: integer
codingChunks:
type: integer
additionalPrinterColumns:
- name: MdsCount
type: string
description: Number of MDSs
JSONPath: .spec.metadataServer.activeCount
- name: Age
type: date
JSONPath: .metadata.creationTimestamp
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: cephnfses.ceph.rook.io
spec:
group: ceph.rook.io
names:
kind: CephNFS
listKind: CephNFSList
plural: cephnfses
singular: cephnfs
shortNames:
- nfs
scope: Namespaced
version: v1
validation:
openAPIV3Schema:
properties:
spec:
properties:
rados:
properties:
pool:
type: string
namespace:
type: string
server:
properties:
active:
type: integer
annotations: {}
placement: {}
resources: {}
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: cephobjectstores.ceph.rook.io
spec:
group: ceph.rook.io
names:
kind: CephObjectStore
listKind: CephObjectStoreList
plural: cephobjectstores
singular: cephobjectstore
scope: Namespaced
version: v1
validation:
openAPIV3Schema:
properties:
spec:
properties:
gateway:
properties:
type:
type: string
sslCertificateRef: {}
port:
type: integer
securePort: {}
instances:
type: integer
annotations: {}
placement: {}
resources: {}
metadataPool:
properties:
failureDomain:
type: string
replicated:
properties:
size:
type: integer
erasureCoded:
properties:
dataChunks:
type: integer
codingChunks:
type: integer
dataPool:
properties:
failureDomain:
type: string
replicated:
properties:
size:
type: integer
erasureCoded:
properties:
dataChunks:
type: integer
codingChunks:
type: integer
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: cephobjectstoreusers.ceph.rook.io
spec:
group: ceph.rook.io
names:
kind: CephObjectStoreUser
listKind: CephObjectStoreUserList
plural: cephobjectstoreusers
singular: cephobjectstoreuser
scope: Namespaced
version: v1
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: cephblockpools.ceph.rook.io
spec:
group: ceph.rook.io
names:
kind: CephBlockPool
listKind: CephBlockPoolList
plural: cephblockpools
singular: cephblockpool
scope: Namespaced
version: v1
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: volumes.rook.io
spec:
group: rook.io
names:
kind: Volume
listKind: VolumeList
plural: volumes
singular: volume
shortNames:
- rv
scope: Namespaced
version: v1alpha2
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: objectbuckets.objectbucket.io
spec:
group: objectbucket.io
versions:
- name: v1alpha1
served: true
storage: true
names:
kind: ObjectBucket
listKind: ObjectBucketList
plural: objectbuckets
singular: objectbucket
shortNames:
- ob
- obs
scope: Cluster
subresources:
status: {}
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: objectbucketclaims.objectbucket.io
spec:
versions:
- name: v1alpha1
served: true
storage: true
group: objectbucket.io
names:
kind: ObjectBucketClaim
listKind: ObjectBucketClaimList
plural: objectbucketclaims
singular: objectbucketclaim
shortNames:
- obc
- obcs
scope: Namespaced
subresources:
status: {}
---
cluster/examples/kubernetes/ceph/upgrade-from-v1.0-create.yaml
+91 -1
View File
@@ -1,4 +1,4 @@
# Service account, role, and role binding to allow cmd-reporter job to run in ceph cluster namespace
# Service account, role, and role binding to allow cmd-reporter job to run in Ceph cluster namespace
---
apiVersion: v1
kind: ServiceAccount
@@ -38,4 +38,94 @@ subjects:
- kind: ServiceAccount
name: rook-ceph-cmd-reporter
namespace: rook-ceph
# CRDs, role, and binding for Object Buckets
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: objectbuckets.objectbucket.io
spec:
group: objectbucket.io
versions:
- name: v1alpha1
served: true
storage: true
names:
kind: ObjectBucket
listKind: ObjectBucketList
plural: objectbuckets
singular: objectbucket
shortNames:
- ob
- obs
scope: Cluster
subresources:
status: {}
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: objectbucketclaims.objectbucket.io
spec:
versions:
- name: v1alpha1
served: true
storage: true
group: objectbucket.io
names:
kind: ObjectBucketClaim
listKind: ObjectBucketClaimList
plural: objectbucketclaims
singular: objectbucketclaim
shortNames:
- obc
- obcs
scope: Namespaced
subresources:
status: {}
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: rook-ceph-object-bucket
labels:
operator: rook
storage-backend: ceph
rbac.ceph.rook.io/aggregate-to-rook-ceph-mgr-cluster: "true"
rules:
- apiGroups:
- ""
verbs:
- "*"
resources:
- secrets
- configmaps
- apiGroups:
- storage.k8s.io
resources:
- storageclasses
verbs:
- get
- list
- watch
- apiGroups:
- "objectbucket.io"
verbs:
- "*"
resources:
- "*"
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: rook-ceph-object-bucket
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: rook-ceph-object-bucket
subjects:
- kind: ServiceAccount
name: rook-ceph-system
namespace: rook-ceph
cluster/examples/kubernetes/ceph/upgrade-from-v1.0-delete.yaml
-1
View File
@@ -1 +0,0 @@
---