feedc0de/my-rook-config
1
0 Fork 0
forked from rook/rook
Code Pull Requests Activity

document the new cluster crd attributes for versioning

Browse Source 
Signed-off-by: travisn <tnielsen@redhat.com>
This commit is contained in:
travisn committed 2018-11-01 11:49:36 -06:00
1 parent ceb47f260d
commit 272f0e9a39
5 files changed
+54 -37
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

Documentation/ceph-cluster-crd.md
+7
View File
@@ -17,6 +17,9 @@ metadata:
name: rook-ceph
namespace: rook-ceph
spec:
cephVersion:
image: ceph/ceph:13.2.2
name: mimic
dataDirHostPath: /var/lib/rook
serviceAccount: rook-ceph-cluster
storage:
@@ -36,6 +39,10 @@ Settings can be specified at the global level to apply to the cluster as a whole
### Cluster Settings
- `cephVersion`: The version information for launching the ceph daemons.
- `image`: The image used for running the ceph daemons. For example, `ceph/ceph:v12.2.7` or `ceph/ceph:v13.2.2`.
- `name`: The name of the major release. Must be `luminous`, `mimic`, or `nautilus`.
- `allowUnsupported`: If `true`, allow an unsupported `name` of the Ceph release. Currently only `luminous` and `mimic` are supported, so `nautilus` would require this to be set to `true`. Should be set to `false` in production.
- `dataDirHostPath`: The path on the host ([hostPath](https://kubernetes.io/docs/concepts/storage/volumes/#hostpath)) where config and data should be stored for each of the services. If the directory does not exist, it will be created. Because this directory persists on the host, it will remain after pods are deleted.
- On **Minikube** environments, use `/data/rook`. Minikube boots into a tmpfs but it provides some [directories](https://github.com/kubernetes/minikube/blob/master/docs/persistent_volumes.md) where files can be persisted across reboots. Using one of these directories will ensure that Rook's data and configuration files are persisted and that enough storage space is available.
- **WARNING**: For test scenarios, if you delete a cluster and start a new cluster on the same hosts, the path used by `dataDirHostPath` must be deleted. Otherwise, stale keys and other config will remain from the previous cluster and the new mons will fail to start.
Documentation/ceph-dashboard.md
+16 -16
View File
@@ -22,19 +22,19 @@ This is the default setting in the example manifests.
enabled: true
```
The Rook operator will enable the ceph-mgr dashboard module to listen on the default port 7000.
The Rook operator will enable the ceph-mgr dashboard module to listen on the default port 8443.
A K8s service will also be created to expose that port inside the cluster.
```bash
kubectl -n rook-ceph get service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
rook-ceph-mgr ClusterIP 10.108.111.192 <none> 9283/TCP 3h
rook-ceph-mgr-dashboard ClusterIP 10.110.113.240 <none> 7000/TCP 3h
rook-ceph-mgr-dashboard ClusterIP 10.110.113.240 <none> 8443/TCP 3h
```
The first service is for reporting the [Prometheus metrics](monitoring.md), while the latter service is for the dashboard.
If you are on a node in the cluster, you will be able to connect to the dashboard by using either the
DNS name of the service at `http://rook-ceph-mgr-dashboard:7000` or by connecting to the cluster IP,
in this example at `http://10.110.113.240:7000`.
DNS name of the service at `https://rook-ceph-mgr-dashboard:8443` or by connecting to the cluster IP,
in this example at `https://10.110.113.240:8443`.
## Viewing the Dashboard External to the Cluster
@@ -46,12 +46,12 @@ You can use an [Ingress Controller](https://kubernetes.io/docs/concepts/services
NodePort, LoadBalancer, or ExternalIPs.
The simplest way to expose the service in minikube or similar environment is using the NodePort to open a port on the
VM that can be accessed by the host. To create a service with the NodePort, save this yaml as `dashboard-external.yaml`:
VM that can be accessed by the host. To create a service with the NodePort, save this yaml as `dashboard-external-https.yaml`:
```yaml
apiVersion: v1
kind: Service
metadata:
name: rook-ceph-mgr-dashboard-external
name: rook-ceph-mgr-dashboard-external-https
namespace: rook-ceph
labels:
app: rook-ceph-mgr
@@ -59,9 +59,9 @@ metadata:
spec:
ports:
- name: dashboard
port: 7000
port: 8443
protocol: TCP
targetPort: 7000
targetPort: 8443
selector:
app: rook-ceph-mgr
rook_cluster: rook-ceph
@@ -71,18 +71,18 @@ spec:
Now create the service:
```bash
$ kubectl create -f dashboard-external.yaml
$ kubectl create -f dashboard-external-https.yaml
```
You will see the new service `rook-ceph-mgr-dashboard-external` created:
```bash
$ kubectl -n rook-ceph get service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
rook-ceph-mgr ClusterIP 10.108.111.192 <none> 9283/TCP 4h
rook-ceph-mgr-dashboard ClusterIP 10.110.113.240 <none> 7000/TCP 4h
rook-ceph-mgr-dashboard-external NodePort 10.101.209.6 <none> 7000:31176/TCP 4h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
rook-ceph-mgr ClusterIP 10.108.111.192 <none> 9283/TCP 4h
rook-ceph-mgr-dashboard ClusterIP 10.110.113.240 <none> 8443/TCP 4h
rook-ceph-mgr-dashboard-external-https NodePort 10.101.209.6 <none> 8443:31176/TCP 4h
```
In this example, port `31176` will be opened to expose port `7000` from the ceph-mgr pod. Find the ip address
of the VM. If using minikube, you can `minikube ssh` to the machine and `ifconfig` to find the ip address.
Now you can enter the URL in your browser such as `http://192.168.99.110:31176` and the dashboard will appear.
In this example, port `31176` will be opened to expose port `8443` from the ceph-mgr pod. Find the ip address
of the VM. If using minikube, you can `minikube ip` to find the ip address accessible to your host.
Now you can enter the URL in your browser such as `http://192.168.99.103:31176` and the dashboard will appear.
Documentation/ceph-quickstart.md
+3
View File
@@ -117,6 +117,9 @@ metadata:
name: rook-ceph
namespace: rook-ceph
spec:
cephVersion:
image: ceph/ceph:v13.2.2
name: mimic
dataDirHostPath: /var/lib/rook
dashboard:
enabled: true
cluster/examples/kubernetes/ceph/cluster.yaml
+2 -2
View File
@@ -57,9 +57,9 @@ metadata:
spec:
cephVersion:
# The container image used to launch the Ceph daemon pods (mon, mgr, osd, mds, rgw)
image: ceph/ceph:v12.2.7
image: ceph/ceph:v13.2.2
# The name of the major release of Ceph. Must be luminous, mimic, or nautilus
name: luminous
name: mimic
# Whether to allow unsupported versions of Ceph. Currently only luminous and mimic are supported.
# After nautilus is released, Rook will be updated to support nautilus.
# Do not set to true in production.
design/decouple-ceph-version.md
+26 -19
View File
@@ -44,7 +44,7 @@ spec:
### Ceph Version
The Ceph version is defined as the property `cephImage` in the Cluster CRD. All Ceph daemon containers launched by the Rook operator will use this image, including the mon, mgr,
The Ceph version is defined under the property `cephVersion` in the Cluster CRD. All Ceph daemon containers launched by the Rook operator will use this image, including the mon, mgr,
osd, rgw, and mds pods. The significance of this approach is that the Rook binary is not included in the daemon containers. All initialization performed by Rook to generate the Ceph config and prepare the daemons must be completed in an [init container](https://github.com/rook/rook/issues/2003). Once the Rook init containers complete their execution, the daemon container will run the Ceph image. The daemon container will no longer have Rook running.
In the following Cluster CRD example, the Ceph version is Mimic 13.2.1.
@@ -56,14 +56,16 @@ metadata:
name: rook-ceph
namespace: rook-ceph
spec:
cephImage: ceph/ceph:v13.2.1
cephVersion:
image: ceph/ceph:v13.2.1
name: mimic
```
### Operator Requirements
The operator needs to run the Ceph client tools to manage the cluster. For example, the `ceph` tool is needed for general Ceph configuration and status, while `radosgw-admin` is required for managing an object store. Therefore, all the necessary client tools will still be included in the Rook image.
The client tools are tested by the Ceph team to be backward and forward compatible by two versions. Therefore, the operator can support a version of Ceph up to two versions older than the client tools it contains.
The client tools are tested by the Ceph team to be backward and forward compatible by two versions. This means the operator can support a version of Ceph up to two versions older than the client tools it contains.
With each Rook release, the tools will be included from the latest release of Ceph. For example, in 0.9 Rook will likely include the Mimic tools. Upgrades would be supported from Luminous to Mimic.
Rook 0.9 can also be tested to support upgrades to Nautilus since they may be released in the same time frame. Since the Ceph tools are forward compatible, the Mimic tools will be sufficient to support upgrading to Nautilus.
If Nautilus is released after Rook 0.9, a patch release can be made to 0.9 so that Rook can officially support the upgrade at that point. The changes in the patch release should be minimal since upgrading to Nautilus could have been mostly planned for in 0.9.
@@ -76,8 +78,8 @@ The operator will be made to understand differences in the Ceph versions that ar
Rook will support a very specific list of major versions. Outside these versions, Rook will not be aware of the needs for configuring and upgrading the cluster.
In v0.9, the supported versions will be:
- Luminous (ceph/ceph:v12.2.x)
- Mimic (ceph/ceph:v13.2.x)
- luminous (ceph/ceph:v12.2.x)
- mimic (ceph/ceph:v13.2.x)
Depending on the timing of the 0.9 and Nautilus releases, Nautilus will likely be supported either in 0.9 or a patch release. Versions not yet officially supported
can be tested with settings in the CRD to be mentioned below.
@@ -89,7 +91,7 @@ various Mimic patch releases.
The flexibility during upgrades will now be improved since the upgrade of Rook will be independent from the upgrade to the Ceph version.
- To upgrade Rook, update the version of the Rook operator container
- To upgrade Ceph, make sure Rook is running the latest release, then update the `cephImage` in the cluster CRD
- To upgrade Ceph, make sure Rook is running the latest release, then update the `cephVersion.image` in the cluster CRD
The versions to be supported during upgrade will be a specific set for each version of Rook. In 0.9, it is anticipated that the only upgrade of Ceph
supported would only be Luminous to Mimic. When Rook officially adds support for a release of Ceph (ie. Nautilus), the upgrade path will also be supported from one previous version.
@@ -117,23 +119,28 @@ To allow more control over the upgrade, we define `upgradePolicy` settings. They
- Allow for testing of future versions that are not officially supported
The settings in the CRD to accommodate the design include:
- `newCephImage`: The version of the image to start applying to the daemons specified in the `upgrade` list. If `newCephImage` is not set, the operator will ensure that all daemons are running the version specified by the `cephImage` setting.
- `upgrade`: A list of daemons that should be upgraded to the version `newCephImage`. The daemons include `mon`, `osd`, `mgr`, `rgw`, and `mds`. The ordering of the list will be ignored as Rook will only support ordering as it determines necessary for a version. If there are special upgrade actions in the future, they could be named and added to this list.
- `allowUnrecognizedVersion`: If `false`, the operator would refuse to upgrade the Ceph version if it doesn't support or recognize that version. If `true`, Rook would go ahead and blindly set the image version and assume the pod specs should match `unrecognizedMajorVersion`. This would allow testing of upgrade to unreleased versions. The default is `false`.
- `unrecognizedMajorVersion`: The version of pod spec to use when orchestrating an unrecognized version. Possible values are the names of versions that Rook understands: `luminous`, `mimic`, or `nautilus`.
- `upgradePolicy.cephVersion`: The version of the image to start applying to the daemons specified in the `components` list.
- `allowUnrecognizedVersion`: If `false`, the operator would refuse to upgrade the Ceph version if it doesn't support or recognize that version. If `true`, Rook would go ahead and blindly set the image version and assume the pod specs should match `unrecognizedMajorVersion`. This would allow testing of upgrade to unreleased versions. The default is `false`.
- `upgradePolicy.components`: A list of daemons or other components that should be upgraded to the version `newCephVersion`. The daemons include `mon`, `osd`, `mgr`, `rgw`, and `mds`. The ordering of the list will be ignored as Rook will only support ordering as it determines necessary for a version. If there are special upgrade actions in the future, they could be named and added to this list.
For example, with the settings below the operator would only upgrade the mons to mimic, while other daemons would remain on luminous. When the admin is ready, he would add more daemons to the list.
```yaml
spec:
cephImage: ceph/ceph:v12.2.7
cephVersion:
image: ceph/ceph:v12.2.7
name: luminous
allowUnrecognizedVersion: false
upgradePolicy:
newCephImage: ceph/ceph:v13.2.1
upgrade:
cephVersion:
image: ceph/ceph:v13.2.1
name: mimic
allowUnrecognizedVersion: false
components:
- mon
```
When the admin is completed with the upgrade or he is ready to allow Rook to complete the full upgrade for all daemons, he would set `cephImage: ceph/ceph:v13.2.1`, and the operator would ignore the `upgradePolicy` since the `cephImage` and `newCephImage` match.
When the admin is completed with the upgrade or he is ready to allow Rook to complete the full upgrade for all daemons, he would set `cephVersion.image: ceph/ceph:v13.2.1`, and the operator would ignore the `upgradePolicy` since the `cephVersion` and `upgradePolicy.cephVersion` match.
If the admin wants to pause or otherwise control the upgrade closely, there are a couple of natural back doors:
- Deleting the operator pod will effectively pause the upgrade. Starting the operator pod up again would resume the upgrade.
@@ -144,15 +151,15 @@ If the admin wants to pause or otherwise control the upgrade closely, there are
If a developer wants to test the upgrade from mimic to nautilus, he would first create the cluster based on mimic. Then he would update the crd with the "unrecognized version" attributes in the CRD to specify nautilus such as:
```yaml
spec:
cephImage: ceph/ceph:v14.1.1
upgradePolicy:
cephVersion:
image: ceph/ceph:v14.1.1
name: nautilus
allowUnrecognizedVersion: true
unrecognizedMajorVersion: nautilus
```
### Default Version
For backward compatibility, if the `cephImage` property is empty, the operator will need to internally set a default version of Ceph.
For backward compatibility, if the `cephVersion` property is not set, the operator will need to internally set a default version of Ceph.
The operator will assume the desired Ceph version is Luminous 12.2.7, which was shipped with Rook v0.8.
This default will allow the Rook upgrade from v0.8 to v0.9 to only impact the Rook version and hold the Ceph version at Luminous.
After the Rook upgrade to v0.9, the user can choose to set the `cephImage` property to some newer version of Ceph such as Mimic.
After the Rook upgrade to v0.9, the user can choose to set the `cephVersion` property to some newer version of Ceph such as mimic.