docs: Add Clarity to Ceph CRDs
- Improve readability of 'Replicated' and 'Erasure Coded' sections - Clarify replicated sample requires 1 OSD located on 3 different nodes - Add whitespace to 'Pool Settings' section - Add link to the k8s 'StorageClass' concept - Rename instances of 'object store' as literal `CephObjectStore` - Rename instances of 'object store user' as literal `CephObjectStoreUser` - Other minor verbiage and syntax clean-up Signed-off-by: Michael Fritch <mfritch@suse.com>
This commit is contained in:
5 files changed
+72
-54
No files matched your search
@@ -12,22 +12,23 @@ indent: true
|
||||
{% endif %}
|
||||
# Block Storage
|
||||
|
||||
Block storage allows you to mount storage to a single pod. This example shows how to build a simple, multi-tier web application on Kubernetes using persistent volumes enabled by Rook.
|
||||
Block storage allows a single pod to mount storage. This guide shows how to create a simple, multi-tier web application on Kubernetes using persistent volumes enabled by Rook.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
This guide assumes you have created a Rook cluster as explained in the main [Quickstart](ceph-quickstart.md) guide.
|
||||
This guide assumes a Rook cluster as explained in the [Quickstart](ceph-quickstart.md).
|
||||
|
||||
## Provision Storage
|
||||
|
||||
Before Rook can start provisioning storage, a StorageClass and its storage pool need to be created. This is needed for Kubernetes to interoperate with Rook for provisioning persistent volumes. For more options on pools, see the documentation on [creating storage pools](ceph-pool-crd.md).
|
||||
Before Rook can provision storage, a [`StorageClass`](https://kubernetes.io/docs/concepts/storage/storage-classes) and [`CephBlockPool`](ceph-pool-crd.md) need to be created. This will allow Kubernetes to interoperate with Rook when provisioning persistent volumes.
|
||||
|
||||
**NOTE** This example requires you to have **at least 3 OSDs each on a different node**.
|
||||
This is because the `replicated.size: 3` will require at least 3 OSDs and as [`failureDomain` setting](ceph-pool-crd.md#spec) to `host` (default), each OSD needs to be on a different node.
|
||||
**NOTE:** This sample requires *at least 1 OSD per node*, with each OSD located on *3 different nodes*.
|
||||
|
||||
Each OSD must be located on a different node, because the [`failureDomain`](ceph-pool-crd.md#spec) is set to `host` and the `replicated.size` is set to `3`.
|
||||
|
||||
**NOTE** This example uses the CSI driver, which is the preferred driver going forward for K8s 1.13 and newer. Examples are found in the [CSI RBD](https://github.com/rook/rook/tree/{{ branchName }}/cluster/examples/kubernetes/ceph/csi/rbd) directory. For an example of a storage class using the flex driver (required for K8s 1.12 or earlier), see the [Flex Driver](#flex-driver) section below, which has examples in the [flex](https://github.com/rook/rook/tree/{{ branchName }}/cluster/examples/kubernetes/ceph/flex) directory.
|
||||
|
||||
Save this storage class definition as `storageclass.yaml`:
|
||||
Save this `StorageClass` definition as `storageclass.yaml`:
|
||||
|
||||
```yaml
|
||||
apiVersion: ceph.rook.io/v1
|
||||
@@ -64,7 +65,7 @@ Create the storage class.
|
||||
kubectl create -f cluster/examples/kubernetes/ceph/csi/rbd/storageclass.yaml
|
||||
```
|
||||
|
||||
**NOTE** As [specified by Kubernetes](https://v1-13.docs.kubernetes.io/docs/concepts/storage/persistent-volumes/#retain), when using the `Retain` reclaim policy, the ceph RBD images that back up `PersistentVolume`s will continue to exist even after the PV is deleted, and have to be cleaned up manually using `rbd rm`.
|
||||
**NOTE** As [specified by Kubernetes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#retain), when using the `Retain` reclaim policy, any Ceph RBD image that is backed by a `PersistentVolume` will continue to exist even after the `PersistentVolume` has been deleted. These Ceph RBD images will need to be cleaned up manually using `rbd rm`.
|
||||
|
||||
## Consume the storage: Wordpress sample
|
||||
|
||||
@@ -167,8 +168,9 @@ To be able to use an erasure coded pool you need to create two pools (as seen be
|
||||
The replicated pool must be specified as the `blockPool` parameter. It is used for the metadata of the RBD images.
|
||||
The erasure coded pool must be set as the `dataBlockPool` parameter below. It is used for the data of the RBD images.
|
||||
|
||||
**NOTE** This example requires you to have **at least 3 bluestore OSDs each on a different node**.
|
||||
This is because the below `erasureCoded` chunk settings require at least 3 bluestore OSDs and as [`failureDomain` setting](ceph-pool-crd.md#spec) to `host` (default), each OSD needs to be on a different nodes.
|
||||
**NOTE:** This example requires *at least 3 bluestore OSDs*, with each OSD located on a *different node*.
|
||||
|
||||
The OSDs must be located on different nodes, because the [`failureDomain`](ceph-pool-crd.md#spec) is set to `host` and the `erasureCoded` chunk settings require at least 3 different OSDs (2 `dataChunks` + 1 `codingChunks`).
|
||||
|
||||
```yaml
|
||||
apiVersion: ceph.rook.io/v1
|
||||
|
||||
@@ -12,16 +12,17 @@ indent: true
|
||||
{% endif %}
|
||||
# Ceph Shared File System CRD
|
||||
|
||||
Rook allows creation and customization of shared file systems through the custom resource definitions (CRDs). The following settings are available
|
||||
for Ceph file systems.
|
||||
Rook allows creation and customization of shared file systems through the custom resource definitions (CRDs). The following settings are available for Ceph file systems.
|
||||
|
||||
## Samples
|
||||
|
||||
### Replicated
|
||||
|
||||
**NOTE** This example requires you to have **at least 3 OSDs each on a different node**.
|
||||
This is because the `replicated.size: 3` (in both defined Pools) will require at least 3 OSDs and as [`failureDomain` setting](ceph-pool-crd.md#spec) to `host` (default), each OSD needs to be on a different nodes.
|
||||
In case you added another location type to your nodes in the [Storage Selection Settings](ceph-cluster-crd.md#storage-selection-settings) (e.g. `rack`), you can also specify this type as your failure domain.
|
||||
**NOTE:** This sample requires *at least 1 OSD per node*, with each OSD located on *3 different nodes*.
|
||||
|
||||
Each OSD must be located on a different node, because both of the defined pools set the [`failureDomain`](ceph-pool-crd.md#spec) to `host` and the `replicated.size` to `3`.
|
||||
|
||||
The `failureDomain` can also be set to another location type (e.g. `rack`), if it has been added as a `location` in the [Storage Selection Settings](ceph-cluster-crd.md#storage-selection-settings).
|
||||
|
||||
```yaml
|
||||
apiVersion: ceph.rook.io/v1
|
||||
@@ -71,11 +72,11 @@ spec:
|
||||
|
||||
### Erasure Coded
|
||||
|
||||
If you want to use erasure coded pool with filesystem, your OSDs must use `bluestore` as their `storeType`.
|
||||
Additionally erasure coded can only be used as a data pool and not as a metadata pool. The metadata pool must still be a replicated pool.
|
||||
Erasure coded pools require the OSDs to use `bluestore` for the configured [`storeType`](ceph-cluster-crd.md#osd-configuration-settings). Additionally, erasure coded pools can only be used with `dataPools`. The `metadataPool` must use a replicated pool.
|
||||
|
||||
The sample below requires that you have at least 3 `bluestore` OSDs on different nodes.
|
||||
For erasure coded to make sense, you need **at least three OSDs for the below `dataPools` config** to work.
|
||||
**NOTE:** This sample requires *at least 3 bluestore OSDs*, with each OSD located on a *different node*.
|
||||
|
||||
The OSDs must be located on different nodes, because the [`failureDomain`](ceph-pool-crd.md#spec) will be set to `host` by default, and the `erasureCoded` chunk settings require at least 3 different OSDs (2 `dataChunks` + 1 `codingChunks`).
|
||||
|
||||
```yaml
|
||||
apiVersion: ceph.rook.io/v1
|
||||
|
||||
@@ -6,13 +6,17 @@ indent: true
|
||||
|
||||
# Ceph Object Store CRD
|
||||
|
||||
Rook allows creation and customization of object stores through the custom resource definitions (CRDs). The following settings are available
|
||||
for Ceph object stores.
|
||||
Rook allows creation and customization of object stores through the custom resource definitions (CRDs). The following settings are available for Ceph object stores.
|
||||
|
||||
## Sample
|
||||
|
||||
**NOTE** This example requires you to have **at least 3 bluestore OSDs each on a different node**.
|
||||
This is because the below `erasureCoded` chunk settings require at least 3 bluestore OSDs and as [`failureDomain` setting](ceph-pool-crd.md#spec) to `host` (default), each OSD needs to be on a different nodes.
|
||||
### Erasure Coded
|
||||
|
||||
Erasure coded pools require the OSDs to use `bluestore` for the configured [`storeType`](ceph-cluster-crd.md#osd-configuration-settings). Additionally, erasure coded pools can only be used with `dataPools`. The `metadataPool` must use a replicated pool.
|
||||
|
||||
**NOTE:** This sample requires *at least 3 bluestore OSDs*, with each OSD located on a *different node*.
|
||||
|
||||
The OSDs must be located on different nodes, because the [`failureDomain`](ceph-pool-crd.md#spec) is set to `host` and the `erasureCoded` chunk settings require at least 3 different OSDs (2 `dataChunks` + 1 `codingChunks`).
|
||||
|
||||
```yaml
|
||||
apiVersion: ceph.rook.io/v1
|
||||
|
||||
@@ -10,15 +10,17 @@ Object storage exposes an S3 API to the storage cluster for applications to put
|
||||
|
||||
## Prerequisites
|
||||
|
||||
This guide assumes you have created a Rook cluster as explained in the main [Kubernetes guide](ceph-quickstart.md)
|
||||
This guide assumes a Rook cluster as explained in the [Quickstart](ceph-quickstart.md).
|
||||
|
||||
## Create an Object Store
|
||||
|
||||
**NOTE** This example requires you to have **at least 3 bluestore OSDs each on a different node**.
|
||||
This is because the below `erasureCoded` chunk settings require at least 3 bluestore OSDs and as [`failureDomain` setting](ceph-pool-crd.md#spec) to `host` (default), each OSD needs to be on a different nodes.
|
||||
The below sample will create a `CephObjectStore` that starts the RGW service in the cluster with an S3 API.
|
||||
|
||||
Now we will create the object store, which starts the RGW service in the cluster with the S3 API.
|
||||
Specify your desired settings for the object store in the `object.yaml`. For more details on the settings see the [Object Store CRD](ceph-object-store-crd.md).
|
||||
**NOTE:** This sample requires *at least 3 bluestore OSDs*, with each OSD located on a *different node*.
|
||||
|
||||
The OSDs must be located on different nodes, because the [`failureDomain`](ceph-pool-crd.md#spec) is set to `host` and the `erasureCoded` chunk settings require at least 3 different OSDs (2 `dataChunks` + 1 `codingChunks`).
|
||||
|
||||
See the [Object Store CRD](ceph-object-store-crd.md#object-store-settings), for more detail on the settings availabe for a `CephObjectStore`.
|
||||
|
||||
```yaml
|
||||
apiVersion: ceph.rook.io/v1
|
||||
@@ -44,7 +46,8 @@ spec:
|
||||
instances: 1
|
||||
```
|
||||
|
||||
When the object store is created the Rook operator will create all the pools and other resources necessary to start the service. This may take a minute to complete.
|
||||
After the `CephObjectStore` is created, the Rook operator will then create all the pools and other resources necessary to start the service. This may take a minute to complete.
|
||||
|
||||
```bash
|
||||
# Create the object store
|
||||
kubectl create -f object.yaml
|
||||
@@ -55,8 +58,9 @@ kubectl -n rook-ceph get pod -l app=rook-ceph-rgw
|
||||
|
||||
## Create a User
|
||||
|
||||
Next we will create the object store user, which calls the RGW service in the cluster with the S3 API.
|
||||
Specify your desired settings for the object store user in the `object-user.yaml`. For more details on the settings see the [Object Store User CRD](ceph-object-store-user-crd.md).
|
||||
Next, create a `CephObjectStoreUser`, which will be used to connect to the RGW service in the cluster using the S3 API.
|
||||
|
||||
See the [Object Store User CRD](ceph-object-store-user-crd.md) for more detail on the settings available for a `CephObjectStoreUser`.
|
||||
|
||||
```yaml
|
||||
apiVersion: ceph.rook.io/v1
|
||||
@@ -69,7 +73,7 @@ spec:
|
||||
displayName: "my display name"
|
||||
```
|
||||
|
||||
When the object store user is created the Rook operator will create the RGW user on the object store specified, and store the Access Key and Secret Key in a kubernetes secret in the same namespace as the object store user.
|
||||
When the `CephObjectStoreUser` is created, the Rook operator will then create the RGW user on the specified `CephObjectStore` and store the Access Key and Secret Key in a kubernetes secret in the same namespace as the `CephObjectStoreUser`.
|
||||
|
||||
```bash
|
||||
# Create the object store user
|
||||
@@ -104,13 +108,13 @@ kubectl -n rook-ceph get secret rook-ceph-object-user-my-store-my-user -o yaml |
|
||||
|
||||
## Consume the Object Storage
|
||||
|
||||
Use an S3 compatible client to create a bucket in the object store.
|
||||
Use an S3 compatible client to create a bucket in the `CephObjectStore`.
|
||||
|
||||
This section will allow you to test connecting to the object store and uploading and downloading from it. Run the following commands after you have connected to the [Rook toolbox](ceph-toolbox.md).
|
||||
This section will allow you to test connecting to the `CephObjectStore` and uploading and downloading from it. Run the following commands after you have connected to the [Rook toolbox](ceph-toolbox.md).
|
||||
|
||||
### Install s3cmd
|
||||
|
||||
To test the object store we will install the `s3cmd` tool into the toobox pod.
|
||||
To test the `CephObjectStore` we will install the `s3cmd` tool into the toobox pod.
|
||||
```bash
|
||||
yum --assumeyes install s3cmd
|
||||
```
|
||||
@@ -144,13 +148,13 @@ The access key and secret key can be retrieved as described in the section above
|
||||
|
||||
Now that the user connection variables were set above, we can proceed to perform operations such as creating buckets.
|
||||
|
||||
Create a bucket in the object store
|
||||
Create a bucket in the `CephObjectStore`
|
||||
|
||||
```bash
|
||||
s3cmd mb --no-ssl --host=${AWS_HOST} --region=":default-placement" --host-bucket="" s3://rookbucket
|
||||
```
|
||||
|
||||
List buckets in the object store
|
||||
List buckets in the `CephObjectStore`
|
||||
|
||||
```bash
|
||||
s3cmd ls --no-ssl --host=${AWS_HOST}
|
||||
@@ -224,4 +228,4 @@ rook-ceph-rgw-my-store ClusterIP 10.104.82.228 <none> 80/
|
||||
rook-ceph-rgw-my-store-external NodePort 10.111.113.237 <none> 80:31536/TCP 39s
|
||||
```
|
||||
|
||||
Internally the rgw service is running on port `80`. The external port in this case is `31536`. Now you can access the object store from anywhere! All you need is the hostname for any machine in the cluster, the external port, and the user credentials.
|
||||
Internally the rgw service is running on port `80`. The external port in this case is `31536`. Now you can access the `CephObjectStore` from anywhere! All you need is the hostname for any machine in the cluster, the external port, and the user credentials.
|
||||
@@ -6,17 +6,18 @@ indent: true
|
||||
|
||||
# Ceph Block Pool CRD
|
||||
|
||||
Rook allows creation and customization of storage pools through the custom resource definitions (CRDs). The following settings are available
|
||||
for pools.
|
||||
Rook allows creation and customization of storage pools through the custom resource definitions (CRDs). The following settings are available for pools.
|
||||
|
||||
## Samples
|
||||
|
||||
### Replicated
|
||||
|
||||
**NOTE** This example requires you to have **at least 3 OSDs each on a different node**.
|
||||
This is because the `replicated.size: 3` will require at least 3 OSDs and as [`failureDomain` setting](ceph-pool-crd.md#spec) to `host` (default), each OSD needs to be on a different nodes.
|
||||
For optimal performance, while also adding redundancy, this sample will configure Ceph to make three full copies of the data on multiple nodes.
|
||||
|
||||
**NOTE:** This sample requires *at least 1 OSD per node*, with each OSD located on *3 different nodes*.
|
||||
|
||||
Each OSD must be located on a different node, because the [`failureDomain`](ceph-pool-crd.md#spec) is set to `host` and the `replicated.size` is set to `3`.
|
||||
|
||||
For optimal performance while adding redundancy to the system, configure the data to be copied in full to multiple locations.
|
||||
```yaml
|
||||
apiVersion: ceph.rook.io/v1
|
||||
kind: CephBlockPool
|
||||
@@ -32,10 +33,12 @@ spec:
|
||||
|
||||
### Erasure Coded
|
||||
|
||||
**NOTE** This example requires you to have **at least 3 bluestore OSDs on one or more nodes**.
|
||||
This is because the below `erasureCoded` chunk settings require at least 3 bluestore OSDs and as [`failureDomain` setting](ceph-pool-crd.md#spec) is set to `osd`, the OSDs can be on one or more nodes to be on a different nodes.
|
||||
This sample will lower the overall storage capacity requirement, while also adding redundancy by using [erasure coding](#erasure-coding).
|
||||
|
||||
**NOTE:** This sample requires *at least 3 bluestore OSDs*.
|
||||
|
||||
The OSDs can be located on a single Ceph node or spread across multiple nodes, because the [`failureDomain`](ceph-pool-crd.md#spec) is set to `osd` and the `erasureCoded` chunk settings require at least 3 different OSDs (2 `dataChunks` + 1 `codingChunks`).
|
||||
|
||||
To lower your storage capacity requirements while adding redundancy, use [erasure coding](#erasure-coding).
|
||||
```yaml
|
||||
apiVersion: ceph.rook.io/v1
|
||||
kind: CephBlockPool
|
||||
@@ -66,18 +69,22 @@ When creating an erasure-coded pool, it is highly recommended to create the pool
|
||||
### Spec
|
||||
|
||||
- `replicated`: Settings for a replicated pool. If specified, `erasureCoded` settings must not be specified.
|
||||
- `size`: The number of copies of the data in the pool.
|
||||
- `size`: The desired number of copies to make of the data in the pool.
|
||||
|
||||
- `erasureCoded`: Settings for an erasure-coded pool. If specified, `replicated` settings must not be specified. See below for more details on [erasure coding](#erasure-coding).
|
||||
- `dataChunks`: Number of chunks to divide the original object into
|
||||
- `codingChunks`: Number of redundant chunks to store
|
||||
- `failureDomain`: The failure domain across which the replicas or chunks of data will be spread. Possible values per default are `osd` or `host`,
|
||||
with the default of `host`. For example, if you have replication of size `3` and the failure domain is `host`, all three copies of the data will be
|
||||
placed on osds that are found on unique hosts. In that case you would be guaranteed to tolerate the failure of two hosts. If the failure domain were `osd`,
|
||||
you would be able to tolerate the loss of two devices. Similarly for erasure coding, the data and coding chunks would be spread across the requested failure domain.
|
||||
<br>In case you added another location type to your nodes in the [Storage Selection Settings](ceph-cluster-crd.md#storage-selection-settings) (e.g. `rack`), you can also specify this type as your failure domain.
|
||||
<br>**NOTE:** Neither Rook nor Ceph will prevent the user from creating a cluster where data (or chunks) cannot be replicated safely;
|
||||
it is Ceph's design to delay checking for OSDs until a write request is made, and the write will hang if there are not sufficient OSDs to satisfy the request.
|
||||
- `codingChunks`: Number of coding chunks to generate
|
||||
|
||||
- `failureDomain`: The failure domain across which the data will be spread. This can be set to a value of either `osd` or `host`, with `host` being the default setting. A failure domain can also be set to a different type (e.g. `rack`), if it is added as a `location` in the [Storage Selection Settings](ceph-cluster-crd.md#storage-selection-settings).
|
||||
|
||||
If a `replicated` pool of size `3` is configured and the `failureDomain` is set to `host`, all three copies of the replicated data will be placed on OSDs located on `3` different Ceph hosts. This case is guaranteed to tolerate a failure of two hosts without a loss of data. Similarly, a failure domain set to `osd`, can tolerate a loss of two OSD devices.
|
||||
|
||||
If erasure coding is used, the data and coding chunks are spread across the configured failure domain.
|
||||
|
||||
**NOTE:** Neither Rook, nor Ceph, prevent the creation of a cluster where the replicated data (or Erasure Coded chunks) can be written safely. By design, Ceph will delay checking for suitable OSDs until a write request is made and this write can hang if there are not sufficient OSDs to satisfy the request.
|
||||
|
||||
- `deviceClass`: Sets up the CRUSH rule for the pool to distribute data only on the specified device class. If left empty or unspecified, the pool will use the cluster's default CRUSH root, which usually distributes data over all OSDs, regardless of their class.
|
||||
|
||||
- `crushRoot`: The root in the crush map to be used by the pool. If left empty or unspecified, the default root will be used. Creating a crush hierarchy for the OSDs currently requires the Rook toolbox to run the Ceph tools described [here](http://docs.ceph.com/docs/master/rados/operations/crush-map/#modifying-the-crush-map).
|
||||
|
||||
### Erasure Coding
|
||||
|
||||
Reference in new issue
Block a user