mkdocs uses a markdown renderer that is hardcoded to 4 spaces per tab
for detecting indentation levels, including ordered- and
unordered-lists. Since we cannot easily change the renderer, begin using
a markdown linter in CI that will fail if official docs do not adhere to
the spacing rules.
As a starting point, the markdownlint config does not begin with the
default set of checks, which might overwhelm attempts to fix them.
Instead, focus on list-tab-spacing rules and a few other highly useful
checks.
markdownlint also has some gaps in its abilities that allow common Rook
doc issues to pass acceptance. However, it allows creating custom
linting plugins. Create 2 such linting plugins to check 2 things:
- all doc lines (except code blocks) must be aligned to a 4-space
boundary, without exception. This ensures that markdown will render
correctly with mkdocs. This unfortunately makes it possible to create
lists that are internally aligned strangely.
- admonitions must all follow the same format of
```
!!! header
body
```
For the strange lists, this is allowed and renders correctly, but it
looks strange:
```md
- first bullet
- second bullet
still second bullet
- third bullet
has a paragraph
of text inside
- last bullet
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
(cherry picked from commit c9d99e01a0)
348 lines
11 KiB
Markdown
348 lines
11 KiB
Markdown
---
|
||
title: PVC Storage Cluster
|
||
---
|
||
|
||
In a "PVC-based cluster", the Ceph persistent data is stored on volumes requested from a storage class of your choice.
|
||
This type of cluster is recommended in a cloud environment where volumes can be dynamically created and also
|
||
in clusters where a local PV provisioner is available.
|
||
|
||
## AWS Storage Example
|
||
|
||
In this example, the mon and OSD volumes are provisioned from the AWS `gp2` storage class. This storage class can be replaced by any storage class that provides `file` mode (for mons) and `block` mode (for OSDs).
|
||
|
||
```yaml
|
||
apiVersion: ceph.rook.io/v1
|
||
kind: CephCluster
|
||
metadata:
|
||
name: rook-ceph
|
||
namespace: rook-ceph
|
||
spec:
|
||
cephVersion:
|
||
image: quay.io/ceph/ceph:v18.2.2
|
||
dataDirHostPath: /var/lib/rook
|
||
mon:
|
||
count: 3
|
||
allowMultiplePerNode: false
|
||
volumeClaimTemplate:
|
||
spec:
|
||
storageClassName: gp2
|
||
resources:
|
||
requests:
|
||
storage: 10Gi
|
||
storage:
|
||
storageClassDeviceSets:
|
||
- name: set1
|
||
count: 3
|
||
portable: false
|
||
encrypted: false
|
||
volumeClaimTemplates:
|
||
- metadata:
|
||
name: data
|
||
spec:
|
||
resources:
|
||
requests:
|
||
storage: 10Gi
|
||
# IMPORTANT: Change the storage class depending on your environment (e.g. local-storage, gp2)
|
||
storageClassName: gp2
|
||
volumeMode: Block
|
||
accessModes:
|
||
- ReadWriteOnce
|
||
onlyApplyOSDPlacement: false
|
||
```
|
||
|
||
### Local Storage Example
|
||
|
||
In the CRD specification below, 3 OSDs (having specific placement and resource values) and 3 mons with each using a 10Gi PVC, are created by Rook using the `local-storage` storage class.
|
||
|
||
```yaml
|
||
apiVersion: ceph.rook.io/v1
|
||
kind: CephCluster
|
||
metadata:
|
||
name: rook-ceph
|
||
namespace: rook-ceph
|
||
spec:
|
||
dataDirHostPath: /var/lib/rook
|
||
mon:
|
||
count: 3
|
||
allowMultiplePerNode: false
|
||
volumeClaimTemplate:
|
||
spec:
|
||
storageClassName: local-storage
|
||
resources:
|
||
requests:
|
||
storage: 10Gi
|
||
cephVersion:
|
||
image: quay.io/ceph/ceph:v18.2.2
|
||
allowUnsupported: false
|
||
dashboard:
|
||
enabled: true
|
||
network:
|
||
hostNetwork: false
|
||
storage:
|
||
storageClassDeviceSets:
|
||
- name: set1
|
||
count: 3
|
||
portable: false
|
||
resources:
|
||
limits:
|
||
memory: "4Gi"
|
||
requests:
|
||
cpu: "500m"
|
||
memory: "4Gi"
|
||
placement:
|
||
podAntiAffinity:
|
||
preferredDuringSchedulingIgnoredDuringExecution:
|
||
- weight: 100
|
||
podAffinityTerm:
|
||
labelSelector:
|
||
matchExpressions:
|
||
- key: "rook.io/cluster"
|
||
operator: In
|
||
values:
|
||
- cluster1
|
||
topologyKey: "topology.kubernetes.io/zone"
|
||
volumeClaimTemplates:
|
||
- metadata:
|
||
name: data
|
||
spec:
|
||
resources:
|
||
requests:
|
||
storage: 10Gi
|
||
storageClassName: local-storage
|
||
volumeMode: Block
|
||
accessModes:
|
||
- ReadWriteOnce
|
||
```
|
||
|
||
## PVC storage only for monitors
|
||
|
||
In the CRD specification below three monitors are created each using a 10Gi PVC
|
||
created by Rook using the `local-storage` storage class. Even while the mons consume PVCs,
|
||
the OSDs in this example will still consume raw devices on the host.
|
||
|
||
```yaml
|
||
apiVersion: ceph.rook.io/v1
|
||
kind: CephCluster
|
||
metadata:
|
||
name: rook-ceph
|
||
namespace: rook-ceph
|
||
spec:
|
||
cephVersion:
|
||
image: quay.io/ceph/ceph:v18.2.2
|
||
dataDirHostPath: /var/lib/rook
|
||
mon:
|
||
count: 3
|
||
allowMultiplePerNode: false
|
||
volumeClaimTemplate:
|
||
spec:
|
||
storageClassName: local-storage
|
||
resources:
|
||
requests:
|
||
storage: 10Gi
|
||
dashboard:
|
||
enabled: true
|
||
storage:
|
||
useAllNodes: true
|
||
useAllDevices: true
|
||
```
|
||
|
||
## Dedicated metadata and wal device for OSD on PVC
|
||
|
||
In the simplest case, Ceph OSD BlueStore consumes a single (primary) storage device.
|
||
BlueStore is the engine used by the OSD to store data.
|
||
|
||
The storage device is normally used as a whole, occupying the full device that is managed directly by BlueStore.
|
||
It is also possible to deploy BlueStore across additional devices such as a DB device.
|
||
This device can be used for storing BlueStore’s internal metadata.
|
||
BlueStore (or rather, the embedded RocksDB) will put as much metadata as it can on the DB device to improve performance.
|
||
If the DB device fills up, metadata will spill back onto the primary device (where it would have been otherwise).
|
||
Again, it is only helpful to provision a DB device if it is faster than the primary device.
|
||
|
||
You can have multiple `volumeClaimTemplates` where each might either represent a device or a metadata device.
|
||
An example of the `storage` section when specifying the metadata device is:
|
||
|
||
```yaml
|
||
storage:
|
||
storageClassDeviceSets:
|
||
- name: set1
|
||
count: 3
|
||
portable: false
|
||
volumeClaimTemplates:
|
||
- metadata:
|
||
name: data
|
||
spec:
|
||
resources:
|
||
requests:
|
||
storage: 10Gi
|
||
# IMPORTANT: Change the storage class depending on your environment (e.g. local-storage, gp2)
|
||
storageClassName: gp2
|
||
volumeMode: Block
|
||
accessModes:
|
||
- ReadWriteOnce
|
||
- metadata:
|
||
name: metadata
|
||
spec:
|
||
resources:
|
||
requests:
|
||
# Find the right size https://docs.ceph.com/docs/master/rados/configuration/bluestore-config-ref/#sizing
|
||
storage: 5Gi
|
||
# IMPORTANT: Change the storage class depending on your environment (e.g. local-storage, io1)
|
||
storageClassName: io1
|
||
volumeMode: Block
|
||
accessModes:
|
||
- ReadWriteOnce
|
||
```
|
||
|
||
!!! note
|
||
Note that Rook only supports three naming convention for a given template:
|
||
|
||
* "data": represents the main OSD block device, where your data is being stored.
|
||
* "metadata": represents the metadata (including block.db and block.wal) device used to store the Ceph Bluestore database for an OSD.
|
||
* "wal": represents the block.wal device used to store the Ceph Bluestore database for an OSD. If this device is set, "metadata" device will refer specifically to block.db device.
|
||
It is recommended to use a faster storage class for the metadata or wal device, with a slower device for the data.
|
||
Otherwise, having a separate metadata device will not improve the performance.
|
||
|
||
The bluestore partition has the following reference combinations supported by the ceph-volume utility:
|
||
|
||
* A single "data" device.
|
||
|
||
```yaml
|
||
storage:
|
||
storageClassDeviceSets:
|
||
- name: set1
|
||
count: 3
|
||
portable: false
|
||
volumeClaimTemplates:
|
||
- metadata:
|
||
name: data
|
||
spec:
|
||
resources:
|
||
requests:
|
||
storage: 10Gi
|
||
# IMPORTANT: Change the storage class depending on your environment (e.g. local-storage, gp2)
|
||
storageClassName: gp2
|
||
volumeMode: Block
|
||
accessModes:
|
||
- ReadWriteOnce
|
||
```
|
||
|
||
* A "data" device and a "metadata" device.
|
||
|
||
```yaml
|
||
storage:
|
||
storageClassDeviceSets:
|
||
- name: set1
|
||
count: 3
|
||
portable: false
|
||
volumeClaimTemplates:
|
||
- metadata:
|
||
name: data
|
||
spec:
|
||
resources:
|
||
requests:
|
||
storage: 10Gi
|
||
# IMPORTANT: Change the storage class depending on your environment (e.g. local-storage, gp2)
|
||
storageClassName: gp2
|
||
volumeMode: Block
|
||
accessModes:
|
||
- ReadWriteOnce
|
||
- metadata:
|
||
name: metadata
|
||
spec:
|
||
resources:
|
||
requests:
|
||
# Find the right size https://docs.ceph.com/docs/master/rados/configuration/bluestore-config-ref/#sizing
|
||
storage: 5Gi
|
||
# IMPORTANT: Change the storage class depending on your environment (e.g. local-storage, io1)
|
||
storageClassName: io1
|
||
volumeMode: Block
|
||
accessModes:
|
||
- ReadWriteOnce
|
||
```
|
||
|
||
* A "data" device and a "wal" device.
|
||
A WAL device can be used for BlueStore’s internal journal or write-ahead log (block.wal), it is only useful to use a WAL device if the device is faster than the primary device (data device).
|
||
There is no separate "metadata" device in this case, the data of main OSD block and block.db located in "data" device.
|
||
|
||
```yaml
|
||
storage:
|
||
storageClassDeviceSets:
|
||
- name: set1
|
||
count: 3
|
||
portable: false
|
||
volumeClaimTemplates:
|
||
- metadata:
|
||
name: data
|
||
spec:
|
||
resources:
|
||
requests:
|
||
storage: 10Gi
|
||
# IMPORTANT: Change the storage class depending on your environment (e.g. local-storage, gp2)
|
||
storageClassName: gp2
|
||
volumeMode: Block
|
||
accessModes:
|
||
- ReadWriteOnce
|
||
- metadata:
|
||
name: wal
|
||
spec:
|
||
resources:
|
||
requests:
|
||
# Find the right size https://docs.ceph.com/docs/master/rados/configuration/bluestore-config-ref/#sizing
|
||
storage: 5Gi
|
||
# IMPORTANT: Change the storage class depending on your environment (e.g. local-storage, io1)
|
||
storageClassName: io1
|
||
volumeMode: Block
|
||
accessModes:
|
||
- ReadWriteOnce
|
||
```
|
||
|
||
* A "data" device, a "metadata" device and a "wal" device.
|
||
|
||
```yaml
|
||
storage:
|
||
storageClassDeviceSets:
|
||
- name: set1
|
||
count: 3
|
||
portable: false
|
||
volumeClaimTemplates:
|
||
- metadata:
|
||
name: data
|
||
spec:
|
||
resources:
|
||
requests:
|
||
storage: 10Gi
|
||
# IMPORTANT: Change the storage class depending on your environment (e.g. local-storage, gp2)
|
||
storageClassName: gp2
|
||
volumeMode: Block
|
||
accessModes:
|
||
- ReadWriteOnce
|
||
- metadata:
|
||
name: metadata
|
||
spec:
|
||
resources:
|
||
requests:
|
||
# Find the right size https://docs.ceph.com/docs/master/rados/configuration/bluestore-config-ref/#sizing
|
||
storage: 5Gi
|
||
# IMPORTANT: Change the storage class depending on your environment (e.g. local-storage, io1)
|
||
storageClassName: io1
|
||
volumeMode: Block
|
||
accessModes:
|
||
- ReadWriteOnce
|
||
- metadata:
|
||
name: wal
|
||
spec:
|
||
resources:
|
||
requests:
|
||
# Find the right size https://docs.ceph.com/docs/master/rados/configuration/bluestore-config-ref/#sizing
|
||
storage: 5Gi
|
||
# IMPORTANT: Change the storage class depending on your environment (e.g. local-storage, io1)
|
||
storageClassName: io1
|
||
volumeMode: Block
|
||
accessModes:
|
||
- ReadWriteOnce
|
||
```
|
||
|
||
To determine the size of the metadata block follow the [official Ceph sizing guide](https://docs.ceph.com/docs/master/rados/configuration/bluestore-config-ref/#sizing).
|
||
|
||
With the present configuration, each OSD will have its main block allocated a 10GB device as well a 5GB device to act as a bluestore database.
|