diff --git a/Documentation/ceph-cluster-crd.md b/Documentation/ceph-cluster-crd.md index 0b0d04834..15ad2bbec 100644 --- a/Documentation/ceph-cluster-crd.md +++ b/Documentation/ceph-cluster-crd.md @@ -12,10 +12,12 @@ available for a Ceph cluster. Settings can be specified at the global level to apply to the cluster as a whole, while other settings can be specified at more fine-grained levels. If any setting is unspecified, a suitable default will be used automatically. ### Cluster metadata + - `name`: The name that will be used internally for the Ceph cluster. Most commonly the name is the same as the namespace since multiple clusters are not supported in the same namespace. - `namespace`: The Kubernetes namespace that will be created for the Rook cluster. The services, pods, and other resources created by the operator will be added to this namespace. The common scenario is to create a single Rook cluster. If multiple clusters are created, they must not have conflicting devices or host paths. ### Cluster Settings + - `dataDirHostPath`: The path on the host ([hostPath](https://kubernetes.io/docs/concepts/storage/volumes/#hostpath)) where config and data should be stored for each of the services. If the directory does not exist, it will be created. Because this directory persists on the host, it will remain after pods are deleted. - On **Minikube** environments, use `/data/rook`. Minikube boots into a tmpfs but it provides some [directories](https://github.com/kubernetes/minikube/blob/master/docs/persistent_volumes.md) where files can be persisted across reboots. Using one of these directories will ensure that Rook's data and configuration files are persisted and that enough storage space is available. - **WARNING**: For test scenarios, if you delete a cluster and start a new cluster on the same hosts, the path used by `dataDirHostPath` must be deleted. Otherwise, stale keys and other config will remain from the previous cluster and the new mons will fail to start. @@ -34,8 +36,8 @@ For more details on the mons and when to choose a number other than `3`, see the If individual nodes are specified under the `nodes` field below, then `useAllNodes` must be set to `false`. - `nodes`: Names of individual nodes in the cluster that should have their storage included in accordance with either the cluster level configuration specified above or any node specific overrides described in the next section below. `useAllNodes` must be set to `false` to use specific nodes and their config. + - `config`: Config settings applied to all OSDs on the node unless overridden by `devices` or `directories`. See the [config settings](#osd-configuration-settings) below. - [storage selection settings](#storage-selection-settings) - - [storage configuration settings](#storage-configuration-settings) #### Node Updates Nodes can be added and removed over time by updating the Cluster CRD, for example with `kubectl -n rook-ceph edit cluster.ceph.rook.io rook-ceph`. @@ -43,19 +45,21 @@ This will bring up your default text editor and allow you to add and remove stor This feature is only available when `useAllNodes` has been set to `false`. ### Mon Settings + - `count`: set the number of mons to be started. The number should be odd and between `1` and `9`. If not specified the default is set to `3` and `allowMultiplePerNode` is also set to `true`. - `allowMultiplePerNode`: enable (`true`) or disable (`false`) the placement of multiple mons on one node. Default is `false`. ### Node Settings In addition to the cluster level settings specified above, each individual node can also specify configuration to override the cluster level settings and defaults. If a node does not specify any configuration then it will inherit the cluster level settings. + - `name`: The name of the node, which should match its `kubernetes.io/hostname` label. - `config`: Config settings applied to all OSDs on the node unless overridden by `devices` or `directories`. See the [config settings](#osd-configuration-settings) below. - [storage selection settings](#storage-selection-settings) -- [storage configuration settings](#storage-configuration-settings) ### Storage Selection Settings Below are the settings available, both at the cluster and individual node level, for selecting which storage resources will be included in the cluster. + - `useAllDevices`: `true` or `false`, indicating whether all devices found on nodes in the cluster should be automatically consumed by OSDs. **Not recommended** unless you have a very controlled environment where you will not risk formatting of devices with existing data. When `true`, all devices will be used except those with partitions created or a local filesystem. Is overridden by `deviceFilter` if specified. - `deviceFilter`: A regular expression that allows selection of devices to be consumed by OSDs. If individual devices have been specified for a node then this filter will be ignored. This field uses [golang regular expression syntax](https://golang.org/pkg/regexp/syntax/). For example: - `sdb`: Only selects the `sdb` device if found @@ -74,16 +78,18 @@ Below are the settings available, both at the cluster and individual node level, ### OSD Configuration Settings The following storage selection settings are specific to Ceph and do not apply to other backends. All variables are key-value pairs represented as strings. - - `metadataDevice`: Name of a device to use for the metadata of OSDs on each node. Performance can be improved by using a low latency device (such as SSD or NVMe) as the metadata device, while other spinning platter (HDD) devices on a node are used to store data. - - `storeType`: `filestore` or `bluestore`, the underlying storage format to use for each OSD. The default is set dynamically to `bluestore` for devices, while `filestore` is the default for directories. Set this store type explicitly to override the default. Warning: Bluestore is **not** recommended for directories in production. Bluestore does not purge data from the directory and over time will grow without the ability to compact or shrink. - - `databaseSizeMB`: The size in MB of a bluestore database. Include quotes around the size. - - `walSizeMB`: The size in MB of a bluestore write ahead log (WAL). Include quotes around the size. - - `journalSizeMB`: The size in MB of a filestore journal. Include quotes around the size. + +- `metadataDevice`: Name of a device to use for the metadata of OSDs on each node. Performance can be improved by using a low latency device (such as SSD or NVMe) as the metadata device, while other spinning platter (HDD) devices on a node are used to store data. +- `storeType`: `filestore` or `bluestore`, the underlying storage format to use for each OSD. The default is set dynamically to `bluestore` for devices, while `filestore` is the default for directories. Set this store type explicitly to override the default. Warning: Bluestore is **not** recommended for directories in production. Bluestore does not purge data from the directory and over time will grow without the ability to compact or shrink. +- `databaseSizeMB`: The size in MB of a bluestore database. Include quotes around the size. +- `walSizeMB`: The size in MB of a bluestore write ahead log (WAL). Include quotes around the size. +- `journalSizeMB`: The size in MB of a filestore journal. Include quotes around the size. ### Placement Configuration Settings Placement configuration for the cluster services. It includes the following keys: `mgr`, `mon`, `osd` and `all`. Each service will have its placement configuration generated by merging the generic configuration under `all` with the most specific one (which will override any attributes). A Placement configuration is specified (according to the kubernetes PodSpec) as: + - `nodeAffinity`: kubernetes [NodeAffinity](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#node-affinity-beta-feature) - `podAffinity`: kubernetes [PodAffinity](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#inter-pod-affinity-and-anti-affinity-beta-feature) - `podAntiAffinity`: kubernetes [PodAntiAffinity](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#inter-pod-affinity-and-anti-affinity-beta-feature) @@ -98,12 +104,14 @@ Resources should be specified so that the rook components are handled after [Kub This allows to keep rook components running when for example a node runs out of memory and the rook components are not killed depending on their Quality of Service class. You can set resource requests/limits for rook components through the [Resource Requirements/Limits](#resource-requirementslimits) structure in the following keys: + - `mgr`: Set resource requests/limits for MGRs. - `mon`: Set resource requests/limits for Mons. - `osd`: Set resource requests/limits for OSDs. ### Resource Requirements/Limits For more information on resource requests/limits see the official Kubernetes documentation: [Kubernetes - Managing Compute Resources for Containers](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#resource-requests-and-limits-of-pod-and-container) + - `requests`: Requests for cpu or memory. - `cpu`: Request for CPU (example: one CPU core `1`, 50% of one CPU core `500m`). - `memory`: Limit for Memory (example: one gigabyte of memory `1Gi`, half a gigabyte of memory `512Mi`).