design doc for provisioning OSDs with ceph-volume
Signed-off-by: travisn <tnielsen@redhat.com>
This commit is contained in:
1 file changed
+143
@@ -0,0 +1,143 @@
|
||||
# ceph-volume OSD Provisioning
|
||||
|
||||
**Targeted for v0.9**
|
||||
|
||||
Provisioning OSDs today is done directly by Rook. This needs to be simplified and improved by building
|
||||
on the functionality provided by the `ceph-volume` tool that is included in the ceph image.
|
||||
|
||||
## Legacy Design
|
||||
|
||||
As Rook is implemented today, the provisioning has a lot of complexity around:
|
||||
|
||||
- Partitioning of devices for bluestore
|
||||
- Partitioning and configuration of a `metadata` device where the WAL and DB are placed on a different device from the data
|
||||
- Support for both directories and devices
|
||||
- Support for bluestore and filestore
|
||||
|
||||
Since this is mostly handled by `ceph-volume` now, Rook should replace its own provisioning code and rely on `ceph-volume`.
|
||||
|
||||
## ceph-volume Design
|
||||
|
||||
`ceph-volume` is a CLI tool included in the `ceph/ceph` image that will be used to configure and run Ceph OSDs.
|
||||
`ceph-volume` will replace the OSD provisioning mentioned previously in the legacy design.
|
||||
|
||||
At a high level this flow remains unchanged from the flow in the [one-osd-per-pod design](dedicated-osd-pod.md#create-new-osds).
|
||||
No new jobs or pods need to be launched from what we have today. The sequence of events in the OSD provisioning will be the following.
|
||||
|
||||
- The cluster CRD specifies what nodes/devices to configure with OSDs
|
||||
- The operator starts a provisioning job on each node where OSDs are to be configured
|
||||
- The provisioning job:
|
||||
- Detects what devices should be configured
|
||||
- Calls `ceph-volume lvm batch` to prepare the OSDs on the node. A single call is made with all of the devices unless more specific settings are included for LVM and partitions.
|
||||
- Calls `ceph-volume lvm list` to retrieve the results of the OSD configuration. Store the results in a configmap for the operator to take the next step.
|
||||
- The operator starts a deployment for each OSD that was provisioned. `rook` is the entrypoint for the container.
|
||||
- The configmap with the osd configuration is loaded with info such as ID, FSID, bluestore/filestore, etc
|
||||
- `ceph-volume lvm activate` is called to activate the osd, which mounts the config directory such as `/var/lib/ceph/osd-0`, using a tempfs mount. The OSD options such as `--bluestore`, `--filestore`, `OSD_ID`, and `OSD_FSID` are passed to the command as necessary.
|
||||
- The OSD daemon is started with `ceph-osd`
|
||||
- When `ceph-osd` exits, `rook` will exit and the pod will be restarted by K8s.
|
||||
|
||||
### New Features
|
||||
|
||||
`ceph-volume` enables rook to expose several new features:
|
||||
|
||||
- Multiple OSDs for a single device, which is ideal for NVME devices.
|
||||
- Configure OSDs on LVM, either consuming the existing LVM or automatically configuring LVM on the raw devices.
|
||||
- Encrypt the OSD data with dmcrypt
|
||||
|
||||
The Cluster CRD will be updated with the following settings to enable these features. All of these settings can be specified
|
||||
globally if under the `storage` element as in this example. The `config` element can also be specified under individual
|
||||
nodes or devices.
|
||||
```yaml
|
||||
storage:
|
||||
config:
|
||||
# whether to encrypt the contents of the OSD with dmcrypt
|
||||
encryptDevice: true
|
||||
# how many OSDs should be configured on each device. only recommended to be greater than 1 for NVME devices
|
||||
osdsPerDevice: 1
|
||||
# the class name for the OSD(s) on devices
|
||||
crushDeviceClass: ssd
|
||||
```
|
||||
|
||||
If more flexibility is needed that consuming raw devices, LVM or partition names can also be used for specific nodes.
|
||||
Properties are shown for both bluestore and filestore OSDs.
|
||||
|
||||
```yaml
|
||||
storage:
|
||||
nodes:
|
||||
- name: node2
|
||||
# OSDs on LVM (open design question: need to re-evaluate the logicalDevice settings when they are implemented after 0.9 and whether they should be under the more general storage node "config" settings)
|
||||
logicalDevices:
|
||||
# bluestore: the DB, WAL, and Data are on separate LVs
|
||||
- db: db_lv1
|
||||
wal: wal_lv1
|
||||
data: data_lv1
|
||||
dbVolumeGroup: db_vg
|
||||
walVolumeGroup: wal_vg
|
||||
dataVolumeGroup: data_vg
|
||||
# bluestore: the DB, WAL, and Data are all on the same LV
|
||||
- volume: my_lv1
|
||||
volumeGroup: my_vg
|
||||
# filestore: data and journal on the same LV
|
||||
- data: my_lv2
|
||||
dataVolumeGroup: my_vg
|
||||
# filestore: data and journal on different LVs
|
||||
- data: data_lv3
|
||||
dataVolumeGroup: data_vg
|
||||
journal: journal_lv3
|
||||
journalVolumeGroup: journal_vg
|
||||
# devices support both filestore and bluestore configurations based on the "config.storeType" setting at the global, node, or device level
|
||||
devices:
|
||||
# OSD on a raw device
|
||||
- name: sdd
|
||||
# OSD on a partition (partition support is new)
|
||||
- name: sdf1
|
||||
# Multiple OSDs on a high performance device
|
||||
- name: nvme01
|
||||
config:
|
||||
osdsPerDevice: 5
|
||||
```
|
||||
|
||||
The above options for LVM and partitions look very tedious. Questions:
|
||||
|
||||
- Is it useful at this level of complexity?
|
||||
- Is there a simpler way users would configure LVM?
|
||||
- Do users need all this flexibility? This looks like too many options to maintain.
|
||||
|
||||
### Backward compatibility
|
||||
|
||||
Rook will need to continue supporting clusters that are running different types of OSDs. All of the v0.8 OSDs must continue running
|
||||
after Rook is upgraded to v0.9 and beyond, whether they were filestore or bluestore running on directories or devices.
|
||||
|
||||
Since `ceph-volume` only supports devices that have **not** been previously configured by Rook:
|
||||
|
||||
- Rook will continue to provision OSDs directly when a `directory` is specified in the CRD
|
||||
- Support for creating new OSDs on directories will be deprecated. While directories might still be used for test scenarios,
|
||||
it's not a mainline scenario. With the legacy design, directories were commonly used on LVM, but LVM is now directly supported.
|
||||
In v0.9, support for directories will remain, but documentation will encourage users to provision devices.
|
||||
- For existing devices configured by Rook, `ceph-volume` will be skipped and the OSDs will be started as previously
|
||||
- New devices will be provisioned with `ceph-volume`
|
||||
|
||||
### Versioning
|
||||
|
||||
Rook relies on very recent developments in `ceph-volume` that are not yet available in luminous or mimic releases.
|
||||
For example, rook needs to run the command:
|
||||
```
|
||||
ceph-volume lvm batch --prepare <devices>
|
||||
```
|
||||
|
||||
The `batch` command and the flag `--prepare` have been added recently.
|
||||
While the latest `ceph-volume` changes will soon be merged to luminous and mimic, Rook needs to know if it is running an image that contains the required functionality.
|
||||
|
||||
To detect if `ceph-volume` supports the required options, Rook will run the
|
||||
command with all the flags that are required. To avoid side effects when testing for the version of `ceph-volume`, no devices
|
||||
are passed to the `batch` command.
|
||||
```
|
||||
ceph-volume lvm batch --prepare
|
||||
```
|
||||
|
||||
- If the flags are supported, `ceph-volume` has an exit code of `0`.
|
||||
- If the flags are not supported, `ceph-volume` has an exit code of `2`.
|
||||
|
||||
Since Rook orchestrates different versions of Ceph, Rook (at least initially) will need to support running images that may not
|
||||
have the features necessary from `ceph-volume`. When a supported version of `ceph-volume` is not detected, Rook will
|
||||
execute the legacy code to provision devices.
|
||||
Reference in new issue
Block a user