osd: allow LVM logical volumes as OSD data devices
remov restrictions that prevented LVM logical volume from being
provisioned as OSDs. LV devices with a metadatadevice were rejected at
the filtering stage. LV devices without one were incorrectly being
routed to ceph-volume raw mode which was resuting in error. This PR:
- removes metadatadevice restriction on LV devices.
- Adds LVM time check to isSafeToUseRAWMode to prevent raw mode on lvm
Signed-off-by: Santosh <sapillai@redhat.com>
(cherry picked from commit c6d48178dd)
This commit is contained in:
1 parent
7d779c6048
commit
c64524b96d
5 files changed
+73
-89
No files matched your search
@@ -366,7 +366,7 @@ See the table in [OSD Configuration Settings](#osd-configuration-settings) to kn
|
||||
|
||||
The following storage selection settings are specific to Ceph and do not apply to other backends. All variables are key-value pairs represented as strings.
|
||||
|
||||
* `metadataDevice`: Name of a device, [partition](#limitations-of-metadata-device) or lvm to use for the metadata of OSDs on each node. Performance can be improved by using a low latency device (such as SSD or NVMe) as the metadata device, while other spinning platter (HDD) devices on a node are used to store data. Provisioning will fail if the user specifies a `metadataDevice` but that device is not used as a metadata device by Ceph. Notably, `ceph-volume` will not use a device of the same device class (HDD, SSD, NVMe) as OSD devices for metadata, resulting in this failure.
|
||||
* `metadataDevice`: Name of a device, [partition](#Metadata device type and provisioning behavior) or lvm to use for the metadata of OSDs on each node. Performance can be improved by using a low latency device (such as SSD or NVMe) as the metadata device, while other spinning platter (HDD) devices on a node are used to store data. Provisioning will fail if the user specifies a `metadataDevice` but that device is not used as a metadata device by Ceph. Notably, `ceph-volume` will not use a device of the same device class (HDD, SSD, NVMe) as OSD devices for metadata, resulting in this failure.
|
||||
* `databaseSizeMB`: The size in MB of a bluestore database. Include quotes around the size.
|
||||
* `walSizeMB`: The size in MB of a bluestore write ahead log (WAL). Include quotes around the size.
|
||||
* `deviceClass`: The [CRUSH device class](https://ceph.io/community/new-luminous-crush-device-classes/) to use for this selection of storage devices. (By default, if a device's class has not already been set, OSDs will automatically set a device's class to either `hdd`, `ssd`, or `nvme` based on the hardware properties exposed by the Linux kernel.) These storage classes can then be used to select the devices backing a storage pool by specifying them as the value of [the pool spec's `deviceClass` field](../Block-Storage/ceph-block-pool-crd.md#spec). If updating the device class of an OSD after the OSD is already created, `allowDeviceClassUpdate: true` must be set. Otherwise updates to this `deviceClass` will be ignored.
|
||||
@@ -384,14 +384,17 @@ Supported configurations are:
|
||||
| :---------------- | :------------------------------------------------------------------------------------------------ | :------------------------------------------------------------------------------ |
|
||||
| disk | supported | supported |
|
||||
| part | `encryptedDevice` must be `false` | `encrypted` must be `false` |
|
||||
| lvm | `metadataDevice` must be `""`, `osdsPerDevice` must be `1`, and `encryptedDevice` must be `false` | `metadata.name` must not be `metadata` or `wal` and `encrypted` must be `false` |
|
||||
| lvm | `osdsPerDevice` must be `1` and `encryptedDevice` must be `false` | `metadata.name` must not be `metadata` or `wal` and `encrypted` must be `false` |
|
||||
| crypt | not supported | supported |
|
||||
| mpath | supported | supported |
|
||||
|
||||
#### Limitations of metadata device
|
||||
#### Metadata device type and provisioning behavior
|
||||
|
||||
- If `metadataDevice` is specified in the global OSD configuration or in the node level OSD configuration, the metadata device will be shared between all OSDs on the same node. In other words, OSDs will be initialized by `lvm batch`. In this case, we can't use partition device.
|
||||
- If `metadataDevice` is specified in the device local configuration, we can use partition as metadata device. In other words, OSDs are initialized by `lvm prepare`.
|
||||
The type of metadata device determines how OSDs are provisioned:
|
||||
|
||||
- A **disk** metadata device can be shared between multiple data devices (global or node-level `metadataDevice`). OSDs are initialized with `ceph-volume lvm batch --db-devices`.
|
||||
- A **partition** metadata device can only be used by a single data device and must be specified in the device-local configuration, not the global or node-level configuration. OSDs are initialized with `ceph-volume lvm prepare --block.db`.
|
||||
- An **LVM logical volume** metadata device can only be used by a single data device. It can be specified in either the device-local or global/node-level configuration, but if multiple data devices share the same LVM metadata device, only the first will be provisioned. OSDs are initialized with `ceph-volume lvm prepare --block.db`.
|
||||
|
||||
### Annotations and Labels
|
||||
|
||||
|
||||
@@ -515,8 +515,8 @@ func getAvailableDevices(context *clusterd.Context, agent *OsdAgent) (*DeviceOsd
|
||||
matched = false
|
||||
continue
|
||||
}
|
||||
if desiredDevice.MetadataDevice != "" {
|
||||
logger.Infof("logical volume %q is not picked because OSD on LV with metadata device is not allowed", device.Name)
|
||||
if desiredDevice.OSDsPerDevice > 1 {
|
||||
logger.Infof("logical volume %q is not picked because osdsPerDevice > 1 on LV is not supported", device.Name)
|
||||
matched = false
|
||||
continue
|
||||
}
|
||||
|
||||
@@ -445,6 +445,19 @@ NAME="sdb1" SIZE="30" TYPE="part" PKNAME="sdb"`, nil
|
||||
assert.Equal(t, 1, len(mapping.Entries))
|
||||
assert.Equal(t, -1, mapping.Entries["dm-0"].Data)
|
||||
|
||||
// select an exact logical volume with a metadata device
|
||||
agent.devices = []DesiredDevice{{Name: "/dev/mapper/vg1-lv1", MetadataDevice: "/dev/mapper/vg1-lv2"}}
|
||||
mapping, err = getAvailableDevices(context, agent)
|
||||
assert.Nil(t, err)
|
||||
assert.Equal(t, 1, len(mapping.Entries))
|
||||
assert.Equal(t, -1, mapping.Entries["dm-0"].Data)
|
||||
|
||||
// logical volume with osdsPerDevice > 1 is rejected
|
||||
agent.devices = []DesiredDevice{{Name: "/dev/mapper/vg1-lv1", OSDsPerDevice: 2}}
|
||||
mapping, err = getAvailableDevices(context, agent)
|
||||
assert.Nil(t, err)
|
||||
assert.Equal(t, 0, len(mapping.Entries))
|
||||
|
||||
// select all devices except those that have a prefix of "s"
|
||||
agent.devices = []DesiredDevice{{Name: "^[^s]", IsFilter: true}}
|
||||
mapping, err = getAvailableDevices(context, agent)
|
||||
|
||||
@@ -445,6 +445,12 @@ func (a *OsdAgent) allowRawMode(context *clusterd.Context) (bool, error) {
|
||||
|
||||
// test if safe to use raw mode for a particular device
|
||||
func isSafeToUseRawMode(device *DeviceOsdIDEntry) bool {
|
||||
// ceph-volume raw mode does not support LVM logical volumes
|
||||
if device.DeviceInfo != nil && device.DeviceInfo.Type == sys.LVMType {
|
||||
logger.Debugf("won't use raw mode for disk %q since it is a logical volume", device.Config.Name)
|
||||
return false
|
||||
}
|
||||
|
||||
// ceph-volume raw mode does not support more than one OSD per disk
|
||||
if device.Config.OSDsPerDevice > 1 {
|
||||
logger.Debugf("won't use raw mode for disk %q since osd per device is %d", device.Config.Name, device.Config.OSDsPerDevice)
|
||||
@@ -461,10 +467,6 @@ func isSafeToUseRawMode(device *DeviceOsdIDEntry) bool {
|
||||
}
|
||||
|
||||
func lvmModeAllowed(device *DeviceOsdIDEntry, storeConfig *config.StoreConfig) bool {
|
||||
if device.DeviceInfo.Type == sys.LVMType {
|
||||
logger.Infof("skipping device %q for lvm mode since LVM logical volumes don't support `metadataDevice` or `osdsPerDevice` > 1", device.Config.Name)
|
||||
return false
|
||||
}
|
||||
if device.DeviceInfo.Type == sys.PartType && storeConfig.EncryptedDevice {
|
||||
logger.Infof("skipping partition %q for lvm mode since encryption is not supported on partitions with a `metadataDevice` or `osdsPerDevice > 1`", device.Config.Name)
|
||||
return false
|
||||
@@ -614,6 +616,16 @@ func (a *OsdAgent) initializeDevicesLVMMode(context *clusterd.Context, devices *
|
||||
// `ceph-volume lvm batch` does not support kernel names (e.g. `/dev/dm-X`)
|
||||
// for multipath devices. We should use real names (e.g. `/dev/mapper/mpathX`) instead.
|
||||
deviceArg = device.DeviceInfo.RealPath
|
||||
} else if device.DeviceInfo.Type == sys.LVMType {
|
||||
// `ceph-volume lvm prepare` needs the VG/LV path (e.g. `/dev/vg/lv`), not
|
||||
// the kernel dm name (e.g. `/dev/dm-0`). Find it from the device links.
|
||||
deviceArg = path.Join("/dev", name)
|
||||
for _, link := range strings.Split(device.DeviceInfo.DevLinks, " ") {
|
||||
if strings.HasPrefix(link, "/dev/") && !strings.HasPrefix(link, "/dev/disk/") && !strings.HasPrefix(link, "/dev/mapper/") {
|
||||
deviceArg = link
|
||||
break
|
||||
}
|
||||
}
|
||||
} else {
|
||||
deviceArg = path.Join("/dev", name)
|
||||
}
|
||||
@@ -646,8 +658,15 @@ func (a *OsdAgent) initializeDevicesLVMMode(context *clusterd.Context, devices *
|
||||
if metadataDevice == nil {
|
||||
return errors.Errorf("metadata device %s is not found", md)
|
||||
}
|
||||
// lvm device format is /dev/<vg>/<lv>
|
||||
if metadataDevice.Type != sys.LVMType {
|
||||
if metadataDevice.Type == sys.LVMType {
|
||||
// Find the /dev/<vg>/<lv> path from DevLinks for ceph-volume compatibility
|
||||
for _, link := range strings.Split(metadataDevice.DevLinks, " ") {
|
||||
if strings.HasPrefix(link, "/dev/") && !strings.HasPrefix(link, "/dev/disk/") && !strings.HasPrefix(link, "/dev/mapper/") {
|
||||
md = link
|
||||
break
|
||||
}
|
||||
}
|
||||
} else {
|
||||
md = metadataDevice.Name
|
||||
}
|
||||
|
||||
@@ -672,6 +691,9 @@ func (a *OsdAgent) initializeDevicesLVMMode(context *clusterd.Context, devices *
|
||||
}
|
||||
metadataDevices[md]["part"] = "true" // ceph-volume lvm batch only supports disk and lvm
|
||||
}
|
||||
if metadataDevice.Type == sys.LVMType {
|
||||
metadataDevices[md]["part"] = "true" // ceph-volume lvm batch does not support existing LVs as --db-devices
|
||||
}
|
||||
deviceDBSizeMB := getDatabaseSize(a.storeConfig.DatabaseSizeMB, device.Config.DatabaseSizeMB)
|
||||
if a.storeConfig.IsValidStoreType() && deviceDBSizeMB > 0 {
|
||||
if deviceDBSizeMB < cephVolumeMinDBSize {
|
||||
|
||||
@@ -1063,7 +1063,7 @@ func TestInitializeBlock(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// Test with two metadata partition devices, one is lvm, one is partition
|
||||
// Test with two metadata devices, one is lvm, one is partition — both use lvm prepare
|
||||
{
|
||||
metadataDevicePath := "/dev/test-rook-vg/test-rook-lv"
|
||||
devices := &DeviceOsdMapping{
|
||||
@@ -1077,53 +1077,23 @@ func TestInitializeBlock(t *testing.T) {
|
||||
MockExecuteCommand: func(command string, args ...string) error {
|
||||
logger.Infof("%s %v", command, args)
|
||||
|
||||
var err error
|
||||
// Validate base common args
|
||||
if args[8] == "--yes" {
|
||||
// lvm will use `ceph-volume lvm batch`
|
||||
err = testBaseArgs(args)
|
||||
} else {
|
||||
// partition will use `ceph-volume lvm prepare`
|
||||
err = testBasePrepareArgs(args)
|
||||
}
|
||||
err := testBasePrepareArgs(args)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// for partition
|
||||
// partition metadata device
|
||||
if args[7] == "--data" && args[8] == "/dev/sda" && args[9] == "--block.db" && args[10] == "/dev/sdb1" {
|
||||
return nil
|
||||
}
|
||||
|
||||
// First command for lvm
|
||||
if args[9] == "--osds-per-device" && args[10] == "1" && args[11] == "/dev/sdc" && args[12] == "--db-devices" && args[13] == metadataDevicePath {
|
||||
return nil
|
||||
}
|
||||
|
||||
// Second command for lvm
|
||||
if args[9] == "--osds-per-device" && args[10] == "1" && args[11] == "/dev/sdc" && args[12] == "--db-devices" && args[13] == metadataDevicePath && args[14] == "--report" {
|
||||
// lvm metadata device
|
||||
if args[7] == "--data" && args[8] == "/dev/sdc" && args[9] == "--block.db" && args[10] == metadataDevicePath {
|
||||
return nil
|
||||
}
|
||||
|
||||
return errors.Errorf("unknown command %s %s", command, args)
|
||||
},
|
||||
|
||||
MockExecuteCommandWithOutput: func(command string, args ...string) (string, error) {
|
||||
logger.Infof("%s %v", command, args)
|
||||
|
||||
// Validate base common args
|
||||
err := testBaseArgs(args)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
// First command
|
||||
if args[9] == "--osds-per-device" && args[10] == "1" && args[11] == "/dev/sdc" && args[12] == "--db-devices" && args[13] == metadataDevicePath {
|
||||
return fmt.Sprintf(`[{"block_db": "%s", "data": "%s"}]`, metadataDevicePath, "/dev/sdc"), nil
|
||||
}
|
||||
|
||||
return "", errors.Errorf("unknown command %s %s", command, args)
|
||||
},
|
||||
}
|
||||
|
||||
a := &OsdAgent{clusterInfo: &cephclient.ClusterInfo{CephVersion: cephver.CephVersion{Major: 17, Minor: 2, Extra: 0}}, nodeName: "node1", storeConfig: config.StoreConfig{StoreType: "bluestore"}}
|
||||
@@ -1425,7 +1395,7 @@ func TestInitializeBlock(t *testing.T) {
|
||||
logger.Info("success, go to next test")
|
||||
}
|
||||
|
||||
// Test with metadata devices with lvm
|
||||
// Test with metadata devices with lvm — should use lvm prepare
|
||||
{
|
||||
metadataDevicePath := "/dev/test-rook-vg/test-rook-lv"
|
||||
devices := &DeviceOsdMapping{
|
||||
@@ -1446,41 +1416,18 @@ func TestInitializeBlock(t *testing.T) {
|
||||
executor.MockExecuteCommand = func(command string, args ...string) error {
|
||||
logger.Infof("%s %v", command, args)
|
||||
|
||||
// Validate base common args
|
||||
err := testBaseArgs(args)
|
||||
err := testBasePrepareArgs(args)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// First command
|
||||
if args[9] == "--osds-per-device" && args[10] == "1" && args[11] == "/dev/sda" && args[12] == "--db-devices" && args[13] == metadataDevicePath {
|
||||
return nil
|
||||
}
|
||||
|
||||
// Second command
|
||||
if args[9] == "--osds-per-device" && args[10] == "1" && args[11] == "/dev/sda" && args[12] == "--db-devices" && args[13] == metadataDevicePath && args[14] == "--report" {
|
||||
// lvm prepare with LV metadata device
|
||||
if args[7] == "--data" && args[8] == "/dev/sda" && args[9] == "--block.db" && args[10] == metadataDevicePath {
|
||||
return nil
|
||||
}
|
||||
|
||||
return errors.Errorf("unknown command %s %s", command, args)
|
||||
}
|
||||
|
||||
executor.MockExecuteCommandWithOutput = func(command string, args ...string) (string, error) {
|
||||
logger.Infof("%s %v", command, args)
|
||||
|
||||
// Validate base common args
|
||||
err := testBaseArgs(args)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
// First command
|
||||
if args[9] == "--osds-per-device" && args[10] == "1" && args[11] == "/dev/sda" && args[12] == "--db-devices" && args[13] == metadataDevicePath {
|
||||
return fmt.Sprintf(`[{"block_db": "%s", "data": "%s"}]`, metadataDevicePath, "/dev/sda"), nil
|
||||
}
|
||||
|
||||
return "", errors.Errorf("unknown command %s %s", command, args)
|
||||
}
|
||||
agent := &OsdAgent{
|
||||
clusterInfo: &cephclient.ClusterInfo{
|
||||
CephVersion: cephver.CephVersion{Major: 17, Minor: 2, Extra: 0},
|
||||
@@ -1995,7 +1942,7 @@ func TestInitializeBlockWithMD(t *testing.T) {
|
||||
assert.NoError(t, err, "failed default behavior test")
|
||||
}
|
||||
|
||||
// Test initialize with LV as metadata devices
|
||||
// Test initialize with LV as metadata device — should use lvm prepare
|
||||
{
|
||||
devices := &DeviceOsdMapping{
|
||||
Entries: map[string]*DeviceOsdIDEntry{
|
||||
@@ -2006,27 +1953,18 @@ func TestInitializeBlockWithMD(t *testing.T) {
|
||||
executor.MockExecuteCommand = func(command string, args ...string) error {
|
||||
logger.Infof("%s %v", command, args)
|
||||
|
||||
// Validate base common args
|
||||
err := testBaseArgs(args)
|
||||
err := testBasePrepareArgs(args)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Second command
|
||||
if args[9] == "--osds-per-device" && args[10] == "1" && args[11] == "/dev/sda" && args[12] == "--db-devices" && args[13] == "/dev/vg0/lv0" {
|
||||
// lvm prepare with LV metadata device
|
||||
if args[7] == "--data" && args[8] == "/dev/sda" && args[9] == "--block.db" && args[10] == "/dev/vg0/lv0" {
|
||||
return nil
|
||||
}
|
||||
|
||||
return errors.Errorf("unknown command %s %s", command, args)
|
||||
}
|
||||
executor.MockExecuteCommandWithOutput = func(command string, args ...string) (string, error) {
|
||||
// First command
|
||||
if args[9] == "--osds-per-device" && args[10] == "1" && args[11] == "/dev/sda" && args[12] == "--db-devices" && args[13] == "/dev/vg0/lv0" && args[14] == "--report" {
|
||||
return `[{"block_db": "vg0/lv0", "encryption": "None", "data": "/dev/sda", "data_size": "100.00 GB", "block_db_size": "10.00 GB"}]`, nil
|
||||
}
|
||||
|
||||
return "", errors.Errorf("unknown command %s %s", command, args)
|
||||
}
|
||||
a := &OsdAgent{clusterInfo: &cephclient.ClusterInfo{CephVersion: cephver.CephVersion{Major: 17, Minor: 2, Extra: 4}}, nodeName: "node1", storeConfig: config.StoreConfig{StoreType: "bluestore"}}
|
||||
context := &clusterd.Context{
|
||||
Executor: executor,
|
||||
@@ -2202,6 +2140,14 @@ func TestIsSafeToUseRawMode(t *testing.T) {
|
||||
assert.True(t, isSafeToUseRawMode(device))
|
||||
})
|
||||
|
||||
t.Run("not safe if device is a logical volume", func(t *testing.T) {
|
||||
device.Config.OSDsPerDevice = 1
|
||||
device.Config.MetadataDevice = ""
|
||||
device.DeviceInfo.Type = sys.LVMType
|
||||
assert.False(t, isSafeToUseRawMode(device))
|
||||
device.DeviceInfo.Type = sys.DiskType
|
||||
})
|
||||
|
||||
t.Run("not safe if OSDs per device > 1", func(t *testing.T) {
|
||||
device.Config.OSDsPerDevice = 2
|
||||
assert.False(t, isSafeToUseRawMode(device))
|
||||
@@ -2229,7 +2175,7 @@ func TestLVMModeAllowed(t *testing.T) {
|
||||
|
||||
// lvm
|
||||
device.DeviceInfo.Type = sys.LVMType
|
||||
assert.False(t, lvmModeAllowed(device, storeConfig))
|
||||
assert.True(t, lvmModeAllowed(device, storeConfig))
|
||||
|
||||
// non-encrypted part
|
||||
device.DeviceInfo.Type = sys.PartType
|
||||
|
||||
Reference in new issue
Block a user