Merge pull request #8319 from BlaineEXE/disable-raw-mode-for-disks
ceph: disable raw mode for disks
This commit is contained in:
10 files changed
+174
-225
No files matched your search
@@ -434,7 +434,8 @@ Below are the settings for host-based cluster. This type of cluster can specify
|
||||
* `name`: The name of the device (e.g., `sda`), or full udev path (e.g. `/dev/disk/by-id/ata-ST4000DM004-XXXX` - this will not change after reboots).
|
||||
* `config`: Device-specific config settings. See the [config settings](#osd-configuration-settings) below
|
||||
|
||||
Host-based cluster only supports raw device and partition.
|
||||
Host-based cluster only supports raw device and partition. Be sure to see the
|
||||
[Ceph quickstart doc prerequisites](ceph-quickstart.md#prerequisites) for additional considerations.
|
||||
|
||||
Below are the settings for a PVC-based cluster.
|
||||
|
||||
|
||||
@@ -958,74 +958,69 @@ You need to create the CRDs found in `cluster/examples/kubernetes/ceph/pre-k8s-1
|
||||
## Unexpected partitions created
|
||||
|
||||
### Symptoms
|
||||
**Rook versions v1.6.0-v1.6.6 may create unwanted OSDs on partitions that appear unexpectedly and
|
||||
seemingly randomly, which will corrupt existing OSDs.**
|
||||
**Users running Rook versions v1.6.0-v1.6.7 may observe unwanted OSDs on partitions that appear
|
||||
unexpectedly and seemingly randomly, which can corrupt existing OSDs.**
|
||||
|
||||
Unexpected partitions are created on host disks that are used by Ceph OSDs. This happens more often
|
||||
on SSDs than HDDs and usually only on disks that are 1TB or larger. Many tools like `lsblk`,
|
||||
`blkid`, and `udevadm` will not show a partition table type for the partition; however, `parted`
|
||||
will reliably list the partition as `atari` (see the example command and output below).
|
||||
on SSDs than HDDs and usually only on disks that are 875GB or larger. Many tools like `lsblk`,
|
||||
`blkid`, `udevadm`, and `parted` will not show a partition table type for the partition. Newer
|
||||
versions of `blkid` are generally able to recognize the type as "atari".
|
||||
|
||||
```console
|
||||
# parted --script /dev/sdb1 print
|
||||
Model: Unknown (unknown)
|
||||
Disk /dev/sdb1: 1509kB
|
||||
Sector size (logical/physical): 512B/512B
|
||||
Partition Table: atari
|
||||
Disk Flags:
|
||||
|
||||
Number Start End Size Type File system Flags
|
||||
|
||||
```
|
||||
|
||||
The underlying issue causing this is Atari (sometimes identified as AHDI) partition support in the
|
||||
Linux kernel. Atari partitions have very relaxed specifications compared to modern partition types,
|
||||
The underlying issue causing this is Atari partition (sometimes identified as AHDI) support in the
|
||||
Linux kernel. Atari partitions have very relaxed specifications compared to other partition types,
|
||||
and it is relatively easy for random data written to a disk to appear as an Atari partition to the
|
||||
Linux kernel. Ceph's Bluestore OSDs have an anecdotally high probability of writing data on to disks
|
||||
that can appear to the kernel as an Atari partition.
|
||||
|
||||
Below is an example of `lsblk` output from a node where phantom Atari partitions are present. Note
|
||||
that `sdX1` is never present for the phantom partitions, and `sdX2` is 48G on all disks. `sdX3`
|
||||
is a variable size and may not always be present. It is possible for `sdX4` to appear, though it is
|
||||
an anecdotally rare event.
|
||||
```
|
||||
# lsblk
|
||||
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
|
||||
sdb 8:16 0 3T 0 disk
|
||||
├─sdb2 8:18 0 48G 0 part
|
||||
└─sdb3 8:19 0 6.1M 0 part
|
||||
sdc 8:32 0 3T 0 disk
|
||||
├─sdc2 8:34 0 48G 0 part
|
||||
└─sdc3 8:35 0 6.2M 0 part
|
||||
sdd 8:48 0 3T 0 disk
|
||||
├─sdd2 8:50 0 48G 0 part
|
||||
└─sdd3 8:51 0 6.3M 0 part
|
||||
```
|
||||
|
||||
You can see https://github.com/rook/rook/issues/7940 for more detailed information and discussion.
|
||||
|
||||
### Solution
|
||||
#### Recover from corruption (v1.6.0-v1.6.6)
|
||||
If you are using Rook v1.6, you must update to v1.6.7 or higher to avoid further incidents of OSD
|
||||
corruption caused by these Atari partitions.
|
||||
#### Recover from corruption (v1.6.0-v1.6.7)
|
||||
If you are using Rook v1.6, you must first update to v1.6.8 or higher to avoid further incidents of
|
||||
OSD corruption caused by these Atari partitions.
|
||||
|
||||
If there are already OSDs created on the unexpected partitions, OSD corruption has already occurred.
|
||||
To recover from this, immediately update to v1.6.7 or higher. After the update, no further OSD
|
||||
corruption should occur. Next, to get back to a healthy Ceph cluster state, focus on one corruped
|
||||
disk at a time and [remove all OSDs on each corrupted disk](ceph-osd-mgmt.md#remove-an-osd) one disk
|
||||
at a time.
|
||||
An old workaround suggested using `deviceFilter: ^sd[a-z]+$`, but this still results in unexpected
|
||||
partitions. Rook will merely stop creating new OSDs on the partitions. It does not fix a related
|
||||
issue that `ceph-volume` that is unaware of the Atari partition problem. Users who used this
|
||||
workaround are still at risk for OSD failures in the future.
|
||||
|
||||
As an example, you may have `/dev/sdb` with two unexpected partitions (`/dev/sdb1` and `/dev/sdb2`)
|
||||
as well as a second corrupted disk `/dev/sde` with one unexpected partition (`/dev/sde1`).
|
||||
1. First, remove the 3 OSDs associated with `/dev/sdb`, `/dev/sdb1`, and `/dev/sdb2`.
|
||||
2. Then wipe clean `/dev/sdb`.
|
||||
3. After this, allow the Rook operator to deploy a new OSD to `/dev/sdb`; this will allow Ceph to
|
||||
To resolve the issue, immediately update to v1.6.8 or higher. After the update, no corruption should
|
||||
occur on OSDs created in the future. Next, to get back to a healthy Ceph cluster state, focus on one
|
||||
corruped disk at a time and [remove all OSDs on each corrupted disk](ceph-osd-mgmt.md#remove-an-osd)
|
||||
one disk at a time.
|
||||
|
||||
As an example, you may have `/dev/sdb` with two unexpected partitions (`/dev/sdb2` and `/dev/sdb3`)
|
||||
as well as a second corrupted disk `/dev/sde` with one unexpected partition (`/dev/sde2`).
|
||||
1. First, remove the OSDs associated with `/dev/sdb`, `/dev/sdb2`, and `/dev/sdb3`. There might be
|
||||
only one, or up to 3 OSDs depending on how your system was affected. Again see the
|
||||
[OSD management doc](ceph-osd-mgmt.md#remove-an-osd).
|
||||
2. Use `dd` to wipe the first sectors of the partitions followed by the disk itself. E.g.,
|
||||
- `dd if=/dev/zero of=/dev/sdb2 bs=1M`
|
||||
- `dd if=/dev/zero of=/dev/sdb3 bs=1M`
|
||||
- `dd if=/dev/zero of=/dev/sdb bs=1M`
|
||||
3. Then wipe clean `/dev/sdb` to prepare it for a new OSD.
|
||||
See [the teardown document](ceph-teardown.md#zapping-devices) for details.
|
||||
4. After this, scale up the Rook operator to deploy a new OSD to `/dev/sdb`. This will allow Ceph to
|
||||
use `/dev/sdb` for data recovery and replication while the next OSDs are removed.
|
||||
4. Now Repeat steps 1-3 for `/dev/sde` and `/dev/sde1`, and continue for any other corruped disks.
|
||||
5. Now Repeat steps 1-4 for `/dev/sde` and `/dev/sde2`, and continue for any other corruped disks.
|
||||
|
||||
If your Rook-Ceph cluster does not have any critical data stored in it, it may be simpler to
|
||||
uninstall Rook completely and redeploy with v1.6.7 or higher.
|
||||
|
||||
#### Handling randomly appearing partitions (v1.6.7+)
|
||||
If you are using Rook v1.6.7 or higher, it is safe to ignore these unexpected `atari` partitions.
|
||||
Rook will not create OSDs on them.
|
||||
|
||||
Not all Linux distributions build kernel with Atari support enabled. If possible, we recommend
|
||||
switching to a kernel built without Atari support enabled. The example below shows how to determine
|
||||
if your kernel was built with Atari support enabled.
|
||||
|
||||
```console
|
||||
$ grep ATARI /boot/config-$(uname -r)
|
||||
CONFIG_ATARI_PARTITION=y
|
||||
```
|
||||
|
||||
The following distributions are known to have Atari support enabled (problematic):
|
||||
- Ubuntu
|
||||
- openSUSE
|
||||
|
||||
The following distribution are known to have Atari support disabled:
|
||||
- CentOS 7 and 8
|
||||
- RHEL 8
|
||||
- openSUSE (kvmsmall kernel on Leap 15.2 and above)
|
||||
uninstall Rook completely and redeploy with v1.6.8 or higher.
|
||||
@@ -24,8 +24,10 @@ To make sure you have a Kubernetes cluster that is ready for `Rook`, you can [fo
|
||||
|
||||
In order to configure the Ceph storage cluster, at least one of these local storage options are required:
|
||||
- Raw devices (no partitions or formatted filesystems)
|
||||
- This requires `lvm2` to be installed on the host.
|
||||
To avoid this dependency, you can create a single full-disk partition on the disk (see below)
|
||||
- Raw partitions (no formatted filesystem)
|
||||
- PVs available from a storage class in `block` mode
|
||||
- Persistent Volumes available from a storage class in `block` mode
|
||||
|
||||
You can confirm whether your partitions or devices are formatted filesystems with the following command.
|
||||
|
||||
|
||||
@@ -324,21 +324,6 @@ func getAvailableDevices(context *clusterd.Context, agent *OsdAgent) (*DeviceOsd
|
||||
continue
|
||||
}
|
||||
|
||||
// raw disks or partitions provisioned with bluestore can sometimes appear to have
|
||||
// Atari (AHDI) partitions created on them. These are phantom partitions that weren't
|
||||
// actually created but appear because the Atari partition spec is too broad. If we
|
||||
// detect an Atari partition, we ignore it because it is likely there is already an OSD
|
||||
// provisioned and running on the disk. If we provision another, we will corrupt the
|
||||
// already-running OSD.
|
||||
isAtari, err := sys.PartitionIsAtari(context.Executor, device.Name)
|
||||
if err != nil {
|
||||
logger.Errorf("failed to determine if partition %q is Atari; skipping it to be safe. %v", device.Name, err)
|
||||
continue
|
||||
} else if isAtari {
|
||||
logger.Infof("skipping Atari partition %q to avoid corrupting an already-running OSD", device.Name)
|
||||
continue
|
||||
}
|
||||
|
||||
device, err := clusterd.PopulateDeviceUdevInfo(device.Name, context.Executor, device)
|
||||
if err != nil {
|
||||
logger.Errorf("failed to get udev info of partition %q. %v", device.Name, err)
|
||||
@@ -390,10 +375,10 @@ func getAvailableDevices(context *clusterd.Context, agent *OsdAgent) (*DeviceOsd
|
||||
var deviceInfo *DeviceOsdIDEntry
|
||||
if agent.metadataDevice != "" && agent.metadataDevice == device.Name {
|
||||
// current device is desired as the metadata device
|
||||
deviceInfo = &DeviceOsdIDEntry{Data: unassignedOSDID, Metadata: []int{}}
|
||||
deviceInfo = &DeviceOsdIDEntry{Data: unassignedOSDID, Metadata: []int{}, DeviceInfo: device}
|
||||
} else if len(desiredDevices) == 1 && desiredDevices[0].Name == "all" {
|
||||
// user has specified all devices, use the current one for data
|
||||
deviceInfo = &DeviceOsdIDEntry{Data: unassignedOSDID}
|
||||
deviceInfo = &DeviceOsdIDEntry{Data: unassignedOSDID, DeviceInfo: device}
|
||||
} else if len(desiredDevices) > 0 {
|
||||
var matched bool
|
||||
var matchedDevice DesiredDevice
|
||||
@@ -460,18 +445,18 @@ func getAvailableDevices(context *clusterd.Context, agent *OsdAgent) (*DeviceOsd
|
||||
if err == nil && matched {
|
||||
// the current device matches the user specifies filter/list, use it for data
|
||||
logger.Infof("device %q is selected by the device filter/name %q", device.Name, matchedDevice.Name)
|
||||
deviceInfo = &DeviceOsdIDEntry{Data: unassignedOSDID, Config: matchedDevice, PersistentDevicePaths: strings.Fields(device.DevLinks)}
|
||||
deviceInfo = &DeviceOsdIDEntry{Data: unassignedOSDID, Config: matchedDevice, PersistentDevicePaths: strings.Fields(device.DevLinks), DeviceInfo: device}
|
||||
|
||||
// set that this is not an OSD but a metadata device
|
||||
if device.Type == pvcMetadataTypeDevice {
|
||||
logger.Infof("metadata device %q is selected by the device filter/name %q", device.Name, matchedDevice.Name)
|
||||
deviceInfo = &DeviceOsdIDEntry{Config: matchedDevice, PersistentDevicePaths: strings.Fields(device.DevLinks), Metadata: []int{1}}
|
||||
deviceInfo = &DeviceOsdIDEntry{Config: matchedDevice, PersistentDevicePaths: strings.Fields(device.DevLinks), Metadata: []int{1}, DeviceInfo: device}
|
||||
}
|
||||
|
||||
// set that this is not an OSD but a wal device
|
||||
if device.Type == pvcWalTypeDevice {
|
||||
logger.Infof("wal device %q is selected by the device filter/name %q", device.Name, matchedDevice.Name)
|
||||
deviceInfo = &DeviceOsdIDEntry{Config: matchedDevice, PersistentDevicePaths: strings.Fields(device.DevLinks), Metadata: []int{2}}
|
||||
deviceInfo = &DeviceOsdIDEntry{Config: matchedDevice, PersistentDevicePaths: strings.Fields(device.DevLinks), Metadata: []int{2}, DeviceInfo: device}
|
||||
}
|
||||
} else {
|
||||
logger.Infof("skipping device %q that does not match the device filter/list (%v). %v", device.Name, desiredDevices, err)
|
||||
|
||||
@@ -170,12 +170,6 @@ USEC_INITIALIZED=1128667
|
||||
]
|
||||
}
|
||||
`
|
||||
|
||||
partedAtariOutput = `BYT;
|
||||
/dev/sdb1:1509kB:unknown:512:512:atari:Unknown:;`
|
||||
|
||||
partedUnknownOutput = `BYT;
|
||||
/dev/sdc1:10.7GB:unknown:512:512:unknown:Unknown:;`
|
||||
)
|
||||
|
||||
func TestAvailableDevices(t *testing.T) {
|
||||
@@ -249,13 +243,6 @@ NAME="sdb1" SIZE="30" TYPE="part" PKNAME="sdb"`, nil
|
||||
}
|
||||
return "{}", nil
|
||||
|
||||
} else if command == "parted" {
|
||||
if strings.Contains(args[2], "sdu1") {
|
||||
return partedAtariOutput, nil
|
||||
} else if strings.Contains(args[2], "sdt1") {
|
||||
// parted returns an error if the partition type is unknown
|
||||
return partedUnknownOutput, errors.New("fake unknown part type err")
|
||||
}
|
||||
}
|
||||
|
||||
return "", errors.Errorf("unknown command %s %s", command, args)
|
||||
@@ -273,7 +260,6 @@ NAME="sdb1" SIZE="30" TYPE="part" PKNAME="sdb"`, nil
|
||||
{Name: "rda", RealPath: "/dev/rda"},
|
||||
{Name: "rdb", RealPath: "/dev/rdb"},
|
||||
{Name: "sdt1", RealPath: "/dev/sdt1", Type: sys.PartType},
|
||||
{Name: "sdu1", RealPath: "/dev/sdu1", Type: sys.PartType}, // is atari
|
||||
{Name: "sdv1", RealPath: "/dev/sdv1", Type: sys.PartType, Filesystem: "ext2"}, // has filesystem
|
||||
}
|
||||
|
||||
@@ -302,7 +288,6 @@ NAME="sdb1" SIZE="30" TYPE="part" PKNAME="sdb"`, nil
|
||||
assert.Equal(t, -1, mapping.Entries["sdt1"].Data)
|
||||
assert.NotContains(t, mapping.Entries, "sdb") // sdb is in use (has a partition)
|
||||
assert.NotContains(t, mapping.Entries, "sdc") // sdc is too small
|
||||
assert.NotContains(t, mapping.Entries, "sdu1") // sdu1 is atari
|
||||
assert.NotContains(t, mapping.Entries, "sdv1") // sdv1 has a filesystem
|
||||
|
||||
// Partition is skipped
|
||||
|
||||
@@ -18,6 +18,8 @@ package osd
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
|
||||
"github.com/rook/rook/pkg/util/sys"
|
||||
)
|
||||
|
||||
const (
|
||||
@@ -58,6 +60,7 @@ type DeviceOsdIDEntry struct {
|
||||
Metadata []int // OSD IDs (multiple) that have metadata stored here
|
||||
Config DesiredDevice // Device specific config options
|
||||
PersistentDevicePaths []string
|
||||
DeviceInfo *sys.LocalDisk // low-level info about the device
|
||||
}
|
||||
|
||||
func (m *DeviceOsdMapping) String() string {
|
||||
|
||||
@@ -234,20 +234,9 @@ func (a *OsdAgent) configureCVDevices(context *clusterd.Context, devices *Device
|
||||
return nil, errors.Wrap(err, "failed to initialize devices on PVC")
|
||||
}
|
||||
} else {
|
||||
// Initialize block device OSD without LVM
|
||||
if useRawMode {
|
||||
logger.Info("initializing osd disk with raw mode")
|
||||
err := a.initializeDevicesRawMode(context, devices)
|
||||
if err != nil {
|
||||
return nil, errors.Wrap(err, "failed to initialize raw based osd")
|
||||
}
|
||||
} else {
|
||||
// Initialize block device OSD with LVM
|
||||
logger.Info("initializing osd disk with lvm mode")
|
||||
err := a.initializeDevicesLVMMode(context, devices)
|
||||
if err != nil {
|
||||
return nil, errors.Wrap(err, "failed to initialize lvm based osd")
|
||||
}
|
||||
err := a.initializeDevices(context, devices, useRawMode)
|
||||
if err != nil {
|
||||
return nil, errors.Wrap(err, "failed to initialize osd")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -544,6 +533,46 @@ func (a *OsdAgent) useRawMode(context *clusterd.Context, pvcBacked bool) (bool,
|
||||
return useRawMode, nil
|
||||
}
|
||||
|
||||
func (a *OsdAgent) initializeDevices(context *clusterd.Context, devices *DeviceOsdMapping, allowRawMode bool) error {
|
||||
// it's a little strange to split this into parts, looping here and in the init functions, but
|
||||
// the LVM mode init requires the ability to loop over all the devices looking for metadata.
|
||||
rawDevices := &DeviceOsdMapping{
|
||||
Entries: map[string]*DeviceOsdIDEntry{},
|
||||
}
|
||||
lvmDevices := &DeviceOsdMapping{
|
||||
Entries: map[string]*DeviceOsdIDEntry{},
|
||||
}
|
||||
|
||||
for name, device := range devices.Entries {
|
||||
// Even if we can use raw mode, do NOT use raw mode on disks. Ceph bluestore disks can
|
||||
// sometimes appear as though they have "phantom" Atari (AHDI) partitions created on them
|
||||
// when they don't in reality. This is due to a series of bugs in the Linux kernel when it
|
||||
// is built with Atari support enabled. This behavior does not appear for raw mode OSDs on
|
||||
// partitions, and we need the raw mode to create partition-based OSDs. We cannot merely
|
||||
// skip creating OSDs on "phantom" partitions due to a bug in `ceph-volume raw inventory`
|
||||
// which reports only the phantom partitions (and malformed OSD info) when they exist and
|
||||
// ignores the original (correct) OSDs created on the raw disk.
|
||||
// See: https://github.com/rook/rook/issues/7940
|
||||
if device.DeviceInfo.Type != sys.DiskType && allowRawMode {
|
||||
rawDevices.Entries[name] = device
|
||||
continue
|
||||
}
|
||||
lvmDevices.Entries[name] = device
|
||||
}
|
||||
|
||||
err := a.initializeDevicesRawMode(context, rawDevices)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
err = a.initializeDevicesLVMMode(context, lvmDevices)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func (a *OsdAgent) initializeDevicesRawMode(context *clusterd.Context, devices *DeviceOsdMapping) error {
|
||||
baseCommand := "stdbuf"
|
||||
cephVolumeMode := "raw"
|
||||
@@ -552,7 +581,7 @@ func (a *OsdAgent) initializeDevicesRawMode(context *clusterd.Context, devices *
|
||||
for name, device := range devices.Entries {
|
||||
deviceArg := path.Join("/dev", name)
|
||||
if device.Data == -1 {
|
||||
logger.Infof("configuring new device %q", deviceArg)
|
||||
logger.Infof("configuring new raw device %q", deviceArg)
|
||||
|
||||
immediateExecuteArgs := append(baseArgs, []string{
|
||||
"--data",
|
||||
@@ -612,7 +641,7 @@ func (a *OsdAgent) initializeDevicesLVMMode(context *clusterd.Context, devices *
|
||||
continue
|
||||
}
|
||||
|
||||
logger.Infof("configuring new device %s", name)
|
||||
logger.Infof("configuring new LVM device %s", name)
|
||||
deviceArg := path.Join("/dev", name)
|
||||
// ceph-volume prefers to use /dev/mapper/<name> if the device has this kind of alias
|
||||
for _, devlink := range device.PersistentDevicePaths {
|
||||
|
||||
@@ -31,6 +31,7 @@ import (
|
||||
cephver "github.com/rook/rook/pkg/operator/ceph/version"
|
||||
"github.com/rook/rook/pkg/operator/test"
|
||||
exectest "github.com/rook/rook/pkg/util/exec/test"
|
||||
"github.com/rook/rook/pkg/util/sys"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
@@ -271,6 +272,16 @@ var cephVolumeRAWTestResult = `{
|
||||
}
|
||||
`
|
||||
|
||||
var cephVolumeRawPartitionTestResult = `{
|
||||
"0": {
|
||||
"ceph_fsid": "4bfe8b72-5e69-4330-b6c0-4d914db8ab89",
|
||||
"device": "/dev/vdb1",
|
||||
"osd_id": 0,
|
||||
"osd_uuid": "c03d7353-96e5-4a41-98de-830dfff97d06",
|
||||
"type": "bluestore"
|
||||
}
|
||||
}`
|
||||
|
||||
func createPVCAvailableDevices() *DeviceOsdMapping {
|
||||
devices := &DeviceOsdMapping{
|
||||
Entries: map[string]*DeviceOsdIDEntry{
|
||||
@@ -550,17 +561,16 @@ func TestConfigureCVDevices(t *testing.T) {
|
||||
}
|
||||
|
||||
{
|
||||
// Test case for a raw mode OSD
|
||||
t.Log("Test case for a raw mode OSD")
|
||||
t.Log("Test case for raw mode on partition")
|
||||
executor := &exectest.MockExecutor{}
|
||||
executor.MockExecuteCommandWithOutput = func(command string, args ...string) (string, error) {
|
||||
logger.Infof("[MockExecuteCommandWithOutput] %s %v", command, args)
|
||||
// get lsblk for disks from cephVolumeRAWTestResult var
|
||||
if command == "lsblk" && (args[0] == "/dev/vdb" || args[0] == "/dev/vdc") {
|
||||
return fmt.Sprintf(`SIZE="17179869184" ROTA="1" RO="0" TYPE="disk" PKNAME="" NAME="%s" KNAME="%s"`, args[0], args[0]), nil
|
||||
if command == "lsblk" && (args[0] == "/dev/vdb1") {
|
||||
return fmt.Sprintf(`SIZE="17179869184" ROTA="1" RO="0" TYPE="part" PKNAME="" NAME="%s" KNAME="%s"`, args[0], args[0]), nil
|
||||
}
|
||||
if args[1] == "ceph-volume" && args[4] == "raw" && args[5] == "list" {
|
||||
return cephVolumeRAWTestResult, nil
|
||||
return cephVolumeRawPartitionTestResult, nil
|
||||
}
|
||||
if args[1] == "ceph-volume" && args[4] == "lvm" && args[5] == "list" {
|
||||
return `{}`, nil
|
||||
@@ -589,13 +599,62 @@ func TestConfigureCVDevices(t *testing.T) {
|
||||
agent := &OsdAgent{clusterInfo: clusterInfo, nodeName: nodeName, storeConfig: config.StoreConfig{DeviceClass: "myclass"}}
|
||||
devices := &DeviceOsdMapping{
|
||||
Entries: map[string]*DeviceOsdIDEntry{
|
||||
"vdb": {Data: -1, Metadata: nil, Config: DesiredDevice{Name: "/dev/vdb"}},
|
||||
"vdb1": {Data: -1, Metadata: nil, Config: DesiredDevice{Name: "/dev/vdb1"}, DeviceInfo: &sys.LocalDisk{Type: sys.PartType}},
|
||||
},
|
||||
}
|
||||
_, err := agent.configureCVDevices(context, devices)
|
||||
assert.Nil(t, err)
|
||||
assert.True(t, deviceClassSet)
|
||||
}
|
||||
|
||||
// disabled while raw mode is disabled for disks
|
||||
// {
|
||||
// // Test case for a raw mode OSD
|
||||
// t.Log("Test case for a raw mode OSD")
|
||||
// executor := &exectest.MockExecutor{}
|
||||
// executor.MockExecuteCommandWithOutput = func(command string, args ...string) (string, error) {
|
||||
// logger.Infof("[MockExecuteCommandWithOutput] %s %v", command, args)
|
||||
// // get lsblk for disks from cephVolumeRAWTestResult var
|
||||
// if command == "lsblk" && (args[0] == "/dev/vdb" || args[0] == "/dev/vdc") {
|
||||
// return fmt.Sprintf(`SIZE="17179869184" ROTA="1" RO="0" TYPE="disk" PKNAME="" NAME="%s" KNAME="%s"`, args[0], args[0]), nil
|
||||
// }
|
||||
// if args[1] == "ceph-volume" && args[4] == "raw" && args[5] == "list" {
|
||||
// return cephVolumeRAWTestResult, nil
|
||||
// }
|
||||
// if args[1] == "ceph-volume" && args[4] == "lvm" && args[5] == "list" {
|
||||
// return `{}`, nil
|
||||
// }
|
||||
// if command == "sgdisk" {
|
||||
// return "Disk identifier (GUID): 18484D7E-5287-4CE9-AC73-D02FB69055CE", nil
|
||||
// }
|
||||
// return "", errors.Errorf("unknown command %s %s", command, args)
|
||||
// }
|
||||
// deviceClassSet := false
|
||||
// executor.MockExecuteCommandWithCombinedOutput = func(command string, args ...string) (string, error) {
|
||||
// logger.Infof("[MockExecuteCommandWithCombinedOutput] %s %v", command, args)
|
||||
// if args[1] == "ceph-volume" && args[2] == "raw" && args[3] == "prepare" && args[4] == "--bluestore" && args[7] == "--crush-device-class" {
|
||||
// assert.Equal(t, "myclass", args[8])
|
||||
// deviceClassSet = true
|
||||
// return "", nil
|
||||
// }
|
||||
// return "", errors.Errorf("unknown command %s %s", command, args)
|
||||
// }
|
||||
|
||||
// context := &clusterd.Context{Executor: executor, ConfigDir: cephConfigDir}
|
||||
// clusterInfo := &cephclient.ClusterInfo{
|
||||
// CephVersion: cephver.CephVersion{Major: 16, Minor: 2, Extra: 1}, // It supports raw mode OSD
|
||||
// FSID: clusterFSID,
|
||||
// }
|
||||
// agent := &OsdAgent{clusterInfo: clusterInfo, nodeName: nodeName, storeConfig: config.StoreConfig{DeviceClass: "myclass"}}
|
||||
// devices := &DeviceOsdMapping{
|
||||
// Entries: map[string]*DeviceOsdIDEntry{
|
||||
// "vdb": {Data: -1, Metadata: nil, Config: DesiredDevice{Name: "/dev/vdb"}},
|
||||
// },
|
||||
// }
|
||||
// _, err := agent.configureCVDevices(context, devices)
|
||||
// assert.Nil(t, err)
|
||||
// assert.True(t, deviceClassSet)
|
||||
// }
|
||||
}
|
||||
|
||||
func testBaseArgs(args []string) error {
|
||||
|
||||
@@ -472,38 +472,3 @@ func ListDevicesChild(executor exec.Executor, device string) ([]string, error) {
|
||||
|
||||
return strings.Split(childListRaw, "\n"), nil
|
||||
}
|
||||
|
||||
// PartitionIsAtari returns true if the device partition is an Atari type.
|
||||
// Ceph bluestore raw disks can sometimes appear as though they have ATARI (AHDI)
|
||||
// partitions created on them when they don't in reality.
|
||||
// See: https://github.com/rook/rook/issues/7940
|
||||
func PartitionIsAtari(executor exec.Executor, device string) (bool, error) {
|
||||
// the command error is not processed the same way as most Go errors. parted returns an error
|
||||
// when the partition table type is unknown, so always process the raw output, even in the case
|
||||
// of a command error
|
||||
raw, cmdErr := executor.ExecuteCommandWithOutput("parted", "--machine", "--script", path.Join("/dev", device), "print")
|
||||
// --machine gives machine-parseable output: https://alioth-lists.debian.net/pipermail/parted-devel/2006-December/000573.html
|
||||
// HEADER-TYPE;
|
||||
// DISK-FULL-PATH:...:PART-TABLE-TYPE:PART-TABLE-NAME:FLAGS-SET;
|
||||
// FS INFO;
|
||||
// always 6 fields or more and on the second line, but we only care about the 3rd to last field
|
||||
raw = strings.TrimSpace(raw) // trim begin/end newlines if exist
|
||||
lines := strings.Split(raw, "\n")
|
||||
if len(lines) < 2 {
|
||||
return true, fmt.Errorf("failed to get 'parted' info for device %q. expect at least two lines of output: %q. %v", device, raw, cmdErr)
|
||||
}
|
||||
rawInfo := lines[1]
|
||||
if rawInfo[len(rawInfo)-1] != ';' {
|
||||
// the last char should be a semicolon
|
||||
return true, fmt.Errorf("failed to get 'parted' info for device %q. expect a semicolon at end: %q. %v", device, rawInfo, cmdErr)
|
||||
}
|
||||
rawInfo = strings.TrimRight(rawInfo, ";") // trim the right semicolon
|
||||
items := strings.Split(rawInfo, ":")
|
||||
if len(items) < 6 {
|
||||
// expect at least 6 fields
|
||||
return true, fmt.Errorf("failed to get 'parted' info for device %q. expect at least 6 fields: %q. %v", device, rawInfo, cmdErr)
|
||||
}
|
||||
pType := items[len(items)-3] // 3rd to last field
|
||||
|
||||
return pType == "atari", nil
|
||||
}
|
||||
@@ -196,78 +196,3 @@ func TestListDevicesChildListDevicesChild(t *testing.T) {
|
||||
assert.NoError(t, err)
|
||||
assert.Equal(t, 3, len(child))
|
||||
}
|
||||
func TestPartitionIsAtari(t *testing.T) {
|
||||
// This also implicitly tests PartitionIsAtari
|
||||
|
||||
partedAtariOutput := `BYT;
|
||||
/dev/sdb1:1509kB:unknown:512:512:atari:Unknown:;`
|
||||
|
||||
partedNonAtariOutput := `BYT;
|
||||
/dev/sda2:2048MB:unknown:512:512:loop:Unknown:;
|
||||
1:0.00B:2048MB:2048MB:linux-swap(v1)::;`
|
||||
|
||||
partedUnknownOutput := `BYT;
|
||||
/dev/sdc1:10.7GB:unknown:512:512:unknown:Unknown:;`
|
||||
|
||||
executor := &exectest.MockExecutor{
|
||||
MockExecuteCommandWithOutput: func(command string, arg ...string) (string, error) {
|
||||
t.Log("command:", command, arg)
|
||||
switch arg[2] {
|
||||
case "/dev/vdb1": // is atari
|
||||
return partedAtariOutput, nil
|
||||
case "/dev/vdb2": // not atari
|
||||
return partedNonAtariOutput, nil
|
||||
case "/dev/vdb3": // unknown type (has error)
|
||||
// parted returns an error if the partition type is unknown, but it still outputs
|
||||
// info to stdout that can be parsed successfully
|
||||
return partedUnknownOutput, fmt.Errorf("fake error about unrecognized disk label")
|
||||
case "/dev/vdb4": // too few lines (none), and error (likely in real world)
|
||||
return "", fmt.Errorf("fake error") // too few output lines (none)
|
||||
case "/dev/vdb5": // too few fields, no error (unlikely in real world)
|
||||
return `BYT;
|
||||
/dev/sdc1:Unknown:;`, nil
|
||||
case "/dev/vdb6": // no semicolon
|
||||
return `BYT;
|
||||
there:is:not:a:semicolon:after:this:line:`, nil
|
||||
default:
|
||||
panic("unhandled command")
|
||||
}
|
||||
},
|
||||
}
|
||||
|
||||
t.Run("is atari", func(t *testing.T) {
|
||||
a, err := PartitionIsAtari(executor, "vdb1")
|
||||
assert.NoError(t, err)
|
||||
assert.True(t, a)
|
||||
})
|
||||
|
||||
t.Run("not atari", func(t *testing.T) {
|
||||
a, err := PartitionIsAtari(executor, "vdb2")
|
||||
assert.NoError(t, err)
|
||||
assert.False(t, a)
|
||||
})
|
||||
|
||||
t.Run("unknown partition type output", func(t *testing.T) {
|
||||
a, err := PartitionIsAtari(executor, "vdb3")
|
||||
assert.NoError(t, err)
|
||||
assert.False(t, a)
|
||||
})
|
||||
|
||||
t.Run("too few output lines", func(t *testing.T) {
|
||||
_, err := PartitionIsAtari(executor, "vdb4")
|
||||
assert.Error(t, err)
|
||||
assert.Contains(t, err.Error(), "lines") // minimal check for error message content
|
||||
})
|
||||
|
||||
t.Run("too few output fields", func(t *testing.T) {
|
||||
_, err := PartitionIsAtari(executor, "vdb5")
|
||||
assert.Error(t, err)
|
||||
assert.Contains(t, err.Error(), "fields") // minimal check for error message content
|
||||
})
|
||||
|
||||
t.Run("no semicolon after fields", func(t *testing.T) {
|
||||
_, err := PartitionIsAtari(executor, "vdb6")
|
||||
assert.Error(t, err)
|
||||
assert.Contains(t, err.Error(), "semicolon") // minimal check for error message content
|
||||
})
|
||||
}
|
||||
Reference in new issue
Block a user