diff --git a/Documentation/ceph-cluster-crd.md b/Documentation/ceph-cluster-crd.md index 375d346cc..ab94e0fba 100755 --- a/Documentation/ceph-cluster-crd.md +++ b/Documentation/ceph-cluster-crd.md @@ -434,7 +434,8 @@ Below are the settings for host-based cluster. This type of cluster can specify * `name`: The name of the device (e.g., `sda`), or full udev path (e.g. `/dev/disk/by-id/ata-ST4000DM004-XXXX` - this will not change after reboots). * `config`: Device-specific config settings. See the [config settings](#osd-configuration-settings) below -Host-based cluster only supports raw device and partition. +Host-based cluster only supports raw device and partition. Be sure to see the +[Ceph quickstart doc prerequisites](ceph-quickstart.md#prerequisites) for additional considerations. Below are the settings for a PVC-based cluster. diff --git a/Documentation/ceph-common-issues.md b/Documentation/ceph-common-issues.md index ae0e5e74e..0db2b8310 100644 --- a/Documentation/ceph-common-issues.md +++ b/Documentation/ceph-common-issues.md @@ -958,74 +958,69 @@ You need to create the CRDs found in `cluster/examples/kubernetes/ceph/pre-k8s-1 ## Unexpected partitions created ### Symptoms -**Rook versions v1.6.0-v1.6.6 may create unwanted OSDs on partitions that appear unexpectedly and -seemingly randomly, which will corrupt existing OSDs.** +**Users running Rook versions v1.6.0-v1.6.7 may observe unwanted OSDs on partitions that appear +unexpectedly and seemingly randomly, which can corrupt existing OSDs.** Unexpected partitions are created on host disks that are used by Ceph OSDs. This happens more often -on SSDs than HDDs and usually only on disks that are 1TB or larger. Many tools like `lsblk`, -`blkid`, and `udevadm` will not show a partition table type for the partition; however, `parted` -will reliably list the partition as `atari` (see the example command and output below). +on SSDs than HDDs and usually only on disks that are 875GB or larger. Many tools like `lsblk`, +`blkid`, `udevadm`, and `parted` will not show a partition table type for the partition. Newer +versions of `blkid` are generally able to recognize the type as "atari". -```console -# parted --script /dev/sdb1 print -Model: Unknown (unknown) -Disk /dev/sdb1: 1509kB -Sector size (logical/physical): 512B/512B -Partition Table: atari -Disk Flags: - -Number Start End Size Type File system Flags - -``` - -The underlying issue causing this is Atari (sometimes identified as AHDI) partition support in the -Linux kernel. Atari partitions have very relaxed specifications compared to modern partition types, +The underlying issue causing this is Atari partition (sometimes identified as AHDI) support in the +Linux kernel. Atari partitions have very relaxed specifications compared to other partition types, and it is relatively easy for random data written to a disk to appear as an Atari partition to the Linux kernel. Ceph's Bluestore OSDs have an anecdotally high probability of writing data on to disks that can appear to the kernel as an Atari partition. +Below is an example of `lsblk` output from a node where phantom Atari partitions are present. Note +that `sdX1` is never present for the phantom partitions, and `sdX2` is 48G on all disks. `sdX3` +is a variable size and may not always be present. It is possible for `sdX4` to appear, though it is +an anecdotally rare event. +``` +# lsblk +NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT +sdb 8:16 0 3T 0 disk +├─sdb2 8:18 0 48G 0 part +└─sdb3 8:19 0 6.1M 0 part +sdc 8:32 0 3T 0 disk +├─sdc2 8:34 0 48G 0 part +└─sdc3 8:35 0 6.2M 0 part +sdd 8:48 0 3T 0 disk +├─sdd2 8:50 0 48G 0 part +└─sdd3 8:51 0 6.3M 0 part +``` + You can see https://github.com/rook/rook/issues/7940 for more detailed information and discussion. ### Solution -#### Recover from corruption (v1.6.0-v1.6.6) -If you are using Rook v1.6, you must update to v1.6.7 or higher to avoid further incidents of OSD -corruption caused by these Atari partitions. +#### Recover from corruption (v1.6.0-v1.6.7) +If you are using Rook v1.6, you must first update to v1.6.8 or higher to avoid further incidents of +OSD corruption caused by these Atari partitions. -If there are already OSDs created on the unexpected partitions, OSD corruption has already occurred. -To recover from this, immediately update to v1.6.7 or higher. After the update, no further OSD -corruption should occur. Next, to get back to a healthy Ceph cluster state, focus on one corruped -disk at a time and [remove all OSDs on each corrupted disk](ceph-osd-mgmt.md#remove-an-osd) one disk -at a time. +An old workaround suggested using `deviceFilter: ^sd[a-z]+$`, but this still results in unexpected +partitions. Rook will merely stop creating new OSDs on the partitions. It does not fix a related +issue that `ceph-volume` that is unaware of the Atari partition problem. Users who used this +workaround are still at risk for OSD failures in the future. -As an example, you may have `/dev/sdb` with two unexpected partitions (`/dev/sdb1` and `/dev/sdb2`) -as well as a second corrupted disk `/dev/sde` with one unexpected partition (`/dev/sde1`). -1. First, remove the 3 OSDs associated with `/dev/sdb`, `/dev/sdb1`, and `/dev/sdb2`. -2. Then wipe clean `/dev/sdb`. -3. After this, allow the Rook operator to deploy a new OSD to `/dev/sdb`; this will allow Ceph to +To resolve the issue, immediately update to v1.6.8 or higher. After the update, no corruption should +occur on OSDs created in the future. Next, to get back to a healthy Ceph cluster state, focus on one +corruped disk at a time and [remove all OSDs on each corrupted disk](ceph-osd-mgmt.md#remove-an-osd) +one disk at a time. + +As an example, you may have `/dev/sdb` with two unexpected partitions (`/dev/sdb2` and `/dev/sdb3`) +as well as a second corrupted disk `/dev/sde` with one unexpected partition (`/dev/sde2`). +1. First, remove the OSDs associated with `/dev/sdb`, `/dev/sdb2`, and `/dev/sdb3`. There might be + only one, or up to 3 OSDs depending on how your system was affected. Again see the + [OSD management doc](ceph-osd-mgmt.md#remove-an-osd). +2. Use `dd` to wipe the first sectors of the partitions followed by the disk itself. E.g., + - `dd if=/dev/zero of=/dev/sdb2 bs=1M` + - `dd if=/dev/zero of=/dev/sdb3 bs=1M` + - `dd if=/dev/zero of=/dev/sdb bs=1M` +3. Then wipe clean `/dev/sdb` to prepare it for a new OSD. + See [the teardown document](ceph-teardown.md#zapping-devices) for details. +4. After this, scale up the Rook operator to deploy a new OSD to `/dev/sdb`. This will allow Ceph to use `/dev/sdb` for data recovery and replication while the next OSDs are removed. -4. Now Repeat steps 1-3 for `/dev/sde` and `/dev/sde1`, and continue for any other corruped disks. +5. Now Repeat steps 1-4 for `/dev/sde` and `/dev/sde2`, and continue for any other corruped disks. If your Rook-Ceph cluster does not have any critical data stored in it, it may be simpler to -uninstall Rook completely and redeploy with v1.6.7 or higher. - -#### Handling randomly appearing partitions (v1.6.7+) -If you are using Rook v1.6.7 or higher, it is safe to ignore these unexpected `atari` partitions. -Rook will not create OSDs on them. - -Not all Linux distributions build kernel with Atari support enabled. If possible, we recommend -switching to a kernel built without Atari support enabled. The example below shows how to determine -if your kernel was built with Atari support enabled. - -```console -$ grep ATARI /boot/config-$(uname -r) -CONFIG_ATARI_PARTITION=y -``` - -The following distributions are known to have Atari support enabled (problematic): -- Ubuntu -- openSUSE - -The following distribution are known to have Atari support disabled: -- CentOS 7 and 8 -- RHEL 8 -- openSUSE (kvmsmall kernel on Leap 15.2 and above) +uninstall Rook completely and redeploy with v1.6.8 or higher. diff --git a/Documentation/ceph-quickstart.md b/Documentation/ceph-quickstart.md index 8faeed27d..51fa217cb 100644 --- a/Documentation/ceph-quickstart.md +++ b/Documentation/ceph-quickstart.md @@ -24,8 +24,10 @@ To make sure you have a Kubernetes cluster that is ready for `Rook`, you can [fo In order to configure the Ceph storage cluster, at least one of these local storage options are required: - Raw devices (no partitions or formatted filesystems) + - This requires `lvm2` to be installed on the host. + To avoid this dependency, you can create a single full-disk partition on the disk (see below) - Raw partitions (no formatted filesystem) -- PVs available from a storage class in `block` mode +- Persistent Volumes available from a storage class in `block` mode You can confirm whether your partitions or devices are formatted filesystems with the following command. diff --git a/pkg/daemon/ceph/osd/daemon.go b/pkg/daemon/ceph/osd/daemon.go index f9b3705c4..3eeabea34 100644 --- a/pkg/daemon/ceph/osd/daemon.go +++ b/pkg/daemon/ceph/osd/daemon.go @@ -324,21 +324,6 @@ func getAvailableDevices(context *clusterd.Context, agent *OsdAgent) (*DeviceOsd continue } - // raw disks or partitions provisioned with bluestore can sometimes appear to have - // Atari (AHDI) partitions created on them. These are phantom partitions that weren't - // actually created but appear because the Atari partition spec is too broad. If we - // detect an Atari partition, we ignore it because it is likely there is already an OSD - // provisioned and running on the disk. If we provision another, we will corrupt the - // already-running OSD. - isAtari, err := sys.PartitionIsAtari(context.Executor, device.Name) - if err != nil { - logger.Errorf("failed to determine if partition %q is Atari; skipping it to be safe. %v", device.Name, err) - continue - } else if isAtari { - logger.Infof("skipping Atari partition %q to avoid corrupting an already-running OSD", device.Name) - continue - } - device, err := clusterd.PopulateDeviceUdevInfo(device.Name, context.Executor, device) if err != nil { logger.Errorf("failed to get udev info of partition %q. %v", device.Name, err) @@ -390,10 +375,10 @@ func getAvailableDevices(context *clusterd.Context, agent *OsdAgent) (*DeviceOsd var deviceInfo *DeviceOsdIDEntry if agent.metadataDevice != "" && agent.metadataDevice == device.Name { // current device is desired as the metadata device - deviceInfo = &DeviceOsdIDEntry{Data: unassignedOSDID, Metadata: []int{}} + deviceInfo = &DeviceOsdIDEntry{Data: unassignedOSDID, Metadata: []int{}, DeviceInfo: device} } else if len(desiredDevices) == 1 && desiredDevices[0].Name == "all" { // user has specified all devices, use the current one for data - deviceInfo = &DeviceOsdIDEntry{Data: unassignedOSDID} + deviceInfo = &DeviceOsdIDEntry{Data: unassignedOSDID, DeviceInfo: device} } else if len(desiredDevices) > 0 { var matched bool var matchedDevice DesiredDevice @@ -460,18 +445,18 @@ func getAvailableDevices(context *clusterd.Context, agent *OsdAgent) (*DeviceOsd if err == nil && matched { // the current device matches the user specifies filter/list, use it for data logger.Infof("device %q is selected by the device filter/name %q", device.Name, matchedDevice.Name) - deviceInfo = &DeviceOsdIDEntry{Data: unassignedOSDID, Config: matchedDevice, PersistentDevicePaths: strings.Fields(device.DevLinks)} + deviceInfo = &DeviceOsdIDEntry{Data: unassignedOSDID, Config: matchedDevice, PersistentDevicePaths: strings.Fields(device.DevLinks), DeviceInfo: device} // set that this is not an OSD but a metadata device if device.Type == pvcMetadataTypeDevice { logger.Infof("metadata device %q is selected by the device filter/name %q", device.Name, matchedDevice.Name) - deviceInfo = &DeviceOsdIDEntry{Config: matchedDevice, PersistentDevicePaths: strings.Fields(device.DevLinks), Metadata: []int{1}} + deviceInfo = &DeviceOsdIDEntry{Config: matchedDevice, PersistentDevicePaths: strings.Fields(device.DevLinks), Metadata: []int{1}, DeviceInfo: device} } // set that this is not an OSD but a wal device if device.Type == pvcWalTypeDevice { logger.Infof("wal device %q is selected by the device filter/name %q", device.Name, matchedDevice.Name) - deviceInfo = &DeviceOsdIDEntry{Config: matchedDevice, PersistentDevicePaths: strings.Fields(device.DevLinks), Metadata: []int{2}} + deviceInfo = &DeviceOsdIDEntry{Config: matchedDevice, PersistentDevicePaths: strings.Fields(device.DevLinks), Metadata: []int{2}, DeviceInfo: device} } } else { logger.Infof("skipping device %q that does not match the device filter/list (%v). %v", device.Name, desiredDevices, err) diff --git a/pkg/daemon/ceph/osd/daemon_test.go b/pkg/daemon/ceph/osd/daemon_test.go index 07b43f2d6..12d7315c6 100644 --- a/pkg/daemon/ceph/osd/daemon_test.go +++ b/pkg/daemon/ceph/osd/daemon_test.go @@ -170,12 +170,6 @@ USEC_INITIALIZED=1128667 ] } ` - - partedAtariOutput = `BYT; -/dev/sdb1:1509kB:unknown:512:512:atari:Unknown:;` - - partedUnknownOutput = `BYT; -/dev/sdc1:10.7GB:unknown:512:512:unknown:Unknown:;` ) func TestAvailableDevices(t *testing.T) { @@ -249,13 +243,6 @@ NAME="sdb1" SIZE="30" TYPE="part" PKNAME="sdb"`, nil } return "{}", nil - } else if command == "parted" { - if strings.Contains(args[2], "sdu1") { - return partedAtariOutput, nil - } else if strings.Contains(args[2], "sdt1") { - // parted returns an error if the partition type is unknown - return partedUnknownOutput, errors.New("fake unknown part type err") - } } return "", errors.Errorf("unknown command %s %s", command, args) @@ -273,7 +260,6 @@ NAME="sdb1" SIZE="30" TYPE="part" PKNAME="sdb"`, nil {Name: "rda", RealPath: "/dev/rda"}, {Name: "rdb", RealPath: "/dev/rdb"}, {Name: "sdt1", RealPath: "/dev/sdt1", Type: sys.PartType}, - {Name: "sdu1", RealPath: "/dev/sdu1", Type: sys.PartType}, // is atari {Name: "sdv1", RealPath: "/dev/sdv1", Type: sys.PartType, Filesystem: "ext2"}, // has filesystem } @@ -302,7 +288,6 @@ NAME="sdb1" SIZE="30" TYPE="part" PKNAME="sdb"`, nil assert.Equal(t, -1, mapping.Entries["sdt1"].Data) assert.NotContains(t, mapping.Entries, "sdb") // sdb is in use (has a partition) assert.NotContains(t, mapping.Entries, "sdc") // sdc is too small - assert.NotContains(t, mapping.Entries, "sdu1") // sdu1 is atari assert.NotContains(t, mapping.Entries, "sdv1") // sdv1 has a filesystem // Partition is skipped diff --git a/pkg/daemon/ceph/osd/device.go b/pkg/daemon/ceph/osd/device.go index 29e937487..152666502 100644 --- a/pkg/daemon/ceph/osd/device.go +++ b/pkg/daemon/ceph/osd/device.go @@ -18,6 +18,8 @@ package osd import ( "encoding/json" + + "github.com/rook/rook/pkg/util/sys" ) const ( @@ -58,6 +60,7 @@ type DeviceOsdIDEntry struct { Metadata []int // OSD IDs (multiple) that have metadata stored here Config DesiredDevice // Device specific config options PersistentDevicePaths []string + DeviceInfo *sys.LocalDisk // low-level info about the device } func (m *DeviceOsdMapping) String() string { diff --git a/pkg/daemon/ceph/osd/volume.go b/pkg/daemon/ceph/osd/volume.go index 8dfd8df4a..0dda7c356 100644 --- a/pkg/daemon/ceph/osd/volume.go +++ b/pkg/daemon/ceph/osd/volume.go @@ -234,20 +234,9 @@ func (a *OsdAgent) configureCVDevices(context *clusterd.Context, devices *Device return nil, errors.Wrap(err, "failed to initialize devices on PVC") } } else { - // Initialize block device OSD without LVM - if useRawMode { - logger.Info("initializing osd disk with raw mode") - err := a.initializeDevicesRawMode(context, devices) - if err != nil { - return nil, errors.Wrap(err, "failed to initialize raw based osd") - } - } else { - // Initialize block device OSD with LVM - logger.Info("initializing osd disk with lvm mode") - err := a.initializeDevicesLVMMode(context, devices) - if err != nil { - return nil, errors.Wrap(err, "failed to initialize lvm based osd") - } + err := a.initializeDevices(context, devices, useRawMode) + if err != nil { + return nil, errors.Wrap(err, "failed to initialize osd") } } @@ -544,6 +533,46 @@ func (a *OsdAgent) useRawMode(context *clusterd.Context, pvcBacked bool) (bool, return useRawMode, nil } +func (a *OsdAgent) initializeDevices(context *clusterd.Context, devices *DeviceOsdMapping, allowRawMode bool) error { + // it's a little strange to split this into parts, looping here and in the init functions, but + // the LVM mode init requires the ability to loop over all the devices looking for metadata. + rawDevices := &DeviceOsdMapping{ + Entries: map[string]*DeviceOsdIDEntry{}, + } + lvmDevices := &DeviceOsdMapping{ + Entries: map[string]*DeviceOsdIDEntry{}, + } + + for name, device := range devices.Entries { + // Even if we can use raw mode, do NOT use raw mode on disks. Ceph bluestore disks can + // sometimes appear as though they have "phantom" Atari (AHDI) partitions created on them + // when they don't in reality. This is due to a series of bugs in the Linux kernel when it + // is built with Atari support enabled. This behavior does not appear for raw mode OSDs on + // partitions, and we need the raw mode to create partition-based OSDs. We cannot merely + // skip creating OSDs on "phantom" partitions due to a bug in `ceph-volume raw inventory` + // which reports only the phantom partitions (and malformed OSD info) when they exist and + // ignores the original (correct) OSDs created on the raw disk. + // See: https://github.com/rook/rook/issues/7940 + if device.DeviceInfo.Type != sys.DiskType && allowRawMode { + rawDevices.Entries[name] = device + continue + } + lvmDevices.Entries[name] = device + } + + err := a.initializeDevicesRawMode(context, rawDevices) + if err != nil { + return err + } + + err = a.initializeDevicesLVMMode(context, lvmDevices) + if err != nil { + return err + } + + return nil +} + func (a *OsdAgent) initializeDevicesRawMode(context *clusterd.Context, devices *DeviceOsdMapping) error { baseCommand := "stdbuf" cephVolumeMode := "raw" @@ -552,7 +581,7 @@ func (a *OsdAgent) initializeDevicesRawMode(context *clusterd.Context, devices * for name, device := range devices.Entries { deviceArg := path.Join("/dev", name) if device.Data == -1 { - logger.Infof("configuring new device %q", deviceArg) + logger.Infof("configuring new raw device %q", deviceArg) immediateExecuteArgs := append(baseArgs, []string{ "--data", @@ -612,7 +641,7 @@ func (a *OsdAgent) initializeDevicesLVMMode(context *clusterd.Context, devices * continue } - logger.Infof("configuring new device %s", name) + logger.Infof("configuring new LVM device %s", name) deviceArg := path.Join("/dev", name) // ceph-volume prefers to use /dev/mapper/ if the device has this kind of alias for _, devlink := range device.PersistentDevicePaths { diff --git a/pkg/daemon/ceph/osd/volume_test.go b/pkg/daemon/ceph/osd/volume_test.go index 183b1069c..35a857caf 100644 --- a/pkg/daemon/ceph/osd/volume_test.go +++ b/pkg/daemon/ceph/osd/volume_test.go @@ -31,6 +31,7 @@ import ( cephver "github.com/rook/rook/pkg/operator/ceph/version" "github.com/rook/rook/pkg/operator/test" exectest "github.com/rook/rook/pkg/util/exec/test" + "github.com/rook/rook/pkg/util/sys" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" ) @@ -271,6 +272,16 @@ var cephVolumeRAWTestResult = `{ } ` +var cephVolumeRawPartitionTestResult = `{ + "0": { + "ceph_fsid": "4bfe8b72-5e69-4330-b6c0-4d914db8ab89", + "device": "/dev/vdb1", + "osd_id": 0, + "osd_uuid": "c03d7353-96e5-4a41-98de-830dfff97d06", + "type": "bluestore" + } +}` + func createPVCAvailableDevices() *DeviceOsdMapping { devices := &DeviceOsdMapping{ Entries: map[string]*DeviceOsdIDEntry{ @@ -550,17 +561,16 @@ func TestConfigureCVDevices(t *testing.T) { } { - // Test case for a raw mode OSD - t.Log("Test case for a raw mode OSD") + t.Log("Test case for raw mode on partition") executor := &exectest.MockExecutor{} executor.MockExecuteCommandWithOutput = func(command string, args ...string) (string, error) { logger.Infof("[MockExecuteCommandWithOutput] %s %v", command, args) // get lsblk for disks from cephVolumeRAWTestResult var - if command == "lsblk" && (args[0] == "/dev/vdb" || args[0] == "/dev/vdc") { - return fmt.Sprintf(`SIZE="17179869184" ROTA="1" RO="0" TYPE="disk" PKNAME="" NAME="%s" KNAME="%s"`, args[0], args[0]), nil + if command == "lsblk" && (args[0] == "/dev/vdb1") { + return fmt.Sprintf(`SIZE="17179869184" ROTA="1" RO="0" TYPE="part" PKNAME="" NAME="%s" KNAME="%s"`, args[0], args[0]), nil } if args[1] == "ceph-volume" && args[4] == "raw" && args[5] == "list" { - return cephVolumeRAWTestResult, nil + return cephVolumeRawPartitionTestResult, nil } if args[1] == "ceph-volume" && args[4] == "lvm" && args[5] == "list" { return `{}`, nil @@ -589,13 +599,62 @@ func TestConfigureCVDevices(t *testing.T) { agent := &OsdAgent{clusterInfo: clusterInfo, nodeName: nodeName, storeConfig: config.StoreConfig{DeviceClass: "myclass"}} devices := &DeviceOsdMapping{ Entries: map[string]*DeviceOsdIDEntry{ - "vdb": {Data: -1, Metadata: nil, Config: DesiredDevice{Name: "/dev/vdb"}}, + "vdb1": {Data: -1, Metadata: nil, Config: DesiredDevice{Name: "/dev/vdb1"}, DeviceInfo: &sys.LocalDisk{Type: sys.PartType}}, }, } _, err := agent.configureCVDevices(context, devices) assert.Nil(t, err) assert.True(t, deviceClassSet) } + + // disabled while raw mode is disabled for disks + // { + // // Test case for a raw mode OSD + // t.Log("Test case for a raw mode OSD") + // executor := &exectest.MockExecutor{} + // executor.MockExecuteCommandWithOutput = func(command string, args ...string) (string, error) { + // logger.Infof("[MockExecuteCommandWithOutput] %s %v", command, args) + // // get lsblk for disks from cephVolumeRAWTestResult var + // if command == "lsblk" && (args[0] == "/dev/vdb" || args[0] == "/dev/vdc") { + // return fmt.Sprintf(`SIZE="17179869184" ROTA="1" RO="0" TYPE="disk" PKNAME="" NAME="%s" KNAME="%s"`, args[0], args[0]), nil + // } + // if args[1] == "ceph-volume" && args[4] == "raw" && args[5] == "list" { + // return cephVolumeRAWTestResult, nil + // } + // if args[1] == "ceph-volume" && args[4] == "lvm" && args[5] == "list" { + // return `{}`, nil + // } + // if command == "sgdisk" { + // return "Disk identifier (GUID): 18484D7E-5287-4CE9-AC73-D02FB69055CE", nil + // } + // return "", errors.Errorf("unknown command %s %s", command, args) + // } + // deviceClassSet := false + // executor.MockExecuteCommandWithCombinedOutput = func(command string, args ...string) (string, error) { + // logger.Infof("[MockExecuteCommandWithCombinedOutput] %s %v", command, args) + // if args[1] == "ceph-volume" && args[2] == "raw" && args[3] == "prepare" && args[4] == "--bluestore" && args[7] == "--crush-device-class" { + // assert.Equal(t, "myclass", args[8]) + // deviceClassSet = true + // return "", nil + // } + // return "", errors.Errorf("unknown command %s %s", command, args) + // } + + // context := &clusterd.Context{Executor: executor, ConfigDir: cephConfigDir} + // clusterInfo := &cephclient.ClusterInfo{ + // CephVersion: cephver.CephVersion{Major: 16, Minor: 2, Extra: 1}, // It supports raw mode OSD + // FSID: clusterFSID, + // } + // agent := &OsdAgent{clusterInfo: clusterInfo, nodeName: nodeName, storeConfig: config.StoreConfig{DeviceClass: "myclass"}} + // devices := &DeviceOsdMapping{ + // Entries: map[string]*DeviceOsdIDEntry{ + // "vdb": {Data: -1, Metadata: nil, Config: DesiredDevice{Name: "/dev/vdb"}}, + // }, + // } + // _, err := agent.configureCVDevices(context, devices) + // assert.Nil(t, err) + // assert.True(t, deviceClassSet) + // } } func testBaseArgs(args []string) error { diff --git a/pkg/util/sys/device.go b/pkg/util/sys/device.go index 1cbc8abc6..727bf6187 100644 --- a/pkg/util/sys/device.go +++ b/pkg/util/sys/device.go @@ -472,38 +472,3 @@ func ListDevicesChild(executor exec.Executor, device string) ([]string, error) { return strings.Split(childListRaw, "\n"), nil } - -// PartitionIsAtari returns true if the device partition is an Atari type. -// Ceph bluestore raw disks can sometimes appear as though they have ATARI (AHDI) -// partitions created on them when they don't in reality. -// See: https://github.com/rook/rook/issues/7940 -func PartitionIsAtari(executor exec.Executor, device string) (bool, error) { - // the command error is not processed the same way as most Go errors. parted returns an error - // when the partition table type is unknown, so always process the raw output, even in the case - // of a command error - raw, cmdErr := executor.ExecuteCommandWithOutput("parted", "--machine", "--script", path.Join("/dev", device), "print") - // --machine gives machine-parseable output: https://alioth-lists.debian.net/pipermail/parted-devel/2006-December/000573.html - // HEADER-TYPE; - // DISK-FULL-PATH:...:PART-TABLE-TYPE:PART-TABLE-NAME:FLAGS-SET; - // FS INFO; - // always 6 fields or more and on the second line, but we only care about the 3rd to last field - raw = strings.TrimSpace(raw) // trim begin/end newlines if exist - lines := strings.Split(raw, "\n") - if len(lines) < 2 { - return true, fmt.Errorf("failed to get 'parted' info for device %q. expect at least two lines of output: %q. %v", device, raw, cmdErr) - } - rawInfo := lines[1] - if rawInfo[len(rawInfo)-1] != ';' { - // the last char should be a semicolon - return true, fmt.Errorf("failed to get 'parted' info for device %q. expect a semicolon at end: %q. %v", device, rawInfo, cmdErr) - } - rawInfo = strings.TrimRight(rawInfo, ";") // trim the right semicolon - items := strings.Split(rawInfo, ":") - if len(items) < 6 { - // expect at least 6 fields - return true, fmt.Errorf("failed to get 'parted' info for device %q. expect at least 6 fields: %q. %v", device, rawInfo, cmdErr) - } - pType := items[len(items)-3] // 3rd to last field - - return pType == "atari", nil -} diff --git a/pkg/util/sys/device_test.go b/pkg/util/sys/device_test.go index d81d73043..10342f857 100644 --- a/pkg/util/sys/device_test.go +++ b/pkg/util/sys/device_test.go @@ -196,78 +196,3 @@ func TestListDevicesChildListDevicesChild(t *testing.T) { assert.NoError(t, err) assert.Equal(t, 3, len(child)) } -func TestPartitionIsAtari(t *testing.T) { - // This also implicitly tests PartitionIsAtari - - partedAtariOutput := `BYT; -/dev/sdb1:1509kB:unknown:512:512:atari:Unknown:;` - - partedNonAtariOutput := `BYT; -/dev/sda2:2048MB:unknown:512:512:loop:Unknown:; -1:0.00B:2048MB:2048MB:linux-swap(v1)::;` - - partedUnknownOutput := `BYT; -/dev/sdc1:10.7GB:unknown:512:512:unknown:Unknown:;` - - executor := &exectest.MockExecutor{ - MockExecuteCommandWithOutput: func(command string, arg ...string) (string, error) { - t.Log("command:", command, arg) - switch arg[2] { - case "/dev/vdb1": // is atari - return partedAtariOutput, nil - case "/dev/vdb2": // not atari - return partedNonAtariOutput, nil - case "/dev/vdb3": // unknown type (has error) - // parted returns an error if the partition type is unknown, but it still outputs - // info to stdout that can be parsed successfully - return partedUnknownOutput, fmt.Errorf("fake error about unrecognized disk label") - case "/dev/vdb4": // too few lines (none), and error (likely in real world) - return "", fmt.Errorf("fake error") // too few output lines (none) - case "/dev/vdb5": // too few fields, no error (unlikely in real world) - return `BYT; -/dev/sdc1:Unknown:;`, nil - case "/dev/vdb6": // no semicolon - return `BYT; -there:is:not:a:semicolon:after:this:line:`, nil - default: - panic("unhandled command") - } - }, - } - - t.Run("is atari", func(t *testing.T) { - a, err := PartitionIsAtari(executor, "vdb1") - assert.NoError(t, err) - assert.True(t, a) - }) - - t.Run("not atari", func(t *testing.T) { - a, err := PartitionIsAtari(executor, "vdb2") - assert.NoError(t, err) - assert.False(t, a) - }) - - t.Run("unknown partition type output", func(t *testing.T) { - a, err := PartitionIsAtari(executor, "vdb3") - assert.NoError(t, err) - assert.False(t, a) - }) - - t.Run("too few output lines", func(t *testing.T) { - _, err := PartitionIsAtari(executor, "vdb4") - assert.Error(t, err) - assert.Contains(t, err.Error(), "lines") // minimal check for error message content - }) - - t.Run("too few output fields", func(t *testing.T) { - _, err := PartitionIsAtari(executor, "vdb5") - assert.Error(t, err) - assert.Contains(t, err.Error(), "fields") // minimal check for error message content - }) - - t.Run("no semicolon after fields", func(t *testing.T) { - _, err := PartitionIsAtari(executor, "vdb6") - assert.Error(t, err) - assert.Contains(t, err.Error(), "semicolon") // minimal check for error message content - }) -}