OSDs are created with `ceph-volume lvm batch` if `metadataDevice`
field is specified in CephCluster CR. Since this command does not
support kernel names (e.g. `/dev/dm-X`) for multipath devices,
we should use real names (e.g. `/dev/mapper/mpathX`) instead.
NOTE:
I intentionally keep raw mode OSD creation code as is
because `ceph-volume raw prepare` supports kernel names
for any devices.
Closes: https://github.com/rook/rook/issues/17057
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
Fixes quick disk shredding by using dd to shred data at additional offsets where ceph metadata is duplicated. For full shred, the shred utility remains in use.
Signed-off-by: Vilius Puškunalis <47086537+puskunalis@users.noreply.github.com>
During the OSD preparation step, if the encrypted OSD disk belongs to a
different cluster, then clean the disk if `wipeDeviceFromOtherClusters`
is set to true.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
Allow force creation of OSD on a device previously configured in a different ceph cluster.
The OSD prepare pod job will zap the OSD disk using ceph-volume lvm zap
if that disk has metadata from a different ceph cluster.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
currently device class and device type
labels were clubbed with eachother
create a seperate label, as these two labels
can be seperate
Signed-off-by: parth-gr <partharora1010@gmail.com>
we were adding the device class as the
disk rotational type, but device class
can be user defined, so if user has
added the device class, use that value
Signed-off-by: parth-gr <partharora1010@gmail.com>
When doing the clean-up of raw device OSDs, it was not taking metadata
and wal devices into account. This commit discover them via the output
of `ceph-volume raw list`, which is returning `device_db` and/or
`device_wal` if they are present.
After discovering the metadata and wal device it also clean them and
close the encrypted device if any.
Signed-off-by: Mathias Chapelain <mathias.chapelain@proton.ch>
Currently, when rook provisions OSDs(in the OSD prepare job), rook effectively run a
c-v command such as the following.
```console
ceph-volume lvm batch --prepare <deviceA> <deviceB> <deviceC> --db-devices <metadataDevice>
```
but c-v lvm batch only supports disk and lvm, instead of disk partitions.
We can resort to `ceph-volume lvm prepare` to implement it.
Signed-off-by: Liang Zheng <zhengliang0901@gmail.com>
The validation logic of checking the number of devices is
wrong when `metadataDevice` is set and `osdsPerDevice` > 1.
`len(cvReports)` is the expected number of OSDs and is
the number of specified data devices multiplied
by `osdsPerDevice`.
Closes: https://github.com/rook/rook/issues/13637
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
This reverts commit 093125904b.
Once we introduced encrypted OSD on partition in #12924.
However, even though `encryptedDevice` field is `true`,
an non-encrypted OSD is created. To prevent the creation
of misconfigured OSD, it's better to revert #12924 for now.
Initially, I intended to correct this problem. However, I found
it will take long time because we need to change the KMS-related
code, init containers of both OSD pod and OSD prepare pod.
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
If osd store is updated in the ceph cluster, then
delete OSDs one by one, cleanup disks and provision a new OSD on
the same disk
Signed-off-by: sp98 <sapillai@redhat.com>
few functions got change as they were deprecated
for ex: ioutil.Readfile change to os.Readfile
ioutil.TempFile change to os.CreateTemp
And fixed golang-ci-lint-issues
Signed-off-by: parth-gr <paarora@redhat.com>
It removes the condition that select the devices name prefixed with /dev/mapper as it is causing the ceph-volume command to fail.
Closes: #11301
Signed-off-by: ausias-armesto <ausiasarmesto@gmail.com>
During osd-prepare-job,
The encrypted device is found to be closed in some cases when
the OSD deployment has been removed manually accompanied
by any of following cases:
- node reboot
- csi managed PVC being unmounted etc
while re-opening the block to re-hydrate the OSDInfo,
the dm device `<pvc-name>-block-dmcrypt` clashes with
the one used by OSD pod which is stale by now.
This commit adds cmd to remove this stale dm device.
Error with output "No such device" is ignored.
Signed-off-by: Rakshith R <rar@redhat.com>
This commit implements this corner case during osd-prepare job.
```
The encrypted block is not opened, this is an extreme corner case
The OSD deployment has been removed manually AND the node rebooted
So we need to re-open the block to re-hydrate the OSDInfo.
Handling this case would mean, writing the encryption key on a
temporary file, then call luksOpen to open the encrypted block and
then call ceph-volume to list against the opened encrypted block.
We don't implement this, yet and return an error.
```
When underlying PVC for osd are CSI provisioned, the encrypted device
is closed when PVC is unmounted due to osd pod being deleted.
Therefore, this may occur more frequently and needs to be handled.
This commit implements the fix for the same.
Signed-off-by: Rakshith R <rar@redhat.com>
With octopus coming to end of life, we remove support from
Rook for deploying Ceph Octopus and assume a min version of
Pacific v16. Any checks for octopus or earlier are removed
from the reconciles since they are obsolete.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Once we are done cleaning up the content of the encrypted osd, let's
close the main LUKS device.
This avoids having dm devices on the system after the cleanup.
Closes: https://github.com/rook/rook/issues/10181
Signed-off-by: Sébastien Han <seb@redhat.com>
If a specific disk is configured to have more than one OSD per device,
or if it has a metadata device specified, exclude it from raw mode on an
individual basis.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
The OSD prepare job was succeeding even if the OSDs failed to be listed
by ceph-volume. These errors should be returned and fail the job, resulting
in a retry to retrieve the OSDs. Otherwise, the failure is masked and
the admin needs to dig through logs to find why the OSDs were not configured
on that node.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
We can eliminate the OSD creation code for Ceph < v15.2.0 because Nautilus(v14.2.z)
is no longer supported. In other words, we can now expect that all newly-created
OSDs on PVC are raw mode.
Please note that the existing lvm mode OSDs on PVCs still be supported after this change.
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
In volume.go, remove if condition in `callCephVolume`
method where we always pass false.
In exec.go, removing `ExecuteCommandWithFile*` methods
which was not used anywhere.
Signed-off-by: subhamkrai <srai@redhat.com>
This is handling a tricky scenario where the OSD deployment is manually
removed and the OSD never reconvers. This is unlikely to happen, but
still OSD should be able to run after that action. Essentially after a
manual deletion, we need to run the prepare job again to re-hydrate the
OSD information so that the OSD deployment can be deployed.
On encryption, it is a little bit tricky since ceph-volume list again
the main block won't return anything, so we need to target the encrypted
block to list.
There is another case this PR does not handle, which is the removal of
the OSD deployment and then the node is restarted. This means that the
encrypted container is not opened anymore. However, opening it requires
more work like writing the key on the filesystem (if not coming from the
Kubernete secret, eg,. KMS vault) and then run luksOpen. This is an
extreme corner case probably not worth worrying about for now.
Signed-off-by: Sébastien Han <seb@redhat.com>
Even if we can use raw mode, do NOT use raw mode on disks. Ceph bluestore disks can
sometimes appear as though they have "phantom" Atari (AHDI) partitions created on them
when they don't in reality. This is due to a series of bugs in the Linux kernel when it
is built with Atari support enabled. This behavior does not appear for raw mode OSDs on
partitions, and we need the raw mode to create partition-based OSDs. We cannot merely
skip creating OSDs on "phantom" partitions due to a bug in `ceph-volume raw inventory`
which reports only the phantom partitions (and malformed OSD info) when they exist and
ignores the original (correct) OSDs created on the raw disk.
Resolves https://github.com/rook/rook/issues/7940
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
Currently it is not possible to set resource limits based on device classes
for different OSDs. Now adding an ability to use predefined keys for main
cluster spec Resource section to reflect resource limits for different OSDs
with different device classes.
Closes: https://github.com/rook/rook/issues/8007
Signed-off-by: Denis Egorenko <degorenko@mirantis.com>
When configuring encrypted on-pvc clusters we now set the cluster fsid
in the LUKS header. If needed we can then determine if the encrypted
block is part of our cluster or not. We also attach the pvc_name in case
it might become useful.
Closes: https://github.com/rook/rook/issues/7991
Signed-off-by: Sébastien Han <seb@redhat.com>
The deviceClass was only being set for LVM mode OSDs and was
missed for the raw mode for non-PVC. Now the deviceClass will be set
for all expected scenarios.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
This fixes cases where raw provisioning mode still cannot be used for
Rook v1.6 and Ceph Pacific clusters where json.Unmarshal fails to read
the ceph-volume output after OSD provisioning.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
This mops up the last of the multiple-imports within ceph-related
packages, in pkg/daemon/ceph/agent/flexvolume/manager/ceph and
pkg/daemon/ceph/osd.
Signed-off-by: Lars Lehtonen <lars.lehtonen@gmail.com>
Update OSDs in parallel per the design in
design/ceph/update-osds-in-parallel.md
The max number of OSDs updated in parallel is currently fixed at 20.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
Satoru Takeuchi
huyejia
Santosh Pillai
Vilius Puškunalis
Joshua Hoblitt
parth-gr
Mathias Chapelain
Liang Zheng
parth-gr
ausias-armesto
Rakshith R
Travis Nielsen
Josh Soref
Sébastien Han
subhamkrai
Blaine Gardner
Jonas Zeiger
Denis Egorenko
Lars Lehtonen