Commit Graph
146 Commits
Author SHA1 Message Date
Artem Torubarov 887d533001 osd: implement osd replacement prepare-job
contains osd prepare job edits to support osd replacement flow:
- removes destroyed OSD IDs from node status CM to stop cluster
  controller from reprovisioning destroyed OSD
- adds provisioning logic for OSD replacement to match available
  blank devices to destroyed OSD slots

Signed-off-by: Artem Torubarov <artem.torubarov@sap.com>
2026-07-27 18:42:07 +02:00
Satoru Takeuchi 4105874d9a Merge pull request #17525 from sp98/lv-osd-with-metadata
osd: allow LVM logical volumes as OSD data devices
2026-05-22 19:44:00 +09:00
Santosh c6d48178dd osd: allow LVM logical volumes as OSD data devices
remov restrictions that prevented LVM logical volume from being
provisioned as OSDs. LV devices with a metadatadevice were rejected at
the filtering stage. LV devices without one were incorrectly being
routed to ceph-volume raw mode which was resuting in error. This PR:

- removes metadatadevice restriction on LV devices.
- Adds LVM time check to isSafeToUseRAWMode to prevent raw mode on lvm

Signed-off-by: Santosh <sapillai@redhat.com>
2026-05-14 11:43:45 +05:30
Artem Torubarov b41d63788d osd: honor per-device deviceClass in raw-mode prepare and reconcile
Fixes #16834 and #17384. Per-OSD CR deviceClass was ignored in
osd prepare job when reporting the osd config back to the operator,
and in the operator's reconcile override.
Fixed by resolving the per-device class from the CR devices list
in GetCephVolumeRawOSDs and in makeDeployment.

Signed-off-by: Artem Torubarov <artem.torubarov@sap.com>
2026-05-11 15:37:16 +02:00
huyejia a6239dedfe osd: replace Split in loops with more efficient SplitSeq
Signed-off-by: huyejia <huyejia@outlook.jp>
2025-11-12 16:52:12 +08:00
Santosh Pillai 2c1bb7443a Merge pull request #15796 from sp98/clean-osd-disks-from-old-clusters
osd: clean osd disks before reinstalling cluster
2025-06-09 14:03:49 +05:30
Santosh Pillai 2bcf196b15 osd: clean osd disks before reinstalling cluster
Allow force creation of OSD on a device previously configured in a different ceph cluster.
The OSD prepare pod job will zap the OSD disk using ceph-volume lvm zap
if that disk has metadata from a different ceph cluster.

Signed-off-by: Santosh Pillai <sapillai@redhat.com>
2025-06-06 16:32:18 +05:30
Carlos Barria 030ada45b0 core: fix golangci-lint check QF1003 QF1002
Refactored multiple conditional if-else blocks into switch statements to improve readability and maintainability. Also removed outdated staticcheck rule comments (QF1002, QF1003) from .golangci.yaml.

Signed-off-by: Carlos Barria <cbarria@yahoo.com>
2025-05-27 13:49:49 -04:00
Joshua Hoblitt 3cb343f62a core: run gofumpt on all files
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
2025-03-26 10:41:48 -07:00
Travis Nielsen b665d7a7b7 core: remove support for ceph quincy
Given that Ceph Quincy (v17) is past end of life,
remove Quincy from the supported Ceph versions,
examples, and documentation.

Supported versions now include only Reef and Squid.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2024-10-03 11:12:55 -06:00
sp98 94953f3f7b osd: create config before migration osd
create the ceph config and keyring file in the osd prepare pod
before starting the OSD migration. These files are needed to
run any ceph command.

Signed-off-by: sp98 <sapillai@redhat.com>
2024-01-08 15:35:45 +05:30
Satoru Takeuchi 8e80b21035 Revert "osd: support encrypted OSD on partiton"
This reverts commit 093125904b.

Once we introduced encrypted OSD on partition in #12924.
However, even though `encryptedDevice` field is `true`,
an non-encrypted OSD is created. To prevent the creation
of misconfigured OSD, it's better to revert #12924 for now.

Initially, I intended to correct this problem. However, I found
it will take long time because we need to change the KMS-related
code, init containers of both OSD pod and OSD prepare pod.

Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
2023-11-07 12:37:31 +00:00
Travis Nielsen cb9ffacce5 Merge pull request #12909 from testwill/pkg-import
core: import packages only once
2023-09-21 15:30:04 -06:00
Satoru Takeuchi 093125904b osd: support encrypted OSD on partiton
Let's use raw mode for encrypted OSD. This mode supports encrypted
OSD on partition.

Closes: https://github.com/rook/rook/issues/10984

Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
2023-09-19 13:10:18 +00:00
guoguangwu 235ac293ff core: import packages only once
Signed-off-by: guoguangwu <guoguangwu@magic-shield.com>
2023-09-16 13:40:17 +08:00
sp98 11b8d10a5a osd: replace existing OSDs to use new store
This follow up the #12507
- fixes replacing of encrypted OSDs.
- Updates OSD status at the end of reconcile

Signed-off-by: sp98 <sapillai@redhat.com>
2023-09-01 21:07:38 +05:30
sp98 886bb357ef osd: replace osd to use new backend store
If osd store is updated in the ceph cluster, then
delete OSDs one by one, cleanup disks and provision a new OSD on
the same disk

Signed-off-by: sp98 <sapillai@redhat.com>
2023-08-17 21:40:55 +05:30
Javier 884d5e9855 osd: allow to use filter device
this feature is to allow the use of filters using the deviceFilter flag
by skipping devices that do not match the filter

Closes: #10340
Signed-off-by: Javier <sjavierlopez@gmail.com>
2023-04-21 13:13:59 -06:00
Satoru Takeuchi 3e34ebeff8 osd: handle global or node-local device class configuration correctly
Rook uses global or node-local device class configuration if device-level
configurations doesn't exist. However, after introducing device-class
level resource configuration, rook set the default value of device class.
So global or node-level device class configuration has never been used
after that.

Closes: https://github.com/rook/rook/issues/11871
Closes: https://github.com/rook/rook/issues/11826

Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
2023-03-27 04:28:45 +00:00
Travis Nielsen 8a94bb20ae osd: allow mpath_member filesystem for mpath disks
Multipath devices may have an FSTYPE=mpath_member even though
a valid device for creating an OSD. The device will be allowed
in this case when creating the OSD on a PV since it is a device
specifically provisioned for the OSD.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2022-12-09 09:42:27 -07:00
Satoru Takeuchi 36f8074938 osd: improve log on specifying lv and loop device as osd
We can use LVM logical volumes and loop devices as OSD iff we specify
them in the `devices` field. If we use filters like `deviceFilter`,
these devices are not picked with a log message which let users
to specify the exact device name. It's better to mention which field
users should use in this case.

Closes: https://github.com/rook/rook/issues/11353

Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
2022-11-28 23:18:54 +00:00
Shinya Hayashi 05875a3f4f osd: support loop devices for test clusters
A new variable is added to rook-ceph-operator-config
ConfigMap to allow using loop devices for osd.

This feature is intended to be used for testing purposes only.

Signed-off-by: Shinya Hayashi <shinya-hayashi@cybozu.co.jp>
2022-11-09 06:45:41 +00:00
Satoru Takeuchi b6a6f66ff0 osd: clarify lv is not picked by filters
LVM logical volumes are not picked by `deviceFilter`, `devicePathFilter`,
and `useAllDevices`. It's to prevent old version's users from creating
lv-backed osd during upgrading rook by accident.

It's better to clarify lvs are not picked by filters" in docs and
suggest users to use exact device name instead.

Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
2022-09-28 00:58:04 +00:00
Satoru Takeuchi 9ad82eebbb Merge pull request #10968 from cybozu-go/osd-disallow-to-create-encrypted-osd-on-part
osd: disallow to create encrypted osd on partition
2022-09-14 04:26:15 +09:00
Satoru Takeuchi 39cc6059f1 ci: disallow to create osd on lv with metadata by accident
OSD on LV with metadata device is not supported. So we once we made
a fix to disallow this configuration. However, it's insuffucient
because we forgot to reset `matched` flag.

Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
2022-09-13 13:21:31 +00:00
Satoru Takeuchi 7969450589 ci: disallow to create encrypted osd on lv
Rook doesn't support encrypted OSD on LV. Without disallowing
this configuration, ceph-volume results in failure.

Closes: https://github.com/rook/rook/issues/10782

Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
2022-09-13 12:44:53 +00:00
Satoru Takeuchi 591fc5f68b osd: disallow to create encrypted osd on partition
Rook doesn't support encrypted osd on partition. So we should
skip partitions as candidates of the new encrypted osds.

Closes: https://github.com/rook/rook/issues/10911

Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
2022-09-13 12:04:14 +00:00
Satoru Takeuchi c7ab01b946 Merge pull request #10858 from cybozu-go/osd-refactoring
osd: refactor code
2022-09-03 06:25:17 +09:00
Satoru Takeuchi 00b3d0f323 osd: refactor code
Simplify osd management code.

Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
2022-09-02 11:17:05 +00:00
Satoru Takeuchi 8b513a4a74 osd: disallow to create osd on lv with metadata device
OSD on logical volume is raw mode. This mode can'set metadata device.
We should not mark a logical volume as available in this case.

Closes: https://github.com/rook/rook/issues/10747

Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
2022-09-01 05:12:43 +00:00
taki-d ca73efa1e5 osd: fix osd detection logic for old lsblk
Old lsblk can't detect bluestore signature.
So read the head of device and verify signature
as a workaround.

closes: https://github.com/rook/rook/issues/10665

Signed-off-by: taki-d <mail@orangepla.net>
2022-08-31 11:02:25 +09:00
Travis Nielsen fc541764f1 Merge pull request #10484 from jsoref/spelling
core: fix spelling
2022-07-15 08:31:41 -06:00
Liang Zheng 7609dad72f osd: metadata device supports udev path
This commit adds udev path for metadata device

Closes: https://github.com/rook/rook/issues/10486

Signed-off-by: Liang Zheng <zhengliang0901@gmail.com>
2022-07-14 10:26:33 +08:00
Satoru Takeuchi 7e571f6114 osd: support OSD on logical volume in host-based cluster
Rook supports raw mode OSD in host-based cluster. So we can also
support OSD on logical volume in this kind of cluster.

Logical volumes aren't picked by filters (i.e. `useAllDevices: true`
and `device{Path,}Filter` to avoid unwanted LV consumption on upgrade.

Closes: https://github.com/rook/rook/issues/2047

Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
2022-07-11 21:08:50 +00:00
Josh Soref 6e7b8767f3 core: fix spelling
* another
* are
* availability
* available
* bootstrap
* boundaries
* ceph
* certificate
* class
* codifies
* consuming
* corrupted
* createor
* csi
* deployments
* exceeded
* execute
* filesystem
* healthiness
* heuristics
* immediately
* insecure
* installed
* isolated
* maintained
* maximum
* minute
* monitor
* new
* nginx
* nonexistent
* not
* occurs
* omitempty
* operator
* orchestration
* persistentvolumes
* placement
* preexisting
* prometheus
* protecting
* provisioner
* purposes
* reconcile
* regex
* related
* requests
* returns
* rubbish
* running
* schedulable
* schedule
* serviceaccount
* simulating
* snapshots
* statement
* static
* tenants
* the
* unavailable
* volumeattachment
* waiting
* with
* wrapper
* zonegroup

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
2022-07-07 18:10:47 -04:00
Sébastien Han bfe4f7c111 osd: do not return if deactivate fail
Let it fail further once we run c-v activate. The error reported by LVM
could potentially ignored. If there is a failure, the c-v command will
fail anyways.

Signed-off-by: Sébastien Han <seb@redhat.com>
2022-06-15 15:42:48 +02:00
Sébastien Han 8328ec6d31 core: add mountpoint detection to the prepare pod
Amount various filters let's add a mountpoint property and skip the
device if its filesystem is mounted.

Signed-off-by: Sébastien Han <seb@redhat.com>
2022-05-09 15:26:50 +02:00
Sébastien Han 6bc7827150 osd: only set kek to env var on encryption scenario
There is a corner case where an env variable populated with the prepare
job could influence and trigger some encryption code. We have an env
variable to discover whether the prepare job will encrypt a drive or not
so let's use that instead of running code as a noop and sometimes
trigger a corner case. In this scenario, the prepare job had an "IBM_"
env variable present in the OS image (IBM OS) so the setKEKinEnv()
  function would get called for no reason.

Signed-off-by: Sébastien Han <seb@redhat.com>
2022-04-11 16:54:18 +02:00
Sébastien Han 05775b068d osd: handle removal of encrypted osd deployment
This is handling a tricky scenario where the OSD deployment is manually
removed and the OSD never reconvers. This is unlikely to happen, but
still OSD should be able to run after that action. Essentially after a
manual deletion, we need to run the prepare job again to re-hydrate the
OSD information so that the OSD deployment can be deployed.
On encryption, it is a little bit tricky since ceph-volume list again
the main block won't return anything, so we need to target the encrypted
block to list.
There is another case this PR does not handle, which is the removal of
the OSD deployment and then the node is restarted. This means that the
encrypted container is not opened anymore. However, opening it requires
more work like writing the key on the filesystem (if not coming from the
Kubernete secret, eg,. KMS vault) and then run luksOpen. This is an
extreme corner case probably not worth worrying about for now.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-12-21 10:51:09 +01:00
Yuichiro Ueno 0d8c581aa9 core: add context parameter to k8sutil kvstore
This commit adds context parameter to k8sutil kvstore functions. By
this, we can handle cancellation during API call of kvstore resource.

Signed-off-by: Yuichiro Ueno <y1r.ueno@gmail.com>
2021-12-11 11:17:12 +09:00
subhamkrai 0150966024 ceph: remove ceph nautilus, ceph octopus to default
since rook 1.8, ceph nautilus no longer supported,
ceph octopus will be the minimum ceph version.

Closes: https://github.com/rook/rook/issues/7908
Signed-off-by: subhamkrai <srai@redhat.com>
2021-10-20 14:45:12 +05:30
Sébastien Han b89730d895 ceph: refactor operator initialization sequence
This commit is a large refactor on how the operator starts, stops and
how it starts various sub-components such as the ceph-csi driver. It
also refines the way we cancel orchestrations. We don't use breakpoints
anymore but send our self a SIGUP to reload our controller runtime
manager.
The reload will happen under different circonstances like:

* a new adminission controller secret is created/deleted/changed
* a CephCluster CR is edited

As mentioned earlier, the csi driver now has its own controller, just
like flex. It reacts to change in the operator config map for particular
ROOK_CSI_ fields.

A second new controller for the operator's general config has been
created, it manages:

* the logging level
* the ceph CLI command timeout
* the discovery daemon

The operator reacts much more rapidly to cancellation events by stopping
the manager's context and reloading it.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-09-17 16:57:12 +02:00
Hiroya Onoe 2ff5413b75 ceph: fix error message in UpdateNodeStatus
The error message in UpdateNodeStatus regards the second argument
as node name. However, it is a PVC name in OSD on PVC.

Signed-off-by: Hiroya Onoe <onoehiroya@gmail.com>
2021-09-02 02:43:16 +00:00
Blaine Gardner c8a2db368a ceph: disable raw mode for disks
Even if we can use raw mode, do NOT use raw mode on disks. Ceph bluestore disks can
sometimes appear as though they have "phantom" Atari (AHDI) partitions created on them
when they don't in reality. This is due to a series of bugs in the Linux kernel when it
is built with Atari support enabled. This behavior does not appear for raw mode OSDs on
partitions, and we need the raw mode to create partition-based OSDs. We cannot merely
skip creating OSDs on "phantom" partitions due to a bug in `ceph-volume raw inventory`
which reports only the phantom partitions (and malformed OSD info) when they exist and
ignores the original (correct) OSDs created on the raw disk.

Resolves https://github.com/rook/rook/issues/7940

Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
2021-07-22 09:59:56 -06:00
Blaine Gardner 7f516b9e3d ceph: ignore atari partitions when scanning disks
Ceph bluestore raw disks can sometimes appear as though they have Atari
(AHDI) partitions on them. If a disk has Atari partitions, we just
ignore them as though they don't exist. This should be a safe assumption
since the hardware was last manufactured in 1992 and likely can't run
Kubernetes.

If we don't ignore the Atari partitions, Rook can create a new OSD on a
disk that is already running an OSD, corrupting the first and possibly
also the latest OSD. This can happen an arbitrary number of times per
disk.

More info: https://github.com/rook/rook/issues/7940

Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
2021-06-30 09:42:33 -06:00
Travis Nielsen 06fbf29b3c Merge pull request #8028 from degorenko/device-classes-resource-limits
ceph: add ability to set resource limit for OSDs based on device classes
2021-06-08 09:01:14 -06:00
Denis Egorenko 8a556fd847 ceph: add ability to set resource limit for OSDs based on device classes
Currently it is not possible to set resource limits based on device classes
for different OSDs. Now adding an ability to use predefined keys for main
cluster spec Resource section to reflect resource limits for different OSDs
with different device classes.

Closes: https://github.com/rook/rook/issues/8007
Signed-off-by: Denis Egorenko <degorenko@mirantis.com>
2021-06-08 17:46:41 +04:00
Sébastien Han e38601ac2d ceph: always skip disks with filesystem
Unless they are encrypted disk and have markers for a ceph cluster.

Closes: https://github.com/rook/rook/issues/8046
Signed-off-by: Sébastien Han <seb@redhat.com>
2021-06-03 14:42:29 +02:00
Sébastien Han 0004869f66 ceph: add ceph cluster fsid to LUKS header
When configuring encrypted on-pvc clusters we now set the cluster fsid
in the LUKS header. If needed we can then determine if the encrypted
block is part of our cluster or not. We also attach the pvc_name in case
it might become useful.

Closes: https://github.com/rook/rook/issues/7991
Signed-off-by: Sébastien Han <seb@redhat.com>
2021-05-31 15:01:52 +02:00
Satoru Takeuchi a4fb080590 ceph: continue to get available devices if failed to get a device info
getAvailableDevices() should continue if something wrong happens in a device than returning
immediately with error.

Closes: https://github.com/rook/rook/issues/7543

Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
2021-04-14 03:13:06 +00:00