use ClusterID of the k8s cluster while doing nslookup of the
exported mon or osd service IP
Signed-off-by: sp98 <sapillai@redhat.com>
(cherry picked from commit 9e8e85d68b)
The defaults in CRDs cause an update and trigger a new
reconcile if the value is not set. For the key rotation
setting, the default was only being updated when the kms
feature was enabled. This update was causing the reconcile
to fail in the kms tests. With the default now in code,
the default can remain empty and no cr update will trigger
a new reconcile unexpectedly.
Signed-off-by: travisn <tnielsen@redhat.com>
(cherry picked from commit cd94ed7ddc)
This commit adds util function to create key rotation cron job
such as getKeyRotationContainer, getKeyRotationCronPodTemplateSpec
and makeKeyRotationCronJob.
Signed-off-by: Rakshith R <rar@redhat.com>
(cherry picked from commit ef38aea2a4)
This commit adds functionality to be able to rotate
key encryption key of encrypted PVC backed OSDs.
Necessary changes such as adding update functionality
to kms and rbac changes are made as well.
Signed-off-by: Rakshith R <rar@redhat.com>
(cherry picked from commit 71e011f731)
Add container ports for OSD container only when multi cluster service is
enabled in the cluster spec.
Signed-off-by: sp98 <sapillai@redhat.com>
(cherry picked from commit 12bbc9b1a8)
If a kernel device name change happens and a block device file
in the OSD directory becomes dangling link, this OSD fails
to start continuously. This problem can be resolved by confirming
the validity of the device file and recreating it if necessary.
The original problem happened in host-based clusters. A similar
problem exists in PVC-based clusters. In this case, if a PV,
corresponds to an existing OSD, and points to a missing block
device file, the OSD pod fails to consume this PV. Although this
behavior is undesirable, I don't think this problem should be
handled in Rook. If doing so, we must re-create the existing PV.
Rook shouldn't do such work.
Closes: https://github.com/rook/rook/issues/10860
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
(cherry picked from commit 3a44a394b5)
The osd needs to bind to an ip that will allow both internal
and external access for rbd mirroring scenarios.
THe operator will create a service for each osd and
instruction the osd to bind to the pod ip while
publishing the clusterIP of the service as the
public IP.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
(cherry picked from commit 14e7225e0a18aff85ede1d476fcaddbbfbcbb732)
(cherry picked from commit d655c86b2f)
few functions got change as they were deprecated
for ex: ioutil.Readfile change to os.Readfile
ioutil.TempFile change to os.CreateTemp
And fixed golang-ci-lint-issues
Signed-off-by: parth-gr <paarora@redhat.com>
(cherry picked from commit a84daf9bf0)
This commit adds code to supply variables required
to enable read affinity for rbd volumes.
It leverages `read_from_replica=localize,crush_locations=key:value`
krbd map option from ceph to serve reads from the most local OSD.
refer:
https://docs.ceph.com/en/latest/man/8/rbd/#kernel-rbd-krbd-options
Signed-off-by: Rakshith R <rar@redhat.com>
(cherry picked from commit 8b9ded1f77)
The rook-version tag must not be set on the pod spec labels
since it will result in the ceph daemons being restarted every
time there is a rook version update, even if the ceph version
or pod spec was not otherwise updated.
The rook-version tag was being added to the OSD pod spec
due to a shared pointer to the labels. The code intended
to only add the version label to the deployment labels,
but since the pod labels shared the map variable, the
pod unintentionally also had the version label added.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Environment variables are not recommended for secrets in pods since
they can be easily leaked if the environemnt variables are logged.
By mounting the mon secret as a file, the mgr and osd prepare pods
can read the mon secret from a file for better security.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The volume names of a pod must follow the standard naming convention
which does not allow periods in the name. Since the device set name
is used as the pvc name and then the volume name to attach to the osd
prepare job, we must sanitize the name for the invalid character
and replace it.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
This commit implements this corner case during osd-prepare job.
```
The encrypted block is not opened, this is an extreme corner case
The OSD deployment has been removed manually AND the node rebooted
So we need to re-open the block to re-hydrate the OSDInfo.
Handling this case would mean, writing the encryption key on a
temporary file, then call luksOpen to open the encrypted block and
then call ceph-volume to list against the opened encrypted block.
We don't implement this, yet and return an error.
```
When underlying PVC for osd are CSI provisioned, the encrypted device
is closed when PVC is unmounted due to osd pod being deleted.
Therefore, this may occur more frequently and needs to be handled.
This commit implements the fix for the same.
Signed-off-by: Rakshith R <rar@redhat.com>
The fsid, username, and user secrets were from legacy and no
longer needed on the osd daemon or mon pods. This removes the
unnecessary env vars from the pod specs.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
A new variable is added to rook-ceph-operator-config
ConfigMap to allow using loop devices for osd.
This feature is intended to be used for testing purposes only.
Signed-off-by: Shinya Hayashi <shinya-hayashi@cybozu.co.jp>
The osd arguments had a few duplicate lines that could be
factored out and thus simplified for code maintainability.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Introduce a new env variable ROOK_CSI_IMAGE_PULL_POLICY in rook operator configmap which should be used to
customize the imagePullPolicy for the csi driver and imagePullPolicy property in cephVersionSpec for ceph pods.
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
The cluster-wide encryption feature now
supports KMIP (Key Management Interoperability Protocol)
kms.
KMIP is an extensible communication protocol that defines
message formats for the manipulation of cryptographic keys
on a key management server.
For more information, refer:
https://en.wikipedia.org/wiki/Key_Management_Interoperability_Protocol
Signed-off-by: Rakshith R <rar@redhat.com>
The tool gofmt in go 1.19 requires certain formatting
in the comments section for better rendering.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
During certain maintenance tasks the admin will own running
operations on the ceph mons and osds, and the operator should
not interfere with those operations. If the operator sees
any mon in debug mode, every reconcile and mon health check
will be skipped. Thus, mons will not be updated while any
one of them is in maintenance. During OSD reconcile, individual
OSD deployment updates will only be skipped for OSDs that are
actively being debugged.
The debug mode for osd and mon deployments is signaled by
creating the ceph.rook.io/do-not-reconcile label.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
This commit replaces `os.Setenv` with `t.Setenv` in tests. The
environment variable is automatically restored to its original value
when the test and all its subtests complete.
Reference: https://pkg.go.dev/testing#T.Setenv
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
Some context canceled were missed and not reported correctly. We had
multiple reconcilers running in //. Since they all run in goroutine then
will continue to run until they timeout (if timeout is available,
otherwise would run forever until the op is restarted).
Signed-off-by: Sébastien Han <seb@redhat.com>
With octopus coming to end of life, we remove support from
Rook for deploying Ceph Octopus and assume a min version of
Pacific v16. Any checks for octopus or earlier are removed
from the reconciles since they are obsolete.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Ancient OSDs based on LVM and activated by the Rook binary are still in
production. They fail on the deactivation of the VolumeGroup due to
incorrect filter. Setting the OSD as LVM adjusts the filter and make the
deactivate call working.
Repro without and with the correct filter:
```
[root@ca862389706c /]# vgchange -an -vv ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d
global/use_lvmpolld not found in config: defaulting to 1
devices/sysfs_scan not found in config: defaulting to 1
devices/scan_lvs not found in config: defaulting to 0
devices/multipath_component_detection not found in config: defaulting to 1
devices/md_component_detection not found in config: defaulting to 1
devices/fw_raid_component_detection not found in config: defaulting to 0
devices/ignore_suspended_devices not found in config: defaulting to 0
devices/ignore_lvm_mirrors not found in config: defaulting to 1
devices/scan_lvs not found in config: defaulting to 0
devices/allow_mixed_block_sizes not found in config: defaulting to 0
devices/hints not found in config: defaulting to "all"
activation/activation_mode not found in config: defaulting to "degraded"
metadata/record_lvs_history not found in config: defaulting to 0
devices/search_for_devnames not found in config: defaulting to "auto"
activation/monitoring not found in config: defaulting to 1
global/locking_type not found in config: defaulting to 1
global/wait_for_locks not found in config: defaulting to 1
global/prioritise_write_locks not found in config: defaulting to 1
global/locking_dir not found in config: defaulting to "/run/lock/lvm"
devices/md_component_detection not found in config: defaulting to 1
devices/md_component_checks not found in config: defaulting to "auto"
devices/multipath_wwids_file not found in config: defaulting to "/etc/multipath/wwids"
global/use_lvmlockd not found in config: defaulting to 0
report/output_format not found in config: defaulting to "basic"
log/report_command_log not found in config: defaulting to 0
devices/use_devicesfile not found in config: defaulting to 0
/dev/log: stat failed: No such file or directory
devices/global_filter not found in config: defaulting to global_filter = [ "a|.*|" ]
Setting devices/filter to filter = [ "a|^/mnt/.*|", "r|.*|" ]
devices/global_filter not found in config: defaulting to global_filter = [ "a|.*|" ]
Setting devices/filter to filter = [ "a|^/mnt/.*|", "r|.*|" ]
devices/devicesfile not found in config: defaulting to "system.devices"
Obtaining the complete list of VGs to process
VG name on command line not found in list of VGs: ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d
Processing VG ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d
Locking /run/lock/lvm/V_ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d WB
Unlocking /run/lock/lvm/V_ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d
Volume group "ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d" not found
Cannot process volume group ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d
global/notify_dbus not found in config: defaulting to 1
[root@ca862389706c /]# echo $?
5
[root@ca862389706c /]# vi /etc/lvm/lvm.conf
[root@ca862389706c /]# vgchange -an -vv ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d
global/use_lvmpolld not found in config: defaulting to 1
devices/sysfs_scan not found in config: defaulting to 1
devices/scan_lvs not found in config: defaulting to 0
devices/multipath_component_detection not found in config: defaulting to 1
devices/md_component_detection not found in config: defaulting to 1
devices/fw_raid_component_detection not found in config: defaulting to 0
devices/ignore_suspended_devices not found in config: defaulting to 0
devices/ignore_lvm_mirrors not found in config: defaulting to 1
devices/scan_lvs not found in config: defaulting to 0
devices/allow_mixed_block_sizes not found in config: defaulting to 0
devices/hints not found in config: defaulting to "all"
activation/activation_mode not found in config: defaulting to "degraded"
metadata/record_lvs_history not found in config: defaulting to 0
devices/search_for_devnames not found in config: defaulting to "auto"
activation/monitoring not found in config: defaulting to 1
global/locking_type not found in config: defaulting to 1
global/wait_for_locks not found in config: defaulting to 1
global/prioritise_write_locks not found in config: defaulting to 1
global/locking_dir not found in config: defaulting to "/run/lock/lvm"
devices/md_component_detection not found in config: defaulting to 1
devices/md_component_checks not found in config: defaulting to "auto"
devices/multipath_wwids_file not found in config: defaulting to "/etc/multipath/wwids"
global/use_lvmlockd not found in config: defaulting to 0
report/output_format not found in config: defaulting to "basic"
log/report_command_log not found in config: defaulting to 0
devices/use_devicesfile not found in config: defaulting to 0
/dev/log: stat failed: No such file or directory
/dev/block/7:0: size is 2097152 sectors
/dev/nbd0: size is 0 sectors
/dev/nbd8: size is 0 sectors
/dev/vda: size is 83886080 sectors
/dev/block/7:1: size is 0 sectors
/dev/vda1: size is 83883999 sectors
/dev/block/7:2: size is 0 sectors
/dev/block/7:3: size is 0 sectors
/dev/block/7:4: size is 0 sectors
/dev/block/7:5: size is 0 sectors
/dev/block/7:6: size is 0 sectors
/dev/block/7:7: size is 0 sectors
/dev/vdb: size is 83886080 sectors
/dev/nbd1: size is 0 sectors
/dev/nbd9: size is 0 sectors
/dev/vdc: size is 83886080 sectors
/dev/vdd: size is 83886080 sectors
/dev/nbd2: size is 0 sectors
/dev/nbd10: size is 0 sectors
/dev/nbd3: size is 0 sectors
/dev/nbd11: size is 0 sectors
/dev/nbd4: size is 0 sectors
/dev/nbd12: size is 0 sectors
/dev/nbd5: size is 0 sectors
/dev/nbd13: size is 0 sectors
/dev/nbd6: size is 0 sectors
/dev/nbd14: size is 0 sectors
/dev/nbd7: size is 0 sectors
/dev/nbd15: size is 0 sectors
devices/global_filter not found in config: defaulting to global_filter = [ "a|.*|" ]
Setting devices/filter to filter = [ "a|^/mnt/.*|", "r|^/dev/loop.*|", "a|^/dev/.*|", "r|.*|" ]
/dev/block/7:0: using cached size 2097152 sectors
/dev/block/7:0: using cached size 2097152 sectors
/dev/block/7:0: No lvm label detected
/dev/vda: using cached size 83886080 sectors
/dev/vda1: using cached size 83883999 sectors
/dev/vda1: using cached size 83883999 sectors
/dev/vda1: No lvm label detected
/dev/vdb: using cached size 83886080 sectors
/dev/vdb: using cached size 83886080 sectors
/dev/vdc: using cached size 83886080 sectors
/dev/vdc: using cached size 83886080 sectors
/dev/vdd: using cached size 83886080 sectors
/dev/vdd: using cached size 83886080 sectors
devices/global_filter not found in config: defaulting to global_filter = [ "a|.*|" ]
Setting devices/filter to filter = [ "a|^/mnt/.*|", "r|^/dev/loop.*|", "a|^/dev/.*|", "r|.*|" ]
devices/devicesfile not found in config: defaulting to "system.devices"
/dev/block/7:0: using cached size 2097152 sectors
/dev/nbd0: using cached size 0 sectors
/dev/nbd8: using cached size 0 sectors
/dev/vda: using cached size 83886080 sectors
/dev/block/7:1: using cached size 0 sectors
/dev/vda1: using cached size 83883999 sectors
/dev/block/7:2: using cached size 0 sectors
/dev/block/7:3: using cached size 0 sectors
/dev/block/7:4: using cached size 0 sectors
/dev/block/7:5: using cached size 0 sectors
/dev/block/7:6: using cached size 0 sectors
/dev/block/7:7: using cached size 0 sectors
/dev/vdb: using cached size 83886080 sectors
/dev/nbd1: using cached size 0 sectors
/dev/nbd9: using cached size 0 sectors
/dev/vdc: using cached size 83886080 sectors
/dev/vdd: using cached size 83886080 sectors
/dev/nbd2: using cached size 0 sectors
/dev/nbd10: using cached size 0 sectors
/dev/nbd3: using cached size 0 sectors
/dev/nbd11: using cached size 0 sectors
/dev/nbd4: using cached size 0 sectors
/dev/nbd12: using cached size 0 sectors
/dev/nbd5: using cached size 0 sectors
/dev/nbd13: using cached size 0 sectors
/dev/nbd6: using cached size 0 sectors
/dev/nbd14: using cached size 0 sectors
/dev/nbd7: using cached size 0 sectors
/dev/nbd15: using cached size 0 sectors
Obtaining the complete list of VGs to process
Processing VG ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d VxbTWs-fMrC-KErD-Pc2l-NBs6-xffp-7n1qab
Locking /run/lock/lvm/V_ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d WB
metadata/lvs_history_retention_time not found in config: defaulting to 0
/dev/vdd: using cached size 83886080 sectors
Running command for VG ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d VxbTWs-fMrC-KErD-Pc2l-NBs6-xffp-7n1qab
Deactivating logical volume ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d/osd-block-058f4926-6ee7-4165-9a55-db5a1fd22d2f.
activation/verify_udev_operations not found in config: defaulting to 0
Getting driver version
Removing ceph--a8ae9f01--a440--4a0e--8e4d--592d3bce3a9d-osd--block--058f4926--6ee7--4165--9a55--db5a1fd22d2f (252:2)
Deactivated 1 logical volumes in volume group ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d.
0 logical volume(s) in volume group "ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d" now active
Unlocking /run/lock/lvm/V_ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d
global/notify_dbus not found in config: defaulting to 1
[root@ca862389706c /]# echo $?
0
```
Signed-off-by: Sébastien Han <seb@redhat.com>
Ceph requires all topology label being unique. And it is quite common to
have topology.kubernetes.io/region and topology.kubernetes.io/zone being set
to the same value. In this case, let's just ignore the region label.
See https://github.com/rook/rook/issues/9419#issuecomment-1144311818
Signed-off-by: Cheng XU <git@xuc.me>
During the tini's binary removal you overlooked that scenario and
introduced a bug in the OSD startup line.
We were passing the binary path as an argument, which is obviously wrong
and makes the OSD crash at boot.
Signed-off-by: Sébastien Han <seb@redhat.com>
We don't need to run any `cp` command since a destination path already
exists. Using `--no-clobber` is not needed and it results in a noop
anyways.
Instead, let's print the reason why we exit the container.
Signed-off-by: Sébastien Han <seb@redhat.com>
Once we are done cleaning up the content of the encrypted osd, let's
close the main LUKS device.
This avoids having dm devices on the system after the cleanup.
Closes: https://github.com/rook/rook/issues/10181
Signed-off-by: Sébastien Han <seb@redhat.com>
Support is removed for k8s for various limitations such as
priority classes not working and csi driver feature
incompleteness and missing snapshots. Documentation is updated
with the new min version of k8s 1.17 and also the tests are
updated to run on the min version of 1.17.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
This removes double package imports. Example:
```
"github.com/rook/rook/pkg/apis/ceph.rook.io/v1"
cephv1 "github.com/rook/rook/pkg/apis/ceph.rook.io/v1"
```
Only one is now being used as shown in go-staticcheck ST1019
Signed-off-by: Alexander Trost <galexrt@googlemail.com>
After all OSDs are updated to each major release of Ceph, the
require-osd-release version is set to that release so all the latest
functionality for osds can be enabled by Ceph. In Quincy there is
a warning if all osds are on Quincy and this flag is not set, so
this will resolve the health warning.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>