few functions got change as they were deprecated
for ex: ioutil.Readfile change to os.Readfile
ioutil.TempFile change to os.CreateTemp
And fixed golang-ci-lint-issues
Signed-off-by: parth-gr <paarora@redhat.com>
This commit adds code to supply variables required
to enable read affinity for rbd volumes.
It leverages `read_from_replica=localize,crush_locations=key:value`
krbd map option from ceph to serve reads from the most local OSD.
refer:
https://docs.ceph.com/en/latest/man/8/rbd/#kernel-rbd-krbd-options
Signed-off-by: Rakshith R <rar@redhat.com>
The rook-version tag must not be set on the pod spec labels
since it will result in the ceph daemons being restarted every
time there is a rook version update, even if the ceph version
or pod spec was not otherwise updated.
The rook-version tag was being added to the OSD pod spec
due to a shared pointer to the labels. The code intended
to only add the version label to the deployment labels,
but since the pod labels shared the map variable, the
pod unintentionally also had the version label added.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Environment variables are not recommended for secrets in pods since
they can be easily leaked if the environemnt variables are logged.
By mounting the mon secret as a file, the mgr and osd prepare pods
can read the mon secret from a file for better security.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The volume names of a pod must follow the standard naming convention
which does not allow periods in the name. Since the device set name
is used as the pvc name and then the volume name to attach to the osd
prepare job, we must sanitize the name for the invalid character
and replace it.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
This commit implements this corner case during osd-prepare job.
```
The encrypted block is not opened, this is an extreme corner case
The OSD deployment has been removed manually AND the node rebooted
So we need to re-open the block to re-hydrate the OSDInfo.
Handling this case would mean, writing the encryption key on a
temporary file, then call luksOpen to open the encrypted block and
then call ceph-volume to list against the opened encrypted block.
We don't implement this, yet and return an error.
```
When underlying PVC for osd are CSI provisioned, the encrypted device
is closed when PVC is unmounted due to osd pod being deleted.
Therefore, this may occur more frequently and needs to be handled.
This commit implements the fix for the same.
Signed-off-by: Rakshith R <rar@redhat.com>
The fsid, username, and user secrets were from legacy and no
longer needed on the osd daemon or mon pods. This removes the
unnecessary env vars from the pod specs.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
A new variable is added to rook-ceph-operator-config
ConfigMap to allow using loop devices for osd.
This feature is intended to be used for testing purposes only.
Signed-off-by: Shinya Hayashi <shinya-hayashi@cybozu.co.jp>
The osd arguments had a few duplicate lines that could be
factored out and thus simplified for code maintainability.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Introduce a new env variable ROOK_CSI_IMAGE_PULL_POLICY in rook operator configmap which should be used to
customize the imagePullPolicy for the csi driver and imagePullPolicy property in cephVersionSpec for ceph pods.
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
The cluster-wide encryption feature now
supports KMIP (Key Management Interoperability Protocol)
kms.
KMIP is an extensible communication protocol that defines
message formats for the manipulation of cryptographic keys
on a key management server.
For more information, refer:
https://en.wikipedia.org/wiki/Key_Management_Interoperability_Protocol
Signed-off-by: Rakshith R <rar@redhat.com>
The tool gofmt in go 1.19 requires certain formatting
in the comments section for better rendering.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
During certain maintenance tasks the admin will own running
operations on the ceph mons and osds, and the operator should
not interfere with those operations. If the operator sees
any mon in debug mode, every reconcile and mon health check
will be skipped. Thus, mons will not be updated while any
one of them is in maintenance. During OSD reconcile, individual
OSD deployment updates will only be skipped for OSDs that are
actively being debugged.
The debug mode for osd and mon deployments is signaled by
creating the ceph.rook.io/do-not-reconcile label.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
This commit replaces `os.Setenv` with `t.Setenv` in tests. The
environment variable is automatically restored to its original value
when the test and all its subtests complete.
Reference: https://pkg.go.dev/testing#T.Setenv
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
Some context canceled were missed and not reported correctly. We had
multiple reconcilers running in //. Since they all run in goroutine then
will continue to run until they timeout (if timeout is available,
otherwise would run forever until the op is restarted).
Signed-off-by: Sébastien Han <seb@redhat.com>
With octopus coming to end of life, we remove support from
Rook for deploying Ceph Octopus and assume a min version of
Pacific v16. Any checks for octopus or earlier are removed
from the reconciles since they are obsolete.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Ancient OSDs based on LVM and activated by the Rook binary are still in
production. They fail on the deactivation of the VolumeGroup due to
incorrect filter. Setting the OSD as LVM adjusts the filter and make the
deactivate call working.
Repro without and with the correct filter:
```
[root@ca862389706c /]# vgchange -an -vv ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d
global/use_lvmpolld not found in config: defaulting to 1
devices/sysfs_scan not found in config: defaulting to 1
devices/scan_lvs not found in config: defaulting to 0
devices/multipath_component_detection not found in config: defaulting to 1
devices/md_component_detection not found in config: defaulting to 1
devices/fw_raid_component_detection not found in config: defaulting to 0
devices/ignore_suspended_devices not found in config: defaulting to 0
devices/ignore_lvm_mirrors not found in config: defaulting to 1
devices/scan_lvs not found in config: defaulting to 0
devices/allow_mixed_block_sizes not found in config: defaulting to 0
devices/hints not found in config: defaulting to "all"
activation/activation_mode not found in config: defaulting to "degraded"
metadata/record_lvs_history not found in config: defaulting to 0
devices/search_for_devnames not found in config: defaulting to "auto"
activation/monitoring not found in config: defaulting to 1
global/locking_type not found in config: defaulting to 1
global/wait_for_locks not found in config: defaulting to 1
global/prioritise_write_locks not found in config: defaulting to 1
global/locking_dir not found in config: defaulting to "/run/lock/lvm"
devices/md_component_detection not found in config: defaulting to 1
devices/md_component_checks not found in config: defaulting to "auto"
devices/multipath_wwids_file not found in config: defaulting to "/etc/multipath/wwids"
global/use_lvmlockd not found in config: defaulting to 0
report/output_format not found in config: defaulting to "basic"
log/report_command_log not found in config: defaulting to 0
devices/use_devicesfile not found in config: defaulting to 0
/dev/log: stat failed: No such file or directory
devices/global_filter not found in config: defaulting to global_filter = [ "a|.*|" ]
Setting devices/filter to filter = [ "a|^/mnt/.*|", "r|.*|" ]
devices/global_filter not found in config: defaulting to global_filter = [ "a|.*|" ]
Setting devices/filter to filter = [ "a|^/mnt/.*|", "r|.*|" ]
devices/devicesfile not found in config: defaulting to "system.devices"
Obtaining the complete list of VGs to process
VG name on command line not found in list of VGs: ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d
Processing VG ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d
Locking /run/lock/lvm/V_ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d WB
Unlocking /run/lock/lvm/V_ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d
Volume group "ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d" not found
Cannot process volume group ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d
global/notify_dbus not found in config: defaulting to 1
[root@ca862389706c /]# echo $?
5
[root@ca862389706c /]# vi /etc/lvm/lvm.conf
[root@ca862389706c /]# vgchange -an -vv ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d
global/use_lvmpolld not found in config: defaulting to 1
devices/sysfs_scan not found in config: defaulting to 1
devices/scan_lvs not found in config: defaulting to 0
devices/multipath_component_detection not found in config: defaulting to 1
devices/md_component_detection not found in config: defaulting to 1
devices/fw_raid_component_detection not found in config: defaulting to 0
devices/ignore_suspended_devices not found in config: defaulting to 0
devices/ignore_lvm_mirrors not found in config: defaulting to 1
devices/scan_lvs not found in config: defaulting to 0
devices/allow_mixed_block_sizes not found in config: defaulting to 0
devices/hints not found in config: defaulting to "all"
activation/activation_mode not found in config: defaulting to "degraded"
metadata/record_lvs_history not found in config: defaulting to 0
devices/search_for_devnames not found in config: defaulting to "auto"
activation/monitoring not found in config: defaulting to 1
global/locking_type not found in config: defaulting to 1
global/wait_for_locks not found in config: defaulting to 1
global/prioritise_write_locks not found in config: defaulting to 1
global/locking_dir not found in config: defaulting to "/run/lock/lvm"
devices/md_component_detection not found in config: defaulting to 1
devices/md_component_checks not found in config: defaulting to "auto"
devices/multipath_wwids_file not found in config: defaulting to "/etc/multipath/wwids"
global/use_lvmlockd not found in config: defaulting to 0
report/output_format not found in config: defaulting to "basic"
log/report_command_log not found in config: defaulting to 0
devices/use_devicesfile not found in config: defaulting to 0
/dev/log: stat failed: No such file or directory
/dev/block/7:0: size is 2097152 sectors
/dev/nbd0: size is 0 sectors
/dev/nbd8: size is 0 sectors
/dev/vda: size is 83886080 sectors
/dev/block/7:1: size is 0 sectors
/dev/vda1: size is 83883999 sectors
/dev/block/7:2: size is 0 sectors
/dev/block/7:3: size is 0 sectors
/dev/block/7:4: size is 0 sectors
/dev/block/7:5: size is 0 sectors
/dev/block/7:6: size is 0 sectors
/dev/block/7:7: size is 0 sectors
/dev/vdb: size is 83886080 sectors
/dev/nbd1: size is 0 sectors
/dev/nbd9: size is 0 sectors
/dev/vdc: size is 83886080 sectors
/dev/vdd: size is 83886080 sectors
/dev/nbd2: size is 0 sectors
/dev/nbd10: size is 0 sectors
/dev/nbd3: size is 0 sectors
/dev/nbd11: size is 0 sectors
/dev/nbd4: size is 0 sectors
/dev/nbd12: size is 0 sectors
/dev/nbd5: size is 0 sectors
/dev/nbd13: size is 0 sectors
/dev/nbd6: size is 0 sectors
/dev/nbd14: size is 0 sectors
/dev/nbd7: size is 0 sectors
/dev/nbd15: size is 0 sectors
devices/global_filter not found in config: defaulting to global_filter = [ "a|.*|" ]
Setting devices/filter to filter = [ "a|^/mnt/.*|", "r|^/dev/loop.*|", "a|^/dev/.*|", "r|.*|" ]
/dev/block/7:0: using cached size 2097152 sectors
/dev/block/7:0: using cached size 2097152 sectors
/dev/block/7:0: No lvm label detected
/dev/vda: using cached size 83886080 sectors
/dev/vda1: using cached size 83883999 sectors
/dev/vda1: using cached size 83883999 sectors
/dev/vda1: No lvm label detected
/dev/vdb: using cached size 83886080 sectors
/dev/vdb: using cached size 83886080 sectors
/dev/vdc: using cached size 83886080 sectors
/dev/vdc: using cached size 83886080 sectors
/dev/vdd: using cached size 83886080 sectors
/dev/vdd: using cached size 83886080 sectors
devices/global_filter not found in config: defaulting to global_filter = [ "a|.*|" ]
Setting devices/filter to filter = [ "a|^/mnt/.*|", "r|^/dev/loop.*|", "a|^/dev/.*|", "r|.*|" ]
devices/devicesfile not found in config: defaulting to "system.devices"
/dev/block/7:0: using cached size 2097152 sectors
/dev/nbd0: using cached size 0 sectors
/dev/nbd8: using cached size 0 sectors
/dev/vda: using cached size 83886080 sectors
/dev/block/7:1: using cached size 0 sectors
/dev/vda1: using cached size 83883999 sectors
/dev/block/7:2: using cached size 0 sectors
/dev/block/7:3: using cached size 0 sectors
/dev/block/7:4: using cached size 0 sectors
/dev/block/7:5: using cached size 0 sectors
/dev/block/7:6: using cached size 0 sectors
/dev/block/7:7: using cached size 0 sectors
/dev/vdb: using cached size 83886080 sectors
/dev/nbd1: using cached size 0 sectors
/dev/nbd9: using cached size 0 sectors
/dev/vdc: using cached size 83886080 sectors
/dev/vdd: using cached size 83886080 sectors
/dev/nbd2: using cached size 0 sectors
/dev/nbd10: using cached size 0 sectors
/dev/nbd3: using cached size 0 sectors
/dev/nbd11: using cached size 0 sectors
/dev/nbd4: using cached size 0 sectors
/dev/nbd12: using cached size 0 sectors
/dev/nbd5: using cached size 0 sectors
/dev/nbd13: using cached size 0 sectors
/dev/nbd6: using cached size 0 sectors
/dev/nbd14: using cached size 0 sectors
/dev/nbd7: using cached size 0 sectors
/dev/nbd15: using cached size 0 sectors
Obtaining the complete list of VGs to process
Processing VG ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d VxbTWs-fMrC-KErD-Pc2l-NBs6-xffp-7n1qab
Locking /run/lock/lvm/V_ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d WB
metadata/lvs_history_retention_time not found in config: defaulting to 0
/dev/vdd: using cached size 83886080 sectors
Running command for VG ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d VxbTWs-fMrC-KErD-Pc2l-NBs6-xffp-7n1qab
Deactivating logical volume ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d/osd-block-058f4926-6ee7-4165-9a55-db5a1fd22d2f.
activation/verify_udev_operations not found in config: defaulting to 0
Getting driver version
Removing ceph--a8ae9f01--a440--4a0e--8e4d--592d3bce3a9d-osd--block--058f4926--6ee7--4165--9a55--db5a1fd22d2f (252:2)
Deactivated 1 logical volumes in volume group ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d.
0 logical volume(s) in volume group "ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d" now active
Unlocking /run/lock/lvm/V_ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d
global/notify_dbus not found in config: defaulting to 1
[root@ca862389706c /]# echo $?
0
```
Signed-off-by: Sébastien Han <seb@redhat.com>
Ceph requires all topology label being unique. And it is quite common to
have topology.kubernetes.io/region and topology.kubernetes.io/zone being set
to the same value. In this case, let's just ignore the region label.
See https://github.com/rook/rook/issues/9419#issuecomment-1144311818
Signed-off-by: Cheng XU <git@xuc.me>
During the tini's binary removal you overlooked that scenario and
introduced a bug in the OSD startup line.
We were passing the binary path as an argument, which is obviously wrong
and makes the OSD crash at boot.
Signed-off-by: Sébastien Han <seb@redhat.com>
We don't need to run any `cp` command since a destination path already
exists. Using `--no-clobber` is not needed and it results in a noop
anyways.
Instead, let's print the reason why we exit the container.
Signed-off-by: Sébastien Han <seb@redhat.com>
Once we are done cleaning up the content of the encrypted osd, let's
close the main LUKS device.
This avoids having dm devices on the system after the cleanup.
Closes: https://github.com/rook/rook/issues/10181
Signed-off-by: Sébastien Han <seb@redhat.com>
Support is removed for k8s for various limitations such as
priority classes not working and csi driver feature
incompleteness and missing snapshots. Documentation is updated
with the new min version of k8s 1.17 and also the tests are
updated to run on the min version of 1.17.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
This removes double package imports. Example:
```
"github.com/rook/rook/pkg/apis/ceph.rook.io/v1"
cephv1 "github.com/rook/rook/pkg/apis/ceph.rook.io/v1"
```
Only one is now being used as shown in go-staticcheck ST1019
Signed-off-by: Alexander Trost <galexrt@googlemail.com>
After all OSDs are updated to each major release of Ceph, the
require-osd-release version is set to that release so all the latest
functionality for osds can be enabled by Ceph. In Quincy there is
a warning if all osds are on Quincy and this flag is not set, so
this will resolve the health warning.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Previously, the struct maintaining the list of cluster was still
initialized with a cluster item. Then the monitoring check will see that
the cluster is part of the struct already and thus won't run the
monitoring go routine again.
Now each time we cancel the context, we also remove the cluster item
from the map so that when the controller runs again, the monitoring
struct is re-populated and the go routine runs and statuses are updated.
Closes: https://github.com/rook/rook/issues/9911
Signed-off-by: Sébastien Han <seb@redhat.com>
adding observedGeneration field in the cephcluster cr
status for having better control on reconciling,
as observedGeneration field will be updated by the controller
Closes: https://github.com/rook/rook/issues/9673
Signed-off-by: parth-gr <paarora@redhat.com>
The purge job is intended to clean up all the resources
related to an OSD that is to be removed. An OSD with
a metadata, wal, and data PVC was only cleaning up the
data PVC. Now the metadata and wal PVCs will also be
cleaned up.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Added unit test for probes changes and
did re-naming of liveness probes file
as it will contain multiple types of probes
Signed-off-by: parth-gr <paarora@redhat.com>
Prior to this, we were comparing a pointer (the memory address) with a
struct. This was obviously always failing and returned false. We must
dereference the pointer to access the data contained at that memory
location.
Closes: https://github.com/rook/rook/issues/9544
Signed-off-by: Sébastien Han <seb@redhat.com>
This patch brings a mechanism to define arbitrary environment
variables in OSD containers (both prepare and main ones).
It bases on idea proposed by Sebastien Han to use an optional
`ConfigMap` instance named `aook-ceph-daemon-env-override`
as the source of these settings.
The need for the patch comes from the fact that, although
crimson finally exposes the same CLI interface as the classical
OSD, the broadly used development builds have ASan built in.
As ASan, by default, complains if it isn't the very first loaded
DSO, we need a way to set the `ASAN_OPTIONS` environment variable
to `verify_asan_link_order=0` to mitigate the early aborts.
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
The cluster-wide encryption feature now has a new supported key
management system: IBM Key protect. More information about the backend
can be found in Rook documentation under the
Today's implementation stores OSD encryption keys has "Standard" keys.
Signed-off-by: Sébastien Han <seb@redhat.com>
Allow specifying daemon startup probes where we also allow configuring
liveness probes. Startup probes allow Rook to tolerate when Ceph daemons
occasionally take a long time to start up while not also making
Kubernetes liveness probes slower to detect runtime failures of daemons.
Startup probes are beta in Kubernetes 1.18, so we should not enable
probes by default for earlier Kubernetes versions.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
This is handling a tricky scenario where the OSD deployment is manually
removed and the OSD never reconvers. This is unlikely to happen, but
still OSD should be able to run after that action. Essentially after a
manual deletion, we need to run the prepare job again to re-hydrate the
OSD information so that the OSD deployment can be deployed.
On encryption, it is a little bit tricky since ceph-volume list again
the main block won't return anything, so we need to target the encrypted
block to list.
There is another case this PR does not handle, which is the removal of
the OSD deployment and then the node is restarted. This means that the
encrypted container is not opened anymore. However, opening it requires
more work like writing the key on the filesystem (if not coming from the
Kubernete secret, eg,. KMS vault) and then run luksOpen. This is an
extreme corner case probably not worth worrying about for now.
Signed-off-by: Sébastien Han <seb@redhat.com>
parth-grandBlaine Gardner
Rakshith R
Travis Nielsen
Avan Thakkar
Yuma Ogami
Satoru Takeuchi
Shinya Hayashi
Eng Zer Jun
Sébastien Han
Josh Soref
Cheng XU
Alexander Trost
Divyansh Kamboj
Radoslaw Zarzynski
subhamkrai