The cluster-wide encryption feature now
supports KMIP (Key Management Interoperability Protocol)
kms.
KMIP is an extensible communication protocol that defines
message formats for the manipulation of cryptographic keys
on a key management server.
For more information, refer:
https://en.wikipedia.org/wiki/Key_Management_Interoperability_Protocol
Signed-off-by: Rakshith R <rar@redhat.com>
The tool gofmt in go 1.19 requires certain formatting
in the comments section for better rendering.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
During certain maintenance tasks the admin will own running
operations on the ceph mons and osds, and the operator should
not interfere with those operations. If the operator sees
any mon in debug mode, every reconcile and mon health check
will be skipped. Thus, mons will not be updated while any
one of them is in maintenance. During OSD reconcile, individual
OSD deployment updates will only be skipped for OSDs that are
actively being debugged.
The debug mode for osd and mon deployments is signaled by
creating the ceph.rook.io/do-not-reconcile label.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
This commit replaces `os.Setenv` with `t.Setenv` in tests. The
environment variable is automatically restored to its original value
when the test and all its subtests complete.
Reference: https://pkg.go.dev/testing#T.Setenv
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
Some context canceled were missed and not reported correctly. We had
multiple reconcilers running in //. Since they all run in goroutine then
will continue to run until they timeout (if timeout is available,
otherwise would run forever until the op is restarted).
Signed-off-by: Sébastien Han <seb@redhat.com>
With octopus coming to end of life, we remove support from
Rook for deploying Ceph Octopus and assume a min version of
Pacific v16. Any checks for octopus or earlier are removed
from the reconciles since they are obsolete.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Ancient OSDs based on LVM and activated by the Rook binary are still in
production. They fail on the deactivation of the VolumeGroup due to
incorrect filter. Setting the OSD as LVM adjusts the filter and make the
deactivate call working.
Repro without and with the correct filter:
```
[root@ca862389706c /]# vgchange -an -vv ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d
global/use_lvmpolld not found in config: defaulting to 1
devices/sysfs_scan not found in config: defaulting to 1
devices/scan_lvs not found in config: defaulting to 0
devices/multipath_component_detection not found in config: defaulting to 1
devices/md_component_detection not found in config: defaulting to 1
devices/fw_raid_component_detection not found in config: defaulting to 0
devices/ignore_suspended_devices not found in config: defaulting to 0
devices/ignore_lvm_mirrors not found in config: defaulting to 1
devices/scan_lvs not found in config: defaulting to 0
devices/allow_mixed_block_sizes not found in config: defaulting to 0
devices/hints not found in config: defaulting to "all"
activation/activation_mode not found in config: defaulting to "degraded"
metadata/record_lvs_history not found in config: defaulting to 0
devices/search_for_devnames not found in config: defaulting to "auto"
activation/monitoring not found in config: defaulting to 1
global/locking_type not found in config: defaulting to 1
global/wait_for_locks not found in config: defaulting to 1
global/prioritise_write_locks not found in config: defaulting to 1
global/locking_dir not found in config: defaulting to "/run/lock/lvm"
devices/md_component_detection not found in config: defaulting to 1
devices/md_component_checks not found in config: defaulting to "auto"
devices/multipath_wwids_file not found in config: defaulting to "/etc/multipath/wwids"
global/use_lvmlockd not found in config: defaulting to 0
report/output_format not found in config: defaulting to "basic"
log/report_command_log not found in config: defaulting to 0
devices/use_devicesfile not found in config: defaulting to 0
/dev/log: stat failed: No such file or directory
devices/global_filter not found in config: defaulting to global_filter = [ "a|.*|" ]
Setting devices/filter to filter = [ "a|^/mnt/.*|", "r|.*|" ]
devices/global_filter not found in config: defaulting to global_filter = [ "a|.*|" ]
Setting devices/filter to filter = [ "a|^/mnt/.*|", "r|.*|" ]
devices/devicesfile not found in config: defaulting to "system.devices"
Obtaining the complete list of VGs to process
VG name on command line not found in list of VGs: ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d
Processing VG ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d
Locking /run/lock/lvm/V_ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d WB
Unlocking /run/lock/lvm/V_ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d
Volume group "ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d" not found
Cannot process volume group ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d
global/notify_dbus not found in config: defaulting to 1
[root@ca862389706c /]# echo $?
5
[root@ca862389706c /]# vi /etc/lvm/lvm.conf
[root@ca862389706c /]# vgchange -an -vv ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d
global/use_lvmpolld not found in config: defaulting to 1
devices/sysfs_scan not found in config: defaulting to 1
devices/scan_lvs not found in config: defaulting to 0
devices/multipath_component_detection not found in config: defaulting to 1
devices/md_component_detection not found in config: defaulting to 1
devices/fw_raid_component_detection not found in config: defaulting to 0
devices/ignore_suspended_devices not found in config: defaulting to 0
devices/ignore_lvm_mirrors not found in config: defaulting to 1
devices/scan_lvs not found in config: defaulting to 0
devices/allow_mixed_block_sizes not found in config: defaulting to 0
devices/hints not found in config: defaulting to "all"
activation/activation_mode not found in config: defaulting to "degraded"
metadata/record_lvs_history not found in config: defaulting to 0
devices/search_for_devnames not found in config: defaulting to "auto"
activation/monitoring not found in config: defaulting to 1
global/locking_type not found in config: defaulting to 1
global/wait_for_locks not found in config: defaulting to 1
global/prioritise_write_locks not found in config: defaulting to 1
global/locking_dir not found in config: defaulting to "/run/lock/lvm"
devices/md_component_detection not found in config: defaulting to 1
devices/md_component_checks not found in config: defaulting to "auto"
devices/multipath_wwids_file not found in config: defaulting to "/etc/multipath/wwids"
global/use_lvmlockd not found in config: defaulting to 0
report/output_format not found in config: defaulting to "basic"
log/report_command_log not found in config: defaulting to 0
devices/use_devicesfile not found in config: defaulting to 0
/dev/log: stat failed: No such file or directory
/dev/block/7:0: size is 2097152 sectors
/dev/nbd0: size is 0 sectors
/dev/nbd8: size is 0 sectors
/dev/vda: size is 83886080 sectors
/dev/block/7:1: size is 0 sectors
/dev/vda1: size is 83883999 sectors
/dev/block/7:2: size is 0 sectors
/dev/block/7:3: size is 0 sectors
/dev/block/7:4: size is 0 sectors
/dev/block/7:5: size is 0 sectors
/dev/block/7:6: size is 0 sectors
/dev/block/7:7: size is 0 sectors
/dev/vdb: size is 83886080 sectors
/dev/nbd1: size is 0 sectors
/dev/nbd9: size is 0 sectors
/dev/vdc: size is 83886080 sectors
/dev/vdd: size is 83886080 sectors
/dev/nbd2: size is 0 sectors
/dev/nbd10: size is 0 sectors
/dev/nbd3: size is 0 sectors
/dev/nbd11: size is 0 sectors
/dev/nbd4: size is 0 sectors
/dev/nbd12: size is 0 sectors
/dev/nbd5: size is 0 sectors
/dev/nbd13: size is 0 sectors
/dev/nbd6: size is 0 sectors
/dev/nbd14: size is 0 sectors
/dev/nbd7: size is 0 sectors
/dev/nbd15: size is 0 sectors
devices/global_filter not found in config: defaulting to global_filter = [ "a|.*|" ]
Setting devices/filter to filter = [ "a|^/mnt/.*|", "r|^/dev/loop.*|", "a|^/dev/.*|", "r|.*|" ]
/dev/block/7:0: using cached size 2097152 sectors
/dev/block/7:0: using cached size 2097152 sectors
/dev/block/7:0: No lvm label detected
/dev/vda: using cached size 83886080 sectors
/dev/vda1: using cached size 83883999 sectors
/dev/vda1: using cached size 83883999 sectors
/dev/vda1: No lvm label detected
/dev/vdb: using cached size 83886080 sectors
/dev/vdb: using cached size 83886080 sectors
/dev/vdc: using cached size 83886080 sectors
/dev/vdc: using cached size 83886080 sectors
/dev/vdd: using cached size 83886080 sectors
/dev/vdd: using cached size 83886080 sectors
devices/global_filter not found in config: defaulting to global_filter = [ "a|.*|" ]
Setting devices/filter to filter = [ "a|^/mnt/.*|", "r|^/dev/loop.*|", "a|^/dev/.*|", "r|.*|" ]
devices/devicesfile not found in config: defaulting to "system.devices"
/dev/block/7:0: using cached size 2097152 sectors
/dev/nbd0: using cached size 0 sectors
/dev/nbd8: using cached size 0 sectors
/dev/vda: using cached size 83886080 sectors
/dev/block/7:1: using cached size 0 sectors
/dev/vda1: using cached size 83883999 sectors
/dev/block/7:2: using cached size 0 sectors
/dev/block/7:3: using cached size 0 sectors
/dev/block/7:4: using cached size 0 sectors
/dev/block/7:5: using cached size 0 sectors
/dev/block/7:6: using cached size 0 sectors
/dev/block/7:7: using cached size 0 sectors
/dev/vdb: using cached size 83886080 sectors
/dev/nbd1: using cached size 0 sectors
/dev/nbd9: using cached size 0 sectors
/dev/vdc: using cached size 83886080 sectors
/dev/vdd: using cached size 83886080 sectors
/dev/nbd2: using cached size 0 sectors
/dev/nbd10: using cached size 0 sectors
/dev/nbd3: using cached size 0 sectors
/dev/nbd11: using cached size 0 sectors
/dev/nbd4: using cached size 0 sectors
/dev/nbd12: using cached size 0 sectors
/dev/nbd5: using cached size 0 sectors
/dev/nbd13: using cached size 0 sectors
/dev/nbd6: using cached size 0 sectors
/dev/nbd14: using cached size 0 sectors
/dev/nbd7: using cached size 0 sectors
/dev/nbd15: using cached size 0 sectors
Obtaining the complete list of VGs to process
Processing VG ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d VxbTWs-fMrC-KErD-Pc2l-NBs6-xffp-7n1qab
Locking /run/lock/lvm/V_ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d WB
metadata/lvs_history_retention_time not found in config: defaulting to 0
/dev/vdd: using cached size 83886080 sectors
Running command for VG ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d VxbTWs-fMrC-KErD-Pc2l-NBs6-xffp-7n1qab
Deactivating logical volume ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d/osd-block-058f4926-6ee7-4165-9a55-db5a1fd22d2f.
activation/verify_udev_operations not found in config: defaulting to 0
Getting driver version
Removing ceph--a8ae9f01--a440--4a0e--8e4d--592d3bce3a9d-osd--block--058f4926--6ee7--4165--9a55--db5a1fd22d2f (252:2)
Deactivated 1 logical volumes in volume group ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d.
0 logical volume(s) in volume group "ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d" now active
Unlocking /run/lock/lvm/V_ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d
global/notify_dbus not found in config: defaulting to 1
[root@ca862389706c /]# echo $?
0
```
Signed-off-by: Sébastien Han <seb@redhat.com>
Ceph requires all topology label being unique. And it is quite common to
have topology.kubernetes.io/region and topology.kubernetes.io/zone being set
to the same value. In this case, let's just ignore the region label.
See https://github.com/rook/rook/issues/9419#issuecomment-1144311818
Signed-off-by: Cheng XU <git@xuc.me>
During the tini's binary removal you overlooked that scenario and
introduced a bug in the OSD startup line.
We were passing the binary path as an argument, which is obviously wrong
and makes the OSD crash at boot.
Signed-off-by: Sébastien Han <seb@redhat.com>
We don't need to run any `cp` command since a destination path already
exists. Using `--no-clobber` is not needed and it results in a noop
anyways.
Instead, let's print the reason why we exit the container.
Signed-off-by: Sébastien Han <seb@redhat.com>
Once we are done cleaning up the content of the encrypted osd, let's
close the main LUKS device.
This avoids having dm devices on the system after the cleanup.
Closes: https://github.com/rook/rook/issues/10181
Signed-off-by: Sébastien Han <seb@redhat.com>
Support is removed for k8s for various limitations such as
priority classes not working and csi driver feature
incompleteness and missing snapshots. Documentation is updated
with the new min version of k8s 1.17 and also the tests are
updated to run on the min version of 1.17.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
This removes double package imports. Example:
```
"github.com/rook/rook/pkg/apis/ceph.rook.io/v1"
cephv1 "github.com/rook/rook/pkg/apis/ceph.rook.io/v1"
```
Only one is now being used as shown in go-staticcheck ST1019
Signed-off-by: Alexander Trost <galexrt@googlemail.com>
After all OSDs are updated to each major release of Ceph, the
require-osd-release version is set to that release so all the latest
functionality for osds can be enabled by Ceph. In Quincy there is
a warning if all osds are on Quincy and this flag is not set, so
this will resolve the health warning.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Previously, the struct maintaining the list of cluster was still
initialized with a cluster item. Then the monitoring check will see that
the cluster is part of the struct already and thus won't run the
monitoring go routine again.
Now each time we cancel the context, we also remove the cluster item
from the map so that when the controller runs again, the monitoring
struct is re-populated and the go routine runs and statuses are updated.
Closes: https://github.com/rook/rook/issues/9911
Signed-off-by: Sébastien Han <seb@redhat.com>
adding observedGeneration field in the cephcluster cr
status for having better control on reconciling,
as observedGeneration field will be updated by the controller
Closes: https://github.com/rook/rook/issues/9673
Signed-off-by: parth-gr <paarora@redhat.com>
The purge job is intended to clean up all the resources
related to an OSD that is to be removed. An OSD with
a metadata, wal, and data PVC was only cleaning up the
data PVC. Now the metadata and wal PVCs will also be
cleaned up.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Added unit test for probes changes and
did re-naming of liveness probes file
as it will contain multiple types of probes
Signed-off-by: parth-gr <paarora@redhat.com>
Prior to this, we were comparing a pointer (the memory address) with a
struct. This was obviously always failing and returned false. We must
dereference the pointer to access the data contained at that memory
location.
Closes: https://github.com/rook/rook/issues/9544
Signed-off-by: Sébastien Han <seb@redhat.com>
This patch brings a mechanism to define arbitrary environment
variables in OSD containers (both prepare and main ones).
It bases on idea proposed by Sebastien Han to use an optional
`ConfigMap` instance named `aook-ceph-daemon-env-override`
as the source of these settings.
The need for the patch comes from the fact that, although
crimson finally exposes the same CLI interface as the classical
OSD, the broadly used development builds have ASan built in.
As ASan, by default, complains if it isn't the very first loaded
DSO, we need a way to set the `ASAN_OPTIONS` environment variable
to `verify_asan_link_order=0` to mitigate the early aborts.
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
The cluster-wide encryption feature now has a new supported key
management system: IBM Key protect. More information about the backend
can be found in Rook documentation under the
Today's implementation stores OSD encryption keys has "Standard" keys.
Signed-off-by: Sébastien Han <seb@redhat.com>
Allow specifying daemon startup probes where we also allow configuring
liveness probes. Startup probes allow Rook to tolerate when Ceph daemons
occasionally take a long time to start up while not also making
Kubernetes liveness probes slower to detect runtime failures of daemons.
Startup probes are beta in Kubernetes 1.18, so we should not enable
probes by default for earlier Kubernetes versions.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
This is handling a tricky scenario where the OSD deployment is manually
removed and the OSD never reconvers. This is unlikely to happen, but
still OSD should be able to run after that action. Essentially after a
manual deletion, we need to run the prepare job again to re-hydrate the
OSD information so that the OSD deployment can be deployed.
On encryption, it is a little bit tricky since ceph-volume list again
the main block won't return anything, so we need to target the encrypted
block to list.
There is another case this PR does not handle, which is the removal of
the OSD deployment and then the node is restarted. This means that the
encrypted container is not opened anymore. However, opening it requires
more work like writing the key on the filesystem (if not coming from the
Kubernete secret, eg,. KMS vault) and then run luksOpen. This is an
extreme corner case probably not worth worrying about for now.
Signed-off-by: Sébastien Han <seb@redhat.com>
For now, we must run the container with UID 0 and privileged for
multiple reasons:
* the rook binary writes ceph config to /var/lib/rook which is owned by
root
* it's difficult to use /etc/ceph since it will conflict with the
rook-ceph-override configmap AND is also owned by root since it's a
mounted configmap.
* using /etc/ceph might be possible but has other issues with rook's
exec package since the ceph config is built from /var/lib/rook
Closes: https://github.com/rook/rook/issues/9385
Signed-off-by: Sébastien Han <seb@redhat.com>
This commit adds context parameter to k8sutil pvc functions. By this, we
can handle cancellation during API call of pvc resource.
Signed-off-by: Yuichiro Ueno <y1r.ueno@gmail.com>
This commit adds context parameter to k8sutil kvstore functions. By
this, we can handle cancellation during API call of kvstore resource.
Signed-off-by: Yuichiro Ueno <y1r.ueno@gmail.com>
Adding Recommended Labels on the resources created by rook
and using Recommended Labels in the helm chart,
for better visuals and management of k8s object
Closes: https://github.com/rook/rook/issues/8400
Signed-off-by: parth-gr <paarora@redhat.com>
In K8s 1.22 there is a bug in the job name generation that
the job name is truncated an additional 10 characters. This can cause an issue
in the generated pod name if it then ends in a non-alphanumeric character. In that case,
we more aggressively generate a hashed job name.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Skipping upgrade checks was not being honored for OSDs.
Now the flag will be checked and allow the OSDs to be upgraded
without checking for the ok-to-stop condition.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
If we apply useAllNodes to false for the current deployment,
the OSDs should get updated with the individual nodes values and config,
The deviceClass was not updating to the existing OSDs because there was
bug in the check.
The check osdInfo.DeviceClass == "" which should be
checked like this osdInfo.DeviceClass == "None"
Updated the code so OSDs can make use of the devices present
Signed-off-by: parth-gr <paarora@redhat.com>
The cluster info is important context for the cluster controller to
create the cluster, and all the fields must be properly set.
A test cluster name was being set temporarily, resulting in
mons incorrectly getting the wrong cluster CR name. There is no
known issue from the temporary value, it was just exposed by
https://github.com/rook/rook/pull/8678 setting the value to a label.
Now the functions are more clearly named so only unit and
integration tests should be using the test value for the cluster
name where it is not important.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The blockdevmapper securityContext was changed to request a minimal set of
required capabilities for its operation and drop running as privileged.
While the base change works and is valid in terms of the container's copy operation,
it turns out that OpenShift may require some additional configuration not
currently covered by the limited securityContext and the capabilities granted.
To not break those OpenShift deployments, make the blkdevmapper securityContext
listen to the ROOK_HOSTPATH_REQUIRES_PRIVILEGED flag again to set privileged mode.
This flag is true on OpenShift deployments and running as privileged
works around the (missing) configuration problem for now.
To properly drop privileged completely some additional investigation needs
to be done on OpenShift deployments without relying on privileged execution.
Signed-off-by: Omar Pakker <Omar007@users.noreply.github.com>
From ceph v16.2.6 onwards the vault TLS suppport in RGW was added,
include similar changes for RGW.
Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
This commit adds context parameter to k8sutil node functions. By this,
we can handle cancellation during API call of node resource.
Signed-off-by: Yuichiro Ueno <y1r.ueno@gmail.com>
This commit adds context parameter to k8sutil job functions. By this, we
can handle cancellation during API call of job resource.
Signed-off-by: Yuichiro Ueno <y1r.ueno@gmail.com>
The OSD blkdevmapper init container relies on the MKNOD capability,
which it does not actually request.
As a result, deployments fail on Kubernetes clusters that do not
happen to assign this capability to all containers by default.
Solve this by updating the container spec securityContext to
explicitly request the capability it relies on.
Closes: https://github.com/rook/rook/issues/9156
Signed-off-by: Omar Pakker <Omar007@users.noreply.github.com>
Rakshith R
Satoru Takeuchi
Travis Nielsen
Eng Zer Jun
Sébastien Han
Josh Soref
Cheng XU
Alexander Trost
Divyansh Kamboj
parth-gr
Radoslaw Zarzynski
subhamkrai
Blaine Gardner
Yuichiro Ueno
Omar Pakker
Jiffin Tony Thottan