The rbd mirror daemon logs to two files, one for the
lodal mirror daemon and one log from the other cluster
where the mirroring is taking place. The log rotation
was only rotating one of the logs. Now we make the
log rotation more general to catch all of the rbd mirror
logs from different clusters.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
v1beta1 cron jobs have not been created by rook since k8s 1.21
and now are fully removed in 1.25. As long as we still support
older than 1.21 we still need to attempt to delete the v1beta1
cron jobs in case they had been created before an upgrade.
If the deletion fails, now we just log a debug statement since
it is a normal condition, and continue with the v1 cron job
creation.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Mon failover reliability is improved in two scenarios with these changes.
First, if the failover is aborted because of failing to start the new mon
and wait for it to join quorum, the new mon will be fully removed and
the failover rolled back. Previously, the operator would leave the new
mon pod running even if it never joined quorum. Now the timeout for the failover
is increased to wait longer for the new mon, but if it times out, there
are no leftovers from the cancelled failover.
Second, if the operator is failed in the middle of a failover
before the old mon is removed, the operator needs to remove an extra
mon. The operator was only selecting a random mon for removal instead
of honoring the stretch cluster topology or looking for mons
on the same node that could be removed to restore the correct
desired number of mons.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
we are already deleting v1beta1 cronJob while creating v1 cronJob,
so no need to delete v1beta while v1 cronJob deltion.
Signed-off-by: subhamkrai <srai@redhat.com>
The tool gofmt in go 1.19 requires certain formatting
in the comments section for better rendering.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
During certain maintenance tasks the admin will own running
operations on the ceph mons and osds, and the operator should
not interfere with those operations. If the operator sees
any mon in debug mode, every reconcile and mon health check
will be skipped. Thus, mons will not be updated while any
one of them is in maintenance. During OSD reconcile, individual
OSD deployment updates will only be skipped for OSDs that are
actively being debugged.
The debug mode for osd and mon deployments is signaled by
creating the ceph.rook.io/do-not-reconcile label.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The RGW support server side encryption with help of s3 protocol, till
now the `sse:kms` was support in which keys will be provided by the user
and but it will be saved in external management service like vault. Now
the support for `sse:s3` is added so the entire encryption key
management is performed by RGW itsels.
Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
This commit replaces `os.Setenv` with `t.Setenv` in tests. The
environment variable is automatically restored to its original value
when the test and all its subtests complete.
Reference: https://pkg.go.dev/testing#T.Setenv
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
When the mgr services are updated by the operator to set the
active mgr properly, skip services that do not currently
have a selector set.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Some context canceled were missed and not reported correctly. We had
multiple reconcilers running in //. Since they all run in goroutine then
will continue to run until they timeout (if timeout is available,
otherwise would run forever until the op is restarted).
Signed-off-by: Sébastien Han <seb@redhat.com>
With octopus coming to end of life, we remove support from
Rook for deploying Ceph Octopus and assume a min version of
Pacific v16. Any checks for octopus or earlier are removed
from the reconciles since they are obsolete.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
In new deployments we always want to disable the insecure global id.
If the mgr deployment is newly created, we know it's a new cluster
and the mons are in quorum, so we can go ahead and disable the
insecure global IDs.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
earlier we're deleting v1beta1 always before creating
v1 job. Now, we delete only when k8s version is
latest(which supports v1 job) and v1 job is not created
which means this is the first time and this is the only time we'll
delete v1beta1.
Signed-off-by: subhamkrai <srai@redhat.com>
Ancient OSDs based on LVM and activated by the Rook binary are still in
production. They fail on the deactivation of the VolumeGroup due to
incorrect filter. Setting the OSD as LVM adjusts the filter and make the
deactivate call working.
Repro without and with the correct filter:
```
[root@ca862389706c /]# vgchange -an -vv ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d
global/use_lvmpolld not found in config: defaulting to 1
devices/sysfs_scan not found in config: defaulting to 1
devices/scan_lvs not found in config: defaulting to 0
devices/multipath_component_detection not found in config: defaulting to 1
devices/md_component_detection not found in config: defaulting to 1
devices/fw_raid_component_detection not found in config: defaulting to 0
devices/ignore_suspended_devices not found in config: defaulting to 0
devices/ignore_lvm_mirrors not found in config: defaulting to 1
devices/scan_lvs not found in config: defaulting to 0
devices/allow_mixed_block_sizes not found in config: defaulting to 0
devices/hints not found in config: defaulting to "all"
activation/activation_mode not found in config: defaulting to "degraded"
metadata/record_lvs_history not found in config: defaulting to 0
devices/search_for_devnames not found in config: defaulting to "auto"
activation/monitoring not found in config: defaulting to 1
global/locking_type not found in config: defaulting to 1
global/wait_for_locks not found in config: defaulting to 1
global/prioritise_write_locks not found in config: defaulting to 1
global/locking_dir not found in config: defaulting to "/run/lock/lvm"
devices/md_component_detection not found in config: defaulting to 1
devices/md_component_checks not found in config: defaulting to "auto"
devices/multipath_wwids_file not found in config: defaulting to "/etc/multipath/wwids"
global/use_lvmlockd not found in config: defaulting to 0
report/output_format not found in config: defaulting to "basic"
log/report_command_log not found in config: defaulting to 0
devices/use_devicesfile not found in config: defaulting to 0
/dev/log: stat failed: No such file or directory
devices/global_filter not found in config: defaulting to global_filter = [ "a|.*|" ]
Setting devices/filter to filter = [ "a|^/mnt/.*|", "r|.*|" ]
devices/global_filter not found in config: defaulting to global_filter = [ "a|.*|" ]
Setting devices/filter to filter = [ "a|^/mnt/.*|", "r|.*|" ]
devices/devicesfile not found in config: defaulting to "system.devices"
Obtaining the complete list of VGs to process
VG name on command line not found in list of VGs: ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d
Processing VG ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d
Locking /run/lock/lvm/V_ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d WB
Unlocking /run/lock/lvm/V_ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d
Volume group "ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d" not found
Cannot process volume group ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d
global/notify_dbus not found in config: defaulting to 1
[root@ca862389706c /]# echo $?
5
[root@ca862389706c /]# vi /etc/lvm/lvm.conf
[root@ca862389706c /]# vgchange -an -vv ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d
global/use_lvmpolld not found in config: defaulting to 1
devices/sysfs_scan not found in config: defaulting to 1
devices/scan_lvs not found in config: defaulting to 0
devices/multipath_component_detection not found in config: defaulting to 1
devices/md_component_detection not found in config: defaulting to 1
devices/fw_raid_component_detection not found in config: defaulting to 0
devices/ignore_suspended_devices not found in config: defaulting to 0
devices/ignore_lvm_mirrors not found in config: defaulting to 1
devices/scan_lvs not found in config: defaulting to 0
devices/allow_mixed_block_sizes not found in config: defaulting to 0
devices/hints not found in config: defaulting to "all"
activation/activation_mode not found in config: defaulting to "degraded"
metadata/record_lvs_history not found in config: defaulting to 0
devices/search_for_devnames not found in config: defaulting to "auto"
activation/monitoring not found in config: defaulting to 1
global/locking_type not found in config: defaulting to 1
global/wait_for_locks not found in config: defaulting to 1
global/prioritise_write_locks not found in config: defaulting to 1
global/locking_dir not found in config: defaulting to "/run/lock/lvm"
devices/md_component_detection not found in config: defaulting to 1
devices/md_component_checks not found in config: defaulting to "auto"
devices/multipath_wwids_file not found in config: defaulting to "/etc/multipath/wwids"
global/use_lvmlockd not found in config: defaulting to 0
report/output_format not found in config: defaulting to "basic"
log/report_command_log not found in config: defaulting to 0
devices/use_devicesfile not found in config: defaulting to 0
/dev/log: stat failed: No such file or directory
/dev/block/7:0: size is 2097152 sectors
/dev/nbd0: size is 0 sectors
/dev/nbd8: size is 0 sectors
/dev/vda: size is 83886080 sectors
/dev/block/7:1: size is 0 sectors
/dev/vda1: size is 83883999 sectors
/dev/block/7:2: size is 0 sectors
/dev/block/7:3: size is 0 sectors
/dev/block/7:4: size is 0 sectors
/dev/block/7:5: size is 0 sectors
/dev/block/7:6: size is 0 sectors
/dev/block/7:7: size is 0 sectors
/dev/vdb: size is 83886080 sectors
/dev/nbd1: size is 0 sectors
/dev/nbd9: size is 0 sectors
/dev/vdc: size is 83886080 sectors
/dev/vdd: size is 83886080 sectors
/dev/nbd2: size is 0 sectors
/dev/nbd10: size is 0 sectors
/dev/nbd3: size is 0 sectors
/dev/nbd11: size is 0 sectors
/dev/nbd4: size is 0 sectors
/dev/nbd12: size is 0 sectors
/dev/nbd5: size is 0 sectors
/dev/nbd13: size is 0 sectors
/dev/nbd6: size is 0 sectors
/dev/nbd14: size is 0 sectors
/dev/nbd7: size is 0 sectors
/dev/nbd15: size is 0 sectors
devices/global_filter not found in config: defaulting to global_filter = [ "a|.*|" ]
Setting devices/filter to filter = [ "a|^/mnt/.*|", "r|^/dev/loop.*|", "a|^/dev/.*|", "r|.*|" ]
/dev/block/7:0: using cached size 2097152 sectors
/dev/block/7:0: using cached size 2097152 sectors
/dev/block/7:0: No lvm label detected
/dev/vda: using cached size 83886080 sectors
/dev/vda1: using cached size 83883999 sectors
/dev/vda1: using cached size 83883999 sectors
/dev/vda1: No lvm label detected
/dev/vdb: using cached size 83886080 sectors
/dev/vdb: using cached size 83886080 sectors
/dev/vdc: using cached size 83886080 sectors
/dev/vdc: using cached size 83886080 sectors
/dev/vdd: using cached size 83886080 sectors
/dev/vdd: using cached size 83886080 sectors
devices/global_filter not found in config: defaulting to global_filter = [ "a|.*|" ]
Setting devices/filter to filter = [ "a|^/mnt/.*|", "r|^/dev/loop.*|", "a|^/dev/.*|", "r|.*|" ]
devices/devicesfile not found in config: defaulting to "system.devices"
/dev/block/7:0: using cached size 2097152 sectors
/dev/nbd0: using cached size 0 sectors
/dev/nbd8: using cached size 0 sectors
/dev/vda: using cached size 83886080 sectors
/dev/block/7:1: using cached size 0 sectors
/dev/vda1: using cached size 83883999 sectors
/dev/block/7:2: using cached size 0 sectors
/dev/block/7:3: using cached size 0 sectors
/dev/block/7:4: using cached size 0 sectors
/dev/block/7:5: using cached size 0 sectors
/dev/block/7:6: using cached size 0 sectors
/dev/block/7:7: using cached size 0 sectors
/dev/vdb: using cached size 83886080 sectors
/dev/nbd1: using cached size 0 sectors
/dev/nbd9: using cached size 0 sectors
/dev/vdc: using cached size 83886080 sectors
/dev/vdd: using cached size 83886080 sectors
/dev/nbd2: using cached size 0 sectors
/dev/nbd10: using cached size 0 sectors
/dev/nbd3: using cached size 0 sectors
/dev/nbd11: using cached size 0 sectors
/dev/nbd4: using cached size 0 sectors
/dev/nbd12: using cached size 0 sectors
/dev/nbd5: using cached size 0 sectors
/dev/nbd13: using cached size 0 sectors
/dev/nbd6: using cached size 0 sectors
/dev/nbd14: using cached size 0 sectors
/dev/nbd7: using cached size 0 sectors
/dev/nbd15: using cached size 0 sectors
Obtaining the complete list of VGs to process
Processing VG ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d VxbTWs-fMrC-KErD-Pc2l-NBs6-xffp-7n1qab
Locking /run/lock/lvm/V_ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d WB
metadata/lvs_history_retention_time not found in config: defaulting to 0
/dev/vdd: using cached size 83886080 sectors
Running command for VG ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d VxbTWs-fMrC-KErD-Pc2l-NBs6-xffp-7n1qab
Deactivating logical volume ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d/osd-block-058f4926-6ee7-4165-9a55-db5a1fd22d2f.
activation/verify_udev_operations not found in config: defaulting to 0
Getting driver version
Removing ceph--a8ae9f01--a440--4a0e--8e4d--592d3bce3a9d-osd--block--058f4926--6ee7--4165--9a55--db5a1fd22d2f (252:2)
Deactivated 1 logical volumes in volume group ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d.
0 logical volume(s) in volume group "ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d" now active
Unlocking /run/lock/lvm/V_ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d
global/notify_dbus not found in config: defaulting to 1
[root@ca862389706c /]# echo $?
0
```
Signed-off-by: Sébastien Han <seb@redhat.com>
For the ceph telemetry report the csi version, k8s version,
and metrics from the desired state in the cephcluster CR such
as mon count, stretch cluster, number of device sets, and
so on.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Ceph requires all topology label being unique. And it is quite common to
have topology.kubernetes.io/region and topology.kubernetes.io/zone being set
to the same value. In this case, let's just ignore the region label.
See https://github.com/rook/rook/issues/9419#issuecomment-1144311818
Signed-off-by: Cheng XU <git@xuc.me>
During the tini's binary removal you overlooked that scenario and
introduced a bug in the OSD startup line.
We were passing the binary path as an argument, which is obviously wrong
and makes the OSD crash at boot.
Signed-off-by: Sébastien Han <seb@redhat.com>
The Rook and Ceph teams settled on using slashes to separate Rook
telemetry items into a hierarchy in https://github.com/rook/rook/pull/10254.
Rename the config-key used to `rook/version`, and create a `telemetry`
package to define the telemetry key constants.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
Ceph upstream telemetry (which Rook users could optionally enable via
Ceph CLI or Dashboard) does not easily identify if a cluster is managed
by Rook. Identifying Rook clusters is possible only if users also enable
the rook mgr module, which is optional and not often used.
To get better upstream insights into Ceph clusters managed by Rook, set
a new 'rook-version' field in Ceph's generic k/v store. This field can
be used to determine if a cluster is managed by Rook by its presence.
It additionally provides information about the Rook version used to
manage the cluster, which is another piece of potentially useful
information.
Regarding Ceph cluster identification via telemetry:
Ceph telemetry is anonymous and uses a different ID from the Ceph
cluster ID. Identifying the Rook version in telemetry does not add
directly identifying information. Therefore, there should be little
risk that adding the Rook version field will decrease anonymity of Ceph
clusters via telemetry.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
We don't need to run any `cp` command since a destination path already
exists. Using `--no-clobber` is not needed and it results in a noop
anyways.
Instead, let's print the reason why we exit the container.
Signed-off-by: Sébastien Han <seb@redhat.com>
Once we are done cleaning up the content of the encrypted osd, let's
close the main LUKS device.
This avoids having dm devices on the system after the cleanup.
Closes: https://github.com/rook/rook/issues/10181
Signed-off-by: Sébastien Han <seb@redhat.com>
The CSI package needs to load clusterInfo, today this code is in the mon
package which makes the call of LoadClusterInfo impossible without
having a circular import.
Signed-off-by: Sébastien Han <seb@redhat.com>
Support is removed for k8s for various limitations such as
priority classes not working and csi driver feature
incompleteness and missing snapshots. Documentation is updated
with the new min version of k8s 1.17 and also the tests are
updated to run on the min version of 1.17.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
This removes double package imports. Example:
```
"github.com/rook/rook/pkg/apis/ceph.rook.io/v1"
cephv1 "github.com/rook/rook/pkg/apis/ceph.rook.io/v1"
```
Only one is now being used as shown in go-staticcheck ST1019
Signed-off-by: Alexander Trost <galexrt@googlemail.com>
If a mon is added or removed in a ceph
cluster all the CSI clusterID's belongs
to the ceph cluster should also gets updated.
Added a new cluster identifier to the csi configmap,
i.e ceph clusterNamespace which helps to identify
the clusterID's belongs to that ceph cluster.
Note:- to support the upgraded clusters,we
can ask users to update the mon IP's for the
problamatic clusterID as it wont be updated
by the operator.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
After all OSDs are updated to each major release of Ceph, the
require-osd-release version is set to that release so all the latest
functionality for osds can be enabled by Ceph. In Quincy there is
a warning if all osds are on Quincy and this flag is not set, so
this will resolve the health warning.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
This introduces a new CRD to add the ability
to create rados namespace for a given
ceph block pool. Typically the name of the pool
is the name of the blockpool created by rook.
Closes: #7035
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
Previously, the struct maintaining the list of cluster was still
initialized with a cluster item. Then the monitoring check will see that
the cluster is part of the struct already and thus won't run the
monitoring go routine again.
Now each time we cancel the context, we also remove the cluster item
from the map so that when the controller runs again, the monitoring
struct is re-populated and the go routine runs and statuses are updated.
Closes: https://github.com/rook/rook/issues/9911
Signed-off-by: Sébastien Han <seb@redhat.com>
The prometheus rules had been previously created if the cephcluster CR
setting monitoring.enabled was set to true. The rules were not customizable
and therefore not flexible enough. Now the rules are installed by the helm
chart. To customize the rules, a post-processor can be applied to the helm
chart.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Older kernels still fail to connect for rbd/cephfs with
the msgr2 port due to a difference in http protocol expected.
For now, we keep the default port as msgr1 for the csi driver
and other client connections, unless msgr2 is strictly required
for encryption or compression.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
adding observedGeneration field in the cephcluster cr
status for having better control on reconciling,
as observedGeneration field will be updated by the controller
Closes: https://github.com/rook/rook/issues/9673
Signed-off-by: parth-gr <paarora@redhat.com>
Satoru Takeuchi
Travis Nielsen
subhamkrai
Joseph Lee
Jiffin Tony Thottan
Eng Zer Jun
Sébastien Han
Josh Soref
Cheng XU
Blaine Gardner
FengJiankui
Alexander Trost
Madhu Rajanna
Divyansh Kamboj
parth-gr