we have noticed multiple failures because of the probe
failing, most of the time it's due to fewer resources.
But increasing timeout fixes that, so increasing the
probe timeout to 2s from default 1s so that it will
give more time to probe before failing.
Signed-off-by: subhamkrai <srai@redhat.com>
Ancient OSDs based on LVM and activated by the Rook binary are still in
production. They fail on the deactivation of the VolumeGroup due to
incorrect filter. Setting the OSD as LVM adjusts the filter and make the
deactivate call working.
Repro without and with the correct filter:
```
[root@ca862389706c /]# vgchange -an -vv ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d
global/use_lvmpolld not found in config: defaulting to 1
devices/sysfs_scan not found in config: defaulting to 1
devices/scan_lvs not found in config: defaulting to 0
devices/multipath_component_detection not found in config: defaulting to 1
devices/md_component_detection not found in config: defaulting to 1
devices/fw_raid_component_detection not found in config: defaulting to 0
devices/ignore_suspended_devices not found in config: defaulting to 0
devices/ignore_lvm_mirrors not found in config: defaulting to 1
devices/scan_lvs not found in config: defaulting to 0
devices/allow_mixed_block_sizes not found in config: defaulting to 0
devices/hints not found in config: defaulting to "all"
activation/activation_mode not found in config: defaulting to "degraded"
metadata/record_lvs_history not found in config: defaulting to 0
devices/search_for_devnames not found in config: defaulting to "auto"
activation/monitoring not found in config: defaulting to 1
global/locking_type not found in config: defaulting to 1
global/wait_for_locks not found in config: defaulting to 1
global/prioritise_write_locks not found in config: defaulting to 1
global/locking_dir not found in config: defaulting to "/run/lock/lvm"
devices/md_component_detection not found in config: defaulting to 1
devices/md_component_checks not found in config: defaulting to "auto"
devices/multipath_wwids_file not found in config: defaulting to "/etc/multipath/wwids"
global/use_lvmlockd not found in config: defaulting to 0
report/output_format not found in config: defaulting to "basic"
log/report_command_log not found in config: defaulting to 0
devices/use_devicesfile not found in config: defaulting to 0
/dev/log: stat failed: No such file or directory
devices/global_filter not found in config: defaulting to global_filter = [ "a|.*|" ]
Setting devices/filter to filter = [ "a|^/mnt/.*|", "r|.*|" ]
devices/global_filter not found in config: defaulting to global_filter = [ "a|.*|" ]
Setting devices/filter to filter = [ "a|^/mnt/.*|", "r|.*|" ]
devices/devicesfile not found in config: defaulting to "system.devices"
Obtaining the complete list of VGs to process
VG name on command line not found in list of VGs: ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d
Processing VG ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d
Locking /run/lock/lvm/V_ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d WB
Unlocking /run/lock/lvm/V_ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d
Volume group "ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d" not found
Cannot process volume group ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d
global/notify_dbus not found in config: defaulting to 1
[root@ca862389706c /]# echo $?
5
[root@ca862389706c /]# vi /etc/lvm/lvm.conf
[root@ca862389706c /]# vgchange -an -vv ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d
global/use_lvmpolld not found in config: defaulting to 1
devices/sysfs_scan not found in config: defaulting to 1
devices/scan_lvs not found in config: defaulting to 0
devices/multipath_component_detection not found in config: defaulting to 1
devices/md_component_detection not found in config: defaulting to 1
devices/fw_raid_component_detection not found in config: defaulting to 0
devices/ignore_suspended_devices not found in config: defaulting to 0
devices/ignore_lvm_mirrors not found in config: defaulting to 1
devices/scan_lvs not found in config: defaulting to 0
devices/allow_mixed_block_sizes not found in config: defaulting to 0
devices/hints not found in config: defaulting to "all"
activation/activation_mode not found in config: defaulting to "degraded"
metadata/record_lvs_history not found in config: defaulting to 0
devices/search_for_devnames not found in config: defaulting to "auto"
activation/monitoring not found in config: defaulting to 1
global/locking_type not found in config: defaulting to 1
global/wait_for_locks not found in config: defaulting to 1
global/prioritise_write_locks not found in config: defaulting to 1
global/locking_dir not found in config: defaulting to "/run/lock/lvm"
devices/md_component_detection not found in config: defaulting to 1
devices/md_component_checks not found in config: defaulting to "auto"
devices/multipath_wwids_file not found in config: defaulting to "/etc/multipath/wwids"
global/use_lvmlockd not found in config: defaulting to 0
report/output_format not found in config: defaulting to "basic"
log/report_command_log not found in config: defaulting to 0
devices/use_devicesfile not found in config: defaulting to 0
/dev/log: stat failed: No such file or directory
/dev/block/7:0: size is 2097152 sectors
/dev/nbd0: size is 0 sectors
/dev/nbd8: size is 0 sectors
/dev/vda: size is 83886080 sectors
/dev/block/7:1: size is 0 sectors
/dev/vda1: size is 83883999 sectors
/dev/block/7:2: size is 0 sectors
/dev/block/7:3: size is 0 sectors
/dev/block/7:4: size is 0 sectors
/dev/block/7:5: size is 0 sectors
/dev/block/7:6: size is 0 sectors
/dev/block/7:7: size is 0 sectors
/dev/vdb: size is 83886080 sectors
/dev/nbd1: size is 0 sectors
/dev/nbd9: size is 0 sectors
/dev/vdc: size is 83886080 sectors
/dev/vdd: size is 83886080 sectors
/dev/nbd2: size is 0 sectors
/dev/nbd10: size is 0 sectors
/dev/nbd3: size is 0 sectors
/dev/nbd11: size is 0 sectors
/dev/nbd4: size is 0 sectors
/dev/nbd12: size is 0 sectors
/dev/nbd5: size is 0 sectors
/dev/nbd13: size is 0 sectors
/dev/nbd6: size is 0 sectors
/dev/nbd14: size is 0 sectors
/dev/nbd7: size is 0 sectors
/dev/nbd15: size is 0 sectors
devices/global_filter not found in config: defaulting to global_filter = [ "a|.*|" ]
Setting devices/filter to filter = [ "a|^/mnt/.*|", "r|^/dev/loop.*|", "a|^/dev/.*|", "r|.*|" ]
/dev/block/7:0: using cached size 2097152 sectors
/dev/block/7:0: using cached size 2097152 sectors
/dev/block/7:0: No lvm label detected
/dev/vda: using cached size 83886080 sectors
/dev/vda1: using cached size 83883999 sectors
/dev/vda1: using cached size 83883999 sectors
/dev/vda1: No lvm label detected
/dev/vdb: using cached size 83886080 sectors
/dev/vdb: using cached size 83886080 sectors
/dev/vdc: using cached size 83886080 sectors
/dev/vdc: using cached size 83886080 sectors
/dev/vdd: using cached size 83886080 sectors
/dev/vdd: using cached size 83886080 sectors
devices/global_filter not found in config: defaulting to global_filter = [ "a|.*|" ]
Setting devices/filter to filter = [ "a|^/mnt/.*|", "r|^/dev/loop.*|", "a|^/dev/.*|", "r|.*|" ]
devices/devicesfile not found in config: defaulting to "system.devices"
/dev/block/7:0: using cached size 2097152 sectors
/dev/nbd0: using cached size 0 sectors
/dev/nbd8: using cached size 0 sectors
/dev/vda: using cached size 83886080 sectors
/dev/block/7:1: using cached size 0 sectors
/dev/vda1: using cached size 83883999 sectors
/dev/block/7:2: using cached size 0 sectors
/dev/block/7:3: using cached size 0 sectors
/dev/block/7:4: using cached size 0 sectors
/dev/block/7:5: using cached size 0 sectors
/dev/block/7:6: using cached size 0 sectors
/dev/block/7:7: using cached size 0 sectors
/dev/vdb: using cached size 83886080 sectors
/dev/nbd1: using cached size 0 sectors
/dev/nbd9: using cached size 0 sectors
/dev/vdc: using cached size 83886080 sectors
/dev/vdd: using cached size 83886080 sectors
/dev/nbd2: using cached size 0 sectors
/dev/nbd10: using cached size 0 sectors
/dev/nbd3: using cached size 0 sectors
/dev/nbd11: using cached size 0 sectors
/dev/nbd4: using cached size 0 sectors
/dev/nbd12: using cached size 0 sectors
/dev/nbd5: using cached size 0 sectors
/dev/nbd13: using cached size 0 sectors
/dev/nbd6: using cached size 0 sectors
/dev/nbd14: using cached size 0 sectors
/dev/nbd7: using cached size 0 sectors
/dev/nbd15: using cached size 0 sectors
Obtaining the complete list of VGs to process
Processing VG ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d VxbTWs-fMrC-KErD-Pc2l-NBs6-xffp-7n1qab
Locking /run/lock/lvm/V_ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d WB
metadata/lvs_history_retention_time not found in config: defaulting to 0
/dev/vdd: using cached size 83886080 sectors
Running command for VG ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d VxbTWs-fMrC-KErD-Pc2l-NBs6-xffp-7n1qab
Deactivating logical volume ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d/osd-block-058f4926-6ee7-4165-9a55-db5a1fd22d2f.
activation/verify_udev_operations not found in config: defaulting to 0
Getting driver version
Removing ceph--a8ae9f01--a440--4a0e--8e4d--592d3bce3a9d-osd--block--058f4926--6ee7--4165--9a55--db5a1fd22d2f (252:2)
Deactivated 1 logical volumes in volume group ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d.
0 logical volume(s) in volume group "ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d" now active
Unlocking /run/lock/lvm/V_ceph-a8ae9f01-a440-4a0e-8e4d-592d3bce3a9d
global/notify_dbus not found in config: defaulting to 1
[root@ca862389706c /]# echo $?
0
```
Signed-off-by: Sébastien Han <seb@redhat.com>
Block deletion of CephFilesystems when there are any raw Ceph
subvolumegroups present that have subvolumes in them. Empty
subvolumegroups will not block deletion.
One important subvolume group is "csi" which is the default location
where CSI subvolumes are kept. If this group is empty, it means that
there are no PVCs created based on the CephFilesystem in question. This
also holds true if there are external consumers of the filesystem in
external cluster mode.
Similarly, if there are any subvolumegroups (for example "_nogroup",
which includes subvolumes in the filesystem root) that contain
manually-created subvolumes, Rook will also see this and block deletion.
This comes into play currently with manually-created NFS exports.
A work-in progress aims to create a Ceph-CSI NFS export provisioner
which will likely create subvolumes in the "csi" group as well. This
implementation will catch this case also.
Rook still checks for CephFilesystemSubVolumeGroups explicitly in
addition to the check added here. This is to ensure that even empty
groups will block deletion if they are created via this CR type.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
For the ceph telemetry report the csi version, k8s version,
and metrics from the desired state in the cephcluster CR such
as mon count, stretch cluster, number of device sets, and
so on.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
updating node-driver-registrar to v2.5.1 release
, nfsplugin to 4.0.0 and csi-snapshotter to v6.0.1
Note:- with snapshotter v6.0.1 v1beta1 snapshot CRD
is not supported. If you want betav1 snapshot support
please switch to v5.x.x snapshotter.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
If csi is configured not the use the
hostnetworking, deploy the holder
pod for executing the commands with nsenter.
The implementation is same as multus.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
Ceph requires all topology label being unique. And it is quite common to
have topology.kubernetes.io/region and topology.kubernetes.io/zone being set
to the same value. In this case, let's just ignore the region label.
See https://github.com/rook/rook/issues/9419#issuecomment-1144311818
Signed-off-by: Cheng XU <git@xuc.me>
Return the error instead of hiding it, err could be non-nil for a lot of
reasons, eg: a canceled context or a non-existing object.
Signed-off-by: Sébastien Han <seb@redhat.com>
If some clusters don't have their monitors up and running, we don't have
a clusterInfo (internal detail struct). We don't want to block the csi
controller reconcile if all the clusters are not ready. So let's proceed
with the elements that are ready and reconcile again to see if others
have made progress.
Signed-off-by: Sébastien Han <seb@redhat.com>
During deletion of a CephBlockPool CR, the underlying ceph
pool was not being deleted. During the deletion sequence
this is due to the spec being refreshed upon a call to
check for dependents in ReportDeletionNotBlockedDueToDependents().
The pool name was then coming up blank during the deletion
request and rook of course then wasn't finding the pool
to delete.
Now Rook properly sets the pool name in the named spec
every time it is converted internally from a CephBlockPool
type to a NamedPoolSpec.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
This argument in csi-provisioner and csi-snapshotter sidecar allows us
to receive additional metadata in the CreateVolume and CreateSnapshot()
request.
For ex:
as part of CreateVolume request:
csi.storage.k8s.io/pvc/name
csi.storage.k8s.io/pvc/namespace
csi.storage.k8s.io/pv/name
and
as part of CreateSnapshot request:
csi.storage.k8s.io/volumesnapshot/name
csi.storage.k8s.io/volumesnapshot/namespace
csi.storage.k8s.io/volumesnapshotcontent/name
This is useful information which can be used depend on the use case we have
at our csi driver level. The features like adding metadata to CephFS subvolume,
RBD image or snapshot can consume this based on the need.
Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
Ceph added a check recently that prevents creation of the
built-in pools that start with a dot, such as .nfs, .mgr,
and .rgw.root. These pools require a --yes-i-really-mean-it
flag to be passed to force the creation now.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
During the tini's binary removal you overlooked that scenario and
introduced a bug in the OSD startup line.
We were passing the binary path as an argument, which is obviously wrong
and makes the OSD crash at boot.
Signed-off-by: Sébastien Han <seb@redhat.com>
The Rook and Ceph teams settled on using slashes to separate Rook
telemetry items into a hierarchy in https://github.com/rook/rook/pull/10254.
Rename the config-key used to `rook/version`, and create a `telemetry`
package to define the telemetry key constants.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
Ceph upstream telemetry (which Rook users could optionally enable via
Ceph CLI or Dashboard) does not easily identify if a cluster is managed
by Rook. Identifying Rook clusters is possible only if users also enable
the rook mgr module, which is optional and not often used.
To get better upstream insights into Ceph clusters managed by Rook, set
a new 'rook-version' field in Ceph's generic k/v store. This field can
be used to determine if a cluster is managed by Rook by its presence.
It additionally provides information about the Rook version used to
manage the cluster, which is another piece of potentially useful
information.
Regarding Ceph cluster identification via telemetry:
Ceph telemetry is anonymous and uses a different ID from the Ceph
cluster ID. Identifying the Rook version in telemetry does not add
directly identifying information. Therefore, there should be little
risk that adding the Rook version field will decrease anonymity of Ceph
clusters via telemetry.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
The startup probe for the OSD has been too aggressive to kill the OSD
in case the OSD is taking some time to start. The OSD may be self-optimizing,
scrubbing, or some other internal operation before it is ready to start.
Rather than disable the startup probe completely, the default is now
to retry for two hours in case the OSD is performing those operations.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
We don't need to run any `cp` command since a destination path already
exists. Using `--no-clobber` is not needed and it results in a noop
anyways.
Instead, let's print the reason why we exit the container.
Signed-off-by: Sébastien Han <seb@redhat.com>
Since Quincy is released, Ceph master builds will soon
be tagged with the reef v18 version. This adds the capability
for rook to check for the reef version of Ceph.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Once we are done cleaning up the content of the encrypted osd, let's
close the main LUKS device.
This avoids having dm devices on the system after the cleanup.
Closes: https://github.com/rook/rook/issues/10181
Signed-off-by: Sébastien Han <seb@redhat.com>
Check the radosgw-admin realm user list per object store instead of relying
on the ceph dashboard get-rgw-api- command.
Resolves#9099
Signed-off-by: Alexander Trost <galexrt@googlemail.com>
The CSI package needs to load clusterInfo, today this code is in the mon
package which makes the call of LoadClusterInfo impossible without
having a circular import.
Signed-off-by: Sébastien Han <seb@redhat.com>
Support is removed for k8s for various limitations such as
priority classes not working and csi driver feature
incompleteness and missing snapshots. Documentation is updated
with the new min version of k8s 1.17 and also the tests are
updated to run on the min version of 1.17.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The access/secrets keys for the user struct need to allocate only if
ceph user creation succeeds.
Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
This removes double package imports. Example:
```
"github.com/rook/rook/pkg/apis/ceph.rook.io/v1"
cephv1 "github.com/rook/rook/pkg/apis/ceph.rook.io/v1"
```
Only one is now being used as shown in go-staticcheck ST1019
Signed-off-by: Alexander Trost <galexrt@googlemail.com>
Blaine Gardner
Travis Nielsen
xakdwch
subhamkrai
Niels de Vos
Sébastien Han
Madhu Rajanna
Satoru Takeuchi
Cheng XU
Prasanna Kumar Kalever
FengJiankui
Alexander Trost
Jiffin Tony Thottan