When the clusters reach full, nearfull, or backfill full thresholds
ceph will raise health warnings and stop allowing IO or backfill
depending on the threshold. These settings require special ceph
commands instead of being generic ceph config. Allow these settings
to be set from the CephCluster CR in the spec.storage
section.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
(cherry picked from commit e479b051bc)
cephfs subvolumegroup supports creating svg
with quota and the datapool, This PR adds the
support for the same.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
(cherry picked from commit f57a8b6bbe)
The upgrade test should always upgrade from the
previous minor release to the latest master. With 1.14
releasing soon, now we upgrade from 1.13 to master,
to confirm if there are any upgrade issues to 1.14.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The pg_autoscaler is always enabled by ceph and cannot be disabled.
In a previous PR the pg_autoscaler config was removed from the main example.
Now the remaining config for the pg_autoscaler is removed as well.
Signed-off-by: travisn <tnielsen@redhat.com>
The upgrade test should always upgrade from the previous
minor release to the latest master. With 1.13 releasing
soon, now we uprade from 1.12 to master, to confirm
if there are any upgrade issues to 1.13.
Signed-off-by: travisn <tnielsen@redhat.com>
The toolbox image tag was always being set from #12625
even when the image was not set in the test. If the image
is not set, skip replacing the image name to use the default
test version.
Signed-off-by: travisn <tnielsen@redhat.com>
The test is currently only using the ceph version that is
included with toolbox.yaml, which may be different
from the version of the ceph cluster being tested.
Now the version will be replaced to match the desired
ceph test version.
Signed-off-by: travisn <tnielsen@redhat.com>
Adding CephCOSIDriver CRD and controller. The controller will bring up
the ceph cosi driver when first object store is created in the rook
operator namespace. Then admin can defined COSI CRDs like BucketClass
and BucketAccessClass for different object stores deployed via Rook.
Using the BucketClass and BucketAccessClass, user can define
BucketAccess for backend bucket in the RGW. The CephCOSIDriver CRD
defines configuration options for ceph cosi driver. In the first version
its usability is minimal. Even if it is not defined Rook will bring up
the ceph cosi driver with default values.
Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
With the upcoming v1.12 release, Rook upgrade tests
should upgrade from v1.11.x to master instead of
from v1.10.x to master.
Signed-off-by: Sheetal Pamecha <spamecha@redhat.com>
The journal size was only applicable to the filestore OSD
format which has not been supported by rook since v1.2.
Remove the remaining obsolete setting from the examples and
code.
Signed-off-by: travisn <tnielsen@redhat.com>
Enabling msgr v2 and disabling msgr v1 currently requires enabling
either encryption on the wire or compression on the wire.
As more clients are running on the latest kernel, allow
the clients to run on v2 even when encryption and compression
are not enabled. Clusters that are fully running on v2
will more easily be able to change configuration between
enabling or disabling msgr v2 features.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
With the v1.11 release approaching, the upgrade tests in
master are not updated to upgrade starting from v1.10
instead of from v1.9.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The machine disruption budgets for handling openshift
machines and machinesets are now removed since they
have been unused and unmaintained since implemented.
This feature is expected to be handled with the more
common Pod Disruption Budgets. A workaround is for the
cluster admin to set up their machine sets so they
match the zone topology. See the original design
doc from the feature here:
https://github.com/rook/rook/blob/master/design/ceph/ceph-openshift-fencing-mitigation.md
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
enabling logCollector by default will enable the coredump
generation in case a process terminates with a segmentation fault.
Signed-off-by: gauravsitlani <gauravsitlani@riseup.net>
Remove the health checker for CephObjectStore. The liveness and
readiness probes go through the same code paths in RGW as creating
buckets without as much affect on the storage backend.
Full discussion: https://github.com/rook/rook/issues/11031
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
This commit anabled nfs csi ci and
add fixes/improvements to it like the
following:
- verify deletion of cephnfs and .nfs pool before proceeding
- verify pv deletion
- do not enable rook module
- reduce activeCount to 1 to save resources
- run cephnfs ci before cephfs ci since it cephfs
ci is more resource intensive.
Signed-off-by: Rakshith R <rar@redhat.com>
Rook will support the most recent seven K8s releases,
so we update the min version to 1.19 for Rook v1.10.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Long ago the gateway type for s3 was removed from the object
store CRD, and k8s 1.25 is now complaining about the obsolete
setting still included in the test.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
With the pending release of 1.10, the upgrade tests
in master should be testing the upgrade from 1.9
to the master branch.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
"nolock" mountOptions has been added to prevent the following error in ci:
Output: mount.nfs: rpc.statd is not running but is required for remote locking.
mount.nfs: Either use '-o nolock' to keep locks local, or start statd.
Signed-off-by: Rakshith R <rar@redhat.com>
Block deletion of CephFilesystems when there are any raw Ceph
subvolumegroups present that have subvolumes in them. Empty
subvolumegroups will not block deletion.
One important subvolume group is "csi" which is the default location
where CSI subvolumes are kept. If this group is empty, it means that
there are no PVCs created based on the CephFilesystem in question. This
also holds true if there are external consumers of the filesystem in
external cluster mode.
Similarly, if there are any subvolumegroups (for example "_nogroup",
which includes subvolumes in the filesystem root) that contain
manually-created subvolumes, Rook will also see this and block deletion.
This comes into play currently with manually-created NFS exports.
A work-in progress aims to create a Ceph-CSI NFS export provisioner
which will likely create subvolumes in the "csi" group as well. This
implementation will catch this case also.
Rook still checks for CephFilesystemSubVolumeGroups explicitly in
addition to the check added here. This is to ensure that even empty
groups will block deletion if they are created via this CR type.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
updating node-driver-registrar to v2.5.1 release
, nfsplugin to 4.0.0 and csi-snapshotter to v6.0.1
Note:- with snapshotter v6.0.1 v1beta1 snapshot CRD
is not supported. If you want betav1 snapshot support
please switch to v5.x.x snapshotter.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
By default, we should set the priority class to one of the built-in
priority class names to ensure that pods critical to the storage
will be able to remain running when resources are low. Otherwise,
critical rook pods could be evicted and affect many other pods
that rely on the storage to continue functioning. The options have
been available in the CRs, but until now we have just not set the
defaults in the examples. Critical rook components are now set to
the priority class system-node-critical if they are generally pinned
to a node, and system-cluster-critical if they are critical to the storage.
Some pods such as the operator and crash collector do not have a
default priority class set in the examples since they don't affect
the data path.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The aws go lang sdk needs value for region, it is set differently in
various part of current code. With PR the value is always `us-east-1` so
that it will work RGW server without any issues.
This reverts commit 280c29f330.
Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
The upgrade integration test had been testing from v1.7
to master. Now with the v1.8 release coming up we upgrade
from v1.8 to master.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
msgr2 allows for encryption and/or compression across the wire.
Settings for enabling the encryption and compression are now
available in the cluster CR to that ceph will be automatically
configured with these settings when desired.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The encryption on the wire requires either the 5.11 kernel
or the nbd driver. Until the kernel update is available in minikube
we use the rbd-nbd.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
To ensure the mgr is not the single point of failure, the mgr
daemon count is now set to 2 by default in the cluster examples.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The built-in pools device_health_metrics and .nfs created by ceph
need to be configured for replicas, failure domain, etc.
To support this, we allow the pool to be created as a CR.
Since K8s does not support underscores in the resource names
the operator must translate this special pool name into
the name expected by ceph.
This also sets the basis for allowing filesystem data
pools to specify the desired pool name instead of requiring
a generated name.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The upgrade integration test was from rook v1.6 to the latest master.
This was necessary until we are ready for the v1.8 release, from which
time we want to focus the upgrade testing from v1.7 to the latest
master.
The duplication in the test CRs and other resources is now reduced
by the upgrade calling a thin wrapper to forward a call to the
master version of the resource. When a new feature is added that
needs to be differentiated from the previous version, the method
then can be implemented instead of wrapping the master implementation.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Using a default value for CompressionMode to none effectively overrides
any values for Parameters. It is deprecated but still takes precedence.
Which means that in its previous form, Parameters was always ignored
since CompressionMode was always set to none when empty.
Signed-off-by: Sébastien Han <seb@redhat.com>
In Rook v1.8 the min version of K8s supported is updated to 1.16.
Users running on older versions of K8s are recommended to update
to 1.16 or newer before updating to Rook v1.8.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The flex driver has been fully deprecated and thus removed from Rook.
Before upgrading to v1.8, users will need to convert existing flex volumes
from flex to csi volumes.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The integration tests must always be run against the local
build of rook, and an image should never be pulled from dockerhub.
To prevent pulling a release or master tag, the local build
will use a tag specific to the build and not ever published
elsewhere.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
(cherry picked from commit a8a40428b0)
The release version needs to be the same in the example/test manifests
as it is in the local build image. The github actions are different in
this regard than the Jenkins builds were. The Jenkins builds always
locally used the master tag instead of a release-specific tag.
Now that the github actions use the release-specific tag, the test
framework no longer should be using the master tag in release branches.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Travis Nielsen
Madhu Rajanna
subhamkrai
Jiffin Tony Thottan
Sheetal Pamecha
Blaine Gardner
gauravsitlani
Rakshith R
Yuval Manor
Yuval Lifshitz
Sébastien Han
parth-gr