Normally the device class of an OSD is determined at provisioning
and is not updated thereafter. In some scenarios the admin may
want to force update the device class to a new value. The device
classes can be updated by first setting allowDeviceClassUpdate
in the storage spec of the cephcluster, then updating the
device class specified on the deviceSets or other OSDs.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
initially, we have the limit of 40 char set on
storageclassdeviceset.Name but seems like that is not enough and
upgrade clusters can face issue with this. So, increasing the limit to
50.
Signed-off-by: subhamkrai <srai@redhat.com>
this will be disabled by default but in scenarios where the user want to
update failureDomain, DeviceClass etc, this option can be enabled, to be
noted this can lead to lot of data rebalancing and remapping. Use with
caution
Signed-off-by: Deepika Upadhyay <deepika.upadhyay@clyso.com>
When the clusters reach full, nearfull, or backfill full thresholds
ceph will raise health warnings and stop allowing IO or backfill
depending on the threshold. These settings require special ceph
commands instead of being generic ceph config. Allow these settings
to be set from the CephCluster CR in the spec.storage
section.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
This PR adds a `settings` option in the `Module` struct
which can help users to futher customize the `balancer`
module with a differet mode.
Signed-off-by: sp98 <sapillai@redhat.com>
LVM-based OSDs on PVCs are legacy and are not well tested.
Therefore, we add status to the cephcluster CR that at least
gives a clue that the OSDs should be replaced.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
there are some cases where storageClassDeviceSets length
are more 63 chars and osd provisioned failed due to this.
with error
```
failed to run osd provisioning job for PVC "ocs-deviceset-*-local-volume-storageclass-0-data-*": Job.batch "rook-ceph-osd-prepare-*" is invalid: [spec.template.spec.volumes[10].name: Invalid value: "ocs-deviceset-*-local-volume-storageclass-0-data-*": must be no more than 63 characters, spec.template.spec.containers[0].volumeMounts[9].name: Not found: "ocs-deviceset-*-local-volume-storageclass-0-data-*",
```
Using x-validator to limit the maxLength to the deviceClassSet.Name to
40 chars and adding code check that overall pvc name should be less than
63 chars.
Signed-off-by: subhamkrai <srai@redhat.com>
The following printcolumns were added
CephBlockPoolRadosNamespace: Phase,
BlockPoolName, Age
CephFilesystemSubVolumeGroup: Phase,
FilesystemName, PinningConfig, Age
Also improved the error messages in
SubVolumeGroupClient
Signed-off-by: NymanRobin <nyman.robin@gmail.com>
For check if cluster PGs are healthy before osd updates
Which helps gainning more confidence in the osd upgrades
Signed-off-by: mmaoyu <wupiaoyu61@gmail.com>
cephfs subvolumegroup supports creating svg
with quota and the datapool, This PR adds the
support for the same.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
make the kubectl output for crds more verbose.
Following changes was done for each crd:
CephBlockPool:
Added print columns:
- Type
- FailureDomain
- Replication
- EC-CodingChunks
- EC-DataChunks
- Age
Extended info field of status to include:
type and failureDomain
CephObjectStore:
Added print columns:
- Endpoint
- SecureEndpoint
- Age
Added Age to print columns of following crds:
CephObjectStoreUser, CephObjectZoneGroup,
CephObjectZone, CephBucketTopic, CephClient,
CephRBDMirror, CephFilesystemMirror,
CephFilesystemSubVolumeGroup
Signed-off-by: NymanRobin <nyman.robin@gmail.com>
Few day's back go 1.22 is release. Let's add that
as valid version to build on. Also, adding 1.22 as
test matric for linux build CI.
https://tip.golang.org/doc/go1.22
Signed-off-by: subhamkrai <srai@redhat.com>
Until now, an object store would create all the necessary
metadata pools and the data pool that were exclusively
for its own object store. When isolation between object
stores is necessary, this would cause many pools and
PGs to be created in the cluster, which was not
manageable.
Now one set of pools can be created to be shared
by any number of object stores. The metadata and data
between each object store is isolated by
RADOS namespaces, which by design will keep the
data safe for multi-tenancy.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Reduce CRD size by removing some descriptions for things that
(a) are long, (b) are repeated, and (c) are commong kubernetes types.
Remove the `Status` field from VolumeClaimTemplates (a.k.a. PVCs) in
types.go to remove all unnecessary status-related fields.
Use `yq` to go through generated `crds.yaml` to strip out descriptions
for well-known k8s types that are long and repeated where we can't
remove them from types.go. This is mostly placement descriptions and
volumeSource descriptions.
The overall `crds.yaml` file is reduced to less than 60% of its original
size.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
Signed-off-by: Redouane Kachach <rkachach@redhat.com>
Co-authored-by: Blaine Gardner <blaine.gardner@ibm.com>
Rook has been setting the application automatically on all
pools to rbd for CephBlockPools, rook-ceph-rgw for
CephObjectStores, mgr on the built-in .mgr pool,
and nfs on the built-in .nfs pool.
The legacy pool device_health_metrics is long gone
from Pacific which is no longer supported, so we can
remove special handling for that pool in the upgrade
guide and in the code.
The application setting is now available on the pool spec
although it is not expected to commonly need to override
the default applications set by Rook.
The application for CephFilesystem pools is now being
set to cephfs, where it was previously blank.
Signed-off-by: travisn <tnielsen@redhat.com>
Fixes: #13692
Since the introduction of the "host" network provider, the legacy
"hostNetwork" setting is intended to be used only in combination with
the default network provider (""), but the code did not enforce this.
This change adds the required validation checks to throw errors
in invalid constellations.
These checks are added both in the operator's input validation code
and as kubernetes x-validation admission policies in the Cepcluster CRD.
Signed-off-by: Michael Adam <obnox@samba.org>
The mds max daemons had been set arbitrarily to 10.
Allow more mds daemons for clusters that have a
scenario for large numbers of mds to manage the
filesystem metadata.
Signed-off-by: travisn <tnielsen@redhat.com>
Long CRD descriptions were pushing the size of the CRDs manifest
over 1MB, which causes issues when creating the CRDs. To shorten
the manifest, the descriptions are now limited to 100 chars.
This still gives some ability for dev tools to show a brief
description if needed. The generated CRD docs still contain
the full descriptions for reference.
Signed-off-by: travisn <tnielsen@redhat.com>
This implements the "Ceph Config via Ceph Cluster CRD" design document
as a `cephConfig:` structure on the CRD.
This also fixes the `yq` commands used to manipulate the
`cluster-test.yaml` that caused CI issues for this PR and potentially
unknowingly others.
Signed-off-by: Alexander Trost <galexrt@googlemail.com>
this commit adds validating admission policy
for cephcluster cr according to webhook rules.
Not all the webhook can be moved to validating
admission policy for example checking multus
selector validation.
Signed-off-by: subhamkrai <srai@redhat.com>
Originally we create it using this cmd
ceph fs subvolume create <vol_name> <subvol_name>
So we can have 2 variables filesystem and subvolume name,
Currently the CR doesn't allow us to make subvolume-name
as constant as needed to "csi" because of k8s limitations
Signed-off-by: parth-gr <paarora@redhat.com>
This commit adds new CSIDriverOptions section in
cephCluster CR. This section contains settings
for read affinity and kernel+fuse Mount options
These settings will be injected directly into
rook-ceph-csi-config cm to be applicable per
ceph cluster.
Signed-off-by: Rakshith R <rar@redhat.com>
This patch adds `pgHealthyRegex` field to DisruptionManagementSpec.
`pgHealthyRegex` is a regular expression that is used to determine which
PG states should be considered healthy. The default value of
`pgHealthyRegex` is:
^(active\+clean|active\+clean\+scrubbing|active\+clean\+scrubbing\+deep)$
which is effectively the same as before.
Signed-off-by: Ryotaro Banno <ryotaro.banno@gmail.com>
Allow end-user to have full control on NFS-Ganesha liveness probe:
- Disabled
- Enabled, using user-provided definition
- Enabled, using Rook's default liveness probe for Ganesha.
Signed-off-by: Shachar Sharon <ssharon@redhat.com>
Using the new configuration parameters prometheusEndpoint and
prometheusEndpointSSLVerify users can configure dashboard to
point to their prometheus instance.
closes: #12876
Signed-off-by: Redouane Kachach <rkachach@redhat.com>
When OSDs flap, ceph stops the OSD daemon if its marked down greater than
5 times in 600 seconds. But OSD pod restarts and marks the OSD `up` again.
This causes the PGs mapped to these OSDs to peer. While the PGs are peering,
IO to these PGs are blocked.
So we need to ensure that if ceph is marking OSD `down` due to flapping, OSD pod
should not restart to mark the OSDs `up` again.
This PR adds a sleep to the OSD pod if the container returned with a 0 exit code
Default behavior is to sleep for 6 hrs. But user can configure it from the
ceph cluster spec.
Signed-off-by: sp98 <sapillai@redhat.com>
While two mgrs is considered sufficient, more mgr daemons are
now allowed in case the admin wants even more fault tolerance
to the active mgr going down, in case multiple mgrs go down.
Up to five mgr pods will be allowed. All mgrs will be in
standby mode except one active mgr. All mgr pods have a sidecar
that will update their respective pod specs with the active
or passive label.
Signed-off-by: travisn <tnielsen@redhat.com>
Change how Rook detects network CIDRs for Multus networks. The IPAM
configuration is only defined as an arbitrary string JSON blob with a
"type" field and nothing more. Rook's detection of CIDRs for whereabouts
had already grown out of date since the initial implementation.
Additionally, Rook did not support DHCP IPAM, which is a reasonable
choice for users. And more, Rook did not support CNI plugin chaining,
which further complicates NADs. Based on the CNI spec, network chaning
can result in any changes to network CIDRs from the first-given plugin.
All these problems make it more and more difficult for Rook to support
Multus by inspecting the NAD itself to predict network CIDRs. Instead,
it is better for Rook to treat the CNI process as a black box. To
preserve legacy functionality of auto-detecting networks and to make
that as robust as possible, change to a canary-style architecture like
that used for Ceph mons, from which Rook will detect the network CIDRs
if possible.
Also allow users to specify overrides for CIDR ranges. This allows Rook
to still support esoteric and unexpected NAD or network configurations
where a CIDR range is not detectable or where the range detected would
be incomplete. Because it may be impossible for Rook to understand the
network CIDRs wholistically while residing only on a portion of the
network, this feature should have been present from Multus's inception.
Improving CIDR auto-detection and allowing users to specify overrides
for auto-detected CIDRs rounds out Rook's Multus support for CephCluster
(core/RADOS) installations. No further architectural changes should be
needed for CephClusters as regards application of public/cluster network
CIDRs for Multus networks.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
Travis Nielsen
subhamkrai
Deepika Upadhyay
sp98
NymanRobin
mmaoyu
Madhu Rajanna
Liang Zheng
subham rai
Jiffin Tony Thottan
Blaine Gardner
Redouane KachachandBlaine Gardner
Michael Adam
dependabot[bot]
parth-gr
parth-gr
Alexander Trost
Rakshith R
Ryotaro Banno
Shachar Sharon