When OSDs flap, ceph stops the OSD daemon if its marked down greater than
5 times in 600 seconds. But OSD pod restarts and marks the OSD `up` again.
This causes the PGs mapped to these OSDs to peer. While the PGs are peering,
IO to these PGs are blocked.
So we need to ensure that if ceph is marking OSD `down` due to flapping, OSD pod
should not restart to mark the OSDs `up` again.
This PR adds a sleep to the OSD pod if the container returned with a 0 exit code
Default behavior is to sleep for 6 hrs. But user can configure it from the
ceph cluster spec.
Signed-off-by: sp98 <sapillai@redhat.com>
While two mgrs is considered sufficient, more mgr daemons are
now allowed in case the admin wants even more fault tolerance
to the active mgr going down, in case multiple mgrs go down.
Up to five mgr pods will be allowed. All mgrs will be in
standby mode except one active mgr. All mgr pods have a sidecar
that will update their respective pod specs with the active
or passive label.
Signed-off-by: travisn <tnielsen@redhat.com>
Change how Rook detects network CIDRs for Multus networks. The IPAM
configuration is only defined as an arbitrary string JSON blob with a
"type" field and nothing more. Rook's detection of CIDRs for whereabouts
had already grown out of date since the initial implementation.
Additionally, Rook did not support DHCP IPAM, which is a reasonable
choice for users. And more, Rook did not support CNI plugin chaining,
which further complicates NADs. Based on the CNI spec, network chaning
can result in any changes to network CIDRs from the first-given plugin.
All these problems make it more and more difficult for Rook to support
Multus by inspecting the NAD itself to predict network CIDRs. Instead,
it is better for Rook to treat the CNI process as a black box. To
preserve legacy functionality of auto-detecting networks and to make
that as robust as possible, change to a canary-style architecture like
that used for Ceph mons, from which Rook will detect the network CIDRs
if possible.
Also allow users to specify overrides for CIDR ranges. This allows Rook
to still support esoteric and unexpected NAD or network configurations
where a CIDR range is not detectable or where the range detected would
be incomplete. Because it may be impossible for Rook to understand the
network CIDRs wholistically while residing only on a portion of the
network, this feature should have been present from Multus's inception.
Improving CIDR auto-detection and allowing users to specify overrides
for auto-detected CIDRs rounds out Rook's Multus support for CephCluster
(core/RADOS) installations. No further architectural changes should be
needed for CephClusters as regards application of public/cluster network
CIDRs for Multus networks.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
The object user was previously required to be created in the
same namespace as the object store and the cluster. Now,
the object user can be reconciled even in a different namespace
from the cluster and object store. The namespace would be specified
in the object user CR.
Signed-off-by: travisn <tnielsen@redhat.com>
Signed-off-by: Michael Adam <obnox@samba.org>
this change introduces a new 'make target 'docs'that generates part of the documentation
It particular, it runs the `helm-docs` target.
Additionally, `make docs` updates the multus validation job manifest
by inserting the output of `rook multus validation run --help` between
two newly introduced markers.
The result of running `make docs` is intended to be committed to git after running
Fixes: #12706
Signed-off-by: Michael Adam <obnox@samba.org>
Signed-off-by: Blaine Gardner <b.blaine.gardner@gmail.com>
Co-authored-by: Blaine Gardner <b.blaine.gardner@gmail.com>
Co-authored-by: Michael Adam <obnox@samba.org>
Update build/deploy/generate-deploy-examples.sh
Signed-off-by: Michael Adam <obnox@samba.org>
The CRDs were not generated for the latest since #12384
was merged without rebasing, and other updates with a newer
dependency required the newer descriptions.
Signed-off-by: travisn <tnielsen@redhat.com>
If osd store is updated in the ceph cluster, then
delete OSDs one by one, cleanup disks and provision a new OSD on
the same disk
Signed-off-by: sp98 <sapillai@redhat.com>
It's better to move most of discover daemon setting
from env to configmap rook-ceph-operator-config.
Although, we are moving to configmap, we keep reading settings
from env but the priority will be configmap settings.
Signed-off-by: subhamkrai <srai@redhat.com>
added a sample job manifest named `multus-validation` that
validates the multus configuration in the cluster.
Signed-off-by: Nikhil-Ladha <nikhilladha1999@gmail.com>
update vault and other indirect imports. Also,
updating k8s package to `v0.28.0-rc.0` since with
v0.27.4 it was through build error.
Signed-off-by: subhamkrai <srai@redhat.com>
Adding maxUnavailable parameter to the cephfs csi daemonset. The
default value of maxUnavailable = 1 can make the upgrade process
slower in environments with large number of nodes.
Fixes: #12636
Signed-off-by: karthik-us <ksubrahm@redhat.com>
add a new toolbox yaml manifest which will use the
rook image instead of ceph image
for running s5 cmd container needs to run with rook image
closes: https://github.com/rook/rook/issues/12227
Signed-off-by: parth-gr <paarora@redhat.com>
Update the ceph cosi driver image to v0.1.1, so that it can
work on both amd and arm platforms.
Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
We need to create the namespace with the import script,
As per the documentation
The import script creates the secrets and
cm which is later used by the operator for cluster creation.
Signed-off-by: parth-gr <paarora@redhat.com>
Adding CephCOSIDriver CRD and controller. The controller will bring up
the ceph cosi driver when first object store is created in the rook
operator namespace. Then admin can defined COSI CRDs like BucketClass
and BucketAccessClass for different object stores deployed via Rook.
Using the BucketClass and BucketAccessClass, user can define
BucketAccess for backend bucket in the RGW. The CephCOSIDriver CRD
defines configuration options for ceph cosi driver. In the first version
its usability is minimal. Even if it is not defined Rook will bring up
the ceph cosi driver with default values.
Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
When the ceph.conf is not in a well-known location, it is typical
for a custom ceph.keyring to be required as well. Currently, the
create-external-cluster-resources.py only tries to load the keyring
from well-known paths, and fails.
This commit adds a new optional argument `--keyring` to help in this
situation. When specified, create-external-cluster-resources.py will
attempt to use this as the keyring to authenticate with Ceph.
Signed-off-by: Angelos Kolaitis <neoaggelos@gmail.com>
Some users want to deploy two CephObjectStores for a single Zone. The first
configures RGWs to process the synchronization of the data, while the second
CephObjectStore configures the client RGWs.
Currently, this can be implemented by setting the RGW option
'rgw_run_sync_thread' in the 'rook-config-override' ConfigMap, though it is not
really user friendly.
Ref: https://docs.ceph.com/en/latest/radosgw/config-ref/#confval-rgw_run_sync_thread
This commit adds a new option in the CephObjectStore CRD as defined in issue
https://github.com/rook/rook/issues/12272. The new option
'disableMultisiteSyncTraffic' determine whether the operator should disable the
multisite sync threads for the RGWs.
If the option is set to 'false', or if the option is not specified, the operator
does nothing. This ensures that the multisite sync threads will not be enabled
for users that disabled explicitely the multisite sync threads either manually
or with the 'rook-config-override' ConfigMap.
This commit also recommends to use two objectstore when scaling Ceph Objectstore
Multisite replication, with one objectstore configured with disabled replication
traffic.
Signed-off-by: Lucas Henry <polyedre@disroot.org>
set the csi subvolume to pin type distributed and pin setting 1
So to statically balance the PVs across all MDS ranks
Signed-off-by: parth-gr <paarora@redhat.com>
in the existing node watcher, we'll check for node update
event and see if there are `out-of-service` taints are applied
and `ROOK_WATCH_FOR_NODE_FAILURE` is enabled in rook-ceph-operator-configmap,
if then we'll create the networkFence cr and delete the cr if nodes come back.
And, added the unit test too.
Signed-off-by: subhamkrai <srai@redhat.com>
adding drop `ALL` capabilities in rook operator container
as this is not required and will remove warning in ocp cluster.
Signed-off-by: subhamkrai <srai@redhat.com>
Travis Nielsen
sp98
Micah Nagel
Blaine Gardner
parth-gr
Michael Adam
subhamkrai
Nikhil-Ladha
karthik-us
Deepika Upadhyay
Jiffin Tony Thottan
Angelos Kolaitis
Lucas Henry
Javier