When OSDs flap, ceph stops the OSD daemon if its marked down greater than
5 times in 600 seconds. But OSD pod restarts and marks the OSD `up` again.
This causes the PGs mapped to these OSDs to peer. While the PGs are peering,
IO to these PGs are blocked.
So we need to ensure that if ceph is marking OSD `down` due to flapping, OSD pod
should not restart to mark the OSDs `up` again.
This PR adds a sleep to the OSD pod if the container returned with a 0 exit code
Default behavior is to sleep for 6 hrs. But user can configure it from the
ceph cluster spec.
Signed-off-by: sp98 <sapillai@redhat.com>
While two mgrs is considered sufficient, more mgr daemons are
now allowed in case the admin wants even more fault tolerance
to the active mgr going down, in case multiple mgrs go down.
Up to five mgr pods will be allowed. All mgrs will be in
standby mode except one active mgr. All mgr pods have a sidecar
that will update their respective pod specs with the active
or passive label.
Signed-off-by: travisn <tnielsen@redhat.com>
Previously ceph-exporter would only bind to IPv4 interfaces. Now if the
CephCluster is configured with `dualStack: true` and/or `ipFamily: IPv6`
an additional flag (`--addrs ::`) will be added to the ceph-exporter
container to make it listen both IPv6 and IPv4 interfaces.
Signed-off-by: Matthew Penner <me@matthewp.io>
Change how Rook detects network CIDRs for Multus networks. The IPAM
configuration is only defined as an arbitrary string JSON blob with a
"type" field and nothing more. Rook's detection of CIDRs for whereabouts
had already grown out of date since the initial implementation.
Additionally, Rook did not support DHCP IPAM, which is a reasonable
choice for users. And more, Rook did not support CNI plugin chaining,
which further complicates NADs. Based on the CNI spec, network chaning
can result in any changes to network CIDRs from the first-given plugin.
All these problems make it more and more difficult for Rook to support
Multus by inspecting the NAD itself to predict network CIDRs. Instead,
it is better for Rook to treat the CNI process as a black box. To
preserve legacy functionality of auto-detecting networks and to make
that as robust as possible, change to a canary-style architecture like
that used for Ceph mons, from which Rook will detect the network CIDRs
if possible.
Also allow users to specify overrides for CIDR ranges. This allows Rook
to still support esoteric and unexpected NAD or network configurations
where a CIDR range is not detectable or where the range detected would
be incomplete. Because it may be impossible for Rook to understand the
network CIDRs wholistically while residing only on a portion of the
network, this feature should have been present from Multus's inception.
Improving CIDR auto-detection and allowing users to specify overrides
for auto-detected CIDRs rounds out Rook's Multus support for CephCluster
(core/RADOS) installations. No further architectural changes should be
needed for CephClusters as regards application of public/cluster network
CIDRs for Multus networks.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
ceph-exporter's ServiceMonitor and Service CRD contain different port
name, which results in no metrics being collected by prometheus.
this commit makes GetCephMonitor configurable, which we use to set
consistent port names in ServiceMonitor and Service for ceph-exporter.
Signed-off-by: Divyansh Kamboj <dkamboj@redhat.com>
We need to restart all the ceph daemons whenever
cephCluster network settings are modified like
requiremsgr2, encryption and compression. This
required for Ceph to consider the new settings
it require new ceph daemons all over.
Signed-off-by: subhamkrai <srai@redhat.com>
we need to return a valid CIDR IP
to get it fenced properly, the previously
returned one was like `10.63.0.5:0/32` which
is not a valid CIDR, removing `:0` to look like
proper IP CIDR.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
If osd store is updated in the ceph cluster, then
delete OSDs one by one, cleanup disks and provision a new OSD on
the same disk
Signed-off-by: sp98 <sapillai@redhat.com>
It's better to move most of discover daemon setting
from env to configmap rook-ceph-operator-config.
Although, we are moving to configmap, we keep reading settings
from env but the priority will be configmap settings.
Signed-off-by: subhamkrai <srai@redhat.com>
golangci linter was throughing error `k8s.io/utils/pointer`
package is deprecated. So, I have removed that.
Signed-off-by: subhamkrai <srai@redhat.com>
Add nfs pods label to the list so that Rook
can get all the pods on the failed node and
force delete the pods stuck in terminating state.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
Add holder pods label to the list so that Rook
can get all the pods on the failed node and
force delete the pods stuck in terminating state.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
* fixes mon zonal deployment of 2 per zone to 1 if zones field is
specified.
* adds zones field doc to CRDs Documentation.
* add code for picking up failureDomainLabel if specified for zones
field in mon deployment
Signed-off-by: Deepika Upadhyay <deepika@koor.tech>
The function ParseMonEndpoints existed twice, once in the mon package
and once in the controller package. It has now been deleted from the
mon package. Usages in the mon package now reference the function
in the controller package.
Signed-off-by: Henry Davies <henrydavies@hotmail.co.uk>
Currently if we use the msgr2 port,
cs-cluster-config-json looks like
"monitors":["192.59.109.11:3300","192.59.109.13:6789","192.59.109.12:6789"]
The first monitor is passed by python script and rest are
Discovered by rook operator, so if the first mon is using the v2 port
lets update the other mons also by setting up the RequireMsgr2 flag
Signed-off-by: parth-gr <paarora@redhat.com>
when node fencing is triggered and a negative case where
one deployment pod consuming pv not provisioned by csi
and another deployment pod with rbd rwo are on the same
node and node fencing is triggered.
In this case rook operator panics since it is trigger to get
field `pv.Spec.CSI.Driver` which will be nil.
Also, skip static pv as there is no `imageName` in the spec
rbd status command will fail.
Signed-off-by: subhamkrai <srai@redhat.com>
During faster node loss recovery,
Rook needs to check and blocklist only rbd pvs
that do not support multinode access modes such as
ReadOnlyMany(ROX) and ReadWriteMany(RWX).
This commit adds check for the same.
Signed-off-by: Rakshith R <rar@redhat.com>
in the existing node watcher, we'll check for node update
event and see if there are `out-of-service` taints are applied
and `ROOK_WATCH_FOR_NODE_FAILURE` is enabled in rook-ceph-operator-configmap,
if then we'll create the networkFence cr and delete the cr if nodes come back.
And, added the unit test too.
Signed-off-by: subhamkrai <srai@redhat.com>
If the ceph version does not suppport the exporter, the operator
will attempt to delete the service monitor related to the exporter
to ensure it does not exist for a node. If the expected rbac does
not exist, this will cause unnecessary errors since there would anyway
be no service monitor to delete. So we ignore the error of deleting
the service monitor for the exporter.
The context is also passed to the exporter so a new kubeconfig does
not need to be initiated and cause unnecessary logging about
invalid options for the config.
Signed-off-by: travisn <tnielsen@redhat.com>
This PR makes Rook operator to add a label on PVC that contains an image version of Ceph when creating an OSD.
Signed-off-by: YZ775 <yuzuki-mimura@cybozu.co.jp>
Signed-off-by: avanthakkar <avanjohn@gmail.com>
Volume "ceph-daemons-sock-dir" is coming empty in case if dataDirHostPath,
which is the case for osd onPVC. Fix the volume creation by using the
ceph cluster spec dataDirHostPath, which allows to run socket commands
on osd containers.
Cleanup exporter daemon even if ceph version is not supported along with other resources like
metrics service and service monitor.
Signed-off-by: avanthakkar <avanjohn@gmail.com>
When deploying a cluster with Helm Chart, if the namespace is not the default value `rook-ceph`, and if the monitoring feature is enabled, then the generated ServiceMonitor's `rook_cluster` selector now follows the namespace, not the hard-coded value `rook-ceph`.
Signed-off-by: Ho Kim <ho.kim@ulagbulag.io>
The rbd mirror reconcile was not re-queuing the reconcile
if the cephcluster was not initialized. All other controllers
waiting for the initialization are requeuing the event,
just not the rbd mirror controller.
Signed-off-by: travisn <tnielsen@redhat.com>
sp98
travisn
Matthew Penner
Blaine Gardner
Divyansh Kamboj
subhamkrai
Madhu Rajanna
Deepika Upadhyay
Henry Davies
parth-gr
Sheetal Pamecha
Javier
Rakshith R
YZ775
avanthakkar
Ho Kim
Satoru Takeuchi