This PR fails over mon when the mon path is changed from hostPath to PVC or vice versa
Signed-off-by: sp98 <sapillai@redhat.com>
(cherry picked from commit 4b6fd89365)
Fixes: #13167
Previously, the mgr did not honor the flag
ContinueUpgradeAfterChecksEvenIfNotHealthy
from the cluster spec. Only osd, mds, and rgw did.
To render the update behavior correct and complete across the daemons, this
change implements the honoring of the flag for the mgr.
Signed-off-by: Michael Adam <obnox@samba.org>
(cherry picked from commit 93bf6540b2)
This patch adds `pgHealthyRegex` field to DisruptionManagementSpec.
`pgHealthyRegex` is a regular expression that is used to determine which
PG states should be considered healthy. The default value of
`pgHealthyRegex` is:
^(active\+clean|active\+clean\+scrubbing|active\+clean\+scrubbing\+deep)$
which is effectively the same as before.
Signed-off-by: Ryotaro Banno <ryotaro.banno@gmail.com>
(cherry picked from commit 0ad88e9d68)
ceph dashboard uses radosgw-admin for certain tasks that
aren't accessible via the rgw REST API. Due to the absence of
a valid ceph.conf file at /etc/ceph/ceph.conf within the mgr pod,
radosgw-admin fails to operate, resulting in 500 errors across
various 'Object Gateway' views on the dashboard. This change
adds CEPH_ARGS environment variable to the mgr pod enabling
its propagation and utilization by the dashboard/radosgw-admin
for executing rgw commands.
closes: https://github.com/rook/rook/issues/13255
Signed-off-by: Redouane Kachach <rkachach@redhat.com>
(cherry picked from commit afc485fa03)
Restarting the exporter using RollingRelease causes a race condition,
that results in exporter crashing and the ceph health to show a warning.
Signed-off-by: Divyansh Kamboj <dkamboj@redhat.com>
(cherry picked from commit f9d6cd7f3a)
Fix an issue in the network address detection job where placement was
only retreived from osd and not merged with all.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
(cherry picked from commit 0a538bfc37)
Use K8s LivenessProbe mechanism to check OK-status of nfs-ganesha
container. A user may define his own lineness-probe, or a default one
which expects NFS TCP-port 2049 to be active; that is, willing to accept
new connections: for Ceph>=18.2.1 issue 'rpcinfo' call on local pod;
otherwise use standard K8s TCP-socket liveness probe mechanism.
Define permissive values to liveness-probe to ensure that the NFS
service is defined in failed-state only when it has non-recoverable
error.
The current default definition of LivenessProbe is expected to guard the
nfs pod from at least the following two cases:
- Deadlocks: where an nfs-ganesha server is running, but unable serve
new connections due to internal bad-state.
- Resource exhaustion on the host node (e.g. OOM) which prevents the
server from accepting new connections and reply to NULL RPC request.
In both cases we expect K8s to reschedule the nfs pod, most likely on
different host node.
Refs rook issue #12719
Signed-off-by: Shachar Sharon <ssharon@redhat.com>
(cherry picked from commit e05184d0a7)
Use the Rook image (defined by the operator pod) to detect the Multus
network address ranges. It is reasonable for users to want to have a
minimal Ceph image that does not have the `ip` utility installed, which
is used for detecting the address ranges of multus interfaces. Instead,
use the Rook image, which Rook can ensure has the `ip` tool if Ceph ever
removes it from their image.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
(cherry picked from commit db1ca8c93e)
when creating networkFence, rbd command was loading
admin config and hence running rbd command use client.admin
in case of external cluster also. With this commit instead
of client.admin user it will use what is being passed to
config.
Signed-off-by: subhamkrai <srai@redhat.com>
(cherry picked from commit ef4dd76df1)
This reverts commit fab23d3407.
The mgr requires rw access for the cron job that collects
the crashes.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
(cherry picked from commit a421143ddd)
As seen in https://github.com/rook/rook/issues/1988, it's a common
mistake to configure rook nodes with names that don't match Kubernetes's
node label. This PR prints more detailed message to help debugging
problem.
Signed-off-by: Bin Wang <bin.wang@mail.binwang.me>
Add break according to review comment
Co-authored-by: Travis Nielsen <tnielsen@redhat.com>
Update log message according to review comment
Co-authored-by: Travis Nielsen <tnielsen@redhat.com>
(cherry picked from commit 4a705b46b0)
Fixed the indentation for the mountinfo
or else an emptyDir is getting created
for it.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
(cherry picked from commit 9f2b7d572d)
If `FlappingRestartIntervalHours` for OSD is > 0 in the cephCluster CR,
then start OSD inside a bash script.
Signed-off-by: sp98 <sapillai@redhat.com>
(cherry picked from commit ffd69291f1)
The /tmp dir is not always available in some environments.
The only directories the operator should rely on
are the directories that are in its pod spec.
For assimilate-conf now we use the /var/lib/rook
directory to generate files since we know it is
a valid emptydir mounted into the operator pod.
Signed-off-by: travisn <tnielsen@redhat.com>
(cherry picked from commit 5359c4f4f5)
If mountinfo path is available cephcsi will
try to recover the fuse client. adding the
support for the same in Rook.
fixes: #13009
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
(cherry picked from commit 54ce2a96da)
Several messages were showing up in the operator log
more often than necessary, so let's change them to
debug level.
Signed-off-by: travisn <tnielsen@redhat.com>
(cherry picked from commit 29fb879675)
Create each cosi user for each object store and secret which holds
credentials.
Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
(cherry picked from commit a941b3c33f)
The activeStandby property of the filesystem CR was not
taking effect when changed from true to false. The standby
was only being enabled when true, but never applied when
changed to false.
Signed-off-by: travisn <tnielsen@redhat.com>
(cherry picked from commit f7094d1d13)
The ceph documentation indicates to use the crash profile
instead of specifying rw for the mgr access.
Signed-off-by: travisn <tnielsen@redhat.com>
(cherry picked from commit fab23d3407)
Using the new configuration parameters prometheusEndpoint and
prometheusEndpointSSLVerify users can configure dashboard to
point to their prometheus instance.
closes: #12876
Signed-off-by: Redouane Kachach <rkachach@redhat.com>
(cherry picked from commit c763935436)
When OSDs flap, ceph stops the OSD daemon if its marked down greater than
5 times in 600 seconds. But OSD pod restarts and marks the OSD `up` again.
This causes the PGs mapped to these OSDs to peer. While the PGs are peering,
IO to these PGs are blocked.
So we need to ensure that if ceph is marking OSD `down` due to flapping, OSD pod
should not restart to mark the OSDs `up` again.
This PR adds a sleep to the OSD pod if the container returned with a 0 exit code
Default behavior is to sleep for 6 hrs. But user can configure it from the
ceph cluster spec.
Signed-off-by: sp98 <sapillai@redhat.com>
(cherry picked from commit 4eb9f62205)
While two mgrs is considered sufficient, more mgr daemons are
now allowed in case the admin wants even more fault tolerance
to the active mgr going down, in case multiple mgrs go down.
Up to five mgr pods will be allowed. All mgrs will be in
standby mode except one active mgr. All mgr pods have a sidecar
that will update their respective pod specs with the active
or passive label.
Signed-off-by: travisn <tnielsen@redhat.com>
(cherry picked from commit 66547d27be)
Previously ceph-exporter would only bind to IPv4 interfaces. Now if the
CephCluster is configured with `dualStack: true` and/or `ipFamily: IPv6`
an additional flag (`--addrs ::`) will be added to the ceph-exporter
container to make it listen both IPv6 and IPv4 interfaces.
Signed-off-by: Matthew Penner <me@matthewp.io>
(cherry picked from commit d09a43df37)
For an OBC's name we cannot simply use the OBC's namespace and name,
because they can be reused, while the preceding bucket might be
retained by reclaimPolicy. (Commit 2733375ca4 also didn't solve
this issue, as it is an intra-cluster issue.) Therefore we instead
use the OBC's UID which should be unique within the cluster and
across clusters.
Signed-off-by: Sebastian Hasler <sebastian.hasler@stuvus.uni-stuttgart.de>
(cherry picked from commit 041fc1a71c)
Change how Rook detects network CIDRs for Multus networks. The IPAM
configuration is only defined as an arbitrary string JSON blob with a
"type" field and nothing more. Rook's detection of CIDRs for whereabouts
had already grown out of date since the initial implementation.
Additionally, Rook did not support DHCP IPAM, which is a reasonable
choice for users. And more, Rook did not support CNI plugin chaining,
which further complicates NADs. Based on the CNI spec, network chaning
can result in any changes to network CIDRs from the first-given plugin.
All these problems make it more and more difficult for Rook to support
Multus by inspecting the NAD itself to predict network CIDRs. Instead,
it is better for Rook to treat the CNI process as a black box. To
preserve legacy functionality of auto-detecting networks and to make
that as robust as possible, change to a canary-style architecture like
that used for Ceph mons, from which Rook will detect the network CIDRs
if possible.
Also allow users to specify overrides for CIDR ranges. This allows Rook
to still support esoteric and unexpected NAD or network configurations
where a CIDR range is not detectable or where the range detected would
be incomplete. Because it may be impossible for Rook to understand the
network CIDRs wholistically while residing only on a portion of the
network, this feature should have been present from Multus's inception.
Improving CIDR auto-detection and allowing users to specify overrides
for auto-detected CIDRs rounds out Rook's Multus support for CephCluster
(core/RADOS) installations. No further architectural changes should be
needed for CephClusters as regards application of public/cluster network
CIDRs for Multus networks.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
(cherry picked from commit 3c43268d0a)
Correctly set discoveryInterval default value by passing defaultDiscoverInterval to k8sutil.GetValue
Allow to set ROOK_DISCOVER_DEVICES_INTERVAL env var through discoveryDaemonInterval helm value
Signed-off-by: Louis Bailleul <louis.bailleul@pgs.com>
(cherry picked from commit e56122cbce)
It's better to move most of discover daemon setting
from env to configmap rook-ceph-operator-config.
Although, we are moving to configmap, we keep reading settings
from env but the priority will be configmap settings.
Signed-off-by: subhamkrai <srai@redhat.com>
(cherry picked from commit fb39580067)
This follow up the #12507
- fixes replacing of encrypted OSDs.
- Updates OSD status at the end of reconcile
Signed-off-by: sp98 <sapillai@redhat.com>
(cherry picked from commit 11b8d10a5a)
ceph-exporter's ServiceMonitor and Service CRD contain different port
name, which results in no metrics being collected by prometheus.
this commit makes GetCephMonitor configurable, which we use to set
consistent port names in ServiceMonitor and Service for ceph-exporter.
Signed-off-by: Divyansh Kamboj <dkamboj@redhat.com>
(cherry picked from commit 708dd29a97)
We need to restart all the ceph daemons whenever
cephCluster network settings are modified like
requiremsgr2, encryption and compression. This
required for Ceph to consider the new settings
it require new ceph daemons all over.
Signed-off-by: subhamkrai <srai@redhat.com>
(cherry picked from commit 29d2b6a071)
when deploying csi, we'll check if csi-addons crds
are deployed we'll enable csi-addons sidecar by default.
Signed-off-by: subhamkrai <srai@redhat.com>
(cherry picked from commit a755ec156a)
If bucket exists for Provision(), then check whether user in the OBC and
owner of bucket are same.
Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
(cherry picked from commit b39e813290)
we need to return a valid CIDR IP
to get it fenced properly, the previously
returned one was like `10.63.0.5:0/32` which
is not a valid CIDR, removing `:0` to look like
proper IP CIDR.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
(cherry picked from commit eb304a5073)
sp98
Michael Adam
Ryotaro Banno
Redouane Kachach
Divyansh Kamboj
parth-gr
Blaine Gardner
Shachar Sharon
subhamkrai
travisn
Bin Wang
Madhu Rajanna
Jiffin Tony Thottan
guoguangwu
Matthew Penner
Sebastian Hasler
Louis Bailleul