The exporter pod is ephemeral and frequently being deleted
and re-created when ceph daemons are created. Therefore,
we need to skip reconciling based on the deletion
of the exporter deployment.
Signed-off-by: travisn <tnielsen@redhat.com>
(cherry picked from commit 995a64fb3a)
Loop variables cannot be reliably uses since they will
change with each iteration. Update these loop variable
uses to be safe by indexing the slice rather than
using the loop variable directly.
Also suppress the linter issues for passwords used
in tests.
Signed-off-by: travisn <tnielsen@redhat.com>
(cherry picked from commit 6f8e42422d)
Use Quay as the source for the nginx-unprivileged image used for the
Multus validation tool because Quay does not rate limit image pulls,
which are a common complaint for users of the tool.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
(cherry picked from commit cd253d0218)
This PR fails over mon when the mon path is changed from hostPath to PVC or vice versa
Signed-off-by: sp98 <sapillai@redhat.com>
(cherry picked from commit 4b6fd89365)
Fixes: #13167
Previously, the mgr did not honor the flag
ContinueUpgradeAfterChecksEvenIfNotHealthy
from the cluster spec. Only osd, mds, and rgw did.
To render the update behavior correct and complete across the daemons, this
change implements the honoring of the flag for the mgr.
Signed-off-by: Michael Adam <obnox@samba.org>
(cherry picked from commit 93bf6540b2)
This patch adds `pgHealthyRegex` field to DisruptionManagementSpec.
`pgHealthyRegex` is a regular expression that is used to determine which
PG states should be considered healthy. The default value of
`pgHealthyRegex` is:
^(active\+clean|active\+clean\+scrubbing|active\+clean\+scrubbing\+deep)$
which is effectively the same as before.
Signed-off-by: Ryotaro Banno <ryotaro.banno@gmail.com>
(cherry picked from commit 0ad88e9d68)
ceph dashboard uses radosgw-admin for certain tasks that
aren't accessible via the rgw REST API. Due to the absence of
a valid ceph.conf file at /etc/ceph/ceph.conf within the mgr pod,
radosgw-admin fails to operate, resulting in 500 errors across
various 'Object Gateway' views on the dashboard. This change
adds CEPH_ARGS environment variable to the mgr pod enabling
its propagation and utilization by the dashboard/radosgw-admin
for executing rgw commands.
closes: https://github.com/rook/rook/issues/13255
Signed-off-by: Redouane Kachach <rkachach@redhat.com>
(cherry picked from commit afc485fa03)
Restarting the exporter using RollingRelease causes a race condition,
that results in exporter crashing and the ceph health to show a warning.
Signed-off-by: Divyansh Kamboj <dkamboj@redhat.com>
(cherry picked from commit f9d6cd7f3a)
Due to multi-site Ceph object resources (realms, zonegroups, zones)
reconciliation CLI commands being proxied through CephCluster's Mgr pods
in Multus-based scenarios, return codes were being additionally wrapped
by K8S' command streams and weren't adequately handled in-code, causing
reconciliation of CephObject(Realm|ZoneGroup|Zone) resources to fail, as
an exit code of 0, instead of the expected syscall.ENOENT (2) was returned.
Fixes#12833
Signed-off-by: zer0def <zv0uzqbqnncivw0afjsx79jmd19shpps3r5f2vjc6yv@definedaszero.xyz>
(cherry picked from commit 9b69b5cb46)
Fix an issue in the network address detection job where placement was
only retreived from osd and not merged with all.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
(cherry picked from commit 0a538bfc37)
Use K8s LivenessProbe mechanism to check OK-status of nfs-ganesha
container. A user may define his own lineness-probe, or a default one
which expects NFS TCP-port 2049 to be active; that is, willing to accept
new connections: for Ceph>=18.2.1 issue 'rpcinfo' call on local pod;
otherwise use standard K8s TCP-socket liveness probe mechanism.
Define permissive values to liveness-probe to ensure that the NFS
service is defined in failed-state only when it has non-recoverable
error.
The current default definition of LivenessProbe is expected to guard the
nfs pod from at least the following two cases:
- Deadlocks: where an nfs-ganesha server is running, but unable serve
new connections due to internal bad-state.
- Resource exhaustion on the host node (e.g. OOM) which prevents the
server from accepting new connections and reply to NULL RPC request.
In both cases we expect K8s to reschedule the nfs pod, most likely on
different host node.
Refs rook issue #12719
Signed-off-by: Shachar Sharon <ssharon@redhat.com>
(cherry picked from commit e05184d0a7)
Allow end-user to have full control on NFS-Ganesha liveness probe:
- Disabled
- Enabled, using user-provided definition
- Enabled, using Rook's default liveness probe for Ganesha.
Signed-off-by: Shachar Sharon <ssharon@redhat.com>
(cherry picked from commit 7c3e91390e)
This reverts commit 093125904b.
Once we introduced encrypted OSD on partition in #12924.
However, even though `encryptedDevice` field is `true`,
an non-encrypted OSD is created. To prevent the creation
of misconfigured OSD, it's better to revert #12924 for now.
Initially, I intended to correct this problem. However, I found
it will take long time because we need to change the KMS-related
code, init containers of both OSD pod and OSD prepare pod.
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
(cherry picked from commit 8e80b21035)
Use the Rook image (defined by the operator pod) to detect the Multus
network address ranges. It is reasonable for users to want to have a
minimal Ceph image that does not have the `ip` utility installed, which
is used for detecting the address ranges of multus interfaces. Instead,
use the Rook image, which Rook can ensure has the `ip` tool if Ceph ever
removes it from their image.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
(cherry picked from commit db1ca8c93e)
Allow the flakiness threshold window to be tuned from the cli
Signed-off-by: Sheetal Pamecha <spamecha@redhat.com>
(cherry picked from commit 06c176524a)
when creating networkFence, rbd command was loading
admin config and hence running rbd command use client.admin
in case of external cluster also. With this commit instead
of client.admin user it will use what is being passed to
config.
Signed-off-by: subhamkrai <srai@redhat.com>
(cherry picked from commit ef4dd76df1)
This reverts commit fab23d3407.
The mgr requires rw access for the cron job that collects
the crashes.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
(cherry picked from commit a421143ddd)
As seen in https://github.com/rook/rook/issues/1988, it's a common
mistake to configure rook nodes with names that don't match Kubernetes's
node label. This PR prints more detailed message to help debugging
problem.
Signed-off-by: Bin Wang <bin.wang@mail.binwang.me>
Add break according to review comment
Co-authored-by: Travis Nielsen <tnielsen@redhat.com>
Update log message according to review comment
Co-authored-by: Travis Nielsen <tnielsen@redhat.com>
(cherry picked from commit 4a705b46b0)
Fixed the indentation for the mountinfo
or else an emptyDir is getting created
for it.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
(cherry picked from commit 9f2b7d572d)
If `FlappingRestartIntervalHours` for OSD is > 0 in the cephCluster CR,
then start OSD inside a bash script.
Signed-off-by: sp98 <sapillai@redhat.com>
(cherry picked from commit ffd69291f1)
The /tmp dir is not always available in some environments.
The only directories the operator should rely on
are the directories that are in its pod spec.
For assimilate-conf now we use the /var/lib/rook
directory to generate files since we know it is
a valid emptydir mounted into the operator pod.
Signed-off-by: travisn <tnielsen@redhat.com>
(cherry picked from commit 5359c4f4f5)
If mountinfo path is available cephcsi will
try to recover the fuse client. adding the
support for the same in Rook.
fixes: #13009
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
(cherry picked from commit 54ce2a96da)
Several messages were showing up in the operator log
more often than necessary, so let's change them to
debug level.
Signed-off-by: travisn <tnielsen@redhat.com>
(cherry picked from commit 29fb879675)
Add the ability to specify node profiles in the multus validation test.
This addresses a few points of early feedback on the validation tool.
Statements below critique the tool's behavior before this patch.
1. The tool assumes all daemons are on public and cluster network, which
means users who have a significantly smaller cluster net (a
design choice) cannot run a single test to determine if Rook is
likely to install correctly.
2. The tool does not have placement options to select only a subset of
Kubernetes nodes to run validation on.
3. Users of multus seem to have a dedicated pool of storage nodes more
often than the average Rook install. This makes sense for security-
and perforance-minded users. The tool cannot run a single test to
verify storage-only and general-workload nodes at one time.
These points are addressed by allowing users to specify configurations
for different "NodeTypes."
Each NodeType config has options for selecting the number of OSDs as
well as the number of other (non-OSD) Ceph daemons. This limits the
unnecessary exhaustion of cluster network addresses from critique 1.
Each NodeType config has its own placement (critique 2).
Users can define as many NodeTypes as needed to test the network for
their planned CephCluster. Specifically, this allows the tool to test
storage-only nodes and generalized-workload nodes at the same time. An
arbitrary number of NodeTypes are allowed to support even more highly
specialized cluster setups, such as multiple tiers of storage nodes
where some storage-only nodes may run more OSDs than others.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
(cherry picked from commit 0c721e05d5)
Create each cosi user for each object store and secret which holds
credentials.
Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
(cherry picked from commit a941b3c33f)
The activeStandby property of the filesystem CR was not
taking effect when changed from true to false. The standby
was only being enabled when true, but never applied when
changed to false.
Signed-off-by: travisn <tnielsen@redhat.com>
(cherry picked from commit f7094d1d13)
The ceph documentation indicates to use the crash profile
instead of specifying rw for the mgr access.
Signed-off-by: travisn <tnielsen@redhat.com>
(cherry picked from commit fab23d3407)
Using the new configuration parameters prometheusEndpoint and
prometheusEndpointSSLVerify users can configure dashboard to
point to their prometheus instance.
closes: #12876
Signed-off-by: Redouane Kachach <rkachach@redhat.com>
(cherry picked from commit c763935436)
When OSDs flap, ceph stops the OSD daemon if its marked down greater than
5 times in 600 seconds. But OSD pod restarts and marks the OSD `up` again.
This causes the PGs mapped to these OSDs to peer. While the PGs are peering,
IO to these PGs are blocked.
So we need to ensure that if ceph is marking OSD `down` due to flapping, OSD pod
should not restart to mark the OSDs `up` again.
This PR adds a sleep to the OSD pod if the container returned with a 0 exit code
Default behavior is to sleep for 6 hrs. But user can configure it from the
ceph cluster spec.
Signed-off-by: sp98 <sapillai@redhat.com>
(cherry picked from commit 4eb9f62205)
While two mgrs is considered sufficient, more mgr daemons are
now allowed in case the admin wants even more fault tolerance
to the active mgr going down, in case multiple mgrs go down.
Up to five mgr pods will be allowed. All mgrs will be in
standby mode except one active mgr. All mgr pods have a sidecar
that will update their respective pod specs with the active
or passive label.
Signed-off-by: travisn <tnielsen@redhat.com>
(cherry picked from commit 66547d27be)