The /run/udev directory is now mounted in the OSD init container. This change
is necessary because `ceph-volume activate` needs access to /run/udev [1] in order
to properly handle devices. Without this mount, `ceph-volume activate` could
fail to discover devices during OSD initialization.
[1] https://github.com/ceph/ceph/commit/c2e8c295dc46f1bd74861feb163d9d82f8bb5509
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>
(cherry picked from commit dbcd0d46dd)
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
Resolves CIS benchmark rule 5.7.3, Pods part. SecurityContext
should be explicitly defined in pod level of Pod spec section.
It is sufficient to specify empty securityContext to satisfy
CIS 5.7.3 rule.
5.7.3 Apply Security Context to Your Pods and Containers
When designing your containers and pods, make sure
that you configure the security context for your pods,
containers, and volumes.
Signed-off-by: Peter Razumovsky <prazumovsky@mirantis.com>
(cherry picked from commit 516eab4d8a)
Disabling the RBD and CephFS fencing
in Rook for now as its having bugs where
Rook is blocklisting wrong IP address
due to timing issues.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
(cherry picked from commit 1d1ed5e962)
When the MDS liveness probe times out, it should not fail the probe. If
the cluster has a network partition or other issue that causes the Ceph
mon cluster to become unstable, `ceph ...` commands can hang and cause
a timeout. In this case, the MDS should not be restarted so as to not
cause cascading cluster disruption.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
(cherry picked from commit ad1bae9e1b)
The crash collector daemon had missed the ability to
add custom annotations via the cephcluster CR, as already
supported for all the other daemons.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
(cherry picked from commit fa2941d098)
Previously, Rook would fail to create monitors for a `CephCluster` spec that
required them to be located in distinct failure domains:
```yaml
spec:
mon:
count: 3
failureDomainLabel: topology.kubernetes.io/zone
zones:
- eu-central-1a
- eu-central-1b
- eu-central-1c
```
This was due to the fact that Rook was mistakenly getting the
`failureDomainLabel` from `spec.mon.stretchCluster.failureDomainLabel`.
This commit ensures `spec.mon.failureDomainLabel` is being used instead, which
fixes the cluster deployment for the spec above.
Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>
(cherry picked from commit 1c45677f5e)
Alerting on controller-runtime's workqueue_depth can be useful for
debugging controllers. Also having a prometheus target for a pod gives
another data point that the system is working as expected. It is useful
for uptime alerts.
Make the bind address configurable via the configmap while still retaining the default
behavior that it is disabled.
Resolves: #14538
Signed-off-by: Justin Cichra <jcichra@cloudflare.com>
(cherry picked from commit 74a79b24b3)
The jobs to detect the ceph and csi versions now can have
custom annotationso and labels added to them. They are
very short-lived jobs, but they may need custom annotations
in some environments to run.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
(cherry picked from commit 5014d5bcf2)
Since Reef v18.2.4 is released, set it to the default
ceph version that is deployed with Rook.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
(cherry picked from commit e26920a483)
# Conflicts:
# deploy/examples/images.txt
If the dashboard cert already exists when ssl is enabled,
we need to not restart the dashboard module. The module
should only be restarted when there is a change to the
dashboard configuration. When ssl is enabled, the dashboard
is essentially being restarted every time there is a reconcile
which is causing issues with the mgr getting blocklisted
unnecessarily by ceph during failover.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
(cherry picked from commit 6ce616bf96)
This commit adds the flexibility to configure kube apiserver qps
as per the user requirement and also keeps the existing values as
the default one.
Signed-off-by: yite.gu <yitegu0@gmail.com>
(cherry picked from commit 80541a23c9)
Reset the Multus validation tool debounce time to its intended 30 second
value. It was changed to 5 for testing, and the change was accidentally
committed.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
(cherry picked from commit 5884f1c900)
In order to help users check that they have implemented the newly-added
Multus host configuration prerequisites, add a check to the validation
tool to verify connectivity.
Because users who are already running clusters with Multus enabled, add
a flag that allows users to only check for host configuration
prerequisites. This mode will not start the large number of clients that
would normally be started because those clients could disrupt a running
Rook cluster negatively.
Host checking pods require host network access. Many Kubernetes
distributions have pod security features enabled. In order to allow
non-Vanilla distros to run this tool, allow specifying a service account
that pods will run as, which can be configured by the admin to allow
test pods.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
(cherry picked from commit 33f5407dd4)
# Conflicts:
# Documentation/CRDs/Cluster/network-providers.md
# deploy/examples/multus-validation.yaml
Pools in stretch clusters must all specify the same
CRUSH rule. No pools can use a different rule. When there
is a change in the device class, we do not even expect to update
the crush rules in a stretch cluster. Different device classes
are not supported in stretch clusters, and it's expected to be
a homogenous environment. Therefore, skip all crush rule updates
in stretch clusters.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
(cherry picked from commit b92bbba671)
this commit updates go-retryablehttp from 0.7.6 to 0.7.7
and without this govulcheck ci is failing.
Signed-off-by: subhamkrai <srai@redhat.com>
(cherry picked from commit 9fbf3d3)
Signed-off-by: subhamkrai <srai@redhat.com>
skip creating networkFence when 'ROOK_CSI_DISABLE_DRIVER' is set to
'false'.
Signed-off-by: subhamkrai <srai@redhat.com>
(cherry picked from commit 4be450ae4b)
Block devices can move between reboots. In corner cases, an OSDs block
device might move to a lower-indexed device while the previous device
does not exist. For example, an OSD on /dev/sde might move to /dev/sdd
on reboot if the original /dev/sdd died. There would be no /dev/sde
after that.
Users report that NVMe drives commonly change names, even when there are
no disk failures.
For these cases, ensure the activate script properly handles cases where
the previous disk is not present on the node and where the OSD is still
available on a different disk.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
(cherry picked from commit f2304bf8b8)
While adding a new encryption key to slot 1
if there exists a key in slot 1 which is not
equal to the one we want to update it with,
we kill the slot and then add the new key to it.
While killing the slot, the existing code uses
the new key, which is not valid in such cases.
This patch modifies the code to use the key in
slot 0 (the one that we know works) to kill the slot.
Signed-off-by: Niraj Yadav <niryadav@redhat.com>
(cherry picked from commit b8b7af3495)
initially, we have the limit of 40 char set on
storageclassdeviceset.Name but seems like that is not enough and
upgrade clusters can face issue with this. So, increasing the limit to
50.
Signed-off-by: subhamkrai <srai@redhat.com>
(cherry picked from commit 1dcc99ec1d)
The endpoint property honorLabels: true should be set so that
different rgw instances can show separate prometheus
values instead of being aggregated.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
(cherry picked from commit eed1033485)
The snyk security check started failing with a warning
about the package being outdated:
github.com/libopenstorage/secrets/azure which depends on
github.com/Azure/azure-sdk-for-go/sdk/azidentity.
These and related packages are now updated.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
(cherry picked from commit c5fd5cd0b5)
this lead to creation of 2 crush rules because of difference in
deviceClass names:
eg:
creating a new crush rule for changed deviceClass
("default~hdd"-->"hdd") on crush rule "test-crush-bug-az-ab-4"
Signed-off-by: Deepika Upadhyay <deepikaupadhyay01@gmail.com>
(cherry picked from commit 41116be919)
For an advanced network configuration, the mon endpoints
expected for the ceph public network may be different from
the public IP address that is set on the K8s node. When
host networking is enabled in rook, the IP address of the
node was only allowed previously. Now, the IP address
used for a mon when scheduled on a node can be determined
by setting an annotation on each node where mons can run:
network.rook.io/mon-ip: <ip-address>
This annotation must be set before the mons are created.
Mon endpoints cannot be changed after they are already
running.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
(cherry picked from commit e90ed3b912)
Add IPv6 support for multus validation tool. Also test that IPv6 support
works by specifying one of the NetAttachDefs with an IPv6 address range.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
(cherry picked from commit 08a27940ed)
When the clusters reach full, nearfull, or backfill full thresholds
ceph will raise health warnings and stop allowing IO or backfill
depending on the threshold. These settings require special ceph
commands instead of being generic ceph config. Allow these settings
to be set from the CephCluster CR in the spec.storage
section.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
(cherry picked from commit e479b051bc)
This PR adds a `settings` option in the `Module` struct
which can help users to futher customize the `balancer`
module with a differet mode.
Signed-off-by: sp98 <sapillai@redhat.com>
(cherry picked from commit 1ad37ecce9)
This reduces the safe-to-remove retry interval from 1m to 15s.
Closes#14247
Signed-off-by: Brian Dowling <bjd-dev@simplicity.net>
(cherry picked from commit ef0ed93134)
fixing 'secert' to `secret` and adding workds like
`NotIn, addin` to skip work list.
Signed-off-by: subhamkrai <srai@redhat.com>
(cherry picked from commit 85c25bbc6e)
When IPv6-only network mode is enabled and when mons are configured to
listen only on msgrV2 port, Rook has to add the port to the mon
--public-bind-addr in order to force mons to stop listening on the
msgrV1 port. This can cause problems for IPv6 clusters where adding
':<port>' to the end of an IP addr looks like a valid IPv6 address.
In IPv6 mode, Rook needs to set this to '[<ip>]:<port>' with the
brackets to be a non-ambiguous IPv6 address.
In IPv4 mode, Rook needs to keep its existing behavior. IPv4 mode should
also continue to be the defualt assumption if the network config isn't
specified.
If 'dualStack: true' is specified, Rook is informed that the environment
is dual stack, but Rook doesn't know which IP family to expect from the
pod IP. In this scenario, it is safest for Rook not to apply the port,
since assuming either IPv4 or IPv6 format could be incorrect. In this
scenario, mons may still listen on msgrV1 ports, but this is preferable
to an environment where mons perpetually crash.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
(cherry picked from commit 573a47e736)
SVG cleanup job uses the cephSubvolumeGroup env name
from spec.Name. But sometimes it can be empty. In that
case rook should use the CR name.
Signed-off-by: sp98 <sapillai@redhat.com>
(cherry picked from commit 33ce8f0c8e)
Since pacific is no longer supported, we can remove pacific specific
code.
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
(cherry picked from commit f776c0b35e)
LVM-based OSDs on PVCs are legacy and are not well tested.
Therefore, we add status to the cephcluster CR that at least
gives a clue that the OSDs should be replaced.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
(cherry picked from commit 3628e83892)
fix UpdateActiveMgrLabel to retry label update on failure. Previously,
the function always returned the current manager, even if update
failed, leading to inconsistent labels. This fix ensures retries on
failures.
Fixes: https://github.com/rook/rook/issues/13601
Signed-off-by: Redouane Kachach <rkachach@ibm.com>
(cherry picked from commit ee7c71c89e)
Someone testing the Multus holder pod removal feature encountered an
issue where the migration process failed to lead to a system state where
PVCs could be created successfully.
The root cause was found to be a ceph csi config map wherein the primary
CephCluster entry was lacking a value for the "namespace" field.
I observed this once in my development on the holder pod removal
feature, but I was unable to reproduce and assumed it was my own error.
Since this has been seen in a user environment, it must be that the
error is a race condition, and I am unable to determine the exact source
of the bug.
I do not believe this bug would be present if the code that updates the
CSI configmap were properly idempotent, but it has many conditions based
on prior states, and I was unable to determine how to resolve this
underlying impelementation pattern issue.
Instead, I opted to separate the `clusterKey` parameter into two clear
parts:
1. `clusterID` for when `clusterKey` is used as an analogue for
`clusterID`
2. `clusterNamespace` for when `clusterKey` is used as an analogue for
`clusterNamespace`
I added unit tests to ensure that SaveClusterConfig() will be able to
detect when the namespace is currently missing, and using the new
`clusterNamespace` field, it should always know what value to use as
input when correcting the bug in already-installed systems.
I also verified that this update works when the function simultaneously
removes netNamespaceFilePath entries, and that those entries are removed
properly.
Finally, manual testing also verifies the change.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
(cherry picked from commit 4ff881240f)
there are some cases where storageClassDeviceSets length
are more 63 chars and osd provisioned failed due to this.
with error
```
failed to run osd provisioning job for PVC "ocs-deviceset-*-local-volume-storageclass-0-data-*": Job.batch "rook-ceph-osd-prepare-*" is invalid: [spec.template.spec.volumes[10].name: Invalid value: "ocs-deviceset-*-local-volume-storageclass-0-data-*": must be no more than 63 characters, spec.template.spec.containers[0].volumeMounts[9].name: Not found: "ocs-deviceset-*-local-volume-storageclass-0-data-*",
```
Using x-validator to limit the maxLength to the deviceClassSet.Name to
40 chars and adding code check that overall pvc name should be less than
63 chars.
Signed-off-by: subhamkrai <srai@redhat.com>
(cherry picked from commit 6f5cc9b01d)
The mon was listening on both v1 and v2 ports inside the daemon
even though the service endpoint would not redirect any traffic
to the v1 port. When msgr2 is required, disable the mon from
listening on the v1 port for reduced security footprint.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
(cherry picked from commit 954b000ba6)
With the removal of Ceph Pacific support, no need to check if Quincy
is avaiable to support encryption on the wire.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
(cherry picked from commit 25d36c4c75)
Guillaume Abrioux
Peter Razumovsky
Madhu Rajanna
Blaine Gardner
Travis Nielsen
Benoît Knecht
Justin Cichra
Artem Torubarov
yite.gu
subham rai
subhamkrai
Niraj Yadav
Deepika Upadhyay
Michael Adam
sp98
Brian Dowling
Satoru Takeuchi
Redouane Kachach