1) Make the csi rbd container logs persisted in a file
(csi plugin, csi provisioner, csi addons sidecar)
2) Use the cephcluster api specs to configure the log rotate
3) Add log rotation to rotate the log file and
Add a sidecar log collector container
part-of: https://github.com/rook/rook/issues/12809
Signed-off-by: parth-gr <partharora1010@gmail.com>
Block devices can move between reboots. In corner cases, an OSDs block
device might move to a lower-indexed device while the previous device
does not exist. For example, an OSD on /dev/sde might move to /dev/sdd
on reboot if the original /dev/sdd died. There would be no /dev/sde
after that.
Users report that NVMe drives commonly change names, even when there are
no disk failures.
For these cases, ensure the activate script properly handles cases where
the previous disk is not present on the node and where the OSD is still
available on a different disk.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
While adding a new encryption key to slot 1
if there exists a key in slot 1 which is not
equal to the one we want to update it with,
we kill the slot and then add the new key to it.
While killing the slot, the existing code uses
the new key, which is not valid in such cases.
This patch modifies the code to use the key in
slot 0 (the one that we know works) to kill the slot.
Signed-off-by: Niraj Yadav <niryadav@redhat.com>
initially, we have the limit of 40 char set on
storageclassdeviceset.Name but seems like that is not enough and
upgrade clusters can face issue with this. So, increasing the limit to
50.
Signed-off-by: subhamkrai <srai@redhat.com>
The endpoint property honorLabels: true should be set so that
different rgw instances can show separate prometheus
values instead of being aggregated.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The snyk security check started failing with a warning
about the package being outdated:
github.com/libopenstorage/secrets/azure which depends on
github.com/Azure/azure-sdk-for-go/sdk/azidentity.
These and related packages are now updated.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
this lead to creation of 2 crush rules because of difference in
deviceClass names:
eg:
creating a new crush rule for changed deviceClass
("default~hdd"-->"hdd") on crush rule "test-crush-bug-az-ab-4"
Signed-off-by: Deepika Upadhyay <deepikaupadhyay01@gmail.com>
this will be disabled by default but in scenarios where the user want to
update failureDomain, DeviceClass etc, this option can be enabled, to be
noted this can lead to lot of data rebalancing and remapping. Use with
caution
Signed-off-by: Deepika Upadhyay <deepika.upadhyay@clyso.com>
For an advanced network configuration, the mon endpoints
expected for the ceph public network may be different from
the public IP address that is set on the K8s node. When
host networking is enabled in rook, the IP address of the
node was only allowed previously. Now, the IP address
used for a mon when scheduled on a node can be determined
by setting an annotation on each node where mons can run:
network.rook.io/mon-ip: <ip-address>
This annotation must be set before the mons are created.
Mon endpoints cannot be changed after they are already
running.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Add IPv6 support for multus validation tool. Also test that IPv6 support
works by specifying one of the NetAttachDefs with an IPv6 address range.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
When the clusters reach full, nearfull, or backfill full thresholds
ceph will raise health warnings and stop allowing IO or backfill
depending on the threshold. These settings require special ceph
commands instead of being generic ceph config. Allow these settings
to be set from the CephCluster CR in the spec.storage
section.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Currently we allow upgrade of other OSDs while migration
of OSD, due to change in backing store, is pending. This
will not work if the updated OSD ceph version does not support
the currently applied backing store. So rook will skip any
OSD upgrade if the OSD migration is pending.
Signed-off-by: sp98 <sapillai@redhat.com>
This PR adds a `settings` option in the `Module` struct
which can help users to futher customize the `balancer`
module with a differet mode.
Signed-off-by: sp98 <sapillai@redhat.com>
When IPv6-only network mode is enabled and when mons are configured to
listen only on msgrV2 port, Rook has to add the port to the mon
--public-bind-addr in order to force mons to stop listening on the
msgrV1 port. This can cause problems for IPv6 clusters where adding
':<port>' to the end of an IP addr looks like a valid IPv6 address.
In IPv6 mode, Rook needs to set this to '[<ip>]:<port>' with the
brackets to be a non-ambiguous IPv6 address.
In IPv4 mode, Rook needs to keep its existing behavior. IPv4 mode should
also continue to be the defualt assumption if the network config isn't
specified.
If 'dualStack: true' is specified, Rook is informed that the environment
is dual stack, but Rook doesn't know which IP family to expect from the
pod IP. In this scenario, it is safest for Rook not to apply the port,
since assuming either IPv4 or IPv6 format could be incorrect. In this
scenario, mons may still listen on msgrV1 ports, but this is preferable
to an environment where mons perpetually crash.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
currently node predicate watches for resourceVersion and heartbeat
changes, which are not critical things. So with the PR we will add
a early-exit condition for the predicate as an optimization and to
ignore considering for a reconcile of cephcluster if updates are
not crucial
closes: https://github.com/rook/rook/issues/14070
Signed-off-by: parth-gr <partharora1010@gmail.com>
Ceph-CSI support for fscrypt encryption of cephfs.
To achieve this commit add capability of mounting the
required `rook-ceph-csi-kms-config` configmap into
csi-cephfsplugin-provisioner and nodeplugin pods.
Further it modifies the ClusterRoles `cephfs-csi-nodeplugin` and
`cephfs-external-provisioner-runner` to grant privileges
required for reading encryption configuration and fetching
encryption secrets from either kubernetes secrets or
from a Key Management System (KMS).
These privileges are essential for the proper functioning of
ceph-csi-cephfs with fscrypt encryption.
The following privileges have been added:
- `secrets/get`: Allows reading of secrets for encryption.
- `configmaps/get`: Grants access to configuration maps,
this is used to read encryption configuration.
- `serviceaccounts/get`: Enables retrieval of service accounts for
authentication to KMS and for retrieving encryption secrets
stored there.
- `serviceaccounts/token/create`: Allows creation of service account tokens,
which are required for authenticating requests to KMS
when retrieving encryption secrets.
The commit also updated the csi documentation to include cephfs
in the encryption section, with examples updated accordingly.
Signed-off-by: NymanRobin <robin.nyman@est.tech>
SVG cleanup job uses the cephSubvolumeGroup env name
from spec.Name. But sometimes it can be empty. In that
case rook should use the CR name.
Signed-off-by: sp98 <sapillai@redhat.com>
LVM-based OSDs on PVCs are legacy and are not well tested.
Therefore, we add status to the cephcluster CR that at least
gives a clue that the OSDs should be replaced.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
fix UpdateActiveMgrLabel to retry label update on failure. Previously,
the function always returned the current manager, even if update
failed, leading to inconsistent labels. This fix ensures retries on
failures.
Fixes: https://github.com/rook/rook/issues/13601
Signed-off-by: Redouane Kachach <rkachach@ibm.com>
there are some cases where storageClassDeviceSets length
are more 63 chars and osd provisioned failed due to this.
with error
```
failed to run osd provisioning job for PVC "ocs-deviceset-*-local-volume-storageclass-0-data-*": Job.batch "rook-ceph-osd-prepare-*" is invalid: [spec.template.spec.volumes[10].name: Invalid value: "ocs-deviceset-*-local-volume-storageclass-0-data-*": must be no more than 63 characters, spec.template.spec.containers[0].volumeMounts[9].name: Not found: "ocs-deviceset-*-local-volume-storageclass-0-data-*",
```
Using x-validator to limit the maxLength to the deviceClassSet.Name to
40 chars and adding code check that overall pvc name should be less than
63 chars.
Signed-off-by: subhamkrai <srai@redhat.com>
Someone testing the Multus holder pod removal feature encountered an
issue where the migration process failed to lead to a system state where
PVCs could be created successfully.
The root cause was found to be a ceph csi config map wherein the primary
CephCluster entry was lacking a value for the "namespace" field.
I observed this once in my development on the holder pod removal
feature, but I was unable to reproduce and assumed it was my own error.
Since this has been seen in a user environment, it must be that the
error is a race condition, and I am unable to determine the exact source
of the bug.
I do not believe this bug would be present if the code that updates the
CSI configmap were properly idempotent, but it has many conditions based
on prior states, and I was unable to determine how to resolve this
underlying impelementation pattern issue.
Instead, I opted to separate the `clusterKey` parameter into two clear
parts:
1. `clusterID` for when `clusterKey` is used as an analogue for
`clusterID`
2. `clusterNamespace` for when `clusterKey` is used as an analogue for
`clusterNamespace`
I added unit tests to ensure that SaveClusterConfig() will be able to
detect when the namespace is currently missing, and using the new
`clusterNamespace` field, it should always know what value to use as
input when correcting the bug in already-installed systems.
I also verified that this update works when the function simultaneously
removes netNamespaceFilePath entries, and that those entries are removed
properly.
Finally, manual testing also verifies the change.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
The mon was listening on both v1 and v2 ports inside the daemon
even though the service endpoint would not redirect any traffic
to the v1 port. When msgr2 is required, disable the mon from
listening on the v1 port for reduced security footprint.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
With the removal of Ceph Pacific support, no need to check if Quincy
is avaiable to support encryption on the wire.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Travis Nielsen
Blaine Gardner
subhamkrai
parth-gr
subham rai
Niraj Yadav
Deepika Upadhyay
dependabot[bot]
Michael Adam
Santosh Pillai
Brian Dowling
parth-gr
NymanRobin
Satoru Takeuchi
Redouane Kachach