Someone testing the Multus holder pod removal feature encountered an
issue where the migration process failed to lead to a system state where
PVCs could be created successfully.
The root cause was found to be a ceph csi config map wherein the primary
CephCluster entry was lacking a value for the "namespace" field.
I observed this once in my development on the holder pod removal
feature, but I was unable to reproduce and assumed it was my own error.
Since this has been seen in a user environment, it must be that the
error is a race condition, and I am unable to determine the exact source
of the bug.
I do not believe this bug would be present if the code that updates the
CSI configmap were properly idempotent, but it has many conditions based
on prior states, and I was unable to determine how to resolve this
underlying impelementation pattern issue.
Instead, I opted to separate the `clusterKey` parameter into two clear
parts:
1. `clusterID` for when `clusterKey` is used as an analogue for
`clusterID`
2. `clusterNamespace` for when `clusterKey` is used as an analogue for
`clusterNamespace`
I added unit tests to ensure that SaveClusterConfig() will be able to
detect when the namespace is currently missing, and using the new
`clusterNamespace` field, it should always know what value to use as
input when correcting the bug in already-installed systems.
I also verified that this update works when the function simultaneously
removes netNamespaceFilePath entries, and that those entries are removed
properly.
Finally, manual testing also verifies the change.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
(cherry picked from commit 4ff881240f)
In the primary CSI reconcile, ensure the CSI config map
(`rook-ceph-csi-config`) has the correct owner info.
This corrects any pre-existing config maps that might have incorrect
owner info, which has observed to include references to CephClusters.
The config map should only have a single reference, and it should refer
to the operator deployment.
If any existing Rook clusters have a CSI config map which has a
CephCluster as an owner, this change will ensure that the config map is
not deleted when the CephCluster is deleted. This is especially
important for any environments with multiple CephClusters installed.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
(cherry picked from commit 605b9633ea)
There have been some issues with non-CSI Rook controllers that are
creating the CSI config configmap (`rook-ceph-csi-config`). This causes
problems with the K8s OwnerReference. The primary CSI reconciler
(controller) creates the configmap with the correct owner reference,
which is supposed to be the operator deployment.
Other instances were creating the configmap with owner references set to
the CephCluster. This is a minor bug, but it can result in this
configmap being deleted along with the first CephCluster that initially
created it.
To fix this issue, remove all instances of `CreateCsiConfigMap()` except
the single usage which the CSI reconcile uses to initially create the
configmap. Other controllers that might have attempted to create this
configmap previously will return an error indicating that it is waiting
for the configmap to be created.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
(cherry picked from commit eb3278d575)
Add new CSI_DISABLE_HOLDER_PODS option for rook-ceph-operator.
This option will disable holder pods when set to "true".
In the long term, Rook plans to deprecate the holder pods entirely.
This new option will allow users to choose to migrate their clusters to
non-holder clusters when they are ready and able, giving them time to
gracefully migrate before the holders are permanently removed.
This option is set to "false" by default so that upgrading users don't
have their CSI pods modified unexpectedly.
Example manifests are modified to set this value to true so that new
clusters will not deploy holder pods.
Migrating users are provided with documentation to instruct them about
the new requirements they need to satisfy to successfully remove holder
pods, a procedure for migrating pods from holder to non-holder mounts,
and a way to delete holder pods once they are no longer in use.
When users set CSI_DISABLE_HOLDER_PODS="true", the CSI controller will
no longer deploy or update the holder pod Daemonsets, but it does not
delete any existing Daemonsets. This allows already-attached PVCs to
continue operating normally with their network connection continuing to
exist in the current holder pod. This is critical to avoid causing
ia cluster-wide storage outage.
More info: https://github.com/rook/rook/issues/13055
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
During cluster creation, csi config map was first filled with
mon ips and without CSIDriverOptions.
This commit makes sure CSIDriverOptions are added at the begining
when the entry is first created.
Signed-off-by: Rakshith R <rar@redhat.com>
This commit resolves the issue where the netNamespaceFilePath
was not being updated for all clusterIDs present in the csi configMap.
Signed-off-by: Praveen M <m.praveen@ibm.com>
The updateCSIDriverOptions() func was overwriting the ClusterInfo.CephFS
struct. Due to which multus and csi hostnetwork canary tests were hanging.
Signed-off-by: Praveen M <m.praveen@ibm.com>
This commit uses an csi config map API exposed by Ceph-CSI.
This will reduce the efforts of keeping the configuration in
sync with Ceph-CSI.
Signed-off-by: Praveen M <m.praveen@ibm.com>
This commit adds new CSIDriverOptions section in
cephCluster CR. This section contains settings
for read affinity and kernel+fuse Mount options
These settings will be injected directly into
rook-ceph-csi-config cm to be applicable per
ceph cluster.
Signed-off-by: Rakshith R <rar@redhat.com>
The msgr2 conversion assumed the endpoints were formatted
with a single :, which is not compatible with ipv6.
Now if the endpoint ends with the v1 port, it will
simply update the prefix to v2.
Signed-off-by: travisn <tnielsen@redhat.com>
When msgr2 is required, ensure the mon endpoints passed to
the csi configmap are on port 3300. The mons will be
listening on both 6789 and 3300. Existing volumes can
continue using port 6789 while new volumes will use
port 3300.
Signed-off-by: travisn <tnielsen@redhat.com>
This commit adds holder pod support
for nfs to be able to work in non-host
networking scenario, without being
affected by restarts of csi-nfsplugin.
Signed-off-by: Rakshith R <rar@redhat.com>
If a mon is added or removed in a ceph
cluster all the CSI clusterID's belongs
to the ceph cluster should also gets updated.
Added a new cluster identifier to the csi configmap,
i.e ceph clusterNamespace which helps to identify
the clusterID's belongs to that ceph cluster.
Note:- to support the upgraded clusters,we
can ask users to update the mon IP's for the
problamatic clusterID as it wont be updated
by the operator.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
If the csi driver was enabled at a later time, the mon endpoints
would not be populated. The populate the csi config, either a new
cluster reconcile or restarting the operator would be necessary.
It is simpler just to update the csi configmap with every reconcile
even if the csi driver is not enabled.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
If the pod namespace was not found by the csi helper to update
the mon endpoints configmap, the reconcile was failing. To allow
for the update to continue in unit tests and ignore the env var,
we only log the warning instead of failing the reconcile.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The rook-ceph-csi-config config map now includes CephFS subvolumegroups
when a subvolumegroup CR is created. It is useful for ceph-csi to
understand which subvolumegroup to use for a given cluster.
Signed-off-by: Sébastien Han <seb@redhat.com>
This commit is a large refactor on how the operator starts, stops and
how it starts various sub-components such as the ceph-csi driver. It
also refines the way we cancel orchestrations. We don't use breakpoints
anymore but send our self a SIGUP to reload our controller runtime
manager.
The reload will happen under different circonstances like:
* a new adminission controller secret is created/deleted/changed
* a CephCluster CR is edited
As mentioned earlier, the csi driver now has its own controller, just
like flex. It reacts to change in the operator config map for particular
ROOK_CSI_ fields.
A second new controller for the operator's general config has been
created, it manages:
* the logging level
* the ceph CLI command timeout
* the discovery daemon
The operator reacts much more rapidly to cancellation events by stopping
the manager's context and reloading it.
Signed-off-by: Sébastien Han <seb@redhat.com>
It's better to validate ownerReferences when setting them. In addition, we should use
controllerrutil.Set{Controller,Owner}Reference, that have such validation, as possible.
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
The ceph config and client packages are conceptually the same.
To avoid circular dependencies in some cases, we simplify by combining
the packages into the client package.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Prior attempts to delete CSI drivers using owner references to the
Ceph ConfigMap resulted in garbage collection of the driver
resources as described in rook#4590. This patch manually deletes these
resources by name as a stopgap to provide an appropriate user
experience while the team investigates this issue further.
Resolves: rook#4824
Signed-off-by: Elise Gafford <egafford@redhat.com>
We create CSI resources when we create the first Ceph cluster.
This change deletes all CSI resources when we delete the last Ceph
cluster.
Partially resolves: #4234
Signed-off-by: Elise Gafford <egafford@redhat.com>
In order to facilitate the deletion of CSI resources, this patch
adds ownerReferences to all CSI deployments, daemonsets, and
services. This ownerRef refers to the CSI ConfigMap in all cases.
Partially resolves#4234
Signed-off-by: Elise Gafford <egafford@redhat.com>
Return an error if the function that updates the csi config map
can not determine a valid namespace for the config map.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
Blaine Gardner
parth-gr
Rakshith R
Praveen M
guoguangwu
travisn
Abirdcfly
Sébastien Han
Madhu Rajanna
Divyansh Kamboj
Satoru Takeuchi
Arun Kumar Mohan
Elise Gafford
John Mulligan