Change how Rook detects network CIDRs for Multus networks. The IPAM
configuration is only defined as an arbitrary string JSON blob with a
"type" field and nothing more. Rook's detection of CIDRs for whereabouts
had already grown out of date since the initial implementation.
Additionally, Rook did not support DHCP IPAM, which is a reasonable
choice for users. And more, Rook did not support CNI plugin chaining,
which further complicates NADs. Based on the CNI spec, network chaning
can result in any changes to network CIDRs from the first-given plugin.
All these problems make it more and more difficult for Rook to support
Multus by inspecting the NAD itself to predict network CIDRs. Instead,
it is better for Rook to treat the CNI process as a black box. To
preserve legacy functionality of auto-detecting networks and to make
that as robust as possible, change to a canary-style architecture like
that used for Ceph mons, from which Rook will detect the network CIDRs
if possible.
Also allow users to specify overrides for CIDR ranges. This allows Rook
to still support esoteric and unexpected NAD or network configurations
where a CIDR range is not detectable or where the range detected would
be incomplete. Because it may be impossible for Rook to understand the
network CIDRs wholistically while residing only on a portion of the
network, this feature should have been present from Multus's inception.
Improving CIDR auto-detection and allowing users to specify overrides
for auto-detected CIDRs rounds out Rook's Multus support for CephCluster
(core/RADOS) installations. No further architectural changes should be
needed for CephClusters as regards application of public/cluster network
CIDRs for Multus networks.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
(cherry picked from commit 3c43268d0a)
We need to restart all the ceph daemons whenever
cephCluster network settings are modified like
requiremsgr2, encryption and compression. This
required for Ceph to consider the new settings
it require new ceph daemons all over.
Signed-off-by: subhamkrai <srai@redhat.com>
(cherry picked from commit 29d2b6a071)
* fixes mon zonal deployment of 2 per zone to 1 if zones field is
specified.
* adds zones field doc to CRDs Documentation.
* add code for picking up failureDomainLabel if specified for zones
field in mon deployment
Signed-off-by: Deepika Upadhyay <deepika@koor.tech>
(cherry picked from commit b67bf2d6f0)
The function ParseMonEndpoints existed twice, once in the mon package
and once in the controller package. It has now been deleted from the
mon package. Usages in the mon package now reference the function
in the controller package.
Signed-off-by: Henry Davies <henrydavies@hotmail.co.uk>
(cherry picked from commit 2c8949e195)
When msgr2 is required, ensure the mon endpoints passed to
the csi configmap are on port 3300. The mons will be
listening on both 6789 and 3300. Existing volumes can
continue using port 6789 while new volumes will use
port 3300.
Signed-off-by: travisn <tnielsen@redhat.com>
Mons that are configured on msgr1 will be using the v1 port
in the csi driver since only one port is allowed in the configmap.
To allow mons to be started on v2 while keeping quorum with the
v1 mons, the v1 mons must continue running on the v1 port.
So even if msgr2 is required, allow existing v1 mons to continue
running on that port.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Environment variables are not recommended for secrets in pods since
they can be easily leaked if the environemnt variables are logged.
By mounting the mon secret as a file, the mgr and osd prepare pods
can read the mon secret from a file for better security.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The fsid, username, and user secrets were from legacy and no
longer needed on the osd daemon or mon pods. This removes the
unnecessary env vars from the pod specs.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The mons that are out of quorum may cause ceph commands to timeout
or fail unnecessarily trying to connect to a mon that is no longer
online. Now the mon health check will update the mon endpoints configmap
when a mon is detected out of quorum. This also means that if the
operator is restarted during a mon failover, the failed mon will no
longer remain in the ceph.conf, thus allowing the quorum to be more
likely to respond to the mons that are still in quorum.
The update for mons out of quorum only applies if other mons are in
quorum. If quorum is down, the configmap will not keep track of the offline
mons since too many are offline.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
If the host network setting is changed in the cluster CR,
the existing mons must continue using the same network
settings or else the operator would update their pod
specs with the incorrect settings and cause mon quorum
to go down. Now the network setting is preserved so
the admin could switch between host and non-host network
configuration without reinstalling.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Introduce a new env variable ROOK_CSI_IMAGE_PULL_POLICY in rook operator configmap which should be used to
customize the imagePullPolicy for the csi driver and imagePullPolicy property in cephVersionSpec for ceph pods.
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
if Multus is enabled the clusterinfo should be updated with
network as multus as to run the ceph cmds in remote
executor
Signed-off-by: parth-gr <paarora@redhat.com>
Mon failover reliability is improved in two scenarios with these changes.
First, if the failover is aborted because of failing to start the new mon
and wait for it to join quorum, the new mon will be fully removed and
the failover rolled back. Previously, the operator would leave the new
mon pod running even if it never joined quorum. Now the timeout for the failover
is increased to wait longer for the new mon, but if it times out, there
are no leftovers from the cancelled failover.
Second, if the operator is failed in the middle of a failover
before the old mon is removed, the operator needs to remove an extra
mon. The operator was only selecting a random mon for removal instead
of honoring the stretch cluster topology or looking for mons
on the same node that could be removed to restore the correct
desired number of mons.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The tool gofmt in go 1.19 requires certain formatting
in the comments section for better rendering.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
During certain maintenance tasks the admin will own running
operations on the ceph mons and osds, and the operator should
not interfere with those operations. If the operator sees
any mon in debug mode, every reconcile and mon health check
will be skipped. Thus, mons will not be updated while any
one of them is in maintenance. During OSD reconcile, individual
OSD deployment updates will only be skipped for OSDs that are
actively being debugged.
The debug mode for osd and mon deployments is signaled by
creating the ceph.rook.io/do-not-reconcile label.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
This commit replaces `os.Setenv` with `t.Setenv` in tests. The
environment variable is automatically restored to its original value
when the test and all its subtests complete.
Reference: https://pkg.go.dev/testing#T.Setenv
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
Some context canceled were missed and not reported correctly. We had
multiple reconcilers running in //. Since they all run in goroutine then
will continue to run until they timeout (if timeout is available,
otherwise would run forever until the op is restarted).
Signed-off-by: Sébastien Han <seb@redhat.com>
With octopus coming to end of life, we remove support from
Rook for deploying Ceph Octopus and assume a min version of
Pacific v16. Any checks for octopus or earlier are removed
from the reconciles since they are obsolete.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
For the ceph telemetry report the csi version, k8s version,
and metrics from the desired state in the cephcluster CR such
as mon count, stretch cluster, number of device sets, and
so on.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The CSI package needs to load clusterInfo, today this code is in the mon
package which makes the call of LoadClusterInfo impossible without
having a circular import.
Signed-off-by: Sébastien Han <seb@redhat.com>
This removes double package imports. Example:
```
"github.com/rook/rook/pkg/apis/ceph.rook.io/v1"
cephv1 "github.com/rook/rook/pkg/apis/ceph.rook.io/v1"
```
Only one is now being used as shown in go-staticcheck ST1019
Signed-off-by: Alexander Trost <galexrt@googlemail.com>
If a mon is added or removed in a ceph
cluster all the CSI clusterID's belongs
to the ceph cluster should also gets updated.
Added a new cluster identifier to the csi configmap,
i.e ceph clusterNamespace which helps to identify
the clusterID's belongs to that ceph cluster.
Note:- to support the upgraded clusters,we
can ask users to update the mon IP's for the
problamatic clusterID as it wont be updated
by the operator.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
Previously, the struct maintaining the list of cluster was still
initialized with a cluster item. Then the monitoring check will see that
the cluster is part of the struct already and thus won't run the
monitoring go routine again.
Now each time we cancel the context, we also remove the cluster item
from the map so that when the controller runs again, the monitoring
struct is re-populated and the go routine runs and statuses are updated.
Closes: https://github.com/rook/rook/issues/9911
Signed-off-by: Sébastien Han <seb@redhat.com>
Older kernels still fail to connect for rbd/cephfs with
the msgr2 port due to a difference in http protocol expected.
For now, we keep the default port as msgr1 for the csi driver
and other client connections, unless msgr2 is strictly required
for encryption or compression.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The canary pods are very short-lived pods and are not
running ceph in any form. The startup probe was mistakenly
enabled for the canary pod, which could cause the pod
to fail unexpectedly if the startup probe was triggered
after the delay timeout.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
msgr2 allows for encryption and/or compression across the wire.
Settings for enabling the encryption and compression are now
available in the cluster CR to that ceph will be automatically
configured with these settings when desired.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
guoguangwu
Travis Nielsen
Blaine Gardner
subhamkrai
Deepika Upadhyay
Henry Davies
sp98
parth-grandBlaine Gardner
Avan Thakkar
Eng Zer Jun
Sébastien Han
Josh Soref
FengJiankui
Alexander Trost
Madhu Rajanna
Divyansh Kamboj