When IPv6-only network mode is enabled and when mons are configured to
listen only on msgrV2 port, Rook has to add the port to the mon
--public-bind-addr in order to force mons to stop listening on the
msgrV1 port. This can cause problems for IPv6 clusters where adding
':<port>' to the end of an IP addr looks like a valid IPv6 address.
In IPv6 mode, Rook needs to set this to '[<ip>]:<port>' with the
brackets to be a non-ambiguous IPv6 address.
In IPv4 mode, Rook needs to keep its existing behavior. IPv4 mode should
also continue to be the defualt assumption if the network config isn't
specified.
If 'dualStack: true' is specified, Rook is informed that the environment
is dual stack, but Rook doesn't know which IP family to expect from the
pod IP. In this scenario, it is safest for Rook not to apply the port,
since assuming either IPv4 or IPv6 format could be incorrect. In this
scenario, mons may still listen on msgrV1 ports, but this is preferable
to an environment where mons perpetually crash.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
(cherry picked from commit 573a47e736)
The mon was listening on both v1 and v2 ports inside the daemon
even though the service endpoint would not redirect any traffic
to the v1 port. When msgr2 is required, disable the mon from
listening on the v1 port for reduced security footprint.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
(cherry picked from commit 954b000ba6)
Change how Rook detects network CIDRs for Multus networks. The IPAM
configuration is only defined as an arbitrary string JSON blob with a
"type" field and nothing more. Rook's detection of CIDRs for whereabouts
had already grown out of date since the initial implementation.
Additionally, Rook did not support DHCP IPAM, which is a reasonable
choice for users. And more, Rook did not support CNI plugin chaining,
which further complicates NADs. Based on the CNI spec, network chaning
can result in any changes to network CIDRs from the first-given plugin.
All these problems make it more and more difficult for Rook to support
Multus by inspecting the NAD itself to predict network CIDRs. Instead,
it is better for Rook to treat the CNI process as a black box. To
preserve legacy functionality of auto-detecting networks and to make
that as robust as possible, change to a canary-style architecture like
that used for Ceph mons, from which Rook will detect the network CIDRs
if possible.
Also allow users to specify overrides for CIDR ranges. This allows Rook
to still support esoteric and unexpected NAD or network configurations
where a CIDR range is not detectable or where the range detected would
be incomplete. Because it may be impossible for Rook to understand the
network CIDRs wholistically while residing only on a portion of the
network, this feature should have been present from Multus's inception.
Improving CIDR auto-detection and allowing users to specify overrides
for auto-detected CIDRs rounds out Rook's Multus support for CephCluster
(core/RADOS) installations. No further architectural changes should be
needed for CephClusters as regards application of public/cluster network
CIDRs for Multus networks.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
We need to restart all the ceph daemons whenever
cephCluster network settings are modified like
requiremsgr2, encryption and compression. This
required for Ceph to consider the new settings
it require new ceph daemons all over.
Signed-off-by: subhamkrai <srai@redhat.com>
* fixes mon zonal deployment of 2 per zone to 1 if zones field is
specified.
* adds zones field doc to CRDs Documentation.
* add code for picking up failureDomainLabel if specified for zones
field in mon deployment
Signed-off-by: Deepika Upadhyay <deepika@koor.tech>
Mons that are configured on msgr1 will be using the v1 port
in the csi driver since only one port is allowed in the configmap.
To allow mons to be started on v2 while keeping quorum with the
v1 mons, the v1 mons must continue running on the v1 port.
So even if msgr2 is required, allow existing v1 mons to continue
running on that port.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
If the host network setting is changed in the cluster CR,
the existing mons must continue using the same network
settings or else the operator would update their pod
specs with the incorrect settings and cause mon quorum
to go down. Now the network setting is preserved so
the admin could switch between host and non-host network
configuration without reinstalling.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Introduce a new env variable ROOK_CSI_IMAGE_PULL_POLICY in rook operator configmap which should be used to
customize the imagePullPolicy for the csi driver and imagePullPolicy property in cephVersionSpec for ceph pods.
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
This removes double package imports. Example:
```
"github.com/rook/rook/pkg/apis/ceph.rook.io/v1"
cephv1 "github.com/rook/rook/pkg/apis/ceph.rook.io/v1"
```
Only one is now being used as shown in go-staticcheck ST1019
Signed-off-by: Alexander Trost <galexrt@googlemail.com>
msgr2 allows for encryption and/or compression across the wire.
Settings for enabling the encryption and compression are now
available in the cluster CR to that ceph will be automatically
configured with these settings when desired.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Allow specifying daemon startup probes where we also allow configuring
liveness probes. Startup probes allow Rook to tolerate when Ceph daemons
occasionally take a long time to start up while not also making
Kubernetes liveness probes slower to detect runtime failures of daemons.
Startup probes are beta in Kubernetes 1.18, so we should not enable
probes by default for earlier Kubernetes versions.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
Adding Recommended Labels on the resources created by rook
and using Recommended Labels in the helm chart,
for better visuals and management of k8s object
Closes: https://github.com/rook/rook/issues/8400
Signed-off-by: parth-gr <paarora@redhat.com>
This commit adds context parameter to k8sutil deployment functions. By
this, we can handle cancellation during API call of deployment resource.
Signed-off-by: Yuichiro Ueno <y1r.ueno@gmail.com>
correct typo in logging, it was showing `ok-to-stop`
instead of `ok-to-continue` when 'continueUpgradeAfterChecksEvenIfNotHealthy' is true
Co-Authored-by: Zeaone <zeaone@ZeaonedeMacBook-Pro.local>
Signed-off-by: subhamkrai <srai@redhat.com>
The mon daemon in a stretch cluster now can have its location set
as a CLI param instead of setting it with a separate command.
This enables mon failover to set the location of a mon immediately
when it is joining quorum instead of having a delayed command
to set the location.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The rook.io/v1 package was only an internal implementation detail and
does not have any CRDs that rely on it. The CRD deserialization should
handle the change in internal types without any issue. This separation
gives more flexibility for the storage providers to implement exactly
what is needed for their storage provider instead of forcing to use the
same types and risk affecting another storage provider.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
When a private docker registry is used and an image pull secret is specified in the chart, the pods with default Service Account fail to pull the image due to authentication issues.
Added rook-ceph-default service account and modify the pods specifications by adding the serviceAccountName.
Closes: https://github.com/rook/rook/issues/6673
Co-authored-by: Tareq Sharafy <tareq.sha@gmail.com>
Signed-off-by: parth-gr <partharora1010@gmail.com>
It's better to validate ownerReferences when setting them. In addition, we should use
controllerrutil.Set{Controller,Owner}Reference, that have such validation, as possible.
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
If there are multiple mgrs running, the operator will automatically
add pod antiaffinity across hosts by default. For stretch clusters,
antiaffinity across the stretch failure domain (e.g. zones)
will be added. Required antiaffinity will be created unless the
test setting allowMultiplePerNode is set to true.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The mgr daemon may be failed over by ceph if the active mgr is not
responding and the standby mgr is available. If the active mgr changes
the services for the dashboard and metrics will be updated with a
label selector for the new active mgr. The services cannot direct
traffic to the standby mgr or else they will be incorrectly redirected
to the active mgr.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
By setting the working directory of a Ceph daemon to the log direcor
(which is bindmounted to the host), we can ensure that the coredumps
will be available.
On CentOS 7 **only**, the kernel is configured with:
```
cat /proc/sys/kernel/core_pattern
core
```
which means that the coredumps will end up in the process working
directory and will be named "core".
On CentOS 8, everything is different and this won't work but still
CentOS will be able to consume this and changing the working directory
is harmless.
Signed-off-by: Sébastien Han <seb@redhat.com>
We can now collect logs directly into a side-car container.
A new CRD spec has been added:
spec:
logCollector:
enabled: true
periodicity: 24h
Every 24h we will rotate log files for each Ceph daemon.
Signed-off-by: Sébastien Han <seb@redhat.com>
Instead of directly defining our own constants, we should be using the k8s
constants for the well-known topology labels.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
In clusters where only two datacenters (or similar failure domains)
are available, a different mon and osd approach is needed to deal
with the network partitions or some other reason for one of the failure
domains going down. The Ceph stretched cluster makes the mons aware
of the failure domains by configuring one as the arbiter in a third
zone, while keeping two replicas of the data in each of the data
zoens.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The selector labels used for mons cannot change during upgrade
or else the upgrade will hang when the cluster is checking
for mon status immediately after the upgrade begins.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
When the mons are based on pvcs, the pvc_name label was not being set
as expected. Now the label will always be set on both the canary and
the daemon pods.
The mon pvcs were also never being deleted during canary cleanup. This
code is now removed since we must be very careful when the pvcs are
removed.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
This allows users to specify additional labels to be added to the components
created by the Rook Ceph operator.
Additionally the rbd mirrors are now getting their annotations and
labels set.
Signed-off-by: Alexander Trost <galexrt@googlemail.com>
The labels cannot be updated on a deployment match selector.
In v1.4 a new label was added for ceph_daemon_type that is intended
to be on the pod labels, but cannot be applied to the match selectors.
Therefore, we suppress any new labels that are added to the daemons.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Rename PodLabels function to CephDaemonAppLabels for more clarity about
what the function's purpose is.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
The mon was starting with a partial spec. There was a race condition
that was causing incorrect initialization in the health check goroutine
that could remove a mon upon operator restart. Now the mon will be
properly initialized with the full spec.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
To provide more context for executing commands in a ceph cluster,
the full clusterInfo is now passed to the ceph execution commands.
All information about the cluster will now be available throughout
all the areas of the operator. The namespace, ceph credentials,
mon endpoints, and other info is a core part of that cluster info.
Arguments passed through the controllers are also simplified for
mons, mgr, osds, and other daemons where the parameters had
become too complex.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
This commit allows us to configure status check for each daemon:
* "mon": health check on the ceph monitors (quorum)
* "osd": health check on the ceph osds
* "status": ceph health status check
Each check is controlled by the following settings:
* disabled: whether to disable the check (default: false)
* internal: interval to run the check
* timeout: only valid for mons, is the timeout for unresponsive mon
before failling over.
Example to disable the status health check:
```yaml
healthCheck:
daemonHealth:
status:
disabled: true
```
As part of that, pod's livenessprobe can now be configured via the
following settings:
* disabled: whether to enable or not
* probe: override the current probe in place by a new one
```yaml
healthCheck:
livenessProbe:
mon:
disabled: true
```
Closes: https://github.com/rook/rook/issues/5772
Signed-off-by: Sébastien Han <seb@redhat.com>
When integrating rook with istio, service port names need to be prefixed by their protocol.
Trying to fix this by removing static port names in favor of configurable names.
Closes: #5587
add protocol prefix to service port names, hardcoded as discussed. renamed container ports
Signed-off-by: Alex Flom <alexander.flom@gmail.com>
You can now use Rook along with Multus. Multus must be up and running
and the right ressources must exist such as NetworkAttachmentDefinition
CR.
The Cluster CR spec has new fields to work with multus:
network:
provider: multus (or 'host' for hostNetworking)
selectors:
public: NetworkAttachmentDefinition name
cluster: NetworkAttachmentDefinition name
If only a single NetworkAttachmentDefinition is provided Rook will use
both anyway for the Ceph traffic.
Please refer to the doc to learn more.
Closes: https://github.com/rook/rook/issues/4716
Signed-off-by: Sébastien Han <seb@redhat.com>
Blaine Gardner
Travis Nielsen
parth-gr
subhamkrai
Deepika Upadhyay
sp98
Avan Thakkar
Alexander Trost
parth-gr
Blaine Gardner
Yuichiro Ueno
Nitin Goyal
Lars Lehtonen
Satoru Takeuchi
Sébastien Han
Yusei Nishiyama
subhamkrai
Blaine Gardner
Alex Flom