Implement CephX key rotation for Rook's client.admin user.
Admin user rotation is risky, so this has been tested extensively both
in unit tests as well as by manually injecting failures during runtime.
In testing, all failures were able to be recovered by the recovery
routine.
A mutex is also added to help ensure that two simultaneous admin key
rotation processes cannot be running simultaneously for any given
namespace. The mutex is tested in unit tests, and it was verified during
runtime via manual testing.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
Allow Rook to rotate CephX keys for Ceph v19.2.3 and up with the
exception of Ceph mons. While v19.2.3 has the `ceph auth rotate` command
that Rook needs to perform key rotation, there is still ongoing work in
Ceph to change how the `mon.` key is tracked internally. Rook's mon key
rotation requires those currently-unreleased Ceph changes.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
fixes a nil pointer dereference in GetProbeWithDefaults when
startupProbe is set in cephfilesystem and currentProbe is nil.
adds panic recovery in cephfilesystem reconcile loop to improve
error visibility and avoid silent reconcile restarts.
Signed-off-by: Oded Viner <oviner@redhat.com>
This is the first implementation of CephX key rotation in Rook.
Adds key rotation API from design PR 15915.
Also add helper methods for rotating keys, determining when keys
need to be rotated, and for generating CephX key statuses. These will be
reusable for other Rook reconciles beyond object/RGW.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
This change adds support for skipping reconciliation of CephNFS daemons
that are labeled with `ceph.rook.io/skip-reconcile=true`.
Similar to MDS, MGR, and RGW components, this allows cluster operators
to prevent Rook from modifying specific NFS daemon deployments.
Signed-off-by: Patryk Rostkowski <patrostkowski@gmail.com>
Implements #15460. If ObjectStore reading value from a secret
provided by user, the secret will be annotated with ObjectStore
name and ObjectStore will be reconciled on secret change.
Signed-off-by: Artem Torubarov <artem.torubarov@clyso.com>
Implements #14733. Allows to set IDs of external mons to
Cluster CRD. Rook will not remove external mons from quorum
and will add external mon addresses to mon endpoints.
Use-case for external mon is to maintain quorum for 2-AZ
k8s cluster in case of zone outage.
Signed-off-by: Artem Torubarov <artem.torubarov@clyso.com>
Given that Ceph Quincy (v17) is past end of life,
remove Quincy from the supported Ceph versions,
examples, and documentation.
Supported versions now include only Reef and Squid.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
we dont set removeMgrDaemonConfiguration to true unless all the
delete operations went successfull.
Signed-off-by: Redouane Kachach <rkachach@redhat.com>
This implements the "Ceph Config via Ceph Cluster CRD" design document
as a `cephConfig:` structure on the CRD.
This also fixes the `yq` commands used to manipulate the
`cluster-test.yaml` that caused CI issues for this PR and potentially
unknowingly others.
Signed-off-by: Alexander Trost <galexrt@googlemail.com>
Similar to the ceph crash collector daemon that generates a keyring with more
restrictive privileges, the exporter should also generate and use a more limited keyring.
Signed-off-by: avanthakkar <avanjohn@gmail.com>
The /tmp dir is not always available in some environments.
The only directories the operator should rely on
are the directories that are in its pod spec.
For assimilate-conf now we use the /var/lib/rook
directory to generate files since we know it is
a valid emptydir mounted into the operator pod.
Signed-off-by: travisn <tnielsen@redhat.com>
Change how Rook detects network CIDRs for Multus networks. The IPAM
configuration is only defined as an arbitrary string JSON blob with a
"type" field and nothing more. Rook's detection of CIDRs for whereabouts
had already grown out of date since the initial implementation.
Additionally, Rook did not support DHCP IPAM, which is a reasonable
choice for users. And more, Rook did not support CNI plugin chaining,
which further complicates NADs. Based on the CNI spec, network chaning
can result in any changes to network CIDRs from the first-given plugin.
All these problems make it more and more difficult for Rook to support
Multus by inspecting the NAD itself to predict network CIDRs. Instead,
it is better for Rook to treat the CNI process as a black box. To
preserve legacy functionality of auto-detecting networks and to make
that as robust as possible, change to a canary-style architecture like
that used for Ceph mons, from which Rook will detect the network CIDRs
if possible.
Also allow users to specify overrides for CIDR ranges. This allows Rook
to still support esoteric and unexpected NAD or network configurations
where a CIDR range is not detectable or where the range detected would
be incomplete. Because it may be impossible for Rook to understand the
network CIDRs wholistically while residing only on a portion of the
network, this feature should have been present from Multus's inception.
Improving CIDR auto-detection and allowing users to specify overrides
for auto-detected CIDRs rounds out Rook's Multus support for CephCluster
(core/RADOS) installations. No further architectural changes should be
needed for CephClusters as regards application of public/cluster network
CIDRs for Multus networks.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
enable flags with --default prefix for --log-to-stderr, --mon-cluster-log-to-stderr, --err-to-stderr, and --log-stderr-prefix
Signed-off-by: Javier <sjavierlopez@gmail.com>
if the configuration is already set in the ceph
database, using `ceph config assimilate-conf`
will add the key and value if its missing but
it wont update the value if the key is already
present, To fix this problem we can remove the
key and add all the configurations once again.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
Mons that are configured on msgr1 will be using the v1 port
in the csi driver since only one port is allowed in the configmap.
To allow mons to be started on v2 while keeping quorum with the
v1 mons, the v1 mons must continue running on the v1 port.
So even if msgr2 is required, allow existing v1 mons to continue
running on that port.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Based on the latest info from the Ceph RGW team, update all probes. Get
rid of the liveness probe entirely. Update startup and readiness probes
to support return codes for misconfiguration (500) and for server-side
throttling (498 or 503).
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
The monstore has a SetAll method for applying multiple settings
that was applying them individually. Now we use the assimilate
config option to make this more efficient.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
With octopus coming to end of life, we remove support from
Rook for deploying Ceph Octopus and assume a min version of
Pacific v16. Any checks for octopus or earlier are removed
from the reconciles since they are obsolete.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
In new deployments we always want to disable the insecure global id.
If the mgr deployment is newly created, we know it's a new cluster
and the mons are in quorum, so we can go ahead and disable the
insecure global IDs.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Ceph upstream telemetry (which Rook users could optionally enable via
Ceph CLI or Dashboard) does not easily identify if a cluster is managed
by Rook. Identifying Rook clusters is possible only if users also enable
the rook mgr module, which is optional and not often used.
To get better upstream insights into Ceph clusters managed by Rook, set
a new 'rook-version' field in Ceph's generic k/v store. This field can
be used to determine if a cluster is managed by Rook by its presence.
It additionally provides information about the Rook version used to
manage the cluster, which is another piece of potentially useful
information.
Regarding Ceph cluster identification via telemetry:
Ceph telemetry is anonymous and uses a different ID from the Ceph
cluster ID. Identifying the Rook version in telemetry does not add
directly identifying information. Therefore, there should be little
risk that adding the Rook version field will decrease anonymity of Ceph
clusters via telemetry.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
msgr2 allows for encryption and/or compression across the wire.
Settings for enabling the encryption and compression are now
available in the cluster CR to that ceph will be automatically
configured with these settings when desired.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Added unit test for probes changes and
did re-naming of liveness probes file
as it will contain multiple types of probes
Signed-off-by: parth-gr <paarora@redhat.com>
Instead of setting the default values in the Rook
we should use the defaults provided by the ceph
it self. This configuration is causing problem
for the rbd image creation, this is not allowing
the rbd images to have default image features
which comes with librbd.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
add annotations to secrets and configmaps to use
different tools to sync secrets and configmap.
Currently, these annotation is applied only on
the `rook-ceph-mon-endpoints` configmap and
the secrets `rook-ceph-mon` and `rook-ceph-admin-keyring`.
Closes: https://github.com/rook/rook/issues/8999
Signed-off-by: subhamkrai <srai@redhat.com>
Allow specifying daemon startup probes where we also allow configuring
liveness probes. Startup probes allow Rook to tolerate when Ceph daemons
occasionally take a long time to start up while not also making
Kubernetes liveness probes slower to detect runtime failures of daemons.
Startup probes are beta in Kubernetes 1.18, so we should not enable
probes by default for earlier Kubernetes versions.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
The cluster info is important context for the cluster controller to
create the cluster, and all the fields must be properly set.
A test cluster name was being set temporarily, resulting in
mons incorrectly getting the wrong cluster CR name. There is no
known issue from the temporary value, it was just exposed by
https://github.com/rook/rook/pull/8678 setting the value to a label.
Now the functions are more clearly named so only unit and
integration tests should be using the test value for the cluster
name where it is not important.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
rgw doesn't respond `livenessProbe` if the number of connection reaches its
limit (by default, 1000). Then rgw is out of service but still live.
Hense the current `livenessProbe` logic is suitiable for `readinessProbe`.
`tcpSocket` is enough for `livenessProbe`.
Closes: #8407
Signed-off-by: Yuzuki Mimura <yuzuki725.m@gmail.com>
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
The flex driver has been fully deprecated and thus removed from Rook.
Before upgrading to v1.8, users will need to convert existing flex volumes
from flex to csi volumes.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
This commit is a large refactor on how the operator starts, stops and
how it starts various sub-components such as the ceph-csi driver. It
also refines the way we cancel orchestrations. We don't use breakpoints
anymore but send our self a SIGUP to reload our controller runtime
manager.
The reload will happen under different circonstances like:
* a new adminission controller secret is created/deleted/changed
* a CephCluster CR is edited
As mentioned earlier, the csi driver now has its own controller, just
like flex. It reacts to change in the operator config map for particular
ROOK_CSI_ fields.
A second new controller for the operator's general config has been
created, it manages:
* the logging level
* the ceph CLI command timeout
* the discovery daemon
The operator reacts much more rapidly to cancellation events by stopping
the manager's context and reloading it.
Signed-off-by: Sébastien Han <seb@redhat.com>
The mon config had two different implementations that have evolved
over the lifetime of the project. This is a simple refactor to remove
the SetConfig() option and stick with the MonStore as a single
implementation for updating the mon store.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Both `ExecuteCommandWithOutputFileTimeout()` and
`ExecuteCommandWithOutputFile()` generate unnecessary system calls by
creating/reading/removing files where the stream output of the command
can simply be used. So sticking with `ExecuteCommandWithOutput()` and
`ExecuteCommandWithCombinedOutput()` for reading outputs is sufficient.
Closes: https://github.com/rook/rook/issues/8343
Signed-off-by: Sébastien Han <seb@redhat.com>
When the Operator set the livenessProbe of the POD, it overwrote the contents of the shared
desiredProbe. Therefore, the wrongly configured livenessProbe was applied to other PODs,
resulting in a CrashLoop. In this PR, I created another instance to avoid overwriting the
desiredProbe.
Closes: https://github.com/rook/rook/issues/8196
Signed-off-by: Yuji Ito <llamerada.jp@gmail.com>
Blaine Gardner
Oded Viner
Travis Nielsen
Patryk Rostkowski
Carlos Barria
Joshua Hoblitt
Artem Torubarov
Eng Zer Jun
ArtemandTravis Nielsen
Redouane Kachach
Alexander Trost
avanthakkar
Javier
Madhu Rajanna
Thomas Way
Blaine Gardner
subhamkrai
Divyansh Kamboj
parth-gr
Yuzuki Mimura
Sébastien Han
Yuji Ito