Commit Graph
1080 Commits
Author SHA1 Message Date
Travis Nielsen 427996a7c0 osd: increase wait timeout for osd prepare cleanup
When a reconcile is started for OSDs, the prepare jobs are first
deleted from a previous reconcile. The timeout for the osd prepare
job deletion was only 40s. After that timeout, the reconcile attempts
to continue waiting for the pod, but of course will never complete
since the OSD prepare was not running in the first place, causing the
reconcile to wait indefinitely. In the reported issue, the osd prepare
jobs were actually deleted successfully, the timeout just wasn't long
enough. Pods need at least a minute to be forcefully deleted,
so we increase the timeout to 90s to give it some extra buffer.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-11-05 14:20:43 -06:00
Travis Nielsen 6dc0cc54b9 Merge pull request #8426 from yuvalif/ceph-bucket-notifications
ceph: support bucket notifications for object storage
2021-11-04 17:35:08 -06:00
Yuval Lifshitz 71ed45b69b rgw: implement bucket notifications for object storage
following the design from here:
https://github.com/rook/rook/blob/master/design/ceph/object/ceph-bucket-notification-crd.md

Closes: https://github.com/rook/rook/issues/5313
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>
2021-11-04 11:20:40 +02:00
Travis Nielsen 0c6ed25c4e core: allow downgrade of all daemons consistently
In the event a ceph image is specified that is lower than the current running
version of the daemons, the downgrade is allowed, even if not technically
supported. All of the core daemons (mon,mgr,osd) were being downgraded,
but the daemons for other controllers (rgw,mds,rbdmirror) were not being
downgraded, resulting in an inconsistent cluster. Now we log that the downgrade
is not supported and all all of the daemons to be downgraded.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-11-03 17:03:39 -06:00
Sébastien Han bccc84efa2 mgr: do not set the balancer mode on pacific
On Pacific, Ceph's default is "upmap", so we should let it be
like this.
This lets the user change the mode is desired.
On Octopus though, Rook continues to force the mode to "upmap".

Closes: https://github.com/rook/rook/issues/9062
Signed-off-by: Sébastien Han <seb@redhat.com>
2021-10-29 15:43:29 +02:00
Travis Nielsen 33072bf2a7 Merge pull request #9041 from travisn/cluster-cleanup
core: Treat cluster as not existing if the cleanup policy is set
2021-10-27 14:06:35 -06:00
Travis Nielsen fd10d98dc6 core: treat cluster as not existing if the cleanup policy is set
The cluster CR can be forcefully deleted and cleanup the
cluster resources if the yes-really-destroy-data policy
is set on the CR. In this case, the other controllers should
treat the cluster CR as not existing and allow the finalizers
to be removed on those resources if they are requested for
deletion.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-10-27 10:25:06 -06:00
Sébastien Han 4f2c685ad9 ceph: ability to set label to crash collector
We can now set labels to the crash collector deployment by editing the
CephCluster CR with:

```yaml
spec:
  labels:
    crashcollector:
```

Closes: https://github.com/rook/rook/issues/9039
Signed-off-by: Sébastien Han <seb@redhat.com>
2021-10-27 12:29:50 +02:00
Sébastien Han 3b5bbf28d5 Merge pull request #9003 from leseb/vault-k8s-auth
osd: add support for k8s with vault kms
2021-10-25 18:02:48 +02:00
Blaine Gardner 2f850b6ae6 Merge pull request #8613 from subhamkrai/remove-nautilus
ceph: remove ceph nautilus, ceph octopus to default
2021-10-25 09:09:18 -06:00
Yuichiro Ueno 3fd86f83ae core: add context parameter to opcontroller
This commit adds context parameter to utilities in opcontroller to
remove context.TODO use in opcontroller. By this, we can handle
cancellation of reconcilers in a fine-grained way.

Signed-off-by: Yuichiro Ueno <y1r.ueno@gmail.com>
2021-10-25 20:45:06 +09:00
Sébastien Han 18a4047679 osd: add support for k8s with vault kms
Rook cluster-wide encryption can now use the native Kubernetes
authentication to interact with vault KMS instead of using the token
method.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-10-21 13:59:28 +02:00
subhamkrai 0150966024 ceph: remove ceph nautilus, ceph octopus to default
since rook 1.8, ceph nautilus no longer supported,
ceph octopus will be the minimum ceph version.

Closes: https://github.com/rook/rook/issues/7908
Signed-off-by: subhamkrai <srai@redhat.com>
2021-10-20 14:45:12 +05:30
Blaine Gardner 01e2feaef5 ceph: get rid of dynamic clientset
Stop using the dynamic clientset in favor of the controller-runtime
clientset.

Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
2021-10-19 17:05:38 -06:00
Sébastien Han 4017f94464 osd: use rook binary to fetch key encryption key
When deploying a cluster-wde encrypted cluster we now use the rook
binary to execute some code to fetch the key encryption key.

Using cURL all the time to fetch the key has its limitations. The
incoming integration with Kubernetes Authentication through service
accounts is leading the usage of cURL to its end.
The logic is really complex and prone to errors. Re-implementing the
Go logic into Bash is not a viable option.

Also, using the lib in a binary allows us to keep a consistent behavior
throughout the life cycle of our code.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-10-19 17:29:25 +02:00
Travis Nielsen 12689bd119 ceph: enable mon failover for the arbiter in stretch mode
Prior to ceph v16.2.7 the failover of the arbiter mon was
not supported. Now the new tiebreaker mon can be set during
the failover event and provide more dynamic stability to
the mon quorum if another node is available in the arbiter
zone.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-10-18 08:58:25 -06:00
Travis Nielsen c628edec85 Merge pull request #8939 from leseb/fix-8928
mon: run ceph commands to mon with timeout
2021-10-08 10:00:21 -06:00
Sébastien Han 8da68bfb78 mon: run ceph commands to mon with timeout
If the mons are not in quorum yet the commands interacting with mon
config store will stale for a very long time.

Closes: https://github.com/rook/rook/issues/8928
Signed-off-by: Sébastien Han <seb@redhat.com>
2021-10-08 11:57:16 +02:00
parth-gr 7c99858a77 ceph: add finalizers to rook-ceph-mon secrets and configmap
Adding finalizers to rook-ceph-mon secrets
and rook-ceph-mon-endpoints configmap
We don't want to delete this resources during disaster
because these details are needed during disaster recovery

Closes: https://github.com/rook/rook/issues/8369
Signed-off-by: parth-gr <paarora@redhat.com>
2021-10-07 19:44:17 +00:00
Sébastien Han 121c2987e3 ceph: stop using tini
We don't need to use tini.
We don't have anything in the rook operator that would
either create zombie processes (no threads) or use
exec (to fork). The Go binary has a really good
signal handling mechanism.

Closes: https://github.com/rook/rook/issues/8794
Signed-off-by: Sébastien Han <seb@redhat.com>
2021-09-27 10:54:16 +02:00
Travis Nielsen 0a0b9c98bd build: remove obsolete flex driver
The flex driver has been fully deprecated and thus removed from Rook.
Before upgrading to v1.8, users will need to convert existing flex volumes
from flex to csi volumes.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-09-23 16:17:20 -06:00
Sébastien Han 470fbfd341 Merge pull request #8743 from leseb/next-pacific
ceph: use next ceph v16.2.6 pacific version
2021-09-21 16:41:45 +02:00
Sébastien Han 0c33493f27 ceph: bump manifests to ceph pacific 16.2.6
New version is out so let's use it.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-09-21 09:16:32 +02:00
Juan Miguel Olmo Martínez 7fbd9f2225 ceph: do not use http for mgr liveness probe
When private/public network have been defined in the Ceph rook cluster it is not
possible to configure properly the ip address of the liveness probe
for the manager.
Changes in the manager in Pacific introduced this regression.

This change replaces the http probe by a command probe, avoiding thus to
determine what is going to be the ip address of the manager before launching
the pod.

fixes: https://github.com/rook/rook/issues/8510

Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
2021-09-20 13:28:51 +00:00
Sébastien Han b89730d895 ceph: refactor operator initialization sequence
This commit is a large refactor on how the operator starts, stops and
how it starts various sub-components such as the ceph-csi driver. It
also refines the way we cancel orchestrations. We don't use breakpoints
anymore but send our self a SIGUP to reload our controller runtime
manager.
The reload will happen under different circonstances like:

* a new adminission controller secret is created/deleted/changed
* a CephCluster CR is edited

As mentioned earlier, the csi driver now has its own controller, just
like flex. It reacts to change in the operator config map for particular
ROOK_CSI_ fields.

A second new controller for the operator's general config has been
created, it manages:

* the logging level
* the ceph CLI command timeout
* the discovery daemon

The operator reacts much more rapidly to cancellation events by stopping
the manager's context and reloading it.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-09-17 16:57:12 +02:00
Sébastien Han 031be4be19 Merge pull request #8675 from subhamkrai/ok-continue
ceph: modify the log info when ok to continue fails
2021-09-09 17:22:13 +02:00
subhamkraiandZeaone 0657804491 ceph: modify the log info when ok to continue fails
correct typo in logging, it was showing `ok-to-stop`
instead of `ok-to-continue` when 'continueUpgradeAfterChecksEvenIfNotHealthy' is true

Co-Authored-by: Zeaone <zeaone@ZeaonedeMacBook-Pro.local>
Signed-off-by: subhamkrai <srai@redhat.com>
2021-09-09 19:42:06 +05:30
JrCs c606f4c488 ceph: use node externalIP if no internalIP defined
In some cases node internalIP is not defined. Then use externalIP if it
exists.

Signed-off-by: JrCs <90z7oey02@sneakemail.com>
2021-09-08 16:26:05 +02:00
Travis Nielsen 1cb97574ea ceph: allow an even number of mons
While an even number of mons can cause lower availability of
mon quorum, it also can provide higher durability for the cluster.
Mon quorum can be restored from a single mon according to the
disaster recovery guide, so there may be scenarios where
an even number of mons may be preferable.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-09-03 07:43:32 -06:00
parth-gr e50d83771e ceph: add pdb for mgr
Currently PDB for MGR is not been created,
Creating the PDB for MGR if its count is 2

Closes: https://github.com/rook/rook/issues/8275
Signed-off-by: parth-gr <paarora@redhat.com>
2021-09-03 14:30:58 +05:30
Travis Nielsen 8af24cf6d1 Merge pull request #8629 from llamerada-jp/ceph-fix-error-message
ceph: fix error message in UpdateNodeStatus
2021-09-02 11:17:10 -06:00
Sébastien Han 072c8f43df ceph: do not reconcile if the op is not ready
When the op is initializing, the ceph config is not ready and thus we
should reconcile. This avoids a double reconcile.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-09-02 13:50:32 +02:00
Hiroya Onoe 2ff5413b75 ceph: fix error message in UpdateNodeStatus
The error message in UpdateNodeStatus regards the second argument
as node name. However, it is a PVC name in OSD on PVC.

Signed-off-by: Hiroya Onoe <onoehiroya@gmail.com>
2021-09-02 02:43:16 +00:00
Sébastien Han d4dd0577f9 Merge pull request #8529 from sp98/id-mapping
ceph: add ClusterID and PoolID mappings between local and peer cluster
2021-08-31 17:32:14 +02:00
Santosh Pillai 3f8abec403 ceph: add ClusterID and PoolID mappings between local and peer cluster
During disaster recovery/migration of a cluster, as part of the failover, the
kubernetes artifacts like deployment, PVC, PV, etc will be restored to a new
cluster by the admin. Even if the kubernetes objects are restored the
corresponding RBD/CephFS subvolume cannot be retrieved during CSI operations as
the clusterID and poolID are not the same in both clusters

This PR creates a mapping between Cluster ID and RBD Pool ID between
local cluster and peer cluster.

Signed-off-by: Santosh Pillai <sapillai@redhat.com>
2021-08-31 20:08:56 +05:30
Sébastien Han d675969567 ceph: fix vault kv secret engine auto-detection
Passing struct by value essentially gives you a copy, so when modified
within a function, the scope is then reduced to that function. Using
pointers solves that you mutate the struct as many times as you want from
anywhere.
As a result, the auto-detection of the Vault KV backend was not working
correctly.
Also, added a ton of unit tests for Vault.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-08-31 15:29:04 +02:00
Travis Nielsen 42717d2fca Merge pull request #8566 from subhamkrai/merge-tolerations
ceph: merge toleration for osd/prepareOSD pod if specified both places
2021-08-26 08:55:27 -06:00
Travis Nielsen b528310b79 Merge pull request #8590 from travisn/mon-store-config
ceph: Consolidate the calls to set mon config
2021-08-25 08:31:20 -06:00
Travis Nielsen edb8aa1834 Merge pull request #8584 from parth-gr/string-instance1
core: convert util.NewSet() to sets.NewString()
2021-08-25 08:30:24 -06:00
subhamkrai e1f232eeb7 ceph: merge toleration for osd/prepareOSD pod if specified both places
earlier, `ApplyToPodSpec()` was only taking one toleration and ignoring
tolerations from `placement.ALL()`.

this commit merge toleration for Mgr,Mon,Osd pod
example, for osd it will merge
spec.placement.all and
storageDeviceClassSets.Placement(in case of pvc) or
spec.placement.osd(in case of non-pvc's).

Signed-off-by: subhamkrai <srai@redhat.com>
2021-08-25 19:34:28 +05:30
parth-gr 77ba1ef11b core: convert util.NewSet() to sets.NewString()
Converting util.NewSet() instance to use sets.NewString() instance

Closes: https://github.com/rook/rook/issues/8479
Signed-off-by: parth-gr <paarora@redhat.com>
2021-08-25 13:13:05 +05:30
Travis Nielsen cdfe1982d1 ceph: consolidate the calls to set mon config
The mon config had two different implementations that have evolved
over the lifetime of the project. This is a simple refactor to remove
the SetConfig() option and stick with the MonStore as a single
implementation for updating the mon store.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-08-24 14:38:26 -06:00
Sébastien Han 656dd0f334 ceph: move the admission webhook to the operator
Our admission webhooks will now run as part of the Operator container
and not an additional deployment. This has the advantage of consuming
fewer resources in the cluster and not having to manage affinities and
tolerations. This only drawback is that the Secret containing the
certificates is not mounted anymore and the content needs to be written
inside the Operator. This is not practical since we also need to watch
for the Secret content to change. Meaning that the certificates have
been renewed and the webhook server needs to use them.
A new approach is on its way to hopefully simplify this last issue and
implement a watcher for the Secret.
In the meantime, users need to use the cert-manager or renew
certificates manually. Additionally, they must update the
ValidatingWebhookConfiguration object with the new CA bundle.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-08-24 19:07:04 +02:00
parth-gr 1601ba1c1c core: convert util.NewSet() to sets.NewString()
Converting util.NewSet() instance to use sets.NewString() instance

Closes: https://github.com/rook/rook/issues/8479
Signed-off-by: parth-gr <paarora@redhat.com>
2021-08-20 18:56:25 +05:30
Travis Nielsen f9f322683a ceph: refuse to failover the arbiter mon on stretch clusters
Stretch clusters in Ceph do not yet support failing over the arbiter
mon, so we now disable the arbiter mon from failing over. Soon
Ceph will support the arbiter mon failover and we will reenable
this failover feature.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-08-10 16:26:47 -06:00
Sébastien Han e4ac38e555 Merge pull request #8482 from leseb/restrict-manager-cache
ceph: move scheme initialization to the same place
2021-08-06 17:45:30 +02:00
Sébastien Han 2d55e69416 ceph: move scheme initialization to the same place
Let's initialize the schemes in a single place instead of doing it
when each controller initializes.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-08-06 11:33:10 +02:00
Sébastien Han 07d14a93a2 ceph: remove cli unused flags
I don't know why these flags are there but it's not like we run the
operator with them and with a different value.
So removing for clarity.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-08-06 11:05:20 +02:00
Travis Nielsen cc16b60f5f Merge pull request #8381 from subhamkrai/osd-placement-merge
ceph: add an option to enable/disable merge all placement
2021-08-04 07:23:33 -06:00
subhamkrai 9838313398 ceph: add an option to enable/disable merge all placement
earlier, we had the issue for osd placement regarding
whether we should merge the placements or not.

Now, we have option `skipApplyAllPlacement` to
enable/disable osd placement. By default it is false
which means it will merge both placement.All() and
storageClassDeviceSets.Placement, when true
it will only apply storageClassDeviceSets.Placement.

Also, adding unit test.

Closes: https://github.com/rook/rook/issues/8135
Signed-off-by: subhamkrai <srai@redhat.com>
2021-08-03 19:47:40 +05:30