Commit Graph
479 Commits
Author SHA1 Message Date
Sébastien Han 05506e7a68 core: reload go routine after CR is edited
Previously, the struct maintaining the list of cluster was still
initialized with a cluster item. Then the monitoring check will see that
the cluster is part of the struct already and thus won't run the
monitoring go routine again.
Now each time we cancel the context, we also remove the cluster item
from the map so that when the controller runs again, the monitoring
struct is re-populated and the go routine runs and statuses are updated.

Closes: https://github.com/rook/rook/issues/9911
Signed-off-by: Sébastien Han <seb@redhat.com>
2022-03-31 08:49:42 +02:00
Divyansh Kamboj 9008409f87 core: add context parameter to functions
This commit adds context parameter to various functions, and remove the
usage of context.TODO.

Closes: https://github.com/rook/rook/issues/8701
Signed-off-by: Divyansh Kamboj <dkamboj@redhat.com>
2022-03-22 08:19:07 +05:30
parth-gr 2dfd64a97c core: add observedGeneration to CR status
adding observedGeneration field in the cephcluster cr
status for having better control on reconciling,
as observedGeneration field will be updated by the controller

Closes: https://github.com/rook/rook/issues/9673

Signed-off-by: parth-gr <paarora@redhat.com>
2022-03-16 19:58:35 +05:30
Travis Nielsen b3ca0f1b87 osd: purge job will remove all pvcs for the osd
The purge job is intended to clean up all the resources
related to an OSD that is to be removed. An OSD with
a metadata, wal, and data PVC was only cleaning up the
data PVC. Now the metadata and wal PVCs will also be
cleaned up.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2022-02-24 12:05:15 -07:00
parth-gr 413662d475 core: change livenessProbe file name to probes
Added unit test for probes changes and
did re-naming of liveness probes file
as it will contain multiple types of probes

Signed-off-by: parth-gr <paarora@redhat.com>
2022-02-23 17:20:07 +05:30
parth-gr 1679af4a02 mds: override mds startup liveness probes from CephFilesystem CR
Rook code allows overriding the probes for mds from the CephCluster,
but this should really come from the CephFilesystem CR.

Closes: https://github.com/rook/rook/issues/9519

Signed-off-by: parth-gr <paarora@redhat.com>
2022-02-22 20:36:40 +05:30
Sébastien Han 5404ec13a2 core: dereference pointer before trying to compare with deepequal
Prior to this, we were comparing a pointer (the memory address) with a
struct. This was obviously always failing and returned false. We must
dereference the pointer to access the data contained at that memory
location.

Closes: https://github.com/rook/rook/issues/9544
Signed-off-by: Sébastien Han <seb@redhat.com>
2022-01-27 17:41:42 +01:00
Radoslaw Zarzynski 17ca41f557 osd: allow for injecting extra env. variables via ConfigMap
This patch brings a mechanism to define arbitrary environment
variables in OSD containers (both prepare and main ones).
It bases on idea proposed by Sebastien Han to use an optional
`ConfigMap` instance named `aook-ceph-daemon-env-override`
as the source of these settings.

The need for the patch comes from the fact that, although
crimson finally exposes the same CLI interface as the classical
OSD, the broadly used development builds have ASan built in.
As ASan, by default, complains if it isn't the very first loaded
DSO, we need a way to set the `ASAN_OPTIONS` environment variable
to `verify_asan_link_order=0` to mitigate the early aborts.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2022-01-26 15:17:27 +00:00
subhamkrai d87be97a50 osd: correct bluestore compression min blob size for ssd
Update bluestore compression min blob size for ssd to 8192.
It was typo which was added long time back.

Signed-off-by: subhamkrai <srai@redhat.com>
2022-01-17 15:13:15 +05:30
Sébastien Han a7acf490b6 osd: add ibm key protect kms
The cluster-wide encryption feature now has a new supported key
management system: IBM Key protect. More information about the backend
can be found in Rook documentation under the

Today's implementation stores OSD encryption keys has "Standard" keys.

Signed-off-by: Sébastien Han <seb@redhat.com>
2022-01-13 16:02:10 +01:00
Blaine Gardner 5b9ac16670 Merge pull request #9468 from BlaineEXE/startup-probes
core: rgw: allow specifying daemon startup probes
2022-01-04 11:03:12 -07:00
Blaine Gardner c07d89d9ea core: rgw: allow specifying daemon startup probes
Allow specifying daemon startup probes where we also allow configuring
liveness probes. Startup probes allow Rook to tolerate when Ceph daemons
occasionally take a long time to start up while not also making
Kubernetes liveness probes slower to detect runtime failures of daemons.

Startup probes are beta in Kubernetes 1.18, so we should not enable
probes by default for earlier Kubernetes versions.

Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
2021-12-21 15:12:41 -07:00
Sébastien Han 05775b068d osd: handle removal of encrypted osd deployment
This is handling a tricky scenario where the OSD deployment is manually
removed and the OSD never reconvers. This is unlikely to happen, but
still OSD should be able to run after that action. Essentially after a
manual deletion, we need to run the prepare job again to re-hydrate the
OSD information so that the OSD deployment can be deployed.
On encryption, it is a little bit tricky since ceph-volume list again
the main block won't return anything, so we need to target the encrypted
block to list.
There is another case this PR does not handle, which is the removal of
the OSD deployment and then the node is restarted. This means that the
encrypted container is not opened anymore. However, opening it requires
more work like writing the key on the filesystem (if not coming from the
Kubernete secret, eg,. KMS vault) and then run luksOpen. This is an
extreme corner case probably not worth worrying about for now.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-12-21 10:51:09 +01:00
Travis Nielsen 7a6a62f43b Merge pull request #9386 from leseb/mgr-active-side-car-permissions
mgr: run the watch side car config on /etc/ceph
2021-12-15 12:20:28 -07:00
Sébastien Han 5c1e459a6a mgr: run active-watch as root and privileged
For now, we must run the container with UID 0 and privileged for
multiple reasons:

* the rook binary writes ceph config to /var/lib/rook which is owned by
  root
* it's difficult to use /etc/ceph since it will conflict with the
  rook-ceph-override configmap AND is also owned by root since it's a
  mounted configmap.
* using /etc/ceph might be possible but has other issues with rook's
  exec package since the ceph config is built from /var/lib/rook

Closes: https://github.com/rook/rook/issues/9385
Signed-off-by: Sébastien Han <seb@redhat.com>
2021-12-13 14:02:33 +01:00
Sébastien Han 200fb569f4 Merge pull request #9395 from y1r/add-context-k8sutil-kvstore
core: add context parameter to k8sutil kvstore
2021-12-13 11:44:58 +01:00
Yuichiro Ueno 5352cf58a7 core: add context parameter to k8sutil pvc
This commit adds context parameter to k8sutil pvc functions. By this, we
can handle cancellation during API call of pvc resource.

Signed-off-by: Yuichiro Ueno <y1r.ueno@gmail.com>
2021-12-11 11:32:46 +09:00
Yuichiro Ueno 0d8c581aa9 core: add context parameter to k8sutil kvstore
This commit adds context parameter to k8sutil kvstore functions. By
this, we can handle cancellation during API call of kvstore resource.

Signed-off-by: Yuichiro Ueno <y1r.ueno@gmail.com>
2021-12-11 11:17:12 +09:00
Travis Nielsen c953032d09 Merge pull request #8678 from parth-gr/Recommended-labels
core: create rook resources with k8s recommended labels
2021-12-07 11:53:33 -07:00
parth-gr 0a86d26b2e core: create rook resources with k8s recommended labels
Adding Recommended Labels on the resources created by rook
    and using Recommended Labels in the helm chart,
    for better visuals and management of k8s object

Closes: https://github.com/rook/rook/issues/8400
Signed-off-by: parth-gr <paarora@redhat.com>
2021-12-07 18:16:44 +05:30
Travis Nielsen ee83ca74af osd: truncate osd prepare job names further
In K8s 1.22 there is a bug in the job name generation that
the job name is truncated an additional 10 characters. This can cause an issue
in the generated pod name if it then ends in a non-alphanumeric character. In that case,
we more aggressively generate a hashed job name.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-12-06 08:02:04 -07:00
Travis Nielsen 32a884ac18 osd: honor skipUpgradeChecks for osds
Skipping upgrade checks was not being honored for OSDs.
Now the flag will be checked and allow the OSDs to be upgraded
without checking for the ok-to-stop condition.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-12-01 14:41:53 -07:00
parth-gr cbe505d122 osd: update existing OSDs with deviceClass
If we apply useAllNodes to false for the current deployment,
the OSDs should get updated with the individual nodes values and config,
The deviceClass was not updating to the existing OSDs because there was
bug in the check.
The check osdInfo.DeviceClass == "" which should be
checked like this osdInfo.DeviceClass == "None"

Updated the code so OSDs can make use of the devices present

Signed-off-by: parth-gr <paarora@redhat.com>
2021-11-29 21:40:02 +05:30
Travis Nielsen 1afd322650 core: ensure cluster name is available on cluster info
The cluster info is important context for the cluster controller to
create the cluster, and all the fields must be properly set.
A test cluster name was being set temporarily, resulting in
mons incorrectly getting the wrong cluster CR name. There is no
known issue from the temporary value, it was just exposed by
https://github.com/rook/rook/pull/8678 setting the value to a label.

Now the functions are more clearly named so only unit and
integration tests should be using the test value for the cluster
name where it is not important.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-11-18 15:30:26 -07:00
Blaine Gardner cd832e5832 Merge pull request #8579 from thotz/vaultsslsupport
ceph: add support in RGW to communicate vault with TLS
2021-11-18 12:32:38 -07:00
Omar Pakker 8f9055809f osd: add privileged support (back) to blkdevmapper securityContext (work-around)
The blockdevmapper securityContext was changed to request a minimal set of
required capabilities for its operation and drop running as privileged.
While the base change works and is valid in terms of the container's copy operation,
it turns out that OpenShift may require some additional configuration not
currently covered by the limited securityContext and the capabilities granted.

To not break those OpenShift deployments, make the blkdevmapper securityContext
listen to the ROOK_HOSTPATH_REQUIRES_PRIVILEGED flag again to set privileged mode.
This flag is true on OpenShift deployments and running as privileged
works around the (missing) configuration problem for now.
To properly drop privileged completely some additional investigation needs
to be done on OpenShift deployments without relying on privileged execution.

Signed-off-by: Omar Pakker <Omar007@users.noreply.github.com>
2021-11-17 12:25:13 +01:00
Jiffin Tony Thottan aba50d3ca9 object: add support in RGW to communicate vault with TLS
From ceph v16.2.6 onwards the vault TLS suppport in RGW was added,
include similar changes for RGW.

Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
2021-11-17 10:19:28 +05:30
Sébastien Han c5783a77cf Merge pull request #9163 from y1r/add-context-k8sutil-node
core: add context parameter to k8sutil node
2021-11-15 16:25:13 +01:00
Travis Nielsen 716baf7785 Merge pull request #9158 from Omar007/fix/blkdevmapper-capabilities
osd: set blkdevmapper capabilities
2021-11-15 08:05:12 -07:00
Yuichiro Ueno 4cc716a7ca core: add context parameter to k8sutil node
This commit adds context parameter to k8sutil node functions. By this,
we can handle cancellation during API call of node resource.

Signed-off-by: Yuichiro Ueno <y1r.ueno@gmail.com>
2021-11-15 22:45:59 +09:00
Yuichiro Ueno 3799542356 core: add context parameter to k8sutil job
This commit adds context parameter to k8sutil job functions. By this, we
can handle cancellation during API call of job resource.

Signed-off-by: Yuichiro Ueno <y1r.ueno@gmail.com>
2021-11-15 22:39:08 +09:00
Omar Pakker 4726d39688 osd: set blkdevmapper capabilities
The OSD blkdevmapper init container relies on the MKNOD capability,
which it does not actually request.
As a result, deployments fail on Kubernetes clusters that do not
happen to assign this capability to all containers by default.
Solve this by updating the container spec securityContext to
explicitly request the capability it relies on.

Closes: https://github.com/rook/rook/issues/9156
Signed-off-by: Omar Pakker <Omar007@users.noreply.github.com>
2021-11-15 13:11:19 +01:00
Yuichiro Ueno 0b575703c7 core: add context parameter to k8sutil deployment
This commit adds context parameter to k8sutil deployment functions. By
this, we can handle cancellation during API call of deployment resource.

Signed-off-by: Yuichiro Ueno <y1r.ueno@gmail.com>
2021-11-13 14:58:13 +09:00
Travis Nielsen 427996a7c0 osd: increase wait timeout for osd prepare cleanup
When a reconcile is started for OSDs, the prepare jobs are first
deleted from a previous reconcile. The timeout for the osd prepare
job deletion was only 40s. After that timeout, the reconcile attempts
to continue waiting for the pod, but of course will never complete
since the OSD prepare was not running in the first place, causing the
reconcile to wait indefinitely. In the reported issue, the osd prepare
jobs were actually deleted successfully, the timeout just wasn't long
enough. Pods need at least a minute to be forcefully deleted,
so we increase the timeout to 90s to give it some extra buffer.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-11-05 14:20:43 -06:00
Sébastien Han 3b5bbf28d5 Merge pull request #9003 from leseb/vault-k8s-auth
osd: add support for k8s with vault kms
2021-10-25 18:02:48 +02:00
Blaine Gardner 2f850b6ae6 Merge pull request #8613 from subhamkrai/remove-nautilus
ceph: remove ceph nautilus, ceph octopus to default
2021-10-25 09:09:18 -06:00
Yuichiro Ueno 3fd86f83ae core: add context parameter to opcontroller
This commit adds context parameter to utilities in opcontroller to
remove context.TODO use in opcontroller. By this, we can handle
cancellation of reconcilers in a fine-grained way.

Signed-off-by: Yuichiro Ueno <y1r.ueno@gmail.com>
2021-10-25 20:45:06 +09:00
Sébastien Han 18a4047679 osd: add support for k8s with vault kms
Rook cluster-wide encryption can now use the native Kubernetes
authentication to interact with vault KMS instead of using the token
method.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-10-21 13:59:28 +02:00
subhamkrai 0150966024 ceph: remove ceph nautilus, ceph octopus to default
since rook 1.8, ceph nautilus no longer supported,
ceph octopus will be the minimum ceph version.

Closes: https://github.com/rook/rook/issues/7908
Signed-off-by: subhamkrai <srai@redhat.com>
2021-10-20 14:45:12 +05:30
Sébastien Han 4017f94464 osd: use rook binary to fetch key encryption key
When deploying a cluster-wde encrypted cluster we now use the rook
binary to execute some code to fetch the key encryption key.

Using cURL all the time to fetch the key has its limitations. The
incoming integration with Kubernetes Authentication through service
accounts is leading the usage of cURL to its end.
The logic is really complex and prone to errors. Re-implementing the
Go logic into Bash is not a viable option.

Also, using the lib in a binary allows us to keep a consistent behavior
throughout the life cycle of our code.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-10-19 17:29:25 +02:00
Sébastien Han 121c2987e3 ceph: stop using tini
We don't need to use tini.
We don't have anything in the rook operator that would
either create zombie processes (no threads) or use
exec (to fork). The Go binary has a really good
signal handling mechanism.

Closes: https://github.com/rook/rook/issues/8794
Signed-off-by: Sébastien Han <seb@redhat.com>
2021-09-27 10:54:16 +02:00
Travis Nielsen 0a0b9c98bd build: remove obsolete flex driver
The flex driver has been fully deprecated and thus removed from Rook.
Before upgrading to v1.8, users will need to convert existing flex volumes
from flex to csi volumes.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
2021-09-23 16:17:20 -06:00
Sébastien Han b89730d895 ceph: refactor operator initialization sequence
This commit is a large refactor on how the operator starts, stops and
how it starts various sub-components such as the ceph-csi driver. It
also refines the way we cancel orchestrations. We don't use breakpoints
anymore but send our self a SIGUP to reload our controller runtime
manager.
The reload will happen under different circonstances like:

* a new adminission controller secret is created/deleted/changed
* a CephCluster CR is edited

As mentioned earlier, the csi driver now has its own controller, just
like flex. It reacts to change in the operator config map for particular
ROOK_CSI_ fields.

A second new controller for the operator's general config has been
created, it manages:

* the logging level
* the ceph CLI command timeout
* the discovery daemon

The operator reacts much more rapidly to cancellation events by stopping
the manager's context and reloading it.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-09-17 16:57:12 +02:00
Hiroya Onoe 2ff5413b75 ceph: fix error message in UpdateNodeStatus
The error message in UpdateNodeStatus regards the second argument
as node name. However, it is a PVC name in OSD on PVC.

Signed-off-by: Hiroya Onoe <onoehiroya@gmail.com>
2021-09-02 02:43:16 +00:00
Travis Nielsen 42717d2fca Merge pull request #8566 from subhamkrai/merge-tolerations
ceph: merge toleration for osd/prepareOSD pod if specified both places
2021-08-26 08:55:27 -06:00
subhamkrai e1f232eeb7 ceph: merge toleration for osd/prepareOSD pod if specified both places
earlier, `ApplyToPodSpec()` was only taking one toleration and ignoring
tolerations from `placement.ALL()`.

this commit merge toleration for Mgr,Mon,Osd pod
example, for osd it will merge
spec.placement.all and
storageDeviceClassSets.Placement(in case of pvc) or
spec.placement.osd(in case of non-pvc's).

Signed-off-by: subhamkrai <srai@redhat.com>
2021-08-25 19:34:28 +05:30
parth-gr 77ba1ef11b core: convert util.NewSet() to sets.NewString()
Converting util.NewSet() instance to use sets.NewString() instance

Closes: https://github.com/rook/rook/issues/8479
Signed-off-by: parth-gr <paarora@redhat.com>
2021-08-25 13:13:05 +05:30
parth-gr 1601ba1c1c core: convert util.NewSet() to sets.NewString()
Converting util.NewSet() instance to use sets.NewString() instance

Closes: https://github.com/rook/rook/issues/8479
Signed-off-by: parth-gr <paarora@redhat.com>
2021-08-20 18:56:25 +05:30
subhamkrai 9838313398 ceph: add an option to enable/disable merge all placement
earlier, we had the issue for osd placement regarding
whether we should merge the placements or not.

Now, we have option `skipApplyAllPlacement` to
enable/disable osd placement. By default it is false
which means it will merge both placement.All() and
storageClassDeviceSets.Placement, when true
it will only apply storageClassDeviceSets.Placement.

Also, adding unit test.

Closes: https://github.com/rook/rook/issues/8135
Signed-off-by: subhamkrai <srai@redhat.com>
2021-08-03 19:47:40 +05:30
Sébastien Han d55fb8d13b Merge pull request #8351 from leseb/refact-exec-helpers
ceph: remove unnecessary exec helpers
2021-07-23 18:12:57 +02:00