This commit replaces `os.Setenv` with `t.Setenv` in tests. The
environment variable is automatically restored to its original value
when the test and all its subtests complete.
Reference: https://pkg.go.dev/testing#T.Setenv
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
Rook supports raw mode OSD in host-based cluster. So we can also
support OSD on logical volume in this kind of cluster.
Logical volumes aren't picked by filters (i.e. `useAllDevices: true`
and `device{Path,}Filter` to avoid unwanted LV consumption on upgrade.
Closes: https://github.com/rook/rook/issues/2047
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
With octopus coming to end of life, we remove support from
Rook for deploying Ceph Octopus and assume a min version of
Pacific v16. Any checks for octopus or earlier are removed
from the reconciles since they are obsolete.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Let it fail further once we run c-v activate. The error reported by LVM
could potentially ignored. If there is a failure, the c-v command will
fail anyways.
Signed-off-by: Sébastien Han <seb@redhat.com>
Return the error instead of hiding it, err could be non-nil for a lot of
reasons, eg: a canceled context or a non-existing object.
Signed-off-by: Sébastien Han <seb@redhat.com>
Once we are done cleaning up the content of the encrypted osd, let's
close the main LUKS device.
This avoids having dm devices on the system after the cleanup.
Closes: https://github.com/rook/rook/issues/10181
Signed-off-by: Sébastien Han <seb@redhat.com>
There is a corner case where an env variable populated with the prepare
job could influence and trigger some encryption code. We have an env
variable to discover whether the prepare job will encrypt a drive or not
so let's use that instead of running code as a noop and sometimes
trigger a corner case. In this scenario, the prepare job had an "IBM_"
env variable present in the OS image (IBM OS) so the setKEKinEnv()
function would get called for no reason.
Signed-off-by: Sébastien Han <seb@redhat.com>
This commits makes Rook usable on NixOS.
NixOS system holds "installed" binaries in `/run/current-system/sw/{bin,sbin}`.
The only binaries residing outside those paths are `/usr/bin/env` and `/bin/sh`.
I got a suggestion on Slack to simply add those NixOS specific paths
Signed-off-by: Krzysztof Nazarewski <3494992+nazarewk@users.noreply.github.com>
When kube authentication is used, let's clarify the error message and
hopefully lead to a better understanding on where to look for.
It could be misleading to print "token auth" when using kube auth,
although internally a token is used (the JWT from the Service
Account).
Signed-off-by: Sébastien Han <seb@redhat.com>
If a specific disk is configured to have more than one OSD per device,
or if it has a metadata device specified, exclude it from raw mode on an
individual basis.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
The osd destroy command leaves the osd id in use instead
of fully purging the osd id. If the osd id is not also
removed, it gives the impression that there is still
something that needs to be cleaned up from the old osd.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The purge job is intended to clean up all the resources
related to an OSD that is to be removed. An OSD with
a metadata, wal, and data PVC was only cleaning up the
data PVC. Now the metadata and wal PVCs will also be
cleaned up.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The OSD prepare job was succeeding even if the OSDs failed to be listed
by ceph-volume. These errors should be returned and fail the job, resulting
in a retry to retrieve the OSDs. Otherwise, the failure is masked and
the admin needs to dig through logs to find why the OSDs were not configured
on that node.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
If the KMS connection details for Vault have a VAULT_NAMESPACE we must
use it when building the vault client to auto-detect the KV internal
version.
Signed-off-by: Sébastien Han <seb@redhat.com>
We can eliminate the OSD creation code for Ceph < v15.2.0 because Nautilus(v14.2.z)
is no longer supported. In other words, we can now expect that all newly-created
OSDs on PVC are raw mode.
Please note that the existing lvm mode OSDs on PVCs still be supported after this change.
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
In volume.go, remove if condition in `callCephVolume`
method where we always pass false.
In exec.go, removing `ExecuteCommandWithFile*` methods
which was not used anywhere.
Signed-off-by: subhamkrai <srai@redhat.com>
For better consistency with IBM_KP_SERVICE_API_KEY and use
"ibmkeyprotect" as a provider name for more clarity.
Signed-off-by: Sébastien Han <seb@redhat.com>
The cluster-wide encryption feature now has a new supported key
management system: IBM Key protect. More information about the backend
can be found in Rook documentation under the
Today's implementation stores OSD encryption keys has "Standard" keys.
Signed-off-by: Sébastien Han <seb@redhat.com>
This is handling a tricky scenario where the OSD deployment is manually
removed and the OSD never reconvers. This is unlikely to happen, but
still OSD should be able to run after that action. Essentially after a
manual deletion, we need to run the prepare job again to re-hydrate the
OSD information so that the OSD deployment can be deployed.
On encryption, it is a little bit tricky since ceph-volume list again
the main block won't return anything, so we need to target the encrypted
block to list.
There is another case this PR does not handle, which is the removal of
the OSD deployment and then the node is restarted. This means that the
encrypted container is not opened anymore. However, opening it requires
more work like writing the key on the filesystem (if not coming from the
Kubernete secret, eg,. KMS vault) and then run luksOpen. This is an
extreme corner case probably not worth worrying about for now.
Signed-off-by: Sébastien Han <seb@redhat.com>
This commit adds context parameter to k8sutil kvstore functions. By
this, we can handle cancellation during API call of kvstore resource.
Signed-off-by: Yuichiro Ueno <y1r.ueno@gmail.com>
If multiple removal jobs are fired in parallel, there is a risk of
losing data since we will forcefully remove the OSD. It's also simply
true if a single OSD is not safe to destroy, there is also a risk of
data loss.
So now, we check if the OSD is safe-to-destroy first and then proceed.
The code waits forever and retries every minute unless the
--force-osd-removal flag is passed.
Signed-off-by: Sébastien Han <seb@redhat.com>
The cluster info is important context for the cluster controller to
create the cluster, and all the fields must be properly set.
A test cluster name was being set temporarily, resulting in
mons incorrectly getting the wrong cluster CR name. There is no
known issue from the temporary value, it was just exposed by
https://github.com/rook/rook/pull/8678 setting the value to a label.
Now the functions are more clearly named so only unit and
integration tests should be using the test value for the cluster
name where it is not important.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
From ceph v16.2.6 onwards the vault TLS suppport in RGW was added,
include similar changes for RGW.
Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
This commit adds context parameter to k8sutil job functions. By this, we
can handle cancellation during API call of job resource.
Signed-off-by: Yuichiro Ueno <y1r.ueno@gmail.com>
This commit adds context parameter to k8sutil deployment functions. By
this, we can handle cancellation during API call of deployment resource.
Signed-off-by: Yuichiro Ueno <y1r.ueno@gmail.com>
When a reconcile is started for OSDs, the prepare jobs are first
deleted from a previous reconcile. The timeout for the osd prepare
job deletion was only 40s. After that timeout, the reconcile attempts
to continue waiting for the pod, but of course will never complete
since the OSD prepare was not running in the first place, causing the
reconcile to wait indefinitely. In the reported issue, the osd prepare
jobs were actually deleted successfully, the timeout just wasn't long
enough. Pods need at least a minute to be forcefully deleted,
so we increase the timeout to 90s to give it some extra buffer.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The rook operator as well as the toolbox pod run with the "rook" user
with UID 2016. The UID was chosen based on the year of the initial
commit in the rook/rook repository.
No more root user running.
Closes: https://github.com/rook/rook/issues/8734
Signed-off-by: Sébastien Han <seb@redhat.com>
We can pass bound_service_account_names with a comma separated list of
service accounts. Let's do this instead of remapping new values.
Earlier, we thought a single service account could be added per Vault
role and we were using other variables like
`VAULT_AUTH_KUBERNETES_ROOK_OPERATOR_ROLE` that we were remapping to
`VAULT_AUTH_KUBERNETES_ROLE` internal for the API calls to Vault.
Signed-off-by: Sébastien Han <seb@redhat.com>
Rook cluster-wide encryption can now use the native Kubernetes
authentication to interact with vault KMS instead of using the token
method.
Signed-off-by: Sébastien Han <seb@redhat.com>
When deploying a cluster-wde encrypted cluster we now use the rook
binary to execute some code to fetch the key encryption key.
Using cURL all the time to fetch the key has its limitations. The
incoming integration with Kubernetes Authentication through service
accounts is leading the usage of cURL to its end.
The logic is really complex and prone to errors. Re-implementing the
Go logic into Bash is not a viable option.
Also, using the lib in a binary allows us to keep a consistent behavior
throughout the life cycle of our code.
Signed-off-by: Sébastien Han <seb@redhat.com>
When TLS is used and includes a caert, client key/cert, we need to copy
the content of the secret to a file in the operator's container
filesystem so that we can build the TLS config and thus the HTTP Client,
which reads those files.
Also, removing the files after each API call so they don't persist on
the filesystem forever.
Signed-off-by: Sébastien Han <seb@redhat.com>
Prior, we were returning a nil map and thus the assignment for forced
deletion was not working since we were trying to assign on a nil map.
Signed-off-by: Sébastien Han <seb@redhat.com>
This commit is a large refactor on how the operator starts, stops and
how it starts various sub-components such as the ceph-csi driver. It
also refines the way we cancel orchestrations. We don't use breakpoints
anymore but send our self a SIGUP to reload our controller runtime
manager.
The reload will happen under different circonstances like:
* a new adminission controller secret is created/deleted/changed
* a CephCluster CR is edited
As mentioned earlier, the csi driver now has its own controller, just
like flex. It reacts to change in the operator config map for particular
ROOK_CSI_ fields.
A second new controller for the operator's general config has been
created, it manages:
* the logging level
* the ceph CLI command timeout
* the discovery daemon
The operator reacts much more rapidly to cancellation events by stopping
the manager's context and reloading it.
Signed-off-by: Sébastien Han <seb@redhat.com>
The error message in UpdateNodeStatus regards the second argument
as node name. However, it is a PVC name in OSD on PVC.
Signed-off-by: Hiroya Onoe <onoehiroya@gmail.com>
Passing struct by value essentially gives you a copy, so when modified
within a function, the scope is then reduced to that function. Using
pointers solves that you mutate the struct as many times as you want from
anywhere.
As a result, the auto-detection of the Vault KV backend was not working
correctly.
Also, added a ton of unit tests for Vault.
Signed-off-by: Sébastien Han <seb@redhat.com>
Rook will now auto detect the kv version of the vault server. This
allows users not having to pass the VAULT_BACKEND configuration in the
CephCluster CR.
Signed-off-by: Sébastien Han <seb@redhat.com>
Eng Zer Jun
Travis Nielsen
Liang Zheng
Satoru Takeuchi
Josh Soref
Sébastien Han
subhamkrai
Krzysztof Nazarewski
Divyansh Kamboj
Blaine Gardner
Yuichiro Ueno
Jiffin Tony Thottan
Jonas Zeiger
Hiroya Onoe