If the KMS connection details for Vault have a VAULT_NAMESPACE we must
use it when building the vault client to auto-detect the KV internal
version.
Signed-off-by: Sébastien Han <seb@redhat.com>
The reconcile was skipping updating most pool properties for
object stores. The implementation of pools between the file,
object, and pool controllers had some duplicate code, so
this change also factors out the common code for better
reuse in a single place. Anytime a pool is created or updated,
it will now consistently update all the pool properties
that are expected to be modifiable.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
We can eliminate the OSD creation code for Ceph < v15.2.0 because Nautilus(v14.2.z)
is no longer supported. In other words, we can now expect that all newly-created
OSDs on PVC are raw mode.
Please note that the existing lvm mode OSDs on PVCs still be supported after this change.
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
If the cluster is external, users might still want to take advantage of
the ceph-csi configmap configuration for ceph filesystem subvolume
group. This configuration is handled by the controller whenever a
cephFilesystemSubVolumeGroup CR is created or deleted.
If the present cluster CephCluster is external, the controller
(cephFilesystemSubVolumeGroup) won't handle creation nor deletion of sub volume groups.
It just assumes that the given name is correct and the controller goes ahead with the
ceph-csi configuration.
Closes: #9551
Signed-off-by: Sébastien Han <seb@redhat.com>
In volume.go, remove if condition in `callCephVolume`
method where we always pass false.
In exec.go, removing `ExecuteCommandWithFile*` methods
which was not used anywhere.
Signed-off-by: subhamkrai <srai@redhat.com>
It's easier to read the config if there is a new line before instead
having a truncated version like:
```
2022-01-26 06:39:41.421373 D | cephclient: config file @ /etc/ceph/ceph.conf: [global]
fsid = 0650a1d5-d688-4575-97fc-7b3052a8a3dd
mon initial members = a
mon host = [v2:10.104.192.209:3300,v1:10.104.192.209:6789]
[client.admin]
keyring = /var/lib/rook/rook-ceph/client.admin.keyring
```
Now we have:
```
2022-01-26 06:39:41.421373 D | cephclient: config file @ /etc/ceph/ceph.conf:
[global]
fsid = 0650a1d5-d688-4575-97fc-7b3052a8a3dd
mon initial members = a
mon host = [v2:10.104.192.209:3300,v1:10.104.192.209:6789]
[client.admin]
keyring = /var/lib/rook/rook-ceph/client.admin.keyring
```
Signed-off-by: Sébastien Han <seb@redhat.com>
If we fail fetching the ceph version let's also print the stdout since
ceph tends to put the error here too.
Signed-off-by: Sébastien Han <seb@redhat.com>
If the pool application is already set, skip setting it again
to avoid a warning message being logged that it is already
set.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
For better consistency with IBM_KP_SERVICE_API_KEY and use
"ibmkeyprotect" as a provider name for more clarity.
Signed-off-by: Sébastien Han <seb@redhat.com>
The cluster-wide encryption feature now has a new supported key
management system: IBM Key protect. More information about the backend
can be found in Rook documentation under the
Today's implementation stores OSD encryption keys has "Standard" keys.
Signed-off-by: Sébastien Han <seb@redhat.com>
This introduces a new CRD to add the ability to create subvolumegroup
for a given ceph filesystem volume. Typically the name of the volume is
the name of the filesystem created by rook.
Closes: https://github.com/rook/rook/issues/7036
Signed-off-by: Sébastien Han <seb@redhat.com>
This is handling a tricky scenario where the OSD deployment is manually
removed and the OSD never reconvers. This is unlikely to happen, but
still OSD should be able to run after that action. Essentially after a
manual deletion, we need to run the prepare job again to re-hydrate the
OSD information so that the OSD deployment can be deployed.
On encryption, it is a little bit tricky since ceph-volume list again
the main block won't return anything, so we need to target the encrypted
block to list.
There is another case this PR does not handle, which is the removal of
the OSD deployment and then the node is restarted. This means that the
encrypted container is not opened anymore. However, opening it requires
more work like writing the key on the filesystem (if not coming from the
Kubernete secret, eg,. KMS vault) and then run luksOpen. This is an
extreme corner case probably not worth worrying about for now.
Signed-off-by: Sébastien Han <seb@redhat.com>
The built-in pools device_health_metrics and .nfs created by ceph
need to be configured for replicas, failure domain, etc.
To support this, we allow the pool to be created as a CR.
Since K8s does not support underscores in the resource names
the operator must translate this special pool name into
the name expected by ceph.
This also sets the basis for allowing filesystem data
pools to specify the desired pool name instead of requiring
a generated name.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
This commit adds context parameter to k8sutil kvstore functions. By
this, we can handle cancellation during API call of kvstore resource.
Signed-off-by: Yuichiro Ueno <y1r.ueno@gmail.com>
If cephVersion>= 16.2.7, we run
ceph fs snapshot mirror daemon status
otherwise we run
ceph fs snapshot mirror daemon status <fs_name>
Updating the cmd so both the ceph version can run successfully
Closes: https://github.com/rook/rook/issues/8458
Signed-off-by: parth-gr <paarora@redhat.com>
The failure domain is baked into the crush rule that is
created for a pool. To allow for an updated failure domain
on the pool, create a new crush rule specific for that
failure domain and update the pool with the new rule.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The failover of the arbiter mon in a stretch cluster was sometimes
failing due to the new tiebreaker not being set in ceph.
Rook would repeatedly try to remove the old tiebreaker mon
and keep failing because the new tiebreaker had not been set.
Now we make setting the tiebreaker idempotent in case the operator
restarts in the middle of the operation or some other corner
case causes the expected tiebreaker to be set. In that case,
the next reconcile will also ensure the tiebreaker mon is
set as expected.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Sometimes Ceph uses a different standard output to return errors or
merges standard error to standard out. So let's allow some commands to
return both in the output.
Signed-off-by: Sébastien Han <seb@redhat.com>
Some users have reported issues while adding the token, this is not
always reproducable so perhaps it's a typo when importing the token and
adding trailing spaces.
Closes: https://github.com/rook/rook/issues/9151
Signed-off-by: Sébastien Han <seb@redhat.com>
If multiple removal jobs are fired in parallel, there is a risk of
losing data since we will forcefully remove the OSD. It's also simply
true if a single OSD is not safe to destroy, there is also a risk of
data loss.
So now, we check if the OSD is safe-to-destroy first and then proceed.
The code waits forever and retries every minute unless the
--force-osd-removal flag is passed.
Signed-off-by: Sébastien Han <seb@redhat.com>
For CRD not using the new nfs spec that includes the pool settings,
applying the "size" property won't work since it is set to 0. The pool
still gets created but returns an error. The loop is re-queued but on
the second run the pool is detected so no further configuration is done.
Closes: https://github.com/rook/rook/issues/9205
Signed-off-by: Sébastien Han <seb@redhat.com>
The cluster info is important context for the cluster controller to
create the cluster, and all the fields must be properly set.
A test cluster name was being set temporarily, resulting in
mons incorrectly getting the wrong cluster CR name. There is no
known issue from the temporary value, it was just exposed by
https://github.com/rook/rook/pull/8678 setting the value to a label.
Now the functions are more clearly named so only unit and
integration tests should be using the test value for the cluster
name where it is not important.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
From ceph v16.2.6 onwards the vault TLS suppport in RGW was added,
include similar changes for RGW.
Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
This commit adds context parameter to k8sutil job functions. By this, we
can handle cancellation during API call of job resource.
Signed-off-by: Yuichiro Ueno <y1r.ueno@gmail.com>
This commit adds context parameter to k8sutil pod functions. By this, we
can handle cancellation during API call of pod resource.
Signed-off-by: Yuichiro Ueno <y1r.ueno@gmail.com>
This commit adds context parameter to k8sutil deployment functions. By
this, we can handle cancellation during API call of deployment resource.
Signed-off-by: Yuichiro Ueno <y1r.ueno@gmail.com>
When a reconcile is started for OSDs, the prepare jobs are first
deleted from a previous reconcile. The timeout for the osd prepare
job deletion was only 40s. After that timeout, the reconcile attempts
to continue waiting for the pod, but of course will never complete
since the OSD prepare was not running in the first place, causing the
reconcile to wait indefinitely. In the reported issue, the osd prepare
jobs were actually deleted successfully, the timeout just wasn't long
enough. Pods need at least a minute to be forcefully deleted,
so we increase the timeout to 90s to give it some extra buffer.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The rook operator as well as the toolbox pod run with the "rook" user
with UID 2016. The UID was chosen based on the year of the initial
commit in the rook/rook repository.
No more root user running.
Closes: https://github.com/rook/rook/issues/8734
Signed-off-by: Sébastien Han <seb@redhat.com>
We can pass bound_service_account_names with a comma separated list of
service accounts. Let's do this instead of remapping new values.
Earlier, we thought a single service account could be added per Vault
role and we were using other variables like
`VAULT_AUTH_KUBERNETES_ROOK_OPERATOR_ROLE` that we were remapping to
`VAULT_AUTH_KUBERNETES_ROLE` internal for the API calls to Vault.
Signed-off-by: Sébastien Han <seb@redhat.com>
The monitor list was not sorted, so each time we were reconciling, the
peer secret token will see its content updated with randomized
monitors. This would enter our predicate and trigger a reconcile.
Potentially an endless one, if the randomized list is already different.
Closes: https://github.com/rook/rook/issues/9076
Signed-off-by: Sébastien Han <seb@redhat.com>
Rook cluster-wide encryption can now use the native Kubernetes
authentication to interact with vault KMS instead of using the token
method.
Signed-off-by: Sébastien Han <seb@redhat.com>
When deploying a cluster-wde encrypted cluster we now use the rook
binary to execute some code to fetch the key encryption key.
Using cURL all the time to fetch the key has its limitations. The
incoming integration with Kubernetes Authentication through service
accounts is leading the usage of cURL to its end.
The logic is really complex and prone to errors. Re-implementing the
Go logic into Bash is not a viable option.
Also, using the lib in a binary allows us to keep a consistent behavior
throughout the life cycle of our code.
Signed-off-by: Sébastien Han <seb@redhat.com>
Previously we were merging the stderr even if it was empty, leading to
unmarshall errors.
The error simulation was done here
https://play.golang.org/p/Sk2yw9GUWNu.
Signed-off-by: Sébastien Han <seb@redhat.com>
Prior to ceph v16.2.7 the failover of the arbiter mon was
not supported. Now the new tiebreaker mon can be set during
the failover event and provide more dynamic stability to
the mon quorum if another node is available in the arbiter
zone.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
When TLS is used and includes a caert, client key/cert, we need to copy
the content of the secret to a file in the operator's container
filesystem so that we can build the TLS config and thus the HTTP Client,
which reads those files.
Also, removing the files after each API call so they don't persist on
the filesystem forever.
Signed-off-by: Sébastien Han <seb@redhat.com>
Using a default value for CompressionMode to none effectively overrides
any values for Parameters. It is deprecated but still takes precedence.
Which means that in its previous form, Parameters was always ignored
since CompressionMode was always set to none when empty.
Signed-off-by: Sébastien Han <seb@redhat.com>
Create a new log level for Rook that is hidden from users. This is the
most verbose log level, and it is the level developers would like to use
to get debug logs that are important for debugging but that could leak
senstivie information like credentials in production use.
If a user sets their debug level to "TRACE", they will merely get
"DEBUG" level logs. Only if they set "TRACE_INSECURE" will they get
trace logs, and those are likely to include insecure information. Rook
tries very hard not to leak sensitive information in logs even with
verbose "DEBUG" logs.
Resolves#8778
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
Sébastien Han
Travis Nielsen
Satoru Takeuchi
subhamkrai
Eng Zer Jun
Blaine Gardner
Yuichiro Ueno
parth-gr
Jiffin Tony Thottan