For now, we must run the container with UID 0 and privileged for
multiple reasons:
* the rook binary writes ceph config to /var/lib/rook which is owned by
root
* it's difficult to use /etc/ceph since it will conflict with the
rook-ceph-override configmap AND is also owned by root since it's a
mounted configmap.
* using /etc/ceph might be possible but has other issues with rook's
exec package since the ceph config is built from /var/lib/rook
Closes: https://github.com/rook/rook/issues/9385
Signed-off-by: Sébastien Han <seb@redhat.com>
This commit adds context parameter to k8sutil pvc functions. By this, we
can handle cancellation during API call of pvc resource.
Signed-off-by: Yuichiro Ueno <y1r.ueno@gmail.com>
This commit adds context parameter to k8sutil kvstore functions. By
this, we can handle cancellation during API call of kvstore resource.
Signed-off-by: Yuichiro Ueno <y1r.ueno@gmail.com>
Adding Recommended Labels on the resources created by rook
and using Recommended Labels in the helm chart,
for better visuals and management of k8s object
Closes: https://github.com/rook/rook/issues/8400
Signed-off-by: parth-gr <paarora@redhat.com>
In K8s 1.22 there is a bug in the job name generation that
the job name is truncated an additional 10 characters. This can cause an issue
in the generated pod name if it then ends in a non-alphanumeric character. In that case,
we more aggressively generate a hashed job name.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Skipping upgrade checks was not being honored for OSDs.
Now the flag will be checked and allow the OSDs to be upgraded
without checking for the ok-to-stop condition.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
If we apply useAllNodes to false for the current deployment,
the OSDs should get updated with the individual nodes values and config,
The deviceClass was not updating to the existing OSDs because there was
bug in the check.
The check osdInfo.DeviceClass == "" which should be
checked like this osdInfo.DeviceClass == "None"
Updated the code so OSDs can make use of the devices present
Signed-off-by: parth-gr <paarora@redhat.com>
The cluster info is important context for the cluster controller to
create the cluster, and all the fields must be properly set.
A test cluster name was being set temporarily, resulting in
mons incorrectly getting the wrong cluster CR name. There is no
known issue from the temporary value, it was just exposed by
https://github.com/rook/rook/pull/8678 setting the value to a label.
Now the functions are more clearly named so only unit and
integration tests should be using the test value for the cluster
name where it is not important.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The blockdevmapper securityContext was changed to request a minimal set of
required capabilities for its operation and drop running as privileged.
While the base change works and is valid in terms of the container's copy operation,
it turns out that OpenShift may require some additional configuration not
currently covered by the limited securityContext and the capabilities granted.
To not break those OpenShift deployments, make the blkdevmapper securityContext
listen to the ROOK_HOSTPATH_REQUIRES_PRIVILEGED flag again to set privileged mode.
This flag is true on OpenShift deployments and running as privileged
works around the (missing) configuration problem for now.
To properly drop privileged completely some additional investigation needs
to be done on OpenShift deployments without relying on privileged execution.
Signed-off-by: Omar Pakker <Omar007@users.noreply.github.com>
From ceph v16.2.6 onwards the vault TLS suppport in RGW was added,
include similar changes for RGW.
Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
This commit adds context parameter to k8sutil node functions. By this,
we can handle cancellation during API call of node resource.
Signed-off-by: Yuichiro Ueno <y1r.ueno@gmail.com>
This commit adds context parameter to k8sutil job functions. By this, we
can handle cancellation during API call of job resource.
Signed-off-by: Yuichiro Ueno <y1r.ueno@gmail.com>
The OSD blkdevmapper init container relies on the MKNOD capability,
which it does not actually request.
As a result, deployments fail on Kubernetes clusters that do not
happen to assign this capability to all containers by default.
Solve this by updating the container spec securityContext to
explicitly request the capability it relies on.
Closes: https://github.com/rook/rook/issues/9156
Signed-off-by: Omar Pakker <Omar007@users.noreply.github.com>
This commit adds context parameter to k8sutil deployment functions. By
this, we can handle cancellation during API call of deployment resource.
Signed-off-by: Yuichiro Ueno <y1r.ueno@gmail.com>
When a reconcile is started for OSDs, the prepare jobs are first
deleted from a previous reconcile. The timeout for the osd prepare
job deletion was only 40s. After that timeout, the reconcile attempts
to continue waiting for the pod, but of course will never complete
since the OSD prepare was not running in the first place, causing the
reconcile to wait indefinitely. In the reported issue, the osd prepare
jobs were actually deleted successfully, the timeout just wasn't long
enough. Pods need at least a minute to be forcefully deleted,
so we increase the timeout to 90s to give it some extra buffer.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
This commit adds context parameter to utilities in opcontroller to
remove context.TODO use in opcontroller. By this, we can handle
cancellation of reconcilers in a fine-grained way.
Signed-off-by: Yuichiro Ueno <y1r.ueno@gmail.com>
Rook cluster-wide encryption can now use the native Kubernetes
authentication to interact with vault KMS instead of using the token
method.
Signed-off-by: Sébastien Han <seb@redhat.com>
When deploying a cluster-wde encrypted cluster we now use the rook
binary to execute some code to fetch the key encryption key.
Using cURL all the time to fetch the key has its limitations. The
incoming integration with Kubernetes Authentication through service
accounts is leading the usage of cURL to its end.
The logic is really complex and prone to errors. Re-implementing the
Go logic into Bash is not a viable option.
Also, using the lib in a binary allows us to keep a consistent behavior
throughout the life cycle of our code.
Signed-off-by: Sébastien Han <seb@redhat.com>
We don't need to use tini.
We don't have anything in the rook operator that would
either create zombie processes (no threads) or use
exec (to fork). The Go binary has a really good
signal handling mechanism.
Closes: https://github.com/rook/rook/issues/8794
Signed-off-by: Sébastien Han <seb@redhat.com>
The flex driver has been fully deprecated and thus removed from Rook.
Before upgrading to v1.8, users will need to convert existing flex volumes
from flex to csi volumes.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
This commit is a large refactor on how the operator starts, stops and
how it starts various sub-components such as the ceph-csi driver. It
also refines the way we cancel orchestrations. We don't use breakpoints
anymore but send our self a SIGUP to reload our controller runtime
manager.
The reload will happen under different circonstances like:
* a new adminission controller secret is created/deleted/changed
* a CephCluster CR is edited
As mentioned earlier, the csi driver now has its own controller, just
like flex. It reacts to change in the operator config map for particular
ROOK_CSI_ fields.
A second new controller for the operator's general config has been
created, it manages:
* the logging level
* the ceph CLI command timeout
* the discovery daemon
The operator reacts much more rapidly to cancellation events by stopping
the manager's context and reloading it.
Signed-off-by: Sébastien Han <seb@redhat.com>
The error message in UpdateNodeStatus regards the second argument
as node name. However, it is a PVC name in OSD on PVC.
Signed-off-by: Hiroya Onoe <onoehiroya@gmail.com>
earlier, `ApplyToPodSpec()` was only taking one toleration and ignoring
tolerations from `placement.ALL()`.
this commit merge toleration for Mgr,Mon,Osd pod
example, for osd it will merge
spec.placement.all and
storageDeviceClassSets.Placement(in case of pvc) or
spec.placement.osd(in case of non-pvc's).
Signed-off-by: subhamkrai <srai@redhat.com>
earlier, we had the issue for osd placement regarding
whether we should merge the placements or not.
Now, we have option `skipApplyAllPlacement` to
enable/disable osd placement. By default it is false
which means it will merge both placement.All() and
storageClassDeviceSets.Placement, when true
it will only apply storageClassDeviceSets.Placement.
Also, adding unit test.
Closes: https://github.com/rook/rook/issues/8135
Signed-off-by: subhamkrai <srai@redhat.com>
Both `ExecuteCommandWithOutputFileTimeout()` and
`ExecuteCommandWithOutputFile()` generate unnecessary system calls by
creating/reading/removing files where the stream output of the command
can simply be used. So sticking with `ExecuteCommandWithOutput()` and
`ExecuteCommandWithCombinedOutput()` for reading outputs is sufficient.
Closes: https://github.com/rook/rook/issues/8343
Signed-off-by: Sébastien Han <seb@redhat.com>
Recently, the builds of `ceph/ceph` image moved to quay.io, see
https://github.com/ceph/ceph-build/pull/1883 for more details.
Current images will remain but new builds will happen on quay.io only.
This means that tags such as `v14.2`, `v15.2`,`v16.2` will need to
switch to quay.io to get updates.
Signed-off-by: Sébastien Han <seb@redhat.com>
when osd pods removed, they are not able
to add back due to missing `ceph auth`
or different `ceph auth`.
this commit mounts the secret `rook-ceph-admin-keyring`
inside the osd activate initcontainer which has
admin keyring.
Signed-off-by: subhamkrai <srai@redhat.com>
When evaluating the potential "warning" key from the json output we were
not hiding stdout (1) from the output, so the key was leaked. Also the
warning check for the key was not perfect, it only checked for the
command to succeed, but we should introspect it since the "warning" key
is always present unlike the "error" one.
We now only print the warning or the error is present.
Signed-off-by: Sébastien Han <seb@redhat.com>
We have seen issues with fullchain.pem files containing one self-signed
certificate. At the same time, the failing configuration was not
providing any client certificates or keys.
The error from curl was `"self signed certificate in the chain"`.
The openssl binary is not available so we cannot even c_rehash the
certificate directory when using `--capath`, however this worked already
with a fully signed fullchain.pem along with a client certificate and
key. So we cannot only use `--cacert` which will break that use case.
This is becoming really obscur on why using `--cacert` solves this
problem. I just can't wait to use the `vault` binary, that's all I can
say...
Signed-off-by: Sébastien Han <seb@redhat.com>
The active osd container was always using /var/lib/rook for the
configuration of the OSD. If the dataDirHostPath is anything other
than the default, the config would show up in the wrong place.
Changing this back to the dataDirHostPath does not appear to affect
any upgrade scenarios as the activate container will simply start
extracting the config to the correct place.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Add ability to set device class for existing osd deployments,
based on its crush map device class.
Related-Issue: rook#8028
Signed-off-by: Denis Egorenko <degorenko@mirantis.com>
Implement the first step of `design/ceph/resource-dependencies.md` to
add dependency checking when deleting a CephCluster.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>