In K8s 1.22 there is a bug in the job name generation that
the job name is truncated an additional 10 characters. This can cause an issue
in the generated pod name if it then ends in a non-alphanumeric character. In that case,
we more aggressively generate a hashed job name.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Skipping upgrade checks was not being honored for OSDs.
Now the flag will be checked and allow the OSDs to be upgraded
without checking for the ok-to-stop condition.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
If we apply useAllNodes to false for the current deployment,
the OSDs should get updated with the individual nodes values and config,
The deviceClass was not updating to the existing OSDs because there was
bug in the check.
The check osdInfo.DeviceClass == "" which should be
checked like this osdInfo.DeviceClass == "None"
Updated the code so OSDs can make use of the devices present
Signed-off-by: parth-gr <paarora@redhat.com>
The cluster info is important context for the cluster controller to
create the cluster, and all the fields must be properly set.
A test cluster name was being set temporarily, resulting in
mons incorrectly getting the wrong cluster CR name. There is no
known issue from the temporary value, it was just exposed by
https://github.com/rook/rook/pull/8678 setting the value to a label.
Now the functions are more clearly named so only unit and
integration tests should be using the test value for the cluster
name where it is not important.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The blockdevmapper securityContext was changed to request a minimal set of
required capabilities for its operation and drop running as privileged.
While the base change works and is valid in terms of the container's copy operation,
it turns out that OpenShift may require some additional configuration not
currently covered by the limited securityContext and the capabilities granted.
To not break those OpenShift deployments, make the blkdevmapper securityContext
listen to the ROOK_HOSTPATH_REQUIRES_PRIVILEGED flag again to set privileged mode.
This flag is true on OpenShift deployments and running as privileged
works around the (missing) configuration problem for now.
To properly drop privileged completely some additional investigation needs
to be done on OpenShift deployments without relying on privileged execution.
Signed-off-by: Omar Pakker <Omar007@users.noreply.github.com>
From ceph v16.2.6 onwards the vault TLS suppport in RGW was added,
include similar changes for RGW.
Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
This commit adds context parameter to k8sutil node functions. By this,
we can handle cancellation during API call of node resource.
Signed-off-by: Yuichiro Ueno <y1r.ueno@gmail.com>
This commit adds context parameter to k8sutil job functions. By this, we
can handle cancellation during API call of job resource.
Signed-off-by: Yuichiro Ueno <y1r.ueno@gmail.com>
The OSD blkdevmapper init container relies on the MKNOD capability,
which it does not actually request.
As a result, deployments fail on Kubernetes clusters that do not
happen to assign this capability to all containers by default.
Solve this by updating the container spec securityContext to
explicitly request the capability it relies on.
Closes: https://github.com/rook/rook/issues/9156
Signed-off-by: Omar Pakker <Omar007@users.noreply.github.com>
This commit adds context parameter to k8sutil deployment functions. By
this, we can handle cancellation during API call of deployment resource.
Signed-off-by: Yuichiro Ueno <y1r.ueno@gmail.com>
When a reconcile is started for OSDs, the prepare jobs are first
deleted from a previous reconcile. The timeout for the osd prepare
job deletion was only 40s. After that timeout, the reconcile attempts
to continue waiting for the pod, but of course will never complete
since the OSD prepare was not running in the first place, causing the
reconcile to wait indefinitely. In the reported issue, the osd prepare
jobs were actually deleted successfully, the timeout just wasn't long
enough. Pods need at least a minute to be forcefully deleted,
so we increase the timeout to 90s to give it some extra buffer.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
This commit adds context parameter to utilities in opcontroller to
remove context.TODO use in opcontroller. By this, we can handle
cancellation of reconcilers in a fine-grained way.
Signed-off-by: Yuichiro Ueno <y1r.ueno@gmail.com>
Rook cluster-wide encryption can now use the native Kubernetes
authentication to interact with vault KMS instead of using the token
method.
Signed-off-by: Sébastien Han <seb@redhat.com>
When deploying a cluster-wde encrypted cluster we now use the rook
binary to execute some code to fetch the key encryption key.
Using cURL all the time to fetch the key has its limitations. The
incoming integration with Kubernetes Authentication through service
accounts is leading the usage of cURL to its end.
The logic is really complex and prone to errors. Re-implementing the
Go logic into Bash is not a viable option.
Also, using the lib in a binary allows us to keep a consistent behavior
throughout the life cycle of our code.
Signed-off-by: Sébastien Han <seb@redhat.com>
We don't need to use tini.
We don't have anything in the rook operator that would
either create zombie processes (no threads) or use
exec (to fork). The Go binary has a really good
signal handling mechanism.
Closes: https://github.com/rook/rook/issues/8794
Signed-off-by: Sébastien Han <seb@redhat.com>
The flex driver has been fully deprecated and thus removed from Rook.
Before upgrading to v1.8, users will need to convert existing flex volumes
from flex to csi volumes.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
This commit is a large refactor on how the operator starts, stops and
how it starts various sub-components such as the ceph-csi driver. It
also refines the way we cancel orchestrations. We don't use breakpoints
anymore but send our self a SIGUP to reload our controller runtime
manager.
The reload will happen under different circonstances like:
* a new adminission controller secret is created/deleted/changed
* a CephCluster CR is edited
As mentioned earlier, the csi driver now has its own controller, just
like flex. It reacts to change in the operator config map for particular
ROOK_CSI_ fields.
A second new controller for the operator's general config has been
created, it manages:
* the logging level
* the ceph CLI command timeout
* the discovery daemon
The operator reacts much more rapidly to cancellation events by stopping
the manager's context and reloading it.
Signed-off-by: Sébastien Han <seb@redhat.com>
The error message in UpdateNodeStatus regards the second argument
as node name. However, it is a PVC name in OSD on PVC.
Signed-off-by: Hiroya Onoe <onoehiroya@gmail.com>
earlier, `ApplyToPodSpec()` was only taking one toleration and ignoring
tolerations from `placement.ALL()`.
this commit merge toleration for Mgr,Mon,Osd pod
example, for osd it will merge
spec.placement.all and
storageDeviceClassSets.Placement(in case of pvc) or
spec.placement.osd(in case of non-pvc's).
Signed-off-by: subhamkrai <srai@redhat.com>
earlier, we had the issue for osd placement regarding
whether we should merge the placements or not.
Now, we have option `skipApplyAllPlacement` to
enable/disable osd placement. By default it is false
which means it will merge both placement.All() and
storageClassDeviceSets.Placement, when true
it will only apply storageClassDeviceSets.Placement.
Also, adding unit test.
Closes: https://github.com/rook/rook/issues/8135
Signed-off-by: subhamkrai <srai@redhat.com>
Both `ExecuteCommandWithOutputFileTimeout()` and
`ExecuteCommandWithOutputFile()` generate unnecessary system calls by
creating/reading/removing files where the stream output of the command
can simply be used. So sticking with `ExecuteCommandWithOutput()` and
`ExecuteCommandWithCombinedOutput()` for reading outputs is sufficient.
Closes: https://github.com/rook/rook/issues/8343
Signed-off-by: Sébastien Han <seb@redhat.com>
Recently, the builds of `ceph/ceph` image moved to quay.io, see
https://github.com/ceph/ceph-build/pull/1883 for more details.
Current images will remain but new builds will happen on quay.io only.
This means that tags such as `v14.2`, `v15.2`,`v16.2` will need to
switch to quay.io to get updates.
Signed-off-by: Sébastien Han <seb@redhat.com>
when osd pods removed, they are not able
to add back due to missing `ceph auth`
or different `ceph auth`.
this commit mounts the secret `rook-ceph-admin-keyring`
inside the osd activate initcontainer which has
admin keyring.
Signed-off-by: subhamkrai <srai@redhat.com>
When evaluating the potential "warning" key from the json output we were
not hiding stdout (1) from the output, so the key was leaked. Also the
warning check for the key was not perfect, it only checked for the
command to succeed, but we should introspect it since the "warning" key
is always present unlike the "error" one.
We now only print the warning or the error is present.
Signed-off-by: Sébastien Han <seb@redhat.com>
We have seen issues with fullchain.pem files containing one self-signed
certificate. At the same time, the failing configuration was not
providing any client certificates or keys.
The error from curl was `"self signed certificate in the chain"`.
The openssl binary is not available so we cannot even c_rehash the
certificate directory when using `--capath`, however this worked already
with a fully signed fullchain.pem along with a client certificate and
key. So we cannot only use `--cacert` which will break that use case.
This is becoming really obscur on why using `--cacert` solves this
problem. I just can't wait to use the `vault` binary, that's all I can
say...
Signed-off-by: Sébastien Han <seb@redhat.com>
The active osd container was always using /var/lib/rook for the
configuration of the OSD. If the dataDirHostPath is anything other
than the default, the config would show up in the wrong place.
Changing this back to the dataDirHostPath does not appear to affect
any upgrade scenarios as the activate container will simply start
extracting the config to the correct place.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Add ability to set device class for existing osd deployments,
based on its crush map device class.
Related-Issue: rook#8028
Signed-off-by: Denis Egorenko <degorenko@mirantis.com>
Implement the first step of `design/ceph/resource-dependencies.md` to
add dependency checking when deleting a CephCluster.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
Currently it is not possible to set resource limits based on device classes
for different OSDs. Now adding an ability to use predefined keys for main
cluster spec Resource section to reflect resource limits for different OSDs
with different device classes.
Closes: https://github.com/rook/rook/issues/8007
Signed-off-by: Denis Egorenko <degorenko@mirantis.com>
In some circonstances the VAULT_TOKEN env variable contains a new line.
We don't really know how and why. The saw this when upgrading from
Nautilus CentOS 7 to Octopus CentOS 8. So there might be something
different with bash and variable expansion.
Anyway, let's remove any potential new line and trailing spaces from
that var.
The error was splitting the curl line to something like:
```
+ curl --silent --show-error --request GET --header 'X-Vault-Token: s.aCHdo57tF19ddN1hQmGVyG8a
' --insecure https://vault.default.svc.cluster.local:8200/v1/rook/ver1/rook-ceph-osd-encryption-key-set1-data-0pztgf
curl: (92) HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)
```
Where a correct line will be:
```
+ curl --silent --show-error --request GET --header 'X-Vault-Token: s.aCHdo57tF19ddN1hQmGVyG8a' --insecure https://vault.default.svc.cluster.local:8200/v1/rook/ver1/rook-ceph-osd-encryption-key-set1-data-0pztgf
```
Signed-off-by: Sébastien Han <seb@redhat.com>
We don't need to pass the Volume with projection for TLS when TLS is not enabled
Somehow when this happens and we try to update a deployment spec it fails with:
```
ValidationError(Pod.spec.volumes[7].projected): missing required field "sources"
```
Signed-off-by: Sébastien Han <seb@redhat.com>
When configuring encrypted on-pvc clusters we now set the cluster fsid
in the LUKS header. If needed we can then determine if the encrypted
block is part of our cluster or not. We also attach the pvc_name in case
it might become useful.
Closes: https://github.com/rook/rook/issues/7991
Signed-off-by: Sébastien Han <seb@redhat.com>
The 'ceph-volume raw list' command used to list OSDs on nodes by the
rook-ceph-osd-prepare- jobs return user-friendly device names (e.g.,
/dev/sda) rather than persistent device names. Friendly device names
often change when systems reboot, and even persistent device names can
change under certain circumstances (e.g., a disk controller is replaced
with a different one). To prevent OSD Pods created for raw OSDs on
nodes from crashing repeatedly if the device it was created with is
renamed, scan the list of OSDs on the host with 'ceph-volume raw list'
to find the right OSD to activate in the OSD pod's OSD activation init
container.
Fixes#7918
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
The rook.io/v1 package was only an internal implementation detail and
does not have any CRDs that rely on it. The CRD deserialization should
handle the change in internal types without any issue. This separation
gives more flexibility for the storage providers to implement exactly
what is needed for their storage provider instead of forcing to use the
same types and risk affecting another storage provider.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>