Currently it is not possible to set resource limits based on device classes
for different OSDs. Now adding an ability to use predefined keys for main
cluster spec Resource section to reflect resource limits for different OSDs
with different device classes.
Closes: https://github.com/rook/rook/issues/8007
Signed-off-by: Denis Egorenko <degorenko@mirantis.com>
In some circonstances the VAULT_TOKEN env variable contains a new line.
We don't really know how and why. The saw this when upgrading from
Nautilus CentOS 7 to Octopus CentOS 8. So there might be something
different with bash and variable expansion.
Anyway, let's remove any potential new line and trailing spaces from
that var.
The error was splitting the curl line to something like:
```
+ curl --silent --show-error --request GET --header 'X-Vault-Token: s.aCHdo57tF19ddN1hQmGVyG8a
' --insecure https://vault.default.svc.cluster.local:8200/v1/rook/ver1/rook-ceph-osd-encryption-key-set1-data-0pztgf
curl: (92) HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)
```
Where a correct line will be:
```
+ curl --silent --show-error --request GET --header 'X-Vault-Token: s.aCHdo57tF19ddN1hQmGVyG8a' --insecure https://vault.default.svc.cluster.local:8200/v1/rook/ver1/rook-ceph-osd-encryption-key-set1-data-0pztgf
```
Signed-off-by: Sébastien Han <seb@redhat.com>
We don't need to pass the Volume with projection for TLS when TLS is not enabled
Somehow when this happens and we try to update a deployment spec it fails with:
```
ValidationError(Pod.spec.volumes[7].projected): missing required field "sources"
```
Signed-off-by: Sébastien Han <seb@redhat.com>
When configuring encrypted on-pvc clusters we now set the cluster fsid
in the LUKS header. If needed we can then determine if the encrypted
block is part of our cluster or not. We also attach the pvc_name in case
it might become useful.
Closes: https://github.com/rook/rook/issues/7991
Signed-off-by: Sébastien Han <seb@redhat.com>
The 'ceph-volume raw list' command used to list OSDs on nodes by the
rook-ceph-osd-prepare- jobs return user-friendly device names (e.g.,
/dev/sda) rather than persistent device names. Friendly device names
often change when systems reboot, and even persistent device names can
change under certain circumstances (e.g., a disk controller is replaced
with a different one). To prevent OSD Pods created for raw OSDs on
nodes from crashing repeatedly if the device it was created with is
renamed, scan the list of OSDs on the host with 'ceph-volume raw list'
to find the right OSD to activate in the OSD pod's OSD activation init
container.
Fixes#7918
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
The rook.io/v1 package was only an internal implementation detail and
does not have any CRDs that rely on it. The CRD deserialization should
handle the change in internal types without any issue. This separation
gives more flexibility for the storage providers to implement exactly
what is needed for their storage provider instead of forcing to use the
same types and risk affecting another storage provider.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
A directory is created under dataDirHostPath for each OSD. In host-based cluster,
its name is constructed from kernel device name like _dev_sdb though user specifies
udev persistent device name. We should use unique name among all OSDs.
Closes: https://github.com/rook/rook/issues/7878
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
A user with unbalanced system (e.g., some OSDs use a partition which
resides on a device along-side OS partition) may want to have a
fine-grained control on the overall load of specific OSD. While osd
weight affects write-load, primary-affinity affects read-load.
This patch allow user to define osd's primary-affinity (similar to
'DeviceClass' and 'InitialWeight'), which is sent to the osd (via mon)
as part of deployment sequence, using 'ceph osd primary-affinity' command.
Valid values are within the range [0.0, 1.0). If not set, default value
is 1.
ROOK issue: https://github.com/rook/rook/issues/7773
Signed-off-by: Shachar Sharon <ssharon@redhat.com>
The VolumeSource type was only used internally when loading
the storageClassDeviceSets. The volumeSource element is never
expected to be specified in the CephCluster CR, so it is now
removed from the schema. The VolumeSource type is renamed to
deviceSet for clarity.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The nodeAffinity for portable OSDs is needed to ensure the OSDs are
tied to the same zone or other failure domain where the OSD was
prepared and where it is expected to run. During upgrade the topology
affinity was not being detected since the osd prepare job is skipped
in that case. Now the topology will be detected from the running
OSD daemon pod's node.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
delete updatePVCIfChanged method on "Cluster" from osd package and use
the common one ExpandPVCIfRequired from k8sutil pkg
Signed-off-by: Nitin Goyal <nigoyal@redhat.com>
Ceph support the option '--osd-crush-initial-weight' upon OSD start,
which sets an explicit weight (in TiB units) to specific OSD. Allow
passing this option all the way from the user (similar to
'DeviceClass'), for the special case where end users wants it cluster
to have non-even balance over specific OSDs (e.g., one of the OSDs is
placed over a partition alongside OS-partition).
ROOK issue: https://github.com/rook/rook/issues/7448
Signed-off-by: Shachar Sharon <ssharon@redhat.com>
The metadata name must be unique for a storageClassDeviceSet since
only one each of the sections data, metadata, and wal can be specified.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
OSD directories were removed in v1.2, this is some more cleanup
of dir configuration that is no longer needed.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
With Pacific comes the support for dualstack where ceph daemons can
listen on both ipv4 and ipv6 stacks.
A new field in the network spec has been added: `dualStack`
Signed-off-by: Sébastien Han <seb@redhat.com>
Update OSDs in parallel per the design in
design/ceph/update-osds-in-parallel.md
The max number of OSDs updated in parallel is currently fixed at 20.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
When no VAULT_BACKEND_PATH is specified we default to the API one which
is set to "secret/", we could use a different variable but we don't want
to break the library support. Bonus point this will also fix issue with
users passing a slash at the end of the path.
Signed-off-by: Sébastien Han <seb@redhat.com>
Previously, `interval` was just a string so no special validation was
done by the OpenAPI validator. Now that it is advertised as metav1.Duration
the server will introspect it correctly. Underneath the type is still a
string so no update issue to be worried about.
Signed-off-by: Sébastien Han <seb@redhat.com>
When running encrypted OSDs on multipath the expand container was
failing to find the underlying device of the encrypted block. This was
because the underlying block was located in /dev/mapper and this path
wasn't mounted in the init container "expand-encrypted-bluefs".
This will manifest like so:
```
Underlying device for crypt device ocs-deviceset-block-dev-from-san-1-data-0-5qkmm-block-dmcrypt disappeared.
```
Signed-off-by: Sébastien Han <seb@redhat.com>
Once we are done opening the block device, we should remove the key even
though it is mounted on a tmpfs.
Signed-off-by: Sébastien Han <seb@redhat.com>
In the case of PVC,
We are giving lower priority to all placement.
We want deviceSet placement to applied and
override in case of overlapping settings and
we are merging nodeAffinity if applied in both
all placement and deviceSet.
In case of non-PVC,
we apply spec.placement
Signed-off-by: subhamkrai <srai@redhat.com>
Co-authored-by: Travis Nielsen <tnielsen@redhat.com>
All clusters are configured with bluestore as the store type
therefore we can remove the settings where mentioned in the
docs and dead code.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
It's better to validate ownerReferences when setting them. In addition, we should use
controllerrutil.Set{Controller,Owner}Reference, that have such validation, as possible.
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
The mgr daemon may be failed over by ceph if the active mgr is not
responding and the standby mgr is available. If the active mgr changes
the services for the dashboard and metrics will be updated with a
label selector for the new active mgr. The services cannot direct
traffic to the standby mgr or else they will be incorrectly redirected
to the active mgr.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
For simple OSD scenario that don't require the following:
* an encrypted osd
* an osd with a backing device
* multiple osd on the same block device
The OSD will be prepared using the ceph-volume raw mode which does
not use LVM on non-pvc case.
Closes: https://github.com/rook/rook/issues/4768
Signed-off-by: Sébastien Han <seb@redhat.com>
The conditions on the cephcluster CR were being set to false
when not in progress, which is misleading to the purpose of
conditions. The conditions will now always represent current
state of the cluster. Transient conditions that show progress of
a reconcile will be removed from the status.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
If there is a warning coming from the Vault server we must catch it and
exit immediately to reflect it if no key was retrieved.
Signed-off-by: Sébastien Han <seb@redhat.com>
The vault env variable VAULT_BACKEND is read by the secret library so we
must not change it. This is reverting
ee6a03dfb8 with a few extra fixes for the
rgw code.
Signed-off-by: Sébastien Han <seb@redhat.com>
Remove features and design that supports adding OSDs to Ceph clusters
via `spec:driveGroups`. Update the Ceph upgrade doc that informs users
who currently use Drive Groups (we believe there are none of these
users) how to migrate to using the `spec:storage` config.
Resolves https://github.com/rook/rook/issues/7275
Revert "ceph: fix drive group deployment failure"
This reverts commit 76f1d9944e.
Revert "ceph: osd: add drive groups spec to cluster CR"
This reverts commit 7117fc12b7.
Revert "design: ceph orchestrator module add/remove OSDs"
This reverts commit 178187d035.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
Change the TestOSDsOnPVC unit test to use t.Log()/t.Logf() to avoid
having a special infof() function that was unnecessary since our unit
tests run with `go test -v` where the `-v` flag interleaves t.Log()
output.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
After adopting `ceph-volume`, secret key for
user osd.x will be generated by `ceph-volume` and
thus method `generateKeyring` is not necessary
anymore.
Executing ceph command takes
extra half of second to complete, executing it
in serial is rather expensive, remove it for better
performance.
Signed-off-by: shenjiatong <yshxxsjt715@gmail.com>
This was a race condition where the test would sometimes finish before a
deferred watcher.Stop() command was called and sometimes would not.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
cURL's ca-cert and Golang TLS config for ca-certs have a different
meaning, it seems that the Golang implementation will always lookup the
local certs first and then proceeds with the specificed cert, if the
system's certs works it will never go to the specificied certificate.
cURL however, does the opposite by strictly checking the specified cert.
If the certificate is signed we don't need to force a specific
certificate, thus by using capath instead we let cURL determine what's
best to validate against.
Signed-off-by: Sébastien Han <seb@redhat.com>
The --silent option will remove the progress bar of the curl but also
silence any errors of stderr, so adding --show-error will display the
error even if --silent is used.
Signed-off-by: Sébastien Han <seb@redhat.com>
removed the code for force delete pods when stuck in terminating state
for mons and osds as there is common code to delete pods for stuck components
Signed-off-by: rohan47 <rohgupta@redhat.com>
The osd prepare job detects the failure domain for the OSD by querying
the node topology labels. This topology is then assigned to the OSD
daemon in the CRUSH map. Previously, the affinity was required to be
set in the cluster CR, but it was very difficult to get right. Now the
operator will enforce the correct topology label on the OSD daemon
nodeAffinity by setting the label of the lowest topology in the hierarcy.
For example, if there are region, zone, and rack labels, the rack label
would be used to set the node affinity for the OSD daemon. If an
OSD prepare job is run in rack1, the corresponding OSD daemon will
have node affinity to rack1 to ensure the same topology. Previously,
the OSD could have ended up in rack2 unless the storageClassDeviceSet
placement was very carefully crafted.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
this commit update `osdProps.placement` instead
of `osdProps.getPreparePlacement()` in the osd.go
this was the copy paste issues coming from other pr.
Signed-off-by: subhamkrai <srai@redhat.com>
we don't want to merge the node affinity if specified in both
storageClassDeviceSet and placement. Now, we are passing new
boolean argument in `ApplyToPodSpec` which will be false in
case of osd and prepare pods because we don't want to overlap
placement of osd on PVC's and non-PVC's.
Signed-off-by: subhamkrai <srai@redhat.com>