When running encrypted OSDs on multipath the expand container was
failing to find the underlying device of the encrypted block. This was
because the underlying block was located in /dev/mapper and this path
wasn't mounted in the init container "expand-encrypted-bluefs".
This will manifest like so:
```
Underlying device for crypt device ocs-deviceset-block-dev-from-san-1-data-0-5qkmm-block-dmcrypt disappeared.
```
Signed-off-by: Sébastien Han <seb@redhat.com>
Once we are done opening the block device, we should remove the key even
though it is mounted on a tmpfs.
Signed-off-by: Sébastien Han <seb@redhat.com>
Other resources such as the service monitor need to match
the labels on the services to the active mgr, thus the active
mgr name is now set on the labels for the metrics service
and dashboard service.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
In the case of PVC,
We are giving lower priority to all placement.
We want deviceSet placement to applied and
override in case of overlapping settings and
we are merging nodeAffinity if applied in both
all placement and deviceSet.
In case of non-PVC,
we apply spec.placement
Signed-off-by: subhamkrai <srai@redhat.com>
Co-authored-by: Travis Nielsen <tnielsen@redhat.com>
The integration tests have long been painful to maintain with
settings in various places and copied to multiple types,
inconsistent variable names, and otherwise difficult to maintain
code. Now the settings for a test suite are all in one place and
they remain in the same settings type throughout the test.
The multi-cluster suite is also refactored to use the same install
and uninstall helpers as the other suites.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
For testing purposes, we need to configure the flex driver
and discovery daemon with the operator settings configmap.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
All clusters are configured with bluestore as the store type
therefore we can remove the settings where mentioned in the
docs and dead code.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
deletion of cluster is stuck when cluster is not configured. The PR skips the cleanup and removes the finalizers if the cluster was not configured, that is, if clusterInfo could not be loaded.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
It's better to validate ownerReferences when setting them. In addition, we should use
controllerrutil.Set{Controller,Owner}Reference, that have such validation, as possible.
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
Finally! We can now stop editing manually our crds definition. Simply
run `make crds-gen`. These two files:
* `cluster/examples/kubernetes/ceph/crds.yaml`
* `cluster/charts/rook-ceph/templates/resources.yaml`
will be autogenerated for us.
We rely on the controller-gen tool, it reads our API definitions from
`pkg/apis/` and produces the CRD files accordingly.
It uses "markers" to add extra yaml fields to the CRD, for instance we
have a lot fields with:
```
nullable: true
x-kubernetes-preserve-unknown-fields: true
```
The corresponding API type markers are:
```
// +kubebuilder:pruning:PreserveUnknownFields
// +nullable
```
For more API convention see
https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#optional-vs-required
and for the markers see: https://book.kubebuilder.io/reference/markers/crd-validation.html
Signed-off-by: Sébastien Han <seb@redhat.com>
If the count of mgr daemons is reduced, we remove the extra one.
The check is made during each reconcile.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
In Pacific the simple command to retrieve the active mgr is with
ceph mgr stat instead of retrieving the full ceph mgr dump.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
If there are multiple mgrs running, the operator will automatically
add pod antiaffinity across hosts by default. For stretch clusters,
antiaffinity across the stretch failure domain (e.g. zones)
will be added. Required antiaffinity will be created unless the
test setting allowMultiplePerNode is set to true.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
If there are two mgr daemons, they should be running on different
hosts or zones, depending on the failure domain.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The mgr daemon may be failed over by ceph if the active mgr is not
responding and the standby mgr is available. If the active mgr changes
the services for the dashboard and metrics will be updated with a
label selector for the new active mgr. The services cannot direct
traffic to the standby mgr or else they will be incorrectly redirected
to the active mgr.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The active mgr should match the labels on the services that
are available for prometheus and the dashboard. The selector
labels must be updated whenever there is a new active mgr.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
For simple OSD scenario that don't require the following:
* an encrypted osd
* an osd with a backing device
* multiple osd on the same block device
The OSD will be prepared using the ceph-volume raw mode which does
not use LVM on non-pvc case.
Closes: https://github.com/rook/rook/issues/4768
Signed-off-by: Sébastien Han <seb@redhat.com>
The conditions on the cephcluster CR were being set to false
when not in progress, which is misleading to the purpose of
conditions. The conditions will now always represent current
state of the cluster. Transient conditions that show progress of
a reconcile will be removed from the status.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
If there is a warning coming from the Vault server we must catch it and
exit immediately to reflect it if no key was retrieved.
Signed-off-by: Sébastien Han <seb@redhat.com>
The vault env variable VAULT_BACKEND is read by the secret library so we
must not change it. This is reverting
ee6a03dfb8 with a few extra fixes for the
rgw code.
Signed-off-by: Sébastien Han <seb@redhat.com>
Remove features and design that supports adding OSDs to Ceph clusters
via `spec:driveGroups`. Update the Ceph upgrade doc that informs users
who currently use Drive Groups (we believe there are none of these
users) how to migrate to using the `spec:storage` config.
Resolves https://github.com/rook/rook/issues/7275
Revert "ceph: fix drive group deployment failure"
This reverts commit 76f1d9944e.
Revert "ceph: osd: add drive groups spec to cluster CR"
This reverts commit 7117fc12b7.
Revert "design: ceph orchestrator module add/remove OSDs"
This reverts commit 178187d035.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
Change the TestOSDsOnPVC unit test to use t.Log()/t.Logf() to avoid
having a special infof() function that was unnecessary since our unit
tests run with `go test -v` where the `-v` flag interleaves t.Log()
output.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
After adopting `ceph-volume`, secret key for
user osd.x will be generated by `ceph-volume` and
thus method `generateKeyring` is not necessary
anymore.
Executing ceph command takes
extra half of second to complete, executing it
in serial is rather expensive, remove it for better
performance.
Signed-off-by: shenjiatong <yshxxsjt715@gmail.com>
This was a race condition where the test would sometimes finish before a
deferred watcher.Stop() command was called and sometimes would not.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
cURL's ca-cert and Golang TLS config for ca-certs have a different
meaning, it seems that the Golang implementation will always lookup the
local certs first and then proceeds with the specificed cert, if the
system's certs works it will never go to the specificied certificate.
cURL however, does the opposite by strictly checking the specified cert.
If the certificate is signed we don't need to force a specific
certificate, thus by using capath instead we let cURL determine what's
best to validate against.
Signed-off-by: Sébastien Han <seb@redhat.com>
The --silent option will remove the progress bar of the curl but also
silence any errors of stderr, so adding --show-error will display the
error even if --silent is used.
Signed-off-by: Sébastien Han <seb@redhat.com>
removed the code for force delete pods when stuck in terminating state
for mons and osds as there is common code to delete pods for stuck components
Signed-off-by: rohan47 <rohgupta@redhat.com>
The osd prepare job detects the failure domain for the OSD by querying
the node topology labels. This topology is then assigned to the OSD
daemon in the CRUSH map. Previously, the affinity was required to be
set in the cluster CR, but it was very difficult to get right. Now the
operator will enforce the correct topology label on the OSD daemon
nodeAffinity by setting the label of the lowest topology in the hierarcy.
For example, if there are region, zone, and rack labels, the rack label
would be used to set the node affinity for the OSD daemon. If an
OSD prepare job is run in rack1, the corresponding OSD daemon will
have node affinity to rack1 to ensure the same topology. Previously,
the OSD could have ended up in rack2 unless the storageClassDeviceSet
placement was very carefully crafted.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
this commit update `osdProps.placement` instead
of `osdProps.getPreparePlacement()` in the osd.go
this was the copy paste issues coming from other pr.
Signed-off-by: subhamkrai <srai@redhat.com>
The cleanup job will need to run wherever the ceph daemons
were running. The scheduling is done by rook on the specific
nodes where the daemons were running, so there is no need for
other placement such as node or pod affinity.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
we don't want to merge the node affinity if specified in both
storageClassDeviceSet and placement. Now, we are passing new
boolean argument in `ApplyToPodSpec` which will be false in
case of osd and prepare pods because we don't want to overlap
placement of osd on PVC's and non-PVC's.
Signed-off-by: subhamkrai <srai@redhat.com>
Add a new OSD provisioner status (reported by provisioner configmap)
that denotes that an OSD is preexisting and that OSD prepare does not
need to be run for the OSD. This only applies to OSDs on PVC currently
where existence of a deployment for the OSD indicates no further
provisioning needs to occur. New status is "preexisting".
Allow creating new OSDs on PVCs before updating existing ones by
allowing deferring OSDs during processing of OSD provisioning status
ConfigMaps. Deferral is identified by new "preexisting" status.
Add a unit test to ensure OSD on PVC provisioning occurs as expected
including deferring already-created OSDs to be updated after new PVCs
are created.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
Sébastien Han
Travis Nielsen
subhamkraiandTravis Nielsen
Santosh Pillai
Satoru Takeuchi
ushen
Blaine Gardner
Blaine Gardner
rohan47