When no VAULT_BACKEND_PATH is specified we default to the API one which
is set to "secret/", we could use a different variable but we don't want
to break the library support. Bonus point this will also fix issue with
users passing a slash at the end of the path.
Signed-off-by: Sébastien Han <seb@redhat.com>
When running encrypted OSDs on multipath the expand container was
failing to find the underlying device of the encrypted block. This was
because the underlying block was located in /dev/mapper and this path
wasn't mounted in the init container "expand-encrypted-bluefs".
This will manifest like so:
```
Underlying device for crypt device ocs-deviceset-block-dev-from-san-1-data-0-5qkmm-block-dmcrypt disappeared.
```
Signed-off-by: Sébastien Han <seb@redhat.com>
When running encrypted OSDs on multipath the expand container was
failing to find the underlying device of the encrypted block. This was
because the underlying block was located in /dev/mapper and this path
wasn't mounted in the init container "expand-encrypted-bluefs".
This will manifest like so:
```
Underlying device for crypt device ocs-deviceset-block-dev-from-san-1-data-0-5qkmm-block-dmcrypt disappeared.
```
Signed-off-by: Sébastien Han <seb@redhat.com>
Once we are done opening the block device, we should remove the key even
though it is mounted on a tmpfs.
Signed-off-by: Sébastien Han <seb@redhat.com>
Other resources such as the service monitor need to match
the labels on the services to the active mgr, thus the active
mgr name is now set on the labels for the metrics service
and dashboard service.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
In the case of PVC,
We are giving lower priority to all placement.
We want deviceSet placement to applied and
override in case of overlapping settings and
we are merging nodeAffinity if applied in both
all placement and deviceSet.
In case of non-PVC,
we apply spec.placement
Signed-off-by: subhamkrai <srai@redhat.com>
Co-authored-by: Travis Nielsen <tnielsen@redhat.com>
The integration tests have long been painful to maintain with
settings in various places and copied to multiple types,
inconsistent variable names, and otherwise difficult to maintain
code. Now the settings for a test suite are all in one place and
they remain in the same settings type throughout the test.
The multi-cluster suite is also refactored to use the same install
and uninstall helpers as the other suites.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
If the object store status is not yet initialized, the object store user should
fail its initialization and requeue the reconcile.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
For testing purposes, we need to configure the flex driver
and discovery daemon with the operator settings configmap.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
All clusters are configured with bluestore as the store type
therefore we can remove the settings where mentioned in the
docs and dead code.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
deletion of cluster is stuck when cluster is not configured. The PR skips the cleanup and removes the finalizers if the cluster was not configured, that is, if clusterInfo could not be loaded.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
It's better to validate ownerReferences when setting them. In addition, we should use
controllerrutil.Set{Controller,Owner}Reference, that have such validation, as possible.
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
Finally! We can now stop editing manually our crds definition. Simply
run `make crds-gen`. These two files:
* `cluster/examples/kubernetes/ceph/crds.yaml`
* `cluster/charts/rook-ceph/templates/resources.yaml`
will be autogenerated for us.
We rely on the controller-gen tool, it reads our API definitions from
`pkg/apis/` and produces the CRD files accordingly.
It uses "markers" to add extra yaml fields to the CRD, for instance we
have a lot fields with:
```
nullable: true
x-kubernetes-preserve-unknown-fields: true
```
The corresponding API type markers are:
```
// +kubebuilder:pruning:PreserveUnknownFields
// +nullable
```
For more API convention see
https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#optional-vs-required
and for the markers see: https://book.kubebuilder.io/reference/markers/crd-validation.html
Signed-off-by: Sébastien Han <seb@redhat.com>
This is a follow up to:
https://github.com/rook/rook/pull/7264
which added pool quotas. It was stated that it would be nice to
be able to set pool max_bytes quotas using standard k8s quantity
formats rather than an integer representing the number of bytes.
I was too busy at the time to make the changes. This PR adds the
functionality.
Signed-off-by: Frank Ritchie <12985912+fritchie@users.noreply.github.com>
As a recommendation from the core team we should turn on PG auto repair
when running on Bluestore OSDs. The Quincy release will enable this by
default but in the meantime let's enable it here.
Signed-off-by: Sébastien Han <seb@redhat.com>
Sometimes `radosgw-admin` succeeds after showing logs to stderr. We should
skip non-json strings if parsing output as json.
Here is an example.
```
2021-02-26 04:10:44.190418 I | op-bucket-prov: creating Ceph user "ceph-user-aSzNqgE7"
E0226 04:11:37.901310 8 controller.go:199] error syncing 'logging/loki-bucket': error provisioning bucket: Provision: can't create ceph user: error creating ceph user "ceph-user-aSzNqgE7": failed to unmarshal json. 2021-02-26T04:11:21.425+0000 7f6714be4980 1 robust_notify: If at first you don't succeed: (110) Connection timed out
2021-02-26T04:11:21.426+0000 7f6714be4980 0 ERROR: failed to distribute cache for ceph-hdd-object-store.rgw.meta:users.uid:ceph-user-aSzNqgE7
2021-02-26T04:11:32.168+0000 7f6714be4980 1 robust_notify: If at first you don't succeed: (110) Connection timed out
2021-02-26T04:11:32.168+0000 7f6714be4980 0 ERROR: failed to distribute cache for ceph-hdd-object-store.rgw.meta:users.keys:23Z8GUEXR0TJDO86PSJR
{
"user_id": "ceph-user-aSzNqgE7",
"display_name": "ceph-user-aSzNqgE7",
...
"mfa_ids": []
}: invalid character '-' after top-level value: failed to unmarshal json.
```
In this case, some logs like "robust_notify:..." was shown in stderr.
Unmarsharing was failed due to tried to parse these logs as json.
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
If the count of mgr daemons is reduced, we remove the extra one.
The check is made during each reconcile.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
In Pacific the simple command to retrieve the active mgr is with
ceph mgr stat instead of retrieving the full ceph mgr dump.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
If there are multiple mgrs running, the operator will automatically
add pod antiaffinity across hosts by default. For stretch clusters,
antiaffinity across the stretch failure domain (e.g. zones)
will be added. Required antiaffinity will be created unless the
test setting allowMultiplePerNode is set to true.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
If there are two mgr daemons, they should be running on different
hosts or zones, depending on the failure domain.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The mgr daemon may be failed over by ceph if the active mgr is not
responding and the standby mgr is available. If the active mgr changes
the services for the dashboard and metrics will be updated with a
label selector for the new active mgr. The services cannot direct
traffic to the standby mgr or else they will be incorrectly redirected
to the active mgr.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The active mgr should match the labels on the services that
are available for prometheus and the dashboard. The selector
labels must be updated whenever there is a new active mgr.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The GRPC metrics exposed by both the provisioner pod
and the node plugin pod on some port. Provisioner pod
is running on the pod network and the daemonset pods
run on the host network. sometimes starting the GRPC
metrics by default can lead to node plugin pods
crashloopback state this is due to the port conflict.
Moreover, the GRPC metrics are not for the user it's
for the one which will help to debug the time taken
by cephcsi to serve each GRPC call. Enabling it by
default won't be a good idea. So the plan is to
disable it by default, If someone faces any issue it
can be enabled later at some point in time.
closes#7378
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
Sébastien Han
Satoru Takeuchi
Travis Nielsen
subhamkraiandTravis Nielsen
Frank Ritchie
Santosh Pillai
ushen
Madhu Rajanna