The unit test TestConfigureCVDevices was failing intermittently
due to comparing the first element of the slice of two elements.
The slice ordering is not guaranteed, which caused a failure
intermittently. The test only compares the critical values in the test
and no need to check elements of the slice.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The deviceClass was only being set for LVM mode OSDs and was
missed for the raw mode for non-PVC. Now the deviceClass will be set
for all expected scenarios.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The ValidateConnectionDetails() contains `clusterSpec` assuming `SecuritySpec` only
part to `cephCluster` CRD. But we may need to validate same for `SecuritySpec` in
`cephObjectStore` CRD. Hence modifying the signature of the api.
Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
This fixes cases where raw provisioning mode still cannot be used for
Rook v1.6 and Ceph Pacific clusters where json.Unmarshal fails to read
the ceph-volume output after OSD provisioning.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
Ceph support the option '--osd-crush-initial-weight' upon OSD start,
which sets an explicit weight (in TiB units) to specific OSD. Allow
passing this option all the way from the user (similar to
'DeviceClass'), for the special case where end users wants it cluster
to have non-even balance over specific OSDs (e.g., one of the OSDs is
placed over a partition alongside OS-partition).
ROOK issue: https://github.com/rook/rook/issues/7448
Signed-off-by: Shachar Sharon <ssharon@redhat.com>
This mops up the last of the multiple-imports within ceph-related
packages, in pkg/daemon/ceph/agent/flexvolume/manager/ceph and
pkg/daemon/ceph/osd.
Signed-off-by: Lars Lehtonen <lars.lehtonen@gmail.com>
`ceph osd purge` should remove the OSD auth, but many users have issues
where they are unable to create new OSDs after removing old ones due to
the old OSD auth still being present. Therefore, run `ceph auth del` for
the OSD after purging to fix this.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
Update OSDs in parallel per the design in
design/ceph/update-osds-in-parallel.md
The max number of OSDs updated in parallel is currently fixed at 20.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
When the cluster is growing, the operator always processes all OSDs and
re-generate their Deployment spec. The generated environment variables
for encryption are coming from a map, which by design is unordered. So
once generated the deployment will see their spec change since the
environment variable position will change.
See from the following diff:
```
env: env:
- name: KMS_SERVICE_NAME <
value: vault <
- name: VAULT_NAMESPACE <
value: ocsns <
- name: VAULT_TLS_SERVER_NAME <
- name: VAULT_BACKEND_PATH <
value: n_ocs <
- name: VAULT_ADDR - name: VAULT_ADDR
value: https://vault.qe.rh-ocs.com:8200 value: https://vault.qe.rh-ocs.com:8200
> - name: VAULT_BACKEND_PATH
> value: n_ocs
> - name: VAULT_TLS_SERVER_NAME
- name: KMS_PROVIDER - name: KMS_PROVIDER
value: vault value: vault
> - name: KMS_SERVICE_NAME
> value: vault
> - name: VAULT_NAMESPACE
> value: ocsns
- name: VAULT_TOKEN - name: VAULT_TOKEN
valueFrom: valueFrom:
secretKeyRef: secretKeyRef:
key: token key: token
name: ocs-kms-token name: ocs-kms-token
```
Signed-off-by: Sébastien Han <seb@redhat.com>
It's better to validate ownerReferences when setting them. In addition, we should use
controllerrutil.Set{Controller,Owner}Reference, that have such validation, as possible.
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
The mgr daemon may be failed over by ceph if the active mgr is not
responding and the standby mgr is available. If the active mgr changes
the services for the dashboard and metrics will be updated with a
label selector for the new active mgr. The services cannot direct
traffic to the standby mgr or else they will be incorrectly redirected
to the active mgr.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
When c-v runs after the end of the prepare sequence, it will look for
lvm and raw-based OSDs. The raw mode detects LVM OSDs so we must trim
the list of OSD, only to report the LVM ones if their ID exists.
Closes: https://github.com/rook/rook/issues/7359
Signed-off-by: Sébastien Han <seb@redhat.com>
For simple OSD scenario that don't require the following:
* an encrypted osd
* an osd with a backing device
* multiple osd on the same block device
The OSD will be prepared using the ceph-volume raw mode which does
not use LVM on non-pvc case.
Closes: https://github.com/rook/rook/issues/4768
Signed-off-by: Sébastien Han <seb@redhat.com>
block if not defined was used with osdInfo.Device so not empty anymore
in the loop, which means it will have the same value for all OSDs in the
loop.
This is only called when running on PVC, where a single device is
present so the loop has a single iteration, so not a big issue.
Signed-off-by: Sébastien Han <seb@redhat.com>
The vault env variable VAULT_BACKEND is read by the secret library so we
must not change it. This is reverting
ee6a03dfb8 with a few extra fixes for the
rgw code.
Signed-off-by: Sébastien Han <seb@redhat.com>
Remove features and design that supports adding OSDs to Ceph clusters
via `spec:driveGroups`. Update the Ceph upgrade doc that informs users
who currently use Drive Groups (we believe there are none of these
users) how to migrate to using the `spec:storage` config.
Resolves https://github.com/rook/rook/issues/7275
Revert "ceph: fix drive group deployment failure"
This reverts commit 76f1d9944e.
Revert "ceph: osd: add drive groups spec to cluster CR"
This reverts commit 7117fc12b7.
Revert "design: ceph orchestrator module add/remove OSDs"
This reverts commit 178187d035.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
The osd prepare job detects the failure domain for the OSD by querying
the node topology labels. This topology is then assigned to the OSD
daemon in the CRUSH map. Previously, the affinity was required to be
set in the cluster CR, but it was very difficult to get right. Now the
operator will enforce the correct topology label on the OSD daemon
nodeAffinity by setting the label of the lowest topology in the hierarcy.
For example, if there are region, zone, and rack labels, the rack label
would be used to set the node affinity for the OSD daemon. If an
OSD prepare job is run in rack1, the corresponding OSD daemon will
have node affinity to rack1 to ensure the same topology. Previously,
the OSD could have ended up in rack2 unless the storageClassDeviceSet
placement was very carefully crafted.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
If the cluster was expanded we would iterate over the entire list of
OSDs again, create a new key and override in Vault... The Secret store
is not impacted since we have a handler for "already exist".
Signed-off-by: Sébastien Han <seb@redhat.com>
Add a new OSD provisioner status (reported by provisioner configmap)
that denotes that an OSD is preexisting and that OSD prepare does not
need to be run for the OSD. This only applies to OSDs on PVC currently
where existence of a deployment for the OSD indicates no further
provisioning needs to occur. New status is "preexisting".
Allow creating new OSDs on PVCs before updating existing ones by
allowing deferring OSDs during processing of OSD provisioning status
ConfigMaps. Deferral is identified by new "preexisting" status.
Add a unit test to ensure OSD on PVC provisioning occurs as expected
including deferring already-created OSDs to be updated after new PVCs
are created.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
With 5f57b6fd3a, we introduced a
regression in the validation code.
If the OSD PV comes from another cluster, GetCephVolumeRawOSDs must
return an error.
The prepare job will crash loop but at least with a useful and
comprehensive message.
Signed-off-by: Sébastien Han <seb@redhat.com>
The first patch to configure vault for ceph object store. If the `security.kms` configured in
`clusterSpec` CRD, RGW will be configured with vault kms settings to handle SSE request from s3 clients.
Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
14.2.15 lvm batch command prepare report changes
output format. This commit skips md check if ceph version
is greater than 14.2.13.
Signed-off-by: shenjiatong <yshxxsjt715@gmail.com>
The deviceClass property was being ignored when creating the
non-pvc OSDs. Now the deviceClass will be specified as a property
for individual devices, all devices on a node, or all OSDs in the
cluster, depending on the level where the config is applied in the
cluster CR.
Co-authored-by: shenjiatong <yshxxsjt715@gmail.com>
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The code was assuming that devices were passed by the user as
"/dev/sda", this is bad! We all know people should be using paths like
/dev/disk/by-id so we must support them.
Closes: https://github.com/rook/rook/issues/6685
Signed-off-by: Sébastien Han <seb@redhat.com>
Found by running the following command:
codespell -S .git,*.png,*.jpg -L \
aks,keyserver,atleast,dne,ser,ist,files\',ba,dum,iam,te -f -H
Signed-off-by: Mateusz Gozdek <mgozdekof@gmail.com>
The osd purge job was not removing the osd prepare job or the
osd pvc due to an invalid label query for the pvc. Now the
prepare job and pvc will be removed as expected from the job.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Golang maps order is not always guaranteed to be identical, which makes
the table driven test to fail. Because of the nature of the maps, it is
hard to predict the exact position of an element in the map. This
makes the table driven test flappy.
Reworked the unit test using a contains statement which achieves the
same validation.
Closes: https://github.com/rook/rook/issues/6522
Signed-off-by: Sébastien Han <seb@redhat.com>
When the Ceph cluster runs on PVC and the OSDs are encrypted we can
store LUKS's Key Encryption Key inside a Key Management System. Today,
Rook only supports HashiCorp Vault: https://www.vaultproject.io/
The CephCluster has now a new "security" field which will plug onto the
KMS. Here is an example:
security:
kms:
tokenSecretName: <name of the secret containing a Vault token, used
to authenticate>
connectionDetails: < a map of strings containing connection
information>
Refer to the ceph-cluster-crd documentation to lear more.
Closes: https://github.com/rook/rook/issues/6105
Signed-off-by: Sébastien Han <seb@redhat.com>
Travis Nielsen
Jiffin Tony Thottan
Sébastien Han
Blaine Gardner
Shachar Sharon
Lars Lehtonen
Satoru Takeuchi
Blaine Gardner
crombus
shenjiatong
Arun Kumar Mohan
Mateusz Gozdek
Daichi Sakaue