For simple OSD scenario that don't require the following:
* an encrypted osd
* an osd with a backing device
* multiple osd on the same block device
The OSD will be prepared using the ceph-volume raw mode which does
not use LVM on non-pvc case.
Closes: https://github.com/rook/rook/issues/4768
Signed-off-by: Sébastien Han <seb@redhat.com>
The conditions on the cephcluster CR were being set to false
when not in progress, which is misleading to the purpose of
conditions. The conditions will now always represent current
state of the cluster. Transient conditions that show progress of
a reconcile will be removed from the status.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
If there is a warning coming from the Vault server we must catch it and
exit immediately to reflect it if no key was retrieved.
Signed-off-by: Sébastien Han <seb@redhat.com>
The vault env variable VAULT_BACKEND is read by the secret library so we
must not change it. This is reverting
ee6a03dfb8 with a few extra fixes for the
rgw code.
Signed-off-by: Sébastien Han <seb@redhat.com>
Remove features and design that supports adding OSDs to Ceph clusters
via `spec:driveGroups`. Update the Ceph upgrade doc that informs users
who currently use Drive Groups (we believe there are none of these
users) how to migrate to using the `spec:storage` config.
Resolves https://github.com/rook/rook/issues/7275
Revert "ceph: fix drive group deployment failure"
This reverts commit 76f1d9944e.
Revert "ceph: osd: add drive groups spec to cluster CR"
This reverts commit 7117fc12b7.
Revert "design: ceph orchestrator module add/remove OSDs"
This reverts commit 178187d035.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
Change the TestOSDsOnPVC unit test to use t.Log()/t.Logf() to avoid
having a special infof() function that was unnecessary since our unit
tests run with `go test -v` where the `-v` flag interleaves t.Log()
output.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
After adopting `ceph-volume`, secret key for
user osd.x will be generated by `ceph-volume` and
thus method `generateKeyring` is not necessary
anymore.
Executing ceph command takes
extra half of second to complete, executing it
in serial is rather expensive, remove it for better
performance.
Signed-off-by: shenjiatong <yshxxsjt715@gmail.com>
This was a race condition where the test would sometimes finish before a
deferred watcher.Stop() command was called and sometimes would not.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
cURL's ca-cert and Golang TLS config for ca-certs have a different
meaning, it seems that the Golang implementation will always lookup the
local certs first and then proceeds with the specificed cert, if the
system's certs works it will never go to the specificied certificate.
cURL however, does the opposite by strictly checking the specified cert.
If the certificate is signed we don't need to force a specific
certificate, thus by using capath instead we let cURL determine what's
best to validate against.
Signed-off-by: Sébastien Han <seb@redhat.com>
The --silent option will remove the progress bar of the curl but also
silence any errors of stderr, so adding --show-error will display the
error even if --silent is used.
Signed-off-by: Sébastien Han <seb@redhat.com>
removed the code for force delete pods when stuck in terminating state
for mons and osds as there is common code to delete pods for stuck components
Signed-off-by: rohan47 <rohgupta@redhat.com>
The osd prepare job detects the failure domain for the OSD by querying
the node topology labels. This topology is then assigned to the OSD
daemon in the CRUSH map. Previously, the affinity was required to be
set in the cluster CR, but it was very difficult to get right. Now the
operator will enforce the correct topology label on the OSD daemon
nodeAffinity by setting the label of the lowest topology in the hierarcy.
For example, if there are region, zone, and rack labels, the rack label
would be used to set the node affinity for the OSD daemon. If an
OSD prepare job is run in rack1, the corresponding OSD daemon will
have node affinity to rack1 to ensure the same topology. Previously,
the OSD could have ended up in rack2 unless the storageClassDeviceSet
placement was very carefully crafted.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
this commit update `osdProps.placement` instead
of `osdProps.getPreparePlacement()` in the osd.go
this was the copy paste issues coming from other pr.
Signed-off-by: subhamkrai <srai@redhat.com>
The cleanup job will need to run wherever the ceph daemons
were running. The scheduling is done by rook on the specific
nodes where the daemons were running, so there is no need for
other placement such as node or pod affinity.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
we don't want to merge the node affinity if specified in both
storageClassDeviceSet and placement. Now, we are passing new
boolean argument in `ApplyToPodSpec` which will be false in
case of osd and prepare pods because we don't want to overlap
placement of osd on PVC's and non-PVC's.
Signed-off-by: subhamkrai <srai@redhat.com>
Add a new OSD provisioner status (reported by provisioner configmap)
that denotes that an OSD is preexisting and that OSD prepare does not
need to be run for the OSD. This only applies to OSDs on PVC currently
where existence of a deployment for the OSD indicates no further
provisioning needs to occur. New status is "preexisting".
Allow creating new OSDs on PVCs before updating existing ones by
allowing deferring OSDs during processing of OSD provisioning status
ConfigMaps. Deferral is identified by new "preexisting" status.
Add a unit test to ensure OSD on PVC provisioning occurs as expected
including deferring already-created OSDs to be updated after new PVCs
are created.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
If the Vault server returns an error, we must report it in the init
container. With this we can properly debug a failing init container.
Resulting container log:
```
oc logs rook-ceph-osd-0-dc6d4d4d-445dg -c encryption-kms-get-kek ⏎
['error performing token check: Vault is sealed']%
```
Signed-off-by: Sébastien Han <seb@redhat.com>
When the blkdevmapper init container initializes we must override any
existing special block. This will prevent re-using a different disk on
the OSD on PVC scenario.
Adding "--remove-destination" to the cp line will remove each existing
destination file before attempting to open it.
We **MUST** do this otherwise in environment where PVCs are dynamic, restarting the deployment will cause conflicts
When restarting the OSD, the PVC block might end up with a different Kernel disk allocation
For instance, prior to restart the block was mapped to 8:32 and when re-attached it was on 8:16
The previous "block" is still 8:32 so if we don't override it we will try to initialize on a disk that is not an OSD or worse another OSD
This is mainly because in https://github.com/rook/rook/commit/ae8dcf7cc3b51cf8ca7da22f48b7a58887536c4f we switched to use HostPath to store the OSD data
Since HostPath is not ephemeral, the block file must be re-hydrated each time the deployment starts
The test uses "==" instead of numbers since the stat command will print
an hex number and thus we won't always get integer, e,g: major/minor
value 202:13312 will be return ca:3400.
Example run of the new init container:
+ PVC_SOURCE=/set1-data-168h7m
+ PVC_DEST=/var/lib/ceph/osd/ceph-0/block
+ CP_ARGS=(--archive --dereference --verbose)
+ '[' -b /var/lib/ceph/osd/ceph-0/block ']'
++ stat -c %t%T /set1-data-168h7m
+ PVC_SOURCE_MAJ_MIN=ca6100
++ stat -c %t%T /var/lib/ceph/osd/ceph-0/block
PVC's source major/minor numbers changed
removed '/var/lib/ceph/osd/ceph-0/block'
'/set1-data-168h7m' -> '/var/lib/ceph/osd/ceph-0/block'
+ PVC_DEST_MAJ_MIN=ca4000
+ [[ ca6100 == \c\a\4\0\0\0 ]]
+ echo 'PVC'\''s source major/minor numbers changed'
+ CP_ARGS+=(--remove-destination)
+ cp --archive --dereference --verbose --remove-destination /set1-data-168h7m /var/lib/ceph/osd/ceph-0/block
Test procedure:
$ oc -n rook-ceph scale deployment rook-ceph-osd-0 --replicas=2
deployment.apps/rook-ceph-osd-0 scaled
Now we have 2 OSDs and one is crashlooping, which is expected:
rook-ceph-osd-0-66964f6d-2cwqq 0/1 Init:CrashLoopBackOff 4 2m12s
rook-ceph-osd-0-66964f6d-pmhlq 1/1 Running 0 6m7s
The logs from the failing container is pretty clear:
+ PVC_SOURCE=/set1-data-168h7m
+ PVC_DEST=/var/lib/ceph/osd/ceph-0/block
+ CP_ARGS=(--archive --dereference --verbose)
+ '[' -b /var/lib/ceph/osd/ceph-0/block ']'
++ stat -c %t%T /set1-data-168h7m
+ PVC_SOURCE_MAJ_MIN=ca6100
++ stat -c %t%T /var/lib/ceph/osd/ceph-0/block
+ PVC_DEST_MAJ_MIN=ca6100
+ [[ ca6100 == \c\a\6\1\0\0 ]]
+ CP_ARGS+=(--no-clobber)
+ cp --archive --dereference --verbose --no-clobber /set1-data-168h7m /var/lib/ceph/osd/ceph-0/block
inferring bluefs devices from bluestore path
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.8/rpm/el8/BUILD/ceph-15.2.8/src/os/bluestore/BlueStore.cc: In function 'int BlueStore::expand_devices(std::ostream&)' thread 7f2a52d18240 time 2021-02-09T13:27:46.011397+0000
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.8/rpm/el8/BUILD/ceph-15.2.8/src/os/bluestore/BlueStore.cc: 7090: FAILED ceph_assert(r == 0)
2021-02-09T13:27:46.010+0000 7f2a52d18240 -1 bluestore(/var/lib/ceph/osd/ceph-0) _lock_fsid failed to lock /var/lib/ceph/osd/ceph-0/fsid (is another ceph-osd still running?)(11) Resource temporarily unavailable
ceph version 15.2.8 (bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55) octopus (stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f2a48e236e4]
Acquiring the lock fails, so the OSD won't run.
Now we scale back down:
$ oc -n rook-ceph scale deployment rook-ceph-osd-0 --replicas=1
deployment.apps/rook-ceph-osd-0 scaled
And we can see the last created OSD (the one crash looping) disappears:
rook-ceph-osd-0-66964f6d-2cwqq 0/1 Terminating 5 3m6s
rook-ceph-osd-0-66964f6d-pmhlq 1/1 Running 0 7m1s
Signed-off-by: Sébastien Han <seb@redhat.com>
Some settinc controller reference code can be replaced
with `controllerutil.SetControllerReference`. It's better to use it
since it provides ownerReference verification.
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
With 5f57b6fd3a, we introduced a
regression in the validation code.
If the OSD PV comes from another cluster, GetCephVolumeRawOSDs must
return an error.
The prepare job will crash loop but at least with a useful and
comprehensive message.
Signed-off-by: Sébastien Han <seb@redhat.com>
For the rbd-mirror daemon, We now rely on the "replica" setting in the Deployment spec when the
desired instance count is higher than 1 instead of creating another
deployment.
So now, we don't need to the number of deployments anymore and let
kubernetes handles the number of pods.
Signed-off-by: Sébastien Han <seb@redhat.com>
If Ceph pods were evicted onto other nodes, there is no reason for the
crash collector to remain on those nodes. So let's remove it.
Signed-off-by: Sébastien Han <seb@redhat.com>
The pod IP address is no longer needed for port binding for the dashboard
and prometheus endpoints anymore in the mgr modules. The default binding
to localhost will be sufficient. This will allow the endpoints to be more
reliable in some environments where today the podIP is not working.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Binding the dashboard and prometheus to all interfaces instead of the pod ip
will ensure that they are listening properly. Binding to the pod ip was only
a short term solution that was intended to be removed with Nautilus, which
is already Rook's min supported version.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
completeOSDsForAllNodes() method waits for 20 minutes for all the OSDs to start on all the nodes. This PR checks if
cluster deletion request was sent while waiting for OSDs to start. If yes, then completeOSDsForAllNodes() method will returnwithout waiting for 20 minutes.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
If the ceph cli outputs an error with "error calling conf_read_file"
this means that the operator has not written its ceph configuration
file. Thus ceph cli commands will fail, so we can just ignore that since
the operator will soon write this file in its initialization sequence.
Signed-off-by: Sébastien Han <seb@redhat.com>
If HostPID is enabled, this means the Ceph version is older than Octopus
and we are still running OSD with HostPID due to an internal
inter-process check.
If the log collector is enabled we need to enable ShareProcessNamespace
**only** if HostIPC is disabled.
Both HostPID and ShareProcessNamespace cannot be enabled.
Signed-off-by: Sébastien Han <seb@redhat.com>
If HostIPC is enabled, this means the Ceph version is older than Octopus
and we are still running OSD with HostIPC due to an internal
inter-process check.
If the log collector is enabled we need to enable ShareProcessNamespace
**only** if HostIPC is disabled.
Signed-off-by: Sébastien Han <seb@redhat.com>
With Ceph Pacific, Rook can now deploy the cephfs-mirror daemon.
This initial commit covers the deployment of a single daemon only.
Multiple mirror daemons is currently untested.
Only a single mirror daemon is recommended.
The configuration of peers will come in a later PR since the mgr module
is still pending upstream: https://github.com/ceph/ceph/pull/39050
The same goes for integration tests, they will get added later once we
start testing on Pacific.
Closes: https://github.com/rook/rook/issues/7002
Signed-off-by: Sébastien Han <seb@redhat.com>
Sébastien Han
Travis Nielsen
Blaine Gardner
Blaine Gardner
shenjiatong
rohan47
subhamkrai
Satoru Takeuchi
Santosh Pillai