If the cluster was expanded we would iterate over the entire list of
OSDs again, create a new key and override in Vault... The Secret store
is not impacted since we have a handler for "already exist".
Signed-off-by: Sébastien Han <seb@redhat.com>
The cleanup job will need to run wherever the ceph daemons
were running. The scheduling is done by rook on the specific
nodes where the daemons were running, so there is no need for
other placement such as node or pod affinity.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
This reverts commit 1ed307cb09. In
8aaff235bb we have introduced the ability
to set a specific monitoring port so the block that fails if the port is
not the default 9283 is not needed anymore.
Signed-off-by: Sébastien Han <seb@redhat.com>
we don't want to merge the node affinity if specified in both
storageClassDeviceSet and placement. Now, we are passing new
boolean argument in `ApplyToPodSpec` which will be false in
case of osd and prepare pods because we don't want to overlap
placement of osd on PVC's and non-PVC's.
Signed-off-by: subhamkrai <srai@redhat.com>
The multiple filesystems parameter needed to be quoted for kubernetes to accept the parameter, without quotes the environment variable is rejected.
Signed-off-by: Tom Hellier <me@tomhellier.com>
Add a new OSD provisioner status (reported by provisioner configmap)
that denotes that an OSD is preexisting and that OSD prepare does not
need to be run for the OSD. This only applies to OSDs on PVC currently
where existence of a deployment for the OSD indicates no further
provisioning needs to occur. New status is "preexisting".
Allow creating new OSDs on PVCs before updating existing ones by
allowing deferring OSDs during processing of OSD provisioning status
ConfigMaps. Deferral is identified by new "preexisting" status.
Add a unit test to ensure OSD on PVC provisioning occurs as expected
including deferring already-created OSDs to be updated after new PVCs
are created.
Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
this commit set default values or allow overriding of liveness probe
fields if that individual field is 0 or nil.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Signed-off-by: subhamkrai <srai@redhat.com>
Currently endpoint is filled from `Status.Info["endpoint"]` string map for cephobjectstore user's secret and
have two issues:
1.) endpoint is nil and only secureendpoint is available
2.) Chance of small race window in which endpoint is not filled before this ceph object user is handling.
Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
The Rancher Continuous Delivery Fleet tool expects a true Ready
condition to be listed after a false condition such as
ProgressingCompleted. This commit orders the false condition
before its corresponding true Ready condition.
Closes: https://github.com/rook/rook/issues/6977
Signed-off-by: ron1 <ron18219@gmail.com>
If the Vault server returns an error, we must report it in the init
container. With this we can properly debug a failing init container.
Resulting container log:
```
oc logs rook-ceph-osd-0-dc6d4d4d-445dg -c encryption-kms-get-kek ⏎
['error performing token check: Vault is sealed']%
```
Signed-off-by: Sébastien Han <seb@redhat.com>
this commit add docs for CSI Troubleshooting when
hit with the issue in Provisioning the CephFS/RBD PVC
or Mounting the CephFS/RBD PVC to application pods.
Signed-off-by: subhamkrai srai@redhat.com
co-authored-by: Madhu-1 madhupr007@gmail.com
Signed-off-by: subhamkrai <srai@redhat.com>
When the blkdevmapper init container initializes we must override any
existing special block. This will prevent re-using a different disk on
the OSD on PVC scenario.
Adding "--remove-destination" to the cp line will remove each existing
destination file before attempting to open it.
We **MUST** do this otherwise in environment where PVCs are dynamic, restarting the deployment will cause conflicts
When restarting the OSD, the PVC block might end up with a different Kernel disk allocation
For instance, prior to restart the block was mapped to 8:32 and when re-attached it was on 8:16
The previous "block" is still 8:32 so if we don't override it we will try to initialize on a disk that is not an OSD or worse another OSD
This is mainly because in https://github.com/rook/rook/commit/ae8dcf7cc3b51cf8ca7da22f48b7a58887536c4f we switched to use HostPath to store the OSD data
Since HostPath is not ephemeral, the block file must be re-hydrated each time the deployment starts
The test uses "==" instead of numbers since the stat command will print
an hex number and thus we won't always get integer, e,g: major/minor
value 202:13312 will be return ca:3400.
Example run of the new init container:
+ PVC_SOURCE=/set1-data-168h7m
+ PVC_DEST=/var/lib/ceph/osd/ceph-0/block
+ CP_ARGS=(--archive --dereference --verbose)
+ '[' -b /var/lib/ceph/osd/ceph-0/block ']'
++ stat -c %t%T /set1-data-168h7m
+ PVC_SOURCE_MAJ_MIN=ca6100
++ stat -c %t%T /var/lib/ceph/osd/ceph-0/block
PVC's source major/minor numbers changed
removed '/var/lib/ceph/osd/ceph-0/block'
'/set1-data-168h7m' -> '/var/lib/ceph/osd/ceph-0/block'
+ PVC_DEST_MAJ_MIN=ca4000
+ [[ ca6100 == \c\a\4\0\0\0 ]]
+ echo 'PVC'\''s source major/minor numbers changed'
+ CP_ARGS+=(--remove-destination)
+ cp --archive --dereference --verbose --remove-destination /set1-data-168h7m /var/lib/ceph/osd/ceph-0/block
Test procedure:
$ oc -n rook-ceph scale deployment rook-ceph-osd-0 --replicas=2
deployment.apps/rook-ceph-osd-0 scaled
Now we have 2 OSDs and one is crashlooping, which is expected:
rook-ceph-osd-0-66964f6d-2cwqq 0/1 Init:CrashLoopBackOff 4 2m12s
rook-ceph-osd-0-66964f6d-pmhlq 1/1 Running 0 6m7s
The logs from the failing container is pretty clear:
+ PVC_SOURCE=/set1-data-168h7m
+ PVC_DEST=/var/lib/ceph/osd/ceph-0/block
+ CP_ARGS=(--archive --dereference --verbose)
+ '[' -b /var/lib/ceph/osd/ceph-0/block ']'
++ stat -c %t%T /set1-data-168h7m
+ PVC_SOURCE_MAJ_MIN=ca6100
++ stat -c %t%T /var/lib/ceph/osd/ceph-0/block
+ PVC_DEST_MAJ_MIN=ca6100
+ [[ ca6100 == \c\a\6\1\0\0 ]]
+ CP_ARGS+=(--no-clobber)
+ cp --archive --dereference --verbose --no-clobber /set1-data-168h7m /var/lib/ceph/osd/ceph-0/block
inferring bluefs devices from bluestore path
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.8/rpm/el8/BUILD/ceph-15.2.8/src/os/bluestore/BlueStore.cc: In function 'int BlueStore::expand_devices(std::ostream&)' thread 7f2a52d18240 time 2021-02-09T13:27:46.011397+0000
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.8/rpm/el8/BUILD/ceph-15.2.8/src/os/bluestore/BlueStore.cc: 7090: FAILED ceph_assert(r == 0)
2021-02-09T13:27:46.010+0000 7f2a52d18240 -1 bluestore(/var/lib/ceph/osd/ceph-0) _lock_fsid failed to lock /var/lib/ceph/osd/ceph-0/fsid (is another ceph-osd still running?)(11) Resource temporarily unavailable
ceph version 15.2.8 (bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55) octopus (stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f2a48e236e4]
Acquiring the lock fails, so the OSD won't run.
Now we scale back down:
$ oc -n rook-ceph scale deployment rook-ceph-osd-0 --replicas=1
deployment.apps/rook-ceph-osd-0 scaled
And we can see the last created OSD (the one crash looping) disappears:
rook-ceph-osd-0-66964f6d-2cwqq 0/1 Terminating 5 3m6s
rook-ceph-osd-0-66964f6d-pmhlq 1/1 Running 0 7m1s
Signed-off-by: Sébastien Han <seb@redhat.com>
Some settinc controller reference code can be replaced
with `controllerutil.SetControllerReference`. It's better to use it
since it provides ownerReference verification.
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
Add missing OVC on LV-backed PVC test. Although we already have
the pvc test case, the code path of lv-backed pvc is a bit different
from raw device.
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
With 5f57b6fd3a, we introduced a
regression in the validation code.
If the OSD PV comes from another cluster, GetCephVolumeRawOSDs must
return an error.
The prepare job will crash loop but at least with a useful and
comprehensive message.
Signed-off-by: Sébastien Han <seb@redhat.com>
For the rbd-mirror daemon, We now rely on the "replica" setting in the Deployment spec when the
desired instance count is higher than 1 instead of creating another
deployment.
So now, we don't need to the number of deployments anymore and let
kubernetes handles the number of pods.
Signed-off-by: Sébastien Han <seb@redhat.com>
Add the cephfilesystemmirrors definition for CI runs where the
Kubernetes version is 1.15 and extension v1 extensions are not
supported.
Closes: https://github.com/rook/rook/issues/7168
Signed-off-by: Sébastien Han <seb@redhat.com>
If Ceph pods were evicted onto other nodes, there is no reason for the
crash collector to remain on those nodes. So let's remove it.
Signed-off-by: Sébastien Han <seb@redhat.com>
Blaine Gardner
Travis Nielsen
Sébastien Han
Varsha Rao
Alexander Trost
subhamkrai
Tom Hellier
Blaine Gardner
Jiffin Tony Thottan
ron1
Satoru Takeuchi