If the Vault server returns an error, we must report it in the init
container. With this we can properly debug a failing init container.
Resulting container log:
```
oc logs rook-ceph-osd-0-dc6d4d4d-445dg -c encryption-kms-get-kek ⏎
['error performing token check: Vault is sealed']%
```
Signed-off-by: Sébastien Han <seb@redhat.com>
When the blkdevmapper init container initializes we must override any
existing special block. This will prevent re-using a different disk on
the OSD on PVC scenario.
Adding "--remove-destination" to the cp line will remove each existing
destination file before attempting to open it.
We **MUST** do this otherwise in environment where PVCs are dynamic, restarting the deployment will cause conflicts
When restarting the OSD, the PVC block might end up with a different Kernel disk allocation
For instance, prior to restart the block was mapped to 8:32 and when re-attached it was on 8:16
The previous "block" is still 8:32 so if we don't override it we will try to initialize on a disk that is not an OSD or worse another OSD
This is mainly because in https://github.com/rook/rook/commit/ae8dcf7cc3b51cf8ca7da22f48b7a58887536c4f we switched to use HostPath to store the OSD data
Since HostPath is not ephemeral, the block file must be re-hydrated each time the deployment starts
The test uses "==" instead of numbers since the stat command will print
an hex number and thus we won't always get integer, e,g: major/minor
value 202:13312 will be return ca:3400.
Example run of the new init container:
+ PVC_SOURCE=/set1-data-168h7m
+ PVC_DEST=/var/lib/ceph/osd/ceph-0/block
+ CP_ARGS=(--archive --dereference --verbose)
+ '[' -b /var/lib/ceph/osd/ceph-0/block ']'
++ stat -c %t%T /set1-data-168h7m
+ PVC_SOURCE_MAJ_MIN=ca6100
++ stat -c %t%T /var/lib/ceph/osd/ceph-0/block
PVC's source major/minor numbers changed
removed '/var/lib/ceph/osd/ceph-0/block'
'/set1-data-168h7m' -> '/var/lib/ceph/osd/ceph-0/block'
+ PVC_DEST_MAJ_MIN=ca4000
+ [[ ca6100 == \c\a\4\0\0\0 ]]
+ echo 'PVC'\''s source major/minor numbers changed'
+ CP_ARGS+=(--remove-destination)
+ cp --archive --dereference --verbose --remove-destination /set1-data-168h7m /var/lib/ceph/osd/ceph-0/block
Test procedure:
$ oc -n rook-ceph scale deployment rook-ceph-osd-0 --replicas=2
deployment.apps/rook-ceph-osd-0 scaled
Now we have 2 OSDs and one is crashlooping, which is expected:
rook-ceph-osd-0-66964f6d-2cwqq 0/1 Init:CrashLoopBackOff 4 2m12s
rook-ceph-osd-0-66964f6d-pmhlq 1/1 Running 0 6m7s
The logs from the failing container is pretty clear:
+ PVC_SOURCE=/set1-data-168h7m
+ PVC_DEST=/var/lib/ceph/osd/ceph-0/block
+ CP_ARGS=(--archive --dereference --verbose)
+ '[' -b /var/lib/ceph/osd/ceph-0/block ']'
++ stat -c %t%T /set1-data-168h7m
+ PVC_SOURCE_MAJ_MIN=ca6100
++ stat -c %t%T /var/lib/ceph/osd/ceph-0/block
+ PVC_DEST_MAJ_MIN=ca6100
+ [[ ca6100 == \c\a\6\1\0\0 ]]
+ CP_ARGS+=(--no-clobber)
+ cp --archive --dereference --verbose --no-clobber /set1-data-168h7m /var/lib/ceph/osd/ceph-0/block
inferring bluefs devices from bluestore path
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.8/rpm/el8/BUILD/ceph-15.2.8/src/os/bluestore/BlueStore.cc: In function 'int BlueStore::expand_devices(std::ostream&)' thread 7f2a52d18240 time 2021-02-09T13:27:46.011397+0000
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.8/rpm/el8/BUILD/ceph-15.2.8/src/os/bluestore/BlueStore.cc: 7090: FAILED ceph_assert(r == 0)
2021-02-09T13:27:46.010+0000 7f2a52d18240 -1 bluestore(/var/lib/ceph/osd/ceph-0) _lock_fsid failed to lock /var/lib/ceph/osd/ceph-0/fsid (is another ceph-osd still running?)(11) Resource temporarily unavailable
ceph version 15.2.8 (bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55) octopus (stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f2a48e236e4]
Acquiring the lock fails, so the OSD won't run.
Now we scale back down:
$ oc -n rook-ceph scale deployment rook-ceph-osd-0 --replicas=1
deployment.apps/rook-ceph-osd-0 scaled
And we can see the last created OSD (the one crash looping) disappears:
rook-ceph-osd-0-66964f6d-2cwqq 0/1 Terminating 5 3m6s
rook-ceph-osd-0-66964f6d-pmhlq 1/1 Running 0 7m1s
Signed-off-by: Sébastien Han <seb@redhat.com>
Some settinc controller reference code can be replaced
with `controllerutil.SetControllerReference`. It's better to use it
since it provides ownerReference verification.
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
With 5f57b6fd3a, we introduced a
regression in the validation code.
If the OSD PV comes from another cluster, GetCephVolumeRawOSDs must
return an error.
The prepare job will crash loop but at least with a useful and
comprehensive message.
Signed-off-by: Sébastien Han <seb@redhat.com>
For the rbd-mirror daemon, We now rely on the "replica" setting in the Deployment spec when the
desired instance count is higher than 1 instead of creating another
deployment.
So now, we don't need to the number of deployments anymore and let
kubernetes handles the number of pods.
Signed-off-by: Sébastien Han <seb@redhat.com>
Add the cephfilesystemmirrors definition for CI runs where the
Kubernetes version is 1.15 and extension v1 extensions are not
supported.
Closes: https://github.com/rook/rook/issues/7168
Signed-off-by: Sébastien Han <seb@redhat.com>
If Ceph pods were evicted onto other nodes, there is no reason for the
crash collector to remain on those nodes. So let's remove it.
Signed-off-by: Sébastien Han <seb@redhat.com>
nfs server requires valid pool for storing nfs-ganesha recovery objects. Only
when filesystem exports are created, filesystem should be configured first.
Signed-off-by: Varsha Rao <varao@redhat.com>
The YugabyteDB team has decided to move support of their operator
to a new repo at https://github.com/yugabyte/yugabyte-operator.
The operator in Rook is no longer needed. Further usage of
the YugabyteDB operator is recommended at the new location.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The pod IP address is no longer needed for port binding for the dashboard
and prometheus endpoints anymore in the mgr modules. The default binding
to localhost will be sufficient. This will allow the endpoints to be more
reliable in some environments where today the podIP is not working.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Binding the dashboard and prometheus to all interfaces instead of the pod ip
will ensure that they are listening properly. Binding to the pod ip was only
a short term solution that was intended to be removed with Nautilus, which
is already Rook's min supported version.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
completeOSDsForAllNodes() method waits for 20 minutes for all the OSDs to start on all the nodes. This PR checks if
cluster deletion request was sent while waiting for OSDs to start. If yes, then completeOSDsForAllNodes() method will returnwithout waiting for 20 minutes.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
integration test was failing frequently due to some
intermittent failures.
```
Error from server (InternalError): error when creating
"STDIN": Internal error occurred: failed calling webhook.
```
(This error is not coming from us most probably)
This is kind of workaround so we get green build
as we still don't know the root cause.
Signed-off-by: subhamkrai <srai@redhat.com>
This PR makes create and delete events on pdb a no-op. Update event only triggers reconcile if its the main OSD pdb and allowed disruptions is 0.
This prevents controller to reconcile on pdb events namespaces outside rook ceph.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
Somehow the combined output does not contain the stderr in the errorn
only in the buffer.
For example:
err = "error exit 1"
buf = "Error EBUSY: not enough monitors would be available () after stopping mons [a]"
So we now embed the buf in the error so that the caller does not need to
print the buffer as well as the error.
Also, the caller can more easily decide to ignore the error by
instropecting the string.
Signed-off-by: Sébastien Han <seb@redhat.com>
If the ceph cli outputs an error with "error calling conf_read_file"
this means that the operator has not written its ceph configuration
file. Thus ceph cli commands will fail, so we can just ignore that since
the operator will soon write this file in its initialization sequence.
Signed-off-by: Sébastien Han <seb@redhat.com>
when extra-create-metadata is passed to csi-provisioner
it passes few information like namespace name,pvc
name to the cephcsi driver which is required for
rbd encryption.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
The log-collector container name was exceeding the 63-char limit
if the parent CR name was too long. The container name just needs
to be unique to the pod spec, so we simplify it to remove the parent
name and avoid hitting the limit in the pod spec.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The deviceClass was missing from the object store CRD schema,
disallowing the device class from being specified on
the object store pools.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Sébastien Han
Satoru Takeuchi
Travis Nielsen
Varsha Rao
ᗪєνιη ᗷυнʟ
Santosh Pillai
subhamkrai
Madhu Rajanna
Blaine Gardner