If the Vault server returns an error, we must report it in the init
container. With this we can properly debug a failing init container.
Resulting container log:
```
oc logs rook-ceph-osd-0-dc6d4d4d-445dg -c encryption-kms-get-kek ⏎
['error performing token check: Vault is sealed']%
```
Signed-off-by: Sébastien Han <seb@redhat.com>
When the blkdevmapper init container initializes we must override any
existing special block. This will prevent re-using a different disk on
the OSD on PVC scenario.
Adding "--remove-destination" to the cp line will remove each existing
destination file before attempting to open it.
We **MUST** do this otherwise in environment where PVCs are dynamic, restarting the deployment will cause conflicts
When restarting the OSD, the PVC block might end up with a different Kernel disk allocation
For instance, prior to restart the block was mapped to 8:32 and when re-attached it was on 8:16
The previous "block" is still 8:32 so if we don't override it we will try to initialize on a disk that is not an OSD or worse another OSD
This is mainly because in https://github.com/rook/rook/commit/ae8dcf7cc3b51cf8ca7da22f48b7a58887536c4f we switched to use HostPath to store the OSD data
Since HostPath is not ephemeral, the block file must be re-hydrated each time the deployment starts
The test uses "==" instead of numbers since the stat command will print
an hex number and thus we won't always get integer, e,g: major/minor
value 202:13312 will be return ca:3400.
Example run of the new init container:
+ PVC_SOURCE=/set1-data-168h7m
+ PVC_DEST=/var/lib/ceph/osd/ceph-0/block
+ CP_ARGS=(--archive --dereference --verbose)
+ '[' -b /var/lib/ceph/osd/ceph-0/block ']'
++ stat -c %t%T /set1-data-168h7m
+ PVC_SOURCE_MAJ_MIN=ca6100
++ stat -c %t%T /var/lib/ceph/osd/ceph-0/block
PVC's source major/minor numbers changed
removed '/var/lib/ceph/osd/ceph-0/block'
'/set1-data-168h7m' -> '/var/lib/ceph/osd/ceph-0/block'
+ PVC_DEST_MAJ_MIN=ca4000
+ [[ ca6100 == \c\a\4\0\0\0 ]]
+ echo 'PVC'\''s source major/minor numbers changed'
+ CP_ARGS+=(--remove-destination)
+ cp --archive --dereference --verbose --remove-destination /set1-data-168h7m /var/lib/ceph/osd/ceph-0/block
Test procedure:
$ oc -n rook-ceph scale deployment rook-ceph-osd-0 --replicas=2
deployment.apps/rook-ceph-osd-0 scaled
Now we have 2 OSDs and one is crashlooping, which is expected:
rook-ceph-osd-0-66964f6d-2cwqq 0/1 Init:CrashLoopBackOff 4 2m12s
rook-ceph-osd-0-66964f6d-pmhlq 1/1 Running 0 6m7s
The logs from the failing container is pretty clear:
+ PVC_SOURCE=/set1-data-168h7m
+ PVC_DEST=/var/lib/ceph/osd/ceph-0/block
+ CP_ARGS=(--archive --dereference --verbose)
+ '[' -b /var/lib/ceph/osd/ceph-0/block ']'
++ stat -c %t%T /set1-data-168h7m
+ PVC_SOURCE_MAJ_MIN=ca6100
++ stat -c %t%T /var/lib/ceph/osd/ceph-0/block
+ PVC_DEST_MAJ_MIN=ca6100
+ [[ ca6100 == \c\a\6\1\0\0 ]]
+ CP_ARGS+=(--no-clobber)
+ cp --archive --dereference --verbose --no-clobber /set1-data-168h7m /var/lib/ceph/osd/ceph-0/block
inferring bluefs devices from bluestore path
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.8/rpm/el8/BUILD/ceph-15.2.8/src/os/bluestore/BlueStore.cc: In function 'int BlueStore::expand_devices(std::ostream&)' thread 7f2a52d18240 time 2021-02-09T13:27:46.011397+0000
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.8/rpm/el8/BUILD/ceph-15.2.8/src/os/bluestore/BlueStore.cc: 7090: FAILED ceph_assert(r == 0)
2021-02-09T13:27:46.010+0000 7f2a52d18240 -1 bluestore(/var/lib/ceph/osd/ceph-0) _lock_fsid failed to lock /var/lib/ceph/osd/ceph-0/fsid (is another ceph-osd still running?)(11) Resource temporarily unavailable
ceph version 15.2.8 (bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55) octopus (stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f2a48e236e4]
Acquiring the lock fails, so the OSD won't run.
Now we scale back down:
$ oc -n rook-ceph scale deployment rook-ceph-osd-0 --replicas=1
deployment.apps/rook-ceph-osd-0 scaled
And we can see the last created OSD (the one crash looping) disappears:
rook-ceph-osd-0-66964f6d-2cwqq 0/1 Terminating 5 3m6s
rook-ceph-osd-0-66964f6d-pmhlq 1/1 Running 0 7m1s
Signed-off-by: Sébastien Han <seb@redhat.com>
Some settinc controller reference code can be replaced
with `controllerutil.SetControllerReference`. It's better to use it
since it provides ownerReference verification.
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
With 5f57b6fd3a, we introduced a
regression in the validation code.
If the OSD PV comes from another cluster, GetCephVolumeRawOSDs must
return an error.
The prepare job will crash loop but at least with a useful and
comprehensive message.
Signed-off-by: Sébastien Han <seb@redhat.com>
For the rbd-mirror daemon, We now rely on the "replica" setting in the Deployment spec when the
desired instance count is higher than 1 instead of creating another
deployment.
So now, we don't need to the number of deployments anymore and let
kubernetes handles the number of pods.
Signed-off-by: Sébastien Han <seb@redhat.com>
If Ceph pods were evicted onto other nodes, there is no reason for the
crash collector to remain on those nodes. So let's remove it.
Signed-off-by: Sébastien Han <seb@redhat.com>
The pod IP address is no longer needed for port binding for the dashboard
and prometheus endpoints anymore in the mgr modules. The default binding
to localhost will be sufficient. This will allow the endpoints to be more
reliable in some environments where today the podIP is not working.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Binding the dashboard and prometheus to all interfaces instead of the pod ip
will ensure that they are listening properly. Binding to the pod ip was only
a short term solution that was intended to be removed with Nautilus, which
is already Rook's min supported version.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
completeOSDsForAllNodes() method waits for 20 minutes for all the OSDs to start on all the nodes. This PR checks if
cluster deletion request was sent while waiting for OSDs to start. If yes, then completeOSDsForAllNodes() method will returnwithout waiting for 20 minutes.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
If the ceph cli outputs an error with "error calling conf_read_file"
this means that the operator has not written its ceph configuration
file. Thus ceph cli commands will fail, so we can just ignore that since
the operator will soon write this file in its initialization sequence.
Signed-off-by: Sébastien Han <seb@redhat.com>
If HostPID is enabled, this means the Ceph version is older than Octopus
and we are still running OSD with HostPID due to an internal
inter-process check.
If the log collector is enabled we need to enable ShareProcessNamespace
**only** if HostIPC is disabled.
Both HostPID and ShareProcessNamespace cannot be enabled.
Signed-off-by: Sébastien Han <seb@redhat.com>
If HostIPC is enabled, this means the Ceph version is older than Octopus
and we are still running OSD with HostIPC due to an internal
inter-process check.
If the log collector is enabled we need to enable ShareProcessNamespace
**only** if HostIPC is disabled.
Signed-off-by: Sébastien Han <seb@redhat.com>
With Ceph Pacific, Rook can now deploy the cephfs-mirror daemon.
This initial commit covers the deployment of a single daemon only.
Multiple mirror daemons is currently untested.
Only a single mirror daemon is recommended.
The configuration of peers will come in a later PR since the mgr module
is still pending upstream: https://github.com/ceph/ceph/pull/39050
The same goes for integration tests, they will get added later once we
start testing on Pacific.
Closes: https://github.com/rook/rook/issues/7002
Signed-off-by: Sébastien Han <seb@redhat.com>
when a node fails some pods remains in terminating while other manage to
be rescheduled on another node successfully.
this commit adds code to clean up all the Rook and CSI pods that are stuck in terminating
state on a failed node.
Signed-off-by: rohan47 <rohgupta@redhat.com>
The maxMonId should only track the mons that have been committed to starting.
Currently the maxMonId was tracking all mon scheduling. If the reconcile
was aborted or failed, or the operator restarted in a reconcile, the
maxMonId was being reset by analyzing the mon deployments that existed.
This does not work for scenarios where mon quroum is lost and being
reset to a single mon. Now the maxMonId will only be incremented
when a mon daemon is started, and will never be decremented after
that to ensure that new mons will be incremented to the next id.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
When the node is not found for mon assignment with host networking
enabled, the operator may crash. This change ensures the node is
not nil and returns an error instead of crashing.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
added waitTimeoutForHealthyOSD in the cluster crd that defines the time (in minutes) the operator would wait before an OSD can be stopped for
upgrade. This PR also removes the ok-to-continue logic for osds as its already handled by ok-to-stop. The default value is 10 minutes.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
The settings for tuneDeviceClass and tuneFastDeviceClass were
stored as a map[string]string. When iterating over the map to
append the args, the args were being appended in an unpredictable
order. The order could change every reconcile, resulting in an
unnecessary OSD restart potentially every reconcile. Now we iterate
over a slice that will be ordered predictably and ensure OSDs
will not be restarted when not needed.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
OSDs on PVCs previously were always replaced with PVCs of a given name.
For on-prem scenarios where the OSDs might need to be removed instead
of replaced, the operator now allows holes to exist in the PVC index
names as long as there are a sufficient number of PVCs to meet the
criteria for the deviceSet.count.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
If the CephCluster CR spec is updated to activate the logCollector, we
must reflect that change onto child CRDs, like the mds and rgw since
their configurationn would be impacted too.
Now we watch for the CephCluster object changes from the object/file
controllers and react upon the appropriate event.
Closes: https://github.com/rook/rook/issues/7022
Signed-off-by: Sébastien Han <seb@redhat.com>
Multiple OSD pods for the same OSD might run simultaneously becasue OSD pod
is managed by Deployment resource. However, OSD locking mechanism
doens't work because the lock file (fsid file) exist for each OSD pod.
It resutls in OSD corruption.
Closes: https://github.com/rook/rook/issues/6530
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
This was the last remaining CRD to not use the controller-runtime
library.
Small additions were added with the transition:
* the Kubernetes Secret that contains the CephX key has now an owner
reference to the CephClient object
* the secret name is present in the Status field of the CephClient:
```
status:
info:
secretName: rook-ceph-client-glance
phase: Ready
```
The controller will reconcile on CR updates and also if the Kubernetes
Secret is deleted.
Closes: https://github.com/rook/rook/issues/4938
Signed-off-by: Sébastien Han <seb@redhat.com>
By setting the working directory of a Ceph daemon to the log direcor
(which is bindmounted to the host), we can ensure that the coredumps
will be available.
On CentOS 7 **only**, the kernel is configured with:
```
cat /proc/sys/kernel/core_pattern
core
```
which means that the coredumps will end up in the process working
directory and will be named "core".
On CentOS 8, everything is different and this won't work but still
CentOS will be able to consume this and changing the working directory
is harmless.
Signed-off-by: Sébastien Han <seb@redhat.com>
If ssl was not enabled, the dashboard was being restarted with every
reconcile. Instead, the dashboard should only be restarted when
the settings have changed in order to reduce the frequency that
the dashboard is restarted.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
`.spec.resources.osd` resources and `placement: all:` placements
parameters for the OSD pod did not reflect the value like
it has for other pods mon,mgr.
this commit will allow to set those parameters for the OSD pod also.
Signed-off-by: subhamkrai <srai@redhat.com>
The operator namespace is not used in the StartOperatorSettingsWatch()
method. Instead, the method looks up the namespace from the
POD_NAMESPACE env var.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Currently, mgr deployment has two overlapping environment variables, ROOK_POD_IP.
This causes kube-apiserver to create error logs
Signed-off-by: binoue <banji-inoue@cybozu.co.jp>
Sébastien Han
Satoru Takeuchi
Travis Nielsen
Santosh Pillai
Blaine Gardner
rohan47
Jiffin Tony Thottan
shenjiatong
subhamkrai
binoue