The OSDs on PVCs were changing location in the CRUSH
hierarchy after an operator restart due to the location
property not being re-discovered on the OSD daemon.
Since OSDs on PVCs do not run the prepare pod again
to discover this property, the operator must read it
off the existing osd deployment. The operator was
reading other properties, but now will read the location
as well.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The isUpgrade flag was being set every time the operator
was started. This caused the upgrade checks to be run every
time the operator restarted. Thus, if any mons were down,
you likely would get stuck on the upgrade checks since another
mon could not be stopped. We need to rely on the upgrade being
detected either by a different image name from the cluster CR
update, or from different versions of the daemons running
in case the operator was restarted in the middle of an upgrade.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Enabling pg_autoscalar module fails most probably becasue `ceph mgr module enable <>` is happening very quickly after ceph mgr starts.
This PR adds a retry (5 times) with a sleep interval of 5 seconds (for each interval) to handle failure while enabling mgr modules.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
This commit adds status field for ceph related CRs which will be useful for knowing the ceph component status without checking the logs.
Signed-off-by: Ashish Ranjan <ashishranjan738@gmail.com>
CSI nodeplugins, specifically when using cephfs FUSE or
rbd-nbd as the mounters, when upgraded, will
cause existing mounts to become stale/not-rechable
(usually connection timeout errors).
This is due to losing the mount processes running within the
CSI nodeplugin pods.
This PR add updated the Daemonset update strategy
based on the ENV variable to take care of above issue
with some manual steps
Moreinfo: https://github.com/ceph/ceph-csi/issues/703
Resolves: https://github.com/rook/rook/issues/4248
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
Reduce from 5 minutes to less than 50 seconds the time needed to restart a
mgr and osds(PVCs based),rgw,mds,rbd,nfs and toolbox pod that was running
in a k8s "NotReady" Node.
A new Rook Operator setting has been added to allow the user to set the
time used in pod's toleration <node.kubernetes.io/unreachable>:
ROOK_UNREACHABLE_NODE_TOLERATION_SECONDS = <5>
New setting added in release notes
Addressed @travisn,@leseb, and @blaineEXE suggestions
Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
There are two issues at odds with the cephfs driver. The
first is that older kernels less than 4.17 do not support
quotas so PVC sizes will not be enforced with the kernel
driver in that case. If the FUSE client is used instead, there
are issues during upgrade because the client mount will be
disrupted and the client pods will have to restart.
To prevent client pods from breaking during upgrade, we enable
the kernel driver by default. If the quotas are critical,
the admin will need to take steps during upgrade to prevent
breaking the applications.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
In ceph-csi v1.2.2 we have add a flag `forcecephkernelclient`
to force use Ceph Kernel clients on kernel < 4.17 which
support quotas for Cephfs
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
When using the "errors" package, using `%+v` (extended format),
each Frame of the error's StackTrace will be printed in detail.
Let's only print `%v` to print the error.
If the error has a Cause it will be printed recursively.
Basically `%+v` has been replaced with `%v` for all `error` type
interfaces, whether the logger is Info, Warning or Error.
Signed-off-by: Sébastien Han <seb@redhat.com>
gp2 devices present themselves as SSDs,
but have much more variable performance.
Ceph attempts to adjust some settings based on the drive type,
HDD/SSD
In this case client I/O can be impacted too much by assuming gp2 devices are like other SSDs
(an example is https://bugzilla.redhat.com/show_bug.cgi?id=1765923#c6)
In order to prevent this, rook can set the following for OSDs
if the new CR setting `tuneSlowDeviceClass` is set to `true`:
```
osd_recovery_sleep = 0.1
osd_snap_trim_sleep = 2
osd_delete_sleep = 2
```
Closes: https://github.com/rook/rook/issues/4298 and https://bugzilla.redhat.com/show_bug.cgi?id=1771047
Signed-off-by: Sébastien Han <seb@redhat.com>
An owner reference cannot be set with BlockOwnerDeletion: true
if the operator does not have privs to add a finalizer to the parent.
The discover resources don't need to block deletion based on the
parent so we remove that property from the owner reference.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Removing an OSD unintentionally can result in data loss,
which is to be prevented at all costs. Rook's functionality
to remove OSDs automatically when a node was removed from the
cluster CR has led on several occasions to users having OSDs
unintentionally removed from their cluster.
If the admin decides they really need to remove OSDs they should
do it as a one-time action that they intentionally run
rather than the operator attempting to manage the deletion
with desired state. The risk of misinterpreting the desired
state, even with a typo, is too great.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
It's not 100% safe since before Nautilus osd safe-to-destroy command
didn't have any json output. So added a version check.
Signed-off-by: rohan47 <rohgupta@redhat.com>
Add owner reference to the discover config map so that when the
discover pod goes away we also remove its configmap.
Signed-off-by: Sébastien Han <seb@redhat.com>
Attach owner reference to the discover daemon set so that when the
opetator goes away, the discover pod gets removed too.
Signed-off-by: Sébastien Han <seb@redhat.com>
As of 14.2.5, the crash collector pod will use the client.crash user
instead of the client.admin keyring to post crash dumps.
Closes: https://github.com/rook/rook/issues/4388
Signed-off-by: Sébastien Han <seb@redhat.com>
Both --id and --name do the same thing, so let's use one only: --id
which is shared accross all the daemons already.
Signed-off-by: Sébastien Han <seb@redhat.com>
rgw needs the fronted on its cli line to select the fronted as well the
port. This option is not supported in the mon config store.
Closes: https://github.com/rook/rook/issues/4471
Signed-off-by: Sébastien Han <seb@redhat.com>
As performed with mon, gateways from same ceph object store
must not run on the same host if hostNetwork is set to true
Signed-off-by: n.fraison <n.fraison@criteo.com>
If editing the cluster CR we must update the spec to reflect its new
content. Now the spec is updated so we can update with an unsupported
image as well as setting allowUnsupported: true.
It was not possible before we had to restart the operator, now we can do
it as an update.
Signed-off-by: Sébastien Han <seb@redhat.com>
We must not default the selectors to the labels since labels can be
changed and selector cannot until the deployment is updated.
This way the deployment was failing with:
2019-12-10 10:48:13.414438 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Deployment.apps "rook-ceph-crashcollector-minikube" is invalid: spec.template.metadata.labels: Invalid value: map[string]string{"app":"rook-ceph-crashcollector", "ceph-version":"15.0.0", "ceph_daemon_id":"crash", "crashcollector":"crash", "kubernetes.io/hostname":"minikube", "node_name":"minikube", "rook-version":"v1.2.0-3.g8d3b85306.dirty"}: `selector` does not match template `labels`
Signed-off-by: Sébastien Han <seb@redhat.com>
The loop wasn't using seconds for its retry so the crash controller
would simply crash since the version is used later in the pod spec.
Signed-off-by: Sébastien Han <seb@redhat.com>
Added default regex values.
Created an environment variable and using that variable the user can give custom regex values
Updated the PendingReleaseNotes.md
Signed-off-by: Nizamudeen <nia@redhat.com>
Just as with the mon, mgr, mds, rgw, and rbd-mirror daemons, do not
generate a ceph.conf in an init container for filestore OSDs.
Instead use the commandline to supply all the needed arguments for
running these OSDs.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
This change adds rook-version and ceph-version tags to crash
collector deployments, in keeping with other Ceph deployments.
Resolves: #4357
Signed-off-by: Elise Gafford <egafford@redhat.com>
Travis Nielsen
Sébastien Han
Santosh Pillai
Ashish Ranjan
Madhu Rajanna
Juan Miguel Olmo Martínez
Dmitry Yusupov
Anton Skriptsov
Blaine Gardner
rohan47
Mark Kogan
n.fraison
Nizamudeen
Elise Gafford