When configuring a cephcluster CRD on a different namespace than rook-ceph, like for example when deploying a external cluster, we need to set ROOK_CURRENT_NAMESPACE_ONLY to true so the operator can watch for CRD modifications on other namespaces.
Closes: https://github.com/rook/rook/issues/4667
Signed-off-by: Daniel Parkes <dparkes@redhat.com>
currently we are running plugin container as privileged
in the provisioner pod for both cephfs and rbd.
we are also running snapshotter container as privileged
in the provisioner pod for rbd.
as we are executing the ceph and rbd commands in provisioner
pod we dont need to be privileged, This PR removed the
privileged capability from the templates.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
currently we are making use of hostpath
directory to store the provisioner socket, as this
socket is not needed by anyone else other than containers
inside the provisioner pod using empty directory
to store this socket is the best option.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
On systems with SELinux enabled, non-privileged containers
can't access data of privileged containers.
Since the socket is exposed by privileged containers,
all sidecars must be privileged too.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
The minio operator has not had community support nor
any updates since being added to Rook. Support is being
removed from Rook due to this lack of community interest.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
A new CRD option `continueUpgradeAfterChecksEvenIfNotHealthy` is added.
When upgrading, Rook goes OSD by OSD and then waits for PGs to be clean
before proceeding to the next OSD. Currently, Rook waits for 5 hours but
there might be circumstances where PGs need more time to settle.
Thus setting `continueUpgradeAfterChecksEvenIfNotHealthy` to true will
pursue the upgrade process, even if PGs are not 100% active+clean.
Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1786029
Signed-off-by: Sébastien Han <seb@redhat.com>
* bumping ceph-operator memory limit from 128Mi to 256Mi in order to
avoid OOMkills of the rook-ceph-operator pod
* updating documentation example for operator to denote a limit of 256Mi
instead of 128Mi
When upgrading from rook v1.1.7 to v1.2.0, the rook-ceph-operator pod
continued to get OOMKilled by the kubernetes scheduler. Manually
bumping this limit to something higher than 128Mi (I tried 256Mi and it
was good) prevented the OOMKills.
Signed-off-by: Jeff Billimek <jeff@billimek.com>
If using ceph-fuse or rbd-nbd the application pods will need
to be restarted during upgrade. This adds a description of
the steps necessary to ensure the applications remain healthy
during upgrade.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
CSI nodeplugins, specifically when using cephfs FUSE or
rbd-nbd as the mounters, when upgraded, will
cause existing mounts to become stale/not-rechable
(usually connection timeout errors).
This is due to losing the mount processes running within the
CSI nodeplugin pods.
This PR add updated the Daemonset update strategy
based on the ENV variable to take care of above issue
with some manual steps
Moreinfo: https://github.com/ceph/ceph-csi/issues/703
Resolves: https://github.com/rook/rook/issues/4248
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
Reduce from 5 minutes to less than 50 seconds the time needed to restart a
mgr and osds(PVCs based),rgw,mds,rbd,nfs and toolbox pod that was running
in a k8s "NotReady" Node.
A new Rook Operator setting has been added to allow the user to set the
time used in pod's toleration <node.kubernetes.io/unreachable>:
ROOK_UNREACHABLE_NODE_TOLERATION_SECONDS = <5>
New setting added in release notes
Addressed @travisn,@leseb, and @blaineEXE suggestions
Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
There are two issues at odds with the cephfs driver. The
first is that older kernels less than 4.17 do not support
quotas so PVC sizes will not be enforced with the kernel
driver in that case. If the FUSE client is used instead, there
are issues during upgrade because the client mount will be
disrupted and the client pods will have to restart.
To prevent client pods from breaking during upgrade, we enable
the kernel driver by default. If the quotas are critical,
the admin will need to take steps during upgrade to prevent
breaking the applications.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
In ceph-csi v1.2.2 we have add a flag `forcecephkernelclient`
to force use Ceph Kernel clients on kernel < 4.17 which
support quotas for Cephfs
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
Removed some obsolete instructions from the 1.1 upgrade
and made a few minor fixes to make the upgrade to 1.2 smoother.
With the addition of the CephClient CRD, the CRD needs to be
created before the upgrade or else the operator log will be
filled with errors until it is created. The CSI driver also
had an error when registering after the upgrade due to an RBAC
change.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
gp2 devices present themselves as SSDs,
but have much more variable performance.
Ceph attempts to adjust some settings based on the drive type,
HDD/SSD
In this case client I/O can be impacted too much by assuming gp2 devices are like other SSDs
(an example is https://bugzilla.redhat.com/show_bug.cgi?id=1765923#c6)
In order to prevent this, rook can set the following for OSDs
if the new CR setting `tuneSlowDeviceClass` is set to `true`:
```
osd_recovery_sleep = 0.1
osd_snap_trim_sleep = 2
osd_delete_sleep = 2
```
Closes: https://github.com/rook/rook/issues/4298 and https://bugzilla.redhat.com/show_bug.cgi?id=1771047
Signed-off-by: Sébastien Han <seb@redhat.com>
Added default regex values.
Created an environment variable and using that variable the user can give custom regex values
Updated the PendingReleaseNotes.md
Signed-off-by: Nizamudeen <nia@redhat.com>
ceph client CRD refactoring
Add info about client crd to documentation
Add tests for client controller CRD
Signed-off-by: Mateusz Los <los.mateusz@gmail.com>
The deviceFilter property needs to have a value rather
than be completely empty as it was in the settings.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
There is use case when need to run rook operator on host network since
ceph mon is not accessible from kubernetes pod network.
Signed-off-by: ruslanloman <ruslanloman@gmail.com>
we need to set a ENV `CSI_RBD_ENABLE_SNAPSHOTTER`
variable to "false" in operator template to disable
deployment of snapshotter sidecar in ceph-csi rbd
provisioner.
Fixes: #4401
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
cephcsi v1.2.2 release is out and the image is available
for use, updating rook to use latest available release
image
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
A recent PR #4172 added the ability for rook to create
the CSIDriver object on a kubernetes cluster. The change also
made it so that attachRequired value in the ceph CSI drivers
were set to false.
This leads to the entire attach skipping the AD controller
that deals with RWO attachment enforcement. Thus, if a
CephFS volume is marked RWO, 2 pods on different hosts can
end up mounting the same and consuming it at the same time.
Further for RBD, currently the image watchers are the only ones
that prevent a double mount of an image, other than the AD controller
enforcement of single attach for RWO RBD PVs. This has proven
not to be reliable in the face of MON failures, as the watcher
information is lost, and a second mount is allowed.
Thus, we need to bring back the attach requirement to leverage
kubernetes checks and balances for RWO volume type, and
prevent unwanted multi-attach use of the same.
This PR hence undoes some of the changes introduced by the
PR #4172, to bring the attacher side car back, and also declare
the attachRequired for both RBD and CephFS CSI plugins as true.
NOTE: Initially the attachRequired was set to false as this
improved attach times considerably on kubernetes, as the
entire phase was being avoided.
Signed-off-by: ShyamsundarR <srangana@redhat.com>
OSDs on PVCs need some anti-affinity or other node affinity in order
to attempt to spread the OSDs across nodes. The OSD anti-affinity
is only effective when the number of OSDs matches the node count,
but at least it is a start.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Sébastien Han
Daniel Parkes
Madhu Rajanna
Travis Nielsen
Jeff Billimek
Juan Miguel Olmo Martínez
Dmitry Yusupov
Blaine Gardner
Swati Kale
Anmol Sachan
Nizamudeen
Mateusz Los
Dominic Pearson
Mateusz Gozdek
Sergey Kaydalov
ruslanloman
Kazuhito MATSUDA
RAJAT SINGH
Satoru Takeuchi
d-luu
ShyamsundarR