In ceph-csi v1.2.2 we have add a flag `forcecephkernelclient`
to force use Ceph Kernel clients on kernel < 4.17 which
support quotas for Cephfs
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
Removed some obsolete instructions from the 1.1 upgrade
and made a few minor fixes to make the upgrade to 1.2 smoother.
With the addition of the CephClient CRD, the CRD needs to be
created before the upgrade or else the operator log will be
filled with errors until it is created. The CSI driver also
had an error when registering after the upgrade due to an RBAC
change.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
gp2 devices present themselves as SSDs,
but have much more variable performance.
Ceph attempts to adjust some settings based on the drive type,
HDD/SSD
In this case client I/O can be impacted too much by assuming gp2 devices are like other SSDs
(an example is https://bugzilla.redhat.com/show_bug.cgi?id=1765923#c6)
In order to prevent this, rook can set the following for OSDs
if the new CR setting `tuneSlowDeviceClass` is set to `true`:
```
osd_recovery_sleep = 0.1
osd_snap_trim_sleep = 2
osd_delete_sleep = 2
```
Closes: https://github.com/rook/rook/issues/4298 and https://bugzilla.redhat.com/show_bug.cgi?id=1771047
Signed-off-by: Sébastien Han <seb@redhat.com>
Removal of OSDs is a sensitive area. If you get it wrong you could have data
loss. Now a help topic is provided for manual steps to remove OSDs when
absolutely necessary.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Add documentation to clone the correct release branch of the repo
to reduce confusion from pulling the wrong branch.
removed spaces
Signed-off-by: Alex Punnen <alexcpn@gmail.com>
ceph client CRD refactoring
Add info about client crd to documentation
Add tests for client controller CRD
Signed-off-by: Mateusz Los <los.mateusz@gmail.com>
Add documentation to clone the correct release branch of the repo
to reduce confusion from pulling the wrong branch.
Signed-off-by: Alex Punnen <alexcpn@gmail.com>
"OSD on PVC" doesn't work for PV backed by LV. Fixing this problem
by the following changes.
- Rook accepts LVM disk type.
- If a LV-backed device is passed, Rook/Ceph invokes
"ceph-volume lvm prepare" with "--data vg/lv"
instead of "--data /path/to/device".
- If a LV-backed device is passed, Rook/Ceph suppresses
activation/deactivation of VG that owns this LV.
Fixes: https://github.com/rook/rook/issues/4185
Signed-off-by: dulltz <isrgnoe@gmail.com>
Co-authored-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
we need to set a ENV `CSI_RBD_ENABLE_SNAPSHOTTER`
variable to "false" in operator template to disable
deployment of snapshotter sidecar in ceph-csi rbd
provisioner.
Fixes: #4401
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
cephcsi v1.2.2 release is out and the image is available
for use, updating rook to use latest available release
image
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
In NFS document, the document example explain with
`-l name=nfs-busybox` label, however no corresponding label in
previous section. Fix to the label mentioned in the previous
section.
Signed-off-by: Woohyung Han <techhanx@gmail.com>
A recent PR #4172 added the ability for rook to create
the CSIDriver object on a kubernetes cluster. The change also
made it so that attachRequired value in the ceph CSI drivers
were set to false.
This leads to the entire attach skipping the AD controller
that deals with RWO attachment enforcement. Thus, if a
CephFS volume is marked RWO, 2 pods on different hosts can
end up mounting the same and consuming it at the same time.
Further for RBD, currently the image watchers are the only ones
that prevent a double mount of an image, other than the AD controller
enforcement of single attach for RWO RBD PVs. This has proven
not to be reliable in the face of MON failures, as the watcher
information is lost, and a second mount is allowed.
Thus, we need to bring back the attach requirement to leverage
kubernetes checks and balances for RWO volume type, and
prevent unwanted multi-attach use of the same.
This PR hence undoes some of the changes introduced by the
PR #4172, to bring the attacher side car back, and also declare
the attachRequired for both RBD and CephFS CSI plugins as true.
NOTE: Initially the attachRequired was set to false as this
improved attach times considerably on kubernetes, as the
entire phase was being avoided.
Signed-off-by: ShyamsundarR <srangana@redhat.com>
OSDs on PVCs need some anti-affinity or other node affinity in order
to attempt to spread the OSDs across nodes. The OSD anti-affinity
is only effective when the number of OSDs matches the node count,
but at least it is a start.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
A new controller to bootstrap ceph-crash pod on Ceph nodes running Ceph
pods only.
This implementation is unfortunately a mix of 'hacks' to make the daemon
working correclty. The ceph-crash script faced different issues:
* it's not a daemon
* it does not have any key in cephx
* it runs ceph commands under the hood which need an admin key and a
ceph.conf
On upstream Ceph, ceph-crash needs to grow better, once that happens we
will improve our implementation.
This also enhances osd provisionConfig struct with DataPathMap
Pod volumes and volume mount needs DataPathMap to perform the right
actions on log and crash dir. Exposing DataPathMap makes that possible.
Obviously, this is exposing ceph crash reports on the host as
well as pushing them into the mgr.
Basically, if a daemon fails, it'll put its core dump into
/var/lib/ceph/crash, bindmounting this dir on the host, ensures that the
crashes don't get lost when the pod dies.
Example:
```
[leseb@tarox~/go/src/github.com/rook/rook][rgw-liveprobe !] kubectl -n rook-ceph exec -ti rook-ceph-crashcollector-minikube-574858b99c-zvg4z bash
bash: warning: setlocale: LC_CTYPE: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_COLLATE: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_MESSAGES: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_NUMERIC: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_TIME: cannot change locale (en_US.UTF-8): No such file or directory
[root@rook-ceph-crashcollector-minikube-574858b99c-zvg4z /]# ceph crash ls
[root@rook-ceph-osd-2-7644f99695-cljzh /]# pidof ceph-osd
13258 5415 5397
[root@rook-ceph-osd-2-7644f99695-cljzh /]# kill -SIGABRT 13258
[root@rook-ceph-osd-2-7644f99695-cljzh /]# ls /var/lib/ceph/crash/
2019-11-12_12:56:51.404109Z_39f060e1-776d-4605-8f28-85c97e53de96 posted
... wait maximum 10 min (ceph-crash scraps every 10 minutes)
... the container will exit
[root@rook-ceph-osd-2-7644f99695-cljzh /]# exit
[leseb@tarox~/go/src/github.com/rook/rook][rgw-liveprobe !]
[leseb@tarox~/go/src/github.com/rook/rook][rgw-liveprobe !] kubectl -n rook-ceph exec -ti rook-ceph-crashcollector-minikube-574858b99c-zvg4z bash
bash: warning: setlocale: LC_CTYPE: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_COLLATE: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_MESSAGES: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_NUMERIC: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_TIME: cannot change locale (en_US.UTF-8): No such file or directory
[root@rook-ceph-crashcollector-minikube-574858b99c-zvg4z /]#
[root@rook-ceph-crashcollector-minikube-574858b99c-zvg4z /]#
[root@rook-ceph-crashcollector-minikube-574858b99c-zvg4z /]# ceph crash ls
2019-11-12_12:56:51.404109Z_39f060e1-776d-4605-8f28-85c97e53de96 osd.1
[root@rook-ceph-crashcollector-minikube-574858b99c-zvg4z /]# ls /var/lib/ceph/crash/
posted
[root@rook-ceph-crashcollector-minikube-574858b99c-zvg4z /]# ls /var/lib/ceph/crash/posted/
2019-11-12_12:56:51.404109Z_39f060e1-776d-4605-8f28-85c97e53de96
[root@rook-ceph-crashcollector-minikube-574858b99c-zvg4z /]# tail /var/lib/ceph/crash/posted/2019-11-12_12\:56\:51.404109Z_39f060e1-776d-4605-8f28-85c97e53de96/log
1/ 5 mgr
1/ 5 mgrc
1/ 5 dpdk
1/ 5 eventtrace
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/lib/ceph/crash/2019-11-12_12:56:51.404109Z_39f060e1-776d-4605-8f28-85c97e53de96/log
--- end dump of recent events ---
```
Signed-off-by: Sébastien Han <seb@redhat.com>
Co-authored-by: Rohan CJ <rohantmp@gmail.com>
Since there are many metadata exported while executing command `kubectl get -o yaml`.
kubectl replace will fail since we modified the deployment's YAML, the resource's version
has been bumped and the exported YAML's version will be behind the target deployment.
Use --force option to delete the resource and re-create it.
Signed-off-by: Jeffrey Chu <peihuachu1112@gmail.com>
As in title, the command should be patch instead of edit since it is passing a
patch to the object.
Signed-off-by: Jeffrey Chu <peihuachu1112@gmail.com>
Madhu Rajanna
Travis Nielsen
Sébastien Han
Dmitry Yusupov
Blaine Gardner
taeuk_kim
Alex Punnen
Mateusz Los
Cody Belcher
Sergey Kaydalov
Otto Yiu
dulltzandSatoru Takeuchi
Kazuhito MATSUDA
RAJAT SINGH
Alexander Trost
Zohar Mamedov
ranrubin
Woohyung Han
d-luu
ShyamsundarR
Jeffrey Chu
Anton Skriptsov