forked from rook/rook
Sébastien HanandRohan CJ
90cd8224f4
A new controller to bootstrap ceph-crash pod on Ceph nodes running Ceph pods only. This implementation is unfortunately a mix of 'hacks' to make the daemon working correclty. The ceph-crash script faced different issues: * it's not a daemon * it does not have any key in cephx * it runs ceph commands under the hood which need an admin key and a ceph.conf On upstream Ceph, ceph-crash needs to grow better, once that happens we will improve our implementation. This also enhances osd provisionConfig struct with DataPathMap Pod volumes and volume mount needs DataPathMap to perform the right actions on log and crash dir. Exposing DataPathMap makes that possible. Obviously, this is exposing ceph crash reports on the host as well as pushing them into the mgr. Basically, if a daemon fails, it'll put its core dump into /var/lib/ceph/crash, bindmounting this dir on the host, ensures that the crashes don't get lost when the pod dies. Example: ``` [leseb@tarox~/go/src/github.com/rook/rook][rgw-liveprobe !] kubectl -n rook-ceph exec -ti rook-ceph-crashcollector-minikube-574858b99c-zvg4z bash bash: warning: setlocale: LC_CTYPE: cannot change locale (en_US.UTF-8): No such file or directory bash: warning: setlocale: LC_COLLATE: cannot change locale (en_US.UTF-8): No such file or directory bash: warning: setlocale: LC_MESSAGES: cannot change locale (en_US.UTF-8): No such file or directory bash: warning: setlocale: LC_NUMERIC: cannot change locale (en_US.UTF-8): No such file or directory bash: warning: setlocale: LC_TIME: cannot change locale (en_US.UTF-8): No such file or directory [root@rook-ceph-crashcollector-minikube-574858b99c-zvg4z /]# ceph crash ls [root@rook-ceph-osd-2-7644f99695-cljzh /]# pidof ceph-osd 13258 5415 5397 [root@rook-ceph-osd-2-7644f99695-cljzh /]# kill -SIGABRT 13258 [root@rook-ceph-osd-2-7644f99695-cljzh /]# ls /var/lib/ceph/crash/ 2019-11-12_12:56:51.404109Z_39f060e1-776d-4605-8f28-85c97e53de96 posted ... wait maximum 10 min (ceph-crash scraps every 10 minutes) ... the container will exit [root@rook-ceph-osd-2-7644f99695-cljzh /]# exit [leseb@tarox~/go/src/github.com/rook/rook][rgw-liveprobe !] [leseb@tarox~/go/src/github.com/rook/rook][rgw-liveprobe !] kubectl -n rook-ceph exec -ti rook-ceph-crashcollector-minikube-574858b99c-zvg4z bash bash: warning: setlocale: LC_CTYPE: cannot change locale (en_US.UTF-8): No such file or directory bash: warning: setlocale: LC_COLLATE: cannot change locale (en_US.UTF-8): No such file or directory bash: warning: setlocale: LC_MESSAGES: cannot change locale (en_US.UTF-8): No such file or directory bash: warning: setlocale: LC_NUMERIC: cannot change locale (en_US.UTF-8): No such file or directory bash: warning: setlocale: LC_TIME: cannot change locale (en_US.UTF-8): No such file or directory [root@rook-ceph-crashcollector-minikube-574858b99c-zvg4z /]# [root@rook-ceph-crashcollector-minikube-574858b99c-zvg4z /]# [root@rook-ceph-crashcollector-minikube-574858b99c-zvg4z /]# ceph crash ls 2019-11-12_12:56:51.404109Z_39f060e1-776d-4605-8f28-85c97e53de96 osd.1 [root@rook-ceph-crashcollector-minikube-574858b99c-zvg4z /]# ls /var/lib/ceph/crash/ posted [root@rook-ceph-crashcollector-minikube-574858b99c-zvg4z /]# ls /var/lib/ceph/crash/posted/ 2019-11-12_12:56:51.404109Z_39f060e1-776d-4605-8f28-85c97e53de96 [root@rook-ceph-crashcollector-minikube-574858b99c-zvg4z /]# tail /var/lib/ceph/crash/posted/2019-11-12_12\:56\:51.404109Z_39f060e1-776d-4605-8f28-85c97e53de96/log 1/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/lib/ceph/crash/2019-11-12_12:56:51.404109Z_39f060e1-776d-4605-8f28-85c97e53de96/log --- end dump of recent events --- ``` Signed-off-by: Sébastien Han <seb@redhat.com> Co-authored-by: Rohan CJ <rohantmp@gmail.com>
2.2 KiB
2.2 KiB
Major Themes
Action Required
Notable Features
- Added K8s 1.16 to the test matrix and removed K8s 1.11 from the test matrix.
Ceph
- The job for detecting the Ceph version can be started with node affinity or tolerations according to the same settings in the Cluster CR as the mons.
- A new CR property
skipUpgradeCheckshas been added, which allows you force an upgrade by skipping daemon checks. Use this at YOUR OWN RISK, only if you know what you're doing. To understand Rook's upgrade process of Ceph, read the upgrade doc. - Mon Quorum Disaster Recovery guide has been updated to work with the latest Rook and Ceph release.
- A new CRD property
PreservePoolsOnDeletehas been added to Filesystem(fs) and Object Store(os) resources in order to increase protection against data loss. if it is set totrue, associated pools won't be deleted when the main resource(fs/os) is deleted. Creating again the deleted fs/os with the same name will reuse the preserved pools. - OSDs:
- Ceph OSD's admin socket is now placed under Ceph's default system location
/run/ceph. - The on-host log directory for OSDs was set incorrectly to
<dataDirHostPath>/<namespace>/log; fix this to be<dataDirHostPath>/log/<namespace>, the same as other daemons. - Use the mon configuration database for directory-based OSDs, and do not generate a config
- Ceph OSD's admin socket is now placed under Ceph's default system location
- A new ceph-crashcollector controller has been added, that new pod will run on any node where a Ceph pod is running. Read more about this in the doc
EdgeFS
YugabyteDB
Breaking Changes
Ceph
- The
topologysetting has been removed from the CephCluster CR. To configure the OSD topology, node labels must be applied. See the OSD topology topic. This setting only affects OSDs when they are first created, thus OSDs will not be impacted during upgrade. The topology settings only apply to bluestore OSDs on raw devices. The topology labels are not applied to directory-based OSDs.