The app name was not being set in k8sutil's cmdreporter code, resulting
in jobs being labeled `app=` with no name. Change cmdreporter's `New`
method to validate the contents of the config struct and not the inputs
to the `New` method to help prevent an issue like this from creeping up
in the future.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
- Checking whether drain was incorrectly checked precense in map instead
of string length. The actual zero value was an empty string.
- Unsetting drain did not check if the cluster was clean.
Signed-off-by: Rohan CJ <rohantmp@gmail.com>
A new controller to bootstrap ceph-crash pod on Ceph nodes running Ceph
pods only.
This implementation is unfortunately a mix of 'hacks' to make the daemon
working correclty. The ceph-crash script faced different issues:
* it's not a daemon
* it does not have any key in cephx
* it runs ceph commands under the hood which need an admin key and a
ceph.conf
On upstream Ceph, ceph-crash needs to grow better, once that happens we
will improve our implementation.
This also enhances osd provisionConfig struct with DataPathMap
Pod volumes and volume mount needs DataPathMap to perform the right
actions on log and crash dir. Exposing DataPathMap makes that possible.
Obviously, this is exposing ceph crash reports on the host as
well as pushing them into the mgr.
Basically, if a daemon fails, it'll put its core dump into
/var/lib/ceph/crash, bindmounting this dir on the host, ensures that the
crashes don't get lost when the pod dies.
Example:
```
[leseb@tarox~/go/src/github.com/rook/rook][rgw-liveprobe !] kubectl -n rook-ceph exec -ti rook-ceph-crashcollector-minikube-574858b99c-zvg4z bash
bash: warning: setlocale: LC_CTYPE: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_COLLATE: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_MESSAGES: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_NUMERIC: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_TIME: cannot change locale (en_US.UTF-8): No such file or directory
[root@rook-ceph-crashcollector-minikube-574858b99c-zvg4z /]# ceph crash ls
[root@rook-ceph-osd-2-7644f99695-cljzh /]# pidof ceph-osd
13258 5415 5397
[root@rook-ceph-osd-2-7644f99695-cljzh /]# kill -SIGABRT 13258
[root@rook-ceph-osd-2-7644f99695-cljzh /]# ls /var/lib/ceph/crash/
2019-11-12_12:56:51.404109Z_39f060e1-776d-4605-8f28-85c97e53de96 posted
... wait maximum 10 min (ceph-crash scraps every 10 minutes)
... the container will exit
[root@rook-ceph-osd-2-7644f99695-cljzh /]# exit
[leseb@tarox~/go/src/github.com/rook/rook][rgw-liveprobe !]
[leseb@tarox~/go/src/github.com/rook/rook][rgw-liveprobe !] kubectl -n rook-ceph exec -ti rook-ceph-crashcollector-minikube-574858b99c-zvg4z bash
bash: warning: setlocale: LC_CTYPE: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_COLLATE: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_MESSAGES: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_NUMERIC: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_TIME: cannot change locale (en_US.UTF-8): No such file or directory
[root@rook-ceph-crashcollector-minikube-574858b99c-zvg4z /]#
[root@rook-ceph-crashcollector-minikube-574858b99c-zvg4z /]#
[root@rook-ceph-crashcollector-minikube-574858b99c-zvg4z /]# ceph crash ls
2019-11-12_12:56:51.404109Z_39f060e1-776d-4605-8f28-85c97e53de96 osd.1
[root@rook-ceph-crashcollector-minikube-574858b99c-zvg4z /]# ls /var/lib/ceph/crash/
posted
[root@rook-ceph-crashcollector-minikube-574858b99c-zvg4z /]# ls /var/lib/ceph/crash/posted/
2019-11-12_12:56:51.404109Z_39f060e1-776d-4605-8f28-85c97e53de96
[root@rook-ceph-crashcollector-minikube-574858b99c-zvg4z /]# tail /var/lib/ceph/crash/posted/2019-11-12_12\:56\:51.404109Z_39f060e1-776d-4605-8f28-85c97e53de96/log
1/ 5 mgr
1/ 5 mgrc
1/ 5 dpdk
1/ 5 eventtrace
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/lib/ceph/crash/2019-11-12_12:56:51.404109Z_39f060e1-776d-4605-8f28-85c97e53de96/log
--- end dump of recent events ---
```
Signed-off-by: Sébastien Han <seb@redhat.com>
Co-authored-by: Rohan CJ <rohantmp@gmail.com>
Since I was parsing this file with the previous commits let's add some
cosmetic changes.
%q is more readable since it doubles quotes the variable.
Signed-off-by: Sébastien Han <seb@redhat.com>
WriteConnectionConfig() is called after the controller starts and thus
the 'osd find' command fails to complete because there is no ceph.conf
or key on the operator.
Let's skip this error print if that's the case since it's misleading and
polluting the logs.
Signed-off-by: Sébastien Han <seb@redhat.com>
This change standardizes the ownerRefs of Ceph keyrings to
k8s Deployment resources across all daemons and refactors a
bit of duplicate code on keyring association for ease of
maintenance.
Fixes: #4089
Signed-off-by: egafford <egafford@redhat.com>
When an orchestration is restarted or retriggered and starts new osd prepare pods, they are unable to start because the device is already in use by the osd daemon pods.
This fix simple skips the osd prepare pod creation if the daemon is already running.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
Once the cluster is connected, we need to create the CSI keys so that we
can provide persistent storage to containers, via ceph csi.
Signed-off-by: Sébastien Han <seb@redhat.com>
**Description of your changes:**
This modification adds the information extracted from 'ceph-volume inventory':
command to the device configmaps generated by the discovery daemon when
"rook discover" starts with the new boolean "--use-ceph-volume" parameter.
Resolves #
https://github.com/rook/rook/issues/2606
Now the <cephVolumeData> field contains all the information returned
from <ceph-volume inventory> command.
Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
This will be helpful to list the CSI drivers
deployed in the kubernetes cluster. if we have
N number of CSI drivers deployed in the cluster,
the user can query kubectl get csidriver and get
the list of registered CSI drivers and it requires
kube 1.14+
This removes the deployment of Attacher sidecar container,
as we are setting the attacher required to false during
the CSIDriver object creation.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
Failure to create a keyring prior to creating a deployment
dependent on that keyring created a race conditition resulting
in an intermittent and unnecessary pod failure. This change
creates a keyring prior to deployment creation for MDS
daemons.
Partially fixes: rook#4089
Signed-off-by: egafford <egafford@redhat.com>
Failure to create a keyring prior to creating a deployment
dependent on that keyring created a race conditition resulting
in an intermittent and unnecessary pod failure. This change
creates a keyring prior to any RGW deployment.
Partially fixes: #4089
Signed-off-by: egafford <egafford@redhat.com>
retry ObjectUser creation as long as valid input is provided
or it times out. Retry every 15sec for 5min.
integration test creates ObjectUser before ObjectStore to
ensure that ObjectUser waits for ObjectStore to be created
and available
Closes: https://github.com/rook/rook/issues/3937
Signed-off-by: Umanga Chapagain <chapagainumanga@gmail.com>
The topology of the cluster should be based on the node labels rather
than a setting in the cluster CR. This allows a much richer and more
dynamic topology to be configured. The location will now be ignored
if specified in the cluster CR.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
We removed the topologyAware CRD option since it was redundant with the
use of an OSD being backed by a PVC.
So now, if an OSD is backed by a PVC we assume the topology aware
decision and will discover zone and region labels on that host.
Signed-off-by: Sébastien Han <seb@redhat.com>
The node labels were already supported for zones and regions to add to
the CRUSH map. Now all layers of the CRUSH map will be supported
with the new labels such as topology.rook.io/rack.
The labels will be detected at the osd startup time
similar to the zone and region labels already being detected.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Blaine Gardner
Dmitry Yusupov
Rohan CJ
Sébastien Han
Travis Nielsen
Anton Skriptsov
egafford
Santosh Pillai
RAJAT SINGH
Juan Miguel Olmo Martínez
Madhu Rajanna
Nizamudeen
Umanga Chapagain
d-luu