Many of these problem cases are hard to summarize down to a short phrase that adequately describes the problem. Each problem will start with a bulleted list of symptoms. Keep in mind that all symptoms may not apply depending upon the configuration of the Rook. If the majority of the symptoms are seen there is a fair chance you are experiencing that problem.
If after trying the suggestions found on this page and the problem is not resolved, the Rook team is very happy to help you troubleshoot the issues in their Slack channel. Once you have [registered for the Rook Slack](https://rook-slackin.herokuapp.com/), proceed to the General channel to ask for assistance.
One of the first things that should be done is to start the [rook-tools pod](./toolbox.md) as described in the Toolbox section. Once the pod is up and running one can `kubectl exec` into the pod to execute Ceph commands to evaluate that current state of the cluster. Here is a list of commands that can help one get an understanding of the current state.
Of particular note, the first two status commands provide the overall cluster health. The normal state for cluster operations is HEALTH_OK, but will still function when the state is in a HEALTH_WARN state. If you are in a WARN state, then the cluster is in a condition that it may enter the HEALTH_ERROR state at which point *all* disk I/O operations are halted. If a HEALTH_WARN state is observed, then one should take action to prevent the cluster from halting when it enters the HEALTH_ERROR state.
There is literally a ton of Ceph sub-commands to look at and manipulate Ceph objects. Well beyond the scope of a few troubleshooting techniques and there are other sites and documentation sets that deal more with assisting one with troubleshooting a Ceph environment. In addition, there are other helpful hints and some best practices concerning a Ceph environment located in the [Advanced Configuration section](advanced-configuration.md). Of particular note there are scripts for collecting logs and gathering OSD information there.
*`rook-agent` pod is in a `CrashLoopBackOff` status because it cannot deploy its driver on a read-only filesystem: [Flexvolume configuration pre-reqs](./k8s-pre-reqs.md#flexvolume-configuration)
* Persistent Volume and/or Claim are failing to be created and bound: [Volume Creation](#volume-creation)
*`rook-agent` pod is failing to mount and format the volume: [Rook Agent Mounting](#volume-mounting)
* You are using Kubernetes 1.7.x or earlier and the Kubelet has not been restarted after `rook-agent` is in the `Running` status: [Restart Kubelet](#kubelet-restart)
* You are using Kubernetes 1.6.x and the attach-detach controller has not been disabled: [Disable attach-detach controller](./kubernetes.md#disable-attacher-detacher-controller)
## Investigation Details
If you see some of the symptoms above, it's because the requested Rook storage for your pod is not being created and mounted successfully.
In this walkthrough, we will be looking at the wordpress mysql example pod that is failing to start.
To first confirm there is an issue, you can run commands similar to the following and you should see similar output (note that some of it has been omitted for brevity):
1h 1h 3 default-scheduler Warning FailedScheduling PersistentVolumeClaim is not bound: "mysql-pv-claim" (repeated 2 times)
1h 35s 36 kubelet, 172.17.8.101 Warning FailedMount Unable to mount volumes for pod "wordpress-mysql-918363043-50pjr_default(08d14e75-bd99-11e7-bc4c-001c428b9fc8)": timeout expired waiting for volumes to attach/mount for pod "default"/"wordpress-mysql-918363043-50pjr". list of unattached/unmounted volumes=[mysql-persistent-storage]
1h 35s 36 kubelet, 172.17.8.101 Warning FailedSync Error syncing pod
```
To troubleshoot this, let's walk through the volume provisioning steps in order to confirm where the failure is happening.
### Rook Agent Deployment
The `rook-agent` pods are responsible for mapping and mounting the volume from the cluster onto the node that your pod will be running on.
If the `rook-agent` pod is not running then it cannot perform this function.
Below is an example of the `rook-agent` pods failing to get to the `Running` status because they are in a `CrashLoopBackOff` status:
```console
> kubectl -n rook-system get pod
NAME READY STATUS RESTARTS AGE
rook-agent-ct5pj 0/1 CrashLoopBackOff 16 59m
rook-agent-zb6n9 0/1 CrashLoopBackOff 16 59m
rook-operator-2203999069-pmhzn 1/1 Running 0 59m
```
If you see this occurring, you can get more details about why the `rook-agent` pods are continuing to crash with the following command and its sample output:
```console
> kubectl -n rook-system get pod -l app=rook-agent -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.containerStatuses[0].lastState.terminated.message}{"\n"}{end}'
rook-agent-ct5pj mkdir /usr/libexec/kubernetes: read-only file system
rook-agent-zb6n9 mkdir /usr/libexec/kubernetes: read-only file system
```
From the output above, we can see that the agents were not able to bind mount to `/usr/libexec/kubernetes` on the host they are scheduled to run on.
For some environments, this default path is read-only and therefore a better path must be provided to the agents.
Once the `rook-agent` pods are gone, **follow the instructions in the [Flexvolume configuration pre-reqs](./k8s-pre-reqs.md#flexvolume-configuration)** to ensure a good value for `--volume-plugin-dir` has been provided to the Kubelet.
After that has been configured, and the Kubelet has been restarted, start the agent pods up again by restarting `rook-operator`:
```console
kubectl -n rook-system delete pod -l app=rook-operator
```
### Kubelet Restart
#### **Kubernetes 1.7.x and earlier only**
If the `rook-agent` pods are all in the `Running` state then another thing to confirm is that **if you are running on Kubernetes 1.7.x or earlier**, the Kubelet must be restarted after the `rook-agent` pods are running.
A symptom of this can be found in the Kubelet's log/journal, with the following error saying `no volume plugin matched`:
```console
Oct 30 22:23:03 core-02 kubelet-wrapper[31926]: E1030 22:23:03.524159 31926 desired_state_of_world_populator.go:285] Failed to add volume "mysql-persistent-storage" (specName: "pvc-9f273fbc") for pod "9f2ff89a-bdbf" to desiredStateOfWorld. err=failed to get Plugin from volumeSpec for volume "pvc-9f273fbc" err=no volume plugin matched
```
If you encounter this, just **restart the Kubelet process**, as described in the **[Restart Kubelet](./kubernetes.md#restart-kubelet)** section of the Rook deployment guide.
### Volume Creation
The volume must first be created in the Rook cluster and then bound to a volume claim before it can be mounted to a pod.
Let's confirm that with the following commands and their output:
```console
> kubectl get pv
NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM STORAGECLASS REASON AGE
If the volume is failing to be created, there should be details in the log output, especially those tagged with `op-provisioner`.
### Volume Mounting
The final step in preparing Rook storage for your pod is for the `rook-agent` pod to mount and format it.
If all the preceding sections have been successful or inconclusive, then take a look at the `rook-agent` pod logs for further clues.
You can determine which `rook-agent` is running on the same node that your pod is scheduled on by using the `-o wide` output, then you can get the logs for that `rook-agent` pod similar to the example below:
In the `rook-agent` pod logs, you may see a snippet similar to the following:
```console
2017-11-07 00:04:37.808870 I | rook-flexdriver: WARNING: The node kernel version is 4.4.0-87-generic, which do not support multiple ceph filesystems. The kernel version has to be at least 4.7. If you have multiple ceph filesystems, the result could be inconsistent
```
This will happen in kernels with versions older than 4.7, where the option `mds_namespace` is not supported. This option is used to specify a filesystem namespace.
In this case, if there is only one filesystem in the Rook cluster, there should be no issues and the mount should succeed. If you have more than one filesystem, inconsistent results may arise and the filesystem mounted may not be the one you specified.
If the issue is still not resolved from the steps above, please come chat with us on the **#general** channel of our [Rook Slack](https://rook-slackin.herokuapp.com/).
We want to help you get your storage working and learn from those lessons to prevent users in the future from seeing the same issue.
Create a [rook-tools pod](./toolbox.md) to investigate the current state of CEPH. Here is an example of what one might see. In this case the `ceph status` command would just hang so a CTRL-C needed to be send. The `rookctl status` command is able to give a good amount of detail. In some cases the rook-api pod needs to be restarted for `rookctl` to be able to gather information. If the rook-api is restarted, then the rook-tools pod should be restarted also.
Another indication is when one or more of the MON pods restart frequently. Note the 'mon107' that has only been up for 16 minutes in the following output.
What is happening here is that the MON pods are restarting and one or more of the CEPH daemons are not getting configured with the proper cluster information. This is commonly the result of not specifying a value for `dataDirHostPath` in your Cluster CRD.
The `dataDirHostPath` setting specifies a path on the local host for the CEPH daemons to store configuration and data. Setting this to a path like `/var/lib/rook`, reapplying your Cluster CRD and restarting all the CEPH daemons (MON, MGR, OSD, RGW) should solve this problem. After the CEPH daemons have been restarted, it is advisable to restart the rook-api and [rook-tool Pods](./toolbox.md).
When attempting to reinstall Rook, the rook-operator pod gets started successfully and then the cluster CRD is then loaded. The rook-operator only starts up a single MON (possibly on rare occasion a second MON may be started) and just hangs. Looking at the log output of the rook-operator the last operation that was occuring was a `ceph mon_status`.
Looking at the log or the termination status for the `mon` pod, you will see a message indicating the keyring does not match from a previous deployment.
# the pod shows a termination status that the keyring does not match the existing keyring
$ kubectl -n rook describe pod -l mon=rook-ceph-mon0
...
Last State: Terminated
Reason: Error
Message: The keyring does not match the existing keyring in /var/lib/rook/rook-ceph-mon0/data/keyring.
You may need to delete the contents of dataDirHostPath on the host from a previous deployment.
...
```
If your cluster is larger than a couple nodes, you may get lucky enough that the monitors were able to start and form quorum. However, now the OSDs pods may fail to start due to state
from a previous deployment. Looking at the OSD pod logs you will see an error about the file already existing.
This is a common problem reinitializing the Rook cluster when the local directory used for persistence has **not** been purged.
This directory is the `dataDirHostPath` setting in the cluster CRD and is typically set to `/var/lib/rook`.
To fix the issue you will need to delete all components of Rook and then delete the contents of `/var/lib/rook` (or the directory specified by `dataDirHostPath`) on each of the hosts in the cluster.
Then when the cluster CRD is applied to start a new cluster, the rook-operator should start all the pods as expected.
