Merge pull request #1593 from travisn/default-config-dir
Run ceph tools from any rook pod
This commit is contained in:
6 files changed
+91
-28
No files matched your search
@@ -4,13 +4,14 @@ weight: 78
|
||||
indent: true
|
||||
---
|
||||
|
||||
# Common Problems
|
||||
# Common Issues
|
||||
|
||||
Many of these problem cases are hard to summarize down to a short phrase that adequately describes the problem. Each problem will start with a bulleted list of symptoms. Keep in mind that all symptoms may not apply depending upon the configuration of the Rook. If the majority of the symptoms are seen there is a fair chance you are experiencing that problem.
|
||||
|
||||
If after trying the suggestions found on this page and the problem is not resolved, the Rook team is very happy to help you troubleshoot the issues in their Slack channel. Once you have [registered for the Rook Slack](https://rook-slackin.herokuapp.com/), proceed to the General channel to ask for assistance.
|
||||
|
||||
## Table of Contents
|
||||
- [Troubleshooting Techniques](#troubleshooting-techniques)
|
||||
- [Pod using Rook storage is not running](#pod-using-rook-storage-is-not-running)
|
||||
- [Cluster failing to service requests](#cluster-failing-to-service-requests)
|
||||
- [Only a single monitor pod starts](#only-a-single-monitor-pod-starts)
|
||||
@@ -18,8 +19,48 @@ If after trying the suggestions found on this page and the problem is not resolv
|
||||
- [Node hangs after reboot](#node-hangs-after-reboot)
|
||||
|
||||
# Troubleshooting Techniques
|
||||
One of the first things that should be done is to start the [rook-tools pod](./toolbox.md) as described in the Toolbox section. Once the pod is up and running one can `kubectl exec` into the pod to execute Ceph commands to evaluate that current state of the cluster. Here is a list of commands that can help one get an understanding of the current state.
|
||||
There are two main categories of information you will need to investigate issues in the cluster:
|
||||
1. Kubernetes status and logs
|
||||
1. Ceph status
|
||||
|
||||
## Kubernetes Tools
|
||||
Kubernetes status is the first line of investigating when something goes wrong with the cluster. Here are a few artifacts that are helpful to gather:
|
||||
- Rook pod status:
|
||||
- `kubectl get pod -n rook -o wide`
|
||||
- `kubectl get pod -n rook-system -o wide`
|
||||
- Logs for Rook pods
|
||||
- Logs for the operator: `kubectl logs -n rook-system -l app=rook-operator`
|
||||
- Logs for a specific pod: `kubectl logs -n rook <pod-name>`, or a pod using a label such as mon1: `kubectl logs -n rook -l mon=rook-ceph-mon1`
|
||||
- Logs on a specific node to find why a PVC is failing to mount:
|
||||
- Rook agent errors around the attach/detach: `kubectl logs -n rook-system <rook-agent-pod>`
|
||||
- Connect to the node, then get kubelet logs (if your distro is using systemd): `journalctl -u kubelet`
|
||||
- See the [log collection topic](advanced-configuration.md#log-collection) for a script that will help you gather the logs
|
||||
- Other Rook artifacts:
|
||||
- The monitors that are expected to be in quorum: `kubectl -n rook get configmap rook-ceph-mon-endpoints -o yaml | grep data`
|
||||
- More artifacts in the `rook` namespace: `kubectl -n rook get all`
|
||||
|
||||
## Ceph Tools
|
||||
After you verify the basic health of the running pods, next you will want to run Ceph tools for status of the storage components. There are two ways to run the Ceph tools, either in the Rook toolbox or inside other Rook pods that are already running.
|
||||
|
||||
### Tools in the Rook Toolbox
|
||||
The [rook-tools pod](./toolbox.md) is a one-stop shop for both Ceph tools and other troubleshooting tools. Once the pod is up and running one connect to the pod to execute Ceph commands to evaluate that current state of the cluster.
|
||||
```bash
|
||||
kubectl exec -it rook-tools bash
|
||||
```
|
||||
|
||||
### Tools in other pods
|
||||
The Ceph tools are found in all of the Rook pods where Ceph is running, such as the operator, monitors, and OSDs. Rather than starting the toolbox pod, you can connect to the existing pods to more quickly execute the Ceph tools. For example, to connect to the operator pod:
|
||||
|
||||
```bash
|
||||
kubectl -n rook-system exec -it $(kubectl -n rook-system get pods -l app=rook-operator -o jsonpath='{.items[0].metadata.name}') -- bash
|
||||
```
|
||||
|
||||
Now from inside the operator pod you can execute the Ceph tools.
|
||||
|
||||
It is preferred to connect to the operator pod rather than monitors and OSDs since the operator always has the latest configuration files. If you have multiple Rook clusters, it is preferred to connect to the Rook toolbox for a specific cluster. Otherwise, your ceph commands may connect to the wrong cluster.
|
||||
|
||||
### Ceph Commands
|
||||
Here are some common commands to troubleshoot the cluster.
|
||||
* ceph status
|
||||
* ceph osd status
|
||||
* ceph osd df
|
||||
@@ -28,9 +69,9 @@ One of the first things that should be done is to start the [rook-tools pod](./t
|
||||
* ceph osd tree
|
||||
* ceph pg stat
|
||||
|
||||
Of particular note, the first two status commands provide the overall cluster health. The normal state for cluster operations is HEALTH_OK, but will still function when the state is in a HEALTH_WARN state. If you are in a WARN state, then the cluster is in a condition that it may enter the HEALTH_ERROR state at which point *all* disk I/O operations are halted. If a HEALTH_WARN state is observed, then one should take action to prevent the cluster from halting when it enters the HEALTH_ERROR state.
|
||||
The first two status commands provide the overall cluster health. The normal state for cluster operations is HEALTH_OK, but will still function when the state is in a HEALTH_WARN state. If you are in a WARN state, then the cluster is in a condition that it may enter the HEALTH_ERROR state at which point *all* disk I/O operations are halted. If a HEALTH_WARN state is observed, then one should take action to prevent the cluster from halting when it enters the HEALTH_ERROR state.
|
||||
|
||||
There is literally a ton of Ceph sub-commands to look at and manipulate Ceph objects. Well beyond the scope of a few troubleshooting techniques and there are other sites and documentation sets that deal more with assisting one with troubleshooting a Ceph environment. In addition, there are other helpful hints and some best practices concerning a Ceph environment located in the [Advanced Configuration section](advanced-configuration.md). Of particular note there are scripts for collecting logs and gathering OSD information there.
|
||||
There are many Ceph sub-commands to look at and manipulate Ceph objects, well beyond the scope this document. See the [Ceph documentation]() for more details of gathering information about the health of the cluster. In addition, there are other helpful hints and some best practices located in the [Advanced Configuration section](advanced-configuration.md). Of particular note there are scripts for collecting logs and gathering OSD information there.
|
||||
|
||||
|
||||
# Pod Using Rook Storage Is Not Running
|
||||
@@ -90,3 +90,6 @@ When you are done with the toolbox, remove the pod:
|
||||
```bash
|
||||
kubectl -n rook delete pod rook-tools
|
||||
```
|
||||
|
||||
## Troubleshooting without the Toolbox
|
||||
The Ceph tools will commonly be the only tools needed to troubleshoot a cluster. In that case, you can connect to any of the rook pods and execute the ceph commands in the same way that you would in the toolbox pod. For example, you can connect to the mon, osd, or even the operator pod to execute commands such as `ceph status`.
|
||||
@@ -8,4 +8,4 @@ weight: 70
|
||||
Rook provides a number of tools to help you manage your cluster.
|
||||
- [Toolbox](toolbox.md): A pod from which you can run all of the tools to troubleshoot the storage cluster
|
||||
- [Advanced Configuration](advanced-configuration.md): Tips and tricks for configuring for cluster
|
||||
- [Common Issues](common-problems.md): Common issues and their potential solutions
|
||||
- [Common Issues](common-issues.md): Common issues and their potential solutions
|
||||
@@ -3,6 +3,7 @@
|
||||
## Action Required
|
||||
|
||||
## Notable Features
|
||||
- Ceph tools can be run [from any rook pod](Documentation/common-issues.md#ceph-tools).
|
||||
|
||||
## Breaking Changes
|
||||
|
||||
|
||||
@@ -177,6 +177,6 @@ func checkIfKubeletRestartRequired(context *clusterd.Context) {
|
||||
if err != nil || kubeVersion.LessThan(version.MustParseSemantic(serverVersionV180)) {
|
||||
logger.Warning("NOTE: The Kubelet must be restarted on this node since this pod appears to " +
|
||||
"be running on a Kubernetes version prior to 1.8. More details can be found in the Rook docs at " +
|
||||
"https://rook.io/docs/rook/master/common-problems.html#kubelet-restart")
|
||||
"https://rook.io/docs/rook/master/common-issues.html#kubelet-restart")
|
||||
}
|
||||
}
|
||||
@@ -46,6 +46,9 @@ const (
|
||||
caps osd = "allow *"
|
||||
caps mgr = "allow *"
|
||||
`
|
||||
defaultConfigDir = "/etc/ceph"
|
||||
defaultConfigFile = "ceph.conf"
|
||||
defaultKeyringFile = "keyring"
|
||||
)
|
||||
|
||||
type CephMonitorConfig struct {
|
||||
@@ -102,7 +105,7 @@ func getMonRunDirPath(configDir, monName string) string {
|
||||
|
||||
// get the path of a given monitor's keyring
|
||||
func getMonKeyringPath(configDir, monName string) string {
|
||||
return filepath.Join(getMonRunDirPath(configDir, monName), "keyring")
|
||||
return filepath.Join(getMonRunDirPath(configDir, monName), defaultKeyringFile)
|
||||
}
|
||||
|
||||
// get the path of a given monitor's data dir
|
||||
@@ -142,17 +145,41 @@ func writeMonKeyring(context *clusterd.Context, c *ClusterInfo, name string) err
|
||||
}
|
||||
|
||||
// writes the monitor keyring to disk
|
||||
func writeKeyring(keyring, path string) error {
|
||||
if err := os.MkdirAll(filepath.Dir(path), 0744); err != nil {
|
||||
return fmt.Errorf("failed to create keyring directory for %s: %+v", path, err)
|
||||
func writeKeyring(keyring, keyringPath string) error {
|
||||
// save the keyring to the given path
|
||||
if err := os.MkdirAll(filepath.Dir(keyringPath), 0744); err != nil {
|
||||
return fmt.Errorf("failed to create keyring directory for %s: %+v", keyringPath, err)
|
||||
}
|
||||
if err := ioutil.WriteFile(path, []byte(keyring), 0644); err != nil {
|
||||
return fmt.Errorf("failed to write monitor keyring to %s: %+v", path, err)
|
||||
if err := ioutil.WriteFile(keyringPath, []byte(keyring), 0644); err != nil {
|
||||
return fmt.Errorf("failed to write monitor keyring to %s: %+v", keyringPath, err)
|
||||
}
|
||||
|
||||
// Save the keyring to the default path. This allows the user to any pod to easily execute Ceph commands.
|
||||
// It is recommended to connect to the operator pod rather than monitors and OSDs since the operator always has the latest configuration files.
|
||||
// The mon and OSD pods will only re-create the config files when the pod is restarted. If a monitor fails over, the config
|
||||
// in the other mon and osd pods may be out of date. This could cause your ceph commands to timeout connecting to invalid mons.
|
||||
// Note that the running mon and osd daemons are not affected by this issue because of their live connection to the mon quorum.
|
||||
// If you have multiple Rook clusters, it is preferred to connect to the Rook toolbox for a specific cluster. Otherwise, your ceph commands
|
||||
// may connect to the wrong cluster.
|
||||
if err := os.MkdirAll(defaultConfigDir, 0744); err != nil {
|
||||
logger.Warningf("failed to create default directory %s: %+v", defaultConfigDir, err)
|
||||
return nil
|
||||
}
|
||||
defaultPath := path.Join(defaultConfigDir, defaultKeyringFile)
|
||||
if err := ioutil.WriteFile(defaultPath, []byte(keyring), 0644); err != nil {
|
||||
logger.Warningf("failed to copy keyring to %s: %+v", defaultPath, err)
|
||||
return nil
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func WriteKeyring(keyringPath, keyring string, generateContents func(string) string) error {
|
||||
// write the keyring to disk
|
||||
contents := generateContents(keyring)
|
||||
return writeKeyring(contents, keyringPath)
|
||||
}
|
||||
|
||||
// generates and writes the monitor config file to disk
|
||||
func GenerateConnectionConfigFile(context *clusterd.Context, cluster *ClusterInfo, pathRoot, user, keyringPath string) (string, error) {
|
||||
return GenerateConfigFile(context, cluster, pathRoot, user, keyringPath, nil, nil)
|
||||
@@ -196,6 +223,13 @@ func GenerateConfigFile(context *clusterd.Context, cluster *ClusterInfo, pathRoo
|
||||
return "", fmt.Errorf("failed to save config file %s. %+v", filePath, err)
|
||||
}
|
||||
|
||||
// copy the config to /etc/ceph/ceph.conf
|
||||
defaultPath := path.Join(defaultConfigDir, defaultConfigFile)
|
||||
logger.Infof("copying config to %s", defaultPath)
|
||||
if err := configFile.SaveTo(defaultPath); err != nil {
|
||||
logger.Warningf("failed to save config file %s. %+v", defaultPath, err)
|
||||
}
|
||||
|
||||
return filePath, nil
|
||||
}
|
||||
|
||||
@@ -220,22 +254,6 @@ func CreateKeyring(context *clusterd.Context, clusterName, username, keyringPath
|
||||
return WriteKeyring(keyringPath, key, generateContents)
|
||||
}
|
||||
|
||||
func WriteKeyring(keyringPath, keyring string, generateContents func(string) string) error {
|
||||
// write the keyring to disk
|
||||
keyringDir := filepath.Dir(keyringPath)
|
||||
if err := os.MkdirAll(keyringDir, 0744); err != nil {
|
||||
return fmt.Errorf("failed to create keyring dir at %s: %+v", keyringDir, err)
|
||||
}
|
||||
|
||||
contents := generateContents(keyring)
|
||||
logger.Debugf("Writing keyring to: %s", keyringPath)
|
||||
if err := ioutil.WriteFile(keyringPath, []byte(contents), 0644); err != nil {
|
||||
return fmt.Errorf("failed to write keyring to %s: %+v", keyringPath, err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// prepends "client." if a user namespace is not already specified
|
||||
func getQualifiedUser(user string) string {
|
||||
if strings.Index(user, ".") == -1 {
|
||||
|
||||
Reference in new issue
Block a user