From f8ffd2c2eb206494efda081b0b80ff0661f04a22 Mon Sep 17 00:00:00 2001 From: Travis Nielsen Date: Mon, 20 Apr 2020 15:48:59 -0600 Subject: [PATCH] ceph: toolbox job for running a ceph script As another option to run Ceph commands, a script can be executed with Ceph commands as if running in the toolbox, but will instead run in a job where the logs can be collected and analyzed separately. This would be useful where automation is in place to collect information about the cluster periodically or upon some failure that is detected, without requiring an interactive shell. Signed-off-by: Travis Nielsen --- Documentation/ceph-toolbox.md | 90 +++++++++++++++++-- Documentation/ceph-upgrade.md | 2 +- PendingReleaseNotes.md | 1 + .../examples/kubernetes/ceph/toolbox-job.yaml | 53 +++++++++++ 4 files changed, 138 insertions(+), 8 deletions(-) create mode 100644 cluster/examples/kubernetes/ceph/toolbox-job.yaml diff --git a/Documentation/ceph-toolbox.md b/Documentation/ceph-toolbox.md index 8330f10a2..690d13a30 100644 --- a/Documentation/ceph-toolbox.md +++ b/Documentation/ceph-toolbox.md @@ -9,10 +9,16 @@ indent: true The Rook toolbox is a container with common tools used for rook debugging and testing. The toolbox is based on CentOS, so more tools of your choosing can be easily installed with `yum`. -## Running the Toolbox in Kubernetes +The toolbox can be run in two modes: +1. [Interactive](#interactive-toolbox): Start a toolbox pod where you can connect and execute Ceph commands from a shell +2. [One-time job](#toolbox-job): Run a script with Ceph commands and collect the results from the job log -The rook toolbox can run as a deployment in a Kubernetes cluster. After you ensure you have a running Kubernetes cluster with rook deployed (see the [Kubernetes](ceph-quickstart.md) instructions), -launch the rook-ceph-tools pod. +> Prerequisite: Before running the toolbox you should have a running Rook cluster deployed (see the [Quickstart Guide](ceph-quickstart.md)). + +## Interactive Toolbox + +The rook toolbox can run as a deployment in a Kubernetes cluster where you can connect and +run arbitrary Ceph commands. Save the tools spec as `toolbox.yaml`: @@ -101,8 +107,78 @@ When you are done with the toolbox, you can remove the deployment: kubectl -n rook-ceph delete deployment rook-ceph-tools ``` -## Troubleshooting without the Toolbox +## Toolbox Job -The Ceph tools will commonly be the only tools needed to troubleshoot a cluster. In that case, you can connect to any of the rook pods and execute the ceph commands in the same way that you would in the toolbox pod such as the mon pods or the operator pod. -If connecting to the mon pods, make sure you connect to the mon most recently started. The mons keep the config updated in memory after starting and may not have the latest config on disk. -For example, after starting the cluster connect to the `mon2` pod instead of `mon0`. +If you want to run Ceph commands as a one-time operation and collect the results later from the +logs, you can run a script as a Kubernetes Job. The toolbox job will run a script that is embedded +in the job spec. The script has the full flexibility of a bash script. + +In this example, the `ceph status` command is executed when the job is created. + +```yaml +apiVersion: batch/v1 +kind: Job +metadata: + name: rook-ceph-toolbox-job + namespace: rook-ceph + labels: + app: ceph-toolbox-job +spec: + template: + spec: + initContainers: + - name: config-init + image: rook/ceph:master + command: ["/usr/local/bin/toolbox.sh"] + args: ["--skip-watch"] + imagePullPolicy: IfNotPresent + env: + - name: ROOK_ADMIN_SECRET + valueFrom: + secretKeyRef: + name: rook-ceph-mon + key: admin-secret + volumeMounts: + - mountPath: /etc/ceph + name: ceph-config + - name: mon-endpoint-volume + mountPath: /etc/rook + containers: + - name: script + image: rook/ceph:master + volumeMounts: + - mountPath: /etc/ceph + name: ceph-config + readOnly: true + command: + - "bash" + - "-c" + - | + # Modify this script to run any ceph, rbd, radosgw-admin, or other commands that could + # be run in the toolbox pod. The output of the commands can be seen by getting the pod log. + # + # example: print the ceph status + ceph status + volumes: + - name: mon-endpoint-volume + configMap: + name: rook-ceph-mon-endpoints + items: + - key: data + path: mon-endpoints + - name: ceph-config + emptyDir: {} + restartPolicy: Never +``` + +Create the toolbox job: + +```console +kubectl create -f toolbox-job.yaml +``` + +After the job completes, see the results of the script: + +```console +kubectl -n rook-ceph logs -l job-name=rook-ceph-toolbox-job +``` diff --git a/Documentation/ceph-upgrade.md b/Documentation/ceph-upgrade.md index 78c727816..a39412f1f 100644 --- a/Documentation/ceph-upgrade.md +++ b/Documentation/ceph-upgrade.md @@ -152,7 +152,7 @@ Before we begin the upgrade process, let's first review some ways that you can v your cluster, ensuring that the upgrade is going smoothly after each step. Most of the health verification checks for your cluster during the upgrade process can be performed with the Rook toolbox. For more information about how to run the toolbox, please visit the -[Rook toolbox readme](./ceph-toolbox.md#running-the-toolbox-in-kubernetes). +[Rook toolbox readme](./ceph-toolbox.md). See the common issues pages for troubleshooting and correcting health issues: diff --git a/PendingReleaseNotes.md b/PendingReleaseNotes.md index 385304f10..b627f8c6a 100644 --- a/PendingReleaseNotes.md +++ b/PendingReleaseNotes.md @@ -6,6 +6,7 @@ ### Ceph +- Added a [toolbox job](Documentation/ceph-toolbox.md#toolbox-job) for running a script with Ceph commands, similar to running commands in the Rook toolbox. - Ceph RBD Mirror daemon has been extracted to its own CRD, it has been removed from the `CephCluster` CRD, see the [rbd-mirror crd](Documentation/ceph-rbd-mirror-crd.html). ### EdgeFS diff --git a/cluster/examples/kubernetes/ceph/toolbox-job.yaml b/cluster/examples/kubernetes/ceph/toolbox-job.yaml new file mode 100644 index 000000000..c2888a457 --- /dev/null +++ b/cluster/examples/kubernetes/ceph/toolbox-job.yaml @@ -0,0 +1,53 @@ +apiVersion: batch/v1 +kind: Job +metadata: + name: rook-ceph-toolbox-job + namespace: rook-ceph + labels: + app: ceph-toolbox-job +spec: + template: + spec: + initContainers: + - name: config-init + image: rook/ceph:master + command: ["/usr/local/bin/toolbox.sh"] + args: ["--skip-watch"] + imagePullPolicy: IfNotPresent + env: + - name: ROOK_ADMIN_SECRET + valueFrom: + secretKeyRef: + name: rook-ceph-mon + key: admin-secret + volumeMounts: + - mountPath: /etc/ceph + name: ceph-config + - name: mon-endpoint-volume + mountPath: /etc/rook + containers: + - name: script + image: rook/ceph:master + volumeMounts: + - mountPath: /etc/ceph + name: ceph-config + readOnly: true + command: + - "bash" + - "-c" + - | + # Modify this script to run any ceph, rbd, radosgw-admin, or other commands that could + # be run in the toolbox pod. The output of the commands can be seen by getting the pod log. + # + # example: print the ceph status + ceph status + volumes: + - name: mon-endpoint-volume + configMap: + name: rook-ceph-mon-endpoints + items: + - key: data + path: mon-endpoints + - name: ceph-config + emptyDir: {} + restartPolicy: Never