ceph: toolbox job for running a ceph script

As another option to run Ceph commands, a script can be executed
with Ceph commands as if running in the toolbox, but will instead
run in a job where the logs can be collected and analyzed
separately. This would be useful where automation is in place to collect
information about the cluster periodically or upon some failure
that is detected, without requiring an interactive shell.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>

Documentation/ceph-toolbox.md

@@ -9,10 +9,16 @@ indent: true
 The Rook toolbox is a container with common tools used for rook debugging and testing.
 The toolbox is based on CentOS, so more tools of your choosing can be easily installed with `yum`.
 
-## Running the Toolbox in Kubernetes
+The toolbox can be run in two modes:
+1. [Interactive](#interactive-toolbox): Start a toolbox pod where you can connect and execute Ceph commands from a shell
+2. [One-time job](#toolbox-job): Run a script with Ceph commands and collect the results from the job log
 
-The rook toolbox can run as a deployment in a Kubernetes cluster.  After you ensure you have a running Kubernetes cluster with rook deployed (see the [Kubernetes](ceph-quickstart.md) instructions),
-launch the rook-ceph-tools pod.
+> Prerequisite: Before running the toolbox you should have a running Rook cluster deployed (see the [Quickstart Guide](ceph-quickstart.md)).
+
+## Interactive Toolbox
+
+The rook toolbox can run as a deployment in a Kubernetes cluster where you can connect and
+run arbitrary Ceph commands.
 
 Save the tools spec as `toolbox.yaml`:
 
@@ -101,8 +107,78 @@ When you are done with the toolbox, you can remove the deployment:
 kubectl -n rook-ceph delete deployment rook-ceph-tools
 ```
 
-## Troubleshooting without the Toolbox
+## Toolbox Job
 
-The Ceph tools will commonly be the only tools needed to troubleshoot a cluster. In that case, you can connect to any of the rook pods and execute the ceph commands in the same way that you would in the toolbox pod such as the mon pods or the operator pod.
-If connecting to the mon pods, make sure you connect to the mon most recently started. The mons keep the config updated in memory after starting and may not have the latest config on disk.
-For example, after starting the cluster connect to the `mon2` pod instead of `mon0`.
+If you want to run Ceph commands as a one-time operation and collect the results later from the
+logs, you can run a script as a Kubernetes Job. The toolbox job will run a script that is embedded
+in the job spec. The script has the full flexibility of a bash script.
+
+In this example, the `ceph status` command is executed when the job is created.
+
+```yaml
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: rook-ceph-toolbox-job
+  namespace: rook-ceph
+  labels:
+    app: ceph-toolbox-job
+spec:
+  template:
+    spec:
+      initContainers:
+      - name: config-init
+        image: rook/ceph:master
+        command: ["/usr/local/bin/toolbox.sh"]
+        args: ["--skip-watch"]
+        imagePullPolicy: IfNotPresent
+        env:
+        - name: ROOK_ADMIN_SECRET
+          valueFrom:
+            secretKeyRef:
+              name: rook-ceph-mon
+              key: admin-secret
+        volumeMounts:
+        - mountPath: /etc/ceph
+          name: ceph-config
+        - name: mon-endpoint-volume
+          mountPath: /etc/rook
+      containers:
+      - name: script
+        image: rook/ceph:master
+        volumeMounts:
+        - mountPath: /etc/ceph
+          name: ceph-config
+          readOnly: true
+        command:
+        - "bash"
+        - "-c"
+        - |
+          # Modify this script to run any ceph, rbd, radosgw-admin, or other commands that could
+          # be run in the toolbox pod. The output of the commands can be seen by getting the pod log.
+          #
+          # example: print the ceph status
+          ceph status
+      volumes:
+      - name: mon-endpoint-volume
+        configMap:
+          name: rook-ceph-mon-endpoints
+          items:
+          - key: data
+            path: mon-endpoints
+      - name: ceph-config
+        emptyDir: {}
+      restartPolicy: Never
+```
+
+Create the toolbox job:
+
+```console
+kubectl create -f toolbox-job.yaml
+```
+
+After the job completes, see the results of the script:
+
+```console
+kubectl -n rook-ceph logs -l job-name=rook-ceph-toolbox-job
+```

Documentation/ceph-upgrade.md

@@ -152,7 +152,7 @@ Before we begin the upgrade process, let's first review some ways that you can v
 your cluster, ensuring that the upgrade is going smoothly after each step. Most of the health
 verification checks for your cluster during the upgrade process can be performed with the Rook
 toolbox. For more information about how to run the toolbox, please visit the
-[Rook toolbox readme](./ceph-toolbox.md#running-the-toolbox-in-kubernetes).
+[Rook toolbox readme](./ceph-toolbox.md).
 
 See the common issues pages for troubleshooting and correcting health issues:
 

PendingReleaseNotes.md

@@ -6,6 +6,7 @@
 
 ### Ceph
 
+- Added a [toolbox job](Documentation/ceph-toolbox.md#toolbox-job) for running a script with Ceph commands, similar to running commands in the Rook toolbox.
 - Ceph RBD Mirror daemon has been extracted to its own CRD, it has been removed from the `CephCluster` CRD, see the [rbd-mirror crd](Documentation/ceph-rbd-mirror-crd.html).
 
 ### EdgeFS

cluster/examples/kubernetes/ceph/toolbox-job.yaml

@@ -0,0 +1,53 @@
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: rook-ceph-toolbox-job
+  namespace: rook-ceph
+  labels:
+    app: ceph-toolbox-job
+spec:
+  template:
+    spec:
+      initContainers:
+      - name: config-init
+        image: rook/ceph:master
+        command: ["/usr/local/bin/toolbox.sh"]
+        args: ["--skip-watch"]
+        imagePullPolicy: IfNotPresent
+        env:
+        - name: ROOK_ADMIN_SECRET
+          valueFrom:
+            secretKeyRef:
+              name: rook-ceph-mon
+              key: admin-secret
+        volumeMounts:
+        - mountPath: /etc/ceph
+          name: ceph-config
+        - name: mon-endpoint-volume
+          mountPath: /etc/rook
+      containers:
+      - name: script
+        image: rook/ceph:master
+        volumeMounts:
+        - mountPath: /etc/ceph
+          name: ceph-config
+          readOnly: true
+        command:
+        - "bash"
+        - "-c"
+        - |
+          # Modify this script to run any ceph, rbd, radosgw-admin, or other commands that could
+          # be run in the toolbox pod. The output of the commands can be seen by getting the pod log.
+          #
+          # example: print the ceph status
+          ceph status
+      volumes:
+      - name: mon-endpoint-volume
+        configMap:
+          name: rook-ceph-mon-endpoints
+          items:
+          - key: data
+            path: mon-endpoints
+      - name: ceph-config
+        emptyDir: {}
+      restartPolicy: Never