Merge pull request #11213 from gauravsitlani/live-coredump

docs: added steps to collect coredump and perf troubleshooting info

Documentation/Troubleshooting/performance-profiling.md

@@ -0,0 +1,76 @@
+---
+title: Performance Profiling
+---
+
+## Collect perf data of a ceph process at runtime
+
+!!! warn
+    This is an advanced topic please be aware of the steps you're performing or reach out
+    to the experts for further guidance.
+
+There are some cases where the debug logs are not sufficient to investigate issues
+like high CPU utilization of a Ceph process. In that situation, coredump and perf information
+of a Ceph process is useful to be collected which can be shared with the Ceph team in an issue.
+
+To collect this information, please follow these steps:
+
+* Edit the rook-ceph-operator deployment and set `ROOK_HOSTPATH_REQUIRES_PRIVILEGED` to `true`.
+* Wait for the pods to get reinitialized:
+
+```console
+# watch kubectl -n rook-ceph get pods
+```
+
+* Enter the respective pod of the Ceph process which needs to be investigated. For example:
+
+```console
+# kubectl -n rook-ceph exec -it deploy/rook-ceph-mon-a -- bash
+```
+
+* Install `gdb` , `perf` and `git` inside the pod. For example:
+```console
+# dnf install gdb git perf -y
+```
+
+* Capture perf data of the respective Ceph process:
+
+```console
+# perf record -e cycles --call-graph dwarf -p <pid of the process>
+# perf report > perf_report_<process/thread>
+```
+
+* Grab the `pid` of the respective Ceph process to collect its backtrace at multiple time instances, attach `gdb` to it and share the output `gdb.txt`:
+
+```console
+# gdb -p <pid_of_the_process>
+
+- set pag off
+- set log on
+- thr a a bt full # This captures the complete backtrace of the process
+- backtrace
+- Ctrl+C
+- backtrace
+- Ctrl+C
+- backtrace
+- Ctrl+C
+- backtrace
+- set log off
+- q (to exit out of gdb)
+```
+
+* Grab the live coredump of the respective process using `gcore`:
+
+```console
+# gcore <pid_of_the_process>
+```
+
+* Capture the [Wallclock Profiler](https://github.com/markhpc/gdbpmp) data for the respective
+Ceph process and share the output `gdbpmp.data` generated:
+
+```console
+# git clone https://github.com/markhpc/gdbpmp
+# cd gdbpmp
+# ./gdbpmp.py -p <pid_of_the_process> -n 100 -o gdbpmp.data
+```
+
+* Collect the `perf.data`, `perf_report`, backtrace of the process `gdb.txt` , `core` file and profiler data `gdbpmp.data` and upload it to the tracker issue for troubleshooting purposes.