From 005f17036afd0aabc34d901f9770c26eb5c8f517 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?S=C3=A9bastien=20Han?= Date: Fri, 18 Sep 2020 11:47:20 +0200 Subject: [PATCH] docs: common issue low aio-max-nr setting MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit When the system has a low ow aio-max-nr setting, the osd prepare pod can fail to complete. This value must be increased. Adding a documentation section for it. Closes: https://github.com/rook/rook/issues/6279 Signed-off-by: Sébastien Han --- Documentation/ceph-common-issues.md | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/Documentation/ceph-common-issues.md b/Documentation/ceph-common-issues.md index 9be51c5ac..e7e39f1ad 100644 --- a/Documentation/ceph-common-issues.md +++ b/Documentation/ceph-common-issues.md @@ -28,6 +28,7 @@ If after trying the suggestions found on this page and the problem is not resolv * [A worker node using RBD devices hangs up](#a-worker-node-using-rbd-devices-hangs-up) * [Too few PGs per OSD warning is shown](#too-few-pgs-per-osd-warning-is-shown) * [LVM metadata can be corrupted with OSD on LV-backed PVC](#lvm-metadata-can-be-corrupted-with-osd-on-lv-backed-pvc) +* [OSD Prepare job fails due to low aio-max-nr setting](#osd-prepare-job-fails-due-to-low-aio-max-nr-setting) ## Troubleshooting Techniques @@ -813,7 +814,20 @@ If you still decide to configure an OSD on LVM, please keep the following in min ### Solution - Disable `lvmetad.` -- Avoid configuration of LVs from the host. In addition, don't touch the VGs and physical volumes that back these LVs. +- Avoid configuration of LVs from the host. In addition, don't touch the VGs and physical volumes that back these LVs. - Avoid incrementing the `count` field of `storageClassDeviceSets` and create a new LV that backs a OSD simultaneously. You can know whether the above-mentioned tag exists tag with the command: `sudo lvs -o lv_name,lv_tags`. If the `lv_tag` field is empty in an LV corresponding to the OSD lv_tags, this OSD encountered the problem. In this case, please [retire this OSD](ceph-osd-mgmt.md#remove-an-osd) or replace with other new OSD before restarting. + +## OSD prepare job fails due to low aio-max-nr setting + +If the Kernel is configured with a low [aio-max-nr setting](https://www.kernel.org/doc/Documentation/sysctl/fs.txt), the OSD prepare job might fail with the following error: + +```text +exec: stderr: 2020-09-17T00:30:12.145+0000 7f0c17632f40 -1 bdev(0x56212de88700 /var/lib/ceph/osd/ceph-0//block) _aio_start io_setup(2) failed with EAGAIN; try increasing /proc/sys/fs/aio-max-nr +``` + +To overcome this, you need to increase the value of `fs.aio-max-nr` of your sysctl configuration (typically `/etc/sysctl.conf`). +You can do this with your favorite configuration management system. + +Alternatively, you can have a [DaemonSet](https://github.com/rook/rook/issues/6279#issuecomment-694390514) to apply the configuration for you on all your nodes.