forked from rook/rook
Satoru Takeuchi
9cd42a17a4
Sometimes CephFlexSuite fails with the following backtrace.
```
--- FAIL: TestCephFlexSuite (584.14s)
--- PASS: TestCephFlexSuite/TestBlockStorageMountUnMountForDifferentAccessModes (163.66s)
--- PASS: TestCephFlexSuite/TestBlockStorageMountUnMountForStatefulSets (83.21s)
--- FAIL: TestCephFlexSuite/TestFileSystem (119.51s)
ceph_base_file_test.go:487:
Error Trace: ceph_base_file_test.go:487
ceph_base_file_test.go:265
ceph_flex_test.go:113
Error: Expected nil, but got: &errors.errorString{s:"kubectl exec command bash failed on pod rook-ceph-tools in namespace flex-ns. Failed to run: kubectl [exec -n flex-ns rook-ceph-tools -- bash -c mount -t ceph -o mds_namespace=smoke-test-fs,name=admin,secret=$(grep key /etc/ceph/keyring | awk '{print $3}') $(grep mon_host /etc/ceph/ceph.conf | awk '{print $3}'):/ /tmp/testrook] : exit status 32"}
Test: TestCephFlexSuite/TestFileSystem
```
It's due to insufficient retry of mount.
There is only one difference between CI logs of both "PASS" cases and "FAIL" cases.
The log of PASS cases:
```
2020-09-30 21:47:03.369019 D | exec: Running command: kubectl exec -n flex-ns rook-ceph-tools -- bash -c mount -t ceph -o mds_namespace=smoke-test-fs,name=admin,secret=$(grep key /etc/ceph/keyring | awk '{print $3}') $(grep mon_host /etc/ceph/ceph.conf | awk '{print $3}'):/ /tmp/testrook
2020-09-30 21:47:03.679359 D | exec: Running command: kubectl exec -n flex-ns rook-ceph-tools -- mkdir -p /tmp/testrook/foo
```
The log of FAIL cases:
```
2020-09-30 21:49:21.591702 D | exec: Running command: kubectl exec -n flex-ns rook-ceph-tools -- bash -c mount -t ceph -o mds_namespace=smoke-test-fs,name=admin,secret=$(grep key /etc/ceph/keyring | awk '{print $3}') $(grep mon_host /etc/ceph/ceph.conf | awk '{print $3}'):/ /tmp/testrook
2020-09-30 21:49:36.592161 I | exec: timeout waiting for process kubectl to return. Sending interrupt signal to the process
2020-09-30 21:49:40.355420 E | utils: Failed to execute: kubectl [exec -n flex-ns rook-ceph-tools -- bash -c mount -t ceph -o mds_namespace=smoke-test-fs,name=admin,secret=$(grep key /etc/ceph/keyring | awk '{print $3}') $(grep mon_host /etc/ceph/ceph.conf | awk '{print $3}'):/ /tmp/testrook] : timeout waiting for the command kubectl to return.
```
Everything looks fine before showing the above-mentioned error in FAIL cases.
In addition, this problem hasn't happened in high-spec local machine
over 10 times.
Related issue: 6358
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>