Removed some obsolete instructions from the 1.1 upgrade
and made a few minor fixes to make the upgrade to 1.2 smoother.
With the addition of the CephClient CRD, the CRD needs to be
created before the upgrade or else the operator log will be
filled with errors until it is created. The CSI driver also
had an error when registering after the upgrade due to an RBAC
change.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Disable the dashboard in the upgrade test because upgrading Ceph from
Mimic to Nautilu with Rook v1.0 can stall for a LONG time when updating
the mgr module. We are pretty sure there is very little risk that the
upgrade test will expose problems with the dashboard module that aren't
caught by other integration tests.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
The file-test pod still hangs sometimes when exiting. Instead of
trapping SIGTERM as before, use tini's `-g` option to kill the process
group. According to further reading on bash signal handling [1], Ctrl-C
sends a kill signal to the entire process group in order to kill `sleep`
commands.
[1] Link also noted in code comments:
http://mywiki.wooledge.org/SignalTrap#When_is_the_signal_handled.3F
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
Reset the integration test k8s helper's `RetryLoop` to its original
value, and instead only wait an extra long time to allow the mgr module
updates to take a long time after Ceph is updated from Mimic to Nautilus
as part of Ceph's upgrade integration test.
Updating mgr modules can hang for quite a while, which causes the tests
to time out waiting for the OSDs to be updated. Allow this to take a
long time so the tests aren't as flaky.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
ceph client CRD refactoring
Add info about client crd to documentation
Add tests for client controller CRD
Signed-off-by: Mateusz Los <los.mateusz@gmail.com>
The helper script to start minikube has dead code that is
no longer used. Specifically, the flex driver config isn't needed
anymore and the RBAC rules are all configured in the manifests.
Neither is there a need to have a restart option on minikube.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Before starting an integration test, run a job on all nodes to wipe the
disks so that the integration tests will actually test Rook using the
disks for storage.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
During upgrade tests, Rook should verify that it can still run legacy
OSDs. This includes directory-based OSDs, filestore disk OSDs, and
bluestore disk OSDs installed without ceph-volume (i.e., before mimic
v13.2.2) can still be run after upgrade.
This necessitates running the upgrade test twice; once with filestore
and once with bluestore.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
A new controller to bootstrap ceph-crash pod on Ceph nodes running Ceph
pods only.
This implementation is unfortunately a mix of 'hacks' to make the daemon
working correclty. The ceph-crash script faced different issues:
* it's not a daemon
* it does not have any key in cephx
* it runs ceph commands under the hood which need an admin key and a
ceph.conf
On upstream Ceph, ceph-crash needs to grow better, once that happens we
will improve our implementation.
This also enhances osd provisionConfig struct with DataPathMap
Pod volumes and volume mount needs DataPathMap to perform the right
actions on log and crash dir. Exposing DataPathMap makes that possible.
Obviously, this is exposing ceph crash reports on the host as
well as pushing them into the mgr.
Basically, if a daemon fails, it'll put its core dump into
/var/lib/ceph/crash, bindmounting this dir on the host, ensures that the
crashes don't get lost when the pod dies.
Example:
```
[leseb@tarox~/go/src/github.com/rook/rook][rgw-liveprobe !] kubectl -n rook-ceph exec -ti rook-ceph-crashcollector-minikube-574858b99c-zvg4z bash
bash: warning: setlocale: LC_CTYPE: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_COLLATE: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_MESSAGES: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_NUMERIC: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_TIME: cannot change locale (en_US.UTF-8): No such file or directory
[root@rook-ceph-crashcollector-minikube-574858b99c-zvg4z /]# ceph crash ls
[root@rook-ceph-osd-2-7644f99695-cljzh /]# pidof ceph-osd
13258 5415 5397
[root@rook-ceph-osd-2-7644f99695-cljzh /]# kill -SIGABRT 13258
[root@rook-ceph-osd-2-7644f99695-cljzh /]# ls /var/lib/ceph/crash/
2019-11-12_12:56:51.404109Z_39f060e1-776d-4605-8f28-85c97e53de96 posted
... wait maximum 10 min (ceph-crash scraps every 10 minutes)
... the container will exit
[root@rook-ceph-osd-2-7644f99695-cljzh /]# exit
[leseb@tarox~/go/src/github.com/rook/rook][rgw-liveprobe !]
[leseb@tarox~/go/src/github.com/rook/rook][rgw-liveprobe !] kubectl -n rook-ceph exec -ti rook-ceph-crashcollector-minikube-574858b99c-zvg4z bash
bash: warning: setlocale: LC_CTYPE: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_COLLATE: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_MESSAGES: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_NUMERIC: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_TIME: cannot change locale (en_US.UTF-8): No such file or directory
[root@rook-ceph-crashcollector-minikube-574858b99c-zvg4z /]#
[root@rook-ceph-crashcollector-minikube-574858b99c-zvg4z /]#
[root@rook-ceph-crashcollector-minikube-574858b99c-zvg4z /]# ceph crash ls
2019-11-12_12:56:51.404109Z_39f060e1-776d-4605-8f28-85c97e53de96 osd.1
[root@rook-ceph-crashcollector-minikube-574858b99c-zvg4z /]# ls /var/lib/ceph/crash/
posted
[root@rook-ceph-crashcollector-minikube-574858b99c-zvg4z /]# ls /var/lib/ceph/crash/posted/
2019-11-12_12:56:51.404109Z_39f060e1-776d-4605-8f28-85c97e53de96
[root@rook-ceph-crashcollector-minikube-574858b99c-zvg4z /]# tail /var/lib/ceph/crash/posted/2019-11-12_12\:56\:51.404109Z_39f060e1-776d-4605-8f28-85c97e53de96/log
1/ 5 mgr
1/ 5 mgrc
1/ 5 dpdk
1/ 5 eventtrace
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/lib/ceph/crash/2019-11-12_12:56:51.404109Z_39f060e1-776d-4605-8f28-85c97e53de96/log
--- end dump of recent events ---
```
Signed-off-by: Sébastien Han <seb@redhat.com>
Co-authored-by: Rohan CJ <rohantmp@gmail.com>
Using the current working directory as the base test directory
fails with an unclear error message when running mkfs for OSDs
if tests are run when the k8s master node is not running on
localhost.
This change creates a new flag for the test suite to set the
base test directory, but uses a default that is safe for
virtualized test environments ("/data").
Fixes: #4205
Signed-off-by: egafford <egafford@redhat.com>
This will be helpful to list the CSI drivers
deployed in the kubernetes cluster. if we have
N number of CSI drivers deployed in the cluster,
the user can query kubectl get csidriver and get
the list of registered CSI drivers and it requires
kube 1.14+
This removes the deployment of Attacher sidecar container,
as we are setting the attacher required to false during
the CSIDriver object creation.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
Real mons and canary both have the same rook-ceph-mon label.
It creates a confusion while retrieving list of mon deploymets
using label selector. This commit adds a check to remove canary
deployments from the deployment list.
Signed-off-by: Umanga Chapagain <chapagainumanga@gmail.com>
retry ObjectUser creation as long as valid input is provided
or it times out. Retry every 15sec for 5min.
integration test creates ObjectUser before ObjectStore to
ensure that ObjectUser waits for ObjectStore to be created
and available
Closes: https://github.com/rook/rook/issues/3937
Signed-off-by: Umanga Chapagain <chapagainumanga@gmail.com>
The topology of the cluster should be based on the node labels rather
than a setting in the cluster CR. This allows a much richer and more
dynamic topology to be configured. The location will now be ignored
if specified in the cluster CR.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The integration tests were creating several namespace-specific resources
in the system namespace instead of the namespace with the cluster. This
was only affecting the rook-ceph-osd and rook-ceph-mgr service accounts,
which hadn't been exercised until the nodes were queried by osds for
topology awareness.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
We removed the topologyAware CRD option since it was redundant with the
use of an OSD being backed by a PVC.
So now, if an OSD is backed by a PVC we assume the topology aware
decision and will discover zone and region labels on that host.
Signed-off-by: Sébastien Han <seb@redhat.com>
The block integration tests are a source of intermittent failures
in the CI. The PVCs were not being confirmed as removed before the
pools were deleted. Then the pool would not be removed since there
might still be a block image from the PVC that wasn't deleted yet.
Now the tests will ensure the PVCs and their block images are
removed before the pool is deleted.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
PR #3611 added the possibility of using something other
than docker as the container CLI, but the readme wasn't
updated.
[skip ci]
Signed-off-by: Kristoffer Grönlund <kgronlund@suse.com>
Depending on the state of the orchestration, the operator might trigger
re-creation of the deleted mon. In this case, consider the test successful
rather than wait for the failover which will never occur.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Creating an object user was always waiting at least 15 seconds before
attempting to create the user. With this change the operator will check
immediately if the object store is already initialized instead
of waiting for the initial 15 seconds delay to elapse.
Signed-off-by: travisn <tnielsen@redhat.com>
With the desire to run the integration tests on the five
most recent versions of K8s, 1.16 is added and 1.11
is removed from the integration test matrix.
Signed-off-by: travisn <tnielsen@redhat.com>
Helm requires some changes to be compatible with k8s 1.16.
The deployment needs to use v1 apiVersion and a selector
needs to be added. A tiller service account is also required.
Signed-off-by: travisn <tnielsen@redhat.com>
K8s 1.16 requires apps/v1 instead of the old extensions/v1beta1 for the
apiVersion of the deployment. The tests were failing when testing on
K8s 1.16 without this change
Signed-off-by: travisn <tnielsen@redhat.com>
The timeout of 80 seconds wasn't always sufficient for mon failover.
The timeout is now increased to 150 seconds.
Signed-off-by: travisn <tnielsen@redhat.com>
The two clusters were being created in parallel by the tests
and sometimes causing the test to timeout. The operator handles
the cluster serially so we don't gain anything by starting
them asynchronously. With this change the tests will be started
serially to match the behavior of the operator and avoid
the issues with test timeouts.
Signed-off-by: travisn <tnielsen@redhat.com>
The logs are only collected by default if a test failed.
Ensure that we collect the logs if the test setup step
failed.
Signed-off-by: travisn <tnielsen@redhat.com>
The BlockCreateSuite and BlockMountUnmountSuite suites are almost identical
for running block tests with different types of mounts. We can
consolidate these into a single test suite and eliminate a couple of
the tests that are already covered by other tests.
Signed-off-by: travisn <tnielsen@redhat.com>
Just as with the mon, mgr, mds, rgw, and rbd-mirror daemons, do not
generate a ceph.conf in an init container for directory-based OSDs.
Instead use the mon config database and the commandline to supply all
the needed arguments for running these OSDs.
Also generate a keyring secret to mount to OSD pods. Use this
secret for directory-based OSDs for now with the intention to
use this for other OSDs in the future.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
The tests must ensure that the cluster is cleaned up so that
other test suites will not be affected by cleanup of a previous
test suite. Removing the cluster finalizer is critical to this
cleanup.
Signed-off-by: travisn <tnielsen@redhat.com>
There are two separate integration tests for file. To simplify
and streamline and potentially eliminate some random failures,
these tests now consume the same file system instead of creating
a separate file system.
Signed-off-by: travisn <tnielsen@redhat.com>
The filesystem test deletes the client test pod by deleting the entire yaml
and sometimes hangs the integration tests. This is the most common intermittent
error in the CI. Now the deletion will happen more directly with a pod delete
command. If it does still fail deletion at least the deletion should timeout
sooner.
Signed-off-by: travisn <tnielsen@redhat.com>
Travis Nielsen
Blaine Gardner
Sébastien Han
Mateusz Los
Rohan CJ
egafford
Umanga Chapagain
Madhu Rajanna
Kristoffer Grönlund
Sebastian Wagner