The block integration tests are a source of intermittent failures
in the CI. The PVCs were not being confirmed as removed before the
pools were deleted. Then the pool would not be removed since there
might still be a block image from the PVC that wasn't deleted yet.
Now the tests will ensure the PVCs and their block images are
removed before the pool is deleted.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Depending on the state of the orchestration, the operator might trigger
re-creation of the deleted mon. In this case, consider the test successful
rather than wait for the failover which will never occur.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
With the desire to run the integration tests on the five
most recent versions of K8s, 1.16 is added and 1.11
is removed from the integration test matrix.
Signed-off-by: travisn <tnielsen@redhat.com>
The timeout of 80 seconds wasn't always sufficient for mon failover.
The timeout is now increased to 150 seconds.
Signed-off-by: travisn <tnielsen@redhat.com>
The two clusters were being created in parallel by the tests
and sometimes causing the test to timeout. The operator handles
the cluster serially so we don't gain anything by starting
them asynchronously. With this change the tests will be started
serially to match the behavior of the operator and avoid
the issues with test timeouts.
Signed-off-by: travisn <tnielsen@redhat.com>
The logs are only collected by default if a test failed.
Ensure that we collect the logs if the test setup step
failed.
Signed-off-by: travisn <tnielsen@redhat.com>
The BlockCreateSuite and BlockMountUnmountSuite suites are almost identical
for running block tests with different types of mounts. We can
consolidate these into a single test suite and eliminate a couple of
the tests that are already covered by other tests.
Signed-off-by: travisn <tnielsen@redhat.com>
Just as with the mon, mgr, mds, rgw, and rbd-mirror daemons, do not
generate a ceph.conf in an init container for directory-based OSDs.
Instead use the mon config database and the commandline to supply all
the needed arguments for running these OSDs.
Also generate a keyring secret to mount to OSD pods. Use this
secret for directory-based OSDs for now with the intention to
use this for other OSDs in the future.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
There are two separate integration tests for file. To simplify
and streamline and potentially eliminate some random failures,
these tests now consume the same file system instead of creating
a separate file system.
Signed-off-by: travisn <tnielsen@redhat.com>
The filesystem test deletes the client test pod by deleting the entire yaml
and sometimes hangs the integration tests. This is the most common intermittent
error in the CI. Now the deletion will happen more directly with a pod delete
command. If it does still fail deletion at least the deletion should timeout
sooner.
Signed-off-by: travisn <tnielsen@redhat.com>
There are six test suites and six k8s versions where we currently run
the tests. For efficiency we can restrict the testing to one suite
per k8s version. Bigger or riskier changes should still run the full
set of suites on all versions. To trigger the smaller test matrix, add
[test ceph min] to the PR description
Signed-off-by: travisn <tnielsen@redhat.com>
THe boolean helm settings were only being applied if their value was true.
If the desired value was false, the value would be skipped in the chart
instead of adding it with the value of false.
Signed-off-by: travisn <tnielsen@redhat.com>
This test doesn't really test the behavior of Rook as much as it tests
the behavior of `kubectl create`; therefore, remove the test.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
Some integration tests fail due to resources from a previous integration
run not being cleaned up properly. Instead of using 'kubectl create' --
which fails with an error if the resource already exists -- to create
resources, use 'kubectl apply' -- which does not fail for pre-existing
resources. 'kubectl apply' will give a warning that it should be used to
apply changes to resources created with 'create' or 'apply', but there
is no error.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
If the Cassandra integration tests are run before the Ceph tests
some of the Ceph tests will fail. There is a lurking interaction
between the tests that needs to be tracked down.
Signed-off-by: travisn <tnielsen@redhat.com>
With the 1.1 release we should not be creating bluestore OSDs
with rook's legacy partitioning scheme. All new bluestore OSDs
should be created with ceph-volume. Therefore, the miminum
version allowed is a version of mimic supporting ceph-volume.
This commit also adds consistent version checking between
the Add and Update methods for the operator handling the
CR events.
Signed-off-by: travisn <tnielsen@redhat.com>
Added code to delete secret, cephfs
and rbd pools and to delete cephfs
and rbd storage class.
delete rbd and cephfs pod,pvc
skip CSI test if kube version is less than 1.13.0
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
Add a prefix to the csi drivername when ceph csi is deployed by rook.
This aims to prevent trivial conflict between csi instances when two
rook operators are deployed into the same cluster. By default the
prefix is the "<rook-namespace>." but can by customized on the cli
and via environment vars.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
Now we collect logs for all pods and all their init and main
containers during the integration tests. No longer will we be
missing logs from the integration tests as long as the pods
are available when they are collected at the end of the test.
The pod descriptions are also written to a log file instead
of being included inline with the test output.
Signed-off-by: travisn <tnielsen@redhat.com>
(cherry picked from commit 6e0bc338f90b3f36a89cf37b542ff4ff873b70bd)
With the storage providers growing, we should make it
clear in the source which tests belong to which storage
provider. This change renames the ceph integration tests.
Signed-off-by: travisn <tnielsen@redhat.com>
Since most PRs are focused on a single storage provider,
the tests for other storage providers do not need to be
run. The CI will allow skipping tests for other storage
providers by specifying keywords in the PR description.
Similar to [skip ci], you can now specify [test storage-provider].
For example, [test ceph] will only trigger the test
suites for ceph and none of the other storage providers.
Signed-off-by: travisn <tnielsen@redhat.com>
Logs will be lost if we collect them after pods are uninstalled.
So we need to collect them before purging.
Signed-off-by: Sébastien Han <seb@redhat.com>
Use cmd-reporter to detect Ceph version.
The job to detect Ceph version needs to run in a security account context
capable of getting/creating ConfigMaps, so a new security account, role,
and role binding are created for the cmd-runner jobs. In the future, the
Ceph mgr may create jobs in this service account for running
`ceph-volume` on nodes, and this also keeps Rook operators controlling
multiple Ceph clusters independent.
Fixes#2686 - ceph version job not working correctly with graylog
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
The upgrade test is intended to upgrade from the previous release to the latest.
Given the 1.0 release recently, the upgrade test will now be from 1.0 to
master. There is no longer a need in master to upgrade from 0.9.
Signed-off-by: travisn <tnielsen@redhat.com>
When the operator first starts, the only operation needed
is to watch for new cephcluster crds to be created and
start the discovery to find available devices. The flexvolume
agent, the csi driver, and the volume provisioning can all be delayed
starting until the first cluster is created.
Signed-off-by: travisn <tnielsen@redhat.com>
Travis Nielsen
Sébastien Han
Blaine Gardner
Maksim Nabokikh
Kristoffer Grönlund
Sameer Kulkarni
jiffin
John Mulligan
Madhu Rajanna
Santosh Pillai
Ashish Ranjan