There are the following problems in the cluster health check.
- If the cluster is healthy: Unnecessary long time before exiting from
the health check loop. The test succeeds thanks to the succeeding
assertion by luck.
- If the cluster is unhealthy: Exit from the health check loop immediately.
The test fails in the succeeding assertion by luck too.
It's a regression that is introduced in the following commit.
https://github.com/rook/rook/commit/c0c0dc4ed977a6b373b1884c5b76c9588f0be077
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
In Octopus 15.2.2 there is an rbd change that causes the rbd ls command
to hang if there are not sufficient OSDs to meet the replica requirement.
The smoke suite just started hanging today since the 15.2.2 image was
released with this change. Therefore, the test that resizes the pool
to replica 3 must resize back to 1 before cleaning up.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The PVCs for OSD on PVC test is prepared before running the integration test.
In addition, users are forced to run this script for this by themselves.
To avoid this problem, running this script inside the integration test.
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
Directory-based OSD is not supported in v1.3.0 anymore. Let's remove
the descriptions of this feature. As an exception, it should still
be in the manifests of v1.2.
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
There are several functions that are prefixed by "Is" and return
just error. It's straightfoward to return bool to make the meaning
of these functions clearer.
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
Ceph orchestrator device ls
Ceph orchestrator status
Ceph orchestrator host ls
Ceph orchestrator create OSD
Ceph orchestrator ls
[test ceph]
Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
The name of a CephCluster is commonly the same as the namespace,
but not always. When looking up the ceph cluster we can look for
the first one in the namespace rather than requiring the name
to be the same as the namespace.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Currently, we need to configure the Ceph external Admin keyi
in the Rook deployment to be able to connect to an external Ceph cluster.
If we wanted to run in a multi-tenant fashion
were several K8s/Rook clusters wanted to connect to the same external ceph cluster,
each K8s deployment would have the access to the External Ceph Admin key
and could potentially access or delete the Data
from pools that belong to other k8s/Rook Clusters.
Now the admin key is optional but the helper script create-external-cluster-resources.sh
will help create the necessary keys/users to connect to that cluster.
Closes: https://github.com/rook/rook/issues/4917 and https://github.com/rook/rook/pull/5227
Signed-off-by: Sébastien Han <seb@redhat.com>
Refactor the StartTestCluster method to make it more readable.
I have put most of the parameters used in this function in the struct.
In this way the call to this function is more easy to understand and to use.
The addition of new parameters to this method started to be annoying.
[test ceph]
Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Previously, the rbd-mirror daemon was integrated into the `CephCluster`
CRD. This wasn't really practical since we would have to wait for the
whole orchestration to be done to actually set it up. The same goes for
any CR update. Let's say you want to change the number of daemons, Rook
would go through mons, mgrs and osds until it get to rbd-mirror.
This triggers an undesired full orchestration.
With its own CRD this component just gains a lot more flexibility.
Closes: https://github.com/rook/rook/issues/5084
Signed-off-by: Sébastien Han <seb@redhat.com>
The command line flags parsing inside an init() method was causing
issues at runtime when trying to add certain new features, which
is a limitation of the flag parsing. We now move away from using
any command line flags in the tests and only use environment variables
to simplify adding new parameters to the tests.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
On Octopus, the "ceph fs status" reports on stderr for some reason.
So we need to combined the output to read the filesystem status.
Signed-off-by: Sébastien Han <seb@redhat.com>
The upgrade test will now support upgrading from nautilus
to octopus and that files and be read and written
after the upgrade.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The upgrade test requires the CSI driver to be running so we cannot
run the test on older versions than k8s 1.13.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
(cherry picked from commit f90664209b)
With 1.18 in the test matrix we need to update which versions
will run the different ceph test suites.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
With the release of Octopus we need to run the integration tests
on Octopus in order to have validation as a supported platform.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
In order to ensure proper clean up of all the rook-ceph data when the cluster is deleted, we need to clean up the dataDirHostPath (var/lib/rook)
Signed-off-by: Santosh Pillai <sapillai@redhat.com>
This fix is added to ensure that everything works as expected
when user tries to create ObjectStoreUser before creating
ObjectStore itself. ObjectStoreUser will wait for ObjectStore
to be up and running.
Signed-off-by: Umanga Chapagain <chapagainumanga@gmail.com>
There are many namespace fields in ClusterRole{,Binding}. However,
ClusterRole{,Binding} are not namespaced. So we can remove these.
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
The pool cleanup only needs to happen for an individual pool.
No need to query the block images in all pools. One of the rgw
pools is periodically causing a hang when it is queried,
but there is no need to query for it when we are cleaning
up the pool tests.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Purge the object store properly, otherwise the cephobjectstore CRD won't
be deleted since a finalizer is in place for the CR.
Signed-off-by: Sébastien Han <seb@redhat.com>
The CRD watcher has been replaced by the new controller-runtime
framework.
This brings robustness in our operator, meaning that any resources that
are modified will be reconciled into the desired state.
Closes: https://github.com/rook/rook/issues/4937
Signed-off-by: Sébastien Han <seb@redhat.com>
With a finalizer on the pools, the pools were not always being purged
during the integration tests. Now the multicluster suite will ensure
its pool is purged.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The integration tests have been mostly running on the flex driver
with only a newer test on the csi driver. With the CSI driver being
the preferred driver going forward, now the integration tests will
all be running with the CSI driver with the exception of a test
suite that is only dedicated to the flex driver.
A number of other test improvements are also made for code
readability, test stability, and removing unused options.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The file-test consumer pod intermittently fails to stop during the
integration tests. When this happens, print the pod description
to see if there are events indicating the issue.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The upgrade test had duplicated code for checking that daemons
were upgraded after the Rook upgrade vs the Ceph upgrade.
This is now refactored for consistent checks after upgrade.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The integration tests were always running two MDS daemons active,
with two standby. This is now parameterized so the test can request
how many MDS daemons to run. The smoke suite will run two active
and the rest will just run a single active MDS.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The integration test was disabled due to removing support of pre-ceph-volume
OSDs as well as OSDs on directories. Now we re-enable the tests, with the
following approach:
- The base install in Rook v1.1 with the latest mimic release that
had c-v support
- The upgrade goes from v1.1 to Rook v1.2 then Rook master
- The final upgrade step is from mimic to nautilus
For efficiency the skipUpgradeChecks flag is added.
Logs are also collected between each upgrade step to improve
troubleshooting.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
This commit enables testing OSDs over PVCs by modifying the
multi-cluster test to use PVC for provisioning OSDs when `manual`
storageClass is present in the cluster.
Signed-off-by: Ashish Ranjan <aranjan@redhat.com>
The integration tests only have a single node so the upgrade
checks do not provide any real safety for the cluster during
the upgrade. We will be able to improve the reliability and speed
of the tests by disabling these checks.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Multiple things:
1. We removed all the function/methods/tests that were used to
create and manage rook legacy OSDS as well as bringing support to
Bluestore OSD only.
It also fixes various go-lint issues in the respectives files.
2. use c-v inventory to detect available devices:
Now we rely on the 'ceph-volume inventory' command to tell us if a
device is available or not.
3. implement raw mode for osd on pvc
When an OSD will be bootstrap on a PVC, the new c-v raw mode will be
used. It consists of putting block, db and wal under the same device.
Here LVM is out of the picture and the raw device is used as is. The
implementation is backward compatible so existing OSD on PVC will LVM
will continue to operate.
Closes: https://github.com/rook/rook/issues/4363
Signed-off-by: Sébastien Han <seb@redhat.com>
When the tests run in a PR, they can only run against a single version
of K8s by default. If more than five k8s versions are supported in hte
CI, we will need to run some of the suites on multiple versions.
The versions are comma-separated in the list.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
In the Ceph upgrade integration test, upgrade from Rook v1.0 with Ceph
Mimic v13 to install legacy disk-based OSDs. Then upgrade Ceph to
Nautilus, then Rook to v1.1, then v1.2, then v1.3 (master currently) to
make sure that Rook is able to run legacy OSDs created without
ceph-volume.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
With the release of K8s 1.17 Rook needs to test on this new version.
The integration tests will now run the tests across K8s 1.13-1.17.
The pattern has been to run the tests across the most recent five
versions.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
The file-test pod still hangs sometimes when exiting. Instead of
trapping SIGTERM as before, use tini's `-g` option to kill the process
group. According to further reading on bash signal handling [1], Ctrl-C
sends a kill signal to the entire process group in order to kill `sleep`
commands.
[1] Link also noted in code comments:
http://mywiki.wooledge.org/SignalTrap#When_is_the_signal_handled.3F
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>
Satoru Takeuchi
Travis Nielsen
Juan Miguel Olmo Martínez
Sébastien Han
Santosh Pillai
Umanga Chapagain
Ashish Ranjan
Blaine Gardner
Mateusz Los