This is handling a tricky scenario where the OSD deployment is manually removed and the OSD never reconvers. This is unlikely to happen, but still OSD should be able to run after that action. Essentially after a manual deletion, we need to run the prepare job again to re-hydrate the OSD information so that the OSD deployment can be deployed. On encryption, it is a little bit tricky since ceph-volume list again the main block won't return anything, so we need to target the encrypted block to list. There is another case this PR does not handle, which is the removal of the OSD deployment and then the node is restarted. This means that the encrypted container is not opened anymore. However, opening it requires more work like writing the key on the filesystem (if not coming from the Kubernete secret, eg,. KMS vault) and then run luksOpen. This is an extreme corner case probably not worth worrying about for now. Signed-off-by: Sébastien Han <seb@redhat.com>
Rook Test Framework
The Rook Test Framework is used to run end to end and integration tests on Rook. The framework depends on a running instance of Kubernetes. The framework also provides scripts for starting Kubernetes so users can quickly spin up a Kubernetes cluster. The Test framework is designed to install Rook, run tests, and uninstall Rook.
Install Kubernetes
To install Kubernetes follow Rook's developer guide.
Run Tests
1. Build rook
Now that the Kubernetes cluster is running we need to populate the Docker registry environment variables:
eval $(minikube docker-env -p minikube)
The build process will build and push the images to the Docker registry inside the Minikube virtual machine.
make build
Now tag the newly built images to rook/ceph:local-build for running tests, or rook/ceph:master if creating sample manifests::
docker tag $(docker images|awk '/build-/ {print $1}') rook/ceph:local-build
docker tag rook/ceph:local-build rook/ceph:master
2. Run integration tests
Some settings are available to run the tests under different environments. The settings are all configured with environment variables. See environment.go for the available environment variables. At least you should set the following variables.
export TEST_HELM_PATH=/tmp/rook-tests-scripts-helm/linux-amd64/helm
export TEST_BASE_DIR=WORKING_DIR
export TEST_SCRATCH_DEVICE=/dev/vdb
Please note that the integration tests erase the contents of TEST_SCRATCH_DEVICE.
To run all integration tests:
go test -v -timeout 7200s github.com/rook/rook/tests/integration
After running tests, you can get test logs under "tests/integration/_output".
To run a specific suite (uses regex):
go test -v -timeout 1800s -run CephSmokeSuite github.com/rook/rook/tests/integration
To run specific tests inside a suite:
go test -v -timeout 1800s -run CephSmokeSuite github.com/rook/rook/tests/integration -testify.m TestARookClusterInstallation_SmokeTest
3. To run tests on OpenShift environment
- Setup OpenShift environment and export KUBECONFIG before executing the tests.
- Make sure
ocexecutable file is in the PATH. - Only
CephSmokeSuiteis currently supported on OpenShift. - Set few environment variables:
export TEST_ENV_NAME=openshift
export TEST_STORAGE_CLASS=gp2
export TEST_BASE_DIR=/tmp
export RETRY_MAX=40
To run the CephSmokeSuite:
go test -v -timeout 1800s -run CephSmokeSuite github.com/rook/rook/tests/integration