Satoru Takeuchi
15a159b1ff
If we try to run the integration test with `go test`, it fails as follows. ``` $ go test -v -timeout 1800s -run CephMultiClusterDeploySuite github.com/rook/rook/tests/integration === RUN TestCephMultiClusterDeploySuite 2020-09-02 00:00:31.874738 I | installer: test environment variable (default) "STORAGE_PROVIDER_TESTS"="" 2020-09-02 00:00:31.950226 I | testutil: Loaded kubectl context kubernetes at https://192.168.123.2:6443. secure=false 2020-09-02 00:00:31.951121 I | integrationTest: running all tests 2020-09-02 00:00:32.027964 I | testutil: Loaded kubectl context kubernetes at https://192.168.123.2:6443. secure=false 2020-09-02 00:00:32.028640 I | installer: Rook Version: master 2020-09-02 00:00:32.028667 I | installer: Ceph Version: ceph/ceph:v14 2020-09-02 00:00:32.028686 I | installer: test environment variable "TEST_HELM_PATH"="/tmp/rook-tests-scripts-helm/linux-amd64/helm" 2020-09-02 00:00:32.031927 I | installer: test environment variable "TEST_SCRATCH_DEVICE"="/dev/sdb" 2020-09-02 00:00:32.031982 I | testutil: Running tests/scripts/localPathPV.sh [/dev/sdb] ceph_multi_cluster_test.go:161: Error Trace: ceph_multi_cluster_test.go:161 ceph_multi_cluster_test.go:151 ceph_multi_cluster_test.go:81 suite.go:102 ceph_multi_cluster_test.go:62 Error: Received unexpected error: fork/exec tests/scripts/localPathPV.sh: no such file or directory Test: TestCephMultiClusterDeploySuite --- FAIL: TestCephMultiClusterDeploySuite (0.16s) FAIL FAIL github.com/rook/rook/tests/integration 0.211s FAIL $ ``` Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com> Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
145 lines
5.1 KiB
Markdown
145 lines
5.1 KiB
Markdown
# Rook Test Framework
|
|
|
|
The Rook Test Framework is used to run end to end and integration tests on Rook. The framework depends on a running instance of Kubernetes.
|
|
The framework also provides scripts for starting Kubernetes using `kubeadm` or `minikube` so users can
|
|
quickly spin up a Kubernetes cluster.The Test framework is designed to install Rook, run tests, and uninstall Rook.
|
|
|
|
## Requirements
|
|
1. Docker version => 1.2 && < 17.0 (for other alternatives, see below)
|
|
1. Ubuntu 18 (the framework has only been tested on this version)
|
|
1. kernel >= 4.15
|
|
1. Kubernetes with kubectl configured
|
|
1. Rook
|
|
|
|
## Instructions
|
|
|
|
## Setup
|
|
|
|
### Container Runtime
|
|
|
|
By default, the test suite tries to use the `docker` command line tool. To use
|
|
an alternative container runtime tool, set the `DOCKERCMD` environment variable
|
|
to your command line tool of choice.
|
|
|
|
### Install Kubernetes
|
|
You can choose any Kubernetes flavor of your choice. The test framework only depends on kubectl being configured.
|
|
The framework also provides scripts to install Kubernetes. There are two scripts to start the cluster:
|
|
1. **Minikube** (recommended for MacOS): Run [minikube.sh](/tests/scripts/minikube.sh) to setup a single-node Minikube Kubernetes.
|
|
1. **Kubeadm** (recommended for Ubuntu): run [kubeadm.sh](/tests/scripts/kubeadm.sh) to setup a single-node K8s cluster using kubeadm
|
|
|
|
#### Minikube (recommended for MacOS)
|
|
Starting the cluster on Minikube is as simple as running:
|
|
```console
|
|
tests/scripts/minikube.sh up
|
|
```
|
|
|
|
To copy Rook images generated from your local build into the Minikube VM, run the following commands after `minikube.sh up` succeeded:
|
|
```console
|
|
tests/scripts/minikube.sh update
|
|
tests/scripts/minikube.sh helm
|
|
```
|
|
|
|
Stopping the cluster and destroying the Minikube VM can be done with:
|
|
```console
|
|
tests/scripts/minikube.sh clean
|
|
```
|
|
|
|
#### Kubeadm (recommended for Ubuntu)
|
|
Starting the cluster using `kubeadm` is as simple as running:
|
|
```console
|
|
tests/scripts/kubeadm.sh up
|
|
```
|
|
|
|
`kubeadm.sh` starts Kubernetes on the host, so all the images generated by your local build should be available to the cluster without any additional commands.
|
|
You can change the Kubernetes version (default is v1.15.12) with `KUBE_VERSION` environment variable. Here is the example of using Kubernetes v1.19.0.
|
|
|
|
```console
|
|
export KUBE_VERSION=v1.19.0
|
|
tests/scripts/kubeadm.sh up
|
|
```
|
|
|
|
Please note that this script may downgrade the following deb packages.
|
|
|
|
- kubernetes-cni
|
|
- kubelet
|
|
- kubeadm
|
|
|
|
Stopping the cluster can be done with:
|
|
```console
|
|
tests/scripts/kubeadm.sh clean
|
|
```
|
|
|
|
#### Alternate Kubernetes Versions
|
|
These two scripts can install any version of Kubernetes you wish based on the `KUBE_VERSION` environment variable.
|
|
To use an alternate version, simply set this variable before running the relevant `up` command from above.
|
|
For example, if you wanted to use `v1.12.5`, you would run `export KUBE_VERSION=v1.12.5` first before running `up`.
|
|
|
|
### Install Helm
|
|
Use [helm.sh](/tests/scripts/helm.sh) to install Helm and set up Rook charts defined under `_output/charts` (generated by build):
|
|
|
|
- To install and set up Helm charts for Rook run `tests/scripts/helm.sh up`.
|
|
- To clean up `tests/scripts/helm.sh clean`.
|
|
|
|
**NOTE:** `*kubeadm.sh`, `minikube.sh` and `helm.sh` scripts depend on some artifacts under the `_output/` directory generated during build time,
|
|
these scripts should be run from project root. e.g., `tests/script/kubeadm.sh up`.
|
|
|
|
**NOTE**: If Helm is not available in your `PATH`, Helm will be downloaded to a temporary directory (`/tmp/rook-tests-scripts-helm`) and used from that directory.
|
|
|
|
## Run Tests
|
|
From the project root do the following:
|
|
|
|
### 1. Build rook:
|
|
Run `make build`
|
|
|
|
### 2. Start Kubernetes
|
|
Using one of the following:
|
|
|
|
- Using Kubeadm
|
|
```
|
|
tests/scripts/kubeadm.sh up
|
|
export KUBECONFIG=~/admin.conf
|
|
tests/scripts/helm.sh up
|
|
```
|
|
- Using minikube
|
|
```
|
|
tests/scripts/minikube.sh up
|
|
tests/scripts/minikube.sh helm
|
|
tests/scripts/helm.sh up
|
|
```
|
|
|
|
### 3. Run integration tests:
|
|
|
|
Some settings are available to run the tests under different environments. The settings are all configured with environment variables.
|
|
See [environment.go](/tests/framework/installer/environment.go) for the available environment variables.
|
|
At least you should set the following variables.
|
|
|
|
```console
|
|
export TEST_HELM_PATH=/tmp/rook-tests-scripts-helm/linux-amd64/helm
|
|
export TEST_BASE_DIR=WORKING_DIR
|
|
export TEST_SCRATCH_DEVICE=<block device> # for example, TEST_SCRATCH_DEVICE=/dev/sdb
|
|
```
|
|
|
|
Please note that the integration test erases the contents of TEST_SCRATCH_DEVICE.
|
|
|
|
To run all integration tests:
|
|
```console
|
|
_output/tests/linux_amd64/integration -test.v -test.timeout 1800s
|
|
```
|
|
|
|
You can only run the tests of the specific storage provider. If you want to test only Ceph, run the test as follows.
|
|
|
|
```console
|
|
export STORAGE_PROVIDER_TESTS=ceph
|
|
_output/tests/linux_amd64/integration -test.v -test.timeout 1800s
|
|
```
|
|
|
|
To run a specific suite (uses regex):
|
|
```console
|
|
_output/tests/linux_amd64/integration -test.v -test.timeout 1800s -test.run CephSmokeSuite
|
|
```
|
|
|
|
To run specific tests inside a suite:
|
|
```console
|
|
_output/tests/linux_amd64/integration -test.v -test.timeout 1800s -test.run CephSmokeSuite -testify.m TestARookClusterInstallation_SmokeTest
|
|
```
|