Files
my-rook-config/Documentation/Contributing/rook-test-framework.md
T
Sebastian Riese 85949049a3 rgw: add support for keystone auth + swift/s3
For the specification see:
<https://github.com/rook/rook/blob/master/design/ceph/object/swift-and-keystone-integration.md>

* extend the API object specs for swift and keystone integration

* adapt rgw to the new go-ceph version

  - The parameter lists of the API call have changes, as parameters
    ignored by the RGW Admin Ops API are no longer serialized, therefore
    the mock has to be adapted.

  - There is now validation for the user keys that are passed to the
    User get API, therefore things failed when we had empty keys in our
    User proxy object.

* expand the reconcile loop for the swift and keystone integration

* fix minor mistakes in design document

* add env var to pass extra args to minikube

  Minikube decides CPU cores and memory automatically based on the
  available resources on the machine which may be insufficient to
  run rook. This commit adds an environment variable to add arbitrary
  arguments to the minikube command, so both can be specified if
  desired.

* integration tests for swift and keystone

  The new integration of swift or s3 and keystone support by rook
  does not have any integration tests yet.

  This commit introduces integration tests for swift and keystone. The
  tests are done against a minimal keystone setup (keystone container
  image from Yaook-project (https://yaook.cloud), sqlite as database
  backend, cert-manager and trust-manager for test certificate setup).

  To prevent hardcoded credentials, passwords are generated
  by the tests. The integration tests use the openstack client
  (keystone- and swift-functionality) (https://docs.openstack.org/
  python-openstackclient/ latest/). This was a concious design decision
  to use client tooling as close as possible to the end user instead of
  using other go-libraries (such as gophercloud).

* add documentation on swift and keystone

  Currently there is no documentation on the use of Swift to access
  an object store as well as the use of OpenStack keystone for
  authentication.

  This commit adds documentation on the use of Swift and OpenStack
  keystone, as well as CRD-related documentation and an example setup.

* add integration tests for S3 via keystone

  This commit introduces integration tests for s3 and keystone. The
  tests are run against the same minimal keystone setup that the tests
  for swift and keystone use.

  The integration tests use the aws s3 client to use client tooling as
  close as possible to the end user instead of using other go-libraries.

Co-authored-by: Jan Klippel <jan.klippel@uhurutec.com>
Co-authored-by: Silvio Ankermann <silvio.ankermann@cloudandheat.com>
Signed-off-by: Sebastian Riese <sebastian.riese@cloudandheat.com>
Signed-off-by: Jan Klippel <jan.klippel@uhurutec.com>
Signed-off-by: Silvio Ankermann <silvio.ankermann@cloudandheat.com>
(cherry picked from commit ee8bcad49d)
2024-08-08 15:56:28 +00:00

3.4 KiB

title
title
Rook Test Framework

Integration Tests

The integration tests run end-to-end tests on Rook in a running instance of Kubernetes. The framework includes scripts for starting Kubernetes so users can quickly spin up a Kubernetes cluster. The tests are generally designed to install Rook, run tests, and uninstall Rook.

The CI runs the integration tests with each PR and each master or release branch build. If the tests fail in a PR, access the tmate for debugging.

This document will outline the steps to run the integration tests locally in a minikube environment, should the CI not be sufficient to troubleshoot.

!!! hint The CI is generally much simpler to troubleshoot than running these tests locally. Running the tests locally is rarely necessary.

!!! warning A risk of running the tests locally is that a local disk is required during the tests. If not running in a VM, your laptop or other test machine could be destroyed.

Install Minikube

Follow Rook's developer guide to install Minikube.

Build Rook image

Now that the Kubernetes cluster is running we need to populate the Docker registry to allow local image builds to be easily used inside Minikube.

eval $(minikube docker-env -p minikube)

make build will now build and push the images to the Docker registry inside the Minikube virtual machine.

make build

Tag the newly built images to rook/ceph:local-build for running tests, or rook/ceph:master if creating example manifests::

docker tag $(docker images|awk '/build-/ {print $1}') rook/ceph:local-build
docker tag rook/ceph:local-build rook/ceph:master

Run integration tests

Some settings are available to run the tests under different environments. The settings are all configured with environment variables. See environment.go for the available environment variables.

Set the following variables:

export TEST_HELM_PATH=/tmp/rook-tests-scripts-helm/helm
export TEST_BASE_DIR=WORKING_DIR
export TEST_SCRATCH_DEVICE=/dev/vdb

Set TEST_SCRATCH_DEVICE to the correct block device name based on the driver that's being used.

!!! hint If using the virtualbox minikube driver, the device should be /dev/sdb

!!! warning The integration tests erase the contents of TEST_SCRATCH_DEVICE when the test is completed

To run a specific suite, specify the suite name:

go test -v -timeout 1800s -run CephSmokeSuite github.com/rook/rook/tests/integration

After running tests, see test logs under tests/integration/_output.

To run specific tests inside a suite:

go test -v -timeout 1800s -run CephSmokeSuite github.com/rook/rook/tests/integration -testify.m TestARookClusterInstallation_SmokeTest

!!! info Only the golang test suites are documented to run locally. Canary and other tests have only ever been supported in the CI.

Running tests on OpenShift

  1. Setup OpenShift environment and export KUBECONFIG

  2. Make sure oc executable file is in the PATH.

  3. Only the CephSmokeSuite is currently supported on OpenShift.

  4. Set the following environment variables depending on the environment:

    export TEST_ENV_NAME=openshift
    export TEST_STORAGE_CLASS=gp2
    export TEST_BASE_DIR=/tmp
    
  5. Run the integration tests