Joshua Hoblitt
8ee5d92022
Add a conventions doc and conversion playbook so the remaining old-style object tests (store lifecycle, notifications, COSI, bucket rw/quota/policy/ lifecycle) can be converted to this pattern mechanically. Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
216 lines
11 KiB
Markdown
216 lines
11 KiB
Markdown
# Object storage integration tests
|
||
|
||
This tree holds the new-style integration tests for rook's object storage
|
||
features. It is the model for converting the remaining old-style object tests
|
||
(`testObjectStoreOperations`, store lifecycle, bucket notifications, COSI);
|
||
follow the conventions here exactly so the conversion happens once.
|
||
|
||
## Execution model
|
||
|
||
- One Kubernetes cluster is shared by the whole `TestCephObjectSuite` run. The
|
||
testify suite in `tests/integration/ceph_object_test.go` owns cluster
|
||
lifecycle and dispatches into the packages here; these packages are plain Go
|
||
libraries, **not** standalone `go test` targets.
|
||
- A single shared `CephObjectStore` fixture (`util/sharedstore`) is created
|
||
once per suite pass and torn down after all packages run. It provides the
|
||
store CR, a go-ceph rgw admin client, and an SNS client, and is created with
|
||
or without TLS to match the pass. The packages run in both passes: the
|
||
with/without-TLS split runs them as separate parallel jobs, so the TLS pass
|
||
adds no serial runtime to a single job.
|
||
- Packages run **sequentially** and each test is an ordered script of
|
||
subtests. Do not use `t.Parallel`: the steps depend on prior steps, and the
|
||
cluster is shared.
|
||
|
||
## Layout
|
||
|
||
The tree mirrors `pkg/operator/ceph/object/*`: one top-level directory per
|
||
operator package, with a feature subdirectory layer when multiple test
|
||
packages cover subsets of one operator package. The operator ROOT package
|
||
(the CephObjectStore controller itself) maps to the tree root,
|
||
`tests/integration/object/` itself, not a subdirectory. Leaf package names
|
||
must not collide with std-lib package names (no `io`, no `http`).
|
||
|
||
| test package | operator package | covers |
|
||
|---|---|---|
|
||
| `bucket/owner` | `object/bucket` | OBC `bucketOwner` handling |
|
||
| `topic/kafka` | `object/topic` | CephBucketTopic kafka endpoints |
|
||
| `user/caps` | `object/user` | user capabilities |
|
||
| `user/keys` | `object/user` | explicit S3 key management |
|
||
| `user/opmask` | `object/user` | user op_mask |
|
||
| reserved: `bucket/rw`, `bucket/quota`, `bucket/policy`, `bucket/lifecycle`, `tests/integration/object/lifecycle`, `tests/integration/object/dependents`, `notification/*`, `cosi` | | future conversions |
|
||
|
||
Shared utilities live under `util/`:
|
||
|
||
- `wait4` — waits for object-test state: watch-based `Assert/Require` ×
|
||
`Create/Delete/Condition/Absent` for k8s resources, `Assert/RequireEventually`
|
||
polling for non-k8s state, `Assert/RequirePodLog` for a matching line in a
|
||
pod's log stream, and the shared timeout tiers.
|
||
- `ready` — readiness predicates for the wait4 helpers (`ObjectStoreUser`,
|
||
`BucketTopic`, `OBCBound`, ...).
|
||
- `fixture` — create-with-`t.Cleanup` helpers for pure-cleanup resources
|
||
(namespaces, StorageClasses).
|
||
- `obc` — pure constructors for the resources ObjectBucketClaim tests build
|
||
(the provisioner `StorageClass`; OBCs and per-OBC S3 clients as `bucket/*`
|
||
is converted).
|
||
- `secrets` — verification helpers for the Secret references object CRDs
|
||
publish in their status, shared by more than one package.
|
||
- `sharedstore` — the shared CephObjectStore fixture.
|
||
- `admin`, `sns`, `s3`, `tls` — client builders and TLS cert generation.
|
||
|
||
## Anatomy of a package
|
||
|
||
One exported entry function per package, called from the dispatcher:
|
||
|
||
```go
|
||
func TestObjectStoreUserCaps(t *testing.T, k8sh *utils.K8sHelper, store *sharedstore.Sharedstore) {
|
||
var (
|
||
defaultName = "test-usercaps"
|
||
objectStore = store.ObjectStore()
|
||
adminClient = store.AdminClient()
|
||
|
||
ns = &corev1.Namespace{ObjectMeta: metav1.ObjectMeta{Name: defaultName}}
|
||
|
||
osu1 = cephv1.CephObjectStoreUser{ /* fixture literal or constructor */ }
|
||
|
||
osuClient = k8sh.RookClientset.CephV1().CephObjectStoreUsers(ns.Name)
|
||
)
|
||
|
||
t.Run("ObjectStoreUser caps", func(t *testing.T) {
|
||
ctx := t.Context()
|
||
|
||
fixture.RequireNamespace(t, k8sh, ns)
|
||
|
||
t.Run(fmt.Sprintf("create CephObjectStoreUser %q", osu1.Name), func(t *testing.T) {
|
||
wait4.RequireCreate(ctx, t, osuClient, &osu1, ready.ObjectStoreUser, wait4.TimeoutLong)
|
||
})
|
||
|
||
// ... ordered subtests: act, verify, act, verify ...
|
||
})
|
||
}
|
||
```
|
||
|
||
Rules encoded in that shape:
|
||
|
||
- The entry signature is `(t, k8sh, store)` and never changes. New
|
||
dependencies are added as `Sharedstore` accessors, not parameters.
|
||
- The var block declares everything the test uses: fixture objects (use
|
||
constructors — `obc.StorageClass`, package-local helpers like
|
||
`awsKeySecret`/`objectUserKey` — to keep literals one line each), then the
|
||
typed clients bound to locals (`osuClient`, `obcClient`, `obClient`, ...).
|
||
Spelled-out client chains do not appear in test bodies.
|
||
- Exactly one outer `t.Run` per package, named `"<Subject> <feature>"`. It
|
||
must be unique across all object packages: it is the `-run` filter handle.
|
||
- `ctx := t.Context()` is the first line inside the outer `t.Run` — contexts
|
||
are test-scoped, so nothing outlives its test. (Helpers that wrap their own
|
||
`t.Run` bind their own `ctx := t.Context()` from the subtest's `t`.)
|
||
- `fixture.Require*` calls come next. Fixture teardown runs via `t.Cleanup`,
|
||
so it happens even when the test fails partway. Only PURE-CLEANUP teardown
|
||
belongs there; a deletion that asserts behavior ("deleting the OBC must not
|
||
delete the user") is an ordered subtest.
|
||
- Repeated multi-assert verification goes in a package-local
|
||
`check<Aspect>(t, deps, subject, expected ...)` helper that wraps its own
|
||
`t.Run` and takes the expected state variadically. Promote a helper to a
|
||
shared `util/` package named for its subject (e.g. `util/secrets`) only
|
||
when a second package needs it.
|
||
|
||
## Waiting
|
||
|
||
- State visible to the Kubernetes API uses the watch-based `wait4` verbs:
|
||
`RequireCreate` (create + wait ready), `AssertDelete`/`RequireDelete`
|
||
(delete + wait gone), `RequireCondition`/`AssertCondition` (wait for a
|
||
predicate on an existing resource), `AssertAbsent`/`RequireAbsent` (wait
|
||
for cascade deletion you did not issue). Predicates come from `util/ready`
|
||
or are inlined when test-specific.
|
||
- State NOT visible to the Kubernetes API (rgw admin ops, S3, SNS) uses
|
||
`wait4.AssertEventually`/`RequireEventually`, the assert/require-flavored
|
||
wrappers over the framework's `utils.Eventually` poll loop. Poll closures
|
||
must not call `assert` or `require` — return an error on transient failures
|
||
(it is surfaced in the timeout message) and assert on captured state after
|
||
the wait (capture-on-success: assign an outer variable inside the closure
|
||
just before returning nil). The closure receives a context bounded by the
|
||
wait deadline; thread it into the reads it performs.
|
||
- Log-based state (an operator or endpoint logging a line) uses
|
||
`wait4.AssertPodLog`/`RequirePodLog`, which follows the first matching pod's
|
||
log stream until a line satisfies the predicate or the timeout elapses.
|
||
- Timeouts come from the shared tiers: `wait4.TimeoutShort` (routine status
|
||
changes), `TimeoutMedium` (full reconcile, e.g. topic Ready with ARN),
|
||
`TimeoutLong` (first reconcile that may race rgw startup). Bespoke waits
|
||
(e.g. the shared store teardown ladder) pass explicit durations.
|
||
|
||
## assert vs require
|
||
|
||
- `require` (and `wait4.Require*`) for anything the rest of the (sub)test
|
||
cannot proceed without: creates, fetches of the object about to be
|
||
inspected, decoding, setup waits. Fail fast to avoid cascade noise.
|
||
- `assert` (and `wait4.Assert*`) for the properties under test — so every
|
||
violated property in a subtest is reported together — and for teardown
|
||
deletes, so one stuck finalizer does not strand the rest of cleanup.
|
||
- `wait4.RequireDelete` only when later steps depend on the deletion having
|
||
completed.
|
||
- Never `assert`/`require` inside an `Eventually` poll closure.
|
||
|
||
## Naming
|
||
|
||
- Subtest names are lowercase sentences: imperative for actions
|
||
(`"create CephObjectStoreUser %q"`, `"delete obc %q"`), declarative for
|
||
assertions (`"obc %q has bucketOwner %q set"`, `"no secrets in ns %q"`).
|
||
Resource names are always interpolated with `%q`. The `-v` output of a run
|
||
should read as the full scenario.
|
||
|
||
## Running and filtering
|
||
|
||
CI runs the suite as:
|
||
|
||
```
|
||
go test -v -timeout 2400s -failfast -run CephObjectSuite github.com/rook/rook/tests/integration
|
||
```
|
||
|
||
To select one package's tests through the suite:
|
||
|
||
```
|
||
go test -tags ceph_preview -run 'TestCephObjectSuite/TestWithoutTLS/ObjectStoreUser_keys' ./tests/integration
|
||
```
|
||
|
||
All build/vet/lint commands need the `ceph_preview` build tag or the module
|
||
does not type-check:
|
||
|
||
```
|
||
go build -tags ceph_preview ./tests/...
|
||
go vet -tags ceph_preview ./tests/integration/...
|
||
gofmt -l tests/integration
|
||
```
|
||
|
||
These are cluster-backed integration tests; the only end-to-end validation is
|
||
the CI object suite.
|
||
|
||
## Adding a new package (checklist)
|
||
|
||
1. Create `tests/integration/object/<operator-pkg>/<feature>/` per the layout
|
||
rules above.
|
||
2. Write the entry func following the anatomy section; pick a globally unique
|
||
outer `t.Run` name.
|
||
3. Wire one dispatcher line in `tests/integration/ceph_object_test.go` and,
|
||
if the package creates CephObjectStoreUsers, add its namespace to the
|
||
shared store's `AllowUsersInNamespaces`.
|
||
4. Verify with the build/vet/gofmt commands above.
|
||
|
||
## Conversion playbook (old-style tests)
|
||
|
||
Per old test: enumerate its behaviors; pick or create target package(s);
|
||
express fixtures as var-block constructors; map k8s waits to `wait4` verbs
|
||
and non-k8s polls to `Eventually`; write check* helpers for repeated
|
||
verification; wire the dispatcher; delete the old code; retire
|
||
`tests/framework/clients` methods whose last consumer is gone.
|
||
|
||
| old test | target package(s) | still needs (build in that PR) |
|
||
|---|---|---|
|
||
| `testObjectStoreOperations` S3 I/O + OBC-stays-Bound | `bucket/rw` | per-OBC S3 client (TLS-aware, from OBC secret) |
|
||
| `testObjectStoreOperations` user/bucket quotas | `bucket/quota` | per-OBC S3 client |
|
||
| `testObjectStoreOperations` bucket policy | `bucket/policy` | per-OBC S3 client |
|
||
| `testObjectStoreOperations` bucket lifecycle | `bucket/lifecycle` | per-OBC S3 client, move `lifecycleCmpOpts` out of the dispatcher |
|
||
| `testObjectStoreOperations` deletion-blocked-by-dependents | `tests/integration/object/dependents` | private-store fixture, store condition predicates in `ready` |
|
||
| `createCephObjectStore`/`runObjectE2ETestLite`/deletion asserts + zone.json canary | `tests/integration/object/lifecycle` | store create/health/delete helpers, `Sharedstore.Installer()` accessor |
|
||
| `testBucketNotifications` | `notification/*` | HTTP endpoint fixture, per-OBC S3 client (the pod-log waiter is now `wait4.Assert/RequirePodLog`) |
|
||
| `testCOSIDriver` | `cosi` | `kubectl -k` install fixture, COSI readiness waits |
|
||
| upgrade-suite object usage | stays in upgrade suite | switch to typed clients + `wait4`/`ready` |
|