my-rook-config/tests/integration/object

Object storage integration tests

This tree holds the new-style integration tests for rook's object storage features. It is the model for converting the remaining old-style object tests (testObjectStoreOperations, store lifecycle); follow the conventions here exactly so the conversion happens once.

Execution model

• One Kubernetes cluster is shared by the whole TestCephObjectSuite run. The testify suite in tests/integration/ceph_object_test.go owns cluster lifecycle and dispatches into the packages here; these packages are plain Go libraries, not standalone go test targets.
• A single shared CephObjectStore fixture (util/sharedstore) is created once per suite pass and torn down after all packages run. It provides the store CR, a go-ceph rgw admin client, and an SNS client, and is created with or without TLS to match the pass. The packages run in both passes: the with/without-TLS split runs them as separate parallel jobs, so the TLS pass adds no serial runtime to a single job.
• Packages run sequentially and each test is an ordered script of subtests. Do not use t.Parallel: the steps depend on prior steps, and the cluster is shared.

Layout

The tree mirrors pkg/operator/ceph/object/*: one top-level directory per operator package, with a feature subdirectory layer when multiple test packages cover subsets of one operator package. The operator ROOT package (the CephObjectStore controller itself) maps to the tree root, tests/integration/object/ itself, not a subdirectory. Leaf package names must not collide with std-lib package names (no io, no http).

test package	operator package	covers
bucket/lifecycle	object/bucket	OBC bucketLifecycle management
bucket/owner	object/bucket	OBC bucketOwner handling
bucket/policy	object/bucket	OBC bucketPolicy management
bucket/quota	object/bucket	OBC maxObjects user quota + bucketMaxObjects/bucketMaxSize bucket quota
bucket/rw	object/bucket	OBC S3 read/write/delete + OBC-stays-Bound
cosi	object/cosi	CephCOSIDriver + COSI bucket provisioning
notification	object/notification	CephBucketNotification HTTP endpoint delivery
topic/kafka	object/topic	CephBucketTopic kafka endpoints
user/caps	object/user	user capabilities
user/keys	object/user	explicit S3 key management
user/opmask	object/user	user op_mask
zonepools	object	zone.json pool fields covered by Rook's shared-pool mapping
reserved: tests/integration/object/lifecycle, tests/integration/object/dependents		future conversions

Shared utilities live under util/:

• wait4 — waits for object-test state: watch-based Assert/Require × Create/Delete/Condition/Absent for k8s resources, Assert/RequireEventually polling for non-k8s state, Assert/RequirePodLog for a matching line in a pod's log stream, the readiness predicates (ObjectStoreUser, BucketTopic, OBCBound, ...), and the shared timeout tiers.
• fixture — create-with-t.Cleanup helpers for pure-cleanup resources (namespaces, StorageClasses).
• obc — ObjectBucketClaim helpers: the provisioner StorageClass constructor, the create/bound and delete/absent lifecycle waiters, and a per-OBC S3 client.
• secrets — verification helpers for the Secret references object CRDs publish in their status, shared by more than one package.
• sharedstore — the shared CephObjectStore fixture.
• client — rgw admin, SNS, and S3 client builders and TLS cert generation.

Anatomy of a package

One exported entry function per package, called from the dispatcher:

func TestObjectStoreUserCaps(t *testing.T, k8sh *utils.K8sHelper, store *sharedstore.Sharedstore) {
	var (
		defaultName = "test-usercaps"
		objectStore = store.ObjectStore()
		adminClient = store.AdminClient()

		ns = &corev1.Namespace{ObjectMeta: metav1.ObjectMeta{Name: defaultName}}

		osu1 = cephv1.CephObjectStoreUser{ /* fixture literal or constructor */ }

		osuClient = k8sh.RookClientset.CephV1().CephObjectStoreUsers(ns.Name)
	)

	t.Run("ObjectStoreUser caps", func(t *testing.T) {
		ctx := t.Context()

		fixture.RequireNamespace(t, k8sh, ns)

		t.Run(fmt.Sprintf("create CephObjectStoreUser %q", osu1.Name), func(t *testing.T) {
			wait4.RequireCreate(ctx, t, osuClient, &osu1, wait4.ObjectStoreUser, wait4.TimeoutLong)
		})

		// ... ordered subtests: act, verify, act, verify ...
	})
}

Rules encoded in that shape:

• The entry signature is (t, k8sh, store) and never changes. New dependencies are added as Sharedstore accessors, not parameters.
• The var block declares everything the test uses: fixture objects (use constructors — obc.StorageClass, package-local helpers like awsKeySecret/objectUserKey — to keep literals one line each), then the typed clients bound to locals (osuClient, obcClient, obClient, ...). Spelled-out client chains do not appear in test bodies.
• Exactly one outer t.Run per package, named "<Subject> <feature>". It must be unique across all object packages: it is the -run filter handle.
• ctx := t.Context() is the first line inside the outer t.Run — contexts are test-scoped, so nothing outlives its test. (Helpers that wrap their own t.Run bind their own ctx := t.Context() from the subtest's t.)
• fixture.Require* calls come next. Fixture teardown runs via t.Cleanup, so it happens even when the test fails partway. Only PURE-CLEANUP teardown belongs there; a deletion that asserts behavior ("deleting the OBC must not delete the user") is an ordered subtest.
• Repeated multi-assert verification goes in a package-local check<Aspect>(t, deps, subject, expected ...) helper that wraps its own t.Run and takes the expected state variadically. Promote a helper to a shared util/ package named for its subject (e.g. util/secrets) only when a second package needs it.

Waiting

• State visible to the Kubernetes API uses the watch-based wait4 verbs: RequireCreate (create + wait ready), AssertDelete/RequireDelete (delete + wait gone), RequireCondition/AssertCondition (wait for a predicate on an existing resource), AssertAbsent/RequireAbsent (wait for cascade deletion you did not issue). Predicates come from wait4 or are inlined when test-specific.
• State NOT visible to the Kubernetes API (rgw admin ops, S3, SNS) uses wait4.AssertEventually/RequireEventually, the assert/require-flavored wrappers over the framework's utils.Eventually poll loop. Poll closures must not call assert or require — return an error on transient failures (it is surfaced in the timeout message) and assert on captured state after the wait (capture-on-success: assign an outer variable inside the closure just before returning nil). The closure receives a context bounded by the wait deadline; thread it into the reads it performs.
• Log-based state (an operator or endpoint logging a line) uses wait4.AssertPodLog/RequirePodLog, which follows the first matching pod's log stream until a line satisfies the predicate or the timeout elapses.
• Timeouts come from the shared tiers: wait4.TimeoutShort (routine status changes), TimeoutMedium (full reconcile, e.g. topic Ready with ARN), TimeoutLong (first reconcile that may race rgw startup). Bespoke waits (e.g. the shared store teardown ladder) pass explicit durations.

assert vs require

• require (and wait4.Require*) for anything the rest of the (sub)test cannot proceed without: creates, fetches of the object about to be inspected, decoding, setup waits. Fail fast to avoid cascade noise.
• assert (and wait4.Assert*) for the properties under test — so every violated property in a subtest is reported together — and for teardown deletes, so one stuck finalizer does not strand the rest of cleanup.
• wait4.RequireDelete only when later steps depend on the deletion having completed.
• Never assert/require inside an Eventually poll closure.

Naming

• Subtest names are lowercase sentences: imperative for actions ("create CephObjectStoreUser %q", "delete obc %q"), declarative for assertions ("obc %q has bucketOwner %q set", "no secrets in ns %q"). Resource names are always interpolated with %q. The -v output of a run should read as the full scenario.

Running and filtering

CI runs the suite as:

go test -v -timeout 2400s -failfast -run CephObjectSuite github.com/rook/rook/tests/integration

To select one package's tests through the suite:

go test -tags ceph_preview -run 'TestCephObjectSuite/TestWithoutTLS/ObjectStoreUser_keys' ./tests/integration

All build/vet/lint commands need the ceph_preview build tag or the module does not type-check:

go build -tags ceph_preview ./tests/...
go vet -tags ceph_preview ./tests/integration/...
gofmt -l tests/integration

These are cluster-backed integration tests; the only end-to-end validation is the CI object suite.

Adding a new package (checklist)

1. Create tests/integration/object/<operator-pkg>/<feature>/ per the layout rules above.
2. Write the entry func following the anatomy section; pick a globally unique outer t.Run name.
3. Wire one dispatcher line in tests/integration/ceph_object_test.go and, if the package creates CephObjectStoreUsers, add its namespace to the shared store's AllowUsersInNamespaces.
4. Verify with the build/vet/gofmt commands above.

Conversion playbook (old-style tests)

Per old test: enumerate its behaviors; pick or create target package(s); express fixtures as var-block constructors; map k8s waits to wait4 verbs and non-k8s polls to Eventually; write check* helpers for repeated verification; wire the dispatcher; delete the old code; retire tests/framework/clients methods whose last consumer is gone.

old test	target package(s)	still needs (build in that PR)
testObjectStoreOperations deletion-blocked-by-dependents	tests/integration/object/dependents	private-store fixture, store condition predicates in wait4
createCephObjectStore/runObjectE2ETestLite/deletion asserts	tests/integration/object/lifecycle	store create/health/delete helpers
upgrade-suite object usage	stays in upgrade suite	switch to typed clients + wait4