The canary that checks every RGW zone.json *_pool field is covered by
Rook's zonePoolNSSuffix map was inline in runObjectE2ETest, running
against the legacy per-pass store before the shared store existed.
Move it into tests/integration/object/zonepools as a standalone
shared-store consumer. It now validates the shared store's zone, which
carries the real shared-pool placements the canary is meant to guard,
and runs first among the shared-store packages so it still sees a fresh
zone.
Add a Sharedstore.Installer() accessor so the packaged canary can run
radosgw-admin inside the cluster while keeping the uniform
(t, k8sh, store) package entry signature.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
Add tests/integration/object/bucket/lifecycle, converting the bucket
lifecycle slice of the legacy testObjectStoreOperations onto the shared
store. An OBC is created with a bucketLifecycle additionalConfig, the
rules are verified on the rgw bucket over the S3 API, then updated and
removed with the bucket polled until it reflects each change. The
lifecycleCmpOpts comparison options move out of the suite dispatcher into
the package, and the Ceph v19.2.3 removal-regression skip now discovers
the CephCluster by listing the store's namespace. It runs against the
shared store in both the TLS and non-TLS passes.
Drop the converted lifecycle block from the legacy test.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
Generalize the object suite's sharedstore.Create to accept the target
namespace, store name, and RGW instance count instead of the hardcoded
object-ns / sharedstore / single instance, then use it to stand up the
SmokeSuite object store.
TestObjectStorage_SmokeTest previously created its store with
runObjectE2ETestLite, which applies a CephObjectStore manifest and polls
for health on fixed intervals. It now builds the store through the
typed-client, watch-based sharedstore fixture and tears it down with
Destroy. lite-store is the only object store in the smoke namespace, so
the fixture safely owns the cluster-global .rgw.root realm pool that
Destroy deletes.
The object suite's own call is updated to pass its namespace,
"sharedstore", and one instance, preserving its existing behavior.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
Configure the CephHelmSuite object store as an RGW multisite zone so the
chart's CephObjectRealm, CephObjectZoneGroup, and CephObjectZone templates
are exercised on a Helm install. The operator must reconcile the realm,
zone group, zone, and store to Ready, which the existing install check
already gates on via the RGW pod count.
The realm/zone group/zone and shared-pool layout mirror the fixture in
tests/integration/object/util/sharedstore: a zone backed by explicit
CephBlockPools referenced through sharedPools.poolPlacements. The rgw pools
are appended to the block pools already configured for CSI, and are torn
down along with the multisite CRs when the default storage CRs are removed.
Gate the multisite configuration behind a new UseMultisiteObjectStore
setting enabled only for CephHelmSuite. The other Helm-based suites keep the
plain object store; the upgrade suite in particular installs an older base
chart that lacks the multisite templates and cannot resolve a zone-based
store.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
Add tests/integration/object/bucket/policy, converting the bucket policy
slice of the legacy testObjectStoreOperations onto the shared store: an OBC
created with a bucketPolicy additionalConfig, the policy verified verbatim
on the rgw bucket over the S3 API, then updated and removed with the bucket
polled until it reflects each change. The policy JSON is generated from the
bucket name, which the legacy test hardcoded in the Resource ARN. It runs
against the shared store in both the TLS and non-TLS passes.
Drop the converted policy block from the legacy test.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
Converting the bucket quota tests to typed clients left the framework
helpers and package globals they were the last consumers of unused. Remove
UpdateObc, CheckOBMaxObject, GetAccessKey, and GetSecretKey from the bucket
client, GetEndPointUrl from the object client, and the orphaned object-test
globals.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
Add tests/integration/object/bucket/quota, covering both quota behaviors
from the legacy testObjectStoreOperations on OBCs of its own: user quota
via the OBC maxObjects additionalConfig (enforced, then raised and
re-enforced, synced on the ObjectBucket reflecting the new limit) and
bucket quota via bucketMaxObjects switched to bucketMaxSize. It runs
against the shared store in both the TLS and non-TLS passes.
Drop the converted S3-access and bucket-quota blocks from the legacy test;
the main OBC now exists only as a dependents fixture for the store
deletion checks.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
Add obc.Update, a get/mutate/update helper for live ObjectBucketClaims,
generalized from the notification suite's label updater, which becomes a
thin wrapper over it. The bucket quota conversion needs the same shape for
additionalConfig updates.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
The README still described the util layout from before the package
consolidation, with ready, admin, sns, s3, and tls as standalone packages
and a code sample calling ready.ObjectStoreUser. The readiness predicates
live in wait4 and the client builders in client; update the package list,
the code sample, and the playbook references to match.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
Add tests/integration/object/bucket/rw, the first slice carved from the
legacy testObjectStoreOperations. It provisions an OBC on the shared store,
writes, reads back, and deletes an object over S3, and checks the OBC stays
Bound. It runs against the shared store in both the TLS and non-TLS passes.
Drop the converted put/get and OBC-revert subtests from the legacy test;
the user quota subtest now seeds both of the objects it needs. The
remaining quota, policy, lifecycle, and dependents slices convert in later
PRs.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
Introduce util/obc as the home for ObjectBucketClaim test helpers: the
provisioner StorageClass constructor (moved out of util/fixture, where it
was a misplaced pure constructor), the create/bound and delete/absent
lifecycle waiters, and a per-OBC S3 client. The lifecycle waiters and S3
client are promoted from the notification suite, which built them inline as
their only consumer.
Retarget the StorageClass callers (topic/kafka, bucket/owner) and rework
the notification suite onto the shared helpers.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
Add client.NewS3Agent, which builds an rgw S3 agent for an object store
from a set of credentials and the store's endpoint. The object tests build
S3 clients from several credential sources (the store's dashboard admin
user, an OBC's provisioned secret); this factors out the endpoint and agent
construction they share.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
Fix duplicate words, incorrect articles (a/an), it's/its, and other small
grammar mistakes in Go comments and user-facing messages across pkg/, cmd/,
and tests/.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
Rework the bucket notification integration test onto the shared
CephObjectStore fixture and the wait4 toolkit, following the object
suite conversion playbook. The old test stood up its own object store
and drove everything through tests/framework/clients and kubectl with
fixed-interval polling.
The new tests/integration/object/notification package is an ordered
t.Run script on typed clients and watch-based waits. It deploys an HTTP
sink, drives a CephBucketTopic (HTTP endpoint), CephBucketNotification,
and a notification-labelled OBC, and verifies both end-to-end delivery
(matched in the sink's logs) and the notification configured on the rgw
bucket (read over the S3 API), covering the label add/remove and
notification-before-topic orderings.
Running against the shared store, it no longer creates a dedicated
object store and now exercises both the TLS and non-TLS suite passes.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
The AWS SDK v1 to v2 migration in #17468 left the S3Agent wrapper methods
calling the SDK with context.TODO(), so S3 operations could not be cancelled
when the controller context is cancelled. Add a leading context.Context
parameter to CreateBucket, PutObjectInBucket, GetObjectInBucket,
DeleteObjectInBucket, PutBucketPolicy, and GetBucketPolicy, forward it to the
underlying client, and pass clusterInfo.Context from the bucket provisioner.
Resolves#17526
Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>
Rework the COSI driver integration test onto the shared-store object test
toolkit: an ordered t.Run script with the (t, k8sh, store) signature, typed
clients, and watch-based waits, replacing kubectl-string manifests and
fixed-interval polling.
Drive the COSI bucket resources (BucketClass, BucketClaim, Bucket) with the
upstream sigs.k8s.io/container-object-storage-interface typed client, exposed
as k8sh.COSIClientset alongside the existing OBC client. Create the
CephCOSIDriver and its privileged user through typed clients, wait for the
driver Deployment with wait4, and verify the provisioned bucket through the
shared store's rgw admin client.
Install the COSI CRDs and central controller from the consolidated upstream
repo pinned to v0.2.2 (the former -api and -controller repos are retired) via
a kubectl -k fixture that is removed with t.Cleanup. The driver cannot trust
a TLS RGW endpoint, so the suite skips itself in the TLS pass rather than
being special-cased in the dispatcher.
Retire the now-unused COSIOperation client and the GetCOSIDriver,
GetBucketClass, and GetBucketClaim manifest helpers.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
Add a conventions doc and conversion playbook so the remaining old-style
object tests (store lifecycle, notifications, COSI, bucket rw/quota/policy/
lifecycle) can be converted to this pattern mechanically.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
The object store user (caps/keys/opmask), bucket-owner, and bucket topic
tests already shared a CephObjectStore but polled with count-based
utils.Retry behind a wide entry signature. Move them onto the wait4 toolkit
and a slim (t, k8sh, store) signature: each is an ordered t.Run script using
typed clients and watch-based waits, with per-package check helpers for
repeated verification. Extend the shared store fixture to provision the
realm/zonegroup/zone pools and serve TLS, build its rgw admin and SNS clients
through the consolidated util/client package (replacing the separate
admin/sns/s3 packages), and run every package in both the TLS and non-TLS
passes (now separate parallel jobs).
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
Add the shared helpers the object test packages build on:
- wait4: watch-based Assert/Require waiters (Create/Delete/Condition/Absent)
that block on Kubernetes watch events rather than fixed-interval polling,
plus Eventually for non-k8s state (rgw admin/S3/SNS), a pod-log waiter, and
the readiness predicates used with them.
- secrets: verification of the Secret references CRDs publish in status.
- fixture: create-with-t.Cleanup helpers for pure-cleanup resources, and
constructors for ObjectBucketClaim test resources.
- client: rgw admin and SNS client builders, S3 credentials/endpoint, and the
store's TLS cert plus a verification-skipping HTTP client.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
Several godoc comments led with a stale or incorrect identifier, left
over from renames, exported/unexported changes, copy-paste between
sibling declarations, or plain typos. As a result the documented name no
longer matched the function, method, type, or var it describes. Correct
each leading word to the name of the declaration it documents.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
CephMgrSuite (ceph_mgr_test.go) exercised the Rook orchestrator mgr
module through `ceph orch` commands. Remove the suite together with its
GitHub Actions workflow (integration-test-mgr-suite.yaml), whose sole
purpose was to run this suite, gated behind the run-mgr-suite label and
a nightly cron.
No other code references the deleted suite. Its helpers (logBytesInfo,
the defaultTries constant, and the host/service/serviceStatus types)
were all local to the file, so nothing is left orphaned.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
Installing the external-snapshotter CRDs and controller runs several
kubectl commands against raw.githubusercontent.com manifest URLs with
no retry, and getManifestFromURL did not retry either, nor did it check
the HTTP status code, so an error page could be fed to kubectl as a
manifest. Transient fetch failures showed up in TestCephSmokeSuite as
NFS test failures during snapshot CRD installation.
Retry the URL-based kubectl invocations and the manifest download, fail
on non-200 responses, and bump the snapshot controller wait in the file
test from 75s to 450s to match the block test since the
snapshot-controller image pull can take longer in CI.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
When the deletion condition never appeared within the retry window,
assertObjectStoreDeletion recorded the failure with assert and then
dereferenced the nil condition, panicking and aborting the whole test
binary without teardown or operator log collection. Observed in a
TestCephSmokeSuite run as a SIGSEGV right after
'delete_object_store ... Should not be: 10'.
Use require for the timeout and a nil check before dereferencing the
condition, and extend the retry window from 30s to the standard retry
budget since the operator can take longer to set the deletion condition
when its reconcile queue is busy.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
retryBlockImageCountCheck aborted the whole wait on the first rbd ls
failure. The listing runs through the toolbox under a timeout and can
fail transiently right after the pool is created (observed as exit
status 124 while the pool PGs were still being created), turning a
recoverable delay into an immediate test failure.
Keep retrying on listing errors and double the retry budget: when the
first provisioning attempt fails, the external-provisioner sidecar
backs off exponentially and its next attempt can be several minutes
out, which the previous 275s budget could not ride out.
This commit also bumps the snapshot controller wait in the block test
from 75s to 450s; the snapshot-controller image pull can take longer
than 75s in CI.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
The keystone auth suite installs cert-manager and trust-manager with
helm --wait and then immediately applies Issuer, Certificate, and
Bundle resources. helm --wait returns when the webhook deployments
report ready, but the webhook service endpoints may not be programmed
on the apiserver's node yet, so the apply is rejected with 'failed
calling webhook ... connect: connection refused' and the suite fails
before any test runs. This is currently the dominant failure mode of
the keystone suite, seen on master and on PRs that cannot have caused
it (dependabot github-actions bumps, mergify backports).
Add utils.Eventually, a timeout-based polling primitive intended as
the shared replacement for hand-rolled retry loops in test code:
- cond is func(ctx) error rather than func() bool, so each failed
attempt logs its reason via t.Logf and the final timeout error wraps
the last attempt's error.
- cond runs on the calling goroutine and receives a context carrying
the overall deadline, so cooperative operations stop at the deadline
instead of overrunning it. It is never run on another goroutine: a
hung attempt would keep executing concurrently with its retry, and
require's FailNow is unsafe off the test goroutine.
- utils.AttemptTimeout decorates a cond with a per-attempt deadline
for operations that can hang but would succeed if canceled and
retried. The bound is cooperative; conds that cannot honor a context
must be bounded at the operation level instead.
Expose it as a K8sHelper.ApplyWithRetry method that retries kubectl
apply until the webhooks accept connections, and use it for the
keystone setup applies instead of failing the suite on the first
attempt.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
The CSI operator is installed unconditionally by the integration test
installer, so the EnableCsiOperator setting no longer selects between
CSI management modes and is consumed nowhere in the test framework.
Remove the setting and the three suites' usages of it.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
The CephUpgradeSuite workflow failed ~31% of PR runs vs the ~19%
broken-PR baseline of the other integration suites. Classifying the
failing step of every failed run from the last ~400 PR runs and the
logs of every failure on non-dependabot branches shows the excess
comes from a handful of too-tight waits, unretried network fetches,
and environment races rather than from the upgrade logic itself:
- The biggest class (21 of 44 genuine test-phase failure runs):
"giving up waiting for deployment(s) with label
app=rook-ceph-{rgw,osd},ceph-version!=..." during the
Squid->Tentacle upgrade. The operator updates daemons sequentially
(mons, mgr, osds with PG health gates between them, mds, rgw), so
each later daemon's fixed 275s wait also absorbs the time spent on
every daemon before it; rgw, last in the sequence, failed most
often. The mon wait was already extended for slow image pulls in
98bd03a42; extend the same retry count to all the daemon waits.
- ~7 runs: "snapshot controller is not ready" in the Helm upgrade
path. WaitForSnapshotController(30) allows only 150s, and the logs
show the deployment still converging (readyreplicas 1 < replicas 2)
on the final poll. Raise to 90 retries.
- InstallOrUpgradeHelmRepoChart ran helm with no retry; an observed
failure fetched the ceph-csi-drivers chart tarball from GitHub
release assets and got a 504. Retry up to 5 times, as
InstallLocalHelmChart already does.
- Raise the go test timeouts (2400s->3600s rook, 1800s->2700s helm).
Failed runs ended in "panic: test timed out" during teardown, which
aborts cleanup and log collection; the longer daemon waits above
also need the headroom.
- The "setup cluster resources" composite action (shared by all
suites) accounted for half of all failed jobs. The one steady class
there (8 distinct runs across 7 different days): minikube exits with
K8S_FAIL_CONNECT (code 40) when the requested kubernetes version is
missing from its bundled version list, because it then validates the
version with an anonymous GitHub API request (GITHUB_TOKEN is not
honored on that code path), and anonymous requests from shared
runner IPs are regularly rate-limited. minikube 1.38.x predates
v1.35.5, so only the v1.35.5 jobs hit this class. Pass --force to
minikube start to skip that check: the kubernetes versions used in
CI are pinned constants already validated by the PRs that bump
them, and an invalid version would still fail fast at the kubeadm
download. Also seen twice: dpkg failing on a corrupt cri-dockerd .deb
because curl ran without --fail and saved an HTTP error page as the
package; add --fail and retries.
- On runners without the /mnt resource disk, the fallback OSD disk is
an iSCSI (LIO) device and use_local_disk_for_integration_test
returned early on those runners, skipping the udev nowatch
workaround for the device re-probe storms of rook issue 8975. A burn-in
failure on this PR captured the consequence with full kernel
forensics: ~100 udev change events on the OSD disk, and ceph-volume
activate wedged in uninterruptible sleep on the block device lock
(blkdev_llseek -> rwsem_down_write_slowpath) for 18+ minutes during
the ceph version upgrade while the cluster's only OSD stayed down.
Install the nowatch rule before the early return so it applies to
every runner type, and before the disk is first written rather than
after.
- One burn-in round failed before the upgrade even began: the
pre-upgrade PVC create on the Helm path expired WaitUntilPVCIsBound
(RetryLoop, 275s) while the CSI provisioner pods were still
starting. Set RETRY_MAX=110 for this workflow to double the
framework's base wait budget, instead of extending individual waits
one flake at a time.
- minikube also exits with GUEST_START (code 80) when its internal 6
minute node-ready wait expires on slow runners (5 runs in the
dataset, two more observed while burning in this PR, one of them on
another suite). Pass --wait-timeout=15m to minikube start.
Not addressed (small or episodic classes): OSDs never coming up on
initial deploy (3 runs, possibly the same udev/iSCSI wedge), the
filesystem not becoming active (2 runs), and several day-clustered
minikube incidents (exit codes 65/67/90).
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
rgw quota checks run against per-instance cached stats kept current by
local per-op deltas; the durable user stats object only catches up on
rgw_user_quota_bucket_sync_interval (180s), and a peer instance only
reloads it on the rgw_bucket_quota_ttl refresh cycle. With 3 gateways
behind the service, an S3 reconnect mid-test can land on an instance
that is blind to the objects already written, letting over-quota writes
pass until those clocks line up. A single gateway serves all writes
from one accurate cache, so enforcement is immediate and deterministic.
Multi-instance rgw deployment remains covered by the keystone suite,
and this matches the notification, COSI, and sharedstore fixtures that
already run one instance.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
Remove the AWS SDK v1 (github.com/aws/aws-sdk-go)
dependency entirely. All S3 operations now use AWS SDK v2
exclusively.
- Remove the v1 Client field from S3Agent struct and
rename ClientV2 to Client
- Remove v1 session/client initialization from NewS3Agent
- Update all call sites referencing ClientV2
- Convert integration tests to use v2 API calling
conventions (context parameter) and smithy error handling
- Remove aws-sdk-go v1.55.8 from go.mod
Signed-off-by: Oded Viner <oviner@redhat.com>
Going forward, admin will manage the csi operator
CR's and rook will only manage Ceph Connection cr
and client Profile cr.
The old csi driver is completely removed from Rook
and can no longer be used starting in Rook v1.20.
The upgrade guide will contain the needed transition steps
for managing the csi operator settings.
Signed-off-by: subhamkrai <srai@redhat.com>
The upgrade integration tests for the v1.20 release need to
validate the upgrade starting from v1.19 to the latest.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
migrate bucket provisioner from aws sdk v1 to v2.
replace awserr.Error type assertions with smithy.APIError
using errors.As for v2-style error handling.
switch setBucketPolicy and setBucketLifecycle to use
the v2 s3 client (ClientV2) directly with context.
replace v1 s3 types (BucketLifecycleConfiguration,
GetBucketLifecycleConfigurationOutput) with v2 equivalents
from s3types and s3v2 packages.
add cmpopts.IgnoreUnexported to cmp.Diff calls to handle
unexported noSmithyDocumentSerde fields in v2 sdk types.
add nil guard on lifecycle output before accessing rules
to prevent nil dereference when get returns an error.
depends on #17414
Signed-off-by: Oded Viner <oviner@redhat.com>
This factors out the creation of a single CephObjectStore instance to be
shared between the individually namespaced object suite packages.
This reduces the object suite runtime by the cost of starting up and
tearing down an object store three times.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
since node fencing is disabled in rook for sometime
and this feature is implemented in csi. So, this
commits remove obsolete code related to node loss
Signed-off-by: subhamkrai <srai@redhat.com>
Joshua Hoblitt
Travis Nielsen
Joshua Hoblitt
Anas Khan
Oded Viner
Madhu Rajanna
subhamkrai
Artem Torubarov