Add a conventions doc and conversion playbook so the remaining old-style
object tests (store lifecycle, notifications, COSI, bucket rw/quota/policy/
lifecycle) can be converted to this pattern mechanically.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
The object store user (caps/keys/opmask), bucket-owner, and bucket topic
tests already shared a CephObjectStore but polled with count-based
utils.Retry behind a wide entry signature. Move them onto the wait4 toolkit
and a slim (t, k8sh, store) signature: each is an ordered t.Run script using
typed clients and watch-based waits, with per-package check helpers for
repeated verification. Extend the shared store fixture to provision the
realm/zonegroup/zone pools and serve TLS, build its rgw admin and SNS clients
through the consolidated util/client package (replacing the separate
admin/sns/s3 packages), and run every package in both the TLS and non-TLS
passes (now separate parallel jobs).
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
Add the shared helpers the object test packages build on:
- wait4: watch-based Assert/Require waiters (Create/Delete/Condition/Absent)
that block on Kubernetes watch events rather than fixed-interval polling,
plus Eventually for non-k8s state (rgw admin/S3/SNS), a pod-log waiter, and
the readiness predicates used with them.
- secrets: verification of the Secret references CRDs publish in status.
- fixture: create-with-t.Cleanup helpers for pure-cleanup resources, and
constructors for ObjectBucketClaim test resources.
- client: rgw admin and SNS client builders, S3 credentials/endpoint, and the
store's TLS cert plus a verification-skipping HTTP client.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
Several godoc comments led with a stale or incorrect identifier, left
over from renames, exported/unexported changes, copy-paste between
sibling declarations, or plain typos. As a result the documented name no
longer matched the function, method, type, or var it describes. Correct
each leading word to the name of the declaration it documents.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
build_rook() captures the make output into $o and inspects it for
transient network errors (connection reset, INTERNAL_ERROR, 503, 500)
to decide whether to retry the build. The capture was stdout-only,
but go and make write those errors to stderr, so $o never matched any
retry case. A transient module-proxy failure (e.g. proxy.golang.org
returning INTERNAL_ERROR while downloading modules) therefore fell
through to the "valid failure" branch and failed the job in the build
step, before the integration test even ran.
Redirect stderr into the captured output (2>&1) so the existing retry
logic actually triggers on network failures.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
The raw-disk-with-object, osd-with-metadata-device, and pvc canary jobs
used loopback devices to supply an extra block device beyond the single
spare disk a GitHub runner provides. Loop devices behave differently
from real block devices -- Rook gates them behind
ROOK_CEPH_ALLOW_LOOP_DEVICES and special-cases them in the OSD daemon --
so exercising core scenarios on them is less realistic than using real
SCSI disks.
Provision the extra device(s) over iSCSI (LIO) instead, reusing the
existing create_extra_disk helper: it now creates N LUNs, the new
find_extra_block_devs and find_second_block_dev helpers enumerate them,
and deploy_cluster gains a two_raw_disks mode. The three jobs now run
entirely on real disks, and the canary CI no longer creates loop devices
or relies on the loop-device test helpers.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
These shell scripts under tests/scripts are referenced nowhere in the
repository -- no workflow, Makefile, Go test, documentation, or
.mergify.yml invokes them:
- tests/scripts/multi-node/ (build-rook.sh, rpm-system-prerequisites.sh,
config.rb): the abandoned Vagrant multi-node VM harness, pinned to the
long-EOL centos/7 box.
- tests/scripts/kubeadm-install.sh: an old kubeadm/apt installer for a
pinned Kubernetes version, no longer wired into any test path.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
Three shell functions defined under tests/scripts are never invoked --
not internally, not from another script, and not dispatched by name from
any CI workflow:
- deploy_csi_hostnetwork_disabled_cluster (github-action-helper.sh)
- wait_for_ceph_csi_configmap_to_be_updated (github-action-helper.sh)
- test_demo_pool (validate_cluster.sh): the daemon validation loop never
passes a "pool" daemon, so this branch is unreachable.
Removing them orphans no other helpers; each only called functions that
remain in use elsewhere.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
CephMgrSuite (ceph_mgr_test.go) exercised the Rook orchestrator mgr
module through `ceph orch` commands. Remove the suite together with its
GitHub Actions workflow (integration-test-mgr-suite.yaml), whose sole
purpose was to run this suite, gated behind the run-mgr-suite label and
a nightly cron.
No other code references the deleted suite. Its helpers (logBytesInfo,
the defaultTries constant, and the host/service/serviceStatus types)
were all local to the file, so nothing is left orphaned.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
The bucket sync checkpoint confirms the bucket's data sync markers are
caught up at the RADOS level, but the reading RGW can briefly serve 404
for the freshly synced object until it refreshes its view. The post-
checkpoint read was given only a 60s budget, which is occasionally too
short; give it 180s so a transient RGW serving lag does not fail the
test.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
Installing the external-snapshotter CRDs and controller runs several
kubectl commands against raw.githubusercontent.com manifest URLs with
no retry, and getManifestFromURL did not retry either, nor did it check
the HTTP status code, so an error page could be fed to kubectl as a
manifest. Transient fetch failures showed up in TestCephSmokeSuite as
NFS test failures during snapshot CRD installation.
Retry the URL-based kubectl invocations and the manifest download, fail
on non-200 responses, and bump the snapshot controller wait in the file
test from 75s to 450s to match the block test since the
snapshot-controller image pull can take longer in CI.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
When the deletion condition never appeared within the retry window,
assertObjectStoreDeletion recorded the failure with assert and then
dereferenced the nil condition, panicking and aborting the whole test
binary without teardown or operator log collection. Observed in a
TestCephSmokeSuite run as a SIGSEGV right after
'delete_object_store ... Should not be: 10'.
Use require for the timeout and a nil check before dereferencing the
condition, and extend the retry window from 30s to the standard retry
budget since the operator can take longer to set the deletion condition
when its reconcile queue is busy.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
retryBlockImageCountCheck aborted the whole wait on the first rbd ls
failure. The listing runs through the toolbox under a timeout and can
fail transiently right after the pool is created (observed as exit
status 124 while the pool PGs were still being created), turning a
recoverable delay into an immediate test failure.
Keep retrying on listing errors and double the retry budget: when the
first provisioning attempt fails, the external-provisioner sidecar
backs off exponentially and its next attempt can be several minutes
out, which the previous 275s budget could not ride out.
This commit also bumps the snapshot controller wait in the block test
from 75s to 450s; the snapshot-controller image pull can take longer
than 75s in CI.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
The csi-operator creates the driver deployments asynchronously, so the
ctrlplugin pods may still be pulling images when the ceph daemons are
already up and the integration tests start. If a PVC is created before
the provisioner is ready, the first provisioning attempt fails (the
mounted csi config may also not be visible to the driver yet) and the
external-provisioner sidecar enters exponential backoff, delaying the
volume creation by several minutes and timing out the test wait.
Observed in TestCephSmokeSuite runs as 'timed out waiting for image
count to reach 1' with the rbd ctrlplugin pod cold-starting mid-test
and the first CreateVolume failing with 'failed to fetch monitor list
using clusterID'. In one run the second provisioning attempt landed two
seconds after the test wait expired.
Wait for the ctrlplugin deployments of the enabled drivers to be ready
before declaring the cluster installed.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
The keystone auth suite installs cert-manager and trust-manager with
helm --wait and then immediately applies Issuer, Certificate, and
Bundle resources. helm --wait returns when the webhook deployments
report ready, but the webhook service endpoints may not be programmed
on the apiserver's node yet, so the apply is rejected with 'failed
calling webhook ... connect: connection refused' and the suite fails
before any test runs. This is currently the dominant failure mode of
the keystone suite, seen on master and on PRs that cannot have caused
it (dependabot github-actions bumps, mergify backports).
Add utils.Eventually, a timeout-based polling primitive intended as
the shared replacement for hand-rolled retry loops in test code:
- cond is func(ctx) error rather than func() bool, so each failed
attempt logs its reason via t.Logf and the final timeout error wraps
the last attempt's error.
- cond runs on the calling goroutine and receives a context carrying
the overall deadline, so cooperative operations stop at the deadline
instead of overrunning it. It is never run on another goroutine: a
hung attempt would keep executing concurrently with its retry, and
require's FailNow is unsafe off the test goroutine.
- utils.AttemptTimeout decorates a cond with a per-attempt deadline
for operations that can hang but would succeed if canceled and
retried. The bound is cooperative; conds that cannot honor a context
must be bounded at the operation level instead.
Expose it as a K8sHelper.ApplyWithRetry method that retries kubectl
apply until the webhooks accept connections, and use it for the
keystone setup applies instead of failing the suite on the first
attempt.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
After the sync-readiness gate was added, rgw-multisite-testing still
fails ~25% of master-push runs, now in the new "wait for multisite
sync to be established" step. Diagnosis from the failure diagnostics:
the secondary zone never initializes metadata sync. For the whole
600s its sync status reports "metadata sync failed to read sync
status: (2) No such file or directory".
The master pushes each new RGW configuration period to the secondary
RGW, but until the realm system user is resolvable locally on the
secondary that push is received unauthenticated and rejected with
HTTP 403. The master then re-pushes the same period unchanged every
30s, so a secondary that starts out unable to resolve the user never
recovers. In passing runs the same push later authenticates as
realm-a-system-user and returns 200, at which point the zone converges;
in failing runs it is 403 for the entire window (zero successful
pushes). The generated keys are clean and identical on both clusters,
so this is a fresh-zone bootstrap race in the gateway, not a key
problem.
Rather than wait out a doomed timeout, nudge the secondary to converge
when its metadata sync has not initialized within a couple of minutes:
pull the latest period directly (outbound auth works, so this does not
depend on the rejected inbound push) and restart the secondary RGW so
it re-runs metadata sync init against the now-ready master. This is
what ceph's own multisite QA does (restart zone gateways after the
period is committed). Healthy setups pass the first check in seconds,
so the nudge only runs when sync is genuinely stuck.
Also guard wait_for_sync_status against matching empty output from a
failed exec, and pass the realm/zonegroup/zone to the diagnostics
sync-status dump so the primary cluster no longer reports the unrelated
default zone.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
Add a single example YAML with NetworkPolicy definitions
for all Ceph operand pods (mon, osd, mgr, mds, exporter,
osd-prepare, crashcollector, tools).
Users can apply these policies to restrict egress/ingress
traffic for Rook-Ceph daemon pods.
Signed-off-by: Oded Viner <oviner@redhat.com>
The CSI operator is installed unconditionally by the integration test
installer, so the EnableCsiOperator setting no longer selects between
CSI management modes and is consumed nowhere in the test framework.
Remove the setting and the three suites' usages of it.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
The package-level VersionAtLeast function had no callers anywhere in the
repo (the K8sHelper.VersionAtLeast method remains in use), and
IsKubectlErrorNotFound's only caller was provisioners.go, which is
deleted in this PR. Verified with deadcode and repo-wide grep.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
The CephUpgradeSuite workflow failed ~31% of PR runs vs the ~19%
broken-PR baseline of the other integration suites. Classifying the
failing step of every failed run from the last ~400 PR runs and the
logs of every failure on non-dependabot branches shows the excess
comes from a handful of too-tight waits, unretried network fetches,
and environment races rather than from the upgrade logic itself:
- The biggest class (21 of 44 genuine test-phase failure runs):
"giving up waiting for deployment(s) with label
app=rook-ceph-{rgw,osd},ceph-version!=..." during the
Squid->Tentacle upgrade. The operator updates daemons sequentially
(mons, mgr, osds with PG health gates between them, mds, rgw), so
each later daemon's fixed 275s wait also absorbs the time spent on
every daemon before it; rgw, last in the sequence, failed most
often. The mon wait was already extended for slow image pulls in
98bd03a42; extend the same retry count to all the daemon waits.
- ~7 runs: "snapshot controller is not ready" in the Helm upgrade
path. WaitForSnapshotController(30) allows only 150s, and the logs
show the deployment still converging (readyreplicas 1 < replicas 2)
on the final poll. Raise to 90 retries.
- InstallOrUpgradeHelmRepoChart ran helm with no retry; an observed
failure fetched the ceph-csi-drivers chart tarball from GitHub
release assets and got a 504. Retry up to 5 times, as
InstallLocalHelmChart already does.
- Raise the go test timeouts (2400s->3600s rook, 1800s->2700s helm).
Failed runs ended in "panic: test timed out" during teardown, which
aborts cleanup and log collection; the longer daemon waits above
also need the headroom.
- The "setup cluster resources" composite action (shared by all
suites) accounted for half of all failed jobs. The one steady class
there (8 distinct runs across 7 different days): minikube exits with
K8S_FAIL_CONNECT (code 40) when the requested kubernetes version is
missing from its bundled version list, because it then validates the
version with an anonymous GitHub API request (GITHUB_TOKEN is not
honored on that code path), and anonymous requests from shared
runner IPs are regularly rate-limited. minikube 1.38.x predates
v1.35.5, so only the v1.35.5 jobs hit this class. Pass --force to
minikube start to skip that check: the kubernetes versions used in
CI are pinned constants already validated by the PRs that bump
them, and an invalid version would still fail fast at the kubeadm
download. Also seen twice: dpkg failing on a corrupt cri-dockerd .deb
because curl ran without --fail and saved an HTTP error page as the
package; add --fail and retries.
- On runners without the /mnt resource disk, the fallback OSD disk is
an iSCSI (LIO) device and use_local_disk_for_integration_test
returned early on those runners, skipping the udev nowatch
workaround for the device re-probe storms of rook issue 8975. A burn-in
failure on this PR captured the consequence with full kernel
forensics: ~100 udev change events on the OSD disk, and ceph-volume
activate wedged in uninterruptible sleep on the block device lock
(blkdev_llseek -> rwsem_down_write_slowpath) for 18+ minutes during
the ceph version upgrade while the cluster's only OSD stayed down.
Install the nowatch rule before the early return so it applies to
every runner type, and before the disk is first written rather than
after.
- One burn-in round failed before the upgrade even began: the
pre-upgrade PVC create on the Helm path expired WaitUntilPVCIsBound
(RetryLoop, 275s) while the CSI provisioner pods were still
starting. Set RETRY_MAX=110 for this workflow to double the
framework's base wait budget, instead of extending individual waits
one flake at a time.
- minikube also exits with GUEST_START (code 80) when its internal 6
minute node-ready wait expires on slow runners (5 runs in the
dataset, two more observed while burning in this PR, one of them on
another suite). Pass --wait-timeout=15m to minikube start.
Not addressed (small or episodic classes): OSDs never coming up on
initial deploy (3 runs, possibly the same udev/iSCSI wedge), the
filesystem not becoming active (2 runs), and several day-clustered
minikube incidents (exit codes 65/67/90).
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
Delete three files from tests/framework that are entirely dead code; no
symbol in any of them is referenced anywhere in the repo, including the
build-tagged integration suites (verified with deadcode and repo-wide
grep):
- clients/read_write.go: abandoned NFS test client (ReadWriteOperation
and its methods)
- installer/provisioners.go: hostpath PV helpers orphaned when the
cassandra and nfs installers were removed
- installer/device.go: IsAdditionalDeviceAvailableOnCluster, unused
since the upgrade test rework
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
validate_cluster.sh built its ceph exec command around a toolbox pod
name captured once at script start, without waiting for the toolbox to
be ready. When the toolbox container was still being created, or the
pod had been replaced, every check failed with 'unable to upgrade
connection: container not found' for the entire wait window and the
mon quorum check timed out without ever observing the cluster. This
intermittently failed the 'wait for ceph cluster N to be ready' steps
of canary jobs (observed on master pushes of multi-cluster-mirroring
and the canary job between 2026-03-10 and 2026-04-24, always with the
container-not-found signature filling the whole window).
Wait for the toolbox deployment rollout up front, and exec through
deploy/rook-ceph-tools so each call resolves a currently-ready pod,
matching how the other CI scripts invoke the toolbox.
Also call wait_for_daemon directly instead of 'return $(...)'. The
command substitution ran wait_for_daemon in a subshell and captured
its output, so the 'current status' diagnostics printed on timeout
were never displayed; worse, the captured text became the argument of
'return', which fails with 'numeric argument required' and aborts the
script with a misleading exit code 2. The osd variant's captured
'Return value' debug echo had the same problem and is removed.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
Multus CI failures on master pushes from 2026-04-08 to 2026-06-10 were
dominated by a race in setup-multus.sh: 'kubectl wait' was invoked on a
pod label selector immediately after 'kubectl create' of a daemonset,
and it exits with "error: no matching resources found" when the
daemonset controller has not created any pods yet. This caused 9 of the
10 genuine multus flakes across the standalone multus workflow and the
canary multus-public-and-cluster job (the two share setup-multus.sh).
- setup-multus.sh: wait with 'kubectl rollout status' on the daemonsets
instead of 'kubectl wait' on pod label selectors. The daemonset
object exists as soon as 'kubectl create' returns, and rollout status
correctly waits for all desired pods to be created and become ready.
The old selector wait also silently under-waited: pods created after
its initial LIST were never waited on at all. The stricter wait
revealed that full multi-node convergence can exceed 2 minutes on
busy runners, so the wait timeout is raised to 5 minutes (the waits
return as soon as the rollouts are ready, so this costs nothing on
healthy runs).
- Wait for the host-net-config daemonset rollout in both workflows
before proceeding; it configures the host routing that the multus
public network depends on and was previously not waited on at all.
Pin its jonlabelle/network-tools image.
- test_multus_connections: retry the osd dump / fs dump network checks
for up to 2 minutes. Daemons register their addresses in the mon maps
asynchronously after the cluster reports ready; one canary failure
(2026-04-29) ran the MDS check at fsmap epoch 1, before any MDS had
registered.
- Dump cluster state (pods, daemonsets, events, multus logs) when the
multus workflow job fails; the workflow previously had no failure
diagnostics at all.
Also remove the unused NUMBER_OF_COMPUTE_NODES env var from the multus
workflow (kind-config.yaml creates 3 workers; nothing consumes it).
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
The rgw-multisite-testing canary job has been chronically flaky
(roughly a third of recent canary runs, including on master and
release branches). Job logs and collected artifacts show two recurring
mechanisms, neither of which is addressed by bumping timeouts:
1. Every recent failure of the "write an object" step died within
~16s: the very first "s3cmd mb" against the primary zone got
ECONNREFUSED three times across the 10s retry window. The RGW
process never crashed; RGWs pause their HTTP frontends to reload
the realm whenever the RGW configuration period changes ("rgw
realm reloader: Pausing frontends for realm update..."), which
keeps happening shortly after the second zone joins the zonegroup,
exactly when the test starts writing. While paused, the readiness
probe also times out, the endpoint is removed from the Service, and
connections to the ClusterIP are refused.
2. The "kubectl wait --for=condition=available deployment/..."
barrier added previously is a no-op for these deployments: with one
replica and maxUnavailable=1, the availability threshold is zero
ready pods, so the condition is always true.
Changes:
- Wait on RGW pod readiness instead of deployment availability.
- Add a "wait for multisite sync to be established" step that polls
"radosgw-admin sync status" on both clusters until metadata and
data sync report caught up, mirroring the checkpoints ceph's own
multisite QA performs before asserting replication.
- Probe both RGW endpoints over HTTP until several consecutive
requests succeed before writing, to ride out realm-reload pauses.
- Replace the fixed 3-attempt s3cmd retries with time-budgeted
retries.
- Replace the s3cmd read polling loop and its SIGUSR1 timer (which was
never cancelled on success, silently shrinking the second
direction's time budget) with "radosgw-admin bucket sync
checkpoint", the marker-based barrier used by ceph QA, followed by
a single get and diff.
- Dump multisite diagnostics (pod state, RGW logs, sync status,
period) when the job fails.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
rgw quota checks run against per-instance cached stats kept current by
local per-op deltas; the durable user stats object only catches up on
rgw_user_quota_bucket_sync_interval (180s), and a peer instance only
reloads it on the rgw_bucket_quota_ttl refresh cycle. With 3 gateways
behind the service, an S3 reconnect mid-test can land on an instance
that is blind to the objects already written, letting over-quota writes
pass until those clocks line up. A single gateway serves all writes
from one accurate cache, so enforcement is immediate and deterministic.
Multi-instance rgw deployment remains covered by the keystone suite,
and this matches the notification, COSI, and sharedstore fixtures that
already run one instance.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
having csi-addons enabled by default is causing random
pod restart on non-openshift cluster. Let's disable
it by default.
Signed-off-by: subhamkrai <srai@redhat.com>
Two csi controllers cannot start on minikube, so the CI
was failing in the canary and nvmeof tests when they validated
that the controllers were running. For the tests that validate
the csi controllers, ensure only a single controller is configured.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Remove the AWS SDK v1 (github.com/aws/aws-sdk-go)
dependency entirely. All S3 operations now use AWS SDK v2
exclusively.
- Remove the v1 Client field from S3Agent struct and
rename ClientV2 to Client
- Remove v1 session/client initialization from NewS3Agent
- Update all call sites referencing ClientV2
- Convert integration tests to use v2 API calling
conventions (context parameter) and smithy error handling
- Remove aws-sdk-go v1.55.8 from go.mod
Signed-off-by: Oded Viner <oviner@redhat.com>
The rgw-multisite job is failing frequently. Observed
"connectionRefusedERror: connection Refused" on the s3cmd mb call. It
might be happening because CephObjectSTore CR is ready but RGW gateway
is not fully accepting TCP connections.
This PR wraps s3cmd mb and s3cmd put calls in a retry loop that
will attempt each command up to 3 times with a 5 second sleep between
attemps. This might help fix the flaky CI test.
Signed-off-by: Santosh <sapillai@redhat.com>
this commit update the csi-operator to latest v1.0.0.
we need to manually patch the csi drivers with the new serviceAccount
name that are being creating based on the latest ceph-csi-operator
release v1.0.0 where serviceAccount are being separate from csi-operator
chart.
co-authored-by: Travis Nielsen <tnielsen@redhat.com>
Signed-off-by: subhamkrai <srai@redhat.com>
Joshua Hoblitt
Joshua Hoblitt
Travis Nielsen
daixihegu
dependabot[bot]
subhamkrai
Oded Viner
Santosh