Commit Graph
1577 Commits
Author SHA1 Message Date
Joshua Hoblitt cf54270bf5 Merge pull request #17822 from jhoblitt/ci-minikube-to-kind
ci: convert integration and canary CI from minikube to kind
2026-07-06 12:46:09 -07:00
Joshua Hoblitt d59abb1956 ci: convert integration and canary CI from minikube to kind
The integration and canary suites ran on a single-node minikube
`driver: none` cluster, where the kubelet ran directly on the GitHub
runner, so the host docker daemon doubled as the cluster runtime and
host block devices and host paths were directly visible to pods. Replace
that with a kind cluster.

Every suite creates its cluster through the shared
integration-test-setup-cluster-resources composite action, so the
conversion is centralized there and converts the smoke, object, helm,
keystone, multi-cluster, upgrade, on-release, nightly, encryption-KMS and
all canary jobs at once.

- Replace the setup-minikube step with helm/kind-action, selecting the
  kubernetes version via the kindest/node image tag and creating a
  single-node cluster from a new kind config (kind pinned to v0.32.0 for
  reproducibility).
- Drop the cri-dockerd install; kind nodes use their built-in containerd.
- Add a kind config that bind-mounts the host /dev, /var/lib/rook and
  /run/udev into the node so the existing host-based disk-prep helpers
  (use_local_disk*, create_partitions_for_osds, blockDevicePV.sh,
  localPathPV.sh, ...) keep working unchanged: devices and partitions
  created on the host appear in the node and in the OSD pods that
  hostPath-mount the node /dev, and ceph-volume can read the host udev
  database it needs to inventory disks.
- Prepare the kind node for the host-level operations rook runs against the
  underlying host: remount /sys read-write so CSI's kernel RBD mapping
  (`rbd map --device-type krbd`, which writes /sys/bus/rbd) works, and install
  lvm2 and cryptsetup, which rook runs in the node's mount namespace to
  provision LVM- and encryption-backed OSDs. kindest/node images provide none
  of this; the minikube driver:none runner host did.
- Route the Service and pod CIDRs from the runner to the kind node so
  host-side tests (the `go test` process runs on the runner) can reach
  in-cluster ClusterIPs, e.g. an S3 request to the RGW service. With minikube
  driver:none the runner already shared the cluster network.
- Load locally built images into the cluster. Under minikube `driver: none`
  the built image was already in the cluster runtime; under kind it must be
  imported, so build_rook and create_helm_tag now import their images into
  each node's containerd through a new load_image_into_cluster helper (via the
  node's ctr, which avoids the kind/kindest-node containerd-config version
  skew that breaks `kind load docker-image`).
- Point Vault's kubernetes-auth at the in-cluster API endpoint
  (kubernetes.default.svc) instead of the kubeconfig server URL: kind exposes
  that as https://127.0.0.1:<port>, unreachable from the in-cluster Vault pod,
  so OSD encryption-key retrieval via k8s-auth failed.
- Replace the remaining direct minikube references in the canary workflow: a
  `minikube kubectl` call and the external-cluster topology values.
- Adapt host-name assumptions that only held under driver:none: resolve the
  disk-cleanup job by the k8s node name rather than the runner hostname, and
  let kind-action ignore post-job cluster-teardown failures (the runner is
  ephemeral; nvme/multus devices can wedge `docker rm` of the node).
- Update stale comments that described the CI environment as minikube.
- Move the multus integration test's kind config under tests/config too, so
  both kind cluster configs live in one place.

create-dev-cluster.sh and other local-dev tooling are intentionally left on
minikube.

Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
2026-07-06 08:42:39 -07:00
Joshua Hoblitt 064e37ae8d test: reconcile the object README with the merged util packages
The README still described the util layout from before the package
consolidation, with ready, admin, sns, s3, and tls as standalone packages
and a code sample calling ready.ObjectStoreUser. The readiness predicates
live in wait4 and the client builders in client; update the package list,
the code sample, and the playbook references to match.

Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
2026-07-02 08:41:59 -07:00
Joshua Hoblitt b4d6335368 test: convert the object bucket read/write suite to the shared store
Add tests/integration/object/bucket/rw, the first slice carved from the
legacy testObjectStoreOperations. It provisions an OBC on the shared store,
writes, reads back, and deletes an object over S3, and checks the OBC stays
Bound. It runs against the shared store in both the TLS and non-TLS passes.

Drop the converted put/get and OBC-revert subtests from the legacy test;
the user quota subtest now seeds both of the objects it needs. The
remaining quota, policy, lifecycle, and dependents slices convert in later
PRs.

Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
2026-07-02 08:41:59 -07:00
Joshua Hoblitt 2f82880de7 test: consolidate the OBC test helpers into util/obc
Introduce util/obc as the home for ObjectBucketClaim test helpers: the
provisioner StorageClass constructor (moved out of util/fixture, where it
was a misplaced pure constructor), the create/bound and delete/absent
lifecycle waiters, and a per-OBC S3 client. The lifecycle waiters and S3
client are promoted from the notification suite, which built them inline as
their only consumer.

Retarget the StorageClass callers (topic/kafka, bucket/owner) and rework
the notification suite onto the shared helpers.

Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
2026-07-01 16:26:46 -07:00
Joshua Hoblitt b7725553d0 test: add a generic S3 agent builder to the object test client
Add client.NewS3Agent, which builds an rgw S3 agent for an object store
from a set of credentials and the store's endpoint. The object tests build
S3 clients from several credential sources (the store's dashboard admin
user, an OBC's provisioned secret); this factors out the endpoint and agent
construction they share.

Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
2026-07-01 14:42:39 -07:00
Joshua Hoblitt c4ac54cea0 Merge pull request #17853 from jhoblitt/convert-bucket-notification-test
test: convert the bucket notification suite to the shared store
2026-07-01 09:22:44 -07:00
Joshua Hoblitt c0eb360041 docs: fix typos and grammar in code comments
Fix duplicate words, incorrect articles (a/an), it's/its, and other small
grammar mistakes in Go comments and user-facing messages across pkg/, cmd/,
and tests/.

Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
2026-07-01 08:34:11 -07:00
Joshua Hoblitt 031e449070 test: retire the unused bucket notification framework helpers
Converting the bucket notification test to typed clients left the
framework notification plumbing it was the last consumer of unused.
Remove the NotificationOperation client, TopicClient.CreateHTTPServer,
BucketClient.CreateObcNotification and CheckBucketNotificationSetonRGW
(and the now-dead UpdateObcNotification helpers), and the
GetBucketNotification and GetOBCNotification manifest builders.

Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
2026-07-01 08:30:22 -07:00
Joshua Hoblitt 780f0cceb0 test: convert the bucket notification suite to the shared store
Rework the bucket notification integration test onto the shared
CephObjectStore fixture and the wait4 toolkit, following the object
suite conversion playbook. The old test stood up its own object store
and drove everything through tests/framework/clients and kubectl with
fixed-interval polling.

The new tests/integration/object/notification package is an ordered
t.Run script on typed clients and watch-based waits. It deploys an HTTP
sink, drives a CephBucketTopic (HTTP endpoint), CephBucketNotification,
and a notification-labelled OBC, and verifies both end-to-end delivery
(matched in the sink's logs) and the notification configured on the rgw
bucket (read over the S3 API), covering the label add/remove and
notification-before-topic orderings.

Running against the shared store, it no longer creates a dedicated
object store and now exercises both the TLS and non-TLS suite passes.

Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
2026-07-01 08:30:22 -07:00
Anas Khan 67bf4baab8 object: use passed context for S3 v2 API calls
The AWS SDK v1 to v2 migration in #17468 left the S3Agent wrapper methods
calling the SDK with context.TODO(), so S3 operations could not be cancelled
when the controller context is cancelled. Add a leading context.Context
parameter to CreateBucket, PutObjectInBucket, GetObjectInBucket,
DeleteObjectInBucket, PutBucketPolicy, and GetBucketPolicy, forward it to the
underlying client, and pass clusterInfo.Context from the bucket provisioner.

Resolves #17526

Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>
2026-07-01 13:15:40 +05:30
Joshua Hoblitt e281600bb5 Merge pull request #17855 from jhoblitt/narrow-bare-nolint
core: narrow bare //nolint directives to specific linters
2026-06-30 08:14:46 -07:00
Joshua Hoblitt 5065ea7973 Merge pull request #17829 from jhoblitt/ci-provision-disks-no-extra
ci: provision extra disks when the runner starts with none
2026-06-30 08:13:13 -07:00
Joshua Hoblitt e5c6d75a73 core: narrow bare //nolint directives to specific linters
The //nolint directives at these sites omit the linter name, so each
suppresses every linter on its line rather than the one check it needs.
That hides any unrelated errcheck/gosec/govet finding later introduced
on the same line. Name the specific linter for each:

- staticcheck for the two operator sites: SA4004 (the intentional
  single-iteration loop in the OSD PVC host lookup) and SA1019 (the
  deliberate read of the deprecated S3.Enabled field in the RGW
  API-enable builder).
- errcheck for the rbd-mirror deferred token-file cleanup and the
  test-framework logging helpers (WriteString / writeHeader).

No behavior change.

Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
2026-06-29 16:32:59 -07:00
Joshua Hoblitt 3a4dc64f5b test: convert the COSI driver suite to the object test toolkit
Rework the COSI driver integration test onto the shared-store object test
toolkit: an ordered t.Run script with the (t, k8sh, store) signature, typed
clients, and watch-based waits, replacing kubectl-string manifests and
fixed-interval polling.

Drive the COSI bucket resources (BucketClass, BucketClaim, Bucket) with the
upstream sigs.k8s.io/container-object-storage-interface typed client, exposed
as k8sh.COSIClientset alongside the existing OBC client. Create the
CephCOSIDriver and its privileged user through typed clients, wait for the
driver Deployment with wait4, and verify the provisioned bucket through the
shared store's rgw admin client.

Install the COSI CRDs and central controller from the consolidated upstream
repo pinned to v0.2.2 (the former -api and -controller repos are retired) via
a kubectl -k fixture that is removed with t.Cleanup. The driver cannot trust
a TLS RGW endpoint, so the suite skips itself in the TLS pass rather than
being special-cased in the dispatcher.

Retire the now-unused COSIOperation client and the GetCOSIDriver,
GetBucketClass, and GetBucketClaim manifest helpers.

Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
2026-06-29 15:36:13 -07:00
Joshua Hoblitt 802c4e0e6e Merge pull request #17663 from jhoblitt/maint-integration-sharedstore-improvements
test: improve the shared-store object suite tests
2026-06-29 13:33:28 -07:00
Joshua Hoblitt 8ee5d92022 test: document the object suite test conventions
Add a conventions doc and conversion playbook so the remaining old-style
object tests (store lifecycle, notifications, COSI, bucket rw/quota/policy/
lifecycle) can be converted to this pattern mechanically.

Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
2026-06-29 11:10:52 -07:00
Joshua Hoblitt eb404e6677 test: rework the shared-store user/bucket/topic suites
The object store user (caps/keys/opmask), bucket-owner, and bucket topic
tests already shared a CephObjectStore but polled with count-based
utils.Retry behind a wide entry signature. Move them onto the wait4 toolkit
and a slim (t, k8sh, store) signature: each is an ordered t.Run script using
typed clients and watch-based waits, with per-package check helpers for
repeated verification. Extend the shared store fixture to provision the
realm/zonegroup/zone pools and serve TLS, build its rgw admin and SNS clients
through the consolidated util/client package (replacing the separate
admin/sns/s3 packages), and run every package in both the TLS and non-TLS
passes (now separate parallel jobs).

Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
2026-06-29 11:10:52 -07:00
Joshua Hoblitt b59101d8e8 test: add the object suite test util toolkit
Add the shared helpers the object test packages build on:
- wait4: watch-based Assert/Require waiters (Create/Delete/Condition/Absent)
  that block on Kubernetes watch events rather than fixed-interval polling,
  plus Eventually for non-k8s state (rgw admin/S3/SNS), a pod-log waiter, and
  the readiness predicates used with them.
- secrets: verification of the Secret references CRDs publish in status.
- fixture: create-with-t.Cleanup helpers for pure-cleanup resources, and
  constructors for ObjectBucketClaim test resources.
- client: rgw admin and SNS client builders, S3 credentials/endpoint, and the
  store's TLS cert plus a verification-skipping HTTP client.

Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
2026-06-29 11:10:52 -07:00
Joshua Hoblitt 029b13b000 ci: provision extra disks when the runner starts with none
find_extra_block_devs reads the runner's current extra disks with
`lsblk ... | egrep -v "($boot_dev|loop|nbd)"` to decide how many more to
provision. When the runner has no extra disk, that egrep matches nothing
and exits 1; under the script's `set -eo pipefail` the assignment aborts
the function before it ever calls create_extra_disk, so callers get zero
disks and fail with "expected >= N extra disks, found 0" (or a bare /dev/
path) on those runners.

Guard that first read with `|| true` (as the adjacent device-count line
already is) so an empty result means "no extra disks yet" and the
provisioning path runs. The second read, after create_extra_disk, is left
strict on purpose: if provisioning produced no disk, egrep's non-zero exit
should abort there rather than silently return nothing.

Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
2026-06-26 15:00:37 -07:00
Joshua Hoblitt 49612461a4 docs: fix function comments to match their declaration names
Several godoc comments led with a stale or incorrect identifier, left
over from renames, exported/unexported changes, copy-paste between
sibling declarations, or plain typos. As a result the documented name no
longer matched the function, method, type, or var it describes. Correct
each leading word to the name of the declaration it documents.

Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
2026-06-26 12:52:39 -07:00
Joshua Hoblitt 04734d7150 ci: capture build_rook stderr so the network-retry can see it
build_rook() captures the make output into $o and inspects it for
transient network errors (connection reset, INTERNAL_ERROR, 503, 500)
to decide whether to retry the build. The capture was stdout-only,
but go and make write those errors to stderr, so $o never matched any
retry case. A transient module-proxy failure (e.g. proxy.golang.org
returning INTERNAL_ERROR while downloading modules) therefore fell
through to the "valid failure" branch and failed the job in the build
step, before the integration test even ran.

Redirect stderr into the captured output (2>&1) so the existing retry
logic actually triggers on network failures.

Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
2026-06-25 12:36:33 -07:00
Travis Nielsen 01d49c2763 Merge pull request #17821 from daixihegu/master
core: use slices.Contains to simplify code
2026-06-25 10:40:35 -06:00
Joshua Hoblitt c1594a82ea Merge pull request #17801 from jhoblitt/ci-loop-to-iscsi
ci: replace loop devices with iscsi in canary osd tests
2026-06-24 22:25:16 -07:00
daixihegu 063dcad054 core: use slices.Contains to simplify code
Signed-off-by: daixihegu <daixihegu@163.com>
2026-06-25 10:17:53 +08:00
Joshua Hoblitt d0885370fa Merge pull request #17732 from jhoblitt/ci-smoke-suite-flakes
ci: harden CephSmokeSuite against CSI cold-start and teardown flakes
2026-06-24 08:57:25 -07:00
Joshua Hoblitt 343462981a Merge pull request #17798 from jhoblitt/ci-multisite-sync-converge
ci: converge wedged multisite secondary before asserting sync
2026-06-24 08:56:49 -07:00
Joshua Hoblitt 4b7d769102 ci: replace loop devices with iscsi in canary osd tests
The raw-disk-with-object, osd-with-metadata-device, and pvc canary jobs
used loopback devices to supply an extra block device beyond the single
spare disk a GitHub runner provides. Loop devices behave differently
from real block devices -- Rook gates them behind
ROOK_CEPH_ALLOW_LOOP_DEVICES and special-cases them in the OSD daemon --
so exercising core scenarios on them is less realistic than using real
SCSI disks.

Provision the extra device(s) over iSCSI (LIO) instead, reusing the
existing create_extra_disk helper: it now creates N LUNs, the new
find_extra_block_devs and find_second_block_dev helpers enumerate them,
and deploy_cluster gains a two_raw_disks mode. The three jobs now run
entirely on real disks, and the canary CI no longer creates loop devices
or relies on the loop-device test helpers.

Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
2026-06-24 08:54:54 -07:00
Joshua Hoblitt 335e840079 test: remove orphaned legacy multi-node and kubeadm scripts
These shell scripts under tests/scripts are referenced nowhere in the
repository -- no workflow, Makefile, Go test, documentation, or
.mergify.yml invokes them:

- tests/scripts/multi-node/ (build-rook.sh, rpm-system-prerequisites.sh,
  config.rb): the abandoned Vagrant multi-node VM harness, pinned to the
  long-EOL centos/7 box.
- tests/scripts/kubeadm-install.sh: an old kubeadm/apt installer for a
  pinned Kubernetes version, no longer wired into any test path.

Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
2026-06-23 15:37:11 -07:00
Joshua Hoblitt 6f1e7a7dd5 test: remove unused functions from tests/scripts helpers
Three shell functions defined under tests/scripts are never invoked --
not internally, not from another script, and not dispatched by name from
any CI workflow:

- deploy_csi_hostnetwork_disabled_cluster (github-action-helper.sh)
- wait_for_ceph_csi_configmap_to_be_updated (github-action-helper.sh)
- test_demo_pool (validate_cluster.sh): the daemon validation loop never
  passes a "pool" daemon, so this branch is unreachable.

Removing them orphans no other helpers; each only called functions that
remain in use elsewhere.

Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
2026-06-23 15:36:51 -07:00
Joshua Hoblitt 1c8ec71edb test: remove CephMgrSuite integration test and CI workflow
CephMgrSuite (ceph_mgr_test.go) exercised the Rook orchestrator mgr
module through `ceph orch` commands. Remove the suite together with its
GitHub Actions workflow (integration-test-mgr-suite.yaml), whose sole
purpose was to run this suite, gated behind the run-mgr-suite label and
a nightly cron.

No other code references the deleted suite. Its helpers (logBytesInfo,
the defaultTries constant, and the host/service/serviceStatus types)
were all local to the file, so nothing is left orphaned.

Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
2026-06-23 13:14:17 -07:00
Joshua Hoblitt dec475914e Merge pull request #17744 from jhoblitt/keystone-webhook-eventually
test: add Eventually polling helper for test setup waits
2026-06-23 10:39:46 -07:00
dependabot[bot] 516b46bf34 build(deps): bump the github-dependencies group with 4 updates
Bumps the github-dependencies group with 4 updates: [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2), [github.com/ceph/go-ceph](https://github.com/ceph/go-ceph), [github.com/prometheus-operator/prometheus-operator/pkg/apis/monitoring](https://github.com/prometheus-operator/prometheus-operator) and [github.com/prometheus-operator/prometheus-operator/pkg/client](https://github.com/prometheus-operator/prometheus-operator).

Updates `github.com/aws/aws-sdk-go-v2/service/s3` from 1.103.3 to 1.104.0
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/s3/v1.103.3...service/s3/v1.104.0)

Updates `github.com/ceph/go-ceph` from 0.39.0 to 0.40.0
- [Release notes](https://github.com/ceph/go-ceph/releases)
- [Changelog](https://github.com/ceph/go-ceph/blob/master/docs/release-process.md)
- [Commits](https://github.com/ceph/go-ceph/compare/v0.39.0...v0.40.0)

Updates `github.com/prometheus-operator/prometheus-operator/pkg/apis/monitoring` from 0.91.0 to 0.92.0
- [Release notes](https://github.com/prometheus-operator/prometheus-operator/releases)
- [Changelog](https://github.com/prometheus-operator/prometheus-operator/blob/main/CHANGELOG.md)
- [Commits](https://github.com/prometheus-operator/prometheus-operator/compare/v0.91.0...v0.92.0)

Updates `github.com/prometheus-operator/prometheus-operator/pkg/client` from 0.91.0 to 0.92.0
- [Release notes](https://github.com/prometheus-operator/prometheus-operator/releases)
- [Changelog](https://github.com/prometheus-operator/prometheus-operator/blob/main/CHANGELOG.md)
- [Commits](https://github.com/prometheus-operator/prometheus-operator/compare/v0.91.0...v0.92.0)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/service/s3
  dependency-version: 1.104.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: github-dependencies
- dependency-name: github.com/ceph/go-ceph
  dependency-version: 0.40.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: github-dependencies
- dependency-name: github.com/prometheus-operator/prometheus-operator/pkg/apis/monitoring
  dependency-version: 0.92.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: github-dependencies
- dependency-name: github.com/prometheus-operator/prometheus-operator/pkg/client
  dependency-version: 0.92.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: github-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: subhamkrai <srai@redhat.com>
2026-06-23 20:32:19 +05:30
Joshua Hoblitt e9b59476a6 ci: extend the multisite replication read budget
The bucket sync checkpoint confirms the bucket's data sync markers are
caught up at the RADOS level, but the reading RGW can briefly serve 404
for the freshly synced object until it refreshes its view. The post-
checkpoint read was given only a 60s budget, which is occasionally too
short; give it 180s so a transient RGW serving lag does not fail the
test.

Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
2026-06-22 19:22:24 -07:00
Joshua Hoblitt c85643692e ci: retry snapshot controller and crd manifest fetches
Installing the external-snapshotter CRDs and controller runs several
kubectl commands against raw.githubusercontent.com manifest URLs with
no retry, and getManifestFromURL did not retry either, nor did it check
the HTTP status code, so an error page could be fed to kubectl as a
manifest. Transient fetch failures showed up in TestCephSmokeSuite as
NFS test failures during snapshot CRD installation.

Retry the URL-based kubectl invocations and the manifest download, fail
on non-200 responses, and bump the snapshot controller wait in the file
test from 75s to 450s to match the block test since the
snapshot-controller image pull can take longer in CI.

Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
2026-06-22 16:27:35 -07:00
Joshua Hoblitt 76ec455402 ci: avoid panic in object store deletion assertion
When the deletion condition never appeared within the retry window,
assertObjectStoreDeletion recorded the failure with assert and then
dereferenced the nil condition, panicking and aborting the whole test
binary without teardown or operator log collection. Observed in a
TestCephSmokeSuite run as a SIGSEGV right after
'delete_object_store ... Should not be: 10'.

Use require for the timeout and a nil check before dereferencing the
condition, and extend the retry window from 30s to the standard retry
budget since the operator can take longer to set the deletion condition
when its reconcile queue is busy.

Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
2026-06-22 16:27:35 -07:00
Joshua Hoblitt 12a2c0f905 ci: tolerate transient errors while waiting for block image count
retryBlockImageCountCheck aborted the whole wait on the first rbd ls
failure. The listing runs through the toolbox under a timeout and can
fail transiently right after the pool is created (observed as exit
status 124 while the pool PGs were still being created), turning a
recoverable delay into an immediate test failure.

Keep retrying on listing errors and double the retry budget: when the
first provisioning attempt fails, the external-provisioner sidecar
backs off exponentially and its next attempt can be several minutes
out, which the previous 275s budget could not ride out.

This commit also bumps the snapshot controller wait in the block test
from 75s to 450s; the snapshot-controller image pull can take longer
than 75s in CI.

Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
2026-06-22 16:27:35 -07:00
Joshua Hoblitt 53ab1c5136 ci: wait for csi driver deployments after cluster install
The csi-operator creates the driver deployments asynchronously, so the
ctrlplugin pods may still be pulling images when the ceph daemons are
already up and the integration tests start. If a PVC is created before
the provisioner is ready, the first provisioning attempt fails (the
mounted csi config may also not be visible to the driver yet) and the
external-provisioner sidecar enters exponential backoff, delaying the
volume creation by several minutes and timing out the test wait.

Observed in TestCephSmokeSuite runs as 'timed out waiting for image
count to reach 1' with the rbd ctrlplugin pod cold-starting mid-test
and the first CreateVolume failing with 'failed to fetch monitor list
using clusterID'. In one run the second provisioning attempt landed two
seconds after the test wait expired.

Wait for the ctrlplugin deployments of the enabled drivers to be ready
before declaring the cluster installed.

Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
2026-06-22 16:27:35 -07:00
Joshua Hoblitt 4fb04e638d test: add Eventually polling helper for test setup waits
The keystone auth suite installs cert-manager and trust-manager with
helm --wait and then immediately applies Issuer, Certificate, and
Bundle resources. helm --wait returns when the webhook deployments
report ready, but the webhook service endpoints may not be programmed
on the apiserver's node yet, so the apply is rejected with 'failed
calling webhook ... connect: connection refused' and the suite fails
before any test runs. This is currently the dominant failure mode of
the keystone suite, seen on master and on PRs that cannot have caused
it (dependabot github-actions bumps, mergify backports).

Add utils.Eventually, a timeout-based polling primitive intended as
the shared replacement for hand-rolled retry loops in test code:

- cond is func(ctx) error rather than func() bool, so each failed
  attempt logs its reason via t.Logf and the final timeout error wraps
  the last attempt's error.
- cond runs on the calling goroutine and receives a context carrying
  the overall deadline, so cooperative operations stop at the deadline
  instead of overrunning it. It is never run on another goroutine: a
  hung attempt would keep executing concurrently with its retry, and
  require's FailNow is unsafe off the test goroutine.
- utils.AttemptTimeout decorates a cond with a per-attempt deadline
  for operations that can hang but would succeed if canceled and
  retried. The bound is cooperative; conds that cannot honor a context
  must be bounded at the operation level instead.

Expose it as a K8sHelper.ApplyWithRetry method that retries kubectl
apply until the webhooks accept connections, and use it for the
keystone setup applies instead of failing the suite on the first
attempt.

Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
2026-06-22 16:17:46 -07:00
Joshua Hoblitt 325d314dd7 ci: converge wedged multisite secondary before asserting sync
After the sync-readiness gate was added, rgw-multisite-testing still
fails ~25% of master-push runs, now in the new "wait for multisite
sync to be established" step. Diagnosis from the failure diagnostics:
the secondary zone never initializes metadata sync. For the whole
600s its sync status reports "metadata sync failed to read sync
status: (2) No such file or directory".

The master pushes each new RGW configuration period to the secondary
RGW, but until the realm system user is resolvable locally on the
secondary that push is received unauthenticated and rejected with
HTTP 403. The master then re-pushes the same period unchanged every
30s, so a secondary that starts out unable to resolve the user never
recovers. In passing runs the same push later authenticates as
realm-a-system-user and returns 200, at which point the zone converges;
in failing runs it is 403 for the entire window (zero successful
pushes). The generated keys are clean and identical on both clusters,
so this is a fresh-zone bootstrap race in the gateway, not a key
problem.

Rather than wait out a doomed timeout, nudge the secondary to converge
when its metadata sync has not initialized within a couple of minutes:
pull the latest period directly (outbound auth works, so this does not
depend on the rejected inbound push) and restart the secondary RGW so
it re-runs metadata sync init against the now-ready master. This is
what ceph's own multisite QA does (restart zone gateways after the
period is committed). Healthy setups pass the first check in seconds,
so the nudge only runs when sync is genuinely stuck.

Also guard wait_for_sync_status against matching empty output from a
failed exec, and pass the realm/zonegroup/zone to the diagnostics
sync-status dump so the primary cluster no longer reports the unrelated
default zone.

Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
2026-06-22 12:45:04 -07:00
Travis Nielsen dc50bce4e6 Merge pull request #17559 from OdedViner/network_policy_upstream
security: add example NetworkPolicy CRs for Ceph operand pods
2026-06-17 12:54:24 -06:00
subhamkrai ad26ebdab0 build: update ceph version to latest v20.2.2
updating the ceph version from v20.2.1 to v20.2.2

Signed-off-by: subhamkrai <srai@redhat.com>
2026-06-17 22:00:11 +05:30
Travis Nielsen 393cb7a25a Merge pull request #17711 from jhoblitt/ci-rgw-multisite-deterministic
ci: make rgw-multisite-testing wait for multisite readiness
2026-06-16 15:15:33 -06:00
Travis Nielsen 72dde3db1d Merge pull request #17712 from jhoblitt/ci-multus-setup-flakes
ci: fix multus integration test setup flakes
2026-06-16 15:14:45 -06:00
Travis Nielsen 923302d7e7 Merge pull request #17714 from jhoblitt/ci-upgrade-suite-flakes
ci: reduce CephUpgradeSuite flakes
2026-06-16 15:14:03 -06:00
Travis Nielsen 6dbdd1d517 Merge pull request #17722 from jhoblitt/ci-validate-cluster-toolbox
ci: fix toolbox exec race in validate_cluster.sh
2026-06-16 15:12:17 -06:00
Travis Nielsen 16ac936fc6 Merge pull request #17743 from jhoblitt/ci-remove-enablecsioperator
ci: remove unused EnableCsiOperator test setting
2026-06-16 10:21:45 -06:00
Oded Viner 9dc70051a8 security: add example NetworkPolicy CRs for Ceph operand pods
Add a single example YAML with NetworkPolicy definitions
for all Ceph operand pods (mon, osd, mgr, mds, exporter,
osd-prepare, crashcollector, tools).
Users can apply these policies to restrict egress/ingress
traffic for Rook-Ceph daemon pods.

Signed-off-by: Oded Viner <oviner@redhat.com>
2026-06-16 17:44:34 +03:00
Joshua Hoblitt 95132446f7 Merge pull request #17708 from jhoblitt/single-rgw-instance-only
test: run object suite store with a single rgw instance
2026-06-12 12:05:41 -07:00
Joshua Hoblitt faa2fb684b ci: remove unused EnableCsiOperator test setting
The CSI operator is installed unconditionally by the integration test
installer, so the EnableCsiOperator setting no longer selects between
CSI management modes and is consumed nowhere in the test framework.
Remove the setting and the three suites' usages of it.

Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
2026-06-12 11:32:05 -07:00