Three of the checklist items say "if necessary" without saying what would make
them necessary, and a box ticked without a matching change misleads the reviewer
who reads the checklist as a claim about the diff. The deciding criterion in
each case is simply which folder the change touches.
State that folder next to each item, so the test is visible where the box is
ticked and needs no judgement to apply.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
(cherry picked from commit d900c61b84)
Nothing verified that the commit being tagged for a release actually
carries the intended version. Tagging a release branch before the version
update PR is merged publishes images and manifests that reference the
previous release. Add build/release/validate-tag.sh to check that the tree
is clean and that deploy/examples/images.txt and the helm chart values
match the tag, run it from the release build workflow before any artifact
is published, and document it in the tagging steps. Alpha tags are exempt
as they only mark the creation of a release branch.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
(cherry picked from commit adda6b17d3)
# Conflicts:
# .github/workflows/push-build.yaml
Bump the most recent kubernetes version exercised by the integration and
canary matrices from v1.35.5 to v1.36.1. The kindest/node image for 1.36 is
provided by the kind v0.32.0 pinned by the kind-conversion commit. The older
matrix versions are unchanged.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
(cherry picked from commit a280c3ae2d)
The minikube-to-kind conversion moved the master/release integration
tests to kind, which selects the Kubernetes version via the
kindest/node:<version> image tag. There is no published
kindest/node:v1.32.13 image (Docker Hub returns 404); the latest 1.32
node image kind ever built is v1.32.11, since later kind releases
advanced to 1.33+ and never produced a newer 1.32 patch.
Pin the on-release matrix's 1.32 entry to v1.32.11 so the 1.32-line
coverage keeps a node image that actually exists. The other matrix
versions (v1.31.14, v1.34.8, v1.36.1) already have published images.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
(cherry picked from commit 117e1126c2)
The integration and canary suites ran on a single-node minikube
`driver: none` cluster, where the kubelet ran directly on the GitHub
runner, so the host docker daemon doubled as the cluster runtime and
host block devices and host paths were directly visible to pods. Replace
that with a kind cluster.
Every suite creates its cluster through the shared
integration-test-setup-cluster-resources composite action, so the
conversion is centralized there and converts the smoke, object, helm,
keystone, multi-cluster, upgrade, on-release, nightly, encryption-KMS and
all canary jobs at once.
- Replace the setup-minikube step with helm/kind-action, selecting the
kubernetes version via the kindest/node image tag and creating a
single-node cluster from a new kind config (kind pinned to v0.32.0 for
reproducibility).
- Drop the cri-dockerd install; kind nodes use their built-in containerd.
- Add a kind config that bind-mounts the host /dev, /var/lib/rook and
/run/udev into the node so the existing host-based disk-prep helpers
(use_local_disk*, create_partitions_for_osds, blockDevicePV.sh,
localPathPV.sh, ...) keep working unchanged: devices and partitions
created on the host appear in the node and in the OSD pods that
hostPath-mount the node /dev, and ceph-volume can read the host udev
database it needs to inventory disks.
- Prepare the kind node for the host-level operations rook runs against the
underlying host: remount /sys read-write so CSI's kernel RBD mapping
(`rbd map --device-type krbd`, which writes /sys/bus/rbd) works, and install
lvm2 and cryptsetup, which rook runs in the node's mount namespace to
provision LVM- and encryption-backed OSDs. kindest/node images provide none
of this; the minikube driver:none runner host did.
- Route the Service and pod CIDRs from the runner to the kind node so
host-side tests (the `go test` process runs on the runner) can reach
in-cluster ClusterIPs, e.g. an S3 request to the RGW service. With minikube
driver:none the runner already shared the cluster network.
- Load locally built images into the cluster. Under minikube `driver: none`
the built image was already in the cluster runtime; under kind it must be
imported, so build_rook and create_helm_tag now import their images into
each node's containerd through a new load_image_into_cluster helper (via the
node's ctr, which avoids the kind/kindest-node containerd-config version
skew that breaks `kind load docker-image`).
- Point Vault's kubernetes-auth at the in-cluster API endpoint
(kubernetes.default.svc) instead of the kubeconfig server URL: kind exposes
that as https://127.0.0.1:<port>, unreachable from the in-cluster Vault pod,
so OSD encryption-key retrieval via k8s-auth failed.
- Replace the remaining direct minikube references in the canary workflow: a
`minikube kubectl` call and the external-cluster topology values.
- Adapt host-name assumptions that only held under driver:none: resolve the
disk-cleanup job by the k8s node name rather than the runner hostname, and
let kind-action ignore post-job cluster-teardown failures (the runner is
ephemeral; nvme/multus devices can wedge `docker rm` of the node).
- Update stale comments that described the CI environment as minikube.
- Move the multus integration test's kind config under tests/config too, so
both kind cluster configs live in one place.
create-dev-cluster.sh and other local-dev tooling are intentionally left on
minikube.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
(cherry picked from commit d59abb1956)
The validate-rgw-endpoint step gave each validation loop only 15
seconds. A single attempt of the external script can exceed that
budget when the RGW is slow to respond right after the object store
was created, killing the step before any retry could happen. This
showed up as sporadic canary job failures with exit code 124 and no
diagnostics.
Validation failures are also reported on stderr while the script exits
0, so the until loop never actually retried them. Retry until the
script succeeds with no stderr output, give the loops a generous
timeout, and print the captured stderr when validation does not
succeed in time.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
(cherry picked from commit 0c374266e5)
The raw-disk-with-object, osd-with-metadata-device, and pvc canary jobs
used loopback devices to supply an extra block device beyond the single
spare disk a GitHub runner provides. Loop devices behave differently
from real block devices -- Rook gates them behind
ROOK_CEPH_ALLOW_LOOP_DEVICES and special-cases them in the OSD daemon --
so exercising core scenarios on them is less realistic than using real
SCSI disks.
Provision the extra device(s) over iSCSI (LIO) instead, reusing the
existing create_extra_disk helper: it now creates N LUNs, the new
find_extra_block_devs and find_second_block_dev helpers enumerate them,
and deploy_cluster gains a two_raw_disks mode. The three jobs now run
entirely on real disks, and the canary CI no longer creates loop devices
or relies on the loop-device test helpers.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
(cherry picked from commit 4b7d769102)
Canary jobs upload their logs under an artifact named by job and ceph image, but the job matrix also varies kubernetes-version. When more than one kubernetes version is exercised the upload-artifact step fails on a duplicate artifact name. Append the kubernetes version so each matrix cell produces a unique artifact.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
(cherry picked from commit 288693a622)
actionlint only understands workflow files, so the .github/actionlint.yaml
ignore keeps the composite action definitions out of `make lint.workflows`.
Apply the same shellcheck fixes there anyway, found by running shellcheck over
their run: blocks directly:
- quote unquoted variable expansions (SC2086): the $GITHUB_ENV writes in
collect-logs, tmate_debug and upterm_debug, and ${ARCH} in the cri-dockerd
install in integration-test-setup-cluster-resources
- split `export BLOCK=$(...)` into a separate assignment and export so a
failing command substitution is not masked (SC2155) in
encryption-pvc-kms-ibm-kp
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
(cherry picked from commit 48dacb1894)
Resolve the shellcheck findings actionlint reports so that
`make lint.workflows` passes:
- quote unquoted variable expansions (SC2086) and command substitutions
(SC2046)
- split `export VAR=$(...)` into a separate assignment and export so a
failing command substitution is not masked by export's exit status
(SC2155)
- double-quote kubectl jsonpath arguments inside `sh -c '...'` loops so
the single quotes no longer break out of the wrapper (SC1083)
- annotate intentional deferred-expansion `sh -c '...'` bodies and the
intentional `for ... in $sockets` word splitting with targeted
`# shellcheck disable` directives (SC2016, SC2086)
- drop unused command-output captures (SC2034), simplify a `\! -z` test
(SC2236) and fix nested same-type quoting (SC2140)
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
(cherry picked from commit 826f8944d1)
Run actionlint over the workflow files through the `make lint.workflows`
target (added in PR 17795) so the check behaves identically when run
locally and in CI, and narrow the workflow's path filter to the
lint-relevant files.
Add .github/actionlint.yaml to ignore the composite action definitions
under .github/workflows/<name>/action.{yml,yaml}: actionlint only
understands workflow files and otherwise mis-parses composite actions as
malformed workflows.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
(cherry picked from commit 34cc8751cf)
The reusable canary workflow declares a default ceph_images value that is never used: both callers always pass an explicit image list. A stale default is a drift hazard because a new caller could silently exercise the wrong image. Drop the default and mark the input required so callers must choose deliberately.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
(cherry picked from commit f40167be78)
The workflow already sets defaults.run.shell, yet several steps in the rgw-multisite-testing and two-object-one-zone jobs re-declare the identical shell. Remove the per-step overrides; the workflow default already applies to every run step.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
(cherry picked from commit 990947a597)
Several callers pass a github-token input to composite actions that do not declare it, so it is silently ignored and emits a runner warning at run time. The integration-test-setup-cluster-resources action (called from the nightly jobs and the keystone-auth suite) and the encryption-pvc-kms-ibm-kp action (called from the canary suite) declare only their real inputs. Remove the dead github-token from every such call. If a token is ever needed it should be declared and consumed by the action itself.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
(cherry picked from commit 6a67e4fbaf)
It is intended that the CI test the latest patch release of each
kubernetes version that is tested.
The canary integration test tested v1.35.0 however, while
the latest v1.35 patch releaase is v1.35.5.
This change updates the integration-test CI workflow
to test k8s v1.35.5 instead of v1.35.0 to match the intention.
Signed-off-by: Michael Adam <obnox@samba.org>
(cherry picked from commit b6ce3e2633)
The rgw-multisite-testing canary job has been chronically flaky
(roughly a third of recent canary runs, including on master and
release branches). Job logs and collected artifacts show two recurring
mechanisms, neither of which is addressed by bumping timeouts:
1. Every recent failure of the "write an object" step died within
~16s: the very first "s3cmd mb" against the primary zone got
ECONNREFUSED three times across the 10s retry window. The RGW
process never crashed; RGWs pause their HTTP frontends to reload
the realm whenever the RGW configuration period changes ("rgw
realm reloader: Pausing frontends for realm update..."), which
keeps happening shortly after the second zone joins the zonegroup,
exactly when the test starts writing. While paused, the readiness
probe also times out, the endpoint is removed from the Service, and
connections to the ClusterIP are refused.
2. The "kubectl wait --for=condition=available deployment/..."
barrier added previously is a no-op for these deployments: with one
replica and maxUnavailable=1, the availability threshold is zero
ready pods, so the condition is always true.
Changes:
- Wait on RGW pod readiness instead of deployment availability.
- Add a "wait for multisite sync to be established" step that polls
"radosgw-admin sync status" on both clusters until metadata and
data sync report caught up, mirroring the checkpoints ceph's own
multisite QA performs before asserting replication.
- Probe both RGW endpoints over HTTP until several consecutive
requests succeed before writing, to ride out realm-reload pauses.
- Replace the fixed 3-attempt s3cmd retries with time-budgeted
retries.
- Replace the s3cmd read polling loop and its SIGUSR1 timer (which was
never cancelled on success, silently shrinking the second
direction's time budget) with "radosgw-admin bucket sync
checkpoint", the marker-based barrier used by ceph QA, followed by
a single get and diff.
- Dump multisite diagnostics (pod state, RGW logs, sync status,
period) when the job fails.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
(cherry picked from commit bf00821056)
Multus CI failures on master pushes from 2026-04-08 to 2026-06-10 were
dominated by a race in setup-multus.sh: 'kubectl wait' was invoked on a
pod label selector immediately after 'kubectl create' of a daemonset,
and it exits with "error: no matching resources found" when the
daemonset controller has not created any pods yet. This caused 9 of the
10 genuine multus flakes across the standalone multus workflow and the
canary multus-public-and-cluster job (the two share setup-multus.sh).
- setup-multus.sh: wait with 'kubectl rollout status' on the daemonsets
instead of 'kubectl wait' on pod label selectors. The daemonset
object exists as soon as 'kubectl create' returns, and rollout status
correctly waits for all desired pods to be created and become ready.
The old selector wait also silently under-waited: pods created after
its initial LIST were never waited on at all. The stricter wait
revealed that full multi-node convergence can exceed 2 minutes on
busy runners, so the wait timeout is raised to 5 minutes (the waits
return as soon as the rollouts are ready, so this costs nothing on
healthy runs).
- Wait for the host-net-config daemonset rollout in both workflows
before proceeding; it configures the host routing that the multus
public network depends on and was previously not waited on at all.
Pin its jonlabelle/network-tools image.
- test_multus_connections: retry the osd dump / fs dump network checks
for up to 2 minutes. Daemons register their addresses in the mon maps
asynchronously after the cluster reports ready; one canary failure
(2026-04-29) ran the MDS check at fsmap epoch 1, before any MDS had
registered.
- Dump cluster state (pods, daemonsets, events, multus logs) when the
multus workflow job fails; the workflow previously had no failure
diagnostics at all.
Also remove the unused NUMBER_OF_COMPUTE_NODES env var from the multus
workflow (kind-config.yaml creates 3 workers; nothing consumes it).
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
(cherry picked from commit 4da654bae2)
The CephUpgradeSuite workflow failed ~31% of PR runs vs the ~19%
broken-PR baseline of the other integration suites. Classifying the
failing step of every failed run from the last ~400 PR runs and the
logs of every failure on non-dependabot branches shows the excess
comes from a handful of too-tight waits, unretried network fetches,
and environment races rather than from the upgrade logic itself:
- The biggest class (21 of 44 genuine test-phase failure runs):
"giving up waiting for deployment(s) with label
app=rook-ceph-{rgw,osd},ceph-version!=..." during the
Squid->Tentacle upgrade. The operator updates daemons sequentially
(mons, mgr, osds with PG health gates between them, mds, rgw), so
each later daemon's fixed 275s wait also absorbs the time spent on
every daemon before it; rgw, last in the sequence, failed most
often. The mon wait was already extended for slow image pulls in
98bd03a42; extend the same retry count to all the daemon waits.
- ~7 runs: "snapshot controller is not ready" in the Helm upgrade
path. WaitForSnapshotController(30) allows only 150s, and the logs
show the deployment still converging (readyreplicas 1 < replicas 2)
on the final poll. Raise to 90 retries.
- InstallOrUpgradeHelmRepoChart ran helm with no retry; an observed
failure fetched the ceph-csi-drivers chart tarball from GitHub
release assets and got a 504. Retry up to 5 times, as
InstallLocalHelmChart already does.
- Raise the go test timeouts (2400s->3600s rook, 1800s->2700s helm).
Failed runs ended in "panic: test timed out" during teardown, which
aborts cleanup and log collection; the longer daemon waits above
also need the headroom.
- The "setup cluster resources" composite action (shared by all
suites) accounted for half of all failed jobs. The one steady class
there (8 distinct runs across 7 different days): minikube exits with
K8S_FAIL_CONNECT (code 40) when the requested kubernetes version is
missing from its bundled version list, because it then validates the
version with an anonymous GitHub API request (GITHUB_TOKEN is not
honored on that code path), and anonymous requests from shared
runner IPs are regularly rate-limited. minikube 1.38.x predates
v1.35.5, so only the v1.35.5 jobs hit this class. Pass --force to
minikube start to skip that check: the kubernetes versions used in
CI are pinned constants already validated by the PRs that bump
them, and an invalid version would still fail fast at the kubeadm
download. Also seen twice: dpkg failing on a corrupt cri-dockerd .deb
because curl ran without --fail and saved an HTTP error page as the
package; add --fail and retries.
- On runners without the /mnt resource disk, the fallback OSD disk is
an iSCSI (LIO) device and use_local_disk_for_integration_test
returned early on those runners, skipping the udev nowatch
workaround for the device re-probe storms of rook issue 8975. A burn-in
failure on this PR captured the consequence with full kernel
forensics: ~100 udev change events on the OSD disk, and ceph-volume
activate wedged in uninterruptible sleep on the block device lock
(blkdev_llseek -> rwsem_down_write_slowpath) for 18+ minutes during
the ceph version upgrade while the cluster's only OSD stayed down.
Install the nowatch rule before the early return so it applies to
every runner type, and before the disk is first written rather than
after.
- One burn-in round failed before the upgrade even began: the
pre-upgrade PVC create on the Helm path expired WaitUntilPVCIsBound
(RetryLoop, 275s) while the CSI provisioner pods were still
starting. Set RETRY_MAX=110 for this workflow to double the
framework's base wait budget, instead of extending individual waits
one flake at a time.
- minikube also exits with GUEST_START (code 80) when its internal 6
minute node-ready wait expires on slow runners (5 runs in the
dataset, two more observed while burning in this PR, one of them on
another suite). Pass --wait-timeout=15m to minikube start.
Not addressed (small or episodic classes): OSDs never coming up on
initial deploy (3 runs, possibly the same udev/iSCSI wedge), the
filesystem not becoming active (2 runs), and several day-clustered
minikube incidents (exit codes 65/67/90).
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
(cherry picked from commit cff8b7f311)
The CephObjectSuite ran both the TLS and non-TLS object store tests in a
single job via `-run CephObjectSuite`, executing TestWithTLS and
TestWithoutTLS sequentially. Split every invocation of the suite so the two
variants run as independent jobs, using `-run CephObjectSuite/TestWithoutTLS`
and `-run CephObjectSuite/TestWithTLS`, matching the existing CephUpgradeSuite
split pattern.
Each workflow that runs the suite now has two jobs, one per variant, in the
same file:
- integration-test-object-suite.yaml (PR): TestCephObjectSuite and
TestCephObjectSuiteTLS
- integration-tests-on-release.yaml (push): TestCephObjectSuite and
TestCephObjectSuiteTLS
- daily-nightly-jobs.yml (nightly): object-suite-ceph-main and
object-suite-tls-ceph-main
The TestCephObjectSuite job name is preserved so the existing required
status check is unchanged.
No Go changes are required; ObjectSuite already separates the two methods.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
(cherry picked from commit c123513543)
Add AI guidelines based on discussions from the most recent community
meeting.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
(cherry picked from commit ebf4656481)
Not all of he CI workflows that test on kubernetes tested on the latest
patch release so far.
This change updates the CI to always test on the latest patch release
when testing on kubernetes.
More explicitly, this changes thew CI to use:
* v1.35.5 instead of v1.35.4
* v1.33.12 instead of v1.33.11
As a result, the kubernetes versions tested in the CI are now:
* v1.31.14
* v1.32.13
* v1.33.12
Signed-off-by: Michael Adam <obnox@samba.org>
(cherry picked from commit 8906bcec0a)
The rgw-multisite job is failing frequently. Observed
"connectionRefusedERror: connection Refused" on the s3cmd mb call. It
might be happening because CephObjectSTore CR is ready but RGW gateway
is not fully accepting TCP connections.
This PR wraps s3cmd mb and s3cmd put calls in a retry loop that
will attempt each command up to 3 times with a 5 second sleep between
attemps. This might help fix the flaky CI test.
Signed-off-by: Santosh <sapillai@redhat.com>
(cherry picked from commit 2cc51e8eb0)
Use the latest version of Go in the github actions.
Support for both Go 1.25 and 1.26 is still available,
this only affects the version used by the CI.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
(cherry picked from commit 033f6f732e)
Going forward, admin will manage the csi operator
CR's and rook will only manage Ceph Connection cr
and client Profile cr.
The old csi driver is completely removed from Rook
and can no longer be used starting in Rook v1.20.
The upgrade guide will contain the needed transition steps
for managing the csi operator settings.
Signed-off-by: subhamkrai <srai@redhat.com>
(cherry picked from commit 4eefad42e8)
The canary test was waiting for replicapool
instead of replicapool2, and a more reliable
wait for the toolbox pod start is added.
Signed-off-by: subhamkrai <srai@redhat.com>
(cherry picked from commit b50f1a3d60)
Update the canary tests to run against tentacle with each
PR instead of squid. the daily tests will still cover squid.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
the helm-lint CI workflow calls `make helm.lint` which depended
on several tools like helm and ct being available in the $PATH.
`make helm.lint` has recently been made self-contained so that it
does not need those tools installed any more.
This change simplifies the helm-lint workflow so that it does not
install now unneeded tools anymore.
Signed-off-by: Michael Adam <obnox@samba.org>
rgw account changes require -tags=ceph_preview with go build.
When we are building the whole code, it's reads the tag from
Makefile but the multus test was only building few files so this
tag was missing. Adding this -tags=ceph_preview to fix ci error.
Signed-off-by: subhamkrai <srai@redhat.com>
actions/checkout v6 changed how credentials are persisted. Instead of writing GITHUB_TOKEN as an extraheader in .git/config, it now stores a pointer to a credential helper file in $RUNNER_TEMP. The Makefile's `git config --unset http.https://github.com/.extraheader` no longer clears the token, so the cross-repo push to rook.github.io was failing.
I have updated the checkout action to v6.0.2, set the persist-credentials field to false and removed the unset line from releases/Makefile.
Signed-off-by: Prabhala Tara Aasrita <taraasrita@ibm.com>
Joshua Hoblitt
mergify[bot]
Michael Adam
parth-gr
Blaine Gardner
Santosh
Travis Nielsen
subhamkrai
Prabhala Tara Aasrita
subham rai
dependabot[bot]
Artem Muterko
Niels de Vos