Several callers pass a github-token input to composite actions that do not declare it, so it is silently ignored and emits a runner warning at run time. The integration-test-setup-cluster-resources action (called from the nightly jobs and the keystone-auth suite) and the encryption-pvc-kms-ibm-kp action (called from the canary suite) declare only their real inputs. Remove the dead github-token from every such call. If a token is ever needed it should be declared and consumed by the action itself.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
(cherry picked from commit 6a67e4fbaf)
This adds a Makefile target lint.workflows that checks
the github workflow yaml files with the
actionlint tool.
Signed-off-by: Michael Adam <obnox@samba.org>
(cherry picked from commit 6dd6a23286)
It is intended that the CI test the latest patch release of each
kubernetes version that is tested.
The canary integration test tested v1.35.0 however, while
the latest v1.35 patch releaase is v1.35.5.
This change updates the integration-test CI workflow
to test k8s v1.35.5 instead of v1.35.0 to match the intention.
Signed-off-by: Michael Adam <obnox@samba.org>
(cherry picked from commit b6ce3e2633)
Add a single example YAML with NetworkPolicy definitions
for all Ceph operand pods (mon, osd, mgr, mds, exporter,
osd-prepare, crashcollector, tools).
Users can apply these policies to restrict egress/ingress
traffic for Rook-Ceph daemon pods.
Signed-off-by: Oded Viner <oviner@redhat.com>
(cherry picked from commit 9dc70051a8)
The realm system user's access and secret keys are generated by
GeneratePassword() and then wrapped in base64. GeneratePassword()
deliberately excludes '/' from the access key character set, but the
base64.StdEncoding wrap reintroduces it: its alphabet contains '/' and
'+', and the 14-character input always produces trailing '='. The
encoded string is the literal key used in S3 requests.
An access key containing '/' breaks AWS SigV4 credential scope parsing
("<access-key>/<date>/<region>/<service>/aws4_request" is split on
'/'), so "radosgw-admin realm pull" against the realm endpoint fails
permanently with "request failed: (22) Invalid argument" (HTTP 400),
and a CephObjectRealm pulling that realm can never reconcile.
This is the dominant cause of the "deploy second cluster rook"
failures in the rgw-multisite-testing canary job: every sampled failure
had a generated access key containing '/' and looped on EINVAL for the
whole 600s wait window, while runs with slash-free keys pulled the
realm successfully.
Encode both keys with base64.RawURLEncoding instead, whose alphabet
(A-Za-z0-9-_, unpadded) is safe in credential scopes, URLs, and shell
arguments. Only newly created realm secrets are affected; existing
secrets are not modified by the reconciler.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
(cherry picked from commit 573e2f81e7)
The rgw-multisite-testing canary job has been chronically flaky
(roughly a third of recent canary runs, including on master and
release branches). Job logs and collected artifacts show two recurring
mechanisms, neither of which is addressed by bumping timeouts:
1. Every recent failure of the "write an object" step died within
~16s: the very first "s3cmd mb" against the primary zone got
ECONNREFUSED three times across the 10s retry window. The RGW
process never crashed; RGWs pause their HTTP frontends to reload
the realm whenever the RGW configuration period changes ("rgw
realm reloader: Pausing frontends for realm update..."), which
keeps happening shortly after the second zone joins the zonegroup,
exactly when the test starts writing. While paused, the readiness
probe also times out, the endpoint is removed from the Service, and
connections to the ClusterIP are refused.
2. The "kubectl wait --for=condition=available deployment/..."
barrier added previously is a no-op for these deployments: with one
replica and maxUnavailable=1, the availability threshold is zero
ready pods, so the condition is always true.
Changes:
- Wait on RGW pod readiness instead of deployment availability.
- Add a "wait for multisite sync to be established" step that polls
"radosgw-admin sync status" on both clusters until metadata and
data sync report caught up, mirroring the checkpoints ceph's own
multisite QA performs before asserting replication.
- Probe both RGW endpoints over HTTP until several consecutive
requests succeed before writing, to ride out realm-reload pauses.
- Replace the fixed 3-attempt s3cmd retries with time-budgeted
retries.
- Replace the s3cmd read polling loop and its SIGUSR1 timer (which was
never cancelled on success, silently shrinking the second
direction's time budget) with "radosgw-admin bucket sync
checkpoint", the marker-based barrier used by ceph QA, followed by
a single get and diff.
- Dump multisite diagnostics (pod state, RGW logs, sync status,
period) when the job fails.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
(cherry picked from commit bf00821056)
Multus CI failures on master pushes from 2026-04-08 to 2026-06-10 were
dominated by a race in setup-multus.sh: 'kubectl wait' was invoked on a
pod label selector immediately after 'kubectl create' of a daemonset,
and it exits with "error: no matching resources found" when the
daemonset controller has not created any pods yet. This caused 9 of the
10 genuine multus flakes across the standalone multus workflow and the
canary multus-public-and-cluster job (the two share setup-multus.sh).
- setup-multus.sh: wait with 'kubectl rollout status' on the daemonsets
instead of 'kubectl wait' on pod label selectors. The daemonset
object exists as soon as 'kubectl create' returns, and rollout status
correctly waits for all desired pods to be created and become ready.
The old selector wait also silently under-waited: pods created after
its initial LIST were never waited on at all. The stricter wait
revealed that full multi-node convergence can exceed 2 minutes on
busy runners, so the wait timeout is raised to 5 minutes (the waits
return as soon as the rollouts are ready, so this costs nothing on
healthy runs).
- Wait for the host-net-config daemonset rollout in both workflows
before proceeding; it configures the host routing that the multus
public network depends on and was previously not waited on at all.
Pin its jonlabelle/network-tools image.
- test_multus_connections: retry the osd dump / fs dump network checks
for up to 2 minutes. Daemons register their addresses in the mon maps
asynchronously after the cluster reports ready; one canary failure
(2026-04-29) ran the MDS check at fsmap epoch 1, before any MDS had
registered.
- Dump cluster state (pods, daemonsets, events, multus logs) when the
multus workflow job fails; the workflow previously had no failure
diagnostics at all.
Also remove the unused NUMBER_OF_COMPUTE_NODES env var from the multus
workflow (kind-config.yaml creates 3 workers; nothing consumes it).
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
(cherry picked from commit 4da654bae2)
The CephUpgradeSuite workflow failed ~31% of PR runs vs the ~19%
broken-PR baseline of the other integration suites. Classifying the
failing step of every failed run from the last ~400 PR runs and the
logs of every failure on non-dependabot branches shows the excess
comes from a handful of too-tight waits, unretried network fetches,
and environment races rather than from the upgrade logic itself:
- The biggest class (21 of 44 genuine test-phase failure runs):
"giving up waiting for deployment(s) with label
app=rook-ceph-{rgw,osd},ceph-version!=..." during the
Squid->Tentacle upgrade. The operator updates daemons sequentially
(mons, mgr, osds with PG health gates between them, mds, rgw), so
each later daemon's fixed 275s wait also absorbs the time spent on
every daemon before it; rgw, last in the sequence, failed most
often. The mon wait was already extended for slow image pulls in
98bd03a42; extend the same retry count to all the daemon waits.
- ~7 runs: "snapshot controller is not ready" in the Helm upgrade
path. WaitForSnapshotController(30) allows only 150s, and the logs
show the deployment still converging (readyreplicas 1 < replicas 2)
on the final poll. Raise to 90 retries.
- InstallOrUpgradeHelmRepoChart ran helm with no retry; an observed
failure fetched the ceph-csi-drivers chart tarball from GitHub
release assets and got a 504. Retry up to 5 times, as
InstallLocalHelmChart already does.
- Raise the go test timeouts (2400s->3600s rook, 1800s->2700s helm).
Failed runs ended in "panic: test timed out" during teardown, which
aborts cleanup and log collection; the longer daemon waits above
also need the headroom.
- The "setup cluster resources" composite action (shared by all
suites) accounted for half of all failed jobs. The one steady class
there (8 distinct runs across 7 different days): minikube exits with
K8S_FAIL_CONNECT (code 40) when the requested kubernetes version is
missing from its bundled version list, because it then validates the
version with an anonymous GitHub API request (GITHUB_TOKEN is not
honored on that code path), and anonymous requests from shared
runner IPs are regularly rate-limited. minikube 1.38.x predates
v1.35.5, so only the v1.35.5 jobs hit this class. Pass --force to
minikube start to skip that check: the kubernetes versions used in
CI are pinned constants already validated by the PRs that bump
them, and an invalid version would still fail fast at the kubeadm
download. Also seen twice: dpkg failing on a corrupt cri-dockerd .deb
because curl ran without --fail and saved an HTTP error page as the
package; add --fail and retries.
- On runners without the /mnt resource disk, the fallback OSD disk is
an iSCSI (LIO) device and use_local_disk_for_integration_test
returned early on those runners, skipping the udev nowatch
workaround for the device re-probe storms of rook issue 8975. A burn-in
failure on this PR captured the consequence with full kernel
forensics: ~100 udev change events on the OSD disk, and ceph-volume
activate wedged in uninterruptible sleep on the block device lock
(blkdev_llseek -> rwsem_down_write_slowpath) for 18+ minutes during
the ceph version upgrade while the cluster's only OSD stayed down.
Install the nowatch rule before the early return so it applies to
every runner type, and before the disk is first written rather than
after.
- One burn-in round failed before the upgrade even began: the
pre-upgrade PVC create on the Helm path expired WaitUntilPVCIsBound
(RetryLoop, 275s) while the CSI provisioner pods were still
starting. Set RETRY_MAX=110 for this workflow to double the
framework's base wait budget, instead of extending individual waits
one flake at a time.
- minikube also exits with GUEST_START (code 80) when its internal 6
minute node-ready wait expires on slow runners (5 runs in the
dataset, two more observed while burning in this PR, one of them on
another suite). Pass --wait-timeout=15m to minikube start.
Not addressed (small or episodic classes): OSDs never coming up on
initial deploy (3 runs, possibly the same udev/iSCSI wedge), the
filesystem not becoming active (2 runs), and several day-clustered
minikube incidents (exit codes 65/67/90).
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
(cherry picked from commit cff8b7f311)
validate_cluster.sh built its ceph exec command around a toolbox pod
name captured once at script start, without waiting for the toolbox to
be ready. When the toolbox container was still being created, or the
pod had been replaced, every check failed with 'unable to upgrade
connection: container not found' for the entire wait window and the
mon quorum check timed out without ever observing the cluster. This
intermittently failed the 'wait for ceph cluster N to be ready' steps
of canary jobs (observed on master pushes of multi-cluster-mirroring
and the canary job between 2026-03-10 and 2026-04-24, always with the
container-not-found signature filling the whole window).
Wait for the toolbox deployment rollout up front, and exec through
deploy/rook-ceph-tools so each call resolves a currently-ready pod,
matching how the other CI scripts invoke the toolbox.
Also call wait_for_daemon directly instead of 'return $(...)'. The
command substitution ran wait_for_daemon in a subshell and captured
its output, so the 'current status' diagnostics printed on timeout
were never displayed; worse, the captured text became the argument of
'return', which fails with 'numeric argument required' and aborts the
script with a misleading exit code 2. The osd variant's captured
'Return value' debug echo had the same problem and is removed.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
(cherry picked from commit 98f088b54f)
Updated the prerequisite guide to ensure udev is installed on the rook
ceph cluster.
Signed-off-by: Santosh <sapillai@redhat.com>
(cherry picked from commit 28c84a2209)
rgw quota checks run against per-instance cached stats kept current by
local per-op deltas; the durable user stats object only catches up on
rgw_user_quota_bucket_sync_interval (180s), and a peer instance only
reloads it on the rgw_bucket_quota_ttl refresh cycle. With 3 gateways
behind the service, an S3 reconnect mid-test can land on an instance
that is blind to the objects already written, letting over-quota writes
pass until those clocks line up. A single gateway serves all writes
from one accurate cache, so enforcement is immediate and deterministic.
Multi-instance rgw deployment remains covered by the keystone suite,
and this matches the notification, COSI, and sharedstore fixtures that
already run one instance.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
(cherry picked from commit 8cbec942f4)
The package-level VersionAtLeast function had no callers anywhere in the
repo (the K8sHelper.VersionAtLeast method remains in use), and
IsKubectlErrorNotFound's only caller was provisioners.go, which is
deleted in this PR. Verified with deadcode and repo-wide grep.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
(cherry picked from commit c4f8831a4d)
Delete three files from tests/framework that are entirely dead code; no
symbol in any of them is referenced anywhere in the repo, including the
build-tagged integration suites (verified with deadcode and repo-wide
grep):
- clients/read_write.go: abandoned NFS test client (ReadWriteOperation
and its methods)
- installer/provisioners.go: hostpath PV helpers orphaned when the
cassandra and nfs installers were removed
- installer/device.go: IsAdditionalDeviceAvailableOnCluster, unused
since the upgrade test rework
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
(cherry picked from commit dbfdd818db)
Remove dead code from the k8sutil package; none of these symbols had any
callers anywhere in the repo (verified with deadcode, staticcheck U1000,
and repo-wide grep, tests included):
- patcher.go: delete file (Patcher type, NewPatcher, Patch)
- replicaset.go: delete file (DeleteReplicaSet)
- customresource.go: WatchCR (the CustomResource type remains in use)
- deployment.go: WaitForDeploymentImage
- job.go: WaitForJobCompletion
- k8sutil.go: GetK8SVersion
- pod.go: ConfigOverrideMount, ConfigOverrideVolume, GetPodLog
- prometheus.go: DeleteServiceMonitor
- service.go: DeleteService
- volume.go: NodeConfigURI, BinariesMountInfo, YamlToVolumes,
YamlToVolumeMounts, and the BinariesMountPath const (only used by the
removed BinariesMountInfo)
Now-unused imports are dropped from each file. The shared
deleteResourceAndWait helper remains (still used by DeleteDeployment).
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
(cherry picked from commit b29e89ef67)
The OSDPerfStats struct had no references anywhere in the repo (verified
with repo-wide grep and an exported-symbol audit); the code that consumed
this unmarshal target was removed long ago.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
(cherry picked from commit ef9761d4df)
NewSharedStorageAndWorkerNodesValidationTestConfig was a wrapper that
simply returned NewDefaultValidationTestConfig() and had no callers
anywhere in the repo (verified with deadcode and repo-wide grep, tests
included). The sibling constructors are all referenced by the multus
validation CLI and remain.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
(cherry picked from commit 2433565238)
WriteFile and PathToProjectRoot had no callers anywhere in the repo
(verified with deadcode, staticcheck U1000, and repo-wide grep, tests
included). WriteFile has long been superseded by direct os.WriteFile
usage. Drop the now-unused bytes, fmt, and runtime imports; the file's
remaining helpers (WriteFileToLog, CreateTempFile) are still in use.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
(cherry picked from commit 8df364df98)
Remove four dead functions from pkg/daemon/ceph/client; none had any
callers anywhere in the repo (verified with deadcode, staticcheck U1000,
and repo-wide grep, tests included):
- auth.go: AuthGetOrCreate
- command.go: ExecuteRBDCommandWithTimeout
- filesystem.go: MarkFilesystemAsDown
- status.go: IsClusterCleanError (IsClusterClean itself remains in use)
No imports become unused by these removals.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
(cherry picked from commit 278453e10c)
Upgrade golang.org/x/crypto from v0.51.0 to v0.52.0 to pick up
security fixes in the indirect dependency.
Signed-off-by: subhamkrai <srai@redhat.com>
(cherry picked from commit 6e2aaac0bd)
kmod.go was entirely dead: IsBuiltinKernelModule, LoadKernelModule,
CheckKernelModuleParam, and the getKernelVersion helper had no callers
anywhere in the repo (verified with grep, deadcode, and staticcheck).
Remove the file.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
(cherry picked from commit 0b6d81d83a)
daemonset.go was entirely dead: CreateDaemonSet, DeleteDaemonset,
AddRookVersionLabelToDaemonSet, and GetDaemonsets had no callers anywhere
in the repo (verified with grep, deadcode, and staticcheck). Rook does not
manage any DaemonSets via these helpers. Remove the file.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
(cherry picked from commit c8ad81a3cf)
Make the NVMe-oF gateway image optional in the CR spec.
When not specified, the operator fetches the image from
the Ceph mon config store using the key
mgr/cephadm/container_image_nvmeof. Falls back to the
hardcoded default if the config is unavailable.
Signed-off-by: Oded Viner <oviner@redhat.com>
(cherry picked from commit 8b4fd67e17)
updated the external cluster doc, and helm and manifest
install to make use of csi operator as it is the default
offering now
Signed-off-by: parth-gr <partharora1010@gmail.com>
(cherry picked from commit dbc8d2a541)
Joshua Hoblitt
Joshua Hoblitt
Michael Adam
Travis Nielsen
mergify[bot]
s3onghyun
Oded Viner
subhamkrai
Santosh
parth-gr
subham rai