Integration tests in the CI were recently updated to test on
the latest patch release for kkubernettes 1.35, i. e. v1.35.5.
This change updates the mergify config for the 1.20 release branch
to require these checs on k8s v1.35.5 to succeed.
Signed-off-by: Michael Adam <obnox@samba.org>
It is intended that the CI test the latest patch release of each
kubernetes version that is tested.
The canary integration test tested v1.35.0 however, while
the latest v1.35 patch releaase is v1.35.5.
This change updates the integration-test CI workflow
to test k8s v1.35.5 instead of v1.35.0 to match the intention.
Signed-off-by: Michael Adam <obnox@samba.org>
Add a single example YAML with NetworkPolicy definitions
for all Ceph operand pods (mon, osd, mgr, mds, exporter,
osd-prepare, crashcollector, tools).
Users can apply these policies to restrict egress/ingress
traffic for Rook-Ceph daemon pods.
Signed-off-by: Oded Viner <oviner@redhat.com>
Rook supports six versions of kubernetes but we only want to spend CI
resources for testing four of them.
In order to align with this commitment, this change lets rook run
integration test suites on kubernetes v1.34 instead of v1.33 for release
testing.
As a result, the latest patch releases of kubernetes v1.31, v1.32,
v1.34, and v1.35 are tested for rook releases.
In addition, k8s version specification is made more systematic to
use the pattern v1.3X.Y instead of 1.3X.Y.
Signed-off-by: Michael Adam <obnox@samba.org>
The CSI operator is installed unconditionally by the integration test
installer, so the EnableCsiOperator setting no longer selects between
CSI management modes and is consumed nowhere in the test framework.
Remove the setting and the three suites' usages of it.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
AddCephVersionLabelToDaemonSet had no callers anywhere in the repo
(verified with deadcode and repo-wide grep); rook applies ceph version
labels to Deployments, Jobs, and object metas, but never to DaemonSets.
The shared addCephVersionLabel helper remains in use by the live
siblings.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
NFSDriverName was assigned once in the CSI controller reconcile but never
read anywhere in the repo (verified with repo-wide grep and an
exported-symbol audit). Remove the variable, its assignment, and the
nfsDriverSuffix const that only existed to feed it.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
The package-level VersionAtLeast function had no callers anywhere in the
repo (the K8sHelper.VersionAtLeast method remains in use), and
IsKubectlErrorNotFound's only caller was provisioners.go, which is
deleted in this PR. Verified with deadcode and repo-wide grep.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
The OSDPerfStats struct had no references anywhere in the repo (verified
with repo-wide grep and an exported-symbol audit); the code that consumed
this unmarshal target was removed long ago.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
NewSharedStorageAndWorkerNodesValidationTestConfig was a wrapper that
simply returned NewDefaultValidationTestConfig() and had no callers
anywhere in the repo (verified with deadcode and repo-wide grep, tests
included). The sibling constructors are all referenced by the multus
validation CLI and remain.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
WriteFile and PathToProjectRoot had no callers anywhere in the repo
(verified with deadcode, staticcheck U1000, and repo-wide grep, tests
included). WriteFile has long been superseded by direct os.WriteFile
usage. Drop the now-unused bytes, fmt, and runtime imports; the file's
remaining helpers (WriteFileToLog, CreateTempFile) are still in use.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
The CephUpgradeSuite workflow failed ~31% of PR runs vs the ~19%
broken-PR baseline of the other integration suites. Classifying the
failing step of every failed run from the last ~400 PR runs and the
logs of every failure on non-dependabot branches shows the excess
comes from a handful of too-tight waits, unretried network fetches,
and environment races rather than from the upgrade logic itself:
- The biggest class (21 of 44 genuine test-phase failure runs):
"giving up waiting for deployment(s) with label
app=rook-ceph-{rgw,osd},ceph-version!=..." during the
Squid->Tentacle upgrade. The operator updates daemons sequentially
(mons, mgr, osds with PG health gates between them, mds, rgw), so
each later daemon's fixed 275s wait also absorbs the time spent on
every daemon before it; rgw, last in the sequence, failed most
often. The mon wait was already extended for slow image pulls in
98bd03a42; extend the same retry count to all the daemon waits.
- ~7 runs: "snapshot controller is not ready" in the Helm upgrade
path. WaitForSnapshotController(30) allows only 150s, and the logs
show the deployment still converging (readyreplicas 1 < replicas 2)
on the final poll. Raise to 90 retries.
- InstallOrUpgradeHelmRepoChart ran helm with no retry; an observed
failure fetched the ceph-csi-drivers chart tarball from GitHub
release assets and got a 504. Retry up to 5 times, as
InstallLocalHelmChart already does.
- Raise the go test timeouts (2400s->3600s rook, 1800s->2700s helm).
Failed runs ended in "panic: test timed out" during teardown, which
aborts cleanup and log collection; the longer daemon waits above
also need the headroom.
- The "setup cluster resources" composite action (shared by all
suites) accounted for half of all failed jobs. The one steady class
there (8 distinct runs across 7 different days): minikube exits with
K8S_FAIL_CONNECT (code 40) when the requested kubernetes version is
missing from its bundled version list, because it then validates the
version with an anonymous GitHub API request (GITHUB_TOKEN is not
honored on that code path), and anonymous requests from shared
runner IPs are regularly rate-limited. minikube 1.38.x predates
v1.35.5, so only the v1.35.5 jobs hit this class. Pass --force to
minikube start to skip that check: the kubernetes versions used in
CI are pinned constants already validated by the PRs that bump
them, and an invalid version would still fail fast at the kubeadm
download. Also seen twice: dpkg failing on a corrupt cri-dockerd .deb
because curl ran without --fail and saved an HTTP error page as the
package; add --fail and retries.
- On runners without the /mnt resource disk, the fallback OSD disk is
an iSCSI (LIO) device and use_local_disk_for_integration_test
returned early on those runners, skipping the udev nowatch
workaround for the device re-probe storms of rook issue 8975. A burn-in
failure on this PR captured the consequence with full kernel
forensics: ~100 udev change events on the OSD disk, and ceph-volume
activate wedged in uninterruptible sleep on the block device lock
(blkdev_llseek -> rwsem_down_write_slowpath) for 18+ minutes during
the ceph version upgrade while the cluster's only OSD stayed down.
Install the nowatch rule before the early return so it applies to
every runner type, and before the disk is first written rather than
after.
- One burn-in round failed before the upgrade even began: the
pre-upgrade PVC create on the Helm path expired WaitUntilPVCIsBound
(RetryLoop, 275s) while the CSI provisioner pods were still
starting. Set RETRY_MAX=110 for this workflow to double the
framework's base wait budget, instead of extending individual waits
one flake at a time.
- minikube also exits with GUEST_START (code 80) when its internal 6
minute node-ready wait expires on slow runners (5 runs in the
dataset, two more observed while burning in this PR, one of them on
another suite). Pass --wait-timeout=15m to minikube start.
Not addressed (small or episodic classes): OSDs never coming up on
initial deploy (3 runs, possibly the same udev/iSCSI wedge), the
filesystem not becoming active (2 runs), and several day-clustered
minikube incidents (exit codes 65/67/90).
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
The CephObjectSuite was split into separate TLS and non-TLS jobs. The
non-TLS job kept the TestCephObjectSuite name, so its required check is
unchanged, but the new TestCephObjectSuiteTLS job produces new status
checks that the release-1.20 automerge rule must also require, otherwise
a backport PR could automerge with the TLS variant failing or still
running.
Older release branches are untouched as the job split will not be
backported to them.
This must not merge until the workflow split has been backported to
release-1.20; until then, backport PRs there do not produce the
TestCephObjectSuiteTLS checks and automerge would stall waiting for
them.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
Delete three files from tests/framework that are entirely dead code; no
symbol in any of them is referenced anywhere in the repo, including the
build-tagged integration suites (verified with deadcode and repo-wide
grep):
- clients/read_write.go: abandoned NFS test client (ReadWriteOperation
and its methods)
- installer/provisioners.go: hostpath PV helpers orphaned when the
cassandra and nfs installers were removed
- installer/device.go: IsAdditionalDeviceAvailableOnCluster, unused
since the upgrade test rework
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
validate_cluster.sh built its ceph exec command around a toolbox pod
name captured once at script start, without waiting for the toolbox to
be ready. When the toolbox container was still being created, or the
pod had been replaced, every check failed with 'unable to upgrade
connection: container not found' for the entire wait window and the
mon quorum check timed out without ever observing the cluster. This
intermittently failed the 'wait for ceph cluster N to be ready' steps
of canary jobs (observed on master pushes of multi-cluster-mirroring
and the canary job between 2026-03-10 and 2026-04-24, always with the
container-not-found signature filling the whole window).
Wait for the toolbox deployment rollout up front, and exec through
deploy/rook-ceph-tools so each call resolves a currently-ready pod,
matching how the other CI scripts invoke the toolbox.
Also call wait_for_daemon directly instead of 'return $(...)'. The
command substitution ran wait_for_daemon in a subshell and captured
its output, so the 'current status' diagnostics printed on timeout
were never displayed; worse, the captured text became the argument of
'return', which fails with 'numeric argument required' and aborts the
script with a misleading exit code 2. The osd variant's captured
'Return value' debug echo had the same problem and is removed.
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
dependabot[bot]
Travis Nielsen
Michael Adam
Artem Torubarov
s3onghyun
subhamkrai
Santosh Pillai
Oded Viner
subham rai
Parth Arora
Joshua Hoblitt
Joshua Hoblitt
Blaine Gardner
