Multus CI failures on master pushes from 2026-04-08 to 2026-06-10 were
dominated by a race in setup-multus.sh: 'kubectl wait' was invoked on a
pod label selector immediately after 'kubectl create' of a daemonset,
and it exits with "error: no matching resources found" when the
daemonset controller has not created any pods yet. This caused 9 of the
10 genuine multus flakes across the standalone multus workflow and the
canary multus-public-and-cluster job (the two share setup-multus.sh).
- setup-multus.sh: wait with 'kubectl rollout status' on the daemonsets
instead of 'kubectl wait' on pod label selectors. The daemonset
object exists as soon as 'kubectl create' returns, and rollout status
correctly waits for all desired pods to be created and become ready.
The old selector wait also silently under-waited: pods created after
its initial LIST were never waited on at all. The stricter wait
revealed that full multi-node convergence can exceed 2 minutes on
busy runners, so the wait timeout is raised to 5 minutes (the waits
return as soon as the rollouts are ready, so this costs nothing on
healthy runs).
- Wait for the host-net-config daemonset rollout in both workflows
before proceeding; it configures the host routing that the multus
public network depends on and was previously not waited on at all.
Pin its jonlabelle/network-tools image.
- test_multus_connections: retry the osd dump / fs dump network checks
for up to 2 minutes. Daemons register their addresses in the mon maps
asynchronously after the cluster reports ready; one canary failure
(2026-04-29) ran the MDS check at fsmap epoch 1, before any MDS had
registered.
- Dump cluster state (pods, daemonsets, events, multus logs) when the
multus workflow job fails; the workflow previously had no failure
diagnostics at all.
Also remove the unused NUMBER_OF_COMPUTE_NODES env var from the multus
workflow (kind-config.yaml creates 3 workers; nothing consumes it).
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
(cherry picked from commit 4da654bae2)
Use the latest version of Go in the github actions.
Support for both Go 1.25 and 1.26 is still available,
this only affects the version used by the CI.
Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
(cherry picked from commit 033f6f732e)
rgw account changes require -tags=ceph_preview with go build.
When we are building the whole code, it's reads the tag from
Makefile but the multus test was only building few files so this
tag was missing. Adding this -tags=ceph_preview to fix ci error.
Signed-off-by: subhamkrai <srai@redhat.com>
Update go.mod to 1.25 while having both 1.24 and 1.25
version in build.yaml for supporting 1.24 longer.
Also, updating golangci-lint version 2.6.2.
Signed-off-by: subhamkrai <srai@redhat.com>
This enables interactive debugging of ci checks
with the upterm tool as an alternative to the tmate tool which does not
work entirely anymore.
see https://github.com/rook/rook/issues/16131
addition to tmate, upterm debugging is now enabled via the debug-ci label on
PRs
Signed-off-by: Michael Adam <obnox@samba.org>
The openSSF scorecard report warns that the github actions are not pinned by hash.
This PR aims at improving this by pinning the github actions by hash.
Signed-off-by: harshitasao <harshitasao@gmail.com>
Ceph image no longer has `ip` tool installed. Use a different container
image for the daemonset which sets host IPs and routes for multus hosts.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
Fixes: #14466
The OpenSSF scorecard report
https://scorecard.dev/viewer/?uri=github.com/rook/rook
warns about excessive token-permissions in the rook project's github CI
workflows.
This bulk change aims at improving this by
1. adding top-level read permissions where missing for integration tests.
2. adding write permissions as needed for the auto-assign workflow.
3. removing the warned-about and unused create-tag workflow
Signed-off-by: Michael Adam <obnox@samba.org>
Co-authored-by: Blaine Gardner <b.blaine.gardner@gmail.com>
In order to help users check that they have implemented the newly-added
Multus host configuration prerequisites, add a check to the validation
tool to verify connectivity.
Because users who are already running clusters with Multus enabled, add
a flag that allows users to only check for host configuration
prerequisites. This mode will not start the large number of clients that
would normally be started because those clients could disrupt a running
Rook cluster negatively.
Host checking pods require host network access. Many Kubernetes
distributions have pod security features enabled. In order to allow
non-Vanilla distros to run this tool, allow specifying a service account
that pods will run as, which can be configured by the admin to allow
test pods.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
Add the ability to specify node profiles in the multus validation test.
This addresses a few points of early feedback on the validation tool.
Statements below critique the tool's behavior before this patch.
1. The tool assumes all daemons are on public and cluster network, which
means users who have a significantly smaller cluster net (a
design choice) cannot run a single test to determine if Rook is
likely to install correctly.
2. The tool does not have placement options to select only a subset of
Kubernetes nodes to run validation on.
3. Users of multus seem to have a dedicated pool of storage nodes more
often than the average Rook install. This makes sense for security-
and perforance-minded users. The tool cannot run a single test to
verify storage-only and general-workload nodes at one time.
These points are addressed by allowing users to specify configurations
for different "NodeTypes."
Each NodeType config has options for selecting the number of OSDs as
well as the number of other (non-OSD) Ceph daemons. This limits the
unnecessary exhaustion of cluster network addresses from critique 1.
Each NodeType config has its own placement (critique 2).
Users can define as many NodeTypes as needed to test the network for
their planned CephCluster. Specifically, this allows the tool to test
storage-only nodes and generalized-workload nodes at the same time. An
arbitrary number of NodeTypes are allowed to support even more highly
specialized cluster setups, such as multiple tiers of storage nodes
where some storage-only nodes may run more OSDs than others.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
The GH action to set up Golang (actions/setup-go) has cache
functionality. For cache hits, this saves ~2 minutes for each run. In
order for the cache to hit, caching must not be disabled, and the code
must be checked out before setting up Golang.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
The previous client loader routine assumed KUBECONFIG would be set in
CLI environments. Instead, now take this approach:
1. If KUBECONFIG is set, that is the de-facto override that informs the
tool it is being run in a CLI environment.
2. Otherwise, try creating a client from the default kube config file.
3. If that fails, assume the tool is running in a Kubernetes Pod.
This will continue supporting dev/test environments so that building the
rook container is not necessary for development. It also allows support
for highly opinionated environments (like deployed by OpenShift CSVs)
where it's not possible to install the Rook operator without also
deploying a CephCluster.
Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
Joshua Hoblitt
Travis Nielsen
dependabot[bot]
subhamkrai
Blaine Gardner
Michael Adam
Oded Viner
subham rai
Satoru Takeuchi
harshitasao
