Multus CI failures on master pushes from 2026-04-08 to 2026-06-10 were
dominated by a race in setup-multus.sh: 'kubectl wait' was invoked on a
pod label selector immediately after 'kubectl create' of a daemonset,
and it exits with "error: no matching resources found" when the
daemonset controller has not created any pods yet. This caused 9 of the
10 genuine multus flakes across the standalone multus workflow and the
canary multus-public-and-cluster job (the two share setup-multus.sh).
- setup-multus.sh: wait with 'kubectl rollout status' on the daemonsets
instead of 'kubectl wait' on pod label selectors. The daemonset
object exists as soon as 'kubectl create' returns, and rollout status
correctly waits for all desired pods to be created and become ready.
The old selector wait also silently under-waited: pods created after
its initial LIST were never waited on at all. The stricter wait
revealed that full multi-node convergence can exceed 2 minutes on
busy runners, so the wait timeout is raised to 5 minutes (the waits
return as soon as the rollouts are ready, so this costs nothing on
healthy runs).
- Wait for the host-net-config daemonset rollout in both workflows
before proceeding; it configures the host routing that the multus
public network depends on and was previously not waited on at all.
Pin its jonlabelle/network-tools image.
- test_multus_connections: retry the osd dump / fs dump network checks
for up to 2 minutes. Daemons register their addresses in the mon maps
asynchronously after the cluster reports ready; one canary failure
(2026-04-29) ran the MDS check at fsmap epoch 1, before any MDS had
registered.
- Dump cluster state (pods, daemonsets, events, multus logs) when the
multus workflow job fails; the workflow previously had no failure
diagnostics at all.
Also remove the unused NUMBER_OF_COMPUTE_NODES env var from the multus
workflow (kind-config.yaml creates 3 workers; nothing consumes it).
Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
(cherry picked from commit 4da654bae2)
72 lines
2.5 KiB
YAML
72 lines
2.5 KiB
YAML
####################################################################################################
|
|
# this daemonset configures each minikube or KinD node to route to the multus public net
|
|
# it is not suitable for production
|
|
####################################################################################################
|
|
apiVersion: apps/v1
|
|
kind: DaemonSet
|
|
metadata:
|
|
name: host-net-config
|
|
labels:
|
|
app: host-net-config
|
|
spec:
|
|
selector:
|
|
matchLabels:
|
|
app: host-net-config
|
|
updateStrategy:
|
|
type: RollingUpdate
|
|
rollingUpdate:
|
|
maxUnavailable: 100 # allow updating all at once
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app: host-net-config
|
|
spec:
|
|
hostNetwork: true
|
|
tolerations:
|
|
# also run on control plane nodes for KinD testing
|
|
- key: "node-role.kubernetes.io/control-plane"
|
|
operator: "Exists"
|
|
terminationGracePeriodSeconds: 0 # allow updating/deleting immediately
|
|
containers:
|
|
- name: test
|
|
# tag sha-6257a44 == the image's 'latest' as of 2026-06-10, pinned
|
|
# so the moving tag cannot break or slow CI
|
|
image: jonlabelle/network-tools:sha-6257a44
|
|
env:
|
|
- name: IFACE_NAME
|
|
value: eth0 # IFACE_NAME
|
|
command:
|
|
- sh
|
|
- -x
|
|
- -c
|
|
args:
|
|
- |
|
|
IFACE='$(IFACE_NAME)'
|
|
|
|
NODE_PUBLIC_NET_IP_FIRST3='192.168.29'
|
|
NET_ATT_DEF_PUBLIC_NET_CIDR='192.168.20.0/24'
|
|
|
|
# get the last part of the interface's IP addr
|
|
ip -4 addr show $IFACE
|
|
ip_cidr="$(ip -4 addr show $IFACE | grep inet | awk '{print $2}')" # e.g., 192.168.100.3/24
|
|
ip="${ip_cidr%/*}" # e.g., 192.168.100.3
|
|
last="${ip##*.}" # e.g., 3
|
|
|
|
# add a shim to connect IFACE to the macvlan public network, with a static IP
|
|
# avoid IP conflicts by reusing the last part of the existing IFACE IP
|
|
ip link add public-shim link ${IFACE} type macvlan mode bridge
|
|
ip addr add ${NODE_PUBLIC_NET_IP_FIRST3}.${last}/24 dev public-shim
|
|
ip link set public-shim up
|
|
ip route add ${NET_ATT_DEF_PUBLIC_NET_CIDR} dev public-shim
|
|
|
|
ip addr show $IFACE
|
|
ip addr show public-shim
|
|
ip route show
|
|
|
|
sleep infinity
|
|
resources: {}
|
|
securityContext:
|
|
privileged: true
|
|
capabilities:
|
|
add: ["NET_ADMIN"]
|