Joshua Hoblitt
4da654bae2
Multus CI failures on master pushes from 2026-04-08 to 2026-06-10 were dominated by a race in setup-multus.sh: 'kubectl wait' was invoked on a pod label selector immediately after 'kubectl create' of a daemonset, and it exits with "error: no matching resources found" when the daemonset controller has not created any pods yet. This caused 9 of the 10 genuine multus flakes across the standalone multus workflow and the canary multus-public-and-cluster job (the two share setup-multus.sh). - setup-multus.sh: wait with 'kubectl rollout status' on the daemonsets instead of 'kubectl wait' on pod label selectors. The daemonset object exists as soon as 'kubectl create' returns, and rollout status correctly waits for all desired pods to be created and become ready. The old selector wait also silently under-waited: pods created after its initial LIST were never waited on at all. The stricter wait revealed that full multi-node convergence can exceed 2 minutes on busy runners, so the wait timeout is raised to 5 minutes (the waits return as soon as the rollouts are ready, so this costs nothing on healthy runs). - Wait for the host-net-config daemonset rollout in both workflows before proceeding; it configures the host routing that the multus public network depends on and was previously not waited on at all. Pin its jonlabelle/network-tools image. - test_multus_connections: retry the osd dump / fs dump network checks for up to 2 minutes. Daemons register their addresses in the mon maps asynchronously after the cluster reports ready; one canary failure (2026-04-29) ran the MDS check at fsmap epoch 1, before any MDS had registered. - Dump cluster state (pods, daemonsets, events, multus logs) when the multus workflow job fails; the workflow previously had no failure diagnostics at all. Also remove the unused NUMBER_OF_COMPUTE_NODES env var from the multus workflow (kind-config.yaml creates 3 workers; nothing consumes it). Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
72 lines
2.5 KiB
YAML
72 lines
2.5 KiB
YAML
####################################################################################################
|
|
# this daemonset configures each minikube or KinD node to route to the multus public net
|
|
# it is not suitable for production
|
|
####################################################################################################
|
|
apiVersion: apps/v1
|
|
kind: DaemonSet
|
|
metadata:
|
|
name: host-net-config
|
|
labels:
|
|
app: host-net-config
|
|
spec:
|
|
selector:
|
|
matchLabels:
|
|
app: host-net-config
|
|
updateStrategy:
|
|
type: RollingUpdate
|
|
rollingUpdate:
|
|
maxUnavailable: 100 # allow updating all at once
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app: host-net-config
|
|
spec:
|
|
hostNetwork: true
|
|
tolerations:
|
|
# also run on control plane nodes for KinD testing
|
|
- key: "node-role.kubernetes.io/control-plane"
|
|
operator: "Exists"
|
|
terminationGracePeriodSeconds: 0 # allow updating/deleting immediately
|
|
containers:
|
|
- name: test
|
|
# tag sha-6257a44 == the image's 'latest' as of 2026-06-10, pinned
|
|
# so the moving tag cannot break or slow CI
|
|
image: jonlabelle/network-tools:sha-6257a44
|
|
env:
|
|
- name: IFACE_NAME
|
|
value: eth0 # IFACE_NAME
|
|
command:
|
|
- sh
|
|
- -x
|
|
- -c
|
|
args:
|
|
- |
|
|
IFACE='$(IFACE_NAME)'
|
|
|
|
NODE_PUBLIC_NET_IP_FIRST3='192.168.29'
|
|
NET_ATT_DEF_PUBLIC_NET_CIDR='192.168.20.0/24'
|
|
|
|
# get the last part of the interface's IP addr
|
|
ip -4 addr show $IFACE
|
|
ip_cidr="$(ip -4 addr show $IFACE | grep inet | awk '{print $2}')" # e.g., 192.168.100.3/24
|
|
ip="${ip_cidr%/*}" # e.g., 192.168.100.3
|
|
last="${ip##*.}" # e.g., 3
|
|
|
|
# add a shim to connect IFACE to the macvlan public network, with a static IP
|
|
# avoid IP conflicts by reusing the last part of the existing IFACE IP
|
|
ip link add public-shim link ${IFACE} type macvlan mode bridge
|
|
ip addr add ${NODE_PUBLIC_NET_IP_FIRST3}.${last}/24 dev public-shim
|
|
ip link set public-shim up
|
|
ip route add ${NET_ATT_DEF_PUBLIC_NET_CIDR} dev public-shim
|
|
|
|
ip addr show $IFACE
|
|
ip addr show public-shim
|
|
ip route show
|
|
|
|
sleep infinity
|
|
resources: {}
|
|
securityContext:
|
|
privileged: true
|
|
capabilities:
|
|
add: ["NET_ADMIN"]
|