ci: convert integration and canary CI from minikube to kind

The integration and canary suites ran on a single-node minikube
`driver: none` cluster, where the kubelet ran directly on the GitHub
runner, so the host docker daemon doubled as the cluster runtime and
host block devices and host paths were directly visible to pods. Replace
that with a kind cluster.

Every suite creates its cluster through the shared
integration-test-setup-cluster-resources composite action, so the
conversion is centralized there and converts the smoke, object, helm,
keystone, multi-cluster, upgrade, on-release, nightly, encryption-KMS and
all canary jobs at once.

- Replace the setup-minikube step with helm/kind-action, selecting the
  kubernetes version via the kindest/node image tag and creating a
  single-node cluster from a new kind config (kind pinned to v0.32.0 for
  reproducibility).
- Drop the cri-dockerd install; kind nodes use their built-in containerd.
- Add a kind config that bind-mounts the host /dev, /var/lib/rook and
  /run/udev into the node so the existing host-based disk-prep helpers
  (use_local_disk*, create_partitions_for_osds, blockDevicePV.sh,
  localPathPV.sh, ...) keep working unchanged: devices and partitions
  created on the host appear in the node and in the OSD pods that
  hostPath-mount the node /dev, and ceph-volume can read the host udev
  database it needs to inventory disks.
- Prepare the kind node for the host-level operations rook runs against the
  underlying host: remount /sys read-write so CSI's kernel RBD mapping
  (`rbd map --device-type krbd`, which writes /sys/bus/rbd) works, and install
  lvm2 and cryptsetup, which rook runs in the node's mount namespace to
  provision LVM- and encryption-backed OSDs. kindest/node images provide none
  of this; the minikube driver:none runner host did.
- Route the Service and pod CIDRs from the runner to the kind node so
  host-side tests (the `go test` process runs on the runner) can reach
  in-cluster ClusterIPs, e.g. an S3 request to the RGW service. With minikube
  driver:none the runner already shared the cluster network.
- Load locally built images into the cluster. Under minikube `driver: none`
  the built image was already in the cluster runtime; under kind it must be
  imported, so build_rook and create_helm_tag now import their images into
  each node's containerd through a new load_image_into_cluster helper (via the
  node's ctr, which avoids the kind/kindest-node containerd-config version
  skew that breaks `kind load docker-image`).
- Point Vault's kubernetes-auth at the in-cluster API endpoint
  (kubernetes.default.svc) instead of the kubeconfig server URL: kind exposes
  that as https://127.0.0.1:<port>, unreachable from the in-cluster Vault pod,
  so OSD encryption-key retrieval via k8s-auth failed.
- Replace the remaining direct minikube references in the canary workflow: a
  `minikube kubectl` call and the external-cluster topology values.
- Adapt host-name assumptions that only held under driver:none: resolve the
  disk-cleanup job by the k8s node name rather than the runner hostname, and
  let kind-action ignore post-job cluster-teardown failures (the runner is
  ephemeral; nvme/multus devices can wedge `docker rm` of the node).
- Update stale comments that described the CI environment as minikube.
- Move the multus integration test's kind config under tests/config too, so
  both kind cluster configs live in one place.

create-dev-cluster.sh and other local-dev tooling are intentionally left on
minikube.

Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
This commit is contained in:
Joshua Hoblitt committed 2026-07-06 08:42:39 -07:00
1 parent 3bc3fd63ea
commit d59abb1956
11 files changed
+143 -45

No files matched your search

@@ -280,15 +280,15 @@ jobs:
kubectl wait --for='jsonpath={.status.phase}=Ready' Cephblockpool/replica1c -nrook-ceph
# pass correct flags
kubectl -n rook-ceph exec "$toolbox" -- python3 /etc/ceph/create-external-cluster-resources.py --rbd-data-pool-name replicapool --topology-pools replica1a,replica1b,replica1c --topology-failure-domain-label hostname --topology-failure-domain-values minikube,minikube-m02,minikube-m03
kubectl -n rook-ceph exec "$toolbox" -- python3 /etc/ceph/create-external-cluster-resources.py --rbd-data-pool-name replicapool --topology-pools replica1a,replica1b,replica1c --topology-failure-domain-label hostname --topology-failure-domain-values kind-control-plane,arbitrary,value
# pass the pool which is not exists
if output=$(kubectl -n rook-ceph exec "$toolbox" -- python3 /etc/ceph/create-external-cluster-resources.py --rbd-data-pool-name replicapool --topology-pools ab,cd,ef --topology-failure-domain-label hostname --topology-failure-domain-values minikube,minikube-m02,minikube-m03); then
if output=$(kubectl -n rook-ceph exec "$toolbox" -- python3 /etc/ceph/create-external-cluster-resources.py --rbd-data-pool-name replicapool --topology-pools ab,cd,ef --topology-failure-domain-label hostname --topology-failure-domain-values kind-control-plane,arbitrary,value); then
echo "script run completed with stderr error after passing the wrong pools: $output"
else
echo "script failed because wrong pools doesn't exist"
fi
# dont pass all topology flags
if output=$(kubectl -n rook-ceph exec "$toolbox" -- python3 /etc/ceph/create-external-cluster-resources.py --rbd-data-pool-name replicapool --topology-pools replica1a,replica1b,replica1c --topology-failure-domain-values minikube,minikube-m02,minikube-m03); then
if output=$(kubectl -n rook-ceph exec "$toolbox" -- python3 /etc/ceph/create-external-cluster-resources.py --rbd-data-pool-name replicapool --topology-pools replica1a,replica1b,replica1c --topology-failure-domain-values kind-control-plane,arbitrary,value); then
echo "script run completed with stderr error after passing the wrong flags: $output"
else
echo "script failed because topology-failure-domain-label is missing"
@@ -1831,7 +1831,7 @@ jobs:
timeout 45 sh -c 'until [ "$(kubectl -n rook-ceph exec -t deploy/rook-ceph-fs-mirror -- ls -1 /var/run/ceph/|grep -c asok)" -gt 3 ]; do echo "waiting for connection to peer" && sleep 1; done'
sockets=$(kubectl -n rook-ceph exec -t deploy/rook-ceph-fs-mirror -- ls -1 /var/run/ceph/)
# shellcheck disable=SC2086 # word splitting of $sockets into separate socket paths is intentional
status=$(for socket in $sockets; do minikube kubectl -- -n rook-ceph exec -t deploy/rook-ceph-fs-mirror -- ceph --admin-daemon "/var/run/ceph/$socket" help|awk -F ":" '/get filesystem mirror status/ {print $1}'; done)
status=$(for socket in $sockets; do kubectl -n rook-ceph exec -t deploy/rook-ceph-fs-mirror -- ceph --admin-daemon "/var/run/ceph/$socket" help|awk -F ":" '/get filesystem mirror status/ {print $1}'; done)
if [ "${#status}" -lt 1 ]; then echo "peer addition failed" && exit 1; fi
- name: display cephfilesystem and fs mirror daemon status
@@ -22,39 +22,27 @@ runs:
with:
go-version: "1.26"
- name: Install cri-dockerd
shell: bash --noprofile --norc -eo pipefail -x {0}
run: |
ARCH=$(go env GOARCH)
# --fail so an HTTP error page is not saved as the .deb (dpkg then fails on the
# corrupt archive), and retry transient download errors
curl -fLO --retry 5 --retry-all-errors --retry-delay 10 https://github.com/Mirantis/cri-dockerd/releases/download/v0.3.21/cri-dockerd_0.3.21.3-0.ubuntu-focal_"${ARCH}".deb
sudo dpkg -i cri-dockerd_0.3.21.3-0.ubuntu-focal_"${ARCH}".deb
rm -f cri-dockerd_0.3.21.3-0.ubuntu-focal_"${ARCH}".deb
- name: Setup Minikube
uses: medyagh/setup-minikube@e9e035a86bbc3caea26a450bd4dbf9d0c453682e # v0.0.21
- name: create kind cluster
uses: helm/kind-action@ef37e7f390d99f746eb8b610417061a60e82a6cc # v1.14.0
with:
# lock minikube version to ensure release branch CI doesn't fail when latest minikube no
# longer supports older k8s version we lock to
minikube-version: "1.38.0"
kubernetes-version: ${{ inputs.kubernetes-version }}
driver: none
container-runtime: docker
memory: 6g
cpus: 2
addons: ingress
cni: calico
# --force: when the requested kubernetes version is missing from minikube's compiled-in
# version list (any version released after the pinned minikube), 'minikube start'
# verifies the version with an anonymous GitHub API request, which is regularly
# rate-limited on shared runner IPs and fails the job (K8S_FAIL_CONNECT, exit code 40).
# The versions used here are pinned constants already validated by the PRs that bump
# them, so that check adds nothing in CI; an invalid version would still fail fast at
# the kubeadm/kubelet download.
# --wait-timeout: minikube's default 6m node-ready wait expires on slow runners
# (GUEST_START, exit code 80).
start-args: --force --wait-timeout=15m
# Pin kind; its release sets which kindest/node image versions are available.
version: v0.32.0
# Each matrix kubernetes-version needs a published kindest/node:<version> image.
node_image: kindest/node:${{ inputs.kubernetes-version }}
# See the config; it bind-mounts the host paths the disk-prep helpers need.
config: tests/config/kind-config.yaml
cluster_name: kind
wait: 300s
# Runner is ephemeral; some jobs hold node devices via the /dev mount, wedging teardown.
ignore_failed_clean: "true"
- name: prepare the kind node
shell: bash --noprofile --norc -eo pipefail -x {0}
run: tests/scripts/github-action-helper.sh prepare_kind_node
- name: route host to cluster services
shell: bash --noprofile --norc -eo pipefail -x {0}
run: tests/scripts/github-action-helper.sh add_host_routes_to_cluster
- name: install deps
shell: bash --noprofile --norc -eo pipefail -x {0}
+1 -1
View File
@@ -46,7 +46,7 @@ jobs:
- name: Create KinD Cluster
uses: helm/kind-action@ef37e7f390d99f746eb8b610417061a60e82a6cc # v1.14.0
with:
config: tests/scripts/multus/kind-config.yaml
config: tests/config/multus/kind-config.yaml
cluster_name: kind
- name: consider (pre-job) debugging