ci: convert integration and canary CI from minikube to kind
The integration and canary suites ran on a single-node minikube `driver: none` cluster, where the kubelet ran directly on the GitHub runner, so the host docker daemon doubled as the cluster runtime and host block devices and host paths were directly visible to pods. Replace that with a kind cluster. Every suite creates its cluster through the shared integration-test-setup-cluster-resources composite action, so the conversion is centralized there and converts the smoke, object, helm, keystone, multi-cluster, upgrade, on-release, nightly, encryption-KMS and all canary jobs at once. - Replace the setup-minikube step with helm/kind-action, selecting the kubernetes version via the kindest/node image tag and creating a single-node cluster from a new kind config (kind pinned to v0.32.0 for reproducibility). - Drop the cri-dockerd install; kind nodes use their built-in containerd. - Add a kind config that bind-mounts the host /dev, /var/lib/rook and /run/udev into the node so the existing host-based disk-prep helpers (use_local_disk*, create_partitions_for_osds, blockDevicePV.sh, localPathPV.sh, ...) keep working unchanged: devices and partitions created on the host appear in the node and in the OSD pods that hostPath-mount the node /dev, and ceph-volume can read the host udev database it needs to inventory disks. - Prepare the kind node for the host-level operations rook runs against the underlying host: remount /sys read-write so CSI's kernel RBD mapping (`rbd map --device-type krbd`, which writes /sys/bus/rbd) works, and install lvm2 and cryptsetup, which rook runs in the node's mount namespace to provision LVM- and encryption-backed OSDs. kindest/node images provide none of this; the minikube driver:none runner host did. - Route the Service and pod CIDRs from the runner to the kind node so host-side tests (the `go test` process runs on the runner) can reach in-cluster ClusterIPs, e.g. an S3 request to the RGW service. With minikube driver:none the runner already shared the cluster network. - Load locally built images into the cluster. Under minikube `driver: none` the built image was already in the cluster runtime; under kind it must be imported, so build_rook and create_helm_tag now import their images into each node's containerd through a new load_image_into_cluster helper (via the node's ctr, which avoids the kind/kindest-node containerd-config version skew that breaks `kind load docker-image`). - Point Vault's kubernetes-auth at the in-cluster API endpoint (kubernetes.default.svc) instead of the kubeconfig server URL: kind exposes that as https://127.0.0.1:<port>, unreachable from the in-cluster Vault pod, so OSD encryption-key retrieval via k8s-auth failed. - Replace the remaining direct minikube references in the canary workflow: a `minikube kubectl` call and the external-cluster topology values. - Adapt host-name assumptions that only held under driver:none: resolve the disk-cleanup job by the k8s node name rather than the runner hostname, and let kind-action ignore post-job cluster-teardown failures (the runner is ephemeral; nvme/multus devices can wedge `docker rm` of the node). - Update stale comments that described the CI environment as minikube. - Move the multus integration test's kind config under tests/config too, so both kind cluster configs live in one place. create-dev-cluster.sh and other local-dev tooling are intentionally left on minikube. Signed-off-by: Joshua Hoblitt <josh@hoblitt.com>
This commit is contained in:
1 parent
3bc3fd63ea
commit
d59abb1956
11 files changed
+143
-45
No files matched your search
@@ -280,15 +280,15 @@ jobs:
|
||||
kubectl wait --for='jsonpath={.status.phase}=Ready' Cephblockpool/replica1c -nrook-ceph
|
||||
|
||||
# pass correct flags
|
||||
kubectl -n rook-ceph exec "$toolbox" -- python3 /etc/ceph/create-external-cluster-resources.py --rbd-data-pool-name replicapool --topology-pools replica1a,replica1b,replica1c --topology-failure-domain-label hostname --topology-failure-domain-values minikube,minikube-m02,minikube-m03
|
||||
kubectl -n rook-ceph exec "$toolbox" -- python3 /etc/ceph/create-external-cluster-resources.py --rbd-data-pool-name replicapool --topology-pools replica1a,replica1b,replica1c --topology-failure-domain-label hostname --topology-failure-domain-values kind-control-plane,arbitrary,value
|
||||
# pass the pool which is not exists
|
||||
if output=$(kubectl -n rook-ceph exec "$toolbox" -- python3 /etc/ceph/create-external-cluster-resources.py --rbd-data-pool-name replicapool --topology-pools ab,cd,ef --topology-failure-domain-label hostname --topology-failure-domain-values minikube,minikube-m02,minikube-m03); then
|
||||
if output=$(kubectl -n rook-ceph exec "$toolbox" -- python3 /etc/ceph/create-external-cluster-resources.py --rbd-data-pool-name replicapool --topology-pools ab,cd,ef --topology-failure-domain-label hostname --topology-failure-domain-values kind-control-plane,arbitrary,value); then
|
||||
echo "script run completed with stderr error after passing the wrong pools: $output"
|
||||
else
|
||||
echo "script failed because wrong pools doesn't exist"
|
||||
fi
|
||||
# dont pass all topology flags
|
||||
if output=$(kubectl -n rook-ceph exec "$toolbox" -- python3 /etc/ceph/create-external-cluster-resources.py --rbd-data-pool-name replicapool --topology-pools replica1a,replica1b,replica1c --topology-failure-domain-values minikube,minikube-m02,minikube-m03); then
|
||||
if output=$(kubectl -n rook-ceph exec "$toolbox" -- python3 /etc/ceph/create-external-cluster-resources.py --rbd-data-pool-name replicapool --topology-pools replica1a,replica1b,replica1c --topology-failure-domain-values kind-control-plane,arbitrary,value); then
|
||||
echo "script run completed with stderr error after passing the wrong flags: $output"
|
||||
else
|
||||
echo "script failed because topology-failure-domain-label is missing"
|
||||
@@ -1831,7 +1831,7 @@ jobs:
|
||||
timeout 45 sh -c 'until [ "$(kubectl -n rook-ceph exec -t deploy/rook-ceph-fs-mirror -- ls -1 /var/run/ceph/|grep -c asok)" -gt 3 ]; do echo "waiting for connection to peer" && sleep 1; done'
|
||||
sockets=$(kubectl -n rook-ceph exec -t deploy/rook-ceph-fs-mirror -- ls -1 /var/run/ceph/)
|
||||
# shellcheck disable=SC2086 # word splitting of $sockets into separate socket paths is intentional
|
||||
status=$(for socket in $sockets; do minikube kubectl -- -n rook-ceph exec -t deploy/rook-ceph-fs-mirror -- ceph --admin-daemon "/var/run/ceph/$socket" help|awk -F ":" '/get filesystem mirror status/ {print $1}'; done)
|
||||
status=$(for socket in $sockets; do kubectl -n rook-ceph exec -t deploy/rook-ceph-fs-mirror -- ceph --admin-daemon "/var/run/ceph/$socket" help|awk -F ":" '/get filesystem mirror status/ {print $1}'; done)
|
||||
if [ "${#status}" -lt 1 ]; then echo "peer addition failed" && exit 1; fi
|
||||
|
||||
- name: display cephfilesystem and fs mirror daemon status
|
||||
|
||||
@@ -22,39 +22,27 @@ runs:
|
||||
with:
|
||||
go-version: "1.26"
|
||||
|
||||
- name: Install cri-dockerd
|
||||
shell: bash --noprofile --norc -eo pipefail -x {0}
|
||||
run: |
|
||||
ARCH=$(go env GOARCH)
|
||||
# --fail so an HTTP error page is not saved as the .deb (dpkg then fails on the
|
||||
# corrupt archive), and retry transient download errors
|
||||
curl -fLO --retry 5 --retry-all-errors --retry-delay 10 https://github.com/Mirantis/cri-dockerd/releases/download/v0.3.21/cri-dockerd_0.3.21.3-0.ubuntu-focal_"${ARCH}".deb
|
||||
sudo dpkg -i cri-dockerd_0.3.21.3-0.ubuntu-focal_"${ARCH}".deb
|
||||
rm -f cri-dockerd_0.3.21.3-0.ubuntu-focal_"${ARCH}".deb
|
||||
|
||||
- name: Setup Minikube
|
||||
uses: medyagh/setup-minikube@e9e035a86bbc3caea26a450bd4dbf9d0c453682e # v0.0.21
|
||||
- name: create kind cluster
|
||||
uses: helm/kind-action@ef37e7f390d99f746eb8b610417061a60e82a6cc # v1.14.0
|
||||
with:
|
||||
# lock minikube version to ensure release branch CI doesn't fail when latest minikube no
|
||||
# longer supports older k8s version we lock to
|
||||
minikube-version: "1.38.0"
|
||||
kubernetes-version: ${{ inputs.kubernetes-version }}
|
||||
driver: none
|
||||
container-runtime: docker
|
||||
memory: 6g
|
||||
cpus: 2
|
||||
addons: ingress
|
||||
cni: calico
|
||||
# --force: when the requested kubernetes version is missing from minikube's compiled-in
|
||||
# version list (any version released after the pinned minikube), 'minikube start'
|
||||
# verifies the version with an anonymous GitHub API request, which is regularly
|
||||
# rate-limited on shared runner IPs and fails the job (K8S_FAIL_CONNECT, exit code 40).
|
||||
# The versions used here are pinned constants already validated by the PRs that bump
|
||||
# them, so that check adds nothing in CI; an invalid version would still fail fast at
|
||||
# the kubeadm/kubelet download.
|
||||
# --wait-timeout: minikube's default 6m node-ready wait expires on slow runners
|
||||
# (GUEST_START, exit code 80).
|
||||
start-args: --force --wait-timeout=15m
|
||||
# Pin kind; its release sets which kindest/node image versions are available.
|
||||
version: v0.32.0
|
||||
# Each matrix kubernetes-version needs a published kindest/node:<version> image.
|
||||
node_image: kindest/node:${{ inputs.kubernetes-version }}
|
||||
# See the config; it bind-mounts the host paths the disk-prep helpers need.
|
||||
config: tests/config/kind-config.yaml
|
||||
cluster_name: kind
|
||||
wait: 300s
|
||||
# Runner is ephemeral; some jobs hold node devices via the /dev mount, wedging teardown.
|
||||
ignore_failed_clean: "true"
|
||||
|
||||
- name: prepare the kind node
|
||||
shell: bash --noprofile --norc -eo pipefail -x {0}
|
||||
run: tests/scripts/github-action-helper.sh prepare_kind_node
|
||||
|
||||
- name: route host to cluster services
|
||||
shell: bash --noprofile --norc -eo pipefail -x {0}
|
||||
run: tests/scripts/github-action-helper.sh add_host_routes_to_cluster
|
||||
|
||||
- name: install deps
|
||||
shell: bash --noprofile --norc -eo pipefail -x {0}
|
||||
|
||||
@@ -46,7 +46,7 @@ jobs:
|
||||
- name: Create KinD Cluster
|
||||
uses: helm/kind-action@ef37e7f390d99f746eb8b610417061a60e82a6cc # v1.14.0
|
||||
with:
|
||||
config: tests/scripts/multus/kind-config.yaml
|
||||
config: tests/config/multus/kind-config.yaml
|
||||
cluster_name: kind
|
||||
|
||||
- name: consider (pre-job) debugging
|
||||
|
||||
Reference in new issue
Block a user