mirror of
https://github.com/wolfSSL/wolfssl.git
synced 2026-07-05 09:40:51 +02:00
CI: move Arduino cores from actions/cache to ghcr bundles
arduino.yml's per-core actions/cache layer stored the installed cores and toolchains (~/.arduino15) - several GB, dominated by the esp32 and mbed cores - in the 10 GB Actions cache. For esp32 it was also ineffective: the disk-cleanup step deletes the esp32 toolchain before actions/cache saves it, so esp32 re-downloaded every run anyway. - New arduino-cores-image workflow resolves each of the 9 distinct cores and publishes a tar of ~/.arduino15 + ~/Arduino/libraries to ghcr.io/<owner>/wolfssl-ci-arduino:<core>. It runs monthly: esp32, the fastest-moving core, releases ~monthly and the rest far less often. - New install-arduino-core composite action restores that bundle offline and verifies the core is present, falling back to `arduino-cli core install` when the bundle is unavailable - so nothing breaks until the image is first published and made public. - arduino.yml calls the action in place of the inline core install and the actions/cache step. This takes the flaky espressif / esp8266.com / pjrc.com downloads off the PR critical path and frees the Actions cache of the largest binaries it held.
This commit is contained in:
Juliusz Sosinowicz
@@ -0,0 +1,81 @@
|
||||
name: 'Install Arduino core'
|
||||
description: >
|
||||
Make an Arduino core (and the shared CI libraries) available, preferring a
|
||||
prebuilt bundle pulled from ghcr (published by the arduino-cores-image
|
||||
workflow) and falling back to `arduino-cli core install` when the bundle is
|
||||
unavailable or stale. Assumes arduino-cli is already on PATH.
|
||||
inputs:
|
||||
core-id:
|
||||
description: 'vendor:arch core to make available, e.g. esp32:esp32'
|
||||
required: true
|
||||
board-manager-url:
|
||||
description: >
|
||||
Optional third-party board_manager index URL, used only on the
|
||||
online-install fallback (the ghcr bundle already carries its own).
|
||||
required: false
|
||||
default: ''
|
||||
libs:
|
||||
description: 'Space-separated Arduino libraries to ensure are present'
|
||||
required: false
|
||||
default: 'ArduinoJson WiFiNINA Ethernet Bridge'
|
||||
runs:
|
||||
using: 'composite'
|
||||
steps:
|
||||
# Preferred path: restore ~/.arduino15 (the core + toolchain) and the
|
||||
# shared libraries from a prebuilt tarball pulled from ghcr, so the flaky
|
||||
# board_manager / toolchain downloads are off the PR critical path. The
|
||||
# bundle is published only under the wolfssl org (gated below), so fork PRs
|
||||
# read the public upstream image too. Best-effort: any failure leaves
|
||||
# "satisfied" unset and the online install below runs unchanged.
|
||||
- name: Restore Arduino core from ghcr bundle
|
||||
id: ghcr
|
||||
shell: bash
|
||||
run: |
|
||||
set -u
|
||||
command -v docker >/dev/null 2>&1 || { echo "::notice::docker unavailable; installing core online"; exit 0; }
|
||||
command -v arduino-cli >/dev/null 2>&1 || { echo "::notice::arduino-cli not on PATH; installing core online"; exit 0; }
|
||||
CORE_ID='${{ inputs.core-id }}'
|
||||
TAG=$(echo "$CORE_ID" | tr ':' '-')
|
||||
IMG="ghcr.io/wolfssl/wolfssl-ci-arduino:$TAG"
|
||||
if ! docker pull -q "$IMG" >/dev/null 2>&1; then
|
||||
echo "::notice::ghcr bundle $IMG unavailable; installing core online"
|
||||
exit 0
|
||||
fi
|
||||
cid=$(docker create "$IMG" 2>/dev/null) || { echo "::notice::cannot open bundle; installing core online"; exit 0; }
|
||||
rm -f "$RUNNER_TEMP/arduino-core.tar"
|
||||
docker cp "$cid:/arduino-core.tar" "$RUNNER_TEMP/arduino-core.tar" >/dev/null 2>&1 || true
|
||||
docker rm "$cid" >/dev/null 2>&1 || true
|
||||
test -f "$RUNNER_TEMP/arduino-core.tar" || { echo "::notice::bundle had no tarball; installing core online"; exit 0; }
|
||||
# Entries are stored relative to $HOME (.arduino15/..., Arduino/libraries/...).
|
||||
tar -C "$HOME" -xf "$RUNNER_TEMP/arduino-core.tar" || { echo "::notice::could not unpack bundle; installing core online"; exit 0; }
|
||||
rm -f "$RUNNER_TEMP/arduino-core.tar"
|
||||
if arduino-cli core list 2>/dev/null | awk 'NR>1 {print $1}' | grep -Fxq "$CORE_ID"; then
|
||||
echo "satisfied=true" >> "$GITHUB_OUTPUT"
|
||||
echo "Restored $CORE_ID from $IMG"
|
||||
else
|
||||
echo "::notice::bundle did not yield $CORE_ID; installing core online"
|
||||
fi
|
||||
|
||||
- name: Install Arduino core online
|
||||
if: steps.ghcr.outputs.satisfied != 'true'
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
CORE_ID='${{ inputs.core-id }}'
|
||||
BM_URL='${{ inputs.board-manager-url }}'
|
||||
retry() { local i; for i in 1 2 3 4 5; do "$@" && return 0; sleep $((2**i)); done; "$@"; }
|
||||
|
||||
arduino-cli config init --overwrite
|
||||
# Wait up to 10 minutes for the big toolchain downloads.
|
||||
arduino-cli config set network.connection_timeout 600s
|
||||
# Scope third-party indexes to the one core that needs them: arduino-cli
|
||||
# re-reads every configured index on each call and fails if any is
|
||||
# unreachable, so an unconditional URL makes all jobs depend on it.
|
||||
if [ -n "$BM_URL" ]; then
|
||||
arduino-cli config add board_manager.additional_urls "$BM_URL"
|
||||
fi
|
||||
retry arduino-cli core update-index
|
||||
retry arduino-cli core install "$CORE_ID"
|
||||
for lib in ${{ inputs.libs }}; do
|
||||
retry arduino-cli lib install "$lib"
|
||||
done
|
||||
@@ -0,0 +1,135 @@
|
||||
name: Arduino cores image
|
||||
|
||||
# Builds the prebuilt Arduino core bundles that arduino.yml restores offline
|
||||
# (see .github/actions/install-arduino-core). Each bundle is a tar of
|
||||
# ~/.arduino15 (the installed core + toolchain) and ~/Arduino/libraries (the
|
||||
# shared CI libraries) for one vendor:arch core, published to
|
||||
# ghcr.io/<owner>/wolfssl-ci-arduino:<core> (':' in the core id becomes '-').
|
||||
#
|
||||
# Why: the core/toolchain downloads (espressif, esp8266.com, pjrc.com) are
|
||||
# large and chronically flaky from runner egress, and the old actions/cache
|
||||
# layer both pressed on the 10 GB cache cap and - for esp32 - was deleted by
|
||||
# arduino.yml's disk cleanup before it was ever saved. Resolving each core ONCE
|
||||
# here and pulling it from ghcr on every PR keeps those downloads off the PR
|
||||
# critical path. ghcr storage/bandwidth is free for public images.
|
||||
#
|
||||
# ONE-TIME SETUP: after the first successful run, make the package
|
||||
# `wolfssl-ci-arduino` PUBLIC (repo/org > Packages > Package settings >
|
||||
# Change visibility). Anonymous `docker pull` then works from fork PRs too;
|
||||
# until then install-arduino-core simply installs the core online (no breakage).
|
||||
|
||||
on:
|
||||
schedule:
|
||||
# Monthly (1st). esp32 - the fastest-moving core - releases roughly monthly
|
||||
# and the rest far less often, so a monthly unconditional rebuild tracks
|
||||
# them closely enough; between rebuilds install-arduino-core installs any
|
||||
# newer core online. Each run republishes every bundle.
|
||||
- cron: '0 4 1 * *'
|
||||
workflow_dispatch:
|
||||
|
||||
concurrency:
|
||||
group: arduino-cores-image-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
packages: write
|
||||
|
||||
jobs:
|
||||
build:
|
||||
name: build ${{ matrix.core_id }}
|
||||
if: github.repository_owner == 'wolfssl'
|
||||
# teensy:avr's index lives at pjrc.com, chronically unreachable from runner
|
||||
# egress; let it fail without blocking the other eight bundles.
|
||||
continue-on-error: ${{ matrix.core_id == 'teensy:avr' }}
|
||||
runs-on: ubuntu-24.04
|
||||
timeout-minutes: 30
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
include:
|
||||
# Distinct vendor:arch cores behind arduino.yml's board matrix. The
|
||||
# esp32 and mbed_* cores are the GB-scale toolchains; the AVR/SAM/SAMD
|
||||
# cores are small. board_url is set only for cores whose index is not
|
||||
# in the default board manager.
|
||||
- core_id: arduino:avr
|
||||
- core_id: arduino:samd
|
||||
- core_id: arduino:sam
|
||||
- core_id: arduino:mbed_edge
|
||||
- core_id: arduino:mbed_portenta
|
||||
- core_id: arduino:renesas_uno
|
||||
- core_id: esp32:esp32
|
||||
- core_id: esp8266:esp8266
|
||||
board_url: https://arduino.esp8266.com/stable/package_esp8266com_index.json
|
||||
- core_id: teensy:avr
|
||||
board_url: https://www.pjrc.com/teensy/package_teensy_index.json
|
||||
steps:
|
||||
- name: Free disk space
|
||||
shell: bash
|
||||
run: |
|
||||
sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc /opt/hostedtoolcache/CodeQL
|
||||
sudo apt-get clean
|
||||
df -h
|
||||
|
||||
- name: Install Arduino CLI
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
mkdir -p "$HOME/bin"
|
||||
echo "$HOME/bin" >> "$GITHUB_PATH"
|
||||
curl -fsSL --retry 5 --retry-delay 10 \
|
||||
https://raw.githubusercontent.com/arduino/arduino-cli/master/install.sh \
|
||||
| BINDIR="$HOME/bin" sh
|
||||
"$HOME/bin/arduino-cli" version
|
||||
|
||||
- name: Install the core and shared libraries
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
CORE_ID='${{ matrix.core_id }}'
|
||||
BM_URL='${{ matrix.board_url }}'
|
||||
retry() { local i; for i in 1 2 3 4 5; do "$@" && return 0; sleep $((2**i)); done; "$@"; }
|
||||
|
||||
arduino-cli config init --overwrite
|
||||
arduino-cli config set network.connection_timeout 600s
|
||||
if [ -n "$BM_URL" ]; then
|
||||
arduino-cli config add board_manager.additional_urls "$BM_URL"
|
||||
fi
|
||||
retry arduino-cli core update-index
|
||||
retry arduino-cli core install "$CORE_ID"
|
||||
# Mirror arduino.yml's always-installed libraries so consumers get a
|
||||
# complete bundle.
|
||||
for lib in ArduinoJson WiFiNINA Ethernet Bridge; do
|
||||
retry arduino-cli lib install "$lib"
|
||||
done
|
||||
mkdir -p "$HOME/Arduino/libraries"
|
||||
|
||||
- name: Pack the bundle tarball
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
mkdir -p "$RUNNER_TEMP/ctx"
|
||||
# Paths relative to $HOME so install-arduino-core can `tar -C $HOME -x`
|
||||
# straight back. Drop the staging area and any wolfssl lib (arduino.yml
|
||||
# always installs the latest wolfssl itself).
|
||||
tar --exclude='.arduino15/staging' --exclude='Arduino/libraries/wolfssl' \
|
||||
-C "$HOME" -cf "$RUNNER_TEMP/ctx/arduino-core.tar" .arduino15 Arduino/libraries
|
||||
echo "Tarball size: $(du -h "$RUNNER_TEMP/ctx/arduino-core.tar" | cut -f1)"
|
||||
|
||||
- name: Log in to ghcr
|
||||
shell: bash
|
||||
run: echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u "${{ github.actor }}" --password-stdin
|
||||
|
||||
- name: Build and push bundle
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
OWNER=$(echo "${{ github.repository_owner }}" | tr '[:upper:]' '[:lower:]')
|
||||
TAG=$(echo "${{ matrix.core_id }}" | tr ':' '-')
|
||||
IMG="ghcr.io/$OWNER/wolfssl-ci-arduino:$TAG"
|
||||
# Tiny busybox base so the consumer can `docker cp` the tarball out;
|
||||
# the base size is negligible next to the toolchain.
|
||||
printf 'FROM busybox\nCOPY arduino-core.tar /arduino-core.tar\n' > "$RUNNER_TEMP/ctx/Dockerfile"
|
||||
docker build -t "$IMG" "$RUNNER_TEMP/ctx"
|
||||
docker push "$IMG"
|
||||
echo "Pushed $IMG"
|
||||
@@ -197,53 +197,25 @@ jobs:
|
||||
run: |
|
||||
CORE_ID="$(echo '${{ matrix.fqbn }}' | cut -d: -f1-2)"
|
||||
echo "CORE_ID=$CORE_ID" >> "$GITHUB_ENV"
|
||||
# Third-party board_manager index for the cores that need one. Scoped
|
||||
# to the one CORE_ID that uses it: arduino-cli re-reads every index on
|
||||
# each call and fails if any is unreachable, so an unconditional URL
|
||||
# would make all jobs depend on pjrc.com / esp8266.com. Used only on
|
||||
# the online-install fallback; the ghcr bundle already carries it.
|
||||
case "$CORE_ID" in
|
||||
teensy:avr) echo "BM_URL=https://www.pjrc.com/teensy/package_teensy_index.json" >> "$GITHUB_ENV" ;;
|
||||
esp8266:esp8266) echo "BM_URL=https://arduino.esp8266.com/stable/package_esp8266com_index.json" >> "$GITHUB_ENV" ;;
|
||||
*) echo "BM_URL=" >> "$GITHUB_ENV" ;;
|
||||
esac
|
||||
|
||||
- name: Setup Arduino CLI
|
||||
run: |
|
||||
arduino-cli config init
|
||||
|
||||
# wait 10 minutes for big downloads (or use 0 for no limit)
|
||||
arduino-cli config set network.connection_timeout 600s
|
||||
|
||||
# Only add third-party board_manager URLs for matrix entries that actually need them.
|
||||
# arduino-cli re-reads every configured index on each invocation and fails the whole
|
||||
# step if any one is unreachable, so adding these unconditionally makes all jobs
|
||||
# depend on pjrc.com and esp8266.com -- a single outage there cascades into total
|
||||
# CI failure. Scope each URL to the one CORE_ID that uses it.
|
||||
if [ "$CORE_ID" = "teensy:avr" ]; then
|
||||
arduino-cli config add board_manager.additional_urls https://www.pjrc.com/teensy/package_teensy_index.json
|
||||
fi
|
||||
if [ "$CORE_ID" = "esp8266:esp8266" ]; then
|
||||
arduino-cli config add board_manager.additional_urls https://arduino.esp8266.com/stable/package_esp8266com_index.json
|
||||
fi
|
||||
arduino-cli core update-index
|
||||
|
||||
echo "CORE_ID: $CORE_ID"
|
||||
arduino-cli core install "$CORE_ID"
|
||||
|
||||
# The above is instead of:
|
||||
# arduino-cli core install esp32:esp32 # ESP32
|
||||
# arduino-cli core install arduino:avr # Arduino Uno, Mega, Nano
|
||||
# arduino-cli core install arduino:sam # Arduino Due
|
||||
# arduino-cli core install arduino:samd # Arduino Zero
|
||||
# arduino-cli core install teensy:avr # PJRC Teensy
|
||||
# arduino-cli core install esp8266:esp8266 # ESP8266
|
||||
# arduino-cli core install arduino:mbed_nano # nanorp2040connect
|
||||
# arduino-cli core install arduino:mbed_portenta # portenta_h7_m7
|
||||
# arduino-cli core install arduino:mbed_edge
|
||||
# arduino-cli core install arduino:renesas_uno
|
||||
|
||||
# For reference:
|
||||
|
||||
# mbed nano not yet tested
|
||||
# sudo "/home/$USER/.arduino15/packages/arduino/hardware/mbed_nano/4.2.4/post_install.sh"
|
||||
|
||||
# Always install networking (not part of FQBN matrix)
|
||||
# The first one also creates directory: /home/runner/Arduino/libraries
|
||||
arduino-cli lib install "ArduinoJson" # Example dependency
|
||||
arduino-cli lib install "WiFiNINA" # ARDUINO_SAMD_NANO_33_IOT
|
||||
arduino-cli lib install "Ethernet" # Install Ethernet library
|
||||
arduino-cli lib install "Bridge" # Pseudo-network for things like arduino:samd:tian
|
||||
# Restore the core + toolchain + shared libraries from the prebuilt ghcr
|
||||
# bundle (arduino-cores-image), falling back to `arduino-cli core install`
|
||||
# when it is unavailable. Replaces the old per-core actions/cache layer.
|
||||
- name: Install Arduino core and libraries
|
||||
uses: ./.github/actions/install-arduino-core
|
||||
with:
|
||||
core-id: ${{ env.CORE_ID }}
|
||||
board-manager-url: ${{ env.BM_URL }}
|
||||
|
||||
- name: Set Job Environment Variables
|
||||
run: |
|
||||
@@ -270,27 +242,6 @@ jobs:
|
||||
# WOLFSSL_EXAMPLES_ROOT is the repo root, not example location
|
||||
echo "WOLFSSL_EXAMPLES_ROOT = $WOLFSSL_EXAMPLES_ROOT"
|
||||
|
||||
- name: Cache Arduino Packages
|
||||
uses: actions/cache@v5
|
||||
with:
|
||||
path: |
|
||||
~/.arduino15
|
||||
~/.cache/arduino
|
||||
# Exclude staging directory from cache to save space
|
||||
!~/.arduino15/staging
|
||||
|
||||
# Arduino libraries
|
||||
# Specific to Arduino CI Build (2 of 4) Arduinbo Release wolfSSL for Local Examples
|
||||
# Include all libraries, as the latest Arduino-wolfSSL will only change upon release.
|
||||
~/Arduino/libraries
|
||||
# Ensure wolfssl is not cached, we're always using the latest. See separate cache.
|
||||
!~/Arduino/libraries/wolfssl
|
||||
key: arduino-${{ runner.os }}-${{ env.CORE_ID }}-${{ hashFiles('Arduino/sketches/board_list.txt') }}
|
||||
|
||||
restore-keys: |
|
||||
arduino-${{ runner.os }}-${{ env.CORE_ID }}-
|
||||
arduino-${{ runner.os }}-
|
||||
|
||||
- name: Get wolfssl-examples
|
||||
run: |
|
||||
# Fetch Arduino examples from the wolfssl-examples repo
|
||||
|
||||
Reference in New Issue
Block a user