From 1e64f241f973372cea673a5515ad63968a9774dc Mon Sep 17 00:00:00 2001 From: Ivan Grokhotkov Date: Thu, 26 May 2022 01:03:01 +0200 Subject: [PATCH 1/4] tools/docker: add IDF_CLONE_SHALLOW and IDF_INSTALL_TARGETS arguments These two arguments can be used to reduce the size of the Docker image: - Setting IDF_CLONE_SHALLOW enables shallow cloning. - Setting IDF_INSTALL_TARGETS to the comma separated list of targets results in toolchains being installed only for these targets. --- tools/docker/Dockerfile | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/tools/docker/Dockerfile b/tools/docker/Dockerfile index 5cf2ff4ecc..e46f14cb59 100644 --- a/tools/docker/Dockerfile +++ b/tools/docker/Dockerfile @@ -41,20 +41,28 @@ RUN : \ # It is possibe to combine both, e.g.: # IDF_CLONE_BRANCH_OR_TAG=release/vX.Y # IDF_CHECKOUT_REF=. +# Use IDF_CLONE_SHALLOW=1 to peform shallow clone (i.e. --depth=1 --shallow-submodules) +# Use IDF_INSTALL_TARGETS to install tools only for selected chip targets (CSV) ARG IDF_CLONE_URL=https://github.com/espressif/esp-idf.git ARG IDF_CLONE_BRANCH_OR_TAG=master ARG IDF_CHECKOUT_REF= +ARG IDF_CLONE_SHALLOW= +ARG IDF_INSTALL_TARGETS=all ENV IDF_PATH=/opt/esp/idf ENV IDF_TOOLS_PATH=/opt/esp RUN echo IDF_CHECKOUT_REF=$IDF_CHECKOUT_REF IDF_CLONE_BRANCH_OR_TAG=$IDF_CLONE_BRANCH_OR_TAG && \ git clone --recursive \ + ${IDF_CLONE_SHALLOW:+--depth=1 --shallow-submodules} \ ${IDF_CLONE_BRANCH_OR_TAG:+-b $IDF_CLONE_BRANCH_OR_TAG} \ $IDF_CLONE_URL $IDF_PATH && \ if [ -n "$IDF_CHECKOUT_REF" ]; then \ cd $IDF_PATH && \ + if [ -n "$IDF_CLONE_SHALLOW" ]; then \ + git fetch origin --depth=1 --recurse-submodules ${IDF_CHECKOUT_REF}; \ + fi && \ git checkout $IDF_CHECKOUT_REF && \ git submodule update --init --recursive; \ fi @@ -63,7 +71,7 @@ RUN echo IDF_CHECKOUT_REF=$IDF_CHECKOUT_REF IDF_CLONE_BRANCH_OR_TAG=$IDF_CLONE_B ARG CRYPTOGRAPHY_DONT_BUILD_RUST=1 RUN : \ && update-ca-certificates --fresh \ - && $IDF_PATH/tools/idf_tools.py --non-interactive install required \ + && $IDF_PATH/tools/idf_tools.py --non-interactive install required --targets=${IDF_INSTALL_TARGETS} \ && $IDF_PATH/tools/idf_tools.py --non-interactive install cmake \ && $IDF_PATH/tools/idf_tools.py --non-interactive install-python-env \ && rm -rf $IDF_TOOLS_PATH/dist \ From a1eb5c77e9fd4d1dad9ec8e4b58c2d0f7b5f0c6e Mon Sep 17 00:00:00 2001 From: Ivan Grokhotkov Date: Thu, 26 May 2022 01:04:14 +0200 Subject: [PATCH 2/4] docs: document build arguments of the Docker image --- docs/en/api-guides/tools/idf-docker-image.rst | 21 ++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/docs/en/api-guides/tools/idf-docker-image.rst b/docs/en/api-guides/tools/idf-docker-image.rst index f4b02e1aa6..8b104f8b59 100644 --- a/docs/en/api-guides/tools/idf-docker-image.rst +++ b/docs/en/api-guides/tools/idf-docker-image.rst @@ -59,7 +59,7 @@ The above command explained: To build with a specific docker image tag, specify it as ``espressif/idf:TAG``, for example:: - docker run --rm -v $PWD:/project -w /project espressif/idf:release-v4.0 idf.py build + docker run --rm -v $PWD:/project -w /project espressif/idf:release-v4.4 idf.py build You can check the up-to-date list of available tags at https://hub.docker.com/r/espressif/idf/tags. @@ -100,3 +100,22 @@ Then inside the container, use ``idf.py`` as usual:: .. note:: Commands which communicate with the development board, such as ``idf.py flash`` and ``idf.py monitor`` will not work in the container unless the serial port is passed through into the container. However currently this is not possible with Docker for Windows (https://github.com/docker/for-win/issues/1018) and Docker for Mac (https://github.com/docker/for-mac/issues/900). + +Building custom images +====================== + +The Dockerfile in ESP-IDF repository provides several build arguments which can be used to customize the Docker image: + +- ``IDF_CLONE_URL``: URL of the repository to clone ESP-IDF from. Can be set to a custom URL when working with a fork of ESP-IDF. Default is ``https://github.com/espressif/esp-idf.git``. +- ``IDF_CLONE_BRANCH_OR_TAG``: Name of a git branch or tag use when cloning ESP-IDF. This value is passed to ``git clone`` command using the ``--branch`` argument. Default is ``master``. +- ``IDF_CHECKOUT_REF``: If this argument is set to a non-empty value, ``git checkout $IDF_CHECKOUT_REF`` command will be performed after cloning. This argument can be set to the SHA of the specific commit to check out, for example if some specific commit on a release branch is desired. +- ``IDF_CLONE_SHALLOW``: If this argument is set to a non-empty value, ``--depth=1 --shallow-submodules`` arguments will be used when performing ``git clone``. This significantly reduces the amount of data downloaded and the size of the resulting Docker image. However, if switching to a different branch in such a "shallow" repository is necessary, an additional ``git fetch origin `` command must be executed first. +- ``IDF_INSTALL_TARGETS``: Comma-separated list of IDF targets to install toolchains for, or ``all`` to install toolchains for all targets. Selecting specific targets reduces the amount of data downloaded and the size of the resulting Docker image. Default is ``all``. + +To use these arguments, pass them via the ``--build-arg`` command line option. For example, the following command will build a Docker image with a shallow clone of ESP-IDF v4.4.1 and tools for ESP32-C3, only:: + + docker build -t idf-custom:v4.4.1-esp32c3 \ + --build-arg IDF_CLONE_BRANCH_OR_TAG=v4.4.1 \ + --build-arg IDF_CLONE_SHALLOW=1 \ + --build-arg IDF_INSTALL_TARGETS=esp32c3 \ + tools/docker From cf44cbfdc97460582e483ba5265a5b0c308fb7a7 Mon Sep 17 00:00:00 2001 From: Ivan Grokhotkov Date: Thu, 26 May 2022 01:07:58 +0200 Subject: [PATCH 3/4] ci: build and push Docker images in Github actions Replaces the previously used Docker Hub autobuild infrastructure. This allows for more flexible configuration of the build process, at the expense of some extra maintenance of CI workflow files required. --- .github/workflows/docker.yml | 75 ++++++++++++++++++++++++++++++++++++ tools/ci/executable-list.txt | 1 - tools/docker/hooks/build | 14 ------- 3 files changed, 75 insertions(+), 15 deletions(-) create mode 100644 .github/workflows/docker.yml delete mode 100755 tools/docker/hooks/build diff --git a/.github/workflows/docker.yml b/.github/workflows/docker.yml new file mode 100644 index 0000000000..d1ee4a8d73 --- /dev/null +++ b/.github/workflows/docker.yml @@ -0,0 +1,75 @@ +name: docker + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + +on: + push: + branches: + - 'master' + - 'release/*' + tags: + - 'v*.*' + +env: + # Platforms to build the image for + BUILD_PLATFORMS: linux/amd64 + DOCKERHUB_REPO: ${{ github.repository_owner }}/idf + +jobs: + docker: + # Disable the job in forks + if: ${{ github.repository_owner == 'espressif' }} + + runs-on: ubuntu-latest + steps: + # Depending on the branch/tag, set CLONE_BRANCH_OR_TAG variable (used in the Dockerfile + # as a build arg) and TAG_NAME (used when tagging the image). + # + # The following 3 steps cover the alternatives (tag, release branch, master branch): + - name: Set variables (tags) + if: ${{ github.ref_type == 'tag' }} + run: | + echo "CLONE_BRANCH_OR_TAG=$GITHUB_REF_NAME" >> $GITHUB_ENV + echo "TAG_NAME=$GITHUB_REF_NAME" >> $GITHUB_ENV + - name: Set variables (release branches) + if: ${{ github.ref_type == 'branch' && startsWith(github.ref_name, 'release/') }} + run: | + echo "CLONE_BRANCH_OR_TAG=$GITHUB_REF_NAME" >> $GITHUB_ENV + echo "TAG_NAME=release-${GITHUB_REF_NAME##release/}" >> $GITHUB_ENV + - name: Set variables (main branch) + if: ${{ github.ref_type == 'branch' && github.ref_name == 'master' }} + run: | + echo "CLONE_BRANCH_OR_TAG=master" >> $GITHUB_ENV + echo "TAG_NAME=latest" >> $GITHUB_ENV + + # Display the variables set above, just in case. + - name: Check variables + run: | + echo "CLONE_BRANCH_OR_TAG: $CLONE_BRANCH_OR_TAG" + echo "CHECKOUT_REF: $CHECKOUT_REF" + echo "TAG_NAME: $TAG_NAME" + + # The following steps are the standard boilerplate from + # https://github.com/marketplace/actions/build-and-push-docker-images + - name: Checkout + uses: actions/checkout@v3 + - name: Login to Docker Hub + uses: docker/login-action@v2 + with: + username: ${{ secrets.DOCKERHUB_USERNAME }} + password: ${{ secrets.DOCKERHUB_TOKEN }} + - name: Set up QEMU for multiarch builds + uses: docker/setup-qemu-action@v2 + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v2 + - name: Build and push + uses: docker/build-push-action@v3 + with: + context: tools/docker + push: true + tags: ${{ env.DOCKERHUB_REPO }}:${{ env.TAG_NAME }} + platforms: ${{ env.BUILD_PLATFORMS }} + build-args: | + IDF_CLONE_URL=${{ github.server_url }}/${{ github.repository }}.git + IDF_CLONE_BRANCH_OR_TAG=${{ env.CLONE_BRANCH_OR_TAG }} diff --git a/tools/ci/executable-list.txt b/tools/ci/executable-list.txt index 866d97fbbe..8ac1eb2370 100644 --- a/tools/ci/executable-list.txt +++ b/tools/ci/executable-list.txt @@ -61,7 +61,6 @@ tools/ci/test_configure_ci_environment.sh tools/cmake/convert_to_cmake.py tools/cmake/run_cmake_lint.sh tools/docker/entrypoint.sh -tools/docker/hooks/build tools/esp_app_trace/logtrace_proc.py tools/esp_app_trace/sysviewtrace_proc.py tools/esp_app_trace/test/logtrace/test.sh diff --git a/tools/docker/hooks/build b/tools/docker/hooks/build deleted file mode 100755 index f98295115d..0000000000 --- a/tools/docker/hooks/build +++ /dev/null @@ -1,14 +0,0 @@ -#!/bin/bash - -# This file gets executed to build the image on the Docker Hub. -# See https://docs.docker.com/docker-hub/builds/advanced/#build-hook-examples for details. - -set -euo pipefail - -echo "Building for branch ${SOURCE_BRANCH}, commit ${SOURCE_COMMIT}" - -docker build \ - --build-arg IDF_CLONE_BRANCH_OR_TAG=${SOURCE_BRANCH} \ - --build-arg IDF_CHECKOUT_REF=${SOURCE_COMMIT} \ - -f $DOCKERFILE_PATH \ - -t $IMAGE_NAME . From 7d9aff3913a04c76b59a97615070ba8e41a2f1b7 Mon Sep 17 00:00:00 2001 From: Ivan Grokhotkov Date: Thu, 26 May 2022 01:10:04 +0200 Subject: [PATCH 4/4] tools/docker: add README.md file to be displayed on Docker Hub Closes https://github.com/espressif/esp-idf/issues/7933 --- .github/workflows/docker.yml | 12 +++++++ docs/en/api-guides/tools/idf-docker-image.rst | 3 ++ tools/docker/README.md | 31 +++++++++++++++++++ 3 files changed, 46 insertions(+) create mode 100644 tools/docker/README.md diff --git a/.github/workflows/docker.yml b/.github/workflows/docker.yml index d1ee4a8d73..9a9afbcd44 100644 --- a/.github/workflows/docker.yml +++ b/.github/workflows/docker.yml @@ -73,3 +73,15 @@ jobs: build-args: | IDF_CLONE_URL=${{ github.server_url }}/${{ github.repository }}.git IDF_CLONE_BRANCH_OR_TAG=${{ env.CLONE_BRANCH_OR_TAG }} + + - name: Update Docker Hub repository description (master branch) + if: ${{ github.ref_type == 'branch' && github.ref_name == 'master' }} + uses: peter-evans/dockerhub-description@v3 + with: + username: ${{ secrets.DOCKERHUB_USERNAME }} + # Token based authentication is not supported here: + # https://github.com/peter-evans/dockerhub-description/issues/10 + # https://github.com/docker/roadmap/issues/115#issuecomment-891694974 + password: ${{ secrets.DOCKERHUB_PASSWORD }} + repository: ${{ env.DOCKERHUB_REPO }} + readme-filepath: ./tools/docker/README.md diff --git a/docs/en/api-guides/tools/idf-docker-image.rst b/docs/en/api-guides/tools/idf-docker-image.rst index 8b104f8b59..7ddb2eb4e4 100644 --- a/docs/en/api-guides/tools/idf-docker-image.rst +++ b/docs/en/api-guides/tools/idf-docker-image.rst @@ -2,6 +2,9 @@ IDF Docker Image **************** +.. + When changing this page, please keep tools/docker/README.md in sync. + .. highlight:: bash IDF Docker image (``espressif/idf``) is intended for building applications and libraries with specific versions of ESP-IDF, when doing automated builds. diff --git a/tools/docker/README.md b/tools/docker/README.md new file mode 100644 index 0000000000..e5e12bd88b --- /dev/null +++ b/tools/docker/README.md @@ -0,0 +1,31 @@ + + +# ESP-IDF Docker Image + +This is a Docker image for the [Espressif IoT Development Framework (ESP-IDF)](https://github.com/espressif/esp-idf). It is intended for building applications and libraries with specific versions of ESP-IDF, when doing automated builds. + +This image contains a copy of ESP-IDF and all the tools necessary to build ESP-IDF projects. + +## Tags + +Multiple tags of this image are maintained: + +- `latest`: tracks `master` branch of ESP-IDF +- `vX.Y`: corresponds to ESP-IDF release `vX.Y` +- `release-vX.Y`: tracks `release/vX.Y` branch of ESP-IDF + +## Basic Usage + +Build a project located in the current directory using `idf.py build` command: + +```bash +docker run --rm -v $PWD:/project -w /project espressif/idf:latest idf.py build +``` + +## Documentation + +For more information about this image and the detailed usage instructions, please refer to the ESP-IDF Programming Guide page: [IDF Docker Image](https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-guides/tools/idf-docker-image.html).