Merge branch 'docs/update_idf_build_apps_and_pytest' into 'master'

Docs/update idf build apps and pytest

See merge request espressif/esp-idf!22696

.gitlab/ci/README.md

@@ -238,86 +238,7 @@ To run these commands in shell script locally, place `source tools/ci/utils.sh`
 
 ### Grammar
 
-#### Operands
+We're using the latest version of [idf-build-apps][idf-build-apps]. Please refer to their [documentation][manifest-doc]
 
-- Variables start with `SOC_`. The value would be parsed from components/soc/[TARGET]/include/soc/*_caps.h
+[idf-build-apps]: https://github.com/espressif/idf-build-apps
-- `IDF_TARGET`
+[manifest-doc]: https://docs.espressif.com/projects/idf-build-apps/en/latest/manifest.html
-- `INCLUDE_DEFAULT` (The default value of officially supported targets is 1, otherwise is 0)
-- String, must be double-quoted. e.g. `"esp32"`, `"12345"`
-- Integer, support decimal and hex. e.g. `1`, `0xAB`
-- List with String and Integer inside, the type could be mixed. e.g. `["esp32", 1]`
-
-#### Operators
-
-- `==`, `!=`, `>`, `>=`, `<`, `<=`
-- `and`, `or`
-- `in`, `not in` with list
-- parentheses
-
-#### Limitation:
-
-- all operators are binary operator. For more than two operands, you may use nested parentheses trick. For example,
-  - `A == 1 or (B == 2 and C in [1,2,3])`
-  - `(A == 1 and B == 2) or (C not in ["3", "4", 5])`
-
-### How does it work?
-
-By default, we enable build and test jobs for supported targets. In other words, if an app supports all supported targets, it does not need to be added in a manifest file. The manifest files are files that set the violation rules for apps.
-
-three rules (disable rules are calculated after the `enable` rule):
-- enable: run CI build/test jobs for targets that match any of the specified conditions only
-- disable: will not run CI build/test jobs for targets that match any of the specified conditions
-- disable_test: will not run CI test jobs for targets that match any of the specified conditions
-
-Each key is a test folder. Will apply to all folders inside.
-
-If one sub folder is in a special case, you can overwrite the rules for this folder by adding another entry for this folder itself. Each folder's rules are standalone, and will not inherit its parent's rules. (YAML inheritance is too complicated for reading)
-
-For example in the following codeblock, only `disable` rule exists in `examples/foo/bar`. It's unaware of its parent's `enable` rule. 
-
-```yaml
-examples/foo:
-  enable:
-    - if: IDF_TARGET == "esp32"
-
-examples/foo/bar:
-  disable:
-    - if: IDF_TARGET == "esp32s2"
-```
-
-### Example
-
-```yaml
-examples/foo:
-  enable:
-    - if IDF_TARGET in ["esp32", 1, 2, 3]
-    - if IDF_TARGET not in ["4", "5", 6]
-  # should be run under all targets!
-
-examples/bluetooth:
-  disable:  # disable both build and tests jobs
-    - if: SOC_BT_SUPPORTED != 1
-    # reason is optional if there's no `temporary: true`
-  disable_test:
-    - if: IDF_TARGET == "esp32"
-      temporary: true
-      reason: lack of ci runners  # required when `temporary: true`
-
-examples/bluetooth/test_foo:
-  # each folder's settings are standalone
-  disable:
-    - if: IDF_TARGET == "esp32s2"
-      temporary: true
-      reason: no idea
-  # unlike examples/bluetooth, the apps under this folder would not be build nor test for "no idea" under target esp32s2
-
-examples/get-started/hello_world:
-  enable:
-    - if: IDF_TARGET == "linux"
-      reason: this one only supports linux!
-
-examples/get-started/blink:
-  enable:
-    - if: INCLUDE_DEFAULT == 1 or IDF_TARGET == "linux"
-      reason: This one supports all supported targets and linux
-```

docs/en/contribute/esp-idf-tests-with-pytest.rst

@@ -543,19 +543,51 @@ The binaries in the target test jobs are downloaded from build jobs, the artifac
 Run the Tests Locally
 =====================
 
-The local executing process is the same as the CI process.
+First you need to install ESP-IDF with additional python requirements:
 
-For example, if you want to run all the esp32 tests under the ``$IDF_PATH/examples/system/console/basic`` folder, you may:
+.. code-block:: shell
-
-.. code:: shell
 
    $ cd $IDF_PATH
    $ bash install.sh --enable-pytest
    $ . ./export.sh
+
+By default, the pytest script will look for the build directory in this order:
+
+- ``build_<target>_<sdkconfig>``
+- ``build_<target>``
+- ``build_<sdkconfig>``
+- ``build``
+
+Which means, the simplest way to run pytest is calling ``idf.py build``.
+
+For example, if you want to run all the esp32 tests under the ``$IDF_PATH/examples/get-started/hello_world`` folder, you should run:
+
+.. code-block:: shell
+
+   $ cd examples/get-started/hello_world
+   $ idf.py build
+   $ pytest --target esp32
+
+If you have multiple sdkconfig files in your test app, like those ``sdkconfig.ci.*`` files, the simple ``idf.py build`` won't apply the extra sdkconfig files. Let's take ``$IDF_PATH/examples/system/console/basic`` as an example.
+
+If you want to test this app with config ``history``, and build with ``idf.py build``, you should run
+
+.. code-block:: shell
+
+   $ cd examples/system/console/basic
+   $ idf.py -DSDKCONFIG_DEFAULTS="sdkconfig.defaults;sdkconfig.ci.history" build
+   $ pytest --target esp32 --sdkconfig history
+
+If you want to build and test with all sdkconfig files at the same time, you should use our CI script as an helper script:
+
+.. code-block:: shell
+
    $ cd examples/system/console/basic
    $ python $IDF_PATH/tools/ci/ci_build_apps.py . --target esp32 -vv --pytest-apps
    $ pytest --target esp32
 
+The app with ``sdkconfig.ci.history`` will be built in ``build_esp32_history``, and the app with ``sdkconfig.ci.nohistory`` will be built in ``build_esp32_nohistory``. ``pytest --target esp32`` will run tests on both apps.
+
 Tips and Tricks
 ===============