espressif/esp-idf
1
0
Fork 1
mirror of https://github.com/espressif/esp-idf.git synced 2025-11-02 16:11:41 +01:00
Code Issues Packages Projects Releases Wiki Activity

doc: linux host test documentation

Browse Source
This commit is contained in:
Jakob Hasse
2020-12-24 17:34:03 +08:00
parent 469c137c83
commit 68393c41c4
11 changed files with 239 additions and 36 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
docs/en/api-guides/index.rst
View File

@@ -38,7 +38,8 @@ API Guides
:SOC_ULP_SUPPORTED: ULP Coprocessor <ulp>
:esp32: ULP Coprocessor (Legacy GNU Make) <ulp-legacy>
:SOC_RISCV_COPROC_SUPPORTED: ULP-RISC-V Coprocessor <ulp-risc-v>
Unit Testing <unit-tests>
Unit Testing (Target) <unit-tests>
Unit Testing (Linux Host) <linux-host-testing>
:esp32: Unit Testing (Legacy GNU Make) <unit-tests-legacy>
:SOC_USB_SUPPORTED: USB Console <usb-console>
WiFi Driver <wifi>

65
docs/en/api-guides/linux-host-testing.rst Normal file
View File

@@ -0,0 +1,65 @@
Unit Testing on Linux
=====================
.. note::
Host testing with IDF is experimental for now. We try our best to keep interfaces stable but can't guarantee it for now. Feedback via github or the forum on esp32.com is highly welcome, though and may influence the future design of the host-based tests.
This article provides an overview of unit tests with IDF on Linux. For using unit tests on the target, please refer to :doc:`target based unit testing <unit-tests>`.
Embedded Software Tests
-----------------------
Embedded software tests are challenging due to the following factors:
- Difficulties running tests efficiently.
- Lack of many operating system abstractions when interfacing with hardware, making it difficult to isolate code under test.
To solve these two problems, Linux host-based tests with `CMock <https://www.throwtheswitch.org/cmock>`_ are introduced. Linux host-based tests are more efficient than unit tests on the target since they:
- Compile the necessary code only
- Don't need time to upload to a target
- Run much faster on a host-computer, compared to an ESP
Using the `CMock <https://www.throwtheswitch.org/cmock>`_ framework also solves the problem of hardware dependencies. Through mocking, hardware details are emulated and specified at run time, but only if necessary.
Of course, using code on the host and using mocks does not fully represent the target device. Thus, two kinds of tests are recommended:
1. Unit tests which test program logic on a Linux machine, isolated through mocks.
2. System/Integration tests which test the interaction of components and the whole system. They run on the target, where irrelevant components and code may as well be emulated via mocks.
This documentation is about the first kind of tests. Refer to :doc:`target based unit testing <unit-tests>` for more information on target tests (the second kind of tests).
IDF Unit Tests on Linux Host
----------------------------
The current focus of the Linux host tests is on creating isolated unit tests of components, while mocking the component's dependencies with CMock.
A complete implementation of IDF to run on Linux does not exist currently.
There are currently two examples for running IDF-built code on Linux host:
- An example :example_file:`hello-world application <build_system/cmake/linux_host_app/README.md>`
- A :component_file:`unit test for NVS <nvs_flash/host_test/nvs_page_test/README.md>`.
Inside the component which should be tested, there is a separate directory ``host_test``, besides the "traditional" ``test`` directory or the ``test_apps`` directory. It has one or more subdirectories::
- host_test/
- fixtures/
contains test fixtures (structs/functions to do test case set-up and tear-down).
If there are no fixtures, this can be ommitted.
- <test_name>/
IDF applications which run the tests
- <test_name2>/
Further tests are possible.
The IDF applications inside ``host_test`` set the mocking configuration as described in the :doc:`IDF unit test documentation <unit-tests>`.
The :component_file:`NVS page unit test <nvs_flash/host_test/nvs_page_test/main/nvs_page_test.cpp>` provides some illustration of how to control the mocks.
Requirements
^^^^^^^^^^^^
Besides the usual IDF requirements and CMock requirements, only the host's ``GCC/g++`` is required.
The host tests have been tested on Ubuntu 20.04 with ``GCC`` version 9 and 10.

41
docs/en/api-guides/unit-tests.rst
View File

@@ -2,7 +2,10 @@ Unit Testing in {IDF_TARGET_NAME}
=================================
:link_to_translation:`zh_CN:[中文]`
ESP-IDF comes with a unit test application that is based on the Unity - unit test framework. Unit tests are integrated in the ESP-IDF repository and are placed in the ``test`` subdirectories of each component respectively.
ESP-IDF comes with two possibilities to test software.
- A unit test application which runs on the target and that is based on the Unity - unit test framework. These unit tests are integrated in the ESP-IDF repository and are placed in the ``test`` subdirectories of each component respectively. Target-based unit tests are covered in this document.
- Linux-host based unit tests in which all the hardware is abstracted via mocks. Linux-host based tests are still under development and only a small fraction of IDF components supports them currently. They are covered here: :doc:`target based unit testing <linux-host-testing>`.
Normal Test Cases
------------------
@@ -274,13 +277,24 @@ One limitation of the cache compensated timer is that the task that benchmarked
Mocks
-----
ESP-IDF has a component which integrates the CMock mocking framework. CMock usually uses Unity as a submodule, but due to some Espressif-internal limitations with CI, we still have Unity as an ordinary module in ESP-IDF.
One of the biggest problems for unit testing in embedded systems are the strong hardware dependencies. This is why ESP-IDF has a component which integrates the `CMock <https://www.throwtheswitch.org/cmock>`_ mocking framework. Ideally, all components other than the one which should be tested *(component under test)* are mocked. This way, the test environment has complete control over all the interaction with the component under test. However, if mocking becomes problematic due to the tests becoming too specific, more "real" IDF code can always be included into the tests.
To use the IDF-supplied Unity component which isn't a submodule, the build system needs to pass an environment variable ``UNITY_IDR`` to CMock. This variable simply contains the path to the Unity directory in IDF, e.g. ``export "UNITY_DIR=${IDF_PATH}/components/unity/unity"``.
Besides the usual IDF requirements, ``ruby`` is necessary to generate the mocks. Refer to :component_file:`cmock/CMock/docs/CMock_Summary.md` for more details on how CMock works and how to create and use mocks.
Refer to :component_file:`cmock/CMock/lib/cmock_generator.rb` to see how the Unity directory is determined in CMock.
In IDF, adjustments are necessary inside the component(s) that should be mocked as well as inside the unit test, compared to writing normal components or unit tests without mocking.
An example cmake build command to create mocks of a component inside that component's CMakeLists.txt may look like this:
Adjustments in Mock Component
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The component that should be mocked requires a separate ``mock`` directory containing all additional files needed specifically for the mocking. Most importantly, it contains ``mock_config.yaml`` which configures CMock. For more details on what the options inside that configuration file mean and how to write your own, please take a look at the :component_file:`CMock documentation <cmock/CMock/docs/CMock_Summary.md>`. It may be necessary to have some more files related to mocking which should also be placed inside the `mock` directory.
Furthermore, the component's ``CMakeLists.txt`` needs a switch to build mocks instead of the actual code. This is usually done by checking the component property ``USE_MOCK`` for the particular component. E.g., the ``spi_flash`` component execute the following code in its ``CMakeLists.txt`` to check whether mocks should be built:
.. code-block:: cmake
idf_component_get_property(spi_flash_mock ${COMPONENT_NAME} USE_MOCK)
An example CMake build command to create mocks of a component inside its ``CMakeLists.txt`` may look like this:
.. code-block:: cmake
@@ -290,6 +304,19 @@ An example cmake build command to create mocks of a component inside that compon
COMMAND ${CMAKE_COMMAND} -E env "UNITY_DIR=${IDF_PATH}/components/unity/unity" ruby ${CMOCK_DIR}/lib/cmock.rb -o${CMAKE_CURRENT_SOURCE_DIR}/mock/mock_config.yaml ${MOCK_HEADERS}
)
${MOCK_OUTPUT} contains all CMock generated output files, ${MOCK_HEADERS} contains all headers to be mocked and ${CMOCK_DIR} needs to be set to CMock directory inside IDF. ${CMAKE_COMMAND} is automatically set.
``${MOCK_OUTPUT}`` contains all CMock generated output files, ``${MOCK_HEADERS}`` contains all headers to be mocked and ``${CMOCK_DIR}`` needs to be set to the CMock directory inside IDF. ``${CMAKE_COMMAND}`` is automatically set by the IDF build system.
Refer to :component_file:`cmock/CMock/docs/CMock_Summary.md` for more details on how CMock works and how to create and use mocks.
One aspect of CMock's usage is special here: CMock usually uses Unity as a submodule, but due to some Espressif-internal limitations with CI, IDF still uses Unity as an ordinary module in ESP-IDF. To use the IDF-supplied Unity component, which isn't a submodule, the build system needs to pass an environment variable ``UNITY_IDR`` to CMock. This variable simply contains the path to the Unity directory in IDF, e.g. ``export "UNITY_DIR=${IDF_PATH}/components/unity/unity"``. Refer to :component_file:`cmock/CMock/lib/cmock_generator.rb` to see how the Unity directory is determined in CMock.
An example ``CMakeLists.txt`` which enables mocking exists :component_file:`in spi_flash <spi_flash/CMakeLists.txt>`
Adjustments in Unit Test
^^^^^^^^^^^^^^^^^^^^^^^^
The unit test needs to set the component property ``USE_MOCK`` for the component that should be mocked. This lets the dependent component build the mocks instead of the actual component. E.g., in the nvs host test's :component_file:`CMakeLists.txt <nvs_flash/host_test/nvs_page_test/CMakeLists.txt>`, ``spi_flash`` mocks are enabled by the following line:
.. code-block:: cmake
idf_component_set_property(spi_flash USE_MOCK 1)
Refer to the :component_file:`NVS host unit test <nvs_flash/host_test/nvs_page_test/README.md>` for more information on how to use and control CMock inside a unit test.