feedc0de/platformio-core
1
0
Fork 0
forked from platformio/platformio-core
Code Pull Requests Activity
Files
a28a8273c571a15f01f08db9ea647acc60bd2caf
platformio-core/docs/plus/unit-testing.rst
Ivan Kravets a28a8273c5 Mention CI for Unit Testing
2016-11-02 20:24:11 +02:00

524 lines
18 KiB
ReStructuredText
Raw Blame History

.. Copyright 2014-present PlatformIO <contact@platformio.org>
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
.. |PIOUTE| replace:: **PlatformIO Unit Testing Engine**
.. |PIOUTF| replace:: *PlatformIO Unit Testing Framework*
.. _unit_testing:
Unit Testing
============
.. versionadded:: 3.0 (`PlatformIO Plus <https://pioplus.com>`_)
`Unit Testing (wiki) <https://en.wikipedia.org/wiki/Unit_testing>`_
is a software testing method by which individual units of source code, sets
of one or more MCU program modules together with associated control data,
usage procedures, and operating procedures, are tested to determine whether
they are fit for use. Unit testing finds problems early in the development cycle.
|PIOUTE| supports 2 different test types:
1. **Desktop Test**. PlatformIO wraps test and main program (from
:ref:`projectconf_pio_src_dir`) with own |PIOUTF|, builds final program
using :ref:`platform_native` and run test on a local host machine (desktop).
.. note::
PlatformIO does not install any toolchains automatically for
:ref:`platform_native` and requires ``GCC`` toolchain to be installed
on your local machine.
Please open Terminal and check that ``gcc`` command is installed.
2. **Embedded Test**. PlatformIO wraps test and main firmware (from
:ref:`projectconf_pio_src_dir`) with own |PIOUTF|, builds special firmware
for a target device and upload it. After uploading, PlatformIO connects
to embedded device (board) using :ref:`projectconf_test_port` , starts
test, collects results and shows test results on the local host machine.
Currently, |PIOUTE| supports these frameworks:
* :ref:`framework_arduino`
* :ref:`framework_energia`
* :ref:`framework_mbed`.
.. note::
Please note that |PIOUTF| uses Serial/UART as communication interface
between PlatformIO Unit Test Engine and target device. If you use
``Serial`` in your project, please wrap/hide Serial-based blocks with
``#ifndef UNIT_TEST`` macro.
There are 2 options how to run tests:
1. **Local**. Allows to run tests on local host machine or on the target devices
(boards) that are directly connected to this machine. In this case, need to
use :ref:`cmd_test` command.
2. **Remote**. Allows to run tests on remote machine or remote target device
(board) without any dependencies to OS software, extra software, SSH, VPN
or opening network ports. Remote Unit Testing works in pair with
:ref:`pio_remote`. In this case, need to use special command
:ref:`cmd_remote_test`.
PlatformIO supports multiple :ref:`ci` systems that can run project
unit tests automatically.
Both commands allow to process specific environments or to ignore some tests
using "Glob patterns". Also, you will be able to ignore some tests for
specific environment using :ref:`projectconf_test_ignore` option
from :ref:`projectconf`.
.. contents::
Demo
----
Demo of `Local & Embedded: Calculator <https://github.com/platformio/platformio-examples/tree/develop/unit-testing/calculator>`_.
.. image:: ../_static/pioplus-unit-testing-demo.png
:target: https://youtu.be/bo3VVRZVKhA
.. _unit_testing_design:
Design
------
|PIOUTE| design is based on a few isolated components:
1. **Main Program**. Contains the independent modules, procedures,
functions or methods that will be the target candidates (TC) for testing.
2. **Unit Test**. This a small independent program that is intended to
re-use TC from the main program and apply tests for them.
3. **Test Processor**. The set of approaches and tools that will be used
to apply test for the environments from :ref:`projectconf`.
Workflow
--------
1. Create PlatformIO project using :ref:`cmd_init` command. For Local Unit
Testing (on the host machine), need to use :ref:`platform_native`.
.. code-block:: ini
; PlatformIO Project Configuration File
;
; Build options: build flags, source filter, extra scripting
; Upload options: custom port, speed and extra flags
; Library options: dependencies, extra library storages
;
; Please visit documentation for the other options and examples
; http://docs.platformio.org/en/stable/projectconf.html
;
; Embedded platforms
;
[env:uno]
platform = atmelavr
framework = arduino
board = uno
[env:nodemcu]
platform = espressif8266
framework = arduino
board = nodemcuv2
;
; Local (PC, native) platforms
;
[env:local]
platform = native
2. Place source code of main program to ``src`` directory.
3. Wrap ``main()`` or ``setup()/loop()`` methods of main program in ``UNIT_TEST``
guard:
.. code-block:: c
/**
* Arduino Wiring-based Framework
*/
#ifndef UNIT_TEST
#include <Arduino.h>
void setup () {
// some code...
}
void loop () {
// some code...
}
#endif
.. code-block:: c
/**
* Generic C/C++
*/
#ifndef UNIT_TEST
int main(int argc, char **argv) {
// setup code...
while (1) {
// loop code...
}
return 0
}
#endif
4. Create ``test`` directory in the root of project. See :ref:`projectconf_pio_test_dir`.
5. Write test using :ref:`unit_testing_api`. The each test is a small
independent program with own ``main()`` or ``setup()/loop()`` methods. Also,
test should start from ``UNITY_BEGIN()`` and finish with ``UNITY_END()``.
6. Place test to ``test`` directory. If you have more than one test, split them
into sub-folders. For example, ``test/test_1/*.[c,cpp,h]``,
``test_N/*.[c,cpp,h]``, etc. If no such directory in ``test`` folder, then
|PIOUTE| will treat the source code of ``test`` folder
as SINGLE test.
7. Run tests using :ref:`cmd_test` command.
.. _unit_testing_api:
API
---
The summary of `Unity Test API <https://github.com/ThrowTheSwitch/Unity#unity-test-api>`_:
* `Running Tests <https://github.com/ThrowTheSwitch/Unity#running-tests>`_
- ``RUN_TEST(func, linenum)``
* `Ignoring Tests <https://github.com/ThrowTheSwitch/Unity#ignoring-tests>`_
- ``TEST_IGNORE()``
- ``TEST_IGNORE_MESSAGE (message)``
* `Aborting Tests <https://github.com/ThrowTheSwitch/Unity#aborting-tests>`_
- ``TEST_PROTECT()``
- ``TEST_ABORT()``
* `Basic Validity Tests <https://github.com/ThrowTheSwitch/Unity#basic-validity-tests>`_
- ``TEST_ASSERT_TRUE(condition)``
- ``TEST_ASSERT_FALSE(condition)``
- ``TEST_ASSERT(condition)``
- ``TEST_ASSERT_UNLESS(condition)``
- ``TEST_FAIL()``
- ``TEST_FAIL_MESSAGE(message)``
* `Numerical Assertions: Integers <https://github.com/ThrowTheSwitch/Unity#numerical-assertions-integers>`_
- ``TEST_ASSERT_EQUAL_INT(expected, actual)``
- ``TEST_ASSERT_EQUAL_INT8(expected, actual)``
- ``TEST_ASSERT_EQUAL_INT16(expected, actual)``
- ``TEST_ASSERT_EQUAL_INT32(expected, actual)``
- ``TEST_ASSERT_EQUAL_INT64(expected, actual)``
- ``TEST_ASSERT_EQUAL_UINT(expected, actual)``
- ``TEST_ASSERT_EQUAL_UINT8(expected, actual)``
- ``TEST_ASSERT_EQUAL_UINT16(expected, actual)``
- ``TEST_ASSERT_EQUAL_UINT32(expected, actual)``
- ``TEST_ASSERT_EQUAL_UINT64(expected, actual)``
- ``TEST_ASSERT_EQUAL_HEX(expected, actual)``
- ``TEST_ASSERT_EQUAL_HEX8(expected, actual)``
- ``TEST_ASSERT_EQUAL_HEX16(expected, actual)``
- ``TEST_ASSERT_EQUAL_HEX32(expected, actual)``
- ``TEST_ASSERT_EQUAL_HEX64(expected, actual)``
- ``TEST_ASSERT_EQUAL_HEX8_ARRAY(expected, actual, elements)``
- ``TEST_ASSERT_EQUAL(expected, actual)``
- ``TEST_ASSERT_INT_WITHIN(delta, expected, actual)``
* `Numerical Assertions: Bitwise <https://github.com/ThrowTheSwitch/Unity#numerical-assertions-bitwise>`_
- ``TEST_ASSERT_BITS(mask, expected, actual)``
- ``TEST_ASSERT_BITS_HIGH(mask, actual)``
- ``TEST_ASSERT_BITS_LOW(mask, actual)``
- ``TEST_ASSERT_BIT_HIGH(mask, actual)``
- ``TEST_ASSERT_BIT_LOW(mask, actual)``
* `Numerical Assertions: Floats <https://github.com/ThrowTheSwitch/Unity#numerical-assertions-floats>`_
- ``TEST_ASSERT_FLOAT_WITHIN(delta, expected, actual)``
- ``TEST_ASSERT_EQUAL_FLOAT(expected, actual)``
- ``TEST_ASSERT_EQUAL_DOUBLE(expected, actual)``
* `String Assertions <https://github.com/ThrowTheSwitch/Unity#string-assertions>`_
- ``TEST_ASSERT_EQUAL_STRING(expected, actual)``
- ``TEST_ASSERT_EQUAL_STRING_LEN(expected, actual, len)``
- ``TEST_ASSERT_EQUAL_STRING_MESSAGE(expected, actual, message)``
- ``TEST_ASSERT_EQUAL_STRING_LEN_MESSAGE(expected, actual, len, message)``
* `Pointer Assertions <https://github.com/ThrowTheSwitch/Unity#pointer-assertions>`_
- ``TEST_ASSERT_NULL(pointer)``
- ``TEST_ASSERT_NOT_NULL(pointer)``
* `Memory Assertions <https://github.com/ThrowTheSwitch/Unity#pointer-assertions>`_
- ``TEST_ASSERT_EQUAL_MEMORY(expected, actual, len)``
User Guide (CLI)
----------------
.. toctree::
:maxdepth: 3
platformio account <../userguide/account/index>
platformio test <../userguide/cmd_test>
platformio remote test <../userguide/remote/cmd_test>
--------------
Test "Blink" Project
--------------------
1. Please follow to :ref:`quickstart` and create "Blink Project". According
to the Unit Testing :ref:`unit_testing_design` it is the **Main program**.
2. Create ``test`` directory in that project (on the same level as ``src``)
and place ``test_main.cpp`` file to it (the source code is located below).
3. Wrap ``setup()`` and ``loop()`` methods of main program in ``UNIT_TEST``
guard.
4. Run tests using :ref:`cmd_test` command.
Project structure
~~~~~~~~~~~~~~~~~
.. code-block:: bash
project_dir
├── lib
│   └── readme.txt
├── platformio.ini
├── src
│   └── main.cpp
└── test
└── test_main.cpp
Source files
~~~~~~~~~~~~
* ``platformio.ini``
.. code-block:: ini
; PlatformIO Project Configuration File
;
; Build options: build flags, source filter, extra scripting
; Upload options: custom port, speed and extra flags
; Library options: dependencies, extra library storages
;
; Please visit documentation for the other options and examples
; http://docs.platformio.org/en/stable/projectconf.html
[env:uno]
platform = atmelavr
framework = arduino
board = uno
[env:nodemcu]
platform = espressif8266
framework = arduino
board = nodemcu
[env:teensy31]
platform = teensy
framework = arduino
board = teensy31
* ``src/main.cpp``
.. code-block:: cpp
/*
* Blink
* Turns on an LED on for one second,
* then off for one second, repeatedly.
*/
#include "Arduino.h"
#ifndef UNIT_TEST // IMPORTANT LINE!
void setup()
{
// initialize LED digital pin as an output.
pinMode(LED_BUILTIN, OUTPUT);
}
void loop()
{
// turn the LED on (HIGH is the voltage level)
digitalWrite(LED_BUILTIN, HIGH);
// wait for a second
delay(1000);
// turn the LED off by making the voltage LOW
digitalWrite(LED_BUILTIN, LOW);
// wait for a second
delay(1000);
}
#endif // IMPORTANT LINE!
* ``test/test_main.cpp``
.. code-block:: cpp
#include <Arduino.h>
#include <unity.h>
#ifdef UNIT_TEST
// void setUp(void) {
// // set stuff up here
// }
// void tearDown(void) {
// // clean stuff up here
// }
void test_led_builtin_pin_number(void) {
TEST_ASSERT_EQUAL(LED_BUILTIN, 13);
}
void test_led_state_high(void) {
digitalWrite(LED_BUILTIN, HIGH);
TEST_ASSERT_EQUAL(digitalRead(LED_BUILTIN), HIGH);
}
void test_led_state_low(void) {
digitalWrite(LED_BUILTIN, LOW);
TEST_ASSERT_EQUAL(digitalRead(LED_BUILTIN), LOW);
}
void setup() {
UNITY_BEGIN(); // IMPORTANT LINE!
RUN_TEST(test_led_builtin_pin_number);
pinMode(LED_BUILTIN, OUTPUT);
}
uint8_t i = 0;
uint8_t max_blinks = 5;
void loop() {
if (i < max_blinks)
{
RUN_TEST(test_led_state_high);
delay(500);
RUN_TEST(test_led_state_low);
delay(500);
i++;
}
else if (i == max_blinks) {
UNITY_END(); // stop unit testing
}
}
#endif
Test results
~~~~~~~~~~~~
.. code::
> platformio test -e nodemcu --verbose
PlatformIO Plus (https://pioplus.com) v0.1.0
Verbose mode can be enabled via `-v, --verbose` option
Collected 1 items
============================== [test::*] Building... (1/3) ==============================
[Wed Sep 7 15:16:55 2016] Processing nodemcu (platform: espressif8266, board: nodemcu, framework: arduino)
----------------------------------------------------------------------------------------------------------------------------------------------------------------
Verbose mode can be enabled via `-v, --verbose` option
Collected 34 compatible libraries
Looking for dependencies...
Project does not have dependencies
Compiling .pioenvs/nodemcu/src/main.o
Compiling .pioenvs/nodemcu/test/output_export.o
Compiling .pioenvs/nodemcu/test/test_main.o
Compiling .pioenvs/nodemcu/UnityTestLib/unity.o
Archiving .pioenvs/nodemcu/libFrameworkArduinoVariant.a
Indexing .pioenvs/nodemcu/libFrameworkArduinoVariant.a
Compiling .pioenvs/nodemcu/FrameworkArduino/Esp.o
Compiling .pioenvs/nodemcu/FrameworkArduino/FS.o
Compiling .pioenvs/nodemcu/FrameworkArduino/HardwareSerial.o
Compiling .pioenvs/nodemcu/FrameworkArduino/IPAddress.o
Archiving .pioenvs/nodemcu/libUnityTestLib.a
Indexing .pioenvs/nodemcu/libUnityTestLib.a
Compiling .pioenvs/nodemcu/FrameworkArduino/MD5Builder.o
...
Compiling .pioenvs/nodemcu/FrameworkArduino/umm_malloc/umm_malloc.o
Archiving .pioenvs/nodemcu/libFrameworkArduino.a
Indexing .pioenvs/nodemcu/libFrameworkArduino.a
Linking .pioenvs/nodemcu/firmware.elf
Calculating size .pioenvs/nodemcu/firmware.elf
text data bss dec hex filename
223500 2408 29536 255444 3e5d4 .pioenvs/nodemcu/firmware.elf
Building .pioenvs/nodemcu/firmware.bin
============================== [test::*] Uploading... (2/3) ==============================
[Wed Sep 7 15:17:01 2016] Processing nodemcu (platform: espressif8266, board: nodemcu, framework: arduino)
----------------------------------------------------------------------------------------------------------------------------------------------------------------
Verbose mode can be enabled via `-v, --verbose` option
Collected 34 compatible libraries
Looking for dependencies...
Project does not have dependencies
Linking .pioenvs/nodemcu/firmware.elf
Checking program size .pioenvs/nodemcu/firmware.elf
text data bss dec hex filename
223500 2408 29536 255444 3e5d4 .pioenvs/nodemcu/firmware.elf
Calculating size .pioenvs/nodemcu/firmware.elf
text data bss dec hex filename
223500 2408 29536 255444 3e5d4 .pioenvs/nodemcu/firmware.elf
Looking for upload port...
Auto-detected: /dev/cu.SLAB_USBtoUART
Uploading .pioenvs/nodemcu/firmware.bin
Uploading 230064 bytes from .pioenvs/nodemcu/firmware.bin to flash at 0x00000000
................................................................................ [ 35% ]
................................................................................ [ 71% ]
................................................................. [ 100% ]
=============================== [test::*] Testing... (3/3) ===============================
If you don't see any output for the first 10 secs, please reset board (press reset button)
test/test_main.cpp:41:test_led_state_high [PASSED]
test/test_main.cpp:43:test_led_state_low [PASSED]
test/test_main.cpp:41:test_led_state_high [PASSED]
test/test_main.cpp:43:test_led_state_low [PASSED]
test/test_main.cpp:41:test_led_state_high [PASSED]
test/test_main.cpp:43:test_led_state_low [PASSED]
test/test_main.cpp:41:test_led_state_high [PASSED]
test/test_main.cpp:43:test_led_state_low [PASSED]
-----------------------
11 Tests 1 Failures 0 Ignored
===================================== [TEST SUMMARY] =====================================
test:*/env:nodemcu [PASSED]
================================ [PASSED] Took 38.15 seconds ================================
Examples
--------
* `Embedded: Wiring Blink <https://github.com/platformio/platformio-examples/tree/develop/unit-testing/wiring-blink>`_
* `Local & Embedded: Calculator <https://github.com/platformio/platformio-examples/tree/develop/unit-testing/calculator>`_
For the other examples and source code please follow to
`PlatformIO Unit Testing Examples <https://github.com/platformio/platformio-examples/tree/develop/unit-testing>`_ repository.
View Git Blame Copy Permalink