forked from platformio/platformio-core
Remote Unit Testing
This commit is contained in:
Ivan Kravets committed
1 parent
db4dbeeca7
commit
9b786ba8c7
10 files changed
+271
-44
No files matched your search
+51
-23
@@ -9,6 +9,9 @@
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
|
||||
.. |PIOUTE| replace:: **PlatformIO Unit Testing Engine**
|
||||
.. |PIOUTF| replace:: *PlatformIO Unit Testing Framework*
|
||||
|
||||
.. _unit_testing:
|
||||
|
||||
Unit Testing
|
||||
@@ -22,30 +25,53 @@ of one or more MCU program modules together with associated control data,
|
||||
usage procedures, and operating procedures, are tested to determine whether
|
||||
they are fit for use. Unit testing finds problems early in the development cycle.
|
||||
|
||||
PlatformIO Testing Engine supports 2 different test types:
|
||||
|PIOUTE| supports 2 different test types:
|
||||
|
||||
1. **Local Test** - *[host, native]*, process test on the host machine
|
||||
using :ref:`platform_native`.
|
||||
2. **Embedded Test** - *[remote, hardware]*, prepare special firmware for the
|
||||
target device and upload it. Run test on the embedded device and collect
|
||||
results. Process test results on the host machine.
|
||||
1. **Desktop Test**. PlatformIO wraps test and main program (from
|
||||
:ref:`projectconf_pio_src_dir`) with own |PIOUTF|, builds final program
|
||||
using :ref:`platform_native` and run test on a local host machine (desktop).
|
||||
|
||||
You will be able to run the same test on the different target devices
|
||||
(:ref:`embedded_boards`).
|
||||
.. note::
|
||||
PlatformIO does not install any toolchains automatically for
|
||||
:ref:`platform_native` and requires ``GCC`` toolchain to be installed
|
||||
on your local machine.
|
||||
Please open Terminal and check that ``gcc`` command is installed.
|
||||
|
||||
PlatformIO Testing Engine consists of:
|
||||
2. **Embedded Test**. PlatformIO wraps test and main firmware (from
|
||||
:ref:`projectconf_pio_src_dir`) with own |PIOUTF|, builds special firmware
|
||||
for a target device and upload it. After uploading, PlatformIO connects
|
||||
to embedded device (board) using :ref:`projectconf_test_port` , starts
|
||||
test, collects results and shows test results on the local host machine.
|
||||
|
||||
* Project builder
|
||||
* Test builder
|
||||
* Firmware uploader (is used only for embedded test)
|
||||
* Test processor
|
||||
Currently, |PIOUTE| supports these frameworks:
|
||||
|
||||
There is special command :ref:`cmd_test` to run tests from PlatformIO Project.
|
||||
It allows to process specific environments or to ignore some tests using
|
||||
"Glob patterns".
|
||||
* :ref:`framework_arduino`
|
||||
* :ref:`framework_energia`
|
||||
* :ref:`framework_mbed`.
|
||||
|
||||
Also, is possible to ignore some tests for specific environment using
|
||||
:ref:`projectconf_test_ignore` option from :ref:`projectconf`.
|
||||
.. note::
|
||||
Please note that |PIOUTF| uses Serial/UART as communication interface
|
||||
between PlatformIO Unit Test Engine and target device. If you use
|
||||
``Serial`` in your project, please wrap/hide Serial-based blocks with
|
||||
``#ifndef UNIT_TEST`` macro.
|
||||
|
||||
|
||||
There are 2 options how to run tests:
|
||||
|
||||
1. **Local**. Allows to run tests on local host machine or on the target devices
|
||||
(boards) that are directly connected to this machine. In this case, need to
|
||||
use :ref:`cmd_test` command.
|
||||
|
||||
2. **Remote**. Allows to run tests on remote machine or remote target device
|
||||
(board) without any dependencies to OS software, extra software, SSH, VPN
|
||||
or opening network ports. Remote Unit Testing works in pair with
|
||||
:ref:`pio_remote`. In this case, need to use special command
|
||||
:ref:`cmd_remote_test`.
|
||||
|
||||
Both commands allow to process specific environments or to ignore some tests
|
||||
using "Glob patterns". Also, you will be able to ignore some tests for
|
||||
specific environment using :ref:`projectconf_test_ignore` option
|
||||
from :ref:`projectconf`.
|
||||
|
||||
.. contents::
|
||||
|
||||
@@ -62,13 +88,13 @@ Demo of `Local & Embedded: Calculator <https://github.com/platformio/platformio-
|
||||
Design
|
||||
------
|
||||
|
||||
PlatformIO Testing Engine design is based on a few isolated components:
|
||||
|PIOUTE| design is based on a few isolated components:
|
||||
|
||||
1. **Main program**. Contains the independent modules, procedures,
|
||||
1. **Main Program**. Contains the independent modules, procedures,
|
||||
functions or methods that will be the target candidates (TC) for testing.
|
||||
2. **Unit test**. This a small independent program that is intended to
|
||||
2. **Unit Test**. This a small independent program that is intended to
|
||||
re-use TC from the main program and apply tests for them.
|
||||
3. **Test processor**. The set of approaches and tools that will be used
|
||||
3. **Test Processor**. The set of approaches and tools that will be used
|
||||
to apply test for the environments from :ref:`projectconf`.
|
||||
|
||||
Workflow
|
||||
@@ -153,7 +179,7 @@ Workflow
|
||||
6. Place test to ``test`` directory. If you have more than one test, split them
|
||||
into sub-folders. For example, ``test/test_1/*.[c,cpp,h]``,
|
||||
``test_N/*.[c,cpp,h]``, etc. If no such directory in ``test`` folder, then
|
||||
PlatformIO Testing Engine will treat the source code of ``test`` folder
|
||||
|PIOUTE| will treat the source code of ``test`` folder
|
||||
as SINGLE test.
|
||||
7. Run tests using :ref:`cmd_test` command.
|
||||
|
||||
@@ -248,7 +274,9 @@ User Guide (CLI)
|
||||
.. toctree::
|
||||
:maxdepth: 3
|
||||
|
||||
platformio account <../userguide/account/index>
|
||||
platformio test <../userguide/cmd_test>
|
||||
platformio remote test <../userguide/remote/cmd_test>
|
||||
|
||||
--------------
|
||||
|
||||
|
||||
+20
-4
@@ -648,7 +648,7 @@ This option is used by "uploader" tool when sending firmware to board via
|
||||
If ``upload_port`` isn't specified, then *PlatformIO* will try to detect it
|
||||
automatically.
|
||||
|
||||
To print all available serial ports use :ref:`cmd_device` command.
|
||||
To print all available serial ports use :ref:`cmd_device_list` command.
|
||||
|
||||
This option can be set by global environment variable
|
||||
:envvar:`PLATFORMIO_UPLOAD_PORT`.
|
||||
@@ -841,9 +841,9 @@ Test options
|
||||
.. seealso::
|
||||
Please make sure to read :ref:`unit_testing` guide first.
|
||||
|
||||
Ignore tests where the name matches specified patterns. Multiple names are
|
||||
allowed. Please separate them using comma+space ", ". Also, you can
|
||||
ignore some tests using :option:`platformio test --ignore` command.
|
||||
Ignore :ref:`unit_testing` tests where the name matches specified patterns.
|
||||
Multiple names are allowed. Please separate them using comma+space ", ". Also,
|
||||
you can ignore some tests using :option:`platformio test --ignore` command.
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
@@ -870,6 +870,22 @@ ignore some tests using :option:`platformio test --ignore` command.
|
||||
[env:myenv]
|
||||
test_ignore = footest, bartest_*, test[13]
|
||||
|
||||
.. _projectconf_test_port:
|
||||
|
||||
``test_port``
|
||||
^^^^^^^^^^^^^
|
||||
|
||||
This option is used as communication interface (Serial/UART) between PlatformIO
|
||||
:ref:`unit_testing` Engine and target device. For example,
|
||||
|
||||
* ``/dev/ttyUSB0`` - Unix-based OS
|
||||
* ``COM3`` - Windows OS
|
||||
|
||||
If ``test_port`` isn't specified, then *PlatformIO* will try to detect it
|
||||
automatically.
|
||||
|
||||
To print all available serial ports use :ref:`cmd_device_list` command.
|
||||
|
||||
|
||||
Advanced options
|
||||
~~~~~~~~~~~~~~~~
|
||||
|
||||
@@ -14,7 +14,7 @@
|
||||
platformio test
|
||||
===============
|
||||
|
||||
.. versionadded:: 3.0
|
||||
Helper command for local :ref:`unit_testing`.
|
||||
|
||||
.. contents::
|
||||
|
||||
@@ -28,7 +28,7 @@ Usage
|
||||
Description
|
||||
-----------
|
||||
|
||||
Run tests from PlatformIO based project. More details about PlatformIO
|
||||
Run locally tests from PlatformIO based project. More details about PlatformIO
|
||||
:ref:`unit_testing`.
|
||||
|
||||
This command allows you to apply the tests for the environments specified
|
||||
@@ -75,17 +75,36 @@ For example, ``platformio test --ignore "mytest*" -i "test[13]"``
|
||||
.. option::
|
||||
--upload-port
|
||||
|
||||
Upload port of embedded board. To print all available ports use
|
||||
:ref:`cmd_device` command.
|
||||
A port that is intended for firmware uploading. To list available ports
|
||||
please use :ref:`cmd_device_list` command.
|
||||
|
||||
If upload port is not specified, PlatformIO will try to detect it automatically.
|
||||
|
||||
.. option::
|
||||
--test-port
|
||||
|
||||
A Serial/UART port that PlatformIO uses as communication interface between
|
||||
PlatformIO Unit Test Engine and target device. To list available ports
|
||||
please use :ref:`cmd_device_list` command.
|
||||
|
||||
If test port is not specified, PlatformIO will try to detect it automatically.
|
||||
|
||||
.. option::
|
||||
-d, --project-dir
|
||||
|
||||
Specify the path to project directory. By default, ``--project-dir`` is equal
|
||||
to current working directory (``CWD``).
|
||||
|
||||
.. option::
|
||||
--without-building
|
||||
|
||||
Skip building stage.
|
||||
|
||||
.. option::
|
||||
--without-uploading
|
||||
|
||||
Skip uploading stage
|
||||
|
||||
.. option::
|
||||
-v, --verbose
|
||||
|
||||
|
||||
@@ -0,0 +1,134 @@
|
||||
.. Copyright 2014-present PlatformIO <contact@platformio.org>
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
|
||||
.. _cmd_remote_test:
|
||||
|
||||
platformio remote test
|
||||
======================
|
||||
|
||||
Helper command for remote :ref:`unit_testing`.
|
||||
|
||||
.. contents::
|
||||
|
||||
Usage
|
||||
-----
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
platformio remote test [OPTIONS]
|
||||
|
||||
# run tests on specified PIO Remote Agent
|
||||
platformio remote --agent NAME test [OPTIONS]
|
||||
|
||||
Description
|
||||
-----------
|
||||
|
||||
Run remotely tests from PlatformIO based project. More details about PlatformIO
|
||||
:ref:`unit_testing`.
|
||||
|
||||
This command allows you to apply the tests for the environments specified
|
||||
in :ref:`projectconf`.
|
||||
|
||||
Options
|
||||
-------
|
||||
|
||||
.. program:: platformio remote test
|
||||
|
||||
.. option::
|
||||
-e, --environment
|
||||
|
||||
Process specified environments. More details :option:`platformio run --environment`
|
||||
|
||||
.. option::
|
||||
-i, --ignore
|
||||
|
||||
Ignore tests where the name matches specified patterns. More than one
|
||||
pattern is allowed. If you need to ignore some tests for the specific
|
||||
environment, please take a look at :ref:`projectconf_test_ignore` option from
|
||||
:ref:`projectconf`.
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
|
||||
* - Pattern
|
||||
- Meaning
|
||||
|
||||
* - ``*``
|
||||
- matches everything
|
||||
|
||||
* - ``?``
|
||||
- matches any single character
|
||||
|
||||
* - ``[seq]``
|
||||
- matches any character in seq
|
||||
|
||||
* - ``[!seq]``
|
||||
- matches any character not in seq
|
||||
|
||||
For example, ``platformio remote test --ignore "mytest*" -i "test[13]"``
|
||||
|
||||
.. option::
|
||||
--upload-port
|
||||
|
||||
A port that is intended for firmware uploading. To list available ports
|
||||
please use :ref:`cmd_device_list` command.
|
||||
|
||||
If upload port is not specified, PlatformIO will try to detect it automatically.
|
||||
|
||||
.. option::
|
||||
--test-port
|
||||
|
||||
A Serial/UART port that PlatformIO uses as communication interface between
|
||||
PlatformIO Unit Test Engine and target device. To list available ports
|
||||
please use :ref:`cmd_device_list` command.
|
||||
|
||||
If test port is not specified, PlatformIO will try to detect it automatically.
|
||||
|
||||
.. option::
|
||||
-d, --project-dir
|
||||
|
||||
Specify the path to project directory. By default, ``--project-dir`` is equal
|
||||
to current working directory (``CWD``).
|
||||
|
||||
.. option::
|
||||
-r, --build-remotely
|
||||
|
||||
By default, :ref:`pio_remote` builds project on the local machine and deploy
|
||||
final testing firmware Over-The-Air (OTA) to remote device.
|
||||
|
||||
If you need to build project on remote machine, please use
|
||||
:option:`platformio remote test --build-remotely` option. In this case,
|
||||
:ref:`pio_remote` will automatically deploy your project to remote machine,
|
||||
install required toolchains, frameworks, SDKs, etc and process tests.
|
||||
|
||||
|
||||
.. option::
|
||||
--without-building
|
||||
|
||||
Skip building stage.
|
||||
|
||||
.. option::
|
||||
--without-uploading
|
||||
|
||||
Skip uploading stage
|
||||
|
||||
.. option::
|
||||
-v, --verbose
|
||||
|
||||
Shows detailed information when processing environments.
|
||||
|
||||
This option can be set globally using :ref:`setting_force_verbose` setting
|
||||
or by environment variable :envvar:`PLATFORMIO_SETTING_FORCE_VERBOSE`.
|
||||
|
||||
Examples
|
||||
--------
|
||||
|
||||
For the examples please follow to :ref:`unit_testing` page.
|
||||
@@ -45,3 +45,4 @@ To print all available commands and options use:
|
||||
cmd_agent
|
||||
cmd_device
|
||||
cmd_run
|
||||
cmd_test
|
||||
Reference in new issue
Block a user