forked from platformio/platformio-core
Remote Unit Testing
This commit is contained in:
Ivan Kravets
@ -9,6 +9,9 @@
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
|
||||
.. |PIOUTE| replace:: **PlatformIO Unit Testing Engine**
|
||||
.. |PIOUTF| replace:: *PlatformIO Unit Testing Framework*
|
||||
|
||||
.. _unit_testing:
|
||||
|
||||
Unit Testing
|
||||
@ -22,30 +25,53 @@ of one or more MCU program modules together with associated control data,
|
||||
usage procedures, and operating procedures, are tested to determine whether
|
||||
they are fit for use. Unit testing finds problems early in the development cycle.
|
||||
|
||||
PlatformIO Testing Engine supports 2 different test types:
|
||||
|PIOUTE| supports 2 different test types:
|
||||
|
||||
1. **Local Test** - *[host, native]*, process test on the host machine
|
||||
using :ref:`platform_native`.
|
||||
2. **Embedded Test** - *[remote, hardware]*, prepare special firmware for the
|
||||
target device and upload it. Run test on the embedded device and collect
|
||||
results. Process test results on the host machine.
|
||||
1. **Desktop Test**. PlatformIO wraps test and main program (from
|
||||
:ref:`projectconf_pio_src_dir`) with own |PIOUTF|, builds final program
|
||||
using :ref:`platform_native` and run test on a local host machine (desktop).
|
||||
|
||||
You will be able to run the same test on the different target devices
|
||||
(:ref:`embedded_boards`).
|
||||
.. note::
|
||||
PlatformIO does not install any toolchains automatically for
|
||||
:ref:`platform_native` and requires ``GCC`` toolchain to be installed
|
||||
on your local machine.
|
||||
Please open Terminal and check that ``gcc`` command is installed.
|
||||
|
||||
PlatformIO Testing Engine consists of:
|
||||
2. **Embedded Test**. PlatformIO wraps test and main firmware (from
|
||||
:ref:`projectconf_pio_src_dir`) with own |PIOUTF|, builds special firmware
|
||||
for a target device and upload it. After uploading, PlatformIO connects
|
||||
to embedded device (board) using :ref:`projectconf_test_port` , starts
|
||||
test, collects results and shows test results on the local host machine.
|
||||
|
||||
* Project builder
|
||||
* Test builder
|
||||
* Firmware uploader (is used only for embedded test)
|
||||
* Test processor
|
||||
Currently, |PIOUTE| supports these frameworks:
|
||||
|
||||
There is special command :ref:`cmd_test` to run tests from PlatformIO Project.
|
||||
It allows to process specific environments or to ignore some tests using
|
||||
"Glob patterns".
|
||||
* :ref:`framework_arduino`
|
||||
* :ref:`framework_energia`
|
||||
* :ref:`framework_mbed`.
|
||||
|
||||
Also, is possible to ignore some tests for specific environment using
|
||||
:ref:`projectconf_test_ignore` option from :ref:`projectconf`.
|
||||
.. note::
|
||||
Please note that |PIOUTF| uses Serial/UART as communication interface
|
||||
between PlatformIO Unit Test Engine and target device. If you use
|
||||
``Serial`` in your project, please wrap/hide Serial-based blocks with
|
||||
``#ifndef UNIT_TEST`` macro.
|
||||
|
||||
|
||||
There are 2 options how to run tests:
|
||||
|
||||
1. **Local**. Allows to run tests on local host machine or on the target devices
|
||||
(boards) that are directly connected to this machine. In this case, need to
|
||||
use :ref:`cmd_test` command.
|
||||
|
||||
2. **Remote**. Allows to run tests on remote machine or remote target device
|
||||
(board) without any dependencies to OS software, extra software, SSH, VPN
|
||||
or opening network ports. Remote Unit Testing works in pair with
|
||||
:ref:`pio_remote`. In this case, need to use special command
|
||||
:ref:`cmd_remote_test`.
|
||||
|
||||
Both commands allow to process specific environments or to ignore some tests
|
||||
using "Glob patterns". Also, you will be able to ignore some tests for
|
||||
specific environment using :ref:`projectconf_test_ignore` option
|
||||
from :ref:`projectconf`.
|
||||
|
||||
.. contents::
|
||||
|
||||
@ -62,13 +88,13 @@ Demo of `Local & Embedded: Calculator <https://github.com/platformio/platformio-
|
||||
Design
|
||||
------
|
||||
|
||||
PlatformIO Testing Engine design is based on a few isolated components:
|
||||
|PIOUTE| design is based on a few isolated components:
|
||||
|
||||
1. **Main program**. Contains the independent modules, procedures,
|
||||
1. **Main Program**. Contains the independent modules, procedures,
|
||||
functions or methods that will be the target candidates (TC) for testing.
|
||||
2. **Unit test**. This a small independent program that is intended to
|
||||
2. **Unit Test**. This a small independent program that is intended to
|
||||
re-use TC from the main program and apply tests for them.
|
||||
3. **Test processor**. The set of approaches and tools that will be used
|
||||
3. **Test Processor**. The set of approaches and tools that will be used
|
||||
to apply test for the environments from :ref:`projectconf`.
|
||||
|
||||
Workflow
|
||||
@ -153,7 +179,7 @@ Workflow
|
||||
6. Place test to ``test`` directory. If you have more than one test, split them
|
||||
into sub-folders. For example, ``test/test_1/*.[c,cpp,h]``,
|
||||
``test_N/*.[c,cpp,h]``, etc. If no such directory in ``test`` folder, then
|
||||
PlatformIO Testing Engine will treat the source code of ``test`` folder
|
||||
|PIOUTE| will treat the source code of ``test`` folder
|
||||
as SINGLE test.
|
||||
7. Run tests using :ref:`cmd_test` command.
|
||||
|
||||
@ -248,7 +274,9 @@ User Guide (CLI)
|
||||
.. toctree::
|
||||
:maxdepth: 3
|
||||
|
||||
platformio account <../userguide/account/index>
|
||||
platformio test <../userguide/cmd_test>
|
||||
platformio remote test <../userguide/remote/cmd_test>
|
||||
|
||||
--------------
|
||||
|
||||
|
||||
@ -648,7 +648,7 @@ This option is used by "uploader" tool when sending firmware to board via
|
||||
If ``upload_port`` isn't specified, then *PlatformIO* will try to detect it
|
||||
automatically.
|
||||
|
||||
To print all available serial ports use :ref:`cmd_device` command.
|
||||
To print all available serial ports use :ref:`cmd_device_list` command.
|
||||
|
||||
This option can be set by global environment variable
|
||||
:envvar:`PLATFORMIO_UPLOAD_PORT`.
|
||||
@ -841,9 +841,9 @@ Test options
|
||||
.. seealso::
|
||||
Please make sure to read :ref:`unit_testing` guide first.
|
||||
|
||||
Ignore tests where the name matches specified patterns. Multiple names are
|
||||
allowed. Please separate them using comma+space ", ". Also, you can
|
||||
ignore some tests using :option:`platformio test --ignore` command.
|
||||
Ignore :ref:`unit_testing` tests where the name matches specified patterns.
|
||||
Multiple names are allowed. Please separate them using comma+space ", ". Also,
|
||||
you can ignore some tests using :option:`platformio test --ignore` command.
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
@ -870,6 +870,22 @@ ignore some tests using :option:`platformio test --ignore` command.
|
||||
[env:myenv]
|
||||
test_ignore = footest, bartest_*, test[13]
|
||||
|
||||
.. _projectconf_test_port:
|
||||
|
||||
``test_port``
|
||||
^^^^^^^^^^^^^
|
||||
|
||||
This option is used as communication interface (Serial/UART) between PlatformIO
|
||||
:ref:`unit_testing` Engine and target device. For example,
|
||||
|
||||
* ``/dev/ttyUSB0`` - Unix-based OS
|
||||
* ``COM3`` - Windows OS
|
||||
|
||||
If ``test_port`` isn't specified, then *PlatformIO* will try to detect it
|
||||
automatically.
|
||||
|
||||
To print all available serial ports use :ref:`cmd_device_list` command.
|
||||
|
||||
|
||||
Advanced options
|
||||
~~~~~~~~~~~~~~~~
|
||||
|
||||
@ -14,7 +14,7 @@
|
||||
platformio test
|
||||
===============
|
||||
|
||||
.. versionadded:: 3.0
|
||||
Helper command for local :ref:`unit_testing`.
|
||||
|
||||
.. contents::
|
||||
|
||||
@ -28,7 +28,7 @@ Usage
|
||||
Description
|
||||
-----------
|
||||
|
||||
Run tests from PlatformIO based project. More details about PlatformIO
|
||||
Run locally tests from PlatformIO based project. More details about PlatformIO
|
||||
:ref:`unit_testing`.
|
||||
|
||||
This command allows you to apply the tests for the environments specified
|
||||
@ -75,17 +75,36 @@ For example, ``platformio test --ignore "mytest*" -i "test[13]"``
|
||||
.. option::
|
||||
--upload-port
|
||||
|
||||
Upload port of embedded board. To print all available ports use
|
||||
:ref:`cmd_device` command.
|
||||
A port that is intended for firmware uploading. To list available ports
|
||||
please use :ref:`cmd_device_list` command.
|
||||
|
||||
If upload port is not specified, PlatformIO will try to detect it automatically.
|
||||
|
||||
.. option::
|
||||
--test-port
|
||||
|
||||
A Serial/UART port that PlatformIO uses as communication interface between
|
||||
PlatformIO Unit Test Engine and target device. To list available ports
|
||||
please use :ref:`cmd_device_list` command.
|
||||
|
||||
If test port is not specified, PlatformIO will try to detect it automatically.
|
||||
|
||||
.. option::
|
||||
-d, --project-dir
|
||||
|
||||
Specify the path to project directory. By default, ``--project-dir`` is equal
|
||||
to current working directory (``CWD``).
|
||||
|
||||
.. option::
|
||||
--without-building
|
||||
|
||||
Skip building stage.
|
||||
|
||||
.. option::
|
||||
--without-uploading
|
||||
|
||||
Skip uploading stage
|
||||
|
||||
.. option::
|
||||
-v, --verbose
|
||||
|
||||
|
||||
134
docs/userguide/remote/cmd_test.rst
Normal file
134
docs/userguide/remote/cmd_test.rst
Normal file
@ -0,0 +1,134 @@
|
||||
.. Copyright 2014-present PlatformIO <contact@platformio.org>
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
|
||||
.. _cmd_remote_test:
|
||||
|
||||
platformio remote test
|
||||
======================
|
||||
|
||||
Helper command for remote :ref:`unit_testing`.
|
||||
|
||||
.. contents::
|
||||
|
||||
Usage
|
||||
-----
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
platformio remote test [OPTIONS]
|
||||
|
||||
# run tests on specified PIO Remote Agent
|
||||
platformio remote --agent NAME test [OPTIONS]
|
||||
|
||||
Description
|
||||
-----------
|
||||
|
||||
Run remotely tests from PlatformIO based project. More details about PlatformIO
|
||||
:ref:`unit_testing`.
|
||||
|
||||
This command allows you to apply the tests for the environments specified
|
||||
in :ref:`projectconf`.
|
||||
|
||||
Options
|
||||
-------
|
||||
|
||||
.. program:: platformio remote test
|
||||
|
||||
.. option::
|
||||
-e, --environment
|
||||
|
||||
Process specified environments. More details :option:`platformio run --environment`
|
||||
|
||||
.. option::
|
||||
-i, --ignore
|
||||
|
||||
Ignore tests where the name matches specified patterns. More than one
|
||||
pattern is allowed. If you need to ignore some tests for the specific
|
||||
environment, please take a look at :ref:`projectconf_test_ignore` option from
|
||||
:ref:`projectconf`.
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
|
||||
* - Pattern
|
||||
- Meaning
|
||||
|
||||
* - ``*``
|
||||
- matches everything
|
||||
|
||||
* - ``?``
|
||||
- matches any single character
|
||||
|
||||
* - ``[seq]``
|
||||
- matches any character in seq
|
||||
|
||||
* - ``[!seq]``
|
||||
- matches any character not in seq
|
||||
|
||||
For example, ``platformio remote test --ignore "mytest*" -i "test[13]"``
|
||||
|
||||
.. option::
|
||||
--upload-port
|
||||
|
||||
A port that is intended for firmware uploading. To list available ports
|
||||
please use :ref:`cmd_device_list` command.
|
||||
|
||||
If upload port is not specified, PlatformIO will try to detect it automatically.
|
||||
|
||||
.. option::
|
||||
--test-port
|
||||
|
||||
A Serial/UART port that PlatformIO uses as communication interface between
|
||||
PlatformIO Unit Test Engine and target device. To list available ports
|
||||
please use :ref:`cmd_device_list` command.
|
||||
|
||||
If test port is not specified, PlatformIO will try to detect it automatically.
|
||||
|
||||
.. option::
|
||||
-d, --project-dir
|
||||
|
||||
Specify the path to project directory. By default, ``--project-dir`` is equal
|
||||
to current working directory (``CWD``).
|
||||
|
||||
.. option::
|
||||
-r, --build-remotely
|
||||
|
||||
By default, :ref:`pio_remote` builds project on the local machine and deploy
|
||||
final testing firmware Over-The-Air (OTA) to remote device.
|
||||
|
||||
If you need to build project on remote machine, please use
|
||||
:option:`platformio remote test --build-remotely` option. In this case,
|
||||
:ref:`pio_remote` will automatically deploy your project to remote machine,
|
||||
install required toolchains, frameworks, SDKs, etc and process tests.
|
||||
|
||||
|
||||
.. option::
|
||||
--without-building
|
||||
|
||||
Skip building stage.
|
||||
|
||||
.. option::
|
||||
--without-uploading
|
||||
|
||||
Skip uploading stage
|
||||
|
||||
.. option::
|
||||
-v, --verbose
|
||||
|
||||
Shows detailed information when processing environments.
|
||||
|
||||
This option can be set globally using :ref:`setting_force_verbose` setting
|
||||
or by environment variable :envvar:`PLATFORMIO_SETTING_FORCE_VERBOSE`.
|
||||
|
||||
Examples
|
||||
--------
|
||||
|
||||
For the examples please follow to :ref:`unit_testing` page.
|
||||
@ -45,3 +45,4 @@ To print all available commands and options use:
|
||||
cmd_agent
|
||||
cmd_device
|
||||
cmd_run
|
||||
cmd_test
|
||||
|
||||
2
examples
2
examples
Submodule examples updated: 15e215e33e...5ca4b103ab
@ -214,8 +214,11 @@ class LibBuilderBase(object):
|
||||
for item in self.dependencies:
|
||||
skip = False
|
||||
for key in ("platforms", "frameworks"):
|
||||
if (key in item and not self.items_in_list(
|
||||
self.env["PIO" + key.upper()[:-1]], item[key])):
|
||||
env_key = "PIO" + key.upper()[:-1]
|
||||
if env_key not in self.env:
|
||||
continue
|
||||
if (key in item and
|
||||
not self.items_in_list(self.env[env_key], item[key])):
|
||||
if verbose:
|
||||
sys.stderr.write("Skip %s incompatible dependency %s\n"
|
||||
% (key[:-1], item))
|
||||
@ -507,7 +510,7 @@ class PlatformIOLibBuilder(LibBuilderBase):
|
||||
isdir(join(self.path, "utility"))):
|
||||
inc_dirs.append(join(self.path, "utility"))
|
||||
|
||||
for path in self.env['CPPPATH']:
|
||||
for path in self.env.get("CPPPATH", []):
|
||||
if path not in self.envorigin['CPPPATH']:
|
||||
inc_dirs.append(self.env.subst(path))
|
||||
return inc_dirs
|
||||
|
||||
@ -51,7 +51,7 @@ def remote_agent_list():
|
||||
pioplus_call(sys.argv[1:])
|
||||
|
||||
|
||||
@cli.command("run", short_help="Process project environments")
|
||||
@cli.command("run", short_help="Process project environments remotely")
|
||||
@click.option("-e", "--environment", multiple=True)
|
||||
@click.option("-t", "--target", multiple=True)
|
||||
@click.option("--upload-port")
|
||||
@ -65,26 +65,49 @@ def remote_agent_list():
|
||||
dir_okay=True,
|
||||
writable=True,
|
||||
resolve_path=True))
|
||||
@click.option("--disable-auto-clean", is_flag=True)
|
||||
@click.option("-r", "--build-remotely", is_flag=True)
|
||||
@click.option("-s", "--silent", is_flag=True)
|
||||
@click.option("-v", "--verbose", is_flag=True)
|
||||
@click.option("-r", "--build-remotely", is_flag=True)
|
||||
@click.option("--disable-auto-clean", is_flag=True)
|
||||
def remote_run(**kwargs):
|
||||
pioplus_call(sys.argv[1:])
|
||||
|
||||
|
||||
@cli.group("device", short_help="Monitor device or list existing")
|
||||
@cli.command("test", short_help="Remote Unit Testing")
|
||||
@click.option("--environment", "-e", multiple=True, metavar="<environment>")
|
||||
@click.option("--ignore", "-i", multiple=True, metavar="<pattern>")
|
||||
@click.option("--upload-port")
|
||||
@click.option("--test-port")
|
||||
@click.option(
|
||||
"-d",
|
||||
"--project-dir",
|
||||
default=getcwd,
|
||||
type=click.Path(
|
||||
exists=True,
|
||||
file_okay=False,
|
||||
dir_okay=True,
|
||||
writable=True,
|
||||
resolve_path=True))
|
||||
@click.option("-r", "--build-remotely", is_flag=True)
|
||||
@click.option("--without-building", is_flag=True)
|
||||
@click.option("--without-uploading", is_flag=True)
|
||||
@click.option("--verbose", "-v", is_flag=True)
|
||||
def remote_test(**kwargs):
|
||||
pioplus_call(sys.argv[1:])
|
||||
|
||||
|
||||
@cli.group("device", short_help="Monitor remote device or list existing")
|
||||
def remote_device():
|
||||
pass
|
||||
|
||||
|
||||
@remote_device.command("list", short_help="List devices")
|
||||
@remote_device.command("list", short_help="List remote devices")
|
||||
@click.option("--json-output", is_flag=True)
|
||||
def device_list(json_output):
|
||||
pioplus_call(sys.argv[1:])
|
||||
|
||||
|
||||
@remote_device.command("monitor", short_help="Monitor device (Serial)")
|
||||
@remote_device.command("monitor", short_help="Monitor remote device")
|
||||
@click.option("--port", "-p", help="Port, a number or a device name")
|
||||
@click.option(
|
||||
"--baud", "-b", type=int, default=9600, help="Set baud rate, default=9600")
|
||||
|
||||
@ -126,7 +126,7 @@ class EnvironmentProcessor(object):
|
||||
"upload_port", "upload_protocol", "upload_speed", "upload_flags",
|
||||
"upload_resetmethod", "lib_install", "lib_deps", "lib_force",
|
||||
"lib_ignore", "lib_extra_dirs", "lib_ldf_mode", "lib_compat_mode",
|
||||
"test_ignore", "piotest")
|
||||
"test_ignore", "test_port", "piotest")
|
||||
|
||||
REMAPED_OPTIONS = {"framework": "pioframework", "platform": "pioplatform"}
|
||||
|
||||
|
||||
@ -20,10 +20,11 @@ import click
|
||||
from platformio.pioplus import pioplus_call
|
||||
|
||||
|
||||
@click.command("test", short_help="Unit Testing")
|
||||
@click.command("test", short_help="Local Unit Testing")
|
||||
@click.option("--environment", "-e", multiple=True, metavar="<environment>")
|
||||
@click.option("--ignore", "-i", multiple=True, metavar="<pattern>")
|
||||
@click.option("--upload-port", metavar="<upload port>")
|
||||
@click.option("--upload-port")
|
||||
@click.option("--test-port")
|
||||
@click.option(
|
||||
"-d",
|
||||
"--project-dir",
|
||||
@ -34,6 +35,8 @@ from platformio.pioplus import pioplus_call
|
||||
dir_okay=True,
|
||||
writable=True,
|
||||
resolve_path=True))
|
||||
@click.option("--without-building", is_flag=True)
|
||||
@click.option("--without-uploading", is_flag=True)
|
||||
@click.option("--verbose", "-v", is_flag=True)
|
||||
def cli(*args, **kwargs): # pylint: disable=unused-argument
|
||||
pioplus_call(sys.argv[1:])
|
||||
|
||||
Reference in New Issue
Block a user