Remote Unit Testing

docs/plus/unit-testing.rst

@@ -9,6 +9,9 @@
     See the License for the specific language governing permissions and
     limitations under the License.
 
+.. |PIOUTE| replace:: **PlatformIO Unit Testing Engine**
+.. |PIOUTF| replace:: *PlatformIO Unit Testing Framework*
+
 .. _unit_testing:
 
 Unit Testing
@@ -22,30 +25,53 @@ of one or more MCU program modules together with associated control data,
 usage procedures, and operating procedures, are tested to determine whether
 they are fit for use. Unit testing finds problems early in the development cycle.
 
-PlatformIO Testing Engine supports 2 different test types:
+|PIOUTE| supports 2 different test types:
 
-1. **Local Test** - *[host, native]*, process test on the host machine
+1.  **Desktop Test**. PlatformIO wraps test and main program (from
-   using :ref:`platform_native`.
+    :ref:`projectconf_pio_src_dir`) with own |PIOUTF|, builds final program
-2. **Embedded Test** - *[remote, hardware]*, prepare special firmware for the
+    using :ref:`platform_native` and run test on a local host machine (desktop).
-   target device and upload it. Run test on the embedded device and collect
-   results. Process test results on the host machine.
 
-   You will be able to run the same test on the different target devices
+    .. note::
-   (:ref:`embedded_boards`).
+        PlatformIO does not install any toolchains automatically for
+        :ref:`platform_native` and requires ``GCC`` toolchain to be installed
+        on your local machine.
+        Please open Terminal and check that ``gcc`` command is installed.
 
-PlatformIO Testing Engine consists of:
+2.  **Embedded Test**. PlatformIO wraps test and main firmware (from
+    :ref:`projectconf_pio_src_dir`) with own |PIOUTF|, builds special firmware
+    for a target device and upload it. After uploading, PlatformIO connects
+    to embedded device (board) using :ref:`projectconf_test_port` , starts
+    test, collects results and shows test results on the local host machine.
 
-* Project builder
+    Currently, |PIOUTE| supports these frameworks:
-* Test builder
-* Firmware uploader (is used only for embedded test)
-* Test processor
 
-There is special command :ref:`cmd_test` to run tests from PlatformIO Project.
+    * :ref:`framework_arduino`
-It allows to process specific environments or to ignore some tests using
+    * :ref:`framework_energia`
-"Glob patterns".
+    * :ref:`framework_mbed`.
 
-Also, is possible to ignore some tests for specific environment using
+    .. note::
-:ref:`projectconf_test_ignore` option from :ref:`projectconf`.
+        Please note that |PIOUTF| uses Serial/UART as communication interface
+        between PlatformIO Unit Test Engine and target device. If you use
+        ``Serial`` in your project, please wrap/hide Serial-based blocks with
+        ``#ifndef UNIT_TEST`` macro.
+
+
+There are 2 options how to run tests:
+
+1.  **Local**. Allows to run tests on local host machine or on the target devices
+    (boards) that are directly connected to this machine. In this case, need to
+    use :ref:`cmd_test` command.
+
+2.  **Remote**. Allows to run tests on remote machine or remote target device
+    (board) without any dependencies to OS software, extra software, SSH, VPN
+    or opening network ports. Remote Unit Testing works in pair with
+    :ref:`pio_remote`. In this case, need to use special command
+    :ref:`cmd_remote_test`.
+
+Both commands allow to process specific environments or to ignore some tests
+using "Glob patterns". Also, you will be able to ignore some tests for
+specific environment using :ref:`projectconf_test_ignore` option
+from :ref:`projectconf`.
 
 .. contents::
 
@@ -62,13 +88,13 @@ Demo of `Local & Embedded: Calculator <https://github.com/platformio/platformio-
 Design
 ------
 
-PlatformIO Testing Engine design is based on a few isolated components:
+|PIOUTE| design is based on a few isolated components:
 
-1. **Main program**. Contains the independent modules, procedures,
+1. **Main Program**. Contains the independent modules, procedures,
    functions or methods that will be the target candidates (TC) for testing.
-2. **Unit test**. This a small independent program that is intended to
+2. **Unit Test**. This a small independent program that is intended to
    re-use TC from the main program and apply tests for them.
-3. **Test processor**. The set of approaches and tools that will be used
+3. **Test Processor**. The set of approaches and tools that will be used
    to apply test for the environments from :ref:`projectconf`.
 
 Workflow
@@ -153,7 +179,7 @@ Workflow
 6. Place test to ``test`` directory. If you have more than one test, split them
    into sub-folders. For example, ``test/test_1/*.[c,cpp,h]``,
    ``test_N/*.[c,cpp,h]``, etc. If no such directory in ``test`` folder, then
-   PlatformIO Testing Engine will treat the source code of ``test`` folder
+   |PIOUTE| will treat the source code of ``test`` folder
    as SINGLE test.
 7. Run tests using :ref:`cmd_test` command.
 
@@ -248,7 +274,9 @@ User Guide (CLI)
 .. toctree::
     :maxdepth: 3
 
+    platformio account <../userguide/account/index>
     platformio test <../userguide/cmd_test>
+    platformio remote test <../userguide/remote/cmd_test>
 
 --------------
 

docs/projectconf.rst

@@ -648,7 +648,7 @@ This option is used by "uploader" tool when sending firmware to board via
 If ``upload_port`` isn't specified, then *PlatformIO* will try to detect it
 automatically.
 
-To print all available serial ports use :ref:`cmd_device` command.
+To print all available serial ports use :ref:`cmd_device_list` command.
 
 This option can be set by global environment variable
 :envvar:`PLATFORMIO_UPLOAD_PORT`.
@@ -841,9 +841,9 @@ Test options
 .. seealso::
     Please make sure to read :ref:`unit_testing` guide first.
 
-Ignore tests where the name matches specified patterns. Multiple names are
+Ignore :ref:`unit_testing` tests where the name matches specified patterns.
-allowed. Please separate them using comma+space ", ". Also, you can
+Multiple names are allowed. Please separate them using comma+space ", ". Also,
-ignore some tests using :option:`platformio test --ignore` command.
+you can ignore some tests using :option:`platformio test --ignore` command.
 
 .. list-table::
     :header-rows:  1
@@ -870,6 +870,22 @@ ignore some tests using :option:`platformio test --ignore` command.
   [env:myenv]
   test_ignore = footest, bartest_*, test[13]
 
+.. _projectconf_test_port:
+
+``test_port``
+^^^^^^^^^^^^^
+
+This option is used as communication interface (Serial/UART) between PlatformIO
+:ref:`unit_testing` Engine and target device. For example,
+
+* ``/dev/ttyUSB0`` - Unix-based OS
+* ``COM3`` - Windows OS
+
+If ``test_port`` isn't specified, then *PlatformIO* will try to detect it
+automatically.
+
+To print all available serial ports use :ref:`cmd_device_list` command.
+
 
 Advanced options
 ~~~~~~~~~~~~~~~~

docs/userguide/cmd_test.rst

@@ -14,7 +14,7 @@
 platformio test
 ===============
 
-.. versionadded:: 3.0
+Helper command for local :ref:`unit_testing`.
 
 .. contents::
 
@@ -28,7 +28,7 @@ Usage
 Description
 -----------
 
-Run tests from PlatformIO based project. More details about PlatformIO
+Run locally tests from PlatformIO based project. More details about PlatformIO
 :ref:`unit_testing`.
 
 This command allows you to apply the tests for the environments specified
@@ -75,17 +75,36 @@ For example, ``platformio test --ignore "mytest*" -i "test[13]"``
 .. option::
     --upload-port
 
-Upload port of embedded board. To print all available ports use
+A port that is intended for firmware uploading. To list available ports
-:ref:`cmd_device` command.
+please use :ref:`cmd_device_list` command.
 
 If upload port is not specified, PlatformIO will try to detect it automatically.
 
+.. option::
+    --test-port
+
+A Serial/UART port that PlatformIO uses as communication interface between
+PlatformIO Unit Test Engine and target device. To list available ports
+please use :ref:`cmd_device_list` command.
+
+If test port is not specified, PlatformIO will try to detect it automatically.
+
 .. option::
     -d, --project-dir
 
 Specify the path to project directory. By default, ``--project-dir`` is equal
 to current working directory (``CWD``).
 
+.. option::
+    --without-building
+
+Skip building stage.
+
+.. option::
+    --without-uploading
+
+Skip uploading stage
+
 .. option::
     -v, --verbose
 

docs/userguide/remote/cmd_test.rst

@@ -0,0 +1,134 @@
+..  Copyright 2014-present PlatformIO <contact@platformio.org>
+    Licensed under the Apache License, Version 2.0 (the "License");
+    you may not use this file except in compliance with the License.
+    You may obtain a copy of the License at
+       http://www.apache.org/licenses/LICENSE-2.0
+    Unless required by applicable law or agreed to in writing, software
+    distributed under the License is distributed on an "AS IS" BASIS,
+    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+    See the License for the specific language governing permissions and
+    limitations under the License.
+
+.. _cmd_remote_test:
+
+platformio remote test
+======================
+
+Helper command for remote :ref:`unit_testing`.
+
+.. contents::
+
+Usage
+-----
+
+.. code-block:: bash
+
+    platformio remote test [OPTIONS]
+
+    # run tests on specified PIO Remote Agent
+    platformio remote --agent NAME test [OPTIONS]
+
+Description
+-----------
+
+Run remotely tests from PlatformIO based project. More details about PlatformIO
+:ref:`unit_testing`.
+
+This command allows you to apply the tests for the environments specified
+in :ref:`projectconf`.
+
+Options
+-------
+
+.. program:: platformio remote test
+
+.. option::
+    -e, --environment
+
+Process specified environments. More details :option:`platformio run --environment`
+
+.. option::
+    -i, --ignore
+
+Ignore tests where the name matches specified patterns. More than one
+pattern is allowed. If you need to ignore some tests for the specific
+environment, please take a look at :ref:`projectconf_test_ignore` option from
+:ref:`projectconf`.
+
+.. list-table::
+    :header-rows:  1
+
+    * - Pattern
+      - Meaning
+
+    * - ``*``
+      - matches everything
+
+    * - ``?``
+      - matches any single character
+
+    * - ``[seq]``
+      - matches any character in seq
+
+    * - ``[!seq]``
+      - matches any character not in seq
+
+For example, ``platformio remote test --ignore "mytest*" -i "test[13]"``
+
+.. option::
+    --upload-port
+
+A port that is intended for firmware uploading. To list available ports
+please use :ref:`cmd_device_list` command.
+
+If upload port is not specified, PlatformIO will try to detect it automatically.
+
+.. option::
+    --test-port
+
+A Serial/UART port that PlatformIO uses as communication interface between
+PlatformIO Unit Test Engine and target device. To list available ports
+please use :ref:`cmd_device_list` command.
+
+If test port is not specified, PlatformIO will try to detect it automatically.
+
+.. option::
+    -d, --project-dir
+
+Specify the path to project directory. By default, ``--project-dir`` is equal
+to current working directory (``CWD``).
+
+.. option::
+    -r, --build-remotely
+
+By default, :ref:`pio_remote` builds project on the local machine and deploy
+final testing firmware Over-The-Air (OTA) to remote device.
+
+If you need to build project on remote machine, please use
+:option:`platformio remote test --build-remotely` option. In this case,
+:ref:`pio_remote` will automatically deploy your project to remote machine,
+install required toolchains, frameworks, SDKs, etc and process tests.
+
+
+.. option::
+    --without-building
+
+Skip building stage.
+
+.. option::
+    --without-uploading
+
+Skip uploading stage
+
+.. option::
+    -v, --verbose
+
+Shows detailed information when processing environments.
+
+This option can be set globally using :ref:`setting_force_verbose` setting
+or by environment variable :envvar:`PLATFORMIO_SETTING_FORCE_VERBOSE`.
+
+Examples
+--------
+
+For the examples please follow to :ref:`unit_testing` page.

docs/userguide/remote/index.rst

@@ -45,3 +45,4 @@ To print all available commands and options use:
     cmd_agent
     cmd_device
     cmd_run
+    cmd_test

platformio/builder/tools/piolib.py

@@ -214,8 +214,11 @@ class LibBuilderBase(object):
         for item in self.dependencies:
             skip = False
             for key in ("platforms", "frameworks"):
-                if (key in item and not self.items_in_list(
+                env_key = "PIO" + key.upper()[:-1]
-                        self.env["PIO" + key.upper()[:-1]], item[key])):
+                if env_key not in self.env:
+                    continue
+                if (key in item and
+                        not self.items_in_list(self.env[env_key], item[key])):
                     if verbose:
                         sys.stderr.write("Skip %s incompatible dependency %s\n"
                                          % (key[:-1], item))
@@ -507,7 +510,7 @@ class PlatformIOLibBuilder(LibBuilderBase):
                 isdir(join(self.path, "utility"))):
             inc_dirs.append(join(self.path, "utility"))
 
-        for path in self.env['CPPPATH']:
+        for path in self.env.get("CPPPATH", []):
             if path not in self.envorigin['CPPPATH']:
                 inc_dirs.append(self.env.subst(path))
         return inc_dirs

platformio/commands/remote.py

@@ -51,7 +51,7 @@ def remote_agent_list():
     pioplus_call(sys.argv[1:])
 
 
-@cli.command("run", short_help="Process project environments")
+@cli.command("run", short_help="Process project environments remotely")
 @click.option("-e", "--environment", multiple=True)
 @click.option("-t", "--target", multiple=True)
 @click.option("--upload-port")
@@ -65,26 +65,49 @@ def remote_agent_list():
         dir_okay=True,
         writable=True,
         resolve_path=True))
+@click.option("--disable-auto-clean", is_flag=True)
+@click.option("-r", "--build-remotely", is_flag=True)
 @click.option("-s", "--silent", is_flag=True)
 @click.option("-v", "--verbose", is_flag=True)
-@click.option("-r", "--build-remotely", is_flag=True)
-@click.option("--disable-auto-clean", is_flag=True)
 def remote_run(**kwargs):
     pioplus_call(sys.argv[1:])
 
 
-@cli.group("device", short_help="Monitor device or list existing")
+@cli.command("test", short_help="Remote Unit Testing")
+@click.option("--environment", "-e", multiple=True, metavar="<environment>")
+@click.option("--ignore", "-i", multiple=True, metavar="<pattern>")
+@click.option("--upload-port")
+@click.option("--test-port")
+@click.option(
+    "-d",
+    "--project-dir",
+    default=getcwd,
+    type=click.Path(
+        exists=True,
+        file_okay=False,
+        dir_okay=True,
+        writable=True,
+        resolve_path=True))
+@click.option("-r", "--build-remotely", is_flag=True)
+@click.option("--without-building", is_flag=True)
+@click.option("--without-uploading", is_flag=True)
+@click.option("--verbose", "-v", is_flag=True)
+def remote_test(**kwargs):
+    pioplus_call(sys.argv[1:])
+
+
+@cli.group("device", short_help="Monitor remote device or list existing")
 def remote_device():
     pass
 
 
-@remote_device.command("list", short_help="List devices")
+@remote_device.command("list", short_help="List remote devices")
 @click.option("--json-output", is_flag=True)
 def device_list(json_output):
     pioplus_call(sys.argv[1:])
 
 
-@remote_device.command("monitor", short_help="Monitor device (Serial)")
+@remote_device.command("monitor", short_help="Monitor remote device")
 @click.option("--port", "-p", help="Port, a number or a device name")
 @click.option(
     "--baud", "-b", type=int, default=9600, help="Set baud rate, default=9600")

platformio/commands/run.py

@@ -126,7 +126,7 @@ class EnvironmentProcessor(object):
         "upload_port", "upload_protocol", "upload_speed", "upload_flags",
         "upload_resetmethod", "lib_install", "lib_deps", "lib_force",
         "lib_ignore", "lib_extra_dirs", "lib_ldf_mode", "lib_compat_mode",
-        "test_ignore", "piotest")
+        "test_ignore", "test_port", "piotest")
 
     REMAPED_OPTIONS = {"framework": "pioframework", "platform": "pioplatform"}
 

platformio/commands/test.py

@@ -20,10 +20,11 @@ import click
 from platformio.pioplus import pioplus_call
 
 
-@click.command("test", short_help="Unit Testing")
+@click.command("test", short_help="Local Unit Testing")
 @click.option("--environment", "-e", multiple=True, metavar="<environment>")
 @click.option("--ignore", "-i", multiple=True, metavar="<pattern>")
-@click.option("--upload-port", metavar="<upload port>")
+@click.option("--upload-port")
+@click.option("--test-port")
 @click.option(
     "-d",
     "--project-dir",
@@ -34,6 +35,8 @@ from platformio.pioplus import pioplus_call
         dir_okay=True,
         writable=True,
         resolve_path=True))
+@click.option("--without-building", is_flag=True)
+@click.option("--without-uploading", is_flag=True)
 @click.option("--verbose", "-v", is_flag=True)
 def cli(*args, **kwargs):  # pylint: disable=unused-argument
     pioplus_call(sys.argv[1:])