Add Support for local ("PC") unit tests // Resolve #519

HISTORY.rst

@@ -9,8 +9,9 @@ PlatformIO 3.0
 
 * PlatformIO Plus
 
-  + `Unit Testing <http://docs.platformio.org/en/latest/unit_testing.html>`__ for Embedded
+  + Local and Embedded `Unit Testing <http://docs.platformio.org/en/latest/unit_testing.html>`__
-    (`issue #408 <https://github.com/platformio/platformio/issues/408>`_)
+    (`issue #408 <https://github.com/platformio/platformio/issues/408>`_,
+    `issue #519 <https://github.com/platformio/platformio/issues/519>`_)
 
 * Decentralized Development Platforms
 

docs/projectconf.rst

@@ -656,7 +656,7 @@ Multiple dependencies are allowed (multi-lines).
 
 .. code-block:: ini
 
-  [env:***]
+  [env:myenv]
   lib_deps =
     LIBRARY_1
     LIBRARY_2
@@ -784,6 +784,53 @@ Finder. More details :ref:`ldf_compat_mode`.
 By default, this value is set to ``lib_compat_mode = 1`` and means that LDF
 will check only for framework compatibility.
 
+
+Test options
+~~~~~~~~~~~~
+
+.. contents::
+    :local:
+
+.. _projectconf_test_ignore:
+
+``test_ignore``
+^^^^^^^^^^^^^^^
+
+.. versionadded:: 3.0
+.. seealso::
+    Please make sure to read :ref:`unit_testing` guide first.
+
+Ignore tests where the name matches specified patterns.
+More than one pattern is allowed (multi-lines). Also, you can ignore some
+tests using :option:`platformio test --ignore` command.
+
+.. list-table::
+    :header-rows:  1
+
+    * - Pattern
+      - Meaning
+
+    * - ``*``
+      - matches everything
+
+    * - ``?``
+      - matches any single character
+
+    * - ``[seq]``
+      - matches any character in seq
+
+    * - ``[!seq]``
+      - matches any character not in seq
+
+**Example**
+
+.. code-block:: ini
+
+  [env:myenv]
+  test_ignore =
+    mytest*
+    test[13]
+
 -----------
 
 .. _projectconf_examples:

docs/unit_testing.rst

@@ -22,19 +22,30 @@ of one or more MCU program modules together with associated control data,
 usage procedures, and operating procedures, are tested to determine whether
 they are fit for use. Unit testing finds problems early in the development cycle.
 
-PlatformIO Test System is very interesting for embedded development.
+PlatformIO Test System supports 2 different test types:
-It allows you to write tests locally and run them directly on the target
+
-device (hardware unit testing). Also, you will be able to run the same tests
+1. **Local Test** - *[host, native]*, process test on the host machine
-on the different target devices (:ref:`embedded_boards`).
+   using :ref:`platform_native`.
+2. **Embedded Test** - *[remote, hardware]*, prepare special firmware for the
+   target device and upload it. Run test on the embedded device and collect
+   results. Process test results on the host machine.
+
+   You will be able to run the same test on the different target devices
+   (:ref:`embedded_boards`).
 
 PlatformIO Test System consists of:
 
 * Project builder
 * Test builder
-* Firmware uploader
+* Firmware uploader (is used only for embedded test)
 * Test processor
 
 There is special command :ref:`cmd_test` to run tests from PlatformIO Project.
+It allows to process specific environments or to ignore some tests using
+"Glob patterns".
+
+Also, is possible to ignore some tests for specific environment using
+:ref:`projectconf_test_ignore` option from :ref:`projectconf`.
 
 .. contents::
 
@@ -55,38 +66,74 @@ PlatformIO Test System design is based on a few isolated components:
 Workflow
 --------
 
-1. Create PlatformIO project using :ref:`cmd_init` command.
+1. Create PlatformIO project using :ref:`cmd_init` command. For Local Unit
+   Testing (on the host machine), need to use :ref:`platform_native`.
+
+   .. code-block:: ini
+
+        ; PlatformIO Project Configuration File
+        ;
+        ;   Build options: build flags, source filter, extra scripting
+        ;   Upload options: custom port, speed and extra flags
+        ;   Library options: dependencies, extra library storages
+        ;
+        ; Please visit documentation for the other options and examples
+        ; http://docs.platformio.org/en/stable/projectconf.html
+
+        ;
+        ; Embedded platforms
+        ;
+
+        [env:uno]
+        platform = atmelavr
+        framework = arduino
+        board = uno
+
+        [env:nodemcu]
+        platform = espressif
+        framework = arduino
+        board = nodemcuv2
+
+        ;
+        ; Local (PC, native) platforms
+        ;
+
+        [env:local]
+        platform = native
+
 2. Place source code of main program to ``src`` directory.
 3. Wrap ``main()`` or ``setup()/loop()`` methods of main program in ``UNIT_TEST``
    guard:
 
    .. code-block:: c
 
-       /**
+        /**
         * Arduino Wiring-based Framework
         */
-       #ifndef UNIT_TEST
+        #ifndef UNIT_TEST
-       void setup () {
+        #include <Arduino.h>
+        void setup () {
           // some code...
-       }
+        }
 
-       void loop () {
+        void loop () {
           // some code...
-       }
+        }
-       #endif
+        #endif
 
-       /**
+        /**
         * Generic C/C++
         */
-       #ifndef UNIT_TEST
+        #ifndef UNIT_TEST
-       int main() {
+        int main(int argc, char **argv) {
           // setup code...
 
           while (1) {
               // loop code...
           }
-       }
+          return 0
-       #endif
+        }
+        #endif
 
 4. Create ``test`` directory in the root of project. See :ref:`projectconf_pio_test_dir`.
 5. Write test using :ref:`unit_testing_api`. The each test is a small
@@ -183,8 +230,8 @@ The summary of `Unity Test API <https://github.com/ThrowTheSwitch/Unity#unity-te
 
   - ``TEST_ASSERT_EQUAL_MEMORY(expected, actual, len)``
 
-Example
+Test "Blink" Project
--------
+--------------------
 
 1. Please follow to :ref:`quickstart` and create "Blink Project". According
    to the Unit Testing :ref:`unit_testing_design` it is the **Main program**.
@@ -215,8 +262,15 @@ Source files
 
   .. code-block:: ini
 
-      ; Project Configuration File
+      ; PlatformIO Project Configuration File
-      ; Docs: http://docs.platformio.org/en/latest/projectconf.html
+      ;
+      ;   Build options: build flags, source filter, extra scripting
+      ;   Upload options: custom port, speed and extra flags
+      ;   Library options: dependencies, extra library storages
+      ;
+      ; Please visit documentation for the other options and examples
+      ; http://docs.platformio.org/en/stable/projectconf.html
+
 
       [env:uno]
       platform = atmelavr
@@ -393,7 +447,11 @@ Test results
     test:*/env:uno  PASSED
     ========================= [PASSED] Took 13.35 seconds ========================
 
--------
+Examples
+--------
+
+* `Embedded: Wiring Blink <https://github.com/platformio/platformio-examples/tree/develop/unit-testing/wiring-blink>`_
+* `Local & Embedded: Calculator <https://github.com/platformio/platformio-examples/tree/develop/unit-testing/calculator>`_
 
 For the other examples and source code please follow to
-`PlatformIO Unit Testing Examples <https://github.com/platformio/platformio-examples/tree/feature/platformio-30/unit-testing>`_ repository.
+`PlatformIO Unit Testing Examples <hhttps://github.com/platformio/platformio-examples/tree/develop/unit-testing>`_ repository.

docs/userguide/cmd_test.rst

@@ -45,10 +45,12 @@ Options
 Process specified environments. More details :option:`platformio run --environment`
 
 .. option::
-    --skip
+    -i, --ignore
 
-Skip over tests where the name matches specified patterns. More than one
+Ignore tests where the name matches specified patterns. More than one
-option/pattern is allowed.
+pattern is allowed. If you need to ignore some tests for the specific
+environment, please take a look at :ref:`projectconf_test_ignore` option from
+:ref:`projectconf`.
 
 .. list-table::
     :header-rows:  1
@@ -68,7 +70,7 @@ option/pattern is allowed.
     * - ``[!seq]``
       - matches any character not in seq
 
-For example, ``platformio test --skip "mytest*" -i "test[13]"``
+For example, ``platformio test --ignore "mytest*" -i "test[13]"``
 
 .. option::
     --upload-port

platformio/__init__.py

@@ -14,7 +14,7 @@
 
 import sys
 
-VERSION = (3, 0, "0a3")
+VERSION = (3, 0, "0a4")
 __version__ = ".".join([str(s) for s in VERSION])
 
 __title__ = "platformio"

platformio/builder/tools/piotest.py

@@ -43,6 +43,14 @@ FRAMEWORK_PARAMETERS = {
         "serial_flush": "Serial.flush()",
         "serial_begin": "Serial.begin(9600)",
         "serial_end": "Serial.end()"
+    },
+    "native": {
+        "framework": "stdio.h",
+        "serial_obj": "",
+        "serial_putc": "putchar(a)",
+        "serial_flush": "fflush(stdout)",
+        "serial_begin": "",
+        "serial_end": ""
     }
 }
 
@@ -114,7 +122,10 @@ void output_complete(void)
                 print("Warning: Could not remove temporary file '%s'. "
                       "Please remove it manually." % file_)
 
-    framework = env.subst("$PIOFRAMEWORK").lower()
+    if env['PIOPLATFORM'] == "native":
+        framework = "native"
+    else:
+        framework = env.subst("$PIOFRAMEWORK").lower()
     if framework not in FRAMEWORK_PARAMETERS:
         env.Exit("Error: %s framework doesn't support testing feature!" %
                  framework)

platformio/commands/init.py

@@ -80,9 +80,8 @@ def cli(ctx,  # pylint: disable=R0913
     click.echo("The next files/directories have been created in %s" %
                click.style(
                    project_dir, fg="cyan"))
-    click.echo("%s - Project Configuration File" %
+    click.echo("%s - Project Configuration File" % click.style(
-               click.style(
+        "platformio.ini", fg="cyan"))
-                   "platformio.ini", fg="cyan"))
     click.echo("%s - Put your source files here" % click.style(
         "src", fg="cyan"))
     click.echo("%s - Put here project specific (private) libraries" %
@@ -273,10 +272,7 @@ def init_ci_conf(project_dir):
 
 def init_cvs_ignore(project_dir):
     ignore_path = join(project_dir, ".gitignore")
-    default = [
+    default = [".pioenvs\n", ".piolibdeps\n"]
-        ".pioenvs\n",
-        ".piolibdeps\n"
-    ]
     current = []
     if isfile(ignore_path):
         with open(ignore_path) as fp:

platformio/commands/run.py

@@ -111,14 +111,14 @@ def cli(ctx,  # pylint: disable=R0913,R0914
 
 class EnvironmentProcessor(object):
 
-    KNOWN_OPTIONS = ("platform", "framework", "board", "board_mcu",
+    KNOWN_OPTIONS = (
-                     "board_f_cpu", "board_f_flash", "board_flash_mode",
+        "platform", "framework", "board", "board_mcu", "board_f_cpu",
-                     "build_flags", "src_build_flags", "build_unflags",
+        "board_f_flash", "board_flash_mode", "build_flags", "src_build_flags",
-                     "src_filter", "extra_script", "targets", "upload_port",
+        "build_unflags", "src_filter", "extra_script", "targets",
-                     "upload_protocol", "upload_speed", "upload_flags",
+        "upload_port", "upload_protocol", "upload_speed", "upload_flags",
-                     "upload_resetmethod", "lib_install", "lib_deps",
+        "upload_resetmethod", "lib_install", "lib_deps", "lib_force",
-                     "lib_force", "lib_ignore", "lib_extra_dirs",
+        "lib_ignore", "lib_extra_dirs", "lib_ldf_mode", "lib_compat_mode",
-                     "lib_ldf_mode", "lib_compat_mode", "piotest")
+        "test_ignore", "piotest")
 
     REMAPED_OPTIONS = {"framework": "pioframework", "platform": "pioplatform"}
 

platformio/commands/test.py

@@ -30,7 +30,7 @@ from platformio.managers.platform import PlatformFactory
 
 @click.command("test", short_help="Unit Testing")
 @click.option("--environment", "-e", multiple=True, metavar="<environment>")
-@click.option("--skip", multiple=True, metavar="<pattern>")
+@click.option("--ignore", "-i", multiple=True, metavar="<pattern>")
 @click.option("--upload-port", metavar="<upload port>")
 @click.option(
     "-d",
@@ -44,7 +44,7 @@ from platformio.managers.platform import PlatformFactory
         resolve_path=True))
 @click.option("--verbose", "-v", is_flag=True)
 @click.pass_context
-def cli(ctx, environment, skip, upload_port, project_dir, verbose):
+def cli(ctx, environment, ignore, upload_port, project_dir, verbose):
     with util.cd(project_dir):
         test_dir = util.get_projecttest_dir()
         if not isdir(test_dir):
@@ -59,19 +59,30 @@ def cli(ctx, environment, skip, upload_port, project_dir, verbose):
     start_time = time()
     results = []
     for testname in test_names:
-        for envname in projectconf.sections():
+        for section in projectconf.sections():
-            if not envname.startswith("env:"):
+            if not section.startswith("env:"):
                 continue
-            envname = envname[4:]
+
+            envname = section[4:]
             if environment and envname not in environment:
                 continue
 
-            # check skip patterns
+            # check ignore patterns
-            if testname != "*" and any([fnmatch(testname, p) for p in skip]):
+            _ignore = list(ignore)
+            if projectconf.has_option(section, "test_ignore"):
+                _ignore.extend([p.strip()
+                                for p in projectconf.get(
+                                    section, "test_ignore").split("\n")
+                                if p.strip()])
+            if testname != "*" and \
+               any([fnmatch(testname, p) for p in _ignore]):
                 results.append((None, testname, envname))
                 continue
 
-            tp = TestProcessor(ctx, testname, envname, {
+            cls = (LocalTestProcessor
+                   if projectconf.get(section, "platform") == "native" else
+                   EmbeddedTestProcessor)
+            tp = cls(ctx, testname, envname, {
                 "project_config": projectconf,
                 "project_dir": project_dir,
                 "upload_port": upload_port,
@@ -93,7 +104,7 @@ def cli(ctx, environment, skip, upload_port, project_dir, verbose):
             status_str = click.style("IGNORED", fg="yellow")
 
         click.echo(
-            "test:%s/env:%s\t%s" % (click.style(
+            "test:%s/env:%s\t[%s]" % (click.style(
                 testname, fg="yellow"), click.style(
                     envname, fg="cyan"), status_str),
             err=status is False)
@@ -108,10 +119,7 @@ def cli(ctx, environment, skip, upload_port, project_dir, verbose):
         raise exception.ReturnErrorCode()
 
 
-class TestProcessor(object):
+class TestProcessorBase(object):
-
-    SERIAL_TIMEOUT = 600
-    SERIAL_BAUDRATE = 9600
 
     def __init__(self, cmd_ctx, testname, envname, options):
         self.cmd_ctx = cmd_ctx
@@ -119,24 +127,16 @@ class TestProcessor(object):
         self.test_name = testname
         self.env_name = envname
         self.options = options
+        self._run_failed = False
 
-    def process(self):
+    def print_progress(self, text, is_error=False):
-        self._progress("Building... (1/3)")
-        self._build_or_upload(["test"])
-        self._progress("Uploading... (2/3)")
-        self._build_or_upload(["test", "upload"])
-        self._progress("Testing... (3/3)")
-        sleep(1.0)  # wait while board is starting...
-        return self._run_hardware_test()
-
-    def _progress(self, text, is_error=False):
         print_header(
             "[test::%s] %s" % (click.style(
                 self.test_name, fg="yellow", bold=True), text),
             is_error=is_error)
         click.echo()
 
-    def _build_or_upload(self, target):
+    def build_or_upload(self, target):
         if self.test_name != "*":
             self.cmd_ctx.meta['piotest'] = self.test_name
         return self.cmd_ctx.invoke(
@@ -147,7 +147,54 @@ class TestProcessor(object):
             environment=[self.env_name],
             target=target)
 
-    def _run_hardware_test(self):
+    def run(self):
+        raise NotImplementedError
+
+    def on_run_out(self, line):
+        if line.endswith(":PASS"):
+            click.echo("%s\t[%s]" % (line[:-5], click.style(
+                "PASSED", fg="green")))
+        elif ":FAIL:" in line:
+            self._run_failed = True
+            click.echo("%s\t[%s]" % (line, click.style("FAILED", fg="red")))
+        else:
+            click.echo(line)
+
+
+class LocalTestProcessor(TestProcessorBase):
+
+    def process(self):
+        self.print_progress("Building... (1/2)")
+        self.build_or_upload(["test"])
+        self.print_progress("Testing... (2/2)")
+        return self.run()
+
+    def run(self):
+        with util.cd(self.options['project_dir']):
+            pioenvs_dir = util.get_projectpioenvs_dir()
+        result = util.exec_command(
+            [join(pioenvs_dir, self.env_name, "program")],
+            stdout=util.AsyncPipe(self.on_run_out),
+            stderr=util.AsyncPipe(self.on_run_out))
+        assert "returncode" in result
+        return result['returncode'] == 0 and not self._run_failed
+
+
+class EmbeddedTestProcessor(TestProcessorBase):
+
+    SERIAL_TIMEOUT = 600
+    SERIAL_BAUDRATE = 9600
+
+    def process(self):
+        self.print_progress("Building... (1/3)")
+        self.build_or_upload(["test"])
+        self.print_progress("Uploading... (2/3)")
+        self.build_or_upload(["test", "upload"])
+        self.print_progress("Testing... (3/3)")
+        sleep(1.0)  # wait while board is starting...
+        return self.run()
+
+    def run(self):
         click.echo("If you don't see any output for the first 10 secs, "
                    "please reset board (press reset button)")
         click.echo()
@@ -155,23 +202,15 @@ class TestProcessor(object):
             self.get_serial_port(),
             self.SERIAL_BAUDRATE,
             timeout=self.SERIAL_TIMEOUT)
-        passed = True
         while True:
             line = ser.readline().strip()
             if not line:
                 continue
-            if line.endswith(":PASS"):
+            self.on_run_out(line)
-                click.echo("%s\t%s" % (line[:-5], click.style(
-                    "PASSED", fg="green")))
-            elif ":FAIL:" in line:
-                passed = False
-                click.echo("%s\t%s" % (line, click.style("FAILED", fg="red")))
-            else:
-                click.echo(line)
             if all([l in line for l in ("Tests", "Failures", "Ignored")]):
                 break
         ser.close()
-        return passed
+        return not self._run_failed
 
     def get_serial_port(self):
         config = self.options['project_config']

platformio/managers/platform.py

@@ -362,14 +362,15 @@ class PlatformBase(PlatformPackagesMixin, PlatformRunMixin):
 
     @property
     def packages(self):
-        packages = self._manifest.get("packages", {})
+        if "packages" not in self._manifest:
-        if "tool-scons" not in packages:
+            self._manifest['packages'] = {}
-            packages['tool-scons'] = {
+        if "tool-scons" not in self._manifest['packages']:
+            self._manifest['packages']['tool-scons'] = {
                 "version": self._manifest.get("engines", {}).get(
                     "scons", ">=2.3.0,<2.6.0"),
                 "optional": False
             }
-        return packages
+        return self._manifest['packages']
 
     def get_dir(self):
         return dirname(self.manifest_path)
@@ -476,7 +477,7 @@ class PlatformBase(PlatformPackagesMixin, PlatformRunMixin):
 
         if "test" in targets and "tool-unity" not in self.packages:
             self.packages['tool-unity'] = {
-                "version": "~1.20302.0",
+                "version": "~1.20302.1",
                 "optional": False
             }
 

tests/commands/test_test.py

@@ -0,0 +1,26 @@
+# Copyright 2014-present PlatformIO <contact@platformio.org>
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from os.path import join
+
+from platformio.commands.test import cli as cli_test
+
+
+def test_local_env(clirunner, validate_cliresult):
+    result = clirunner.invoke(
+        cli_test,
+        ["-d", join("examples", "unit-testing", "calculator"), "-e", "local"])
+    result.exit_code == -1
+    assert all(
+        [s in result.output for s in ("[PASSED]", "[IGNORED]", "[FAILED]")])