diff --git a/HISTORY.rst b/HISTORY.rst index 1451fd11..5b7e7d47 100644 --- a/HISTORY.rst +++ b/HISTORY.rst @@ -9,8 +9,9 @@ PlatformIO 3.0 * PlatformIO Plus - + `Unit Testing `__ for Embedded - (`issue #408 `_) + + Local and Embedded `Unit Testing `__ + (`issue #408 `_, + `issue #519 `_) * Decentralized Development Platforms diff --git a/docs/projectconf.rst b/docs/projectconf.rst index 57b3ca86..6ccb5484 100644 --- a/docs/projectconf.rst +++ b/docs/projectconf.rst @@ -656,7 +656,7 @@ Multiple dependencies are allowed (multi-lines). .. code-block:: ini - [env:***] + [env:myenv] lib_deps = LIBRARY_1 LIBRARY_2 @@ -784,6 +784,53 @@ Finder. More details :ref:`ldf_compat_mode`. By default, this value is set to ``lib_compat_mode = 1`` and means that LDF will check only for framework compatibility. + +Test options +~~~~~~~~~~~~ + +.. contents:: + :local: + +.. _projectconf_test_ignore: + +``test_ignore`` +^^^^^^^^^^^^^^^ + +.. versionadded:: 3.0 +.. seealso:: + Please make sure to read :ref:`unit_testing` guide first. + +Ignore tests where the name matches specified patterns. +More than one pattern is allowed (multi-lines). Also, you can ignore some +tests using :option:`platformio test --ignore` command. + +.. list-table:: + :header-rows: 1 + + * - Pattern + - Meaning + + * - ``*`` + - matches everything + + * - ``?`` + - matches any single character + + * - ``[seq]`` + - matches any character in seq + + * - ``[!seq]`` + - matches any character not in seq + +**Example** + +.. code-block:: ini + + [env:myenv] + test_ignore = + mytest* + test[13] + ----------- .. _projectconf_examples: diff --git a/docs/unit_testing.rst b/docs/unit_testing.rst index 5e6555a5..e50b6cdb 100644 --- a/docs/unit_testing.rst +++ b/docs/unit_testing.rst @@ -22,19 +22,30 @@ of one or more MCU program modules together with associated control data, usage procedures, and operating procedures, are tested to determine whether they are fit for use. Unit testing finds problems early in the development cycle. -PlatformIO Test System is very interesting for embedded development. -It allows you to write tests locally and run them directly on the target -device (hardware unit testing). Also, you will be able to run the same tests -on the different target devices (:ref:`embedded_boards`). +PlatformIO Test System supports 2 different test types: + +1. **Local Test** - *[host, native]*, process test on the host machine + using :ref:`platform_native`. +2. **Embedded Test** - *[remote, hardware]*, prepare special firmware for the + target device and upload it. Run test on the embedded device and collect + results. Process test results on the host machine. + + You will be able to run the same test on the different target devices + (:ref:`embedded_boards`). PlatformIO Test System consists of: * Project builder * Test builder -* Firmware uploader +* Firmware uploader (is used only for embedded test) * Test processor There is special command :ref:`cmd_test` to run tests from PlatformIO Project. +It allows to process specific environments or to ignore some tests using +"Glob patterns". + +Also, is possible to ignore some tests for specific environment using +:ref:`projectconf_test_ignore` option from :ref:`projectconf`. .. contents:: @@ -55,38 +66,74 @@ PlatformIO Test System design is based on a few isolated components: Workflow -------- -1. Create PlatformIO project using :ref:`cmd_init` command. +1. Create PlatformIO project using :ref:`cmd_init` command. For Local Unit + Testing (on the host machine), need to use :ref:`platform_native`. + + .. code-block:: ini + + ; PlatformIO Project Configuration File + ; + ; Build options: build flags, source filter, extra scripting + ; Upload options: custom port, speed and extra flags + ; Library options: dependencies, extra library storages + ; + ; Please visit documentation for the other options and examples + ; http://docs.platformio.org/en/stable/projectconf.html + + ; + ; Embedded platforms + ; + + [env:uno] + platform = atmelavr + framework = arduino + board = uno + + [env:nodemcu] + platform = espressif + framework = arduino + board = nodemcuv2 + + ; + ; Local (PC, native) platforms + ; + + [env:local] + platform = native + 2. Place source code of main program to ``src`` directory. 3. Wrap ``main()`` or ``setup()/loop()`` methods of main program in ``UNIT_TEST`` guard: .. code-block:: c - /** + /** * Arduino Wiring-based Framework */ - #ifndef UNIT_TEST - void setup () { + #ifndef UNIT_TEST + #include + void setup () { // some code... - } + } - void loop () { + void loop () { // some code... - } - #endif + } + #endif - /** + /** * Generic C/C++ */ - #ifndef UNIT_TEST - int main() { + #ifndef UNIT_TEST + int main(int argc, char **argv) { // setup code... while (1) { // loop code... } - } - #endif + return 0 + } + #endif 4. Create ``test`` directory in the root of project. See :ref:`projectconf_pio_test_dir`. 5. Write test using :ref:`unit_testing_api`. The each test is a small @@ -183,8 +230,8 @@ The summary of `Unity Test API `_ +* `Local & Embedded: Calculator `_ For the other examples and source code please follow to -`PlatformIO Unit Testing Examples `_ repository. +`PlatformIO Unit Testing Examples `_ repository. diff --git a/docs/userguide/cmd_test.rst b/docs/userguide/cmd_test.rst index c42bda6c..7c74361d 100644 --- a/docs/userguide/cmd_test.rst +++ b/docs/userguide/cmd_test.rst @@ -45,10 +45,12 @@ Options Process specified environments. More details :option:`platformio run --environment` .. option:: - --skip + -i, --ignore -Skip over tests where the name matches specified patterns. More than one -option/pattern is allowed. +Ignore tests where the name matches specified patterns. More than one +pattern is allowed. If you need to ignore some tests for the specific +environment, please take a look at :ref:`projectconf_test_ignore` option from +:ref:`projectconf`. .. list-table:: :header-rows: 1 @@ -68,7 +70,7 @@ option/pattern is allowed. * - ``[!seq]`` - matches any character not in seq -For example, ``platformio test --skip "mytest*" -i "test[13]"`` +For example, ``platformio test --ignore "mytest*" -i "test[13]"`` .. option:: --upload-port diff --git a/platformio/__init__.py b/platformio/__init__.py index e880489e..7114b1f3 100644 --- a/platformio/__init__.py +++ b/platformio/__init__.py @@ -14,7 +14,7 @@ import sys -VERSION = (3, 0, "0a3") +VERSION = (3, 0, "0a4") __version__ = ".".join([str(s) for s in VERSION]) __title__ = "platformio" diff --git a/platformio/builder/tools/piotest.py b/platformio/builder/tools/piotest.py index 3789e73f..3b72f7ce 100644 --- a/platformio/builder/tools/piotest.py +++ b/platformio/builder/tools/piotest.py @@ -43,6 +43,14 @@ FRAMEWORK_PARAMETERS = { "serial_flush": "Serial.flush()", "serial_begin": "Serial.begin(9600)", "serial_end": "Serial.end()" + }, + "native": { + "framework": "stdio.h", + "serial_obj": "", + "serial_putc": "putchar(a)", + "serial_flush": "fflush(stdout)", + "serial_begin": "", + "serial_end": "" } } @@ -114,7 +122,10 @@ void output_complete(void) print("Warning: Could not remove temporary file '%s'. " "Please remove it manually." % file_) - framework = env.subst("$PIOFRAMEWORK").lower() + if env['PIOPLATFORM'] == "native": + framework = "native" + else: + framework = env.subst("$PIOFRAMEWORK").lower() if framework not in FRAMEWORK_PARAMETERS: env.Exit("Error: %s framework doesn't support testing feature!" % framework) diff --git a/platformio/commands/init.py b/platformio/commands/init.py index 8a77a24c..1ccc590a 100644 --- a/platformio/commands/init.py +++ b/platformio/commands/init.py @@ -80,9 +80,8 @@ def cli(ctx, # pylint: disable=R0913 click.echo("The next files/directories have been created in %s" % click.style( project_dir, fg="cyan")) - click.echo("%s - Project Configuration File" % - click.style( - "platformio.ini", fg="cyan")) + click.echo("%s - Project Configuration File" % click.style( + "platformio.ini", fg="cyan")) click.echo("%s - Put your source files here" % click.style( "src", fg="cyan")) click.echo("%s - Put here project specific (private) libraries" % @@ -273,10 +272,7 @@ def init_ci_conf(project_dir): def init_cvs_ignore(project_dir): ignore_path = join(project_dir, ".gitignore") - default = [ - ".pioenvs\n", - ".piolibdeps\n" - ] + default = [".pioenvs\n", ".piolibdeps\n"] current = [] if isfile(ignore_path): with open(ignore_path) as fp: diff --git a/platformio/commands/run.py b/platformio/commands/run.py index 95bd4ca6..e130393f 100644 --- a/platformio/commands/run.py +++ b/platformio/commands/run.py @@ -111,14 +111,14 @@ def cli(ctx, # pylint: disable=R0913,R0914 class EnvironmentProcessor(object): - KNOWN_OPTIONS = ("platform", "framework", "board", "board_mcu", - "board_f_cpu", "board_f_flash", "board_flash_mode", - "build_flags", "src_build_flags", "build_unflags", - "src_filter", "extra_script", "targets", "upload_port", - "upload_protocol", "upload_speed", "upload_flags", - "upload_resetmethod", "lib_install", "lib_deps", - "lib_force", "lib_ignore", "lib_extra_dirs", - "lib_ldf_mode", "lib_compat_mode", "piotest") + KNOWN_OPTIONS = ( + "platform", "framework", "board", "board_mcu", "board_f_cpu", + "board_f_flash", "board_flash_mode", "build_flags", "src_build_flags", + "build_unflags", "src_filter", "extra_script", "targets", + "upload_port", "upload_protocol", "upload_speed", "upload_flags", + "upload_resetmethod", "lib_install", "lib_deps", "lib_force", + "lib_ignore", "lib_extra_dirs", "lib_ldf_mode", "lib_compat_mode", + "test_ignore", "piotest") REMAPED_OPTIONS = {"framework": "pioframework", "platform": "pioplatform"} diff --git a/platformio/commands/test.py b/platformio/commands/test.py index 9ba8e472..a1b73262 100644 --- a/platformio/commands/test.py +++ b/platformio/commands/test.py @@ -30,7 +30,7 @@ from platformio.managers.platform import PlatformFactory @click.command("test", short_help="Unit Testing") @click.option("--environment", "-e", multiple=True, metavar="") -@click.option("--skip", multiple=True, metavar="") +@click.option("--ignore", "-i", multiple=True, metavar="") @click.option("--upload-port", metavar="") @click.option( "-d", @@ -44,7 +44,7 @@ from platformio.managers.platform import PlatformFactory resolve_path=True)) @click.option("--verbose", "-v", is_flag=True) @click.pass_context -def cli(ctx, environment, skip, upload_port, project_dir, verbose): +def cli(ctx, environment, ignore, upload_port, project_dir, verbose): with util.cd(project_dir): test_dir = util.get_projecttest_dir() if not isdir(test_dir): @@ -59,19 +59,30 @@ def cli(ctx, environment, skip, upload_port, project_dir, verbose): start_time = time() results = [] for testname in test_names: - for envname in projectconf.sections(): - if not envname.startswith("env:"): + for section in projectconf.sections(): + if not section.startswith("env:"): continue - envname = envname[4:] + + envname = section[4:] if environment and envname not in environment: continue - # check skip patterns - if testname != "*" and any([fnmatch(testname, p) for p in skip]): + # check ignore patterns + _ignore = list(ignore) + if projectconf.has_option(section, "test_ignore"): + _ignore.extend([p.strip() + for p in projectconf.get( + section, "test_ignore").split("\n") + if p.strip()]) + if testname != "*" and \ + any([fnmatch(testname, p) for p in _ignore]): results.append((None, testname, envname)) continue - tp = TestProcessor(ctx, testname, envname, { + cls = (LocalTestProcessor + if projectconf.get(section, "platform") == "native" else + EmbeddedTestProcessor) + tp = cls(ctx, testname, envname, { "project_config": projectconf, "project_dir": project_dir, "upload_port": upload_port, @@ -93,7 +104,7 @@ def cli(ctx, environment, skip, upload_port, project_dir, verbose): status_str = click.style("IGNORED", fg="yellow") click.echo( - "test:%s/env:%s\t%s" % (click.style( + "test:%s/env:%s\t[%s]" % (click.style( testname, fg="yellow"), click.style( envname, fg="cyan"), status_str), err=status is False) @@ -108,10 +119,7 @@ def cli(ctx, environment, skip, upload_port, project_dir, verbose): raise exception.ReturnErrorCode() -class TestProcessor(object): - - SERIAL_TIMEOUT = 600 - SERIAL_BAUDRATE = 9600 +class TestProcessorBase(object): def __init__(self, cmd_ctx, testname, envname, options): self.cmd_ctx = cmd_ctx @@ -119,24 +127,16 @@ class TestProcessor(object): self.test_name = testname self.env_name = envname self.options = options + self._run_failed = False - def process(self): - self._progress("Building... (1/3)") - self._build_or_upload(["test"]) - self._progress("Uploading... (2/3)") - self._build_or_upload(["test", "upload"]) - self._progress("Testing... (3/3)") - sleep(1.0) # wait while board is starting... - return self._run_hardware_test() - - def _progress(self, text, is_error=False): + def print_progress(self, text, is_error=False): print_header( "[test::%s] %s" % (click.style( self.test_name, fg="yellow", bold=True), text), is_error=is_error) click.echo() - def _build_or_upload(self, target): + def build_or_upload(self, target): if self.test_name != "*": self.cmd_ctx.meta['piotest'] = self.test_name return self.cmd_ctx.invoke( @@ -147,7 +147,54 @@ class TestProcessor(object): environment=[self.env_name], target=target) - def _run_hardware_test(self): + def run(self): + raise NotImplementedError + + def on_run_out(self, line): + if line.endswith(":PASS"): + click.echo("%s\t[%s]" % (line[:-5], click.style( + "PASSED", fg="green"))) + elif ":FAIL:" in line: + self._run_failed = True + click.echo("%s\t[%s]" % (line, click.style("FAILED", fg="red"))) + else: + click.echo(line) + + +class LocalTestProcessor(TestProcessorBase): + + def process(self): + self.print_progress("Building... (1/2)") + self.build_or_upload(["test"]) + self.print_progress("Testing... (2/2)") + return self.run() + + def run(self): + with util.cd(self.options['project_dir']): + pioenvs_dir = util.get_projectpioenvs_dir() + result = util.exec_command( + [join(pioenvs_dir, self.env_name, "program")], + stdout=util.AsyncPipe(self.on_run_out), + stderr=util.AsyncPipe(self.on_run_out)) + assert "returncode" in result + return result['returncode'] == 0 and not self._run_failed + + +class EmbeddedTestProcessor(TestProcessorBase): + + SERIAL_TIMEOUT = 600 + SERIAL_BAUDRATE = 9600 + + def process(self): + self.print_progress("Building... (1/3)") + self.build_or_upload(["test"]) + self.print_progress("Uploading... (2/3)") + self.build_or_upload(["test", "upload"]) + self.print_progress("Testing... (3/3)") + sleep(1.0) # wait while board is starting... + return self.run() + + def run(self): click.echo("If you don't see any output for the first 10 secs, " "please reset board (press reset button)") click.echo() @@ -155,23 +202,15 @@ class TestProcessor(object): self.get_serial_port(), self.SERIAL_BAUDRATE, timeout=self.SERIAL_TIMEOUT) - passed = True while True: line = ser.readline().strip() if not line: continue - if line.endswith(":PASS"): - click.echo("%s\t%s" % (line[:-5], click.style( - "PASSED", fg="green"))) - elif ":FAIL:" in line: - passed = False - click.echo("%s\t%s" % (line, click.style("FAILED", fg="red"))) - else: - click.echo(line) + self.on_run_out(line) if all([l in line for l in ("Tests", "Failures", "Ignored")]): break ser.close() - return passed + return not self._run_failed def get_serial_port(self): config = self.options['project_config'] diff --git a/platformio/managers/platform.py b/platformio/managers/platform.py index aae99081..8bfee505 100644 --- a/platformio/managers/platform.py +++ b/platformio/managers/platform.py @@ -362,14 +362,15 @@ class PlatformBase(PlatformPackagesMixin, PlatformRunMixin): @property def packages(self): - packages = self._manifest.get("packages", {}) - if "tool-scons" not in packages: - packages['tool-scons'] = { + if "packages" not in self._manifest: + self._manifest['packages'] = {} + if "tool-scons" not in self._manifest['packages']: + self._manifest['packages']['tool-scons'] = { "version": self._manifest.get("engines", {}).get( "scons", ">=2.3.0,<2.6.0"), "optional": False } - return packages + return self._manifest['packages'] def get_dir(self): return dirname(self.manifest_path) @@ -476,7 +477,7 @@ class PlatformBase(PlatformPackagesMixin, PlatformRunMixin): if "test" in targets and "tool-unity" not in self.packages: self.packages['tool-unity'] = { - "version": "~1.20302.0", + "version": "~1.20302.1", "optional": False } diff --git a/tests/commands/test_test.py b/tests/commands/test_test.py new file mode 100644 index 00000000..482f20f4 --- /dev/null +++ b/tests/commands/test_test.py @@ -0,0 +1,26 @@ +# Copyright 2014-present PlatformIO +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from os.path import join + +from platformio.commands.test import cli as cli_test + + +def test_local_env(clirunner, validate_cliresult): + result = clirunner.invoke( + cli_test, + ["-d", join("examples", "unit-testing", "calculator"), "-e", "local"]) + result.exit_code == -1 + assert all( + [s in result.output for s in ("[PASSED]", "[IGNORED]", "[FAILED]")])