feedc0de/platformio-core
1
0
Fork 0
forked from platformio/platformio-core
Code Pull Requests Activity

Add Support for local ("PC") unit tests // Resolve #519

Browse Source
This commit is contained in:
Ivan Kravets
2016-08-10 15:50:01 +03:00
parent 9177c6f210
commit a395b171e3
11 changed files with 269 additions and 88 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
HISTORY.rst
View File

@ -9,8 +9,9 @@ PlatformIO 3.0
* PlatformIO Plus
+ `Unit Testing <http://docs.platformio.org/en/latest/unit_testing.html>`__ for Embedded
(`issue #408 <https://github.com/platformio/platformio/issues/408>`_)
+ Local and Embedded `Unit Testing <http://docs.platformio.org/en/latest/unit_testing.html>`__
(`issue #408 <https://github.com/platformio/platformio/issues/408>`_,
`issue #519 <https://github.com/platformio/platformio/issues/519>`_)
* Decentralized Development Platforms

49
docs/projectconf.rst
View File

@ -656,7 +656,7 @@ Multiple dependencies are allowed (multi-lines).
.. code-block:: ini
[env:***]
[env:myenv]
lib_deps =
LIBRARY_1
LIBRARY_2
@ -784,6 +784,53 @@ Finder. More details :ref:`ldf_compat_mode`.
By default, this value is set to ``lib_compat_mode = 1`` and means that LDF
will check only for framework compatibility.
Test options
~~~~~~~~~~~~
.. contents::
:local:
.. _projectconf_test_ignore:
``test_ignore``
^^^^^^^^^^^^^^^
.. versionadded:: 3.0
.. seealso::
Please make sure to read :ref:`unit_testing` guide first.
Ignore tests where the name matches specified patterns.
More than one pattern is allowed (multi-lines). Also, you can ignore some
tests using :option:`platformio test --ignore` command.
.. list-table::
:header-rows: 1
* - Pattern
- Meaning
* - ``*``
- matches everything
* - ``?``
- matches any single character
* - ``[seq]``
- matches any character in seq
* - ``[!seq]``
- matches any character not in seq
**Example**
.. code-block:: ini
[env:myenv]
test_ignore =
mytest*
test[13]
-----------
.. _projectconf_examples:

106
docs/unit_testing.rst
View File

@ -22,19 +22,30 @@ of one or more MCU program modules together with associated control data,
usage procedures, and operating procedures, are tested to determine whether
they are fit for use. Unit testing finds problems early in the development cycle.
PlatformIO Test System is very interesting for embedded development.
It allows you to write tests locally and run them directly on the target
device (hardware unit testing). Also, you will be able to run the same tests
on the different target devices (:ref:`embedded_boards`).
PlatformIO Test System supports 2 different test types:
1. **Local Test** - *[host, native]*, process test on the host machine
using :ref:`platform_native`.
2. **Embedded Test** - *[remote, hardware]*, prepare special firmware for the
target device and upload it. Run test on the embedded device and collect
results. Process test results on the host machine.
You will be able to run the same test on the different target devices
(:ref:`embedded_boards`).
PlatformIO Test System consists of:
* Project builder
* Test builder
* Firmware uploader
* Firmware uploader (is used only for embedded test)
* Test processor
There is special command :ref:`cmd_test` to run tests from PlatformIO Project.
It allows to process specific environments or to ignore some tests using
"Glob patterns".
Also, is possible to ignore some tests for specific environment using
:ref:`projectconf_test_ignore` option from :ref:`projectconf`.
.. contents::
@ -55,38 +66,74 @@ PlatformIO Test System design is based on a few isolated components:
Workflow
--------
1. Create PlatformIO project using :ref:`cmd_init` command.
1. Create PlatformIO project using :ref:`cmd_init` command. For Local Unit
Testing (on the host machine), need to use :ref:`platform_native`.
.. code-block:: ini
; PlatformIO Project Configuration File
;
; Build options: build flags, source filter, extra scripting
; Upload options: custom port, speed and extra flags
; Library options: dependencies, extra library storages
;
; Please visit documentation for the other options and examples
; http://docs.platformio.org/en/stable/projectconf.html
;
; Embedded platforms
;
[env:uno]
platform = atmelavr
framework = arduino
board = uno
[env:nodemcu]
platform = espressif
framework = arduino
board = nodemcuv2
;
; Local (PC, native) platforms
;
[env:local]
platform = native
2. Place source code of main program to ``src`` directory.
3. Wrap ``main()`` or ``setup()/loop()`` methods of main program in ``UNIT_TEST``
guard:
.. code-block:: c
/**
/**
* Arduino Wiring-based Framework
*/
#ifndef UNIT_TEST
void setup () {
#ifndef UNIT_TEST
#include <Arduino.h>
void setup () {
// some code...
}
}
void loop () {
void loop () {
// some code...
}
#endif
}
#endif
/**
/**
* Generic C/C++
*/
#ifndef UNIT_TEST
int main() {
#ifndef UNIT_TEST
int main(int argc, char **argv) {
// setup code...
while (1) {
// loop code...
}
}
#endif
return 0
}
#endif
4. Create ``test`` directory in the root of project. See :ref:`projectconf_pio_test_dir`.
5. Write test using :ref:`unit_testing_api`. The each test is a small
@ -183,8 +230,8 @@ The summary of `Unity Test API <https://github.com/ThrowTheSwitch/Unity#unity-te
- ``TEST_ASSERT_EQUAL_MEMORY(expected, actual, len)``
Example
-------
Test "Blink" Project
--------------------
1. Please follow to :ref:`quickstart` and create "Blink Project". According
to the Unit Testing :ref:`unit_testing_design` it is the **Main program**.
@ -215,8 +262,15 @@ Source files
.. code-block:: ini
; Project Configuration File
; Docs: http://docs.platformio.org/en/latest/projectconf.html
; PlatformIO Project Configuration File
;
; Build options: build flags, source filter, extra scripting
; Upload options: custom port, speed and extra flags
; Library options: dependencies, extra library storages
;
; Please visit documentation for the other options and examples
; http://docs.platformio.org/en/stable/projectconf.html
[env:uno]
platform = atmelavr
@ -393,7 +447,11 @@ Test results
test:*/env:uno PASSED
========================= [PASSED] Took 13.35 seconds ========================
-------
Examples
--------
* `Embedded: Wiring Blink <https://github.com/platformio/platformio-examples/tree/develop/unit-testing/wiring-blink>`_
* `Local & Embedded: Calculator <https://github.com/platformio/platformio-examples/tree/develop/unit-testing/calculator>`_
For the other examples and source code please follow to
`PlatformIO Unit Testing Examples <https://github.com/platformio/platformio-examples/tree/feature/platformio-30/unit-testing>`_ repository.
`PlatformIO Unit Testing Examples <hhttps://github.com/platformio/platformio-examples/tree/develop/unit-testing>`_ repository.

10
docs/userguide/cmd_test.rst
View File

@ -45,10 +45,12 @@ Options
Process specified environments. More details :option:`platformio run --environment`
.. option::
--skip
-i, --ignore
Skip over tests where the name matches specified patterns. More than one
option/pattern is allowed.
Ignore tests where the name matches specified patterns. More than one
pattern is allowed. If you need to ignore some tests for the specific
environment, please take a look at :ref:`projectconf_test_ignore` option from
:ref:`projectconf`.
.. list-table::
:header-rows: 1
@ -68,7 +70,7 @@ option/pattern is allowed.
* - ``[!seq]``
- matches any character not in seq
For example, ``platformio test --skip "mytest*" -i "test[13]"``
For example, ``platformio test --ignore "mytest*" -i "test[13]"``
.. option::
--upload-port

2
platformio/__init__.py
View File

@ -14,7 +14,7 @@
import sys
VERSION = (3, 0, "0a3")
VERSION = (3, 0, "0a4")
__version__ = ".".join([str(s) for s in VERSION])
__title__ = "platformio"

13
platformio/builder/tools/piotest.py
View File

@ -43,6 +43,14 @@ FRAMEWORK_PARAMETERS = {
"serial_flush": "Serial.flush()",
"serial_begin": "Serial.begin(9600)",
"serial_end": "Serial.end()"
},
"native": {
"framework": "stdio.h",
"serial_obj": "",
"serial_putc": "putchar(a)",
"serial_flush": "fflush(stdout)",
"serial_begin": "",
"serial_end": ""
}
}
@ -114,7 +122,10 @@ void output_complete(void)
print("Warning: Could not remove temporary file '%s'. "
"Please remove it manually." % file_)
framework = env.subst("$PIOFRAMEWORK").lower()
if env['PIOPLATFORM'] == "native":
framework = "native"
else:
framework = env.subst("$PIOFRAMEWORK").lower()
if framework not in FRAMEWORK_PARAMETERS:
env.Exit("Error: %s framework doesn't support testing feature!" %
framework)

10
platformio/commands/init.py
View File

@ -80,9 +80,8 @@ def cli(ctx, # pylint: disable=R0913
click.echo("The next files/directories have been created in %s" %
click.style(
project_dir, fg="cyan"))
click.echo("%s - Project Configuration File" %
click.style(
"platformio.ini", fg="cyan"))
click.echo("%s - Project Configuration File" % click.style(
"platformio.ini", fg="cyan"))
click.echo("%s - Put your source files here" % click.style(
"src", fg="cyan"))
click.echo("%s - Put here project specific (private) libraries" %
@ -273,10 +272,7 @@ def init_ci_conf(project_dir):
def init_cvs_ignore(project_dir):
ignore_path = join(project_dir, ".gitignore")
default = [
".pioenvs\n",
".piolibdeps\n"
]
default = [".pioenvs\n", ".piolibdeps\n"]
current = []
if isfile(ignore_path):
with open(ignore_path) as fp:

16
platformio/commands/run.py
View File

@ -111,14 +111,14 @@ def cli(ctx, # pylint: disable=R0913,R0914
class EnvironmentProcessor(object):
KNOWN_OPTIONS = ("platform", "framework", "board", "board_mcu",
"board_f_cpu", "board_f_flash", "board_flash_mode",
"build_flags", "src_build_flags", "build_unflags",
"src_filter", "extra_script", "targets", "upload_port",
"upload_protocol", "upload_speed", "upload_flags",
"upload_resetmethod", "lib_install", "lib_deps",
"lib_force", "lib_ignore", "lib_extra_dirs",
"lib_ldf_mode", "lib_compat_mode", "piotest")
KNOWN_OPTIONS = (
"platform", "framework", "board", "board_mcu", "board_f_cpu",
"board_f_flash", "board_flash_mode", "build_flags", "src_build_flags",
"build_unflags", "src_filter", "extra_script", "targets",
"upload_port", "upload_protocol", "upload_speed", "upload_flags",
"upload_resetmethod", "lib_install", "lib_deps", "lib_force",
"lib_ignore", "lib_extra_dirs", "lib_ldf_mode", "lib_compat_mode",
"test_ignore", "piotest")
REMAPED_OPTIONS = {"framework": "pioframework", "platform": "pioplatform"}

109
platformio/commands/test.py
View File

@ -30,7 +30,7 @@ from platformio.managers.platform import PlatformFactory
@click.command("test", short_help="Unit Testing")
@click.option("--environment", "-e", multiple=True, metavar="<environment>")
@click.option("--skip", multiple=True, metavar="<pattern>")
@click.option("--ignore", "-i", multiple=True, metavar="<pattern>")
@click.option("--upload-port", metavar="<upload port>")
@click.option(
"-d",
@ -44,7 +44,7 @@ from platformio.managers.platform import PlatformFactory
resolve_path=True))
@click.option("--verbose", "-v", is_flag=True)
@click.pass_context
def cli(ctx, environment, skip, upload_port, project_dir, verbose):
def cli(ctx, environment, ignore, upload_port, project_dir, verbose):
with util.cd(project_dir):
test_dir = util.get_projecttest_dir()
if not isdir(test_dir):
@ -59,19 +59,30 @@ def cli(ctx, environment, skip, upload_port, project_dir, verbose):
start_time = time()
results = []
for testname in test_names:
for envname in projectconf.sections():
if not envname.startswith("env:"):
for section in projectconf.sections():
if not section.startswith("env:"):
continue
envname = envname[4:]
envname = section[4:]
if environment and envname not in environment:
continue
# check skip patterns
if testname != "*" and any([fnmatch(testname, p) for p in skip]):
# check ignore patterns
_ignore = list(ignore)
if projectconf.has_option(section, "test_ignore"):
_ignore.extend([p.strip()
for p in projectconf.get(
section, "test_ignore").split("\n")
if p.strip()])
if testname != "*" and \
any([fnmatch(testname, p) for p in _ignore]):
results.append((None, testname, envname))
continue
tp = TestProcessor(ctx, testname, envname, {
cls = (LocalTestProcessor
if projectconf.get(section, "platform") == "native" else
EmbeddedTestProcessor)
tp = cls(ctx, testname, envname, {
"project_config": projectconf,
"project_dir": project_dir,
"upload_port": upload_port,
@ -93,7 +104,7 @@ def cli(ctx, environment, skip, upload_port, project_dir, verbose):
status_str = click.style("IGNORED", fg="yellow")
click.echo(
"test:%s/env:%s\t%s" % (click.style(
"test:%s/env:%s\t[%s]" % (click.style(
testname, fg="yellow"), click.style(
envname, fg="cyan"), status_str),
err=status is False)
@ -108,10 +119,7 @@ def cli(ctx, environment, skip, upload_port, project_dir, verbose):
raise exception.ReturnErrorCode()
class TestProcessor(object):
SERIAL_TIMEOUT = 600
SERIAL_BAUDRATE = 9600
class TestProcessorBase(object):
def __init__(self, cmd_ctx, testname, envname, options):
self.cmd_ctx = cmd_ctx
@ -119,24 +127,16 @@ class TestProcessor(object):
self.test_name = testname
self.env_name = envname
self.options = options
self._run_failed = False
def process(self):
self._progress("Building... (1/3)")
self._build_or_upload(["test"])
self._progress("Uploading... (2/3)")
self._build_or_upload(["test", "upload"])
self._progress("Testing... (3/3)")
sleep(1.0) # wait while board is starting...
return self._run_hardware_test()
def _progress(self, text, is_error=False):
def print_progress(self, text, is_error=False):
print_header(
"[test::%s] %s" % (click.style(
self.test_name, fg="yellow", bold=True), text),
is_error=is_error)
click.echo()
def _build_or_upload(self, target):
def build_or_upload(self, target):
if self.test_name != "*":
self.cmd_ctx.meta['piotest'] = self.test_name
return self.cmd_ctx.invoke(
@ -147,7 +147,54 @@ class TestProcessor(object):
environment=[self.env_name],
target=target)
def _run_hardware_test(self):
def run(self):
raise NotImplementedError
def on_run_out(self, line):
if line.endswith(":PASS"):
click.echo("%s\t[%s]" % (line[:-5], click.style(
"PASSED", fg="green")))
elif ":FAIL:" in line:
self._run_failed = True
click.echo("%s\t[%s]" % (line, click.style("FAILED", fg="red")))
else:
click.echo(line)
class LocalTestProcessor(TestProcessorBase):
def process(self):
self.print_progress("Building... (1/2)")
self.build_or_upload(["test"])
self.print_progress("Testing... (2/2)")
return self.run()
def run(self):
with util.cd(self.options['project_dir']):
pioenvs_dir = util.get_projectpioenvs_dir()
result = util.exec_command(
[join(pioenvs_dir, self.env_name, "program")],
stdout=util.AsyncPipe(self.on_run_out),
stderr=util.AsyncPipe(self.on_run_out))
assert "returncode" in result
return result['returncode'] == 0 and not self._run_failed
class EmbeddedTestProcessor(TestProcessorBase):
SERIAL_TIMEOUT = 600
SERIAL_BAUDRATE = 9600
def process(self):
self.print_progress("Building... (1/3)")
self.build_or_upload(["test"])
self.print_progress("Uploading... (2/3)")
self.build_or_upload(["test", "upload"])
self.print_progress("Testing... (3/3)")
sleep(1.0) # wait while board is starting...
return self.run()
def run(self):
click.echo("If you don't see any output for the first 10 secs, "
"please reset board (press reset button)")
click.echo()
@ -155,23 +202,15 @@ class TestProcessor(object):
self.get_serial_port(),
self.SERIAL_BAUDRATE,
timeout=self.SERIAL_TIMEOUT)
passed = True
while True:
line = ser.readline().strip()
if not line:
continue
if line.endswith(":PASS"):
click.echo("%s\t%s" % (line[:-5], click.style(
"PASSED", fg="green")))
elif ":FAIL:" in line:
passed = False
click.echo("%s\t%s" % (line, click.style("FAILED", fg="red")))
else:
click.echo(line)
self.on_run_out(line)
if all([l in line for l in ("Tests", "Failures", "Ignored")]):
break
ser.close()
return passed
return not self._run_failed
def get_serial_port(self):
config = self.options['project_config']

11
platformio/managers/platform.py
View File

@ -362,14 +362,15 @@ class PlatformBase(PlatformPackagesMixin, PlatformRunMixin):
@property
def packages(self):
packages = self._manifest.get("packages", {})
if "tool-scons" not in packages:
packages['tool-scons'] = {
if "packages" not in self._manifest:
self._manifest['packages'] = {}
if "tool-scons" not in self._manifest['packages']:
self._manifest['packages']['tool-scons'] = {
"version": self._manifest.get("engines", {}).get(
"scons", ">=2.3.0,<2.6.0"),
"optional": False
}
return packages
return self._manifest['packages']
def get_dir(self):
return dirname(self.manifest_path)
@ -476,7 +477,7 @@ class PlatformBase(PlatformPackagesMixin, PlatformRunMixin):
if "test" in targets and "tool-unity" not in self.packages:
self.packages['tool-unity'] = {
"version": "~1.20302.0",
"version": "~1.20302.1",
"optional": False
}

26
tests/commands/test_test.py Normal file
View File

@ -0,0 +1,26 @@
# Copyright 2014-present PlatformIO <contact@platformio.org>
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from os.path import join
from platformio.commands.test import cli as cli_test
def test_local_env(clirunner, validate_cliresult):
result = clirunner.invoke(
cli_test,
["-d", join("examples", "unit-testing", "calculator"), "-e", "local"])
result.exit_code == -1
assert all(
[s in result.output for s in ("[PASSED]", "[IGNORED]", "[FAILED]")])