Merge branch 'release/v3.3.0'

.github/ISSUE_TEMPLATE.md

@@ -7,7 +7,9 @@ What kind of issue is this?
 - [ ] PlatformIO IDE. All issues related to PlatformIO IDE should be reported to appropriate repository
       https://github.com/platformio/platformio-atom-ide/issues
 
-- [ ] Development Platform. All issues related to Development Platform should be reported to appropriate repository. Search it using link below
+- [ ] Development Platform or Board. All issues related to Development Platforms or Embedded Boards
+      should be reported to appropriate repository. 
+      See full list with repositories and search for "platform-xxx" repository related to your hardware
       https://github.com/platformio?query=platform-
 
 - [ ] Feature Request. Start by telling us what problem you’re trying to solve. Often a solution

.isort.cfg

@@ -1,3 +1,3 @@
 [settings]
 line_length=79
-known_third_party=bottle,click,lockfile,pytest,requests,semantic_version,serial,SCons
+known_third_party=arrow,bottle,click,lockfile,pytest,requests,SCons,semantic_version,serial

.travis.yml

@@ -16,7 +16,7 @@ matrix:
       env: TOX_ENV=py27
     - os: osx
       language: generic
-      env: TOX_ENV=py27
+      env: TOX_ENV=skipexamples
 
 install:
   - git submodule update --init --recursive

CONTRIBUTING.md

@@ -1,7 +1,7 @@
 Contributing
 ------------
 
-To get started, <a href="https://www.clahub.com/agreements/platformio/platformio">sign the Contributor License Agreement</a>.
+To get started, <a href="https://www.clahub.com/agreements/platformio/platformio-core">sign the Contributor License Agreement</a>.
 
 1. Fork the repository on GitHub.
 2. Make a branch off of ``develop``

README.rst

@@ -7,8 +7,8 @@ PlatformIO
 .. image:: https://ci.appveyor.com/api/projects/status/unnpw0n3c5k14btn/branch/develop?svg=true
     :target: https://ci.appveyor.com/project/ivankravets/platformio-core
     :alt: AppVeyor.CI Build Status
-.. image:: https://requires.io/github/platformio/platformio/requirements.svg?branch=develop
-    :target: https://requires.io/github/platformio/platformio/requirements/?branch=develop
+.. image:: https://requires.io/github/platformio/platformio-core/requirements.svg?branch=develop
+    :target: https://requires.io/github/platformio/platformio-core/requirements/?branch=develop
     :alt: Requirements Status
 .. image:: https://img.shields.io/pypi/v/platformio.svg
     :target: https://pypi.python.org/pypi/platformio/
@@ -26,7 +26,7 @@ PlatformIO
 **Quick Links:** `Home Page <http://platformio.org>`_ |
 `PlatformIO Plus <https://pioplus.com>`_ |
 `PlatformIO IDE <http://platformio.org/platformio-ide>`_ |
-`Project Examples <https://github.com/platformio/platformio-examples/tree/develop>`_ |
+`Project Examples <https://github.com/platformio/platformio-examples/>`_ |
 `Docs <http://docs.platformio.org>`_ |
 `Donate <http://platformio.org/donate>`_ |
 `Contact Us <https://pioplus.com/contact.html>`_
@@ -98,13 +98,13 @@ settings for most popular `Embedded Boards <http://platformio.org/boards>`_.
 
 * Colourful `command-line output <https://raw.githubusercontent.com/platformio/platformio/develop/examples/platformio-examples.png>`_
 * `IDE Integration <http://docs.platformio.org/en/stable/ide.html>`_ with
-  *Arduino, Atom, Eclipse, Emacs, Energia, Qt Creator, Sublime Text, Vim, Visual Studio*
+  *Cloud9, Codeanywhere, Eclipse Che, Atom, CLion, CodeBlocks, Eclipse, Emacs, NetBeans, Qt Creator, Sublime Text, Vim, Visual Studio*
 * Cloud compiling and `Continuous Integration <http://docs.platformio.org/en/stable/ci/index.html>`_
   with *AppVeyor, Circle CI, Drone, Shippable, Travis CI*
 * Built-in `Serial Port Monitor <http://docs.platformio.org/en/stable/userguide/cmd_serialports.html#platformio-serialports-monitor>`_ and configurable
   `build -flags/-options <http://docs.platformio.org/en/stable/projectconf.html#build-flags>`_
 * Automatic **firmware uploading**
-* Pre-built tool chains, frameworks for the popular `Hardware Platforms <http://platformio.org/platforms>`_
+* Pre-built tool chains, frameworks for the popular `development platforms <http://platformio.org/platforms>`_
 
 .. image:: https://raw.githubusercontent.com/platformio/platformio-web/develop/app/images/platformio-embedded-development.png
     :target: http://platformio.org
@@ -116,12 +116,11 @@ The Missing Library Manager. *It's here!*
 platforms which allows you to organize and have up-to-date external libraries.
 
 * Friendly `Command-Line Interface <http://docs.platformio.org/en/stable/librarymanager/index.html>`_
-* Modern `Web 2.0 Library Search <http://platformio.org/lib>`_
+* Modern `Web 2.0 Library Portal <http://platformio.org/lib>`_
 * Open Source `Library Registry API <https://github.com/platformio/platformio-api>`_
 * Library Crawler based on `library.json <http://docs.platformio.org/en/stable/librarymanager/config.html>`_
   specification
-* Library **dependency management**
-* Automatic library updating
+* Project Dependency Manager with `Semantic Versioning <http://docs.platformio.org/page/librarymanager/index.html>`_ requirements
 
 .. image:: https://raw.githubusercontent.com/platformio/platformio-web/develop/app/images/platformio-library-manager.png
     :target: http://platformio.org
@@ -158,7 +157,8 @@ It has support for the most popular embedded platforms:
 
 * `Atmel AVR <http://platformio.org/platforms/atmelavr>`_
 * `Atmel SAM <http://platformio.org/platforms/atmelsam>`_
-* `Espressif <http://platformio.org/platforms/espressif>`_
+* `Espressif 32 <http://platformio.org/platforms/espressif32>`_
+* `Espressif 8266 <http://platformio.org/platforms/espressif8266>`_
 * `Freescale Kinetis <http://platformio.org/platforms/freescalekinetis>`_
 * `Intel ARC32 <http://platformio.org/platforms/intel_arc32>`_
 * `Lattice iCE40 <http://platformio.org/platforms/lattice_ice40>`_
@@ -169,15 +169,18 @@ It has support for the most popular embedded platforms:
 * `Silicon Labs EFM32 <http://platformio.org/platforms/siliconlabsefm32>`_
 * `Teensy <http://platformio.org/platforms/teensy>`_
 * `TI MSP430 <http://platformio.org/platforms/timsp430>`_
-* `TI TIVA C <http://platformio.org/platforms/titiva>`_
+* `TI TivaVA C <http://platformio.org/platforms/titiva>`_
 
 Frameworks:
 
 * `Arduino <http://platformio.org/frameworks/arduino>`_
+* `ARTIK SDK <http://platformio.org/frameworks/artik-sdk>`_
 * `CMSIS <http://platformio.org/frameworks/cmsis>`_
 * `Energia <http://platformio.org/frameworks/energia>`_
+* `ESP-IDF <http://platformio.org/frameworks/espidf>`_
 * `libOpenCM3 <http://platformio.org/frameworks/libopencm3>`_
 * `mbed <http://platformio.org/frameworks/mbed>`_
+* `Pumbaa <http://platformio.org/frameworks/pumbaa>`_
 * `Simba <http://platformio.org/frameworks/simba>`_
 * `SPL <http://platformio.org/frameworks/spl>`_
 * `WiringPi <http://platformio.org/frameworks/wiringpi>`_

platformio/__init__.py

@@ -14,7 +14,7 @@
 
 import sys
 
-VERSION = (3, 2, 1)
+VERSION = (3, 3, 0)
 __version__ = ".".join([str(s) for s in VERSION])
 
 __title__ = "platformio"

platformio/app.py

@@ -137,8 +137,6 @@ class ContentCache(object):
             return
 
         self.cache_dir = cache_dir or join(util.get_home_dir(), ".cache")
-        if not self.cache_dir:
-            os.makedirs(self.cache_dir)
         self._db_path = join(self.cache_dir, "db.data")
 
     def __enter__(self):
@@ -152,7 +150,7 @@ class ContentCache(object):
                     continue
                 line = line.strip()
                 expire, path = line.split("=")
-                if time() < int(expire):
+                if time() < int(expire) and isfile(path):
                     newlines.append(line)
                     continue
                 found = True
@@ -172,6 +170,8 @@ class ContentCache(object):
         pass
 
     def _lock_dbindex(self):
+        if not self.cache_dir:
+            os.makedirs(self.cache_dir)
         self._lockfile = LockFile(self.cache_dir)
         if self._lockfile.is_locked() and \
                 (time() - getmtime(self._lockfile.lock_file)) > 10:
@@ -200,19 +200,17 @@ class ContentCache(object):
         return h.hexdigest()
 
     def get(self, key):
-        if not self.cache_dir:
-            return None
         cache_path = self.get_cache_path(key)
         if not isfile(cache_path):
             return None
         with open(cache_path, "rb") as fp:
             data = fp.read()
-            if data[0] in ("{", "["):
+            if data and data[0] in ("{", "["):
                 return json.loads(data)
             return data
 
     def set(self, key, data, valid):
-        if not self.cache_dir or not data:
+        if not data:
             return
         if not isdir(self.cache_dir):
             os.makedirs(self.cache_dir)
@@ -238,8 +236,14 @@ class ContentCache(object):
         return True
 
     def clean(self):
-        if self.cache_dir and isdir(self.cache_dir):
-            util.rmtree_(self.cache_dir)
+        if not self.cache_dir or not isdir(self.cache_dir):
+            return
+        util.rmtree_(self.cache_dir)
+
+
+def clean_cache():
+    with ContentCache() as cc:
+        cc.clean()
 
 
 def sanitize_setting(name, value):
@@ -325,7 +329,7 @@ def get_cid():
             except:  # pylint: disable=bare-except
                 pass
         cid = str(
-            uuid.UUID(bytes=hashlib.md5(
-                str(_uid if _uid else uuid.getnode())).digest()))
+            uuid.UUID(bytes=hashlib.md5(str(_uid if _uid else uuid.getnode()))
+                      .digest()))
         set_state_item("cid", cid)
     return cid

platformio/builder/main.py

@@ -117,8 +117,13 @@ elif not int(ARGUMENTS.get("PIOVERBOSE", 0)):
 for var in ("BUILD_FLAGS", "SRC_BUILD_FLAGS", "SRC_FILTER", "EXTRA_SCRIPT",
             "UPLOAD_PORT", "UPLOAD_FLAGS", "LIB_EXTRA_DIRS"):
     k = "PLATFORMIO_%s" % var
-    if environ.get(k):
+    if k not in environ:
+        continue
+    if var in ("UPLOAD_PORT", "EXTRA_SCRIPT") or not env.get(var):
         env[var] = environ.get(k)
+    else:
+        env[var] = "%s%s%s" % (environ.get(k), ", "
+                               if var == "LIB_EXTRA_DIRS" else " ", env[var])
 
 # Parse comma separated items
 for opt in ("PIOFRAMEWORK", "LIB_DEPS", "LIB_IGNORE", "LIB_EXTRA_DIRS"):

platformio/builder/tools/piolib.py

@@ -46,9 +46,8 @@ class LibBuilderFactory(object):
             elif used_frameworks:
                 clsname = "%sLibBuilder" % used_frameworks[0].title()
 
-        obj = getattr(sys.modules[__name__], clsname)(env,
-                                                      path,
-                                                      verbose=verbose)
+        obj = getattr(sys.modules[__name__], clsname)(
+            env, path, verbose=verbose)
         assert isinstance(obj, LibBuilderBase)
         return obj
 
@@ -571,7 +570,7 @@ class PlatformIOLibBuilder(LibBuilderBase):
             inc_dirs.append(join(self.path, "utility"))
 
         for path in self.env.get("CPPPATH", []):
-            if path not in self.envorigin['CPPPATH']:
+            if path not in self.envorigin.get("CPPPATH", []):
                 inc_dirs.append(self.env.subst(path))
         return inc_dirs
 
@@ -591,8 +590,8 @@ def GetLibBuilders(env):  # pylint: disable=too-many-branches
             if verbose:
                 sys.stderr.write("Ignored library %s\n" % lb.path)
             return
-        if compat_mode > 1 and not lb.is_platforms_compatible(env[
-                'PIOPLATFORM']):
+        if compat_mode > 1 and not lb.is_platforms_compatible(
+                env['PIOPLATFORM']):
             if verbose:
                 sys.stderr.write("Platform incompatible library %s\n" %
                                  lb.path)
@@ -614,9 +613,8 @@ def GetLibBuilders(env):  # pylint: disable=too-many-branches
             if item == "__cores__" or not isdir(join(libs_dir, item)):
                 continue
             try:
-                lb = LibBuilderFactory.new(env,
-                                           join(libs_dir, item),
-                                           verbose=verbose)
+                lb = LibBuilderFactory.new(
+                    env, join(libs_dir, item), verbose=verbose)
             except ValueError:
                 if verbose:
                     sys.stderr.write("Skip library with broken manifest: %s\n"

platformio/builder/tools/piomisc.py

@@ -32,6 +32,7 @@ from platformio import util
 class InoToCPPConverter(object):
 
     PROTOTYPE_RE = re.compile(r"""^(
+        (?:template\<.*\>\s*)?      # template
         ([a-z_\d]+\*?\s+){1,2}      # return type
         ([a-z_\d]+\s*)              # name of prototype
         \([a-z_,\.\*\&\[\]\s\d]*\)  # arguments
@@ -180,8 +181,9 @@ class InoToCPPConverter(object):
 
 
 def ConvertInoToCpp(env):
-    ino_nodes = (env.Glob(join("$PROJECTSRC_DIR", "*.ino")) +
-                 env.Glob(join("$PROJECTSRC_DIR", "*.pde")))
+    src_dir = util.glob_escape(env.subst("$PROJECTSRC_DIR"))
+    ino_nodes = (
+        env.Glob(join(src_dir, "*.ino")) + env.Glob(join(src_dir, "*.pde")))
     if not ino_nodes:
         return
     c = InoToCPPConverter(env)
@@ -215,7 +217,7 @@ def DumpIDEData(env):
         for name in p.get_installed_packages():
             if p.get_package_type(name) != "toolchain":
                 continue
-            toolchain_dir = p.get_package_dir(name)
+            toolchain_dir = util.glob_escape(p.get_package_dir(name))
             toolchain_incglobs = [
                 join(toolchain_dir, "*", "include*"),
                 join(toolchain_dir, "lib", "gcc", "*", "*", "include*")

platformio/builder/tools/piotest.py

@@ -16,14 +16,15 @@ from __future__ import absolute_import
 
 from os.path import join, sep
 
+from platformio.managers.core import get_core_package_dir
+
 
 def ProcessTest(env):
     env.Append(
         CPPDEFINES=["UNIT_TEST", "UNITY_INCLUDE_CONFIG_H"],
         CPPPATH=[join("$BUILD_DIR", "UnityTestLib")])
     unitylib = env.BuildLibrary(
-        join("$BUILD_DIR", "UnityTestLib"),
-        env.PioPlatform().get_package_dir("tool-unity"))
+        join("$BUILD_DIR", "UnityTestLib"), get_core_package_dir("tool-unity"))
     env.Prepend(LIBS=[unitylib])
 
     src_filter = None

platformio/builder/tools/platformio.py

@@ -25,7 +25,7 @@ from SCons.Script import (COMMAND_LINE_TARGETS, AlwaysBuild,
                           DefaultEnvironment, SConscript)
 from SCons.Util import case_sensitive_suffixes, is_Sequence
 
-from platformio.util import pioversion_to_intstr
+from platformio.util import glob_escape, pioversion_to_intstr
 
 SRC_BUILD_EXT = ["c", "cpp", "S", "spp", "SPP", "sx", "s", "asm", "ASM"]
 SRC_HEADER_EXT = ["h", "hpp"]
@@ -191,7 +191,7 @@ def MatchSourceFiles(env, src_dir, src_filter=None):
     src_filter = src_filter.replace("/", sep).replace("\\", sep)
     for (action, pattern) in SRC_FILTER_PATTERNS_RE.findall(src_filter):
         items = set()
-        for item in glob(join(src_dir, pattern)):
+        for item in glob(join(glob_escape(src_dir), pattern)):
             if isdir(item):
                 for root, _, files in walk(item, followlinks=True):
                     for f in files:
@@ -266,8 +266,7 @@ def BuildLibrary(env, variant_dir, src_dir, src_filter=None):
     lib = env.Clone()
     return lib.StaticLibrary(
         lib.subst(variant_dir),
-        lib.CollectBuildFiles(
-            variant_dir, src_dir, src_filter=src_filter))
+        lib.CollectBuildFiles(variant_dir, src_dir, src_filter=src_filter))
 
 
 def BuildSources(env, variant_dir, src_dir, src_filter=None):

platformio/commands/account.py

@@ -18,7 +18,7 @@ import sys
 
 import click
 
-from platformio.pioplus import pioplus_call
+from platformio.managers.core import pioplus_call
 
 
 @click.group("account", short_help="Manage PIO Account")

platformio/commands/boards.py

@@ -20,72 +20,63 @@ from platformio.exception import APIRequestError, InternetIsOffline
 from platformio.managers.platform import PlatformManager
 
 
-@click.command("boards", short_help="Pre-configured Embedded Boards")
+@click.command("boards", short_help="Embedded Board Explorer")
 @click.argument("query", required=False)
 @click.option("--installed", is_flag=True)
 @click.option("--json-output", is_flag=True)
 def cli(query, installed, json_output):  # pylint: disable=R0912
     if json_output:
-        return _ouput_boards_json(query, installed)
-
-    BOARDLIST_TPL = ("{type:<30} {mcu:<14} {frequency:<8} "
-                     " {flash:<7} {ram:<6} {name}")
-    terminal_width, _ = click.get_terminal_size()
+        return _print_boards_json(query, installed)
 
     grpboards = {}
     for board in _get_boards(installed):
+        if query and query.lower() not in json.dumps(board).lower():
+            continue
         if board['platform'] not in grpboards:
             grpboards[board['platform']] = []
         grpboards[board['platform']].append(board)
 
-    for (platform, pboards) in sorted(grpboards.items()):
-        if query:
-            search_data = json.dumps(pboards).lower()
-            if query.lower() not in search_data.lower():
-                continue
-
+    terminal_width, _ = click.get_terminal_size()
+    for (platform, boards) in sorted(grpboards.items()):
         click.echo("")
         click.echo("Platform: ", nl=False)
         click.secho(platform, bold=True)
         click.echo("-" * terminal_width)
+        print_boards(boards)
+
+
+def print_boards(boards):
+    terminal_width, _ = click.get_terminal_size()
+    BOARDLIST_TPL = ("{type:<30} {mcu:<14} {frequency:<8} "
+                     " {flash:<7} {ram:<6} {name}")
+    click.echo(
+        BOARDLIST_TPL.format(
+            type=click.style("ID", fg="cyan"),
+            mcu="MCU",
+            frequency="Frequency",
+            flash="Flash",
+            ram="RAM",
+            name="Name"))
+    click.echo("-" * terminal_width)
+
+    for board in boards:
+        ram_size = board['ram']
+        if ram_size >= 1024:
+            if ram_size % 1024:
+                ram_size = "%.1fkB" % (ram_size / 1024.0)
+            else:
+                ram_size = "%dkB" % (ram_size / 1024)
+        else:
+            ram_size = "%dB" % ram_size
+
         click.echo(
             BOARDLIST_TPL.format(
-                type=click.style(
-                    "ID", fg="cyan"),
-                mcu="MCU",
-                frequency="Frequency",
-                flash="Flash",
-                ram="RAM",
-                name="Name"))
-        click.echo("-" * terminal_width)
-
-        for board in sorted(pboards, key=lambda b: b['id']):
-            if query:
-                search_data = "%s %s" % (board['id'],
-                                         json.dumps(board).lower())
-                if query.lower() not in search_data.lower():
-                    continue
-
-            flash_size = "%dkB" % (board['rom'] / 1024)
-
-            ram_size = board['ram']
-            if ram_size >= 1024:
-                if ram_size % 1024:
-                    ram_size = "%.1fkB" % (ram_size / 1024.0)
-                else:
-                    ram_size = "%dkB" % (ram_size / 1024)
-            else:
-                ram_size = "%dB" % ram_size
-
-            click.echo(
-                BOARDLIST_TPL.format(
-                    type=click.style(
-                        board['id'], fg="cyan"),
-                    mcu=board['mcu'],
-                    frequency="%dMhz" % (board['fcpu'] / 1000000),
-                    flash=flash_size,
-                    ram=ram_size,
-                    name=board['name']))
+                type=click.style(board['id'], fg="cyan"),
+                mcu=board['mcu'],
+                frequency="%dMhz" % (board['fcpu'] / 1000000),
+                flash="%dkB" % (board['rom'] / 1024),
+                ram=ram_size,
+                name=board['name']))
 
 
 def _get_boards(installed=False):
@@ -99,10 +90,10 @@ def _get_boards(installed=False):
                     boards.append(board)
         except InternetIsOffline:
             pass
-    return boards
+    return sorted(boards, key=lambda b: b['name'])
 
 
-def _ouput_boards_json(query, installed=False):
+def _print_boards_json(query, installed=False):
     result = []
     try:
         boards = _get_boards(installed)

platformio/commands/ci.py

@@ -152,7 +152,7 @@ def _copy_contents(dst_dir, contents):
 def _exclude_contents(dst_dir, patterns):
     contents = []
     for p in patterns:
-        contents += glob(join(dst_dir, p))
+        contents += glob(join(util.glob_escape(dst_dir), p))
     for path in contents:
         path = abspath(path)
         if isdir(path):

platformio/commands/device.py

@@ -61,12 +61,12 @@ def device_list(json_output):
 @click.option(
     "--rts",
     default=None,
-    type=click.Choice(["0", "1"]),
+    type=click.IntRange(0, 1),
     help="Set initial RTS line state")
 @click.option(
     "--dtr",
     default=None,
-    type=click.Choice(["0", "1"]),
+    type=click.IntRange(0, 1),
     help="Set initial DTR line state")
 @click.option("--echo", is_flag=True, help="Enable local echo, default=Off")
 @click.option(

platformio/commands/init.py

@@ -57,6 +57,7 @@ def validate_boards(ctx, param, value):  # pylint: disable=W0613
     "--ide", type=click.Choice(ProjectGenerator.get_supported_ides()))
 @click.option("-O", "--project-option", multiple=True)
 @click.option("--env-prefix", default="")
+@click.option("-s", "--silent", is_flag=True)
 @click.pass_context
 def cli(
         ctx,  # pylint: disable=R0913
@@ -64,28 +65,29 @@ def cli(
         board,
         ide,
         project_option,
-        env_prefix):
+        env_prefix,
+        silent):
 
-    if project_dir == getcwd():
-        click.secho("\nThe current working directory", fg="yellow", nl=False)
-        click.secho(" %s " % project_dir, fg="cyan", nl=False)
-        click.secho(
-            "will be used for project.\n"
-            "You can specify another project directory via\n"
-            "`platformio init -d %PATH_TO_THE_PROJECT_DIR%` command.",
-            fg="yellow")
-        click.echo("")
+    if not silent:
+        if project_dir == getcwd():
+            click.secho(
+                "\nThe current working directory", fg="yellow", nl=False)
+            click.secho(" %s " % project_dir, fg="cyan", nl=False)
+            click.secho(
+                "will be used for project.\n"
+                "You can specify another project directory via\n"
+                "`platformio init -d %PATH_TO_THE_PROJECT_DIR%` command.",
+                fg="yellow")
+            click.echo("")
 
-    click.echo("The next files/directories have been created in %s" %
-               click.style(
-                   project_dir, fg="cyan"))
-    click.echo("%s - Project Configuration File" % click.style(
-        "platformio.ini", fg="cyan"))
-    click.echo("%s - Put your source files here" % click.style(
-        "src", fg="cyan"))
-    click.echo("%s - Put here project specific (private) libraries" %
-               click.style(
-                   "lib", fg="cyan"))
+        click.echo("The next files/directories have been created in %s" %
+                   click.style(project_dir, fg="cyan"))
+        click.echo("%s - Project Configuration File" % click.style(
+            "platformio.ini", fg="cyan"))
+        click.echo("%s - Put your source files here" % click.style(
+            "src", fg="cyan"))
+        click.echo("%s - Put here project specific (private) libraries" %
+                   click.style("lib", fg="cyan"))
 
     init_base_project(project_dir)
 
@@ -111,16 +113,17 @@ def cli(
         pg = ProjectGenerator(project_dir, ide, board[0])
         pg.generate()
 
-    click.secho(
-        "\nProject has been successfully initialized!\nUseful commands:\n"
-        "`platformio run` - process/build project from the current "
-        "directory\n"
-        "`platformio run --target upload` or `platformio run -t upload` "
-        "- upload firmware to embedded board\n"
-        "`platformio run --target clean` - clean project (remove compiled "
-        "files)\n"
-        "`platformio run --help` - additional information",
-        fg="green")
+    if not silent:
+        click.secho(
+            "\nProject has been successfully initialized!\nUseful commands:\n"
+            "`platformio run` - process/build project from the current "
+            "directory\n"
+            "`platformio run --target upload` or `platformio run -t upload` "
+            "- upload firmware to embedded board\n"
+            "`platformio run --target clean` - clean project (remove compiled "
+            "files)\n"
+            "`platformio run --help` - additional information",
+            fg="green")
 
 
 def get_first_board(project_dir):

platformio/commands/lib.py

@@ -12,14 +12,19 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-import json
-from os.path import join
-from time import sleep
+# pylint: disable=too-many-branches, too-many-locals
 
+import json
+from os.path import isdir, join
+from time import sleep
+from urllib import quote
+
+import arrow
 import click
 
 from platformio import exception, util
 from platformio.managers.lib import LibraryManager
+from platformio.managers.platform import PlatformFactory, PlatformManager
 from platformio.util import get_api_result
 
 
@@ -43,8 +48,9 @@ from platformio.util import get_api_result
     help="Manage custom library storage")
 @click.pass_context
 def cli(ctx, **options):
+    non_storage_cmds = ("search", "show", "register", "stats", "builtin")
     # skip commands that don't need storage folder
-    if ctx.invoked_subcommand in ("search", "register") or \
+    if ctx.invoked_subcommand in non_storage_cmds or \
             (len(ctx.args) == 2 and ctx.args[1] in ("-h", "--help")):
         return
     storage_dir = options['storage_dir']
@@ -106,57 +112,69 @@ def lib_uninstall(lm, libraries):
     "--only-check",
     is_flag=True,
     help="Do not update, only check for new version")
+@click.option("--json-output", is_flag=True)
 @click.pass_obj
-def lib_update(lm, libraries, only_check):
+def lib_update(lm, libraries, only_check, json_output):
     if not libraries:
-        libraries = [str(m.get("id", m['name'])) for m in lm.get_installed()]
-    for library in libraries:
-        lm.update(library, only_check=only_check)
+        libraries = [manifest['__pkg_dir'] for manifest in lm.get_installed()]
+
+    if only_check and json_output:
+        result = []
+        for library in libraries:
+            pkg_dir = library if isdir(library) else None
+            requirements = None
+            url = None
+            if not pkg_dir:
+                name, requirements, url = lm.parse_pkg_input(library)
+                pkg_dir = lm.get_package_dir(name, requirements, url)
+            if not pkg_dir:
+                continue
+            latest = lm.outdated(pkg_dir, requirements)
+            if not latest:
+                continue
+            manifest = lm.load_manifest(pkg_dir)
+            manifest['versionLatest'] = latest
+            result.append(manifest)
+        return click.echo(json.dumps(result))
+    else:
+        for library in libraries:
+            lm.update(library, only_check=only_check)
 
 
-#######
+def print_lib_item(item):
+    click.secho(item['name'], fg="cyan")
+    click.echo("=" * len(item['name']))
+    if "id" in item:
+        click.secho("#ID: %d" % item['id'], bold=True)
+    if "description" in item or "url" in item:
+        click.echo(item.get("description", item.get("url", "")))
+    click.echo()
 
-LIBLIST_TPL = ("[{id:^14}] {name:<25} {compatibility:<30} "
-               "\"{authornames}\": {description}")
+    for key in ("version", "homepage", "license", "keywords"):
+        if key not in item or not item[key]:
+            continue
+        if isinstance(item[key], list):
+            click.echo("%s: %s" % (key.title(), ", ".join(item[key])))
+        else:
+            click.echo("%s: %s" % (key.title(), item[key]))
+
+    for key in ("frameworks", "platforms"):
+        if key not in item:
+            continue
+        click.echo("Compatible %s: %s" % (key, ", ".join(
+            [i['title'] if isinstance(i, dict) else i for i in item[key]])))
+
+    if "authors" in item or "authornames" in item:
+        click.echo("Authors: %s" % ", ".join(
+            item.get("authornames",
+                     [a.get("name", "") for a in item.get("authors", [])])))
+
+    if "__src_url" in item:
+        click.secho("Source: %s" % item['__src_url'])
+    click.echo()
 
 
-def echo_liblist_header():
-    click.echo(
-        LIBLIST_TPL.format(
-            id=click.style(
-                "ID", fg="green"),
-            name=click.style(
-                "Name", fg="cyan"),
-            compatibility=click.style(
-                "Compatibility", fg="yellow"),
-            authornames="Authors",
-            description="Description"))
-
-    terminal_width, _ = click.get_terminal_size()
-    click.echo("-" * terminal_width)
-
-
-def echo_liblist_item(item):
-    description = item.get("description", item.get("url", "")).encode("utf-8")
-    if "version" in item:
-        description += " | @" + click.style(item['version'], fg="yellow")
-
-    click.echo(
-        LIBLIST_TPL.format(
-            id=click.style(
-                str(item.get("id", "-")), fg="green"),
-            name=click.style(
-                item['name'], fg="cyan"),
-            compatibility=click.style(
-                ", ".join(
-                    item.get("frameworks", ["-"]) + item.get("platforms", [])),
-                fg="yellow"),
-            authornames=", ".join(item.get("authornames", ["Unknown"])).encode(
-                "utf-8"),
-            description=description))
-
-
-@cli.command("search", short_help="Search for library")
+@cli.command("search", short_help="Search for a library")
 @click.argument("query", required=False, nargs=-1)
 @click.option("--json-output", is_flag=True)
 @click.option("--page", type=click.INT, default=1)
@@ -181,9 +199,8 @@ def lib_search(query, json_output, page, noninteractive, **filters):
             query.append('%s:"%s"' % (key, value))
 
     result = get_api_result(
-        "/lib/search",
-        dict(
-            query=" ".join(query), page=page),
+        "/v2/lib/search",
+        dict(query=" ".join(query), page=page),
         cache_valid="3d")
 
     if json_output:
@@ -210,12 +227,9 @@ def lib_search(query, json_output, page, noninteractive, **filters):
         "Found %d libraries:\n" % result['total'],
         fg="green" if result['total'] else "yellow")
 
-    if result['total']:
-        echo_liblist_header()
-
     while True:
         for item in result['items']:
-            echo_liblist_item(item)
+            print_lib_item(item)
 
         if (int(result['page']) * int(result['perpage']) >=
                 int(result['total'])):
@@ -232,9 +246,9 @@ def lib_search(query, json_output, page, noninteractive, **filters):
         elif not click.confirm("Show next libraries?"):
             break
         result = get_api_result(
-            "/lib/search",
-            dict(
-                query=" ".join(query), page=int(result['page']) + 1),
+            "/v2/lib/search",
+            {"query": " ".join(query),
+             "page": int(result['page']) + 1},
             cache_valid="3d")
 
 
@@ -245,41 +259,87 @@ def lib_list(lm, json_output):
     items = lm.get_installed()
 
     if json_output:
-        click.echo(json.dumps(items))
-        return
+        return click.echo(json.dumps(items))
 
     if not items:
         return
 
-    echo_liblist_header()
     for item in sorted(items, key=lambda i: i['name']):
-        if "authors" in item:
-            item['authornames'] = [i['name'] for i in item['authors']]
-        echo_liblist_item(item)
+        print_lib_item(item)
 
 
-@cli.command("show", short_help="Show details about installed library")
-@click.pass_obj
+@util.memoized
+def get_builtin_libs(storage_names=None):
+    items = []
+    storage_names = storage_names or []
+    pm = PlatformManager()
+    for manifest in pm.get_installed():
+        p = PlatformFactory.newPlatform(manifest['__pkg_dir'])
+        for storage in p.get_lib_storages():
+            if storage_names and storage['name'] not in storage_names:
+                continue
+            lm = LibraryManager(storage['path'])
+            items.append({
+                "name": storage['name'],
+                "path": storage['path'],
+                "items": lm.get_installed()
+            })
+    return items
+
+
+@cli.command("builtin", short_help="List built-in libraries")
+@click.option("--storage", multiple=True)
+@click.option("--json-output", is_flag=True)
+def lib_builtin(storage, json_output):
+    items = get_builtin_libs(storage)
+    if json_output:
+        return click.echo(json.dumps(items))
+
+    for storage in items:
+        if not storage['items']:
+            continue
+        click.secho(storage['name'], fg="green")
+        click.echo("*" * len(storage['name']))
+        click.echo()
+
+        for item in sorted(storage['items'], key=lambda i: i['name']):
+            print_lib_item(item)
+
+
+@cli.command("show", short_help="Show detailed info about a library")
 @click.argument("library", metavar="[LIBRARY]")
-def lib_show(lm, library):  # pylint: disable=too-many-branches
-    name, requirements, url = lm.parse_pkg_name(library)
-    package_dir = lm.get_package_dir(name, requirements, url)
-    if not package_dir:
-        click.secho(
-            "%s @ %s is not installed" % (name, requirements or "*"),
-            fg="yellow")
-        return
+@click.option("--json-output", is_flag=True)
+def lib_show(library, json_output):
+    lm = LibraryManager()
+    name, requirements, _ = lm.parse_pkg_input(library)
+    lib_id = lm.get_pkg_id_by_name(
+        name, requirements, silent=json_output, interactive=not json_output)
+    lib = get_api_result("/lib/info/%d" % lib_id, cache_valid="1d")
+    if json_output:
+        return click.echo(json.dumps(lib))
 
-    manifest = lm.load_manifest(package_dir)
-
-    click.secho(manifest['name'], fg="cyan")
-    click.echo("=" * len(manifest['name']))
-    if "description" in manifest:
-        click.echo(manifest['description'])
+    click.secho(lib['name'], fg="cyan")
+    click.echo("=" * len(lib['name']))
+    click.secho("#ID: %d" % lib['id'], bold=True)
+    click.echo(lib['description'])
     click.echo()
 
+    click.echo("Version: %s, released %s" %
+               (lib['version']['name'],
+                arrow.get(lib['version']['released']).humanize()))
+    click.echo("Manifest: %s" % lib['confurl'])
+    for key in ("homepage", "repository", "license"):
+        if key not in lib or not lib[key]:
+            continue
+        if isinstance(lib[key], list):
+            click.echo("%s: %s" % (key.title(), ", ".join(lib[key])))
+        else:
+            click.echo("%s: %s" % (key.title(), lib[key]))
+
+    blocks = []
+
     _authors = []
-    for author in manifest.get("authors", []):
+    for author in lib.get("authors", []):
         _data = []
         for key in ("name", "email", "url", "maintainer"):
             if not author[key]:
@@ -292,19 +352,33 @@ def lib_show(lm, library):  # pylint: disable=too-many-branches
                 _data.append(author[key])
         _authors.append(" ".join(_data))
     if _authors:
-        click.echo("Authors: %s" % ", ".join(_authors))
+        blocks.append(("Authors", _authors))
 
-    for key in ("keywords", "frameworks", "platforms", "license", "url",
-                "version"):
-        if key not in manifest:
+    blocks.append(("Keywords", lib['keywords']))
+    for key in ("frameworks", "platforms"):
+        if key not in lib or not lib[key]:
             continue
-        if isinstance(manifest[key], list):
-            click.echo("%s: %s" % (key.title(), ", ".join(manifest[key])))
-        else:
-            click.echo("%s: %s" % (key.title(), manifest[key]))
+        blocks.append(("Compatible %s" % key, [i['title'] for i in lib[key]]))
+    blocks.append(("Headers", lib['headers']))
+    blocks.append(("Examples", lib['examples']))
+    blocks.append(("Versions", [
+        "%s, released %s" % (v['name'], arrow.get(v['released']).humanize())
+        for v in lib['versions']
+    ]))
+    blocks.append(("Unique Downloads", [
+        "Today: %s" % lib['dlstats']['day'], "Week: %s" %
+        lib['dlstats']['week'], "Month: %s" % lib['dlstats']['month']
+    ]))
+
+    for (title, rows) in blocks:
+        click.echo()
+        click.secho(title, bold=True)
+        click.echo("-" * len(title))
+        for row in rows:
+            click.echo(row)
 
 
-@cli.command("register", short_help="Register new library")
+@cli.command("register", short_help="Register a new library")
 @click.argument("config_url")
 def lib_register(config_url):
     if (not config_url.startswith("http://") and
@@ -317,3 +391,76 @@ def lib_register(config_url):
             result['message'],
             fg="green"
             if "successed" in result and result['successed'] else "red")
+
+
+@cli.command("stats", short_help="Library Registry Statistics")
+@click.option("--json-output", is_flag=True)
+def lib_stats(json_output):
+    result = get_api_result("/lib/stats", cache_valid="1h")
+
+    if json_output:
+        return click.echo(json.dumps(result))
+
+    printitem_tpl = "{name:<33} {url}"
+    printitemdate_tpl = "{name:<33} {date:23} {url}"
+
+    def _print_title(title):
+        click.secho(title.upper(), bold=True)
+        click.echo("*" * len(title))
+
+    def _print_header(with_date=False):
+        click.echo((printitemdate_tpl if with_date else printitem_tpl).format(
+            name=click.style("Name", fg="cyan"),
+            date="Date",
+            url=click.style("Url", fg="blue")))
+
+        terminal_width, _ = click.get_terminal_size()
+        click.echo("-" * terminal_width)
+
+    def _print_lib_item(item):
+        click.echo((
+            printitemdate_tpl if "date" in item else printitem_tpl
+        ).format(
+            name=click.style(item['name'], fg="cyan"),
+            date=str(
+                arrow.get(item['date']).humanize() if "date" in item else ""),
+            url=click.style(
+                "http://platformio.org/lib/show/%s/%s" % (item['id'],
+                                                          quote(item['name'])),
+                fg="blue")))
+
+    def _print_tag_item(name):
+        click.echo(
+            printitem_tpl.format(
+                name=click.style(name, fg="cyan"),
+                url=click.style(
+                    "http://platformio.org/lib/search?query=" + quote(
+                        "keyword:%s" % name),
+                    fg="blue")))
+
+    for key in ("updated", "added"):
+        _print_title("Recently " + key)
+        _print_header(with_date=True)
+        for item in result.get(key, []):
+            _print_lib_item(item)
+        click.echo()
+
+    _print_title("Recent keywords")
+    _print_header(with_date=False)
+    for item in result.get("lastkeywords"):
+        _print_tag_item(item)
+    click.echo()
+
+    _print_title("Popular keywords")
+    _print_header(with_date=False)
+    for item in result.get("topkeywords"):
+        _print_tag_item(item)
+    click.echo()
+
+    for key, title in (("dlday", "Today"), ("dlweek", "Week"),
+                       ("dlmonth", "Month")):
+        _print_title("Featured: " + title)
+        _print_header(with_date=False)
+        for item in result.get(key, []):
+            _print_lib_item(item)
+        click.echo()

platformio/commands/platform.py

@@ -13,10 +13,12 @@
 # limitations under the License.
 
 import json
+from os.path import dirname, isdir
 
 import click
 
-from platformio import exception, util
+from platformio import app, exception, util
+from platformio.commands.boards import print_boards
 from platformio.managers.platform import PlatformFactory, PlatformManager
 
 
@@ -28,40 +30,152 @@ def cli():
 def _print_platforms(platforms):
     for platform in platforms:
         click.echo("{name} ~ {title}".format(
-            name=click.style(
-                platform['name'], fg="cyan"),
+            name=click.style(platform['name'], fg="cyan"),
             title=platform['title']))
         click.echo("=" * (3 + len(platform['name'] + platform['title'])))
         click.echo(platform['description'])
         click.echo()
-        click.echo("Home: %s" % "http://platformio.org/platforms/" + platform[
-            'name'])
-        if platform['packages']:
+        if "homepage" in platform:
+            click.echo("Home: %s" % platform['homepage'])
+        if "frameworks" in platform and platform['frameworks']:
+            click.echo("Frameworks: %s" % ", ".join(platform['frameworks']))
+        if "packages" in platform:
             click.echo("Packages: %s" % ", ".join(platform['packages']))
         if "version" in platform:
             click.echo("Version: " + platform['version'])
         click.echo()
 
 
+def _get_registry_platforms():
+    platforms = util.get_api_result("/platforms", cache_valid="30d")
+    pm = PlatformManager()
+    for platform in platforms or []:
+        platform['versions'] = pm.get_all_repo_versions(platform['name'])
+    return platforms
+
+
+def _original_version(version):
+    if version.count(".") != 2:
+        return None
+    _, y = version.split(".")[:2]
+    if int(y) < 100:
+        return None
+    if len(y) % 2 != 0:
+        y = "0" + y
+    parts = [str(int(y[i * 2:i * 2 + 2])) for i in range(len(y) / 2)]
+    return ".".join(parts)
+
+
+def _get_platform_data(*args, **kwargs):
+    try:
+        return _get_installed_platform_data(*args, **kwargs)
+    except exception.UnknownPlatform:
+        return _get_registry_platform_data(*args, **kwargs)
+
+
+def _get_installed_platform_data(platform,
+                                 with_boards=True,
+                                 expose_packages=True):
+    p = PlatformFactory.newPlatform(platform)
+    data = dict(
+        name=p.name,
+        title=p.title,
+        description=p.description,
+        version=p.version,  # comment before dump
+        homepage=p.homepage,
+        repository=p.repository_url,
+        url=p.vendor_url,
+        license=p.license,
+        forDesktop=not p.is_embedded(),
+        frameworks=sorted(p.frameworks.keys() if p.frameworks else []),
+        packages=p.packages.keys() if p.packages else [])
+
+    # if dump to API
+    # del data['version']
+    # return data
+
+    # overwrite VCS version and add extra fields
+    manifest = PlatformManager().load_manifest(dirname(p.manifest_path))
+    assert manifest
+    for key in manifest:
+        if key == "version" or key.startswith("__"):
+            data[key] = manifest[key]
+
+    if with_boards:
+        data['boards'] = [c.get_brief_data() for c in p.get_boards().values()]
+
+    if not data['packages'] or not expose_packages:
+        return data
+
+    data['packages'] = []
+    installed_pkgs = p.get_installed_packages()
+    for name, opts in p.packages.items():
+        item = dict(
+            name=name,
+            type=p.get_package_type(name),
+            requirements=opts.get("version"),
+            optional=opts.get("optional") is True)
+        if name in installed_pkgs:
+            for key, value in installed_pkgs[name].items():
+                if key not in ("url", "version", "description"):
+                    continue
+                item[key] = value
+                if key == "version":
+                    item["originalVersion"] = _original_version(value)
+        data['packages'].append(item)
+
+    return data
+
+
+def _get_registry_platform_data(  # pylint: disable=unused-argument
+        platform,
+        with_boards=True,
+        expose_packages=True):
+    _data = None
+    for p in _get_registry_platforms():
+        if p['name'] == platform:
+            _data = p
+            break
+
+    if not _data:
+        return None
+
+    data = dict(
+        name=_data['name'],
+        title=_data['title'],
+        description=_data['description'],
+        homepage=_data['homepage'],
+        repository=_data['repository'],
+        url=_data['url'],
+        license=_data['license'],
+        forDesktop=_data['forDesktop'],
+        frameworks=_data['frameworks'],
+        packages=_data['packages'],
+        versions=_data['versions'])
+
+    if with_boards:
+        data['boards'] = [
+            board for board in PlatformManager().get_registered_boards()
+            if board['platform'] == _data['name']
+        ]
+
+    return data
+
+
 @cli.command("search", short_help="Search for development platform")
 @click.argument("query", required=False)
 @click.option("--json-output", is_flag=True)
 def platform_search(query, json_output):
     platforms = []
-    for platform in util.get_api_result("/platforms", cache_valid="365d"):
+    for platform in _get_registry_platforms():
         if query == "all":
             query = ""
-
         search_data = json.dumps(platform)
         if query and query.lower() not in search_data.lower():
             continue
-
-        platforms.append({
-            "name": platform['name'],
-            "title": platform['title'],
-            "description": platform['description'],
-            "packages": platform['packages']
-        })
+        platforms.append(
+            _get_registry_platform_data(
+                platform['name'], with_boards=False, expose_packages=False))
 
     if json_output:
         click.echo(json.dumps(platforms))
@@ -69,6 +183,108 @@ def platform_search(query, json_output):
         _print_platforms(platforms)
 
 
+@cli.command("frameworks", short_help="List supported frameworks, SDKs")
+@click.argument("query", required=False)
+@click.option("--json-output", is_flag=True)
+def platform_frameworks(query, json_output):
+    frameworks = []
+    for framework in util.get_api_result("/frameworks", cache_valid="30d"):
+        if query == "all":
+            query = ""
+        search_data = json.dumps(framework)
+        if query and query.lower() not in search_data.lower():
+            continue
+        framework['homepage'] = (
+            "http://platformio.org/frameworks/" + framework['name'])
+        framework['platforms'] = [
+            platform['name'] for platform in _get_registry_platforms()
+            if framework['name'] in platform['frameworks']
+        ]
+        frameworks.append(framework)
+
+    if json_output:
+        click.echo(json.dumps(frameworks))
+    else:
+        _print_platforms(frameworks)
+
+
+@cli.command("list", short_help="List installed development platforms")
+@click.option("--json-output", is_flag=True)
+def platform_list(json_output):
+    platforms = []
+    pm = PlatformManager()
+    for manifest in pm.get_installed():
+        platforms.append(
+            _get_installed_platform_data(
+                manifest['__pkg_dir'],
+                with_boards=False,
+                expose_packages=False))
+    if json_output:
+        click.echo(json.dumps(platforms))
+    else:
+        _print_platforms(platforms)
+
+
+@cli.command("show", short_help="Show details about development platform")
+@click.argument("platform")
+@click.option("--json-output", is_flag=True)
+def platform_show(platform, json_output):  # pylint: disable=too-many-branches
+    data = _get_platform_data(platform)
+    if not data:
+        raise exception.UnknownPlatform(platform)
+    if json_output:
+        return click.echo(json.dumps(data))
+
+    click.echo("{name} ~ {title}".format(
+        name=click.style(data['name'], fg="cyan"), title=data['title']))
+    click.echo("=" * (3 + len(data['name'] + data['title'])))
+    click.echo(data['description'])
+    click.echo()
+    if "version" in data:
+        click.echo("Version: %s" % data['version'])
+    if data['homepage']:
+        click.echo("Home: %s" % data['homepage'])
+    if data['repository']:
+        click.echo("Repository: %s" % data['repository'])
+    if data['url']:
+        click.echo("Vendor: %s" % data['url'])
+    if data['license']:
+        click.echo("License: %s" % data['license'])
+    if data['frameworks']:
+        click.echo("Frameworks: %s" % ", ".join(data['frameworks']))
+
+    if not data['packages']:
+        return
+
+    if not isinstance(data['packages'][0], dict):
+        click.echo("Packages: %s" % ", ".join(data['packages']))
+    else:
+        click.echo()
+        click.secho("Packages", bold=True)
+        click.echo("--------")
+        for item in data['packages']:
+            click.echo()
+            click.echo("Package %s" % click.style(item['name'], fg="yellow"))
+            click.echo("-" * (8 + len(item['name'])))
+            if item['type']:
+                click.echo("Type: %s" % item['type'])
+            click.echo("Requirements: %s" % item['requirements'])
+            click.echo("Installed: %s" % ("Yes" if item.get("version") else
+                                          "No (optional)"))
+            if "version" in item:
+                click.echo("Version: %s" % item['version'])
+            if "originalVersion" in item:
+                click.echo("Original version: %s" % item['originalVersion'])
+            if "description" in item:
+                click.echo("Description: %s" % item['description'])
+
+    if data['boards']:
+        click.echo()
+        click.secho("Boards", bold=True)
+        click.echo("------")
+        print_boards(data['boards'])
+
+
 @cli.command("install", short_help="Install new development platform")
 @click.argument("platforms", nargs=-1, required=True, metavar="[PLATFORM...]")
 @click.option("--with-package", multiple=True)
@@ -108,99 +324,51 @@ def platform_uninstall(platforms):
     "-p",
     "--only-packages",
     is_flag=True,
-    help="Update only platform packages")
+    help="Update only the platform packages")
 @click.option(
     "-c",
     "--only-check",
     is_flag=True,
-    help="Do not update, only check for new version")
-def platform_update(platforms, only_packages, only_check):
-    pm = PlatformManager()
-    if not platforms:
-        platforms = set([m['name'] for m in pm.get_installed()])
-    for platform in platforms:
-        click.echo("Platform %s" % click.style(platform, fg="cyan"))
-        click.echo("--------")
-        pm.update(platform, only_packages=only_packages, only_check=only_check)
-        click.echo()
-
-
-@cli.command("list", short_help="List installed development platforms")
+    help="Do not update, only check for a new version")
 @click.option("--json-output", is_flag=True)
-def platform_list(json_output):
-    platforms = []
+def platform_update(platforms, only_packages, only_check, json_output):
     pm = PlatformManager()
-    for manifest in pm.get_installed():
-        p = PlatformFactory.newPlatform(
-            pm.get_manifest_path(manifest['__pkg_dir']))
-        platforms.append({
-            "name": p.name,
-            "title": p.title,
-            "description": p.description,
-            "version": p.version,
-            "url": p.vendor_url,
-            "packages": p.get_installed_packages().keys(),
-            'forDesktop': any([
-                p.name.startswith(n) for n in ("native", "linux", "windows")
-            ])
-        })
+    pkg_dir_to_name = {}
+    if not platforms:
+        platforms = []
+        for manifest in pm.get_installed():
+            platforms.append(manifest['__pkg_dir'])
+            pkg_dir_to_name[manifest['__pkg_dir']] = manifest.get(
+                "title", manifest['name'])
 
-    if json_output:
-        click.echo(json.dumps(platforms))
+    if only_check and json_output:
+        result = []
+        for platform in platforms:
+            pkg_dir = platform if isdir(platform) else None
+            requirements = None
+            url = None
+            if not pkg_dir:
+                name, requirements, url = pm.parse_pkg_input(platform)
+                pkg_dir = pm.get_package_dir(name, requirements, url)
+            if not pkg_dir:
+                continue
+            latest = pm.outdated(pkg_dir, requirements)
+            if (not latest and not PlatformFactory.newPlatform(pkg_dir)
+                    .are_outdated_packages()):
+                continue
+            data = _get_installed_platform_data(
+                pkg_dir, with_boards=False, expose_packages=False)
+            if latest:
+                data['versionLatest'] = latest
+            result.append(data)
+        return click.echo(json.dumps(result))
     else:
-        _print_platforms(platforms)
-
-
-@cli.command("show", short_help="Show details about installed platform")
-@click.argument("platform")
-def platform_show(platform):
-
-    def _detail_version(version):
-        if version.count(".") != 2:
-            return version
-        _, y = version.split(".")[:2]
-        if int(y) < 100:
-            return version
-        if len(y) % 2 != 0:
-            y = "0" + y
-        parts = [str(int(y[i * 2:i * 2 + 2])) for i in range(len(y) / 2)]
-        return "%s (%s)" % (version, ".".join(parts))
-
-    try:
-        p = PlatformFactory.newPlatform(platform)
-    except exception.UnknownPlatform:
-        raise exception.PlatformNotInstalledYet(platform)
-
-    click.echo("{name} ~ {title}".format(
-        name=click.style(
-            p.name, fg="cyan"), title=p.title))
-    click.echo("=" * (3 + len(p.name + p.title)))
-    click.echo(p.description)
-    click.echo()
-    click.echo("Version: %s" % p.version)
-    if p.homepage:
-        click.echo("Home: %s" % p.homepage)
-    if p.license:
-        click.echo("License: %s" % p.license)
-    if p.frameworks:
-        click.echo("Frameworks: %s" % ", ".join(p.frameworks.keys()))
-
-    if not p.packages:
-        return
-
-    installed_pkgs = p.get_installed_packages()
-    for name, opts in p.packages.items():
-        click.echo()
-        click.echo("Package %s" % click.style(name, fg="yellow"))
-        click.echo("-" * (8 + len(name)))
-        if p.get_package_type(name):
-            click.echo("Type: %s" % p.get_package_type(name))
-        click.echo("Requirements: %s" % opts.get("version"))
-        click.echo("Installed: %s" % ("Yes" if name in installed_pkgs else
-                                      "No (optional)"))
-        if name in installed_pkgs:
-            for key, value in installed_pkgs[name].items():
-                if key in ("url", "version", "description"):
-                    if key == "version":
-                        value = _detail_version(value)
-                    click.echo("%s: %s" % (key.title(), value))
+        # cleanup cached board and platform lists
+        app.clean_cache()
+        for platform in platforms:
+            click.echo("Platform %s" % click.style(
+                pkg_dir_to_name.get(platform, platform), fg="cyan"))
+            click.echo("--------")
+            pm.update(
+                platform, only_packages=only_packages, only_check=only_check)
+            click.echo()

platformio/commands/remote.py

@@ -23,7 +23,7 @@ import click
 
 from platformio import exception, util
 from platformio.commands.device import device_monitor as cmd_device_monitor
-from platformio.pioplus import pioplus_call
+from platformio.managers.core import pioplus_call
 
 # pylint: disable=unused-argument
 
@@ -147,12 +147,12 @@ def device_list(json_output):
 @click.option(
     "--rts",
     default=None,
-    type=click.Choice(["0", "1"]),
+    type=click.IntRange(0, 1),
     help="Set initial RTS line state")
 @click.option(
     "--dtr",
     default=None,
-    type=click.Choice(["0", "1"]),
+    type=click.IntRange(0, 1),
     help="Set initial DTR line state")
 @click.option("--echo", is_flag=True, help="Enable local echo, default=Off")
 @click.option(

platformio/commands/run.py

@@ -22,6 +22,7 @@ import click
 
 from platformio import __version__, exception, telemetry, util
 from platformio.commands.lib import lib_install as cmd_lib_install
+from platformio.commands.lib import get_builtin_libs
 from platformio.commands.platform import \
     platform_install as cmd_platform_install
 from platformio.managers.lib import LibraryManager
@@ -95,7 +96,7 @@ def cli(ctx, environment, target, upload_port, project_dir, silent, verbose,
                 results.append((envname, None))
                 continue
 
-            if results:
+            if not silent and results:
                 click.echo()
 
             options = {}
@@ -108,11 +109,13 @@ def cli(ctx, environment, target, upload_port, project_dir, silent, verbose,
                                       upload_port, silent, verbose)
             results.append((envname, ep.process()))
 
-        if len(results) > 1:
+        found_error = any([status is False for (_, status) in results])
+
+        if (found_error or not silent) and len(results) > 1:
             click.echo()
             print_summary(results, start_time)
 
-        if any([status is False for (_, status) in results]):
+        if found_error:
             raise exception.ReturnErrorCode(1)
         return True
 
@@ -160,18 +163,20 @@ class EnvironmentProcessor(object):
             if "\n" in v:
                 self.options[k] = self.options[k].strip().replace("\n", ", ")
 
-        click.echo("[%s] Processing %s (%s)" % (
-            datetime.now().strftime("%c"), click.style(
-                self.name, fg="cyan", bold=True),
-            ", ".join(["%s: %s" % (k, v) for k, v in self.options.items()])))
-        click.secho("-" * terminal_width, bold=True)
-        if self.silent:
-            click.echo("Please wait...")
+        if not self.silent:
+            click.echo("[%s] Processing %s (%s)" % (
+                datetime.now().strftime("%c"), click.style(
+                    self.name, fg="cyan", bold=True), ", ".join(
+                        ["%s: %s" % (k, v) for k, v in self.options.items()])))
+            click.secho("-" * terminal_width, bold=True)
 
         self.options = self._validate_options(self.options)
         result = self._run()
-
         is_error = result['returncode'] != 0
+
+        if self.silent and not is_error:
+            return True
+
         if is_error or "piotest_processor" not in self.cmd_ctx.meta:
             print_header(
                 "[%s] Took %.2f seconds" % ((click.style(
@@ -275,7 +280,15 @@ def _autoinstall_libdeps(ctx, libraries, verbose=False):
         try:
             ctx.invoke(cmd_lib_install, libraries=[lib], silent=not verbose)
         except exception.LibNotFound as e:
-            click.secho("Warning! %s" % e, fg="yellow")
+            if not _is_builtin_lib(lib):
+                click.secho("Warning! %s" % e, fg="yellow")
+
+
+def _is_builtin_lib(lib_name):
+    for storage in get_builtin_libs():
+        if any([l.get("name") == lib_name for l in storage['items']]):
+            return True
+    return False
 
 
 def _clean_pioenvs_dir(pioenvs_dir):
@@ -329,9 +342,7 @@ def print_summary(results, start_time):
         format_str = (
             "Environment {0:<" + str(envname_max_len + 9) + "}\t[{1}]")
         click.echo(
-            format_str.format(
-                click.style(
-                    envname, fg="cyan"), status_str),
+            format_str.format(click.style(envname, fg="cyan"), status_str),
             err=status is False)
 
     print_header(

platformio/commands/settings.py

@@ -31,11 +31,9 @@ def settings_get(name):
 
     click.echo(
         list_tpl.format(
-            name=click.style(
-                "Name", fg="cyan"),
-            value=(click.style(
-                "Value", fg="green") + click.style(
-                    " [Default]", fg="yellow")),
+            name=click.style("Name", fg="cyan"),
+            value=(click.style("Value", fg="green") + click.style(
+                " [Default]", fg="yellow")),
             description="Description"))
     click.echo("-" * terminal_width)
 
@@ -59,8 +57,7 @@ def settings_get(name):
 
         click.echo(
             list_tpl.format(
-                name=click.style(
-                    _name, fg="cyan"),
+                name=click.style(_name, fg="cyan"),
                 value=_value_str,
                 description=_data['description']))
 

platformio/commands/test.py

@@ -17,7 +17,7 @@ from os import getcwd
 
 import click
 
-from platformio.pioplus import pioplus_call
+from platformio.managers.core import pioplus_call
 
 
 @click.command("test", short_help="Local Unit Testing")
@@ -37,6 +37,20 @@ from platformio.pioplus import pioplus_call
         resolve_path=True))
 @click.option("--without-building", is_flag=True)
 @click.option("--without-uploading", is_flag=True)
+@click.option(
+    "--no-reset",
+    is_flag=True,
+    help="Disable software reset via Serial.DTR/RST")
+@click.option(
+    "--monitor-rts",
+    default=None,
+    type=click.IntRange(0, 1),
+    help="Set initial RTS line state for Serial Monitor")
+@click.option(
+    "--monitor-dtr",
+    default=None,
+    type=click.IntRange(0, 1),
+    help="Set initial DTR line state for Serial Monitor")
 @click.option("--verbose", "-v", is_flag=True)
 def cli(*args, **kwargs):  # pylint: disable=unused-argument
     pioplus_call(sys.argv[1:])

platformio/commands/update.py

@@ -14,25 +14,36 @@
 
 import click
 
+from platformio import app
 from platformio.commands.lib import lib_update as cmd_lib_update
 from platformio.commands.platform import platform_update as cmd_platform_update
+from platformio.managers.core import update_core_packages
 from platformio.managers.lib import LibraryManager
-from platformio.pioplus import pioplus_update
 
 
 @click.command(
-    "update", short_help="Update installed Platforms, Packages and Libraries")
+    "update", short_help="Update installed platforms, packages and libraries")
+@click.option(
+    "--core-packages", is_flag=True, help="Update only the core packages")
 @click.option(
     "-c",
     "--only-check",
     is_flag=True,
     help="Do not update, only check for new version")
 @click.pass_context
-def cli(ctx, only_check):
+def cli(ctx, core_packages, only_check):
+    update_core_packages(only_check)
+
+    if core_packages:
+        return
+
+    # cleanup lib search results, cached board and platform lists
+    app.clean_cache()
+
+    click.echo()
     click.echo("Platform Manager")
     click.echo("================")
     ctx.invoke(cmd_platform_update, only_check=only_check)
-    pioplus_update()
 
     click.echo()
     click.echo("Library Manager")

platformio/commands/upgrade.py

@@ -83,8 +83,8 @@ WARNING! Don't use `sudo` for the rest PlatformIO commands.
                     err=True)
                 raise exception.ReturnErrorCode(1)
             else:
-                raise exception.UpgradeError("\n".join(
-                    [str(cmd), r['out'], r['err']]))
+                raise exception.UpgradeError(
+                    "\n".join([str(cmd), r['out'], r['err']]))
 
 
 def get_latest_version():
@@ -101,9 +101,10 @@ def get_latest_version():
 
 def get_develop_latest_version():
     version = None
-    r = requests.get("https://raw.githubusercontent.com/platformio/platformio"
-                     "/develop/platformio/__init__.py",
-                     headers=util.get_request_defheaders())
+    r = requests.get(
+        "https://raw.githubusercontent.com/platformio/platformio"
+        "/develop/platformio/__init__.py",
+        headers=util.get_request_defheaders())
     r.raise_for_status()
     for line in r.text.split("\n"):
         line = line.strip()
@@ -121,7 +122,8 @@ def get_develop_latest_version():
 
 
 def get_pypi_latest_version():
-    r = requests.get("https://pypi.python.org/pypi/platformio/json",
-                     headers=util.get_request_defheaders())
+    r = requests.get(
+        "https://pypi.python.org/pypi/platformio/json",
+        headers=util.get_request_defheaders())
     r.raise_for_status()
     return r.json()['info']['version']

platformio/downloader.py

@@ -31,9 +31,8 @@ class FileDownloader(object):
 
     def __init__(self, url, dest_dir=None):
         # make connection
-        self._request = requests.get(url,
-                                     stream=True,
-                                     headers=util.get_request_defheaders())
+        self._request = requests.get(
+            url, stream=True, headers=util.get_request_defheaders())
         if self._request.status_code != 200:
             raise FDUnrecognizedStatusCode(self._request.status_code, url)
 

platformio/exception.py

@@ -40,7 +40,12 @@ class AbortedByUser(PlatformioException):
 
 class UnknownPlatform(PlatformioException):
 
-    MESSAGE = "Unknown platform '{0}'"
+    MESSAGE = "Unknown development platform '{0}'"
+
+
+class IncompatiblePlatform(PlatformioException):
+
+    MESSAGE = "Development platform '{0}' is not compatible with PIO Core v{1}"
 
 
 class PlatformNotInstalledYet(PlatformioException):
@@ -53,7 +58,7 @@ class BoardNotDefined(PlatformioException):
 
     MESSAGE = "You need to specify board ID using `-b` or `--board` "\
         "option. Supported boards list is available via "\
-        " `platformio boards` command"
+        "`platformio boards` command"
 
 
 class UnknownBoard(PlatformioException):
@@ -78,7 +83,7 @@ class UnknownPackage(PlatformioException):
 
 class MissingPackageManifest(PlatformioException):
 
-    MESSAGE = "Could not find '{0}' manifest file in the package"
+    MESSAGE = "Could not find one of '{0}' manifest files in the package"
 
 
 class UndefinedPackageVersion(PlatformioException):
@@ -89,8 +94,10 @@ class UndefinedPackageVersion(PlatformioException):
 
 class PackageInstallError(PlatformioException):
 
-    MESSAGE = "Can not install '{0}' with version requirements '{1}' "\
-              "for your system '{2}'"
+    MESSAGE = "Could not install '{0}' with version requirements '{1}' "\
+              "for your system '{2}'.\n"\
+              "If you use Antivirus, it can block PlatformIO Package "\
+              "Manager. Try to disable it for a while."
 
 
 class FDUnrecognizedStatusCode(PlatformioException):

platformio/ide/projectgenerator.py

@@ -69,8 +69,8 @@ class ProjectGenerator(object):
         result = util.exec_command(cmd)
 
         if result['returncode'] != 0 or '"includes":' not in result['out']:
-            raise exception.PlatformioException("\n".join(
-                [result['out'], result['err']]))
+            raise exception.PlatformioException(
+                "\n".join([result['out'], result['err']]))
 
         for line in result['out'].split("\n"):
             line = line.strip()

platformio/ide/tpls/clion/CMakeListsPrivate.txt.tpl

@@ -5,6 +5,7 @@ SET(CMAKE_C_COMPILER "{{cc_path.replace("\\", "/")}}")
 SET(CMAKE_CXX_COMPILER "{{cxx_path.replace("\\", "/")}}")
 SET(CMAKE_CXX_FLAGS_DISTRIBUTION "{{cxx_flags}}")
 SET(CMAKE_C_FLAGS_DISTRIBUTION "{{cc_flags}}")
+set(CMAKE_CXX_STANDARD 11)
 
 % for define in defines:
 add_definitions(-D{{!define}})

platformio/maintenance.py

@@ -29,9 +29,9 @@ from platformio.commands.platform import \
     platform_uninstall as cmd_platform_uninstall
 from platformio.commands.platform import platform_update as cmd_platform_update
 from platformio.commands.upgrade import get_latest_version
+from platformio.managers.core import update_core_packages
 from platformio.managers.lib import LibraryManager
-from platformio.managers.platform import PlatformManager
-from platformio.pioplus import pioplus_update
+from platformio.managers.platform import PlatformFactory, PlatformManager
 
 
 def in_silence(ctx=None):
@@ -42,11 +42,6 @@ def in_silence(ctx=None):
             (ctx.args[0] == "upgrade" or "--json-output" in ctx_args))
 
 
-def clean_cache():
-    with app.ContentCache() as cc:
-        cc.clean()
-
-
 def on_platformio_start(ctx, force, caller):
     if not caller:
         if getenv("PLATFORMIO_CALLER"):
@@ -64,8 +59,6 @@ def on_platformio_start(ctx, force, caller):
     app.set_session_var("caller_id", caller)
     telemetry.on_command()
 
-    if ctx.args and (ctx.args[0] == "upgrade" or "update" in ctx.args):
-        clean_cache()
     if not in_silence(ctx):
         after_upgrade(ctx)
 
@@ -98,8 +91,8 @@ class Upgrader(object):
             util.pepver_to_semver(to_version))
 
         self._upgraders = [
-            (semantic_version.Version("3.0.0-a1"), self._upgrade_to_3_0_0),
-            (semantic_version.Version("3.0.0-b11"), self._upgrade_to_3_0_0)
+            (semantic_version.Version("3.0.0-a.1"), self._upgrade_to_3_0_0),
+            (semantic_version.Version("3.0.0-b.11"), self._upgrade_to_3_0_0b11)
         ]
 
     def run(self, ctx):
@@ -146,9 +139,10 @@ class Upgrader(object):
             m['name'] for m in PlatformManager().get_installed()
         ]
         if "espressif" not in current_platforms:
-            return
+            return True
         ctx.invoke(cmd_platform_install, platforms=["espressif8266"])
         ctx.invoke(cmd_platform_uninstall, platforms=["espressif"])
+        return True
 
 
 def after_upgrade(ctx):
@@ -159,26 +153,19 @@ def after_upgrade(ctx):
     if last_version == "0.0.0":
         app.set_state_item("last_version", __version__)
     else:
-        click.secho("Please wait while upgrading PlatformIO ...", fg="yellow")
-        clean_cache()
+        click.secho("Please wait while upgrading PlatformIO...", fg="yellow")
+        app.clean_cache()
+
+        # Update PlatformIO's Core packages
+        update_core_packages(silent=True)
+
         u = Upgrader(last_version, __version__)
         if u.run(ctx):
             app.set_state_item("last_version", __version__)
-
-            # update development platforms
-            pm = PlatformManager()
-            for manifest in pm.get_installed():
-                # pm.update(manifest['name'], "^" + manifest['version'])
-                pm.update(manifest['name'])
-
-            # update PlatformIO Plus tool if installed
-            pioplus_update()
-
             click.secho(
                 "PlatformIO has been successfully upgraded to %s!\n" %
                 __version__,
                 fg="green")
-
             telemetry.on_event(
                 category="Auto",
                 action="Upgrade",
@@ -196,14 +183,13 @@ def after_upgrade(ctx):
                "on the latest project news > %s" % (click.style(
                    "follow", fg="cyan"), click.style(
                        "https://twitter.com/PlatformIO_Org", fg="cyan")))
-    click.echo("- %s it on GitHub > %s" % (click.style(
-        "star", fg="cyan"), click.style(
-            "https://github.com/platformio/platformio", fg="cyan")))
+    click.echo("- %s it on GitHub > %s" %
+               (click.style("star", fg="cyan"), click.style(
+                   "https://github.com/platformio/platformio", fg="cyan")))
     if not getenv("PLATFORMIO_IDE"):
         click.echo("- %s PlatformIO IDE for IoT development > %s" %
-                   (click.style(
-                       "try", fg="cyan"), click.style(
-                           "http://platformio.org/platformio-ide", fg="cyan")))
+                   (click.style("try", fg="cyan"), click.style(
+                       "http://platformio.org/platformio-ide", fg="cyan")))
     if not util.is_ci():
         click.echo("- %s us with PlatformIO Plus > %s" % (click.style(
             "support", fg="cyan"), click.style(
@@ -267,8 +253,14 @@ def check_internal_updates(ctx, what):
     pm = PlatformManager() if what == "platforms" else LibraryManager()
     outdated_items = []
     for manifest in pm.get_installed():
-        if manifest['name'] not in outdated_items and \
-                pm.is_outdated(manifest['name']):
+        if manifest['name'] in outdated_items:
+            continue
+        conds = [
+            pm.outdated(manifest['__pkg_dir']), what == "platforms" and
+            PlatformFactory.newPlatform(
+                manifest['__pkg_dir']).are_outdated_packages()
+        ]
+        if any(conds):
             outdated_items.append(manifest['name'])
 
     if not outdated_items:

platformio/managers/core.py

@@ -20,21 +20,17 @@ from os.path import join
 from platformio import exception, util
 from platformio.managers.package import PackageManager
 
-PACKAGE_DEPS = {
-    "pysite": {
-        "name": "pysite-pioplus",
-        "requirements": ">=0.3.0,<2"
-    },
-    "tool": {
-        "name": "tool-pioplus",
-        "requirements": ">=0.6.6,<2"
-    }
+CORE_PACKAGES = {
+    "pysite-pioplus": ">=0.3.0,<2",
+    "tool-pioplus": ">=0.6.10,<2",
+    "tool-unity": "~1.20302.1",
+    "tool-scons": "~3.20501.2"
 }
 
-AUTO_UPDATES_MAX = 100
+PIOPLUS_AUTO_UPDATES_MAX = 100
 
 
-class PioPlusPackageManager(PackageManager):
+class CorePackageManager(PackageManager):
 
     def __init__(self):
         PackageManager.__init__(
@@ -46,29 +42,30 @@ class PioPlusPackageManager(PackageManager):
             ])
 
 
-def pioplus_install():
-    pm = PioPlusPackageManager()
-    for item in PACKAGE_DEPS.values():
-        pm.install(item['name'], item['requirements'], silent=True)
+def get_core_package_dir(name):
+    assert name in CORE_PACKAGES
+    requirements = CORE_PACKAGES[name]
+    pm = CorePackageManager()
+    pkg_dir = pm.get_package_dir(name, requirements)
+    if pkg_dir:
+        return pkg_dir
+    return pm.install(name, requirements)
 
 
-def pioplus_update():
-    pm = PioPlusPackageManager()
-    for item in PACKAGE_DEPS.values():
-        package_dir = pm.get_package_dir(item['name'])
-        if package_dir:
-            pm.update(item['name'], item['requirements'])
+def update_core_packages(only_check=False, silent=False):
+    pm = CorePackageManager()
+    for name, requirements in CORE_PACKAGES.items():
+        pkg_dir = pm.get_package_dir(name)
+        if not pkg_dir:
+            continue
+        if not silent or pm.outdated(pkg_dir, requirements):
+            pm.update(name, requirements, only_check=only_check)
 
 
 def pioplus_call(args, **kwargs):
-    pioplus_install()
-    pm = PioPlusPackageManager()
-    pioplus_path = join(
-        pm.get_package_dir(PACKAGE_DEPS['tool']['name'],
-                           PACKAGE_DEPS['tool']['requirements']), "pioplus")
+    pioplus_path = join(get_core_package_dir("tool-pioplus"), "pioplus")
     os.environ['PYTHONEXEPATH'] = util.get_pythonexe_path()
-    os.environ['PYTHONPYSITEDIR'] = pm.get_package_dir(
-        PACKAGE_DEPS['pysite']['name'], PACKAGE_DEPS['pysite']['requirements'])
+    os.environ['PYTHONPYSITEDIR'] = get_core_package_dir("pysite-pioplus")
     util.copy_pythonpath_to_osenv()
     code = subprocess.call([pioplus_path] + args, **kwargs)
 
@@ -82,8 +79,8 @@ def pioplus_call(args, **kwargs):
             setattr(pioplus_call, count_attr, 1)
         count_value += 1
         setattr(pioplus_call, count_attr, count_value)
-        if count_value < AUTO_UPDATES_MAX:
-            pioplus_update()
+        if count_value < PIOPLUS_AUTO_UPDATES_MAX:
+            update_core_packages()
             return pioplus_call(args, **kwargs)
 
     # handle reload request

platformio/managers/lib.py

@@ -15,10 +15,11 @@
 # pylint: disable=too-many-arguments, too-many-locals, too-many-branches
 
 import json
-import os
-from hashlib import md5
-from os.path import dirname, join
+import re
+from glob import glob
+from os.path import isdir, join
 
+import arrow
 import click
 import semantic_version
 
@@ -34,70 +35,93 @@ class LibraryManager(BasePkgManager):
         BasePkgManager.__init__(self, package_dir)
 
     @property
-    def manifest_name(self):
-        return ".library.json"
+    def manifest_names(self):
+        return [
+            ".library.json", "library.json", "library.properties",
+            "module.json"
+        ]
 
-    def check_pkg_structure(self, pkg_dir):
-        try:
-            return BasePkgManager.check_pkg_structure(self, pkg_dir)
-        except exception.MissingPackageManifest:
-            # we will generate manifest automatically
-            pass
+    def get_manifest_path(self, pkg_dir):
+        path = BasePkgManager.get_manifest_path(self, pkg_dir)
+        if path:
+            return path
 
-        manifest = {
-            "name": "Library_" + md5(pkg_dir).hexdigest()[:5],
-            "version": "0.0.0"
-        }
-        manifest_path = self._find_any_manifest(pkg_dir)
-        if manifest_path:
-            _manifest = self._parse_manifest(manifest_path)
-            pkg_dir = dirname(manifest_path)
-            for key in ("name", "version"):
-                if key not in _manifest:
-                    _manifest[key] = manifest[key]
-            manifest = _manifest
-        else:
-            for root, dirs, files in os.walk(pkg_dir):
-                if len(dirs) == 1 and not files:
-                    manifest['name'] = dirs[0]
-                    continue
-                if dirs or files:
-                    pkg_dir = root
-                    break
+        # if library without manifest, returns first source file
+        src_dir = join(util.glob_escape(pkg_dir))
+        if isdir(join(pkg_dir, "src")):
+            src_dir = join(src_dir, "src")
+        chs_files = glob(join(src_dir, "*.[chS]"))
+        if chs_files:
+            return chs_files[0]
+        cpp_files = glob(join(src_dir, "*.cpp"))
+        if cpp_files:
+            return cpp_files[0]
 
-        with open(join(pkg_dir, self.manifest_name), "w") as fp:
-            json.dump(manifest, fp)
-
-        return pkg_dir
-
-    @staticmethod
-    def _find_any_manifest(pkg_dir):
-        manifests = ("library.json", "library.properties", "module.json")
-        for root, _, files in os.walk(pkg_dir):
-            for manifest in manifests:
-                if manifest in files:
-                    return join(root, manifest)
         return None
 
-    @staticmethod
-    def _parse_manifest(path):
-        manifest = {}
-        if path.endswith(".json"):
-            return util.load_json(path)
-        elif path.endswith("library.properties"):
-            with open(path) as fp:
-                for line in fp.readlines():
-                    if "=" not in line:
-                        continue
-                    key, value = line.split("=", 1)
-                    manifest[key.strip()] = value.strip()
+    def load_manifest(self, pkg_dir):
+        manifest = BasePkgManager.load_manifest(self, pkg_dir)
+        if not manifest:
+            return manifest
+
+        # if Arudino library.properties
+        if "sentence" in manifest:
             manifest['frameworks'] = ["arduino"]
-            if "author" in manifest:
-                manifest['authors'] = [{"name": manifest['author']}]
-                del manifest['author']
-            if "sentence" in manifest:
-                manifest['description'] = manifest['sentence']
-                del manifest['sentence']
+            manifest['description'] = manifest['sentence']
+            del manifest['sentence']
+
+        if "author" in manifest:
+            manifest['authors'] = [{"name": manifest['author']}]
+            del manifest['author']
+
+        if "authors" in manifest and not isinstance(manifest['authors'], list):
+            manifest['authors'] = [manifest['authors']]
+
+        if "keywords" not in manifest:
+            keywords = []
+            for keyword in re.split(r"[\s/]+",
+                                    manifest.get("category", "Uncategorized")):
+                keyword = keyword.strip()
+                if not keyword:
+                    continue
+                keywords.append(keyword.lower())
+            manifest['keywords'] = keywords
+            if "category" in manifest:
+                del manifest['category']
+
+        # don't replace VCS URL
+        if "url" in manifest and "description" in manifest:
+            manifest['homepage'] = manifest['url']
+            del manifest['url']
+
+        if "architectures" in manifest:
+            platforms = []
+            platforms_map = {
+                "avr": "atmelavr",
+                "sam": "atmelsam",
+                "samd": "atmelsam",
+                "esp8266": "espressif8266",
+                "arc32": "intel_arc32"
+            }
+            for arch in manifest['architectures'].split(","):
+                arch = arch.strip()
+                if arch == "*":
+                    platforms = "*"
+                    break
+                if arch in platforms_map:
+                    platforms.append(platforms_map[arch])
+            manifest['platforms'] = platforms
+            del manifest['architectures']
+
+        # convert listed items via comma to array
+        for key in ("keywords", "frameworks", "platforms"):
+            if key not in manifest or \
+                    not isinstance(manifest[key], basestring):
+                continue
+            manifest[key] = [
+                i.strip() for i in manifest[key].split(",") if i.strip()
+            ]
+
         return manifest
 
     @staticmethod
@@ -129,13 +153,8 @@ class LibraryManager(BasePkgManager):
     def max_satisfying_repo_version(versions, requirements=None):
 
         def _cmp_dates(datestr1, datestr2):
-            from datetime import datetime
-            assert "T" in datestr1 and "T" in datestr2
-            dateformat = "%Y-%m-%d %H:%M:%S"
-            date1 = datetime.strptime(datestr1[:-1].replace("T", " "),
-                                      dateformat)
-            date2 = datetime.strptime(datestr2[:-1].replace("T", " "),
-                                      dateformat)
+            date1 = arrow.get(datestr1)
+            date2 = arrow.get(datestr2)
             if date1 == date2:
                 return 0
             return -1 if date1 < date2 else 1
@@ -150,7 +169,7 @@ class LibraryManager(BasePkgManager):
         for v in versions:
             specver = None
             try:
-                specver = semantic_version.Version(v['version'], partial=True)
+                specver = semantic_version.Version(v['name'], partial=True)
             except ValueError:
                 pass
 
@@ -158,30 +177,30 @@ class LibraryManager(BasePkgManager):
                 if not specver or specver not in reqspec:
                     continue
                 if not item or semantic_version.Version(
-                        item['version'], partial=True) < specver:
+                        item['name'], partial=True) < specver:
                     item = v
             elif requirements:
-                if requirements == v['version']:
+                if requirements == v['name']:
                     return v
             else:
-                if not item or _cmp_dates(item['date'], v['date']) == -1:
+                if not item or _cmp_dates(item['released'],
+                                          v['released']) == -1:
                     item = v
         return item
 
-    def get_latest_repo_version(self, name, requirements):
+    def get_latest_repo_version(self, name, requirements, silent=False):
         item = self.max_satisfying_repo_version(
             util.get_api_result(
-                "/lib/versions/%d" % self._get_pkg_id_by_name(name,
-                                                              requirements),
-                cache_valid="1h"),
-            requirements)
-        return item['version'] if item else None
+                "/lib/info/%d" % self.get_pkg_id_by_name(
+                    name, requirements, silent=silent),
+                cache_valid="1d")['versions'], requirements)
+        return item['name'] if item else None
 
-    def _get_pkg_id_by_name(self,
-                            name,
-                            requirements,
-                            silent=False,
-                            interactive=False):
+    def get_pkg_id_by_name(self,
+                           name,
+                           requirements,
+                           silent=False,
+                           interactive=False):
         if name.startswith("id="):
             return int(name[3:])
         # try to find ID from installed packages
@@ -196,7 +215,7 @@ class LibraryManager(BasePkgManager):
             }, silent, interactive)['id'])
 
     def _install_from_piorepo(self, name, requirements):
-        assert name.startswith("id=")
+        assert name.startswith("id="), name
         version = self.get_latest_repo_version(name, requirements)
         if not version:
             raise exception.UndefinedPackageVersion(requirements or "latest",
@@ -211,32 +230,32 @@ class LibraryManager(BasePkgManager):
             name, dl_data['url'].replace("http://", "https://")
             if app.get_setting("enable_ssl") else dl_data['url'], requirements)
 
-    def install(self,
-                name,
-                requirements=None,
-                silent=False,
-                trigger_event=True,
-                interactive=False):
-        already_installed = False
-        _name, _requirements, _url = self.parse_pkg_name(name, requirements)
-
+    def install(  # pylint: disable=arguments-differ
+            self,
+            name,
+            requirements=None,
+            silent=False,
+            trigger_event=True,
+            interactive=False):
+        pkg_dir = None
         try:
+            _name, _requirements, _url = self.parse_pkg_input(name,
+                                                              requirements)
             if not _url:
-                _name = "id=%d" % self._get_pkg_id_by_name(
+                name = "id=%d" % self.get_pkg_id_by_name(
                     _name,
                     _requirements,
                     silent=silent,
                     interactive=interactive)
-            already_installed = self.get_package(_name, _requirements, _url)
-            pkg_dir = BasePkgManager.install(
-                self, _name
-                if not _url else name, _requirements, silent, trigger_event)
+                requirements = _requirements
+            pkg_dir = BasePkgManager.install(self, name, requirements, silent,
+                                             trigger_event)
         except exception.InternetIsOffline as e:
             if not silent:
                 click.secho(str(e), fg="yellow")
             return
 
-        if already_installed:
+        if not pkg_dir:
             return
 
         manifest = self.load_manifest(pkg_dir)
@@ -295,34 +314,37 @@ class LibraryManager(BasePkgManager):
 
         lib_info = None
         result = util.get_api_result(
-            "/lib/search", dict(query=" ".join(query)), cache_valid="3d")
+            "/v2/lib/search", dict(query=" ".join(query)), cache_valid="3d")
         if result['total'] == 1:
             lib_info = result['items'][0]
         elif result['total'] > 1:
-            click.secho(
-                "Conflict: More than one library has been found "
-                "by request %s:" % json.dumps(filters),
-                fg="red",
-                err=True)
-            commands.lib.echo_liblist_header()
-            for item in result['items']:
-                commands.lib.echo_liblist_item(item)
-
-            if not interactive:
-                click.secho(
-                    "Automatically chose the first available library "
-                    "(use `--interactive` option to make a choice)",
-                    fg="yellow",
-                    err=True)
+            if silent and not interactive:
                 lib_info = result['items'][0]
             else:
-                deplib_id = click.prompt(
-                    "Please choose library ID",
-                    type=click.Choice([str(i['id']) for i in result['items']]))
+                click.secho(
+                    "Conflict: More than one library has been found "
+                    "by request %s:" % json.dumps(filters),
+                    fg="yellow",
+                    err=True)
                 for item in result['items']:
-                    if item['id'] == int(deplib_id):
-                        lib_info = item
-                        break
+                    commands.lib.print_lib_item(item)
+
+                if not interactive:
+                    click.secho(
+                        "Automatically chose the first available library "
+                        "(use `--interactive` option to make a choice)",
+                        fg="yellow",
+                        err=True)
+                    lib_info = result['items'][0]
+                else:
+                    deplib_id = click.prompt(
+                        "Please choose library ID",
+                        type=click.Choice(
+                            [str(i['id']) for i in result['items']]))
+                    for item in result['items']:
+                        if item['id'] == int(deplib_id):
+                            lib_info = item
+                            break
 
         if not lib_info:
             if filters.keys() == ["name"]:

platformio/managers/package.py

@@ -12,17 +12,19 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+import codecs
+import hashlib
 import json
 import os
 import shutil
-from os.path import basename, dirname, getsize, isdir, isfile, islink, join
+from os.path import basename, getsize, isdir, isfile, islink, join
 from tempfile import mkdtemp
 
 import click
 import requests
 import semantic_version
 
-from platformio import app, exception, telemetry, util
+from platformio import __version__, app, exception, telemetry, util
 from platformio.downloader import FileDownloader
 from platformio.unpacker import FileUnpacker
 from platformio.vcsclient import VCSClientFactory
@@ -73,6 +75,8 @@ class PackageRepoIterator(object):
 
 class PkgRepoMixin(object):
 
+    PIO_VERSION = semantic_version.Version(util.pepver_to_semver(__version__))
+
     @staticmethod
     def max_satisfying_repo_version(versions, requirements=None):
         item = None
@@ -85,9 +89,13 @@ class PkgRepoMixin(object):
                 pass
 
         for v in versions:
-            if ("system" in v and v['system'] not in ("all", "*") and
-                    systype not in v['system']):
+            if "system" in v and v['system'] not in ("all", "*") and \
+                    systype not in v['system']:
                 continue
+            if "platformio" in v.get("engines", {}):
+                if PkgRepoMixin.PIO_VERSION not in semantic_version.Spec(
+                        v['engines']['platformio']):
+                    continue
             specver = semantic_version.Version(v['version'])
             if reqspec and specver not in reqspec:
                 continue
@@ -95,7 +103,11 @@ class PkgRepoMixin(object):
                 item = v
         return item
 
-    def get_latest_repo_version(self, name, requirements):
+    def get_latest_repo_version(  # pylint: disable=unused-argument
+            self,
+            name,
+            requirements,
+            silent=False):
         version = None
         for versions in PackageRepoIterator(name, self.repositories):
             pkgdata = self.max_satisfying_repo_version(versions, requirements)
@@ -106,54 +118,202 @@ class PkgRepoMixin(object):
                 version = pkgdata['version']
         return version
 
+    def get_all_repo_versions(self, name):
+        result = []
+        for versions in PackageRepoIterator(name, self.repositories):
+            result.extend([v['version'] for v in versions])
+        return sorted(set(result))
+
 
 class PkgInstallerMixin(object):
 
-    VCS_MANIFEST_NAME = ".piopkgmanager.json"
+    SRC_MANIFEST_NAME = ".piopkgmanager.json"
 
-    def get_vcs_manifest_path(self, pkg_dir):
+    FILE_CACHE_VALID = "1m"  # 1 month
+    FILE_CACHE_MAX_SIZE = 1024 * 1024
+
+    MEMORY_CACHE = {}
+
+    @staticmethod
+    def cache_get(key, default=None):
+        return PkgInstallerMixin.MEMORY_CACHE.get(key, default)
+
+    @staticmethod
+    def cache_set(key, value):
+        PkgInstallerMixin.MEMORY_CACHE[key] = value
+
+    @staticmethod
+    def cache_reset():
+        PkgInstallerMixin.MEMORY_CACHE = {}
+
+    def read_dirs(self, src_dir):
+        cache_key = "read_dirs-%s" % src_dir
+        result = self.cache_get(cache_key)
+        if result:
+            return result
+        result = [
+            join(src_dir, name) for name in sorted(os.listdir(src_dir))
+            if isdir(join(src_dir, name))
+        ]
+        self.cache_set(cache_key, result)
+        return result
+
+    def download(self, url, dest_dir, sha1=None):
+        cache_key_fname = app.ContentCache.key_from_args(url, "fname")
+        cache_key_data = app.ContentCache.key_from_args(url, "data")
+        if self.FILE_CACHE_VALID:
+            with app.ContentCache() as cc:
+                fname = cc.get(cache_key_fname)
+                cache_path = cc.get_cache_path(cache_key_data)
+                if fname and isfile(cache_path):
+                    dst_path = join(dest_dir, fname)
+                    shutil.copy(cache_path, dst_path)
+                    return dst_path
+
+        fd = FileDownloader(url, dest_dir)
+        fd.start()
+        if sha1:
+            fd.verify(sha1)
+        dst_path = fd.get_filepath()
+        if not self.FILE_CACHE_VALID or getsize(
+                dst_path) > PkgInstallerMixin.FILE_CACHE_MAX_SIZE:
+            return dst_path
+
+        with app.ContentCache() as cc:
+            cc.set(cache_key_fname, basename(dst_path), self.FILE_CACHE_VALID)
+            cc.set(cache_key_data, "DUMMY", self.FILE_CACHE_VALID)
+            shutil.copy(dst_path, cc.get_cache_path(cache_key_data))
+        return dst_path
+
+    @staticmethod
+    def unpack(source_path, dest_dir):
+        fu = FileUnpacker(source_path, dest_dir)
+        return fu.start()
+
+    @staticmethod
+    def get_install_dirname(manifest):
+        name = manifest['name']
+        if "id" in manifest:
+            name += "_ID%d" % manifest['id']
+        return name
+
+    def get_src_manifest_path(self, pkg_dir):
         for item in os.listdir(pkg_dir):
             if not isdir(join(pkg_dir, item)):
                 continue
-            if isfile(join(pkg_dir, item, self.VCS_MANIFEST_NAME)):
-                return join(pkg_dir, item, self.VCS_MANIFEST_NAME)
+            if isfile(join(pkg_dir, item, self.SRC_MANIFEST_NAME)):
+                return join(pkg_dir, item, self.SRC_MANIFEST_NAME)
         return None
 
     def get_manifest_path(self, pkg_dir):
         if not isdir(pkg_dir):
             return None
-        manifest_path = join(pkg_dir, self.manifest_name)
-        if isfile(manifest_path):
-            return manifest_path
-        return self.get_vcs_manifest_path(pkg_dir)
-
-    def manifest_exists(self, pkg_dir):
-        return self.get_manifest_path(pkg_dir) is not None
-
-    def load_manifest(self, path):
-        assert path
-        pkg_dir = path
-        if isdir(path):
-            path = self.get_manifest_path(path)
-        else:
-            pkg_dir = dirname(pkg_dir)
-        if path:
-            if isfile(path) and path.endswith(self.VCS_MANIFEST_NAME):
-                pkg_dir = dirname(dirname(path))
-            manifest = util.load_json(path)
-            manifest['__pkg_dir'] = pkg_dir
-            return manifest
+        for name in self.manifest_names:
+            manifest_path = join(pkg_dir, name)
+            if isfile(manifest_path):
+                return manifest_path
         return None
 
-    def check_pkg_structure(self, pkg_dir):
-        if self.manifest_exists(pkg_dir):
-            return pkg_dir
+    def manifest_exists(self, pkg_dir):
+        return self.get_manifest_path(pkg_dir) or \
+            self.get_src_manifest_path(pkg_dir)
 
-        for root, _, _ in os.walk(pkg_dir):
+    def load_manifest(self, pkg_dir):
+        cache_key = "load_manifest-%s" % pkg_dir
+        result = self.cache_get(cache_key)
+        if result:
+            return result
+
+        manifest_path = self.get_manifest_path(pkg_dir)
+        if not manifest_path:
+            return None
+
+        # if non-registry packages: VCS or archive
+        src_manifest_path = self.get_src_manifest_path(pkg_dir)
+        src_manifest = None
+        if src_manifest_path:
+            src_manifest = util.load_json(src_manifest_path)
+
+        manifest = {}
+        if manifest_path.endswith(".json"):
+            manifest = util.load_json(manifest_path)
+        elif manifest_path.endswith(".properties"):
+            with codecs.open(manifest_path, encoding="utf-8") as fp:
+                for line in fp.readlines():
+                    if "=" not in line:
+                        continue
+                    key, value = line.split("=", 1)
+                    manifest[key.strip()] = value.strip()
+
+        if src_manifest:
+            if "name" not in manifest:
+                manifest['name'] = src_manifest['name']
+            if "version" in src_manifest:
+                manifest['version'] = src_manifest['version']
+            manifest['__src_url'] = src_manifest['url']
+
+        if "name" not in manifest:
+            manifest['name'] = basename(pkg_dir)
+        if "version" not in manifest:
+            manifest['version'] = "0.0.0"
+
+        manifest['__pkg_dir'] = pkg_dir
+        self.cache_set(cache_key, manifest)
+        return manifest
+
+    def get_installed(self):
+        items = []
+        for pkg_dir in self.read_dirs(self.package_dir):
+            manifest = self.load_manifest(pkg_dir)
+            if not manifest:
+                continue
+            assert "name" in manifest
+            items.append(manifest)
+        return items
+
+    def get_package(self, name, requirements=None, url=None):
+        pkg_id = int(name[3:]) if name.startswith("id=") else 0
+        best = None
+        for manifest in self.get_installed():
+            if url:
+                if manifest.get("__src_url") != url:
+                    continue
+            elif pkg_id and manifest.get("id") != pkg_id:
+                continue
+            elif not pkg_id and manifest['name'] != name:
+                continue
+
+            # strict version or VCS HASH
+            if requirements and requirements == manifest['version']:
+                return manifest
+
+            try:
+                if requirements and not semantic_version.Spec(
+                        requirements).match(
+                            semantic_version.Version(
+                                manifest['version'], partial=True)):
+                    continue
+                elif not best or (semantic_version.Version(
+                        manifest['version'], partial=True) >
+                                  semantic_version.Version(
+                                      best['version'], partial=True)):
+                    best = manifest
+            except ValueError:
+                pass
+
+        return best
+
+    def get_package_dir(self, name, requirements=None, url=None):
+        manifest = self.get_package(name, requirements, url)
+        return manifest.get("__pkg_dir") if manifest else None
+
+    def find_pkg_root(self, src_dir):
+        if self.manifest_exists(src_dir):
+            return src_dir
+        for root, _, _ in os.walk(src_dir):
             if self.manifest_exists(root):
                 return root
-
-        raise exception.MissingPackageManifest(self.manifest_name)
+        raise exception.MissingPackageManifest(", ".join(self.manifest_names))
 
     def _install_from_piorepo(self, name, requirements):
         pkg_dir = None
@@ -179,18 +339,25 @@ class PkgInstallerMixin(object):
                                                     util.get_systype())
         return pkg_dir
 
-    def _install_from_url(self, name, url, requirements=None, sha1=None):
+    def _install_from_url(self,
+                          name,
+                          url,
+                          requirements=None,
+                          sha1=None,
+                          track=False):
         pkg_dir = None
-        tmp_dir = mkdtemp("-package", "installing-", self.package_dir)
+        tmp_dir = mkdtemp("-package", "_tmp_installing-", self.package_dir)
+        src_manifest_dir = None
+        src_manifest = {"name": name, "url": url, "requirements": requirements}
 
         try:
             if url.startswith("file://"):
-                url = url[7:]
-                if isfile(url):
-                    self.unpack(url, tmp_dir)
+                _url = url[7:]
+                if isfile(_url):
+                    self.unpack(_url, tmp_dir)
                 else:
                     util.rmtree_(tmp_dir)
-                    shutil.copytree(url, tmp_dir)
+                    shutil.copytree(_url, tmp_dir)
             elif url.startswith(("http://", "https://")):
                 dlpath = self.download(url, tmp_dir, sha1)
                 assert isfile(dlpath)
@@ -199,71 +366,112 @@ class PkgInstallerMixin(object):
             else:
                 vcs = VCSClientFactory.newClient(tmp_dir, url)
                 assert vcs.export()
-                with open(join(vcs.storage_dir, self.VCS_MANIFEST_NAME),
-                          "w") as fp:
-                    json.dump({
-                        "name": name,
-                        "version": vcs.get_current_revision(),
-                        "url": url,
-                        "requirements": requirements
-                    }, fp)
+                src_manifest_dir = vcs.storage_dir
+                src_manifest['version'] = vcs.get_current_revision()
+
+            pkg_dir = self.find_pkg_root(tmp_dir)
+
+            # write source data to a special manifest
+            if track:
+                if not src_manifest_dir:
+                    src_manifest_dir = join(pkg_dir, ".pio")
+                self._update_src_manifest(src_manifest, src_manifest_dir)
 
-            pkg_dir = self.check_pkg_structure(tmp_dir)
             pkg_dir = self._install_from_tmp_dir(pkg_dir, requirements)
         finally:
             if isdir(tmp_dir):
                 util.rmtree_(tmp_dir)
         return pkg_dir
 
-    def _install_from_tmp_dir(self, tmp_dir, requirements=None):
-        tmpmanifest = self.load_manifest(tmp_dir)
-        assert set(["name", "version"]) <= set(tmpmanifest.keys())
-        name = tmpmanifest['name']
-        pkg_dir = join(self.package_dir, name)
-        if "id" in tmpmanifest:
-            name += "_ID%d" % tmpmanifest['id']
-            pkg_dir = join(self.package_dir, name)
+    def _update_src_manifest(self, data, src_dir):
+        if not isdir(src_dir):
+            os.makedirs(src_dir)
+        src_manifest_path = join(src_dir, self.SRC_MANIFEST_NAME)
+        _data = {}
+        if isfile(src_manifest_path):
+            _data = util.load_json(src_manifest_path)
+        _data.update(data)
+        with open(src_manifest_path, "w") as fp:
+            json.dump(_data, fp)
+
+    def _install_from_tmp_dir(  # pylint: disable=too-many-branches
+            self, tmp_dir, requirements=None):
+        tmp_manifest = self.load_manifest(tmp_dir)
+        assert set(["name", "version"]) <= set(tmp_manifest.keys())
+
+        pkg_dirname = self.get_install_dirname(tmp_manifest)
+        pkg_dir = join(self.package_dir, pkg_dirname)
+        cur_manifest = self.load_manifest(pkg_dir)
+
+        tmp_semver = None
+        cur_semver = None
+        try:
+            tmp_semver = semantic_version.Version(
+                tmp_manifest['version'], partial=True)
+            if cur_manifest:
+                cur_semver = semantic_version.Version(
+                    cur_manifest['version'], partial=True)
+        except ValueError:
+            pass
 
         # package should satisfy requirements
         if requirements:
             mismatch_error = (
                 "Package version %s doesn't satisfy requirements %s" % (
-                    tmpmanifest['version'], requirements))
+                    tmp_manifest['version'], requirements))
             try:
-                reqspec = semantic_version.Spec(requirements)
-                tmpmanver = semantic_version.Version(
-                    tmpmanifest['version'], partial=True)
-                assert tmpmanver in reqspec, mismatch_error
+                assert tmp_semver and tmp_semver in semantic_version.Spec(
+                    requirements), mismatch_error
+            except (AssertionError, ValueError):
+                assert tmp_manifest['version'] == requirements, mismatch_error
 
-                if self.manifest_exists(pkg_dir):
-                    curmanifest = self.load_manifest(pkg_dir)
-                    curmanver = semantic_version.Version(
-                        curmanifest['version'], partial=True)
-                    # if current package version < new package, backup it
-                    if tmpmanver > curmanver:
-                        os.rename(pkg_dir,
-                                  join(self.package_dir, "%s@%s" %
-                                       (name, curmanifest['version'])))
-                    elif tmpmanver < curmanver:
-                        pkg_dir = join(self.package_dir, "%s@%s" %
-                                       (name, tmpmanifest['version']))
-            except ValueError:
-                assert tmpmanifest['version'] == requirements, mismatch_error
+        # check if package already exists
+        if cur_manifest:
+            # 0-overwrite, 1-rename, 2-fix to a version
+            action = 0
+            if "__src_url" in cur_manifest:
+                if cur_manifest['__src_url'] != tmp_manifest.get("__src_url"):
+                    action = 1
+            elif "__src_url" in tmp_manifest:
+                action = 2
+            else:
+                if tmp_semver and (not cur_semver or tmp_semver > cur_semver):
+                    action = 1
+                elif tmp_semver and cur_semver and tmp_semver != cur_semver:
+                    action = 2
+
+            # rename
+            if action == 1:
+                target_dirname = "%s@%s" % (pkg_dirname,
+                                            cur_manifest['version'])
+                if "__src_url" in cur_manifest:
+                    target_dirname = "%s@src-%s" % (
+                        pkg_dirname,
+                        hashlib.md5(cur_manifest['__src_url']).hexdigest())
+                os.rename(pkg_dir, join(self.package_dir, target_dirname))
+            # fix to a version
+            elif action == 2:
+                target_dirname = "%s@%s" % (pkg_dirname,
+                                            tmp_manifest['version'])
+                if "__src_url" in tmp_manifest:
+                    target_dirname = "%s@src-%s" % (
+                        pkg_dirname,
+                        hashlib.md5(tmp_manifest['__src_url']).hexdigest())
+                pkg_dir = join(self.package_dir, target_dirname)
 
         # remove previous/not-satisfied package
         if isdir(pkg_dir):
             util.rmtree_(pkg_dir)
         os.rename(tmp_dir, pkg_dir)
         assert isdir(pkg_dir)
+        self.cache_reset()
         return pkg_dir
 
 
 class BasePkgManager(PkgRepoMixin, PkgInstallerMixin):
 
-    _INSTALLED_CACHE = {}
-
-    FILE_CACHE_VALID = "1m"  # 1 month
-    FILE_CACHE_MAX_SIZE = 1024 * 1024
+    # Handle circle dependencies
+    INSTALL_HISTORY = None
 
     def __init__(self, package_dir, repositories=None):
         self.repositories = repositories
@@ -273,57 +481,27 @@ class BasePkgManager(PkgRepoMixin, PkgInstallerMixin):
         assert isdir(self.package_dir)
 
     @property
-    def manifest_name(self):
+    def manifest_names(self):
         raise NotImplementedError()
 
-    def download(self, url, dest_dir, sha1=None):
-        cache_key_fname = app.ContentCache.key_from_args(url, "fname")
-        cache_key_data = app.ContentCache.key_from_args(url, "data")
-        if self.FILE_CACHE_VALID:
-            with app.ContentCache() as cc:
-                fname = cc.get(cache_key_fname)
-                cache_path = cc.get_cache_path(cache_key_data)
-                if fname and isfile(cache_path):
-                    dst_path = join(dest_dir, fname)
-                    shutil.copy(cache_path, dst_path)
-                    return dst_path
-
-        fd = FileDownloader(url, dest_dir)
-        fd.start()
-        if sha1:
-            fd.verify(sha1)
-        dst_path = fd.get_filepath()
-        if not self.FILE_CACHE_VALID or getsize(
-                dst_path) > BasePkgManager.FILE_CACHE_MAX_SIZE:
-            return dst_path
-
-        with app.ContentCache() as cc:
-            cc.set(cache_key_fname, basename(dst_path), self.FILE_CACHE_VALID)
-            cc.set(cache_key_data, "DUMMY", self.FILE_CACHE_VALID)
-            shutil.copy(dst_path, cc.get_cache_path(cache_key_data))
-        return dst_path
-
-    @staticmethod
-    def unpack(source_path, dest_dir):
-        fu = FileUnpacker(source_path, dest_dir)
-        return fu.start()
-
-    def reset_cache(self):
-        if self.package_dir in BasePkgManager._INSTALLED_CACHE:
-            del BasePkgManager._INSTALLED_CACHE[self.package_dir]
-
     def print_message(self, message, nl=True):
         click.echo("%s: %s" % (self.__class__.__name__, message), nl=nl)
 
     @staticmethod
-    def parse_pkg_name(  # pylint: disable=too-many-branches
+    def parse_pkg_input(  # pylint: disable=too-many-branches
             text, requirements=None):
         text = str(text)
-        url_marker = "://"
-        if not any([
-                requirements, "@" not in text, text.startswith("git@"),
-                url_marker in text
-        ]):
+        # git@github.com:user/package.git
+        url_marker = text[:4]
+        if url_marker not in ("git@", "git+") or ":" not in text:
+            url_marker = "://"
+
+        req_conditions = [
+            not requirements, "@" in text,
+            (url_marker != "git@" and "://git@" not in text) or
+            text.count("@") > 1
+        ]
+        if all(req_conditions):
             text, requirements = text.rsplit("@", 1)
         if text.isdigit():
             text = "id=" + text
@@ -339,22 +517,18 @@ class BasePkgManager(PkgRepoMixin, PkgInstallerMixin):
             url.startswith("http") and
             (url.split("#", 1)[0] if "#" in url else url).endswith(".git")
         ]
-
         if any(git_conditions):
             url = "git+" + url
+
         # Handle Developer Mbed URL
         # (https://developer.mbed.org/users/user/code/package/)
-        elif url.startswith("https://developer.mbed.org"):
+        if url.startswith("https://developer.mbed.org"):
             url = "hg+" + url
 
-        # git@github.com:user/package.git
-        if url.startswith("git@"):
-            url_marker = "git@"
-
         if any([s in url for s in ("\\", "/")]) and url_marker not in url:
             if isfile(url) or isdir(url):
                 url = "file://" + url
-            elif url.count("/") == 1 and not url.startswith("git@"):
+            elif url.count("/") == 1 and "git" not in url_marker:
                 url = "git+https://github.com/" + url
 
         # determine name
@@ -364,91 +538,73 @@ class BasePkgManager(PkgRepoMixin, PkgInstallerMixin):
                 _url = _url[:-1]
             name = basename(_url)
             if "." in name and not name.startswith("."):
-                name = name.split(".", 1)[0]
+                name = name.rsplit(".", 1)[0]
 
         if url_marker not in url:
             url = None
         return (name or text, requirements, url)
 
-    def get_installed(self):
-        if self.package_dir in BasePkgManager._INSTALLED_CACHE:
-            return BasePkgManager._INSTALLED_CACHE[self.package_dir]
-        items = []
-        for p in sorted(os.listdir(self.package_dir)):
-            pkg_dir = join(self.package_dir, p)
-            if not isdir(pkg_dir):
-                continue
-            manifest = self.load_manifest(pkg_dir)
-            if not manifest:
-                continue
-            assert set(["name", "version"]) <= set(manifest.keys())
-            items.append(manifest)
-        BasePkgManager._INSTALLED_CACHE[self.package_dir] = items
-        return items
+    def outdated(self, pkg_dir, requirements=None):
+        """
+        Has 3 different results:
+        `None` - unknown package, VCS is fixed to commit
+        `False` - package is up-to-date
+        `String` - a found latest version
+        """
+        assert isdir(pkg_dir)
+        latest = None
+        manifest = self.load_manifest(pkg_dir)
+        # skip a fixed package to a specific version
+        if "@" in pkg_dir and "__src_url" not in manifest:
+            return None
 
-    def get_package(self, name, requirements=None, url=None):
-        pkg_id = int(name[3:]) if name.startswith("id=") else 0
-        best = None
-        reqspec = None
-        if requirements:
+        if "__src_url" in manifest:
             try:
-                reqspec = semantic_version.Spec(requirements)
-            except ValueError:
-                pass
-
-        for manifest in self.get_installed():
-            if pkg_id and manifest.get("id") != pkg_id:
-                continue
-            elif not pkg_id and manifest['name'] != name:
-                continue
-            elif not reqspec and requirements:
-                if requirements == manifest['version']:
-                    best = manifest
-                    break
-                continue
-            try:
-                if reqspec and not reqspec.match(
-                        semantic_version.Version(
-                            manifest['version'], partial=True)):
-                    continue
-                elif not best or (semantic_version.Version(
-                        manifest['version'], partial=True) >
-                                  semantic_version.Version(
-                                      best['version'], partial=True)):
-                    best = manifest
-            except ValueError:
-                pass
-        if best:
-            # check that URL is the same in installed package (VCS)
-            if url and best.get("url") != url:
+                vcs = VCSClientFactory.newClient(
+                    pkg_dir, manifest['__src_url'], silent=True)
+            except (AttributeError, exception.PlatformioException):
+                return None
+            if not vcs.can_be_updated:
+                return None
+            latest = vcs.get_latest_revision()
+        else:
+            try:
+                latest = self.get_latest_repo_version(
+                    "id=%d" % manifest['id']
+                    if "id" in manifest else manifest['name'],
+                    requirements,
+                    silent=True)
+            except (exception.PlatformioException, ValueError):
                 return None
-            return best
-        return None
 
-    def get_package_dir(self, name, requirements=None, url=None):
-        package = self.get_package(name, requirements, url)
-        return package.get("__pkg_dir") if package else None
+        if not latest:
+            return None
 
-    def is_outdated(self, name, requirements=None):
-        package_dir = self.get_package_dir(name, requirements)
-        if not package_dir:
-            click.secho(
-                "%s @ %s is not installed" % (name, requirements or "*"),
-                fg="yellow")
-            return
-        if self.get_vcs_manifest_path(package_dir):
-            return False
-        manifest = self.load_manifest(package_dir)
-        return manifest['version'] != self.get_latest_repo_version(
-            name, requirements)
+        up_to_date = False
+        try:
+            assert "__src_url" not in manifest
+            up_to_date = (semantic_version.Version.coerce(manifest['version'])
+                          >= semantic_version.Version.coerce(latest))
+        except (AssertionError, ValueError):
+            up_to_date = latest == manifest['version']
+
+        return False if up_to_date else latest
 
     def install(self,
                 name,
                 requirements=None,
                 silent=False,
-                trigger_event=True,
-                interactive=False):  # pylint: disable=unused-argument
-        name, requirements, url = self.parse_pkg_name(name, requirements)
+                trigger_event=True):
+
+        # avoid circle dependencies
+        if not self.INSTALL_HISTORY:
+            self.INSTALL_HISTORY = []
+        history_key = "%s-%s" % (name, requirements) if requirements else name
+        if history_key in self.INSTALL_HISTORY:
+            return
+        self.INSTALL_HISTORY.append(history_key)
+
+        name, requirements, url = self.parse_pkg_input(name, requirements)
         package_dir = self.get_package_dir(name, requirements, url)
 
         if not package_dir or not silent:
@@ -465,15 +621,16 @@ class BasePkgManager(PkgRepoMixin, PkgInstallerMixin):
             return package_dir
 
         if url:
-            pkg_dir = self._install_from_url(name, url, requirements)
+            pkg_dir = self._install_from_url(
+                name, url, requirements, track=True)
         else:
             pkg_dir = self._install_from_piorepo(name, requirements)
         if not pkg_dir or not self.manifest_exists(pkg_dir):
             raise exception.PackageInstallError(name, requirements or "*",
                                                 util.get_systype())
 
-        self.reset_cache()
         manifest = self.load_manifest(pkg_dir)
+        assert manifest
 
         if trigger_event:
             telemetry.on_event(
@@ -481,37 +638,48 @@ class BasePkgManager(PkgRepoMixin, PkgInstallerMixin):
                 action="Install",
                 label=manifest['name'])
 
-        click.secho(
-            "{name} @ {version} has been successfully installed!".format(
-                **manifest),
-            fg="green")
+        if not silent:
+            click.secho(
+                "{name} @ {version} has been successfully installed!".format(
+                    **manifest),
+                fg="green")
 
         return pkg_dir
 
-    def uninstall(self, name, requirements=None, trigger_event=True):
-        name, requirements, url = self.parse_pkg_name(name, requirements)
-        package_dir = self.get_package_dir(name, requirements, url)
-        if not package_dir:
-            click.secho(
-                "%s @ %s is not installed" % (name, requirements or "*"),
-                fg="yellow")
-            return
+    def uninstall(self, package, requirements=None, trigger_event=True):
+        if isdir(package):
+            pkg_dir = package
+        else:
+            name, requirements, url = self.parse_pkg_input(package,
+                                                           requirements)
+            pkg_dir = self.get_package_dir(name, requirements, url)
 
-        manifest = self.load_manifest(package_dir)
+        if not pkg_dir:
+            raise exception.UnknownPackage("%s @ %s" %
+                                           (package, requirements or "*"))
+
+        manifest = self.load_manifest(pkg_dir)
         click.echo(
             "Uninstalling %s @ %s: \t" % (click.style(
                 manifest['name'], fg="cyan"), manifest['version']),
             nl=False)
 
-        if isdir(package_dir):
-            if islink(package_dir):
-                os.unlink(package_dir)
-            else:
-                util.rmtree_(package_dir)
+        if islink(pkg_dir):
+            os.unlink(pkg_dir)
+        else:
+            util.rmtree_(pkg_dir)
+        self.cache_reset()
+
+        # unfix package with the same name
+        pkg_dir = self.get_package_dir(manifest['name'])
+        if pkg_dir and "@" in pkg_dir:
+            os.rename(
+                pkg_dir,
+                join(self.package_dir, self.get_install_dirname(manifest)))
+            self.cache_reset()
 
         click.echo("[%s]" % click.style("OK", fg="green"))
 
-        self.reset_cache()
         if trigger_event:
             telemetry.on_event(
                 category=self.__class__.__name__,
@@ -521,77 +689,49 @@ class BasePkgManager(PkgRepoMixin, PkgInstallerMixin):
 
     def update(  # pylint: disable=too-many-return-statements
             self,
-            name,
+            package,
             requirements=None,
             only_check=False):
-        name, requirements, url = self.parse_pkg_name(name, requirements)
-        package_dir = self.get_package_dir(name, None, url)
-        if not package_dir:
-            click.secho(
-                "%s @ %s is not installed" % (name, requirements or "*"),
-                fg="yellow")
+        if isdir(package):
+            pkg_dir = package
+        else:
+            pkg_dir = self.get_package_dir(*self.parse_pkg_input(package))
+
+        if not pkg_dir:
+            raise exception.UnknownPackage("%s @ %s" %
+                                           (package, requirements or "*"))
+
+        manifest = self.load_manifest(pkg_dir)
+        name = manifest['name']
+
+        click.echo(
+            "{} {:<40} @ {:<15}".format(
+                "Checking" if only_check else "Updating",
+                click.style(manifest['name'], fg="cyan"), manifest['version']),
+            nl=False)
+        if not util.internet_on():
+            click.echo("[%s]" % (click.style("Off-line", fg="yellow")))
             return
 
-        is_vcs_pkg = False
-        if self.get_vcs_manifest_path(package_dir):
-            is_vcs_pkg = True
-            manifest_path = self.get_vcs_manifest_path(package_dir)
+        latest = self.outdated(pkg_dir, requirements)
+        if latest:
+            click.echo("[%s]" % (click.style(latest, fg="red")))
+        elif latest is False:
+            click.echo("[%s]" % (click.style("Up-to-date", fg="green")))
         else:
-            manifest_path = self.get_manifest_path(package_dir)
+            click.echo("[%s]" % (click.style("Skip", fg="yellow")))
 
-        manifest = self.load_manifest(manifest_path)
-        click.echo(
-            "%s %s @ %s: \t" % ("Checking"
-                                if only_check else "Updating", click.style(
-                                    manifest['name'], fg="cyan"),
-                                manifest['version']),
-            nl=False)
-        if is_vcs_pkg:
-            if only_check:
-                click.echo("[%s]" % (click.style("Skip", fg="yellow")))
-                return
-            click.echo("[%s]" % (click.style("VCS", fg="yellow")))
-            vcs = VCSClientFactory.newClient(package_dir, manifest['url'])
-            if not vcs.can_be_updated:
-                click.secho(
-                    "Skip update because repository is fixed "
-                    "to %s revision" % manifest['version'],
-                    fg="yellow")
-                return
+        if only_check or not latest:
+            return
+
+        if "__src_url" in manifest:
+            vcs = VCSClientFactory.newClient(pkg_dir, manifest['__src_url'])
             assert vcs.update()
-            with open(manifest_path, "w") as fp:
-                manifest['version'] = vcs.get_current_revision()
-                json.dump(manifest, fp)
+            self._update_src_manifest(
+                dict(version=vcs.get_current_revision()), vcs.storage_dir)
         else:
-            latest_version = None
-            try:
-                latest_version = self.get_latest_repo_version(name,
-                                                              requirements)
-            except exception.PlatformioException:
-                pass
-            if not latest_version:
-                click.echo("[%s]" % (click.style(
-                    "Off-line" if not util.internet_on() else "Unknown",
-                    fg="yellow")))
-                return
-
-            up_to_date = False
-            try:
-                up_to_date = (
-                    semantic_version.Version.coerce(manifest['version']) >=
-                    semantic_version.Version.coerce(latest_version))
-            except ValueError:
-                up_to_date = latest_version == manifest['version']
-
-            if up_to_date:
-                click.echo("[%s]" % (click.style("Up-to-date", fg="green")))
-                return
-
-            click.echo("[%s]" % (click.style("Out-of-date", fg="red")))
-            if only_check:
-                return
-            self.uninstall(name, manifest['version'], trigger_event=False)
-            self.install(name, latest_version, trigger_event=False)
+            self.uninstall(pkg_dir, trigger_event=False)
+            self.install(name, latest, trigger_event=False)
 
         telemetry.on_event(
             category=self.__class__.__name__,
@@ -605,5 +745,5 @@ class PackageManager(BasePkgManager):
     FILE_CACHE_VALID = None  # disable package caching
 
     @property
-    def manifest_name(self):
-        return "package.json"
+    def manifest_names(self):
+        return ["package.json"]

platformio/managers/platform.py

@@ -20,8 +20,10 @@ from multiprocessing import cpu_count
 from os.path import basename, dirname, isdir, isfile, join
 
 import click
+import semantic_version
 
-from platformio import app, exception, util
+from platformio import __version__, app, exception, util
+from platformio.managers.core import get_core_package_dir
 from platformio.managers.package import BasePkgManager, PackageManager
 
 
@@ -41,8 +43,17 @@ class PlatformManager(BasePkgManager):
                                 repositories)
 
     @property
-    def manifest_name(self):
-        return "platform.json"
+    def manifest_names(self):
+        return ["platform.json"]
+
+    def get_manifest_path(self, pkg_dir):
+        if not isdir(pkg_dir):
+            return None
+        for name in self.manifest_names:
+            manifest_path = join(pkg_dir, name)
+            if isfile(manifest_path):
+                return manifest_path
+        return None
 
     def install(self,
                 name,
@@ -50,50 +61,80 @@ class PlatformManager(BasePkgManager):
                 with_packages=None,
                 without_packages=None,
                 skip_default_package=False,
+                trigger_event=True,
+                silent=False,
                 **_):  # pylint: disable=too-many-arguments
-        platform_dir = BasePkgManager.install(self, name, requirements)
-        p = PlatformFactory.newPlatform(self.get_manifest_path(platform_dir))
-        p.install_packages(with_packages, without_packages,
-                           skip_default_package)
+        platform_dir = BasePkgManager.install(
+            self, name, requirements, silent=silent)
+        p = PlatformFactory.newPlatform(platform_dir)
+
+        # @Hook: when 'update' operation (trigger_event is False),
+        # don't cleanup packages or install them
+        if not trigger_event:
+            return True
+        p.install_packages(
+            with_packages,
+            without_packages,
+            skip_default_package,
+            silent=silent)
         self.cleanup_packages(p.packages.keys())
         return True
 
-    def uninstall(self, name, requirements=None, trigger_event=True):
-        name, requirements, _ = self.parse_pkg_name(name, requirements)
-        p = PlatformFactory.newPlatform(name, requirements)
-        BasePkgManager.uninstall(self, name, requirements)
-        # trigger event is disabled when upgrading operation
-        # don't cleanup packages, "install" will do that
-        if trigger_event:
-            self.cleanup_packages(p.packages.keys())
+    def uninstall(self, package, requirements=None, trigger_event=True):
+        if isdir(package):
+            pkg_dir = package
+        else:
+            name, requirements, url = self.parse_pkg_input(package,
+                                                           requirements)
+            pkg_dir = self.get_package_dir(name, requirements, url)
+
+        p = PlatformFactory.newPlatform(pkg_dir)
+        BasePkgManager.uninstall(self, pkg_dir, requirements)
+
+        # @Hook: when 'update' operation (trigger_event is False),
+        # don't cleanup packages or install them
+        if not trigger_event:
+            return True
+
+        self.cleanup_packages(p.packages.keys())
         return True
 
     def update(  # pylint: disable=arguments-differ
             self,
-            name,
+            package,
             requirements=None,
-            only_packages=False,
-            only_check=False):
-        name, requirements, _ = self.parse_pkg_name(name, requirements)
+            only_check=False,
+            only_packages=False):
+        if isdir(package):
+            pkg_dir = package
+        else:
+            name, requirements, url = self.parse_pkg_input(package,
+                                                           requirements)
+            pkg_dir = self.get_package_dir(name, requirements, url)
+
+        p = PlatformFactory.newPlatform(pkg_dir)
+        pkgs_before = pkgs_after = p.get_installed_packages().keys()
+
         if not only_packages:
-            BasePkgManager.update(self, name, requirements, only_check)
-        p = PlatformFactory.newPlatform(name, requirements)
+            BasePkgManager.update(self, pkg_dir, requirements, only_check)
+            p = PlatformFactory.newPlatform(pkg_dir)
+            pkgs_after = p.get_installed_packages().keys()
+
         p.update_packages(only_check)
         self.cleanup_packages(p.packages.keys())
+
+        pkgs_missed = set(pkgs_before) - set(pkgs_after)
+        if pkgs_missed:
+            p.install_packages(
+                with_packages=pkgs_missed, skip_default_package=True)
+
         return True
 
-    def is_outdated(self, name, requirements=None):
-        if BasePkgManager.is_outdated(self, name, requirements):
-            return True
-        p = PlatformFactory.newPlatform(name, requirements)
-        return p.are_outdated_packages()
-
     def cleanup_packages(self, names):
-        self.reset_cache()
+        self.cache_reset()
         deppkgs = {}
         for manifest in PlatformManager().get_installed():
-            p = PlatformFactory.newPlatform(manifest['name'],
-                                            manifest['version'])
+            p = PlatformFactory.newPlatform(manifest['__pkg_dir'])
             for pkgname, pkgmanifest in p.get_installed_packages().items():
                 if pkgname not in deppkgs:
                     deppkgs[pkgname] = set()
@@ -105,32 +146,34 @@ class PlatformManager(BasePkgManager):
                 continue
             if (manifest['name'] not in deppkgs or
                     manifest['version'] not in deppkgs[manifest['name']]):
-                pm.uninstall(
-                    manifest['name'], manifest['version'], trigger_event=False)
+                pm.uninstall(manifest['__pkg_dir'], trigger_event=False)
 
-        self.reset_cache()
+        self.cache_reset()
         return True
 
     def get_installed_boards(self):
         boards = []
         for manifest in self.get_installed():
-            p = PlatformFactory.newPlatform(
-                self.get_manifest_path(manifest['__pkg_dir']))
+            p = PlatformFactory.newPlatform(manifest['__pkg_dir'])
             for config in p.get_boards().values():
-                boards.append(config.get_brief_data())
+                board = config.get_brief_data()
+                if board not in boards:
+                    boards.append(board)
         return boards
 
     @staticmethod
     @util.memoized
     def get_registered_boards():
-        return util.get_api_result("/boards", cache_valid="365d")
+        return util.get_api_result("/boards", cache_valid="30d")
 
-    def board_config(self, id_):
+    def board_config(self, id_, platform=None):
         for manifest in self.get_installed_boards():
-            if manifest['id'] == id_:
+            if manifest['id'] == id_ and (not platform or
+                                          manifest['platform'] == platform):
                 return manifest
         for manifest in self.get_registered_boards():
-            if manifest['id'] == id_:
+            if manifest['id'] == id_ and (not platform or
+                                          manifest['platform'] == platform):
                 return manifest
         raise exception.UnknownBoard(id_)
 
@@ -155,7 +198,10 @@ class PlatformFactory(object):
     @classmethod
     def newPlatform(cls, name, requirements=None):
         platform_dir = None
-        if name.endswith("platform.json") and isfile(name):
+        if isdir(name):
+            platform_dir = name
+            name = PlatformManager().load_manifest(platform_dir)['name']
+        elif name.endswith("platform.json") and isfile(name):
             platform_dir = dirname(name)
             name = util.load_json(name)['name']
         else:
@@ -189,8 +235,8 @@ class PlatformPackagesMixin(object):
                          without_packages=None,
                          skip_default_package=False,
                          silent=False):
-        with_packages = set(self.pkg_types_to_names(with_packages or []))
-        without_packages = set(self.pkg_types_to_names(without_packages or []))
+        with_packages = set(self.find_pkg_names(with_packages or []))
+        without_packages = set(self.find_pkg_names(without_packages or []))
 
         upkgs = with_packages | without_packages
         ppkgs = set(self.packages.keys())
@@ -198,44 +244,86 @@ class PlatformPackagesMixin(object):
             raise exception.UnknownPackage(", ".join(upkgs - ppkgs))
 
         for name, opts in self.packages.items():
+            version = opts.get("version", "")
             if name in without_packages:
                 continue
             elif (name in with_packages or
                   not (skip_default_package or opts.get("optional", False))):
-                if any([s in opts.get("version", "") for s in ("\\", "/")]):
-                    self.pm.install(
-                        "%s=%s" % (name, opts['version']), silent=silent)
+                if self.is_valid_requirements(version):
+                    self.pm.install(name, version, silent=silent)
                 else:
-                    self.pm.install(name, opts.get("version"), silent=silent)
+                    requirements = None
+                    if "@" in version:
+                        version, requirements = version.rsplit("@", 1)
+                    self.pm.install(
+                        "%s=%s" % (name, version), requirements, silent=silent)
 
         return True
 
-    def get_installed_packages(self):
-        items = {}
-        for name, opts in self.packages.items():
-            package = self.pm.get_package(name, opts['version'])
-            if package:
-                items[name] = package
-        return items
+    def find_pkg_names(self, items):
+        result = []
+        for item in items:
+            candidate = item
+
+            # lookup by package types
+            for _name, _opts in self.packages.items():
+                if _opts.get("type") == item:
+                    candidate = _name
+
+            if (self.frameworks and item.startswith("framework-") and
+                    item[10:] in self.frameworks):
+                candidate = self.frameworks[item[10:]]['package']
+
+            result.append(candidate)
+        return result
 
     def update_packages(self, only_check=False):
-        for name in self.get_installed_packages():
-            self.pm.update(name, self.packages[name]['version'], only_check)
+        for name, manifest in self.get_installed_packages().items():
+            version = self.packages[name].get("version", "")
+            if "@" in version:
+                _, version = version.rsplit("@", 1)
+            self.pm.update(manifest['__pkg_dir'], version, only_check)
+
+    def get_installed_packages(self):
+        items = {}
+        for name in self.packages:
+            pkg_dir = self.get_package_dir(name)
+            if pkg_dir:
+                items[name] = self.pm.load_manifest(pkg_dir)
+        return items
 
     def are_outdated_packages(self):
-        for name, opts in self.get_installed_packages().items():
-            if (opts['version'] != self.pm.get_latest_repo_version(
-                    name, self.packages[name].get("version"))):
+        for name, manifest in self.get_installed_packages().items():
+            version = self.packages[name].get("version", "")
+            if "@" in version:
+                _, version = version.rsplit("@", 1)
+            if self.pm.outdated(manifest['__pkg_dir'], version):
                 return True
         return False
 
     def get_package_dir(self, name):
-        return self.pm.get_package_dir(name,
-                                       self.packages[name].get("version"))
+        version = self.packages[name].get("version", "")
+        if self.is_valid_requirements(version):
+            return self.pm.get_package_dir(name, version)
+        else:
+            return self.pm.get_package_dir(*self._parse_pkg_input(name,
+                                                                  version))
 
     def get_package_version(self, name):
-        package = self.pm.get_package(name, self.packages[name].get("version"))
-        return package['version'] if package else None
+        pkg_dir = self.get_package_dir(name)
+        if not pkg_dir:
+            return None
+        return self.pm.load_manifest(pkg_dir).get("version")
+
+    @staticmethod
+    def is_valid_requirements(requirements):
+        return requirements and "://" not in requirements
+
+    def _parse_pkg_input(self, name, version):
+        requirements = None
+        if "@" in version:
+            version, requirements = version.rsplit("@", 1)
+        return self.pm.parse_pkg_input("%s=%s" % (name, version), requirements)
 
 
 class PlatformRunMixin(object):
@@ -270,7 +358,7 @@ class PlatformRunMixin(object):
     def _run_scons(self, variables, targets):
         cmd = [
             util.get_pythonexe_path(),
-            join(self.get_package_dir("tool-scons"), "script", "scons"), "-Q",
+            join(get_core_package_dir("tool-scons"), "script", "scons"), "-Q",
             "-j %d" % self.get_job_nums(), "--warn=no-no-parallel-support",
             "-f", join(util.get_source_dir(), "builder", "main.py")
         ]
@@ -316,8 +404,10 @@ class PlatformRunMixin(object):
             return 1
 
 
-class PlatformBase(PlatformPackagesMixin, PlatformRunMixin):
+class PlatformBase(  # pylint: disable=too-many-public-methods
+        PlatformPackagesMixin, PlatformRunMixin):
 
+    PIO_VERSION = semantic_version.Version(util.pepver_to_semver(__version__))
     _BOARDS_CACHE = {}
 
     def __init__(self, manifest_path):
@@ -332,6 +422,12 @@ class PlatformBase(PlatformPackagesMixin, PlatformRunMixin):
         self.silent = False
         self.verbose = False
 
+        if self.engines and "platformio" in self.engines:
+            if self.PIO_VERSION not in semantic_version.Spec(
+                    self.engines['platformio']):
+                raise exception.IncompatiblePlatform(self.name,
+                                                     str(self.PIO_VERSION))
+
     @property
     def name(self):
         return self._manifest['name']
@@ -356,6 +452,10 @@ class PlatformBase(PlatformPackagesMixin, PlatformRunMixin):
     def vendor_url(self):
         return self._manifest.get("url")
 
+    @property
+    def repository_url(self):
+        return self._manifest.get("repository", {}).get("url")
+
     @property
     def license(self):
         return self._manifest.get("license")
@@ -364,6 +464,10 @@ class PlatformBase(PlatformPackagesMixin, PlatformRunMixin):
     def frameworks(self):
         return self._manifest.get("frameworks")
 
+    @property
+    def engines(self):
+        return self._manifest.get("engines")
+
     @property
     def manifest(self):
         return self._manifest
@@ -435,20 +539,6 @@ class PlatformBase(PlatformPackagesMixin, PlatformRunMixin):
     def get_package_type(self, name):
         return self.packages[name].get("type")
 
-    def pkg_types_to_names(self, types):
-        names = []
-        for type_ in types:
-            name = type_
-            # lookup by package types
-            for _name, _opts in self.packages.items():
-                if _opts.get("type") == type_:
-                    name = None
-                    names.append(_name)
-            # if type is the right name
-            if name:
-                names.append(name)
-        return names
-
     def configure_default_packages(self, variables, targets):
         # enable used frameworks
         frameworks = variables.get("pioframework", [])
@@ -460,8 +550,9 @@ class PlatformBase(PlatformPackagesMixin, PlatformRunMixin):
             framework = framework.lower().strip()
             if not framework or framework not in self.frameworks:
                 continue
-            _pkg_name = self.frameworks[framework]['package']
-            self.packages[_pkg_name]['optional'] = False
+            _pkg_name = self.frameworks[framework].get("package")
+            if _pkg_name:
+                self.packages[_pkg_name]['optional'] = False
 
         # enable upload tools for upload targets
         if any(["upload" in t for t in targets] + ["program" in targets]):
@@ -472,16 +563,29 @@ class PlatformBase(PlatformPackagesMixin, PlatformRunMixin):
                     # skip all packages, allow only upload tools
                     self.packages[_name]['optional'] = True
 
-        if "__test" in targets and "tool-unity" not in self.packages:
-            self.packages['tool-unity'] = {
-                "version": "~1.20302.1",
-                "optional": False
-            }
-        if "tool-scons" not in self.packages:
-            self.packages['tool-scons'] = {
-                "version": "~3.20501.2",
-                "optional": False
-            }
+    def get_lib_storages(self):
+        storages = []
+        for opts in (self.frameworks or {}).values():
+            if "package" not in opts:
+                continue
+            pkg_dir = self.get_package_dir(opts['package'])
+            if not pkg_dir or not isdir(join(pkg_dir, "libraries")):
+                continue
+            libs_dir = join(pkg_dir, "libraries")
+            storages.append({"name": opts['package'], "path": libs_dir})
+            libcores_dir = join(libs_dir, "__cores__")
+            if not isdir(libcores_dir):
+                continue
+            for item in os.listdir(libcores_dir):
+                libcore_dir = join(libcores_dir, item)
+                if not isdir(libcore_dir):
+                    continue
+                storages.append({
+                    "name": "%s-core-%s" % (opts['package'], item),
+                    "path": libcore_dir
+                })
+
+        return storages
 
 
 class PlatformBoardConfig(object):

platformio/telemetry.py

@@ -121,8 +121,8 @@ class MeasurementProtocol(TelemetryBase):
                        "settings", "account"):
             cmd_path = args[:2]
         if args[0] == "lib" and len(args) > 1:
-            lib_subcmds = ("install", "list", "register", "search", "show",
-                           "uninstall", "update")
+            lib_subcmds = ("builtin", "install", "list", "register", "search",
+                           "show", "stats", "uninstall", "update")
             sub_cmd = _first_arg_from_list(args[1:], lib_subcmds)
             if sub_cmd:
                 cmd_path.append(sub_cmd)
@@ -227,8 +227,13 @@ class MPDataPusher(object):
                 timeout=1)
             r.raise_for_status()
             return True
+        except requests.exceptions.HTTPError as e:
+            # skip Bad Request
+            if 400 >= e.response.status_code < 500:
+                return True
         except:  # pylint: disable=W0702
-            self._http_offline = True
+            pass
+        self._http_offline = True
         return False
 
 
@@ -304,7 +309,8 @@ def on_exception(e):
         "Error" in e.__class__.__name__
     ])
     mp = MeasurementProtocol()
-    mp['exd'] = "%s: %s" % (type(e).__name__, format_exc() if is_crash else e)
+    mp['exd'] = ("%s: %s" % (type(e).__name__, format_exc()
+                             if is_crash else e))[:2048]
     mp['exf'] = 1 if is_crash else 0
     mp.send("exception")
 

platformio/util.py

@@ -141,7 +141,12 @@ class memoized(object):
 
     def __get__(self, obj, objtype):
         '''Support instance methods.'''
-        return functools.partial(self.__call__, obj)
+        fn = functools.partial(self.__call__, obj)
+        fn.reset = self._reset
+        return fn
+
+    def _reset(self):
+        self.cache = {}
 
 
 def singleton(cls):
@@ -157,8 +162,12 @@ def singleton(cls):
 
 
 def load_json(file_path):
-    with open(file_path, "r") as f:
-        return json.load(f)
+    try:
+        with open(file_path, "r") as f:
+            return json.load(f)
+    except ValueError:
+        raise exception.PlatformioException("Could not load broken JSON: %s" %
+                                            file_path)
 
 
 def get_systype():
@@ -458,11 +467,12 @@ def _get_api_result(
                 auth=auth,
                 verify=disable_ssl_check)
         else:
-            r = _api_request_session().get(url,
-                                           params=params,
-                                           headers=headers,
-                                           auth=auth,
-                                           verify=disable_ssl_check)
+            r = _api_request_session().get(
+                url,
+                params=params,
+                headers=headers,
+                auth=auth,
+                verify=disable_ssl_check)
         result = r.json()
         r.raise_for_status()
     except requests.exceptions.HTTPError as e:
@@ -558,7 +568,7 @@ def where_is_program(program, envpath=None):
 
 
 def pepver_to_semver(pepver):
-    return re.sub(r"(\.\d+)\.?(dev|a|b|rc|post)", r"\1-\2", pepver, 1)
+    return re.sub(r"(\.\d+)\.?(dev|a|b|rc|post)", r"\1-\2.", pepver, 1)
 
 
 def rmtree_(path):
@@ -568,3 +578,28 @@ def rmtree_(path):
         os.remove(name)
 
     return rmtree(path, onerror=_onerror)
+
+
+#
+# Glob.Escape from Python 3.4
+# https://github.com/python/cpython/blob/master/Lib/glob.py#L161
+#
+
+try:
+    from glob import escape as glob_escape  # pylint: disable=unused-import
+except ImportError:
+    magic_check = re.compile('([*?[])')
+    magic_check_bytes = re.compile(b'([*?[])')
+
+    def glob_escape(pathname):
+        """Escape all special characters.
+        """
+        # Escaping is done by wrapping any of "*?[" between square brackets.
+        # Metacharacters do not work in the drive part and shouldn't be
+        # escaped.
+        drive, pathname = os.path.splitdrive(pathname)
+        if isinstance(pathname, bytes):
+            pathname = magic_check_bytes.sub(br'[\1]', pathname)
+        else:
+            pathname = magic_check.sub(r'[\1]', pathname)
+        return drive + pathname

platformio/vcsclient.py

@@ -25,21 +25,22 @@ from platformio.exception import PlatformioException
 class VCSClientFactory(object):
 
     @staticmethod
-    def newClient(src_dir, remote_url):
+    def newClient(src_dir, remote_url, silent=False):
         result = urlparse(remote_url)
         type_ = result.scheme
+        tag = None
         if not type_ and remote_url.startswith("git@"):
             type_ = "git"
         elif "+" in result.scheme:
             type_, _ = result.scheme.split("+", 1)
             remote_url = remote_url[len(type_) + 1:]
-        if result.fragment:
-            remote_url = remote_url.rsplit("#", 1)[0]
+        if "#" in remote_url:
+            remote_url, tag = remote_url.rsplit("#", 1)
         if not type_:
             raise PlatformioException("VCS: Unknown repository type %s" %
                                       remote_url)
         obj = getattr(modules[__name__], "%sClient" % type_.title())(
-            src_dir, remote_url, result.fragment)
+            src_dir, remote_url, tag, silent)
         assert isinstance(obj, VCSClientBase)
         return obj
 
@@ -48,17 +49,21 @@ class VCSClientBase(object):
 
     command = None
 
-    def __init__(self, src_dir, remote_url=None, tag=None):
+    def __init__(self, src_dir, remote_url=None, tag=None, silent=False):
         self.src_dir = src_dir
         self.remote_url = remote_url
         self.tag = tag
+        self.silent = silent
         self.check_client()
 
     def check_client(self):
         try:
             assert self.command
-            assert self.run_cmd(["--version"])
-        except (AssertionError, OSError):
+            if self.silent:
+                self.get_cmd_output(["--version"])
+            else:
+                assert self.run_cmd(["--version"])
+        except (AssertionError, OSError, PlatformioException):
             raise PlatformioException(
                 "VCS: `%s` client is not installed in your system" %
                 self.command)
@@ -81,6 +86,9 @@ class VCSClientBase(object):
     def get_current_revision(self):
         raise NotImplementedError
 
+    def get_latest_revision(self):
+        return None if self.can_be_updated else self.get_current_revision()
+
     def run_cmd(self, args, **kwargs):
         args = [self.command] + args
         if "cwd" not in kwargs:
@@ -108,6 +116,16 @@ class GitClient(VCSClientBase):
         output = output.replace("*", "")  # fix active branch
         return [b.strip() for b in output.split("\n")]
 
+    def get_current_branch(self):
+        output = self.get_cmd_output(["branch"])
+        for line in output.split("\n"):
+            line = line.strip()
+            if line.startswith("*"):
+                branch = line[1:].strip()
+                if branch != "(no branch)":
+                    return branch
+        return None
+
     def get_tags(self):
         output = self.get_cmd_output(["tag", "-l"])
         return [t.strip() for t in output.split("\n")]
@@ -140,6 +158,19 @@ class GitClient(VCSClientBase):
     def get_current_revision(self):
         return self.get_cmd_output(["rev-parse", "--short", "HEAD"])
 
+    def get_latest_revision(self):
+        if not self.can_be_updated:
+            return self.get_current_revision()
+        branch = self.get_current_branch()
+        if not branch:
+            return self.get_current_revision()
+        result = self.get_cmd_output(["ls-remote"])
+        for line in result.split("\n"):
+            ref_pos = line.strip().find("refs/heads/" + branch)
+            if ref_pos > 0:
+                return line[:ref_pos].strip()[:7]
+        return None
+
 
 class HgClient(VCSClientBase):
 
@@ -159,6 +190,11 @@ class HgClient(VCSClientBase):
     def get_current_revision(self):
         return self.get_cmd_output(["identify", "--id"])
 
+    def get_latest_revision(self):
+        if not self.can_be_updated:
+            return self.get_latest_revision()
+        return self.get_cmd_output(["identify", "--id", self.remote_url])
+
 
 class SvnClient(VCSClientBase):
 
@@ -177,9 +213,8 @@ class SvnClient(VCSClientBase):
         return self.run_cmd(args)
 
     def get_current_revision(self):
-        output = self.get_cmd_output([
-            "info", "--non-interactive", "--trust-server-cert", "-r", "HEAD"
-        ])
+        output = self.get_cmd_output(
+            ["info", "--non-interactive", "--trust-server-cert", "-r", "HEAD"])
         for line in output.split("\n"):
             line = line.strip()
             if line.startswith("Revision:"):

scripts/docspregen.py

@@ -93,16 +93,15 @@ Packages
     :header-rows:  1
 
     * - Name
-      - Contents""")
+      - Description""")
     for name in sorted(packagenames):
         assert name in API_PACKAGES, name
-        contitems = [
-            "`{name} <{url}>`_".format(**item) for item in API_PACKAGES[name]
-        ]
         lines.append("""
-    * - ``{name}``
-      - {contents}""".format(
-            name=name, contents=", ".join(contitems)))
+    * - `{name} <{url}>`__
+      - {description}""".format(
+            name=name,
+            url=API_PACKAGES[name]['url'],
+            description=API_PACKAGES[name]['description']))
 
     if is_embedded:
         lines.append("""
@@ -172,8 +171,8 @@ For more detailed information please visit `vendor site <%s>`_.""" %
     #
     # Packages
     #
-    _packages_content = generate_packages(name, p.packages.keys(),
-                                          p.is_embedded())
+    _packages_content = generate_packages(name,
+                                          p.packages.keys(), p.is_embedded())
     if _packages_content:
         lines.append(_packages_content)
 
@@ -288,10 +287,11 @@ Platforms
             continue
         _found_platform = True
         p = PlatformFactory.newPlatform(manifest['name'])
-        lines.append("""
+        lines.append(
+            """
     * - :ref:`platform_{type_}`
-      - {description}""".format(
-            type_=manifest['name'], description=p.description))
+      - {description}"""
+            .format(type_=manifest['name'], description=p.description))
     if not _found_platform:
         del lines[-1]
 
@@ -347,19 +347,21 @@ Packages
     :header-rows:  1
 
     * - Name
-      - Contents""")
+      - Description""")
     for name, items in sorted(API_PACKAGES.iteritems()):
-        contitems = ["`{name} <{url}>`_".format(**item) for item in items]
         lines.append("""
-    * - ``{name}``
-      - {contents}""".format(
-            name=name, contents=", ".join(contitems)))
+    * - `{name} <{url}>`__
+      - {description}""".format(
+            name=name,
+            url=API_PACKAGES[name]['url'],
+            description=API_PACKAGES[name]['description']))
 
     with open(
             join(util.get_source_dir(), "..", "docs", "platforms",
                  "creating_platform.rst"), "r+") as fp:
         content = fp.read()
-        fp.seek(0, 0)
+        fp.seek(0)
+        fp.truncate()
         fp.write(content[:content.index(".. _platform_creating_packages:")] +
                  "\n".join(lines) + "\n\n" + content[content.index(
                      ".. _platform_creating_manifest_file:"):])

scripts/get-platformio.py

@@ -114,11 +114,7 @@ def install_platformio():
     r = None
     cmd = ["-m", "pip.__main__" if sys.version_info < (2, 7, 0) else "pip"]
     try:
-        # r = exec_python_cmd(cmd + ["install", "-U", "platformio"])
-        r = exec_python_cmd(cmd + [
-            "install", "-U",
-            "https://github.com/platformio/platformio-core/archive/develop.zip"
-        ])
+        r = exec_python_cmd(cmd + ["install", "-U", "platformio"])
         assert r['returncode'] == 0
     except AssertionError:
         r = exec_python_cmd(cmd + ["--no-cache-dir", "install", "-U",

setup.py

@@ -18,13 +18,14 @@ from platformio import (__author__, __description__, __email__, __license__,
                         __title__, __url__, __version__)
 
 install_requires = [
+    "arrow<1",
     "bottle<0.13",
     "click>=5,<6",
-    "lockfile>=0.9.1,<0.13",
-    "requests>=2.4.0,<3",
-    "semantic_version>=2.5.0",
     "colorama",
-    "pyserial>=3,<4"
+    "lockfile>=0.9.1,<0.13",
+    "pyserial>=3,<4",
+    "requests>=2.4.0,<3",
+    "semantic_version>=2.5.0"
 ]
 
 setup(
@@ -69,6 +70,6 @@ setup(
         "iot", "ide", "build", "compile", "library manager",
         "embedded", "ci", "continuous integration", "arduino", "mbed",
         "esp8266", "framework", "ide", "ide integration", "library.json",
-        "make", "cmake", "makefile", "mk", "pic32", "fpga"
+        "make", "cmake", "makefile", "mk", "pic32", "fpga", "artik"
     ]
 )

tests/commands/test_lib.py

@@ -16,7 +16,7 @@ import json
 import re
 from os.path import basename
 
-from platformio import util
+from platformio import exception, util
 from platformio.commands.init import cli as cmd_init
 from platformio.commands.lib import cli as cmd_lib
 
@@ -37,15 +37,35 @@ def test_search(clirunner, validate_cliresult):
 def test_global_install_registry(clirunner, validate_cliresult,
                                  isolated_pio_home):
     result = clirunner.invoke(cmd_lib, [
-        "-g", "install", "58", "OneWire",
+        "-g", "install", "58", "547@2.2.4", "DallasTemperature",
         "http://dl.platformio.org/libraries/archives/3/5174.tar.gz",
-        "ArduinoJson@5.6.7", "ArduinoJson@>5.6"
+        "ArduinoJson@5.6.7", "ArduinoJson@~5.7.0", "1089@fee16e880b"
     ])
     validate_cliresult(result)
+
+    # check lib with duplicate URL
+    result = clirunner.invoke(cmd_lib, [
+        "-g", "install",
+        "http://dl.platformio.org/libraries/archives/3/5174.tar.gz"
+    ])
+    validate_cliresult(result)
+    assert "is already installed" in result.output
+
+    # check lib with duplicate ID
+    result = clirunner.invoke(cmd_lib, ["-g", "install", "305"])
+    validate_cliresult(result)
+    assert "is already installed" in result.output
+
+    # install unknown library
+    result = clirunner.invoke(cmd_lib, ["-g", "install", "Unknown"])
+    assert result.exit_code != 0
+    assert isinstance(result.exception, exception.LibNotFound)
+
     items1 = [d.basename for d in isolated_pio_home.join("lib").listdir()]
     items2 = [
-        "DHT22_ID58", "ArduinoJson_ID64", "ArduinoJson_ID64@5.6.7",
-        "OneWire_ID1", "ESPAsyncTCP_ID305"
+        "ArduinoJson_ID64", "ArduinoJson_ID64@5.6.7", "DallasTemperature_ID54",
+        "DHT22_ID58", "ESPAsyncTCP_ID305", "NeoPixelBus_ID547", "OneWire_ID1",
+        "IRremoteESP8266_ID1089"
     ]
     assert set(items1) == set(items2)
 
@@ -55,11 +75,29 @@ def test_global_install_archive(clirunner, validate_cliresult,
     result = clirunner.invoke(cmd_lib, [
         "-g", "install", "https://github.com/adafruit/Adafruit-ST7735-Library/"
         "archive/master.zip",
+        "http://www.airspayce.com/mikem/arduino/RadioHead/RadioHead-1.62.zip",
+        "https://github.com/bblanchon/ArduinoJson/archive/v5.8.2.zip",
+        "https://github.com/bblanchon/ArduinoJson/archive/v5.8.2.zip@5.8.2"
+    ])
+    validate_cliresult(result)
+
+    # incorrect requirements
+    result = clirunner.invoke(cmd_lib, [
+        "-g", "install",
+        "https://github.com/bblanchon/ArduinoJson/archive/v5.8.2.zip@1.2.3"
+    ])
+    assert result.exit_code != 0
+
+    # check lib with duplicate URL
+    result = clirunner.invoke(cmd_lib, [
+        "-g", "install",
         "http://www.airspayce.com/mikem/arduino/RadioHead/RadioHead-1.62.zip"
     ])
     validate_cliresult(result)
+    assert "is already installed" in result.output
+
     items1 = [d.basename for d in isolated_pio_home.join("lib").listdir()]
-    items2 = ["Adafruit ST7735 Library", "RadioHead"]
+    items2 = ["Adafruit ST7735 Library", "RadioHead-1.62"]
     assert set(items1) >= set(items2)
 
 
@@ -71,14 +109,20 @@ def test_global_install_repository(clirunner, validate_cliresult,
             "-g",
             "install",
             "https://github.com/gioblu/PJON.git#3.0",
+            "https://github.com/gioblu/PJON.git#6.2",
+            "https://github.com/bblanchon/ArduinoJson.git",
             "https://gitlab.com/ivankravets/rs485-nodeproto.git",
             # "https://developer.mbed.org/users/simon/code/TextLCD/",
             "knolleary/pubsubclient"
         ])
     validate_cliresult(result)
     items1 = [d.basename for d in isolated_pio_home.join("lib").listdir()]
-    items2 = ["PJON", "ESPAsyncTCP", "PubSubClient"]
-    assert set(items2) & set(items1)
+    items2 = [
+        "PJON", "PJON@src-79de467ebe19de18287becff0a1fb42d",
+        "ArduinoJson@src-69ebddd821f771debe7ee734d3c7fa81", "rs485-nodeproto",
+        "PubSubClient"
+    ]
+    assert set(items1) >= set(items2)
 
 
 def test_global_lib_list(clirunner, validate_cliresult, isolated_pio_home):
@@ -89,81 +133,119 @@ def test_global_lib_list(clirunner, validate_cliresult, isolated_pio_home):
     result = clirunner.invoke(cmd_lib, ["-g", "list", "--json-output"])
     assert all([
         n in result.output
-        for n in ("PJON", "git+https://github.com/knolleary/pubsubclient")
+        for n in ("PJON", "git+https://github.com/knolleary/pubsubclient",
+                  "https://github.com/bblanchon/ArduinoJson/archive/v5.8.2.zip"
+                  )
     ])
     items1 = [i['name'] for i in json.loads(result.output)]
     items2 = [
         "OneWire", "DHT22", "PJON", "ESPAsyncTCP", "ArduinoJson",
-        "pubsubclient", "rs485-nodeproto", "Adafruit ST7735 Library",
-        "RadioHead"
+        "PubSubClient", "rs485-nodeproto", "Adafruit ST7735 Library",
+        "RadioHead-1.62", "DallasTemperature", "NeoPixelBus", "IRremoteESP8266"
     ]
     assert set(items1) == set(items2)
 
 
-def test_global_lib_show(clirunner, validate_cliresult, isolated_pio_home):
-    result = clirunner.invoke(cmd_lib, ["-g", "show", "64@5.6.7"])
+def test_global_lib_update_check(clirunner, validate_cliresult,
+                                 isolated_pio_home):
+    result = clirunner.invoke(
+        cmd_lib, ["-g", "update", "--only-check", "--json-output"])
     validate_cliresult(result)
-    assert all([
-        s in result.output for s in ("Json", "arduino", "atmelavr", "5.6.7")
-    ])
-
-    result = clirunner.invoke(cmd_lib, ["-g", "show", "ArduinoJson@>5.6.7"])
-    validate_cliresult(result)
-    assert all(
-        [s in result.output for s in ("ArduinoJson", "arduino", "atmelavr")])
-    assert "5.6.7" not in result.output
-
-    result = clirunner.invoke(cmd_lib, ["-g", "show", "1"])
-    validate_cliresult(result)
-    assert "OneWire" in result.output
+    output = json.loads(result.output)
+    assert set(["ArduinoJson", "IRremoteESP8266", "NeoPixelBus"]) == set(
+        [l['name'] for l in output])
 
 
 def test_global_lib_update(clirunner, validate_cliresult, isolated_pio_home):
+    # update library using package directory
+    result = clirunner.invoke(
+        cmd_lib,
+        ["-g", "update", "NeoPixelBus", "--only-check", "--json-output"])
+    validate_cliresult(result)
+    oudated = json.loads(result.output)
+    assert len(oudated) == 1
+    assert "__pkg_dir" in oudated[0]
+    result = clirunner.invoke(cmd_lib,
+                              ["-g", "update", oudated[0]['__pkg_dir']])
+    validate_cliresult(result)
+    assert "Uninstalling NeoPixelBus @ 2.2.4" in result.output
+
+    # update rest libraries
     result = clirunner.invoke(cmd_lib, ["-g", "update"])
     validate_cliresult(result)
-    assert all([s in result.output for s in ("[Up-to-date]", "[VCS]")])
+    validate_cliresult(result)
+    assert result.output.count("[Skip]") == 5
+    assert result.output.count("[Up-to-date]") == 9
+    assert "Uninstalling ArduinoJson @ 5.7.3" in result.output
+    assert "Uninstalling IRremoteESP8266 @ fee16e880b" in result.output
+
+    # update unknown library
+    result = clirunner.invoke(cmd_lib, ["-g", "update", "Unknown"])
+    assert result.exit_code != 0
+    assert isinstance(result.exception, exception.UnknownPackage)
 
 
 def test_global_lib_uninstall(clirunner, validate_cliresult,
                               isolated_pio_home):
+    # uninstall using package directory
+    result = clirunner.invoke(cmd_lib, ["-g", "list", "--json-output"])
+    validate_cliresult(result)
+    items = json.loads(result.output)
+    result = clirunner.invoke(cmd_lib,
+                              ["-g", "uninstall", items[0]['__pkg_dir']])
+    validate_cliresult(result)
+    assert "Uninstalling Adafruit ST7735 Library" in result.output
+
+    # uninstall the rest libraries
     result = clirunner.invoke(cmd_lib, [
-        "-g", "uninstall", "1", "ArduinoJson@!=5.6.7", "TextLCD",
-        "Adafruit ST7735 Library"
+        "-g", "uninstall", "1", "ArduinoJson@!=5.6.7",
+        "https://github.com/bblanchon/ArduinoJson.git", "IRremoteESP8266@>=0.2"
     ])
     validate_cliresult(result)
+
     items1 = [d.basename for d in isolated_pio_home.join("lib").listdir()]
     items2 = [
-        "DHT22_ID58", "ArduinoJson_ID64@5.6.7", "ESPAsyncTCP_ID305",
-        "pubsubclient", "PJON", "rs485-nodeproto", "RadioHead_ID124"
+        "ArduinoJson", "ArduinoJson_ID64@5.6.7", "DallasTemperature_ID54",
+        "DHT22_ID58", "ESPAsyncTCP_ID305", "NeoPixelBus_ID547", "PJON",
+        "PJON@src-79de467ebe19de18287becff0a1fb42d", "PubSubClient",
+        "RadioHead-1.62", "rs485-nodeproto"
     ]
     assert set(items1) == set(items2)
 
+    # uninstall unknown library
+    result = clirunner.invoke(cmd_lib, ["-g", "uninstall", "Unknown"])
+    assert result.exit_code != 0
+    assert isinstance(result.exception, exception.UnknownPackage)
 
-def test_project_lib_complex(clirunner, validate_cliresult, tmpdir):
-    with tmpdir.as_cwd():
-        # init
-        result = clirunner.invoke(cmd_init)
-        validate_cliresult(result)
 
-        # isntall
-        result = clirunner.invoke(cmd_lib, ["install", "54", "ArduinoJson"])
-        validate_cliresult(result)
-        items1 = [
-            d.basename
-            for d in tmpdir.join(basename(util.get_projectlibdeps_dir()))
-            .listdir()
-        ]
-        items2 = ["DallasTemperature_ID54", "OneWire_ID1", "ArduinoJson_ID64"]
-        assert set(items1) == set(items2)
+def test_lib_show(clirunner, validate_cliresult, isolated_pio_home):
+    result = clirunner.invoke(cmd_lib, ["show", "64"])
+    validate_cliresult(result)
+    assert all(
+        [s in result.output for s in ("ArduinoJson", "Arduino", "Atmel AVR")])
+    result = clirunner.invoke(cmd_lib, ["show", "OneWire"])
+    validate_cliresult(result)
+    assert "OneWire" in result.output
 
-        # list
-        result = clirunner.invoke(cmd_lib, ["list", "--json-output"])
-        validate_cliresult(result)
-        items1 = [i['name'] for i in json.loads(result.output)]
-        items2 = ["DallasTemperature", "OneWire", "ArduinoJson"]
-        assert set(items1) == set(items2)
 
-        # update
-        result = clirunner.invoke(cmd_lib, ["update"])
-        validate_cliresult(result)
-        assert "[Up-to-date]" in result.output
+def test_lib_builtin(clirunner, validate_cliresult, isolated_pio_home):
+    result = clirunner.invoke(cmd_lib, ["builtin"])
+    validate_cliresult(result)
+    result = clirunner.invoke(cmd_lib, ["builtin", "--json-output"])
+    validate_cliresult(result)
+
+
+def test_lib_stats(clirunner, validate_cliresult, isolated_pio_home):
+    result = clirunner.invoke(cmd_lib, ["stats"])
+    validate_cliresult(result)
+    assert all([
+        s in result.output
+        for s in ("UPDATED", "ago", "http://platformio.org/lib/show")
+    ])
+
+    result = clirunner.invoke(cmd_lib, ["stats", "--json-output"])
+    validate_cliresult(result)
+    assert set([
+        "dlweek", "added", "updated", "topkeywords", "dlmonth", "dlday",
+        "lastkeywords"
+    ]) == set(json.loads(result.output).keys())

tests/commands/test_platform.py

@@ -13,30 +13,12 @@
 # limitations under the License.
 
 import json
-import os
-from os.path import join
 
-from platformio import exception, util
+from platformio import exception
 from platformio.commands import platform as cli_platform
 
 
-def test_list_json_output(clirunner, validate_cliresult):
-    result = clirunner.invoke(cli_platform.platform_list, ["--json-output"])
-    validate_cliresult(result)
-    list_result = json.loads(result.output)
-    assert isinstance(list_result, list)
-    assert len(list_result)
-    platforms = [item['name'] for item in list_result]
-    assert "titiva" in platforms
-
-
-def test_list_raw_output(clirunner, validate_cliresult):
-    result = clirunner.invoke(cli_platform.platform_list)
-    validate_cliresult(result)
-    assert "teensy" in result.output
-
-
-def test_search_json_output(clirunner, validate_cliresult):
+def test_search_json_output(clirunner, validate_cliresult, isolated_pio_home):
     result = clirunner.invoke(cli_platform.platform_search,
                               ["arduino", "--json-output"])
     validate_cliresult(result)
@@ -47,73 +29,86 @@ def test_search_json_output(clirunner, validate_cliresult):
     assert "atmelsam" in platforms
 
 
-def test_search_raw_output(clirunner, validate_cliresult):
+def test_search_raw_output(clirunner, validate_cliresult, isolated_pio_home):
     result = clirunner.invoke(cli_platform.platform_search, ["arduino"])
     validate_cliresult(result)
     assert "teensy" in result.output
 
 
-def test_install_uknown_from_registry(clirunner, validate_cliresult):
-    result = clirunner.invoke(cli_platform.platform_install,
-                              ["uknown-platform"])
-    assert result.exit_code == -1
-    assert isinstance(result.exception, exception.UnknownPackage)
-
-
-def test_install_uknown_version(clirunner, validate_cliresult):
+def test_install_unknown_version(clirunner, validate_cliresult,
+                                 isolated_pio_home):
     result = clirunner.invoke(cli_platform.platform_install,
                               ["atmelavr@99.99.99"])
     assert result.exit_code == -1
     assert isinstance(result.exception, exception.UndefinedPackageVersion)
 
 
-def test_complex(clirunner, validate_cliresult):
-    with clirunner.isolated_filesystem():
-        os.environ["PLATFORMIO_HOME_DIR"] = os.getcwd()
-        try:
-            result = clirunner.invoke(
-                cli_platform.platform_install,
-                ["teensy", "--with-package", "framework-arduinoteensy"])
-            validate_cliresult(result)
-            assert all([
-                s in result.output
-                for s in ("teensy", "Downloading", "Unpacking")
-            ])
+def test_install_unknown_from_registry(clirunner, validate_cliresult,
+                                       isolated_pio_home):
+    result = clirunner.invoke(cli_platform.platform_install,
+                              ["unknown-platform"])
+    assert result.exit_code == -1
+    assert isinstance(result.exception, exception.UnknownPackage)
 
-            # show platform information
-            result = clirunner.invoke(cli_platform.platform_show, ["teensy"])
-            validate_cliresult(result)
-            assert "teensy" in result.output
 
-            # list platforms
-            result = clirunner.invoke(cli_platform.platform_list,
-                                      ["--json-output"])
-            validate_cliresult(result)
-            list_result = json.loads(result.output)
-            assert isinstance(list_result, list)
-            assert len(list_result) == 1
-            assert list_result[0]["name"] == "teensy"
-            assert list_result[0]["packages"] == ["framework-arduinoteensy"]
+def test_install_known_version(clirunner, validate_cliresult,
+                               isolated_pio_home):
+    result = clirunner.invoke(cli_platform.platform_install, [
+        "atmelavr@1.1.0", "--skip-default-package", "--with-package",
+        "tool-avrdude"
+    ])
+    validate_cliresult(result)
+    assert "atmelavr @ 1.1.0" in result.output
+    assert "Installing tool-avrdude @" in result.output
+    assert len(isolated_pio_home.join("packages").listdir()) == 1
 
-            # try to install again
-            result = clirunner.invoke(cli_platform.platform_install,
-                                      ["teensy"])
-            validate_cliresult(result)
-            assert "is already installed" in result.output
 
-            # try to update
-            for _ in range(2):
-                result = clirunner.invoke(cli_platform.platform_update)
-                validate_cliresult(result)
-                assert "teensy" in result.output
-                assert "Up-to-date" in result.output
-                assert "Out-of-date" not in result.output
+def test_install_from_vcs(clirunner, validate_cliresult, isolated_pio_home):
+    result = clirunner.invoke(cli_platform.platform_install, [
+        "https://github.com/platformio/"
+        "platform-espressif8266.git#feature/stage", "--skip-default-package"
+    ])
+    validate_cliresult(result)
+    assert "espressif8266_stage" in result.output
 
-            # try to uninstall
-            result = clirunner.invoke(cli_platform.platform_uninstall,
-                                      ["teensy"])
-            validate_cliresult(result)
-            for folder in ("platforms", "packages"):
-                assert len(os.listdir(join(util.get_home_dir(), folder))) == 0
-        finally:
-            del os.environ["PLATFORMIO_HOME_DIR"]
+
+def test_list_json_output(clirunner, validate_cliresult, isolated_pio_home):
+    result = clirunner.invoke(cli_platform.platform_list, ["--json-output"])
+    validate_cliresult(result)
+    list_result = json.loads(result.output)
+    assert isinstance(list_result, list)
+    assert len(list_result)
+    platforms = [item['name'] for item in list_result]
+    assert set(["atmelavr", "espressif8266_stage"]) == set(platforms)
+
+
+def test_list_raw_output(clirunner, validate_cliresult, isolated_pio_home):
+    result = clirunner.invoke(cli_platform.platform_list)
+    validate_cliresult(result)
+    assert all(
+        [s in result.output for s in ("atmelavr", "espressif8266_stage")])
+
+
+def test_update_check(clirunner, validate_cliresult, isolated_pio_home):
+    result = clirunner.invoke(cli_platform.platform_update,
+                              ["--only-check", "--json-output"])
+    validate_cliresult(result)
+    output = json.loads(result.output)
+    assert len(output) == 1
+    assert output[0]['name'] == "atmelavr"
+    assert len(isolated_pio_home.join("packages").listdir()) == 1
+
+
+def test_update_raw(clirunner, validate_cliresult, isolated_pio_home):
+    result = clirunner.invoke(cli_platform.platform_update)
+    validate_cliresult(result)
+    assert "Uninstalling atmelavr @ 1.1.0:" in result.output
+    assert "PlatformManager: Installing atmelavr @" in result.output
+    assert len(isolated_pio_home.join("packages").listdir()) == 1
+
+
+def test_uninstall(clirunner, validate_cliresult, isolated_pio_home):
+    result = clirunner.invoke(cli_platform.platform_uninstall,
+                              ["atmelavr", "espressif8266_stage"])
+    validate_cliresult(result)
+    assert len(isolated_pio_home.join("platforms").listdir()) == 0

tests/ino2cpp/basic/basic.ino

@@ -26,12 +26,15 @@ Foo foo(&fooCallback);
 
 //
 
+template<class T> T Add(T n1, T n2) {
+    return n1 + n2;
+}
+
 void setup() {
     struct Item item1;
     myFunction(&item1);
 }
 
-
 void loop() {
 
 }
@@ -40,7 +43,7 @@ void myFunction(struct Item *item) {
 
 }
 
-#warning "Line number is 43"
+#warning "Line number is 46"
 
 void fooCallback(){
 

tests/test_ino2cpp.py

@@ -42,7 +42,7 @@ def test_warning_line(clirunner, validate_cliresult):
     validate_cliresult(result)
     assert ('basic.ino:16:14: warning: #warning "Line number is 16"' in
             result.output)
-    assert ('basic.ino:43:2: warning: #warning "Line number is 43"' in
+    assert ('basic.ino:46:2: warning: #warning "Line number is 46"' in
             result.output)
     result = clirunner.invoke(
         cmd_ci, [join(INOTEST_DIR, "strmultilines"), "-b", "uno"])

tests/test_maintenance.py

@@ -13,6 +13,7 @@
 # limitations under the License.
 
 import json
+import re
 from time import time
 
 from platformio import app, maintenance
@@ -135,7 +136,8 @@ def test_check_and_update_libraries(clirunner, validate_cliresult,
     assert ("There are the new updates for libraries (ArduinoJson)" in
             result.output)
     assert "Please wait while updating libraries" in result.output
-    assert "[Out-of-date]" in result.output
+    assert re.search(r"Updating ArduinoJson\s+@ 5.6.7\s+\[[\d\.]+\]",
+                     result.output)
 
     # check updated version
     result = clirunner.invoke(cli_pio, ["lib", "-g", "list", "--json-output"])
@@ -154,7 +156,7 @@ def test_check_platform_updates(clirunner, validate_cliresult,
     manifest['version'] = "0.0.0"
     manifest_path.write(json.dumps(manifest))
     # reset cached manifests
-    PlatformManager().reset_cache()
+    PlatformManager().cache_reset()
 
     # reset check time
     interval = int(app.get_setting("check_platforms_interval")) * 3600 * 24
@@ -188,7 +190,8 @@ def test_check_and_update_platforms(clirunner, validate_cliresult,
     validate_cliresult(result)
     assert "There are the new updates for platforms (native)" in result.output
     assert "Please wait while updating platforms" in result.output
-    assert "[Out-of-date]" in result.output
+    assert re.search(r"Updating native\s+@ 0.0.0\s+\[[\d\.]+\]",
+                     result.output)
 
     # check updated version
     result = clirunner.invoke(cli_pio, ["platform", "list", "--json-output"])

tests/test_managers.py

@@ -12,70 +12,118 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+import json
+from os.path import join
+
 from platformio import util
-from platformio.managers.package import BasePkgManager
+from platformio.managers.package import PackageManager
 
 
-def test_pkg_name_parser():
+def test_pkg_input_parser():
     items = [
         ["PkgName", ("PkgName", None, None)],
         [("PkgName", "!=1.2.3,<2.0"), ("PkgName", "!=1.2.3,<2.0", None)],
         ["PkgName@1.2.3", ("PkgName", "1.2.3", None)],
         [("PkgName@1.2.3", "1.2.5"), ("PkgName@1.2.3", "1.2.5", None)],
         ["id:13", ("id:13", None, None)],
-        ["id:13@~1.2.3", ("id:13", "~1.2.3", None)], [
+        ["id:13@~1.2.3", ("id:13", "~1.2.3", None)],
+        [
             util.get_home_dir(),
             (".platformio", None, "file://" + util.get_home_dir())
-        ], [
+        ],
+        [
             "LocalName=" + util.get_home_dir(),
             ("LocalName", None, "file://" + util.get_home_dir())
-        ], [
+        ],
+        [
+            "LocalName=%s@>2.3.0" % util.get_home_dir(),
+            ("LocalName", ">2.3.0", "file://" + util.get_home_dir())
+        ],
+        [
             "https://github.com/user/package.git",
             ("package", None, "git+https://github.com/user/package.git")
-        ], [
-            "https://gitlab.com/user/package.git",
-            ("package", None, "git+https://gitlab.com/user/package.git")
-        ], [
+        ],
+        [
+            "MyPackage=https://gitlab.com/user/package.git",
+            ("MyPackage", None, "git+https://gitlab.com/user/package.git")
+        ],
+        [
+            "MyPackage=https://gitlab.com/user/package.git@3.2.1,!=2",
+            ("MyPackage", "3.2.1,!=2",
+             "git+https://gitlab.com/user/package.git")
+        ],
+        [
+            "https://somedomain.com/path/LibraryName-1.2.3.zip",
+            ("LibraryName-1.2.3", None,
+             "https://somedomain.com/path/LibraryName-1.2.3.zip")
+        ],
+        [
             "https://github.com/user/package/archive/branch.zip",
             ("branch", None,
              "https://github.com/user/package/archive/branch.zip")
-        ], [
+        ],
+        [
+            "https://github.com/user/package/archive/branch.zip@~1.2.3",
+            ("branch", "~1.2.3",
+             "https://github.com/user/package/archive/branch.zip")
+        ],
+        [
             "https://github.com/user/package/archive/branch.tar.gz",
-            ("branch", None,
+            ("branch.tar", None,
              "https://github.com/user/package/archive/branch.tar.gz")
-        ], [
+        ],
+        [
+            "https://github.com/user/package/archive/branch.tar.gz@!=5",
+            ("branch.tar", "!=5",
+             "https://github.com/user/package/archive/branch.tar.gz")
+        ],
+        [
             "https://developer.mbed.org/users/user/code/package/",
             ("package", None,
              "hg+https://developer.mbed.org/users/user/code/package/")
-        ], [
+        ],
+        [
             "https://github.com/user/package#v1.2.3",
             ("package", None, "git+https://github.com/user/package#v1.2.3")
-        ], [
+        ],
+        [
             "https://github.com/user/package.git#branch",
             ("package", None, "git+https://github.com/user/package.git#branch")
-        ], [
+        ],
+        [
             "PkgName=https://github.com/user/package.git#a13d344fg56",
             ("PkgName", None,
              "git+https://github.com/user/package.git#a13d344fg56")
-        ], [
+        ],
+        [
+            "user/package",
+            ("package", None, "git+https://github.com/user/package")
+        ],
+        [
             "PkgName=user/package",
             ("PkgName", None, "git+https://github.com/user/package")
-        ], [
+        ],
+        [
             "PkgName=user/package#master",
             ("PkgName", None, "git+https://github.com/user/package#master")
-        ], [
+        ],
+        [
             "git+https://github.com/user/package",
             ("package", None, "git+https://github.com/user/package")
-        ], [
+        ],
+        [
             "hg+https://example.com/user/package",
             ("package", None, "hg+https://example.com/user/package")
-        ], [
+        ],
+        [
             "git@github.com:user/package.git",
             ("package", None, "git@github.com:user/package.git")
-        ], [
+        ],
+        [
             "git@github.com:user/package.git#v1.2.0",
             ("package", None, "git@github.com:user/package.git#v1.2.0")
-        ], [
+        ],
+        [
             "git+ssh://git@gitlab.private-server.com/user/package#1.2.0",
             ("package", None,
              "git+ssh://git@gitlab.private-server.com/user/package#1.2.0")
@@ -83,6 +131,72 @@ def test_pkg_name_parser():
     ]
     for params, result in items:
         if isinstance(params, tuple):
-            assert BasePkgManager.parse_pkg_name(*params) == result
+            assert PackageManager.parse_pkg_input(*params) == result
         else:
-            assert BasePkgManager.parse_pkg_name(params) == result
+            assert PackageManager.parse_pkg_input(params) == result
+
+
+def test_install_packages(isolated_pio_home, tmpdir):
+    packages = [
+        dict(id=1, name="name_1", version="shasum"),
+        dict(id=1, name="name_1", version="2.0.0"),
+        dict(id=1, name="name_1", version="2.1.0"),
+        dict(id=1, name="name_1", version="1.2.0"),
+        dict(id=1, name="name_1", version="1.0.0"),
+        dict(name="name_2", version="1.0.0"),
+        dict(name="name_2", version="2.0.0",
+             __src_url="git+https://github.com"),
+        dict(name="name_2", version="3.0.0",
+             __src_url="git+https://github2.com"),
+        dict(name="name_2", version="4.0.0",
+             __src_url="git+https://github2.com")
+    ]
+
+    pm = PackageManager(join(util.get_home_dir(), "packages"))
+    for package in packages:
+        tmp_dir = tmpdir.mkdir("tmp-package")
+        tmp_dir.join("package.json").write(json.dumps(package))
+        pm._install_from_url(package['name'], "file://%s" % str(tmp_dir))
+        tmp_dir.remove(rec=1)
+
+    assert len(pm.get_installed()) == len(packages) - 1
+
+    pkg_dirnames = [
+        'name_1_ID1', 'name_1_ID1@1.0.0', 'name_1_ID1@1.2.0',
+        'name_1_ID1@2.0.0', 'name_1_ID1@shasum', 'name_2',
+        'name_2@src-177cbce1f0705580d17790fda1cc2ef5',
+        'name_2@src-f863b537ab00f4c7b5011fc44b120e1f'
+    ]
+    assert set([p.basename for p in isolated_pio_home.join(
+        "packages").listdir()]) == set(pkg_dirnames)
+
+
+def test_get_package(isolated_pio_home):
+    tests = [
+        [("unknown", ), None],
+        [("1", ), None],
+        [("id=1", "shasum"), dict(id=1, name="name_1", version="shasum")],
+        [("id=1", "*"), dict(id=1, name="name_1", version="2.1.0")],
+        [("id=1", "^1"), dict(id=1, name="name_1", version="1.2.0")],
+        [("id=1", "^1"), dict(id=1, name="name_1", version="1.2.0")],
+        [("name_1", "<2"), dict(id=1, name="name_1", version="1.2.0")],
+        [("name_1", ">2"), None],
+        [("name_1", "2-0-0"), dict(id=1, name="name_1", version="2.1.0")],
+        [("name_1", "2-0-0"), dict(id=1, name="name_1", version="2.1.0")],
+        [("name_2", ), dict(name="name_2", version="4.0.0")],
+        [("url_has_higher_priority", None, "git+https://github.com"),
+         dict(name="name_2", version="2.0.0",
+              __src_url="git+https://github.com")],
+        [("name_2", None, "git+https://github.com"),
+         dict(name="name_2", version="2.0.0",
+              __src_url="git+https://github.com")],
+    ]
+
+    pm = PackageManager(join(util.get_home_dir(), "packages"))
+    for test in tests:
+        manifest = pm.get_package(*test[0])
+        if test[1] is None:
+            assert manifest is None, test
+            continue
+        for key, value in test[1].items():
+            assert manifest[key] == value, test

tests/test_pkgmanifest.py

@@ -16,19 +16,6 @@ import pytest
 import requests
 
 
-def pytest_generate_tests(metafunc):
-    if "package_data" not in metafunc.fixturenames:
-        return
-    pkgs_manifest = requests.get(
-        "https://dl.bintray.com/platformio/dl-packages/manifest.json").json()
-    assert isinstance(pkgs_manifest, dict)
-    packages = []
-    for _, variants in pkgs_manifest.iteritems():
-        for item in variants:
-            packages.append(item)
-    metafunc.parametrize("package_data", packages)
-
-
 def validate_response(req):
     assert req.status_code == 200
     assert int(req.headers['Content-Length']) > 0
@@ -36,13 +23,22 @@ def validate_response(req):
                                            "application/octet-stream")
 
 
-def test_package(package_data):
-    assert package_data['url'].endswith(".tar.gz")
+def test_packages():
+    pkgs_manifest = requests.get(
+        "https://dl.bintray.com/platformio/dl-packages/manifest.json").json()
+    assert isinstance(pkgs_manifest, dict)
+    items = []
+    for _, variants in pkgs_manifest.iteritems():
+        for item in variants:
+            items.append(item)
 
-    r = requests.head(package_data['url'], allow_redirects=True)
-    validate_response(r)
+    for item in items:
+        assert item['url'].endswith(".tar.gz"), item
 
-    if "X-Checksum-Sha1" not in r.headers:
-        return pytest.skip("X-Checksum-Sha1 is not provided")
+        r = requests.head(item['url'], allow_redirects=True)
+        validate_response(r)
 
-    assert package_data['sha1'] == r.headers.get("X-Checksum-Sha1")
+        if "X-Checksum-Sha1" not in r.headers:
+            return pytest.skip("X-Checksum-Sha1 is not provided")
+
+        assert item['sha1'] == r.headers.get("X-Checksum-Sha1"), item

tox.ini

@@ -24,6 +24,7 @@ deps =
     yapf
     pylint
     pytest
+    show
 commands = python --version
 
 [testenv:docs]
@@ -61,6 +62,13 @@ commands =
     {envpython} --version
     py.test -v --basetemp="{envtmpdir}" tests
 
+[testenv:skipexamples]
+basepython = python2.7
+deps =
+    pytest
+commands =
+    py.test -v --basetemp="{envtmpdir}" tests --ignore tests/test_examples.py
+
 [testenv:coverage]
 basepython = python2.7
 passenv = *