Improve base package manager; VCS and package ID support

docs/userguide/platforms/cmd_install.rst

@@ -22,7 +22,7 @@ Usage
 .. code-block:: bash
 
     # install platform by name
-    platformio platform install [OPTIONS] PLATFORM
+    platformio platform install [OPTIONS] [PLATFORM...]
 
     # install specific platform version using Semantic Versioning
     platformio platform install [OPTIONS] PLATFORM@X.Y.Z
@@ -45,10 +45,16 @@ There are several predefined aliases for packages, such as:
 Local
 ~~~~~
 
-PlatformIO supports installing development platform from local directory. Here
+PlatformIO supports installing development platform from local directory or
-is supported form:
+archive. Need to use ``file://`` prefix before local path. Also, platform
+directory or archive should contain ``platform.json`` manifest.
 
 * ``file:///local/path/to/the/platform/dir``
+* ``file:///local/path/to/the/platform.zip``
+* ``file:///local/path/to/the/platform.tar.gz``
+
+Remote
+~~~~~~
 
 VCS
 ~~~
@@ -135,34 +141,38 @@ Examples
 .. code-block:: bash
 
     $ platformio platform install atmelavr
-    Installing platform atmelavr @ latest:
+    PlatformManager: Installing atmelavr
     Downloading...
     Unpacking  [####################################]  100%
-    Installing package tool-scons @ >=2.3.0,<2.6.0:
+    atmelavr @ 0.0.0 has been successfully installed!
+    PackageManager: Installing tool-scons @ >=2.3.0,<2.6.0
     Downloading  [####################################]  100%
     Unpacking  [####################################]  100%
-    Installing package toolchain-atmelavr @ ~1.40801.0:
+    tool-scons @ 2.4.1 has been successfully installed!
+    PackageManager: Installing toolchain-atmelavr @ ~1.40801.0
     Downloading  [####################################]  100%
     Unpacking  [####################################]  100%
+    toolchain-atmelavr @ 1.40801.0 has been successfully installed!
     The platform 'atmelavr' has been successfully installed!
     The rest of packages will be installed automatically depending on your build environment.
 
-
 2. Install :ref:`platform_atmelavr` with ``uploader`` utility only and skip
    default packages
 
 .. code-block:: bash
 
-    $ platformio platform install atmelavr --skip-default-package --with-package=uploader
+    PlatformManager: Installing atmelavr
-    Installing platform atmelavr @ latest:
     Downloading  [####################################]  100%
     Unpacking  [####################################]  100%
-    Installing package tool-micronucleus @ ~1.200.0:
+    atmelavr @ 0.0.0 has been successfully installed!
+    PackageManager: Installing tool-micronucleus @ ~1.200.0
     Downloading  [####################################]  100%
     Unpacking  [####################################]  100%
-    Installing package tool-avrdude @ >=1.60001.0,<1.60101.0:
+    tool-micronucleus @ 1.200.0 has been successfully installed!
+    PackageManager: Installing tool-avrdude @ ~1.60001.0
     Downloading  [####################################]  100%
     Unpacking  [####################################]  100%
+    tool-avrdude @ 1.60001.1 has been successfully installed!
     The platform 'atmelavr' has been successfully installed!
     The rest of packages will be installed automatically depending on your build environment.
 
@@ -171,27 +181,31 @@ Examples
 .. code-block:: bash
 
     $ platformio platform install https://github.com/platformio/platform-atmelavr.git
-    Installing platform https://github.com/platformio/platform-atmelavr.git @ latest:
+
+    PlatformManager: Installing platform-atmelavr
     git version 2.7.4 (Apple Git-66)
-    Cloning into '/Users/ikravets/.platformio/platforms/installing-XMIsAE-package'...
+    Cloning into '/Volumes/MEDIA/tmp/pio3_test_projects/arduino-digihead-master/home_dir/platforms/installing-U3ucN0-package'...
-    remote: Counting objects: 172, done.
+    remote: Counting objects: 176, done.
-    remote: Compressing objects: 100% (51/51), done.
+    remote: Compressing objects: 100% (55/55), done.
-    remote: Total 172 (delta 109), reused 168 (delta 109), pack-reused 0
+    remote: Total 176 (delta 114), reused 164 (delta 109), pack-reused 0
-    Receiving objects: 100% (172/172), 38.18 KiB | 0 bytes/s, done.
+    Receiving objects: 100% (176/176), 38.86 KiB | 0 bytes/s, done.
-    Resolving deltas: 100% (109/109), done.
+    Resolving deltas: 100% (114/114), done.
     Checking connectivity... done.
     Submodule 'examples/arduino-external-libs/lib/OneWire' (https://github.com/PaulStoffregen/OneWire.git) registered for path 'examples/arduino-external-libs/lib/OneWire'
     Cloning into 'examples/arduino-external-libs/lib/OneWire'...
-    remote: Counting objects: 87, done.
+    remote: Counting objects: 91, done.
-    remote: Total 87 (delta 0), reused 0 (delta 0), pack-reused 87
+    remote: Total 91 (delta 0), reused 0 (delta 0), pack-reused 91
-    Unpacking objects: 100% (87/87), done.
+    Unpacking objects: 100% (91/91), done.
     Checking connectivity... done.
     Submodule path 'examples/arduino-external-libs/lib/OneWire': checked out '57c18c6de80c13429275f70875c7c341f1719201'
-    Installing package tool-scons @ >=2.3.0,<2.6.0:
+    atmelavr @ 0.0.0 has been successfully installed!
+    PackageManager: Installing tool-scons @ >=2.3.0,<2.6.0
     Downloading  [####################################]  100%
     Unpacking  [####################################]  100%
-    Installing package toolchain-atmelavr @ ~1.40801.0:
+    tool-scons @ 2.4.1 has been successfully installed!
+    PackageManager: Installing toolchain-atmelavr @ ~1.40801.0
     Downloading  [####################################]  100%
     Unpacking  [####################################]  100%
+    toolchain-atmelavr @ 1.40801.0 has been successfully installed!
     The platform 'https://github.com/platformio/platform-atmelavr.git' has been successfully installed!
     The rest of packages will be installed automatically depending on your build environment.

docs/userguide/platforms/cmd_uninstall.rst

@@ -21,7 +21,7 @@ Usage
 
 .. code-block:: bash
 
-    platformio platform uninstall PLATFORM
+    platformio platform uninstall [PLATFORM...]
 
     # uninstall specific platform version using Semantic Versioning
     platformio platform uninstall PLATFORM@X.Y.Z
@@ -39,7 +39,7 @@ Examples
 .. code-block:: bash
 
     $ platformio platform uninstall atmelavr
-    Uninstalling platform atmelavr @ latest:    [OK]
+    Uninstalling platform atmelavr @ 0.0.0:    [OK]
-    Uninstalling package tool-scons @ 2.5.0:    [OK]
+    Uninstalling package tool-scons @ 2.4.1:    [OK]
     Uninstalling package toolchain-atmelavr @ 1.40801.0:    [OK]
     The platform 'atmelavr' has been successfully uninstalled!

docs/userguide/platforms/cmd_update.rst

@@ -21,7 +21,10 @@ Usage
 
 .. code-block:: bash
 
-    platformio platform update
+    platformio platform update [OPTIONS] [PLATFORM...]
+
+    # update specific platform version using Semantic Versioning
+    platformio platform update PLATFORM@X.Y.Z
 
 
 Description
@@ -35,64 +38,48 @@ Options
 .. program:: platformio platform update
 
 .. option::
-    --only-packages
+    -p, --only-packages
 
 Update only platform related packages. Do not update development platform
 build scripts, board configs and etc.
 
+.. option::
+    -c, --only-check
+
+Do not update, only check for new version
+
 Examples
 --------
 
 .. code-block:: bash
 
     $ platformio platform update
-    Platform atmelavr @ 0.0.0
+    Platform atmelavr
     --------
-    Updating platform atmelavr @ latest:
+    Updating atmelavr @ 0.0.0:  [Up-to-date]
-    Versions: Current=0.0.0, Latest=0.0.0    [Up-to-date]
+    Updating framework-arduinoavr @ 1.10608.1:  [Up-to-date]
-    Updating package framework-arduinoavr @ ~1.10608.0:
+    Updating tool-avrdude @ 1.60001.1:  [Up-to-date]
-    Versions: Current=1.10608.0, Latest=1.10608.0    [Up-to-date]
+    Updating toolchain-atmelavr @ 1.40801.0:    [Up-to-date]
-    Updating package toolchain-atmelavr @ ~1.40801.0:
+    Updating tool-scons @ 2.4.1:    [Up-to-date]
-    Versions: Current=1.40801.0, Latest=1.40801.0    [Up-to-date]
-    Updating package framework-simba @ ~1.500.0:
-    Versions: Current=1.500.0, Latest=1.500.0    [Up-to-date]
-    Updating package tool-scons @ >=2.3.0,<2.6.0:
-    Versions: Current=2.5.0, Latest=2.5.0    [Up-to-date]
 
-    Platform atmelsam @ 0.0.0
+    Platform espressif
     --------
-    Updating platform atmelsam @ latest:
+    Updating espressif @ 0.0.0:     [Up-to-date]
-    Versions: Current=0.0.0, Latest=0.0.0    [Up-to-date]
+    Updating tool-scons @ 2.4.1:    [Up-to-date]
-    Updating package toolchain-gccarmnoneeabi @ >=1.40803.0,<1.40805.0:
+    Updating toolchain-xtensa @ 1.40802.0:  [Up-to-date]
-    Versions: Current=1.40804.0, Latest=1.40804.0    [Up-to-date]
+    Updating tool-esptool @ 1.409.0:    [Up-to-date]
-    Updating package framework-arduinosam @ ~1.10607.0:
+    Updating tool-mkspiffs @ 1.102.0:   [Up-to-date]
-    Versions: Current=1.10607.0, Latest=1.10607.0    [Up-to-date]
+    Updating framework-arduinoespressif @ 1.20300.0:    [Up-to-date]
-    Updating package framework-simba @ ~1.500.0:
+    Updating sdk-esp8266 @ 1.10502.0:   [Up-to-date]
-    Versions: Current=1.500.0, Latest=1.500.0    [Up-to-date]
-    Updating package framework-mbed @ ~1.117.0:
-    Versions: Current=1.117.0, Latest=1.117.0    [Up-to-date]
-    Updating package tool-scons @ >=2.3.0,<2.6.0:
-    Versions: Current=2.5.0, Latest=2.5.0    [Up-to-date]
-    Updating package tool-bossac @ ~1.10500.0:
-    Versions: Current=1.10500.0, Latest=1.10500.0    [Up-to-date]
 
-    Platform espressif @ 0.0.0
+    Platform teensy
     --------
-    Updating platform espressif @ latest:
+    Updating teensy @ 0.0.0:    [Up-to-date]
-    Versions: Current=0.0.0, Latest=0.0.0    [Up-to-date]
+    Updating framework-arduinoteensy @ 1.128.0:     [Up-to-date]
-    Updating package tool-scons @ >=2.3.0,<2.6.0:
+    Updating tool-teensy @ 1.1.0:   [Up-to-date]
-    Versions: Current=2.5.0, Latest=2.5.0    [Up-to-date]
+    Updating framework-mbed @ 1.121.0:  [Up-to-date]
-    Updating package toolchain-xtensa @ ~1.40802.0:
+    Updating tool-scons @ 2.4.1:    [Up-to-date]
-    Versions: Current=1.40802.0, Latest=1.40802.0    [Up-to-date]
+    Updating toolchain-atmelavr @ 1.40801.0:    [Up-to-date]
-    Updating package framework-simba @ ~1.500.0:
+    Updating toolchain-gccarmnoneeabi @ 1.40804.0:  [Up-to-date]
-    Versions: Current=1.500.0, Latest=1.500.0    [Up-to-date]
-    Updating package tool-esptool @ ~1.408.0:
-    Versions: Current=1.408.0, Latest=1.408.0    [Up-to-date]
-    Updating package tool-mkspiffs @ ~1.102.0:
-    Versions: Current=1.102.0, Latest=1.102.0    [Up-to-date]
-    Updating package framework-arduinoespressif @ ~1.20200.0:
-    Versions: Current=1.20200.0, Latest=1.20200.0    [Up-to-date]
-    Updating package sdk-esp8266 @ ~1.10502.0:
-    Versions: Current=1.10502.0, Latest=1.10502.0    [Up-to-date]
 
     ...

platformio/commands/platform.py

@@ -13,7 +13,6 @@
 # limitations under the License.
 
 import json
-from os.path import basename
 
 import click
 
@@ -29,13 +28,14 @@ def cli():
 def _print_platforms(platforms):
     for platform in platforms:
         click.echo("{name} ~ {title}".format(
-            name=click.style(platform['name'], fg="cyan"),
+            name=click.style(
+                platform['name'], fg="cyan"),
             title=platform['title']))
         click.echo("=" * (3 + len(platform['name'] + platform['title'])))
         click.echo(platform['description'])
         click.echo()
-        click.echo("Home: %s" %
+        click.echo("Home: %s" % "http://platformio.org/platforms/" + platform[
-                   "http://platformio.org/platforms/" + platform['name'])
+            'name'])
         if platform['packages']:
             click.echo("Packages: %s" % ", ".join(platform['packages']))
         if "version" in platform:
@@ -71,22 +71,19 @@ def platform_search(query, json_output):
 
 
 @cli.command("install", short_help="Install new platforms")
-@click.argument("platforms", nargs=-1, required=True)
+@click.argument("platforms", nargs=-1, required=True, metavar="[PLATFORM...]")
 @click.option("--with-package", multiple=True)
 @click.option("--without-package", multiple=True)
 @click.option("--skip-default-package", is_flag=True)
 def platform_install(platforms, with_package, without_package,
                      skip_default_package):
+    pm = PlatformManager()
     for platform in platforms:
-        _platform = platform
+        if pm.install(
-        _version = None
+                name=platform,
-        if any([s in platform for s in ("\\", "/")]):
+                with_packages=with_package,
-            _platform = basename(platform)
+                without_packages=without_package,
-            _version = platform
+                skip_default_package=skip_default_package):
-        elif "@" in platform:
-            _platform, _version = platform.rsplit("@", 1)
-        if PlatformManager().install(_platform, _version, with_package,
-                                     without_package, skip_default_package):
             click.secho(
                 "The platform '%s' has been successfully installed!\n"
                 "The rest of packages will be installed automatically "
@@ -94,12 +91,49 @@ def platform_install(platforms, with_package, without_package,
                 fg="green")
 
 
+@cli.command("uninstall", short_help="Uninstall platforms")
+@click.argument("platforms", nargs=-1, required=True, metavar="[PLATFORM...]")
+def platform_uninstall(platforms):
+    pm = PlatformManager()
+    for platform in platforms:
+        if pm.uninstall(platform):
+            click.secho(
+                "The platform '%s' has been successfully "
+                "uninstalled!" % platform,
+                fg="green")
+
+
+@cli.command("update", short_help="Update installed Platforms")
+@click.argument("platforms", nargs=-1, required=False, metavar="[PLATFORM...]")
+@click.option(
+    "-p",
+    "--only-packages",
+    is_flag=True,
+    help="Update only platform packages")
+@click.option(
+    "-c",
+    "--only-check",
+    is_flag=True,
+    help="Do not update, only check for new version")
+def platform_update(platforms, only_packages, only_check):
+    pm = PlatformManager()
+    if not platforms:
+        platforms = set([m['name'] for m in pm.get_installed()])
+    for platform in platforms:
+        click.echo("Platform %s" % click.style(platform, fg="cyan"))
+        click.echo("--------")
+        pm.update(platform, only_packages=only_packages, only_check=only_check)
+        click.echo()
+
+
 @cli.command("list", short_help="List installed platforms")
 @click.option("--json-output", is_flag=True)
 def platform_list(json_output):
     platforms = []
-    for manifest in PlatformManager().get_installed():
+    pm = PlatformManager()
-        p = PlatformFactory.newPlatform(manifest['_manifest_path'])
+    for manifest in pm.get_installed():
+        p = PlatformFactory.newPlatform(
+            pm.get_manifest_path(manifest['__pkg_dir']))
         platforms.append({
             "name": p.name,
             "title": p.title,
@@ -129,7 +163,8 @@ def platform_show(ctx, platform):
             raise exception.PlatformNotInstalledYet(platform)
 
     click.echo("{name} ~ {title}".format(
-        name=click.style(p.name, fg="cyan"), title=p.title))
+        name=click.style(
+            p.name, fg="cyan"), title=p.title))
     click.echo("=" * (3 + len(p.name + p.title)))
     click.echo(p.description)
     click.echo()
@@ -152,35 +187,9 @@ def platform_show(ctx, platform):
         if p.get_package_type(name):
             click.echo("Type: %s" % p.get_package_type(name))
         click.echo("Requirements: %s" % opts.get("version"))
-        click.echo("Installed: %s" % (
+        click.echo("Installed: %s" % ("Yes" if name in installed_pkgs else
-            "Yes" if name in installed_pkgs else "No (optional)"))
+                                      "No (optional)"))
         if name in installed_pkgs:
             for key, value in installed_pkgs[name].items():
                 if key in ("url", "version", "description"):
                     click.echo("%s: %s" % (key.title(), value))
-
-
-@cli.command("uninstall", short_help="Uninstall platforms")
-@click.argument("platforms", nargs=-1, required=True)
-def platform_uninstall(platforms):
-    for platform in platforms:
-        _platform = platform
-        _version = None
-        if "@" in platform:
-            _platform, _version = platform.rsplit("@", 1)
-        if PlatformManager().uninstall(_platform, _version):
-            click.secho("The platform '%s' has been successfully "
-                        "uninstalled!" % platform, fg="green")
-
-
-@cli.command("update", short_help="Update installed Platforms")
-@click.option("--only-packages", is_flag=True)
-def platform_update(only_packages):
-    pm = PlatformManager()
-    for manifest in pm.get_installed():
-        click.echo("Platform %s @ %s" % (
-            click.style(manifest['name'], fg="cyan"), manifest['version']))
-        click.echo("--------")
-        pm.update(manifest['name'], manifest['version'],
-                  only_packages=only_packages)
-        click.echo()

platformio/commands/run.py

@@ -190,17 +190,13 @@ class EnvironmentProcessor(object):
         if "lib_install" in self.options:
             _autoinstall_libs(self.cmd_ctx, self.options['lib_install'])
 
-        platform = self.options['platform']
-        version = None
-        if "@" in platform:
-            platform, version = platform.rsplit("@", 1)
 
         try:
-            p = PlatformFactory.newPlatform(platform, version)
+            p = PlatformFactory.newPlatform(self.options['platform'])
         except exception.UnknownPlatform:
             self.cmd_ctx.invoke(
                 cmd_platform_install, platforms=[self.options['platform']])
-            p = PlatformFactory.newPlatform(platform, version)
+            p = PlatformFactory.newPlatform(self.options['platform'])
 
         return p.run(build_vars, build_targets, self.verbose)
 

platformio/commands/test.py

@@ -174,11 +174,7 @@ class TestProcessor(object):
         if self.options.get("upload_port", envdata.get("upload_port")):
             return self.options.get("upload_port", envdata.get("upload_port"))
 
-        platform = envdata['platform']
+        p = PlatformFactory.newPlatform(envdata['platform'])
-        version = None
-        if "@" in platform:
-            platform, version = platform.rsplit("@", 1)
-        p = PlatformFactory.newPlatform(platform, version)
         bconfig = p.board_config(envdata['board'])
 
         port = None

platformio/exception.py

@@ -72,26 +72,16 @@ class UnknownPackage(PlatformioException):
 
 class UndefinedPackageVersion(PlatformioException):
 
-    MESSAGE = "Can not find package '{0}' with version requirements '{1}'"\
+    MESSAGE = "Could not find a version that satisfies the requirement '{0}'"\
-        " for your system '{2}'"
+              " for your system '{1}'"
-
-
-class UndefinedPlatformVersion(PlatformioException):
-
-    MESSAGE = "Can not find platform '{0}' with version requirements '{1}'"
 
 
 class PackageInstallError(PlatformioException):
 
-    MESSAGE = "Can not install package '{0}' with version requirements '{1}' "\
+    MESSAGE = "Can not install '{0}' with version requirements '{1}' "\
               "for your system '{2}'"
 
 
-class NonSystemPackage(PlatformioException):
-
-    MESSAGE = "The package '{0}' is not available for your system '{1}'"
-
-
 class FDUnrecognizedStatusCode(PlatformioException):
 
     MESSAGE = "Got an unrecognized status code '{0}' when downloaded {1}"

platformio/maintenance.py

@@ -130,7 +130,7 @@ def after_upgrade(ctx):
             # patch development platforms
             pm = PlatformManager()
             for manifest in pm.get_installed():
-                pm.update(manifest['name'], "~" + manifest['version'])
+                pm.update(manifest['name'], "^" + manifest['version'])
 
             click.secho("PlatformIO has been successfully upgraded to %s!\n" %
                         __version__, fg="green")
@@ -225,9 +225,9 @@ def check_internal_updates(ctx, what):
     if what == "platforms":
         pm = PlatformManager()
         for manifest in pm.get_installed():
-            if pm.is_outdated(manifest['name'], manifest['version']):
+            if manifest['name'] not in outdated_items and \
-                outdated_items.append(
+                    pm.is_outdated(manifest['name']):
-                    "%s@%s" % (manifest['name'], manifest['version']))
+                outdated_items.append(manifest['name'])
     elif what == "libraries":
         lm = LibraryManager()
         outdated_items = lm.get_outdated()

platformio/managers/package.py

@@ -14,7 +14,7 @@
 
 import json
 import os
-from os.path import dirname, isdir, isfile, islink, join
+from os.path import basename, dirname, isdir, isfile, islink, join
 from shutil import copyfile, copytree, rmtree
 from tempfile import mkdtemp
 
@@ -75,18 +75,21 @@ class PkgRepoMixin(object):
     def max_satisfying_repo_version(versions, requirements=None):
         item = None
         systype = util.get_systype()
-        if requirements is not None:
+        reqspec = None
-            requirements = str(requirements)
+        if requirements:
+            try:
+                reqspec = semantic_version.Spec(requirements)
+            except ValueError:
+                pass
+
         for v in versions:
-            if isinstance(v['version'], int):
+            if ("system" in v and v['system'] not in ("all", "*") and
+                    systype not in v['system']):
                 continue
-            if v['system'] not in ("all", "*") and systype not in v['system']:
+            specver = semantic_version.Version(v['version'])
+            if reqspec and specver not in reqspec:
                 continue
-            if requirements and not semantic_version.match(
+            if not item or semantic_version.Version(item['version']) < specver:
-                    requirements, v['version']):
-                continue
-            if item is None or semantic_version.compare(
-                    v['version'], item['version']) == 1:
                 item = v
         return item
 
@@ -96,8 +99,8 @@ class PkgRepoMixin(object):
             pkgdata = self.max_satisfying_repo_version(versions, requirements)
             if not pkgdata:
                 continue
-            if (not version or semantic_version.compare(
+            if not version or semantic_version.compare(pkgdata['version'],
-                    pkgdata['version'], version) == 1):
+                                                       version) == 1:
                 version = pkgdata['version']
         return version
 
@@ -126,77 +129,11 @@ class PkgInstallerMixin(object):
         manifest_path = self.get_manifest_path(pkg_dir)
         if manifest_path:
             manifest = util.load_json(manifest_path)
-            manifest['_manifest_path'] = manifest_path
             manifest['__pkg_dir'] = pkg_dir
             return manifest
         return None
 
-    def _install_from_piorepo(self, name, requirements):
+    def check_pkg_structure(self, pkg_dir):
-        pkg_dir = None
-        pkgdata = None
-        versions = None
-        for versions in PackageRepoIterator(name, self.repositories):
-            pkgdata = self.max_satisfying_repo_version(versions, requirements)
-            if not pkgdata:
-                continue
-            try:
-                pkg_dir = self._install_from_url(
-                    name, pkgdata['url'], requirements, pkgdata.get("sha1"))
-                break
-            except Exception as e:  # pylint: disable=broad-except
-                click.secho("Warning! Package Mirror: %s" % e, fg="yellow")
-                click.secho("Looking for another mirror...", fg="yellow")
-
-        if versions is None:
-            raise exception.UnknownPackage(name)
-        elif not pkgdata:
-            if "platform" in self.manifest_name:
-                raise exception.UndefinedPlatformVersion(
-                    name, requirements or "latest")
-            else:
-                raise exception.UndefinedPackageVersion(
-                    name, requirements or "latest", util.get_systype())
-        return pkg_dir
-
-    def _install_from_url(self, name, url, requirements=None, sha1=None):
-        pkg_dir = None
-        tmp_dir = mkdtemp("-package", "installing-", self.package_dir)
-
-        # Handle GitHub URL (https://github.com/user/repo.git)
-        if url.endswith(".git") and not url.startswith("git"):
-            url = "git+" + url
-
-        try:
-            if url.startswith("file://"):
-                url = url[7:]
-                if isfile(url):
-                    self.unpack(url, tmp_dir)
-                else:
-                    rmtree(tmp_dir)
-                    copytree(url, tmp_dir)
-            elif url.startswith(("http://", "https://", "ftp://")):
-                dlpath = self.download(url, tmp_dir, sha1)
-                assert isfile(dlpath)
-                self.unpack(dlpath, tmp_dir)
-                os.remove(dlpath)
-            else:
-                vcs = VCSClientFactory.newClient(tmp_dir, url)
-                assert vcs.export()
-                with open(join(vcs.storage_dir,
-                               self.VCS_MANIFEST_NAME), "w") as fp:
-                    json.dump({
-                        "name": name,
-                        "version": "0.0.0+rev%s" % vcs.get_latest_revision(),
-                        "url": url}, fp)
-
-            self._check_pkg_structure(tmp_dir)
-            pkg_dir = self._install_from_tmp_dir(tmp_dir, requirements)
-        finally:
-            if isdir(tmp_dir):
-                rmtree(tmp_dir)
-        return pkg_dir
-
-    def _check_pkg_structure(self, pkg_dir):
         if self.manifest_exists(pkg_dir):
             return True
 
@@ -225,28 +162,98 @@ class PkgInstallerMixin(object):
             "Could not find '%s' manifest file in the package" %
             self.manifest_name)
 
+    def _install_from_piorepo(self, name, requirements):
+        pkg_dir = None
+        pkgdata = None
+        versions = None
+        for versions in PackageRepoIterator(name, self.repositories):
+            pkgdata = self.max_satisfying_repo_version(versions, requirements)
+            if not pkgdata:
+                continue
+            try:
+                pkg_dir = self._install_from_url(
+                    name, pkgdata['url'], requirements, pkgdata.get("sha1"))
+                break
+            except Exception as e:  # pylint: disable=broad-except
+                click.secho("Warning! Package Mirror: %s" % e, fg="yellow")
+                click.secho("Looking for the another mirror...", fg="yellow")
+
+        if versions is None:
+            raise exception.UnknownPackage(name)
+        elif not pkgdata:
+            raise exception.UndefinedPackageVersion(requirements or "latest",
+                                                    util.get_systype())
+        return pkg_dir
+
+    def _install_from_url(self, name, url, requirements=None, sha1=None):
+        pkg_dir = None
+        tmp_dir = mkdtemp("-package", "installing-", self.package_dir)
+
+        try:
+            if url.startswith("file://"):
+                url = url[7:]
+                if isfile(url):
+                    self.unpack(url, tmp_dir)
+                else:
+                    rmtree(tmp_dir)
+                    copytree(url, tmp_dir)
+            elif url.startswith(("http://", "https://")):
+                dlpath = self.download(url, tmp_dir, sha1)
+                assert isfile(dlpath)
+                self.unpack(dlpath, tmp_dir)
+                os.remove(dlpath)
+            else:
+                vcs = VCSClientFactory.newClient(tmp_dir, url)
+                assert vcs.export()
+                with open(join(vcs.storage_dir, self.VCS_MANIFEST_NAME),
+                          "w") as fp:
+                    json.dump({
+                        "name": name,
+                        "version": vcs.get_current_revision(),
+                        "url": url,
+                        "requirements": requirements
+                    }, fp)
+
+            self.check_pkg_structure(tmp_dir)
+            pkg_dir = self._install_from_tmp_dir(tmp_dir, requirements)
+        finally:
+            if isdir(tmp_dir):
+                rmtree(tmp_dir)
+        return pkg_dir
+
     def _install_from_tmp_dir(self, tmp_dir, requirements=None):
         tmpmanifest = self.load_manifest(tmp_dir)
         assert set(["name", "version"]) <= set(tmpmanifest.keys())
         name = tmpmanifest['name']
         pkg_dir = join(self.package_dir, name)
+        if "id" in tmpmanifest:
+            name += "_ID%d" % tmpmanifest['id']
+            pkg_dir = join(self.package_dir, name)
 
         # package should satisfy requirements
         if requirements:
-            assert semantic_version.match(
+            mismatch_error = (
-                requirements, tmpmanifest['version'])
+                "Package version %s doesn't satisfy requirements %s" % (
+                    tmpmanifest['version'], requirements))
+            try:
+                reqspec = semantic_version.Spec(requirements)
+                tmpmanver = semantic_version.Version(
+                    tmpmanifest['version'], partial=True)
+                assert tmpmanver in reqspec, mismatch_error
 
                 if self.manifest_exists(pkg_dir):
-            manifest = self.load_manifest(pkg_dir)
+                    curmanifest = self.load_manifest(pkg_dir)
-            cmp_result = semantic_version.compare(
+                    curmanver = semantic_version.Version(
-                tmpmanifest['version'], manifest['version'])
+                        curmanifest['version'], partial=True)
-            if cmp_result == 1:
+                    # if current package version < new package, backup it
-                # if main package version < new package, backup it
+                    if tmpmanver > curmanver:
-                os.rename(pkg_dir, join(
+                        os.rename(pkg_dir, join(self.package_dir, "%s@%s" % (
-                    self.package_dir, "%s@%s" % (name, manifest['version'])))
+                            name, curmanifest['version'])))
-            elif cmp_result == -1:
+                    elif tmpmanver < curmanver:
-                pkg_dir = join(
+                        pkg_dir = join(self.package_dir, "%s@%s" %
-                    self.package_dir, "%s@%s" % (name, tmpmanifest['version']))
+                                       (name, tmpmanifest['version']))
+            except ValueError:
+                assert tmpmanifest['version'] == requirements, mismatch_error
 
         # remove previous/not-satisfied package
         if isdir(pkg_dir):
@@ -291,6 +298,50 @@ class BasePkgManager(PkgRepoMixin, PkgInstallerMixin):
     def print_message(self, message, nl=True):
         click.echo("%s: %s" % (self.__class__.__name__, message), nl=nl)
 
+    @staticmethod
+    def parse_pkg_name(  # pylint: disable=too-many-branches
+            text, requirements=None):
+        text = str(text)
+        if not requirements and "@" in text and not text.startswith("git@"):
+            text, requirements = text.rsplit("@", 1)
+        if text.isdigit():
+            text = "id=" + text
+
+        url_marker = "://"
+        name, url = (None, text)
+        if "=" in text and not text.startswith("id="):
+            name, url = text.split("=", 1)
+
+        # Handle GitHub URL (https://github.com/user/package.git)
+        if url.startswith("https://github.com/") and \
+           not url.endswith((".zip", ".tar.gz")):
+            url = "git+" + url
+        # Handle Developer Mbed URL
+        # (https://developer.mbed.org/users/user/code/package/)
+        if url.startswith("https://developer.mbed.org"):
+            url = "hg+" + url
+
+        # git@github.com:user/package.git
+        if url.startswith("git@"):
+            url_marker = "git@"
+
+        if any([s in url for s in ("\\", "/")]) and url_marker not in url:
+            if isfile(url) or isdir(url):
+                url = "file://" + url
+            elif url.count("/") == 1 and not url.startswith("git@"):
+                url = "git+https://github.com/" + url
+        if url_marker in url and not name:
+            _url = url.split("#", 1)[0] if "#" in url else url
+            if _url.endswith(("\\", "/")):
+                _url = _url[:-1]
+            name = basename(_url)
+            if "." in name and not name.startswith("."):
+                name = name.split(".", 1)[0]
+
+        if url_marker not in url:
+            url = None
+        return (name or text, requirements, url)
+
     def get_installed(self):
         if self.package_dir in BasePkgManager._INSTALLED_CACHE:
             return BasePkgManager._INSTALLED_CACHE[self.package_dir]
@@ -304,26 +355,8 @@ class BasePkgManager(PkgRepoMixin, PkgInstallerMixin):
         BasePkgManager._INSTALLED_CACHE[self.package_dir] = items
         return items
 
-    def is_installed(self, name, requirements=None):
+    def get_installed_dir(self, name, requirements=None, url=None):
-        installed = self.get_installed()
+        pkg_id = int(name[3:]) if name.startswith("id=") else 0
-        reqspec = None
-        if requirements:
-            try:
-                reqspec = semantic_version.Spec(requirements)
-            except ValueError:
-                pass
-
-        if not reqspec:
-            return any([p['name'] == name for p in installed])
-
-        for p in installed:
-            if p['name'] != name:
-                continue
-            elif reqspec.match(semantic_version.Version(p['version'])):
-                return True
-        return None
-
-    def max_installed_version(self, name, requirements=None):
         best = None
         reqspec = None
         if requirements:
@@ -333,123 +366,152 @@ class BasePkgManager(PkgRepoMixin, PkgInstallerMixin):
                 pass
 
         for manifest in self.get_installed():
-            if manifest['name'] != name:
+            if pkg_id and manifest.get("id") != pkg_id:
                 continue
-            elif reqspec and not reqspec.match(
+            elif not pkg_id and manifest['name'] != name:
-                    semantic_version.Version(manifest['version'])):
                 continue
-            elif (not best or semantic_version.compare(
+            elif not reqspec and requirements:
-                    manifest['version'], best['version']) == 1):
+                if requirements == manifest['version']:
                     best = manifest
-
+                    break
+                continue
+            try:
+                if reqspec and not reqspec.match(
+                        semantic_version.Version(
+                            manifest['version'], partial=True)):
+                    continue
+                elif not best or (semantic_version.Version(
+                        manifest['version'], partial=True) >
+                                  semantic_version.Version(
+                                      best['version'], partial=True)):
+                    best = manifest
+            except ValueError:
+                pass
         if best:
+            # check that URL is the same in installed package (VCS)
+            if url and best.get("url") != url:
+                return None
             return best.get("__pkg_dir")
         return None
 
-    def install(self, name, requirements, silent=False, trigger_event=True):
+    def install(self, name, requirements=None, quiet=False,
-        installed = self.is_installed(name, requirements)
+                trigger_event=True):
-        if not installed or not silent:
+        name, requirements, url = self.parse_pkg_name(name, requirements)
-            self.print_message("Installing %s @ %s:" % (
+        installed_dir = self.get_installed_dir(name, requirements, url)
-                click.style(name, fg="cyan"),
-                requirements if requirements else "latest"))
-        if installed:
-            if not silent:
-                click.secho("Already installed", fg="yellow")
-            return self.max_installed_version(
-                name, requirements)
 
-        if (requirements and any([s in requirements for s in ("\\", "/")]) and
+        if not installed_dir or not quiet:
-                "://" not in requirements and (
+            msg = "Installing " + click.style(name, fg="cyan")
-                    isfile(requirements) or isdir(requirements))):
+            if requirements:
-            requirements = "file://" + requirements
+                msg += " @ " + requirements
+            self.print_message(msg)
+        if installed_dir:
+            if not quiet:
+                click.secho(
+                    "{name} @ {version} is already installed".format(
+                        **self.load_manifest(installed_dir)),
+                    fg="yellow")
+            return installed_dir
 
-        if requirements and "://" in requirements:
+        if url:
-            pkg_dir = self._install_from_url(name, requirements)
+            pkg_dir = self._install_from_url(name, url, requirements)
         else:
             pkg_dir = self._install_from_piorepo(name, requirements)
         if not pkg_dir or not self.manifest_exists(pkg_dir):
-            raise exception.PackageInstallError(
+            raise exception.PackageInstallError(name, requirements or "*",
-                name, requirements or "latest", util.get_systype())
+                                                util.get_systype())
 
         self.reset_cache()
+        manifest = self.load_manifest(pkg_dir)
+
         if trigger_event:
             telemetry.on_event(
                 category=self.__class__.__name__,
-                action="Install", label=name)
+                action="Install",
+                label=manifest['name'])
+
+        click.secho(
+            "{name} @ {version} has been successfully installed!".format(
+                **manifest),
+            fg="green")
 
         return pkg_dir
 
     def uninstall(self, name, requirements=None, trigger_event=True):
-        self.print_message("Uninstalling %s @ %s: \t" % (
+        name, requirements, url = self.parse_pkg_name(name, requirements)
-            click.style(name, fg="cyan"),
+        installed_dir = self.get_installed_dir(name, requirements, url)
-            requirements if requirements else "latest"), nl=False)
+        if not installed_dir:
-        found = False
+            click.secho(
-        for manifest in self.get_installed():
+                "%s @ %s is not installed" % (name, requirements or "*"),
-            if manifest['name'] != name:
+                fg="yellow")
-                continue
+            return
-            if (requirements and not semantic_version.match(
-                    requirements, manifest['version'])):
-                continue
-            found = True
-            if isdir(manifest['__pkg_dir']):
-                if islink(manifest['__pkg_dir']):
-                    os.unlink(manifest['__pkg_dir'])
-                else:
-                    rmtree(manifest['__pkg_dir'])
 
-        if not found:
+        manifest = self.load_manifest(installed_dir)
-            click.secho("Not installed", fg="yellow")
+        click.echo(
-            return False
+            "Uninstalling %s @ %s: \t" % (click.style(
+                manifest['name'], fg="cyan"), manifest['version']),
+            nl=False)
+
+        if isdir(installed_dir):
+            if islink(installed_dir):
+                os.unlink(installed_dir)
             else:
+                rmtree(installed_dir)
+
         click.echo("[%s]" % click.style("OK", fg="green"))
 
         self.reset_cache()
         if trigger_event:
             telemetry.on_event(
                 category=self.__class__.__name__,
-                action="Uninstall", label=name)
+                action="Uninstall",
+                label=manifest['name'])
+        return True
 
-    def update(self, name, requirements=None):
+    def update(self, name, requirements=None, only_check=False):
-        self.print_message("Updating %s @ %s:" % (
+        name, requirements, url = self.parse_pkg_name(name, requirements)
-            click.style(name, fg="yellow"),
+        installed_dir = self.get_installed_dir(name, requirements, url)
-            requirements if requirements else "latest"))
+        if not installed_dir:
-
-        latest_version = self.get_latest_repo_version(name, requirements)
-        if latest_version is None:
             click.secho(
-                "Ignored! '%s' is not listed in registry" % name,
+                "%s @ %s is not installed" % (name, requirements or "*"),
                 fg="yellow")
             return
 
-        current = None
+        manifest = self.load_manifest(installed_dir)
-        for manifest in self.get_installed():
+        click.echo(
-            if manifest['name'] != name:
+            "Updating %s @ %s: \t" % (click.style(
-                continue
+                manifest['name'], fg="cyan"), manifest['version']),
-            if (requirements and not semantic_version.match(
+            nl=False)
-                    requirements, manifest['version'])):
+        manifest_path = self.get_manifest_path(installed_dir)
-                continue
+        if manifest_path.endswith(self.VCS_MANIFEST_NAME):
-            if (not current or semantic_version.compare(
+            if only_check:
-                    manifest['version'], current['version']) == 1):
+                click.echo("[%s]" % (click.style("Skip", fg="yellow")))
-                current = manifest
-
-        if current is None:
                 return
-
+            click.echo("[%s]" % (click.style("Checking", fg="yellow")))
-        current_version = current['version']
+            vcs = VCSClientFactory.newClient(installed_dir, manifest['url'])
-        click.echo("Versions: Current=%s, Latest=%s \t " %
+            if not vcs.can_be_updated:
-                   (current_version, latest_version), nl=False)
+                click.secho(
-
+                    "Skip update because repository is fixed "
-        if current_version == latest_version:
+                    "to %s revision" % manifest['version'],
-            click.echo("[%s]" % (click.style("Up-to-date", fg="green")))
+                    fg="yellow")
-            return True
+                return
+            assert vcs.update()
+            with open(manifest_path, "w") as fp:
+                manifest['version'] = vcs.get_current_revision()
+                json.dump(manifest, fp)
         else:
+            latest_version = self.get_latest_repo_version(name, requirements)
+            if manifest['version'] == latest_version:
+                click.echo("[%s]" % (click.style("Up-to-date", fg="green")))
+                return
             click.echo("[%s]" % (click.style("Out-of-date", fg="red")))
-
+            if only_check:
+                return
             self.install(name, latest_version, trigger_event=False)
 
         telemetry.on_event(
             category=self.__class__.__name__,
-            action="Update", label=name)
+            action="Update",
+            label=manifest['name'])
         return True
 
 

platformio/managers/platform.py

@@ -26,17 +26,17 @@ import semantic_version
 from platformio import app, exception, util
 from platformio.managers.package import BasePkgManager, PackageManager
 
-PLATFORMS_DIR = join(util.get_home_dir(), "platforms")
-PACKAGES_DIR = join(util.get_home_dir(), "packages")
-
 
 class PlatformManager(BasePkgManager):
 
     def __init__(self, package_dir=None, repositories=None):
         if not repositories:
-            repositories = ["http://dl.platformio.org/platforms/manifest.json"]
+            repositories = [
-        BasePkgManager.__init__(
+                "https://dl.platformio.org/platforms/manifest.json"
-            self, package_dir or PLATFORMS_DIR, repositories)
+            ]
+        BasePkgManager.__init__(self, package_dir or
+                                join(util.get_home_dir(), "platforms"),
+                                repositories)
 
     @property
     def manifest_name(self):
@@ -44,27 +44,29 @@ class PlatformManager(BasePkgManager):
 
     def install(self,  # pylint: disable=too-many-arguments,arguments-differ
                 name, requirements=None, with_packages=None,
-                without_packages=None, skip_default_packages=False):
+                without_packages=None, skip_default_package=False):
         platform_dir = BasePkgManager.install(self, name, requirements)
         p = PlatformFactory.newPlatform(self.get_manifest_path(platform_dir))
-        p.install_packages(
+        p.install_packages(with_packages, without_packages,
-            with_packages, without_packages, skip_default_packages)
+                           skip_default_package)
         self.cleanup_packages(p.packages.keys())
         return True
 
-    def uninstall(self,  # pylint: disable=arguments-differ
+    def uninstall(  # pylint: disable=arguments-differ
-                  name, requirements=None):
+            self, name, requirements=None):
+        name, requirements, _ = self.parse_pkg_name(name, requirements)
         p = PlatformFactory.newPlatform(name, requirements)
         BasePkgManager.uninstall(self, name, requirements)
         self.cleanup_packages(p.packages.keys())
         return True
 
     def update(self,  # pylint: disable=arguments-differ
-               name, requirements=None, only_packages=False):
+               name, requirements=None, only_packages=False, only_check=False):
+        name, requirements, _ = self.parse_pkg_name(name, requirements)
         if not only_packages:
-            BasePkgManager.update(self, name, requirements)
+            BasePkgManager.update(self, name, requirements, only_check)
         p = PlatformFactory.newPlatform(name, requirements)
-        p.update_packages()
+        p.update_packages(only_check)
         self.cleanup_packages(p.packages.keys())
         return True
 
@@ -77,14 +79,14 @@ class PlatformManager(BasePkgManager):
         self.reset_cache()
         deppkgs = {}
         for manifest in PlatformManager().get_installed():
-            p = PlatformFactory.newPlatform(
+            p = PlatformFactory.newPlatform(manifest['name'],
-                manifest['name'], manifest['version'])
+                                            manifest['version'])
             for pkgname, pkgmanifest in p.get_installed_packages().items():
                 if pkgname not in deppkgs:
                     deppkgs[pkgname] = set()
                 deppkgs[pkgname].add(pkgmanifest['version'])
 
-        pm = PackageManager(PACKAGES_DIR)
+        pm = PackageManager(join(util.get_home_dir(), "packages"))
         for manifest in pm.get_installed():
             if manifest['name'] not in names:
                 continue
@@ -126,35 +128,36 @@ class PlatformFactory(object):
     def load_module(name, path):
         module = None
         try:
-            module = load_source(
+            module = load_source("platformio.managers.platform.%s" % name,
-                "platformio.managers.platform.%s" % name, path)
+                                 path)
         except ImportError:
             raise exception.UnknownPlatform(name)
         return module
 
     @classmethod
     def newPlatform(cls, name, requirements=None):
+        if not requirements and "@" in name:
+            name, requirements = name.rsplit("@", 1)
         platform_dir = None
         if name.endswith("platform.json") and isfile(name):
             platform_dir = dirname(name)
             name = util.load_json(name)['name']
         else:
-            platform_dir = PlatformManager().max_installed_version(
+            platform_dir = PlatformManager().get_installed_dir(name,
-                name, requirements)
+                                                               requirements)
 
         if not platform_dir:
-            raise exception.UnknownPlatform(
+            raise exception.UnknownPlatform(name if not requirements else
-                name if not requirements else "%s@%s" % (name, requirements))
+                                            "%s@%s" % (name, requirements))
 
         platform_cls = None
         if isfile(join(platform_dir, "platform.py")):
             platform_cls = getattr(
                 cls.load_module(name, join(platform_dir, "platform.py")),
-                cls.get_clsname(name)
+                cls.get_clsname(name))
-            )
         else:
             platform_cls = type(
-                str(cls.get_clsname(name)), (PlatformBase,), {})
+                str(cls.get_clsname(name)), (PlatformBase, ), {})
 
         _instance = platform_cls(join(platform_dir, "platform.json"))
         assert isinstance(_instance, PlatformBase)
@@ -187,12 +190,13 @@ class PlatformPackagesMixin(object):
                 items[name] = manifest
         return items
 
-    def install_packages(self, with_packages=None, without_packages=None,
+    def install_packages(self,
-                         skip_default_packages=False, silent=False):
+                         with_packages=None,
-        with_packages = set(
+                         without_packages=None,
-            self.pkg_types_to_names(with_packages or []))
+                         skip_default_package=False,
-        without_packages = set(
+                         quiet=False):
-            self.pkg_types_to_names(without_packages or []))
+        with_packages = set(self.pkg_types_to_names(with_packages or []))
+        without_packages = set(self.pkg_types_to_names(without_packages or []))
 
         upkgs = with_packages | without_packages
         ppkgs = set(self.packages.keys())
@@ -203,14 +207,18 @@ class PlatformPackagesMixin(object):
             if name in without_packages:
                 continue
             elif (name in with_packages or
-                  not (skip_default_packages or opts.get("optional", False))):
+                  not (skip_default_package or opts.get("optional", False))):
-                self.pm.install(name, opts.get("version"), silent=silent)
+                if any([s in opts.get("version", "") for s in ("\\", "/")]):
+                    self.pm.install(
+                        "%s=%s" % (name, opts['version']), quiet=quiet)
+                else:
+                    self.pm.install(name, opts.get("version"), quiet=quiet)
 
         return True
 
-    def update_packages(self):
+    def update_packages(self, only_check=False):
         for name in self.get_installed_packages():
-            self.pm.update(name, self.packages[name]['version'])
+            self.pm.update(name, self.packages[name]['version'], only_check)
 
     def are_outdated_packages(self):
         for name, opts in self.get_installed_packages().items():
@@ -229,7 +237,7 @@ class PlatformRunMixin(object):
         assert isinstance(targets, list)
 
         self.configure_default_packages(variables, targets)
-        self.install_packages(silent=True)
+        self.install_packages(quiet=True)
 
         self._verbose = verbose or app.get_setting("force_verbose")
 
@@ -261,10 +269,8 @@ class PlatformRunMixin(object):
 
         cmd = [
             os.path.normpath(sys.executable),
-            join(self.get_package_dir("tool-scons"), "script", "scons"),
+            join(self.get_package_dir("tool-scons"), "script", "scons"), "-Q",
-            "-Q",
+            "-j %d" % self.get_job_nums(), "--warn=no-no-parallel-support",
-            "-j %d" % self.get_job_nums(),
-            "--warn=no-no-parallel-support",
             "-f", join(util.get_source_dir(), "builder", "main.py")
         ]
         if not self._verbose and "-c" not in targets:
@@ -278,8 +284,7 @@ class PlatformRunMixin(object):
         result = util.exec_command(
             cmd,
             stdout=util.AsyncPipe(self.on_run_out),
-            stderr=util.AsyncPipe(self.on_run_err)
+            stderr=util.AsyncPipe(self.on_run_err))
-        )
         return result
 
     def on_run_out(self, line):
@@ -315,7 +320,8 @@ class PlatformBase(PlatformPackagesMixin, PlatformRunMixin):
         self._manifest = util.load_json(manifest_path)
 
         self.pm = PackageManager(
-            PACKAGES_DIR, self._manifest.get("packageRepositories"))
+            join(util.get_home_dir(), "packages"),
+            self._manifest.get("packageRepositories"))
 
         self._verbose = False
 

platformio/vcsclient.py

@@ -12,12 +12,11 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from os import listdir
+import re
-from os.path import isdir, join
+from os.path import join
-from platform import system
 from subprocess import check_call
 from sys import modules
-from urlparse import urlsplit, urlunsplit
+from urlparse import urlparse
 
 from platformio import util
 from platformio.exception import PlatformioException
@@ -26,28 +25,16 @@ from platformio.exception import PlatformioException
 class VCSClientFactory(object):
 
     @staticmethod
-    def newClient(src_dir, remote_url=None, branch=None):
+    def newClient(src_dir, remote_url):
-        clsnametpl = "%sClient"
+        result = urlparse(remote_url)
-        vcscls = None
+        type_ = result.scheme
-        type_ = None
+        if "+" in result.scheme:
-        if remote_url:
+            type_, _ = result.scheme.split("+", 1)
-            scheme, netloc, path, query, branch = urlsplit(remote_url)
+            remote_url = remote_url[len(type_) + 1:]
-            type_ = scheme
+        if result.fragment:
-            if "+" in type_:
+            remote_url = remote_url.rsplit("#", 1)[0]
-                type_, scheme = type_.split("+", 1)
+        obj = getattr(modules[__name__], "%sClient" % type_.title())(
-            remote_url = urlunsplit((scheme, netloc, path, query, None))
+            src_dir, remote_url, result.fragment)
-            vcscls = getattr(modules[__name__], clsnametpl % type_.title())
-        elif isdir(src_dir):
-            for item in listdir(src_dir):
-                if not isdir(join(src_dir, item)) or not item.startswith("."):
-                    continue
-                try:
-                    vcscls = getattr(
-                        modules[__name__], clsnametpl % item[1:].title())
-                except AttributeError:
-                    pass
-        assert vcscls
-        obj = vcscls(src_dir, remote_url, branch)
         assert isinstance(obj, VCSClientBase)
         return obj
 
@@ -79,19 +66,29 @@ class VCSClientBase(object):
     def export(self):
         raise NotImplementedError
 
-    def get_latest_revision(self):
+    def update(self):
+        raise NotImplementedError
+
+    @property
+    def can_be_updated(self):
+        return not self.branch
+
+    def get_current_revision(self):
         raise NotImplementedError
 
     def run_cmd(self, args, **kwargs):
         args = [self.command] + args
-        kwargs['shell'] = system() == "Windows"
+        if "cwd" not in kwargs:
+            kwargs['cwd'] = self.src_dir
         return check_call(args, **kwargs) == 0
 
     def get_cmd_output(self, args, **kwargs):
         args = [self.command] + args
+        if "cwd" not in kwargs:
+            kwargs['cwd'] = self.src_dir
         result = util.exec_command(args, **kwargs)
         if result['returncode'] == 0:
-            return result['out']
+            return result['out'].strip()
         raise PlatformioException(
             "VCS: Could not receive an output from `%s` command (%s)" % (
                 args, result))
@@ -101,16 +98,42 @@ class GitClient(VCSClientBase):
 
     command = "git"
 
+    def get_branches(self):
+        output = self.get_cmd_output(["branch"])
+        output = output.replace("*", "")  # fix active branch
+        return [b.strip() for b in output.split("\n")]
+
+    def get_tags(self):
+        output = self.get_cmd_output(["tag", "-l"])
+        return [t.strip() for t in output.split("\n")]
+
+    @staticmethod
+    def is_commit_id(text):
+        return text and re.match(r"[0-9a-f]{7,}$", text) is not None
+
+    @property
+    def can_be_updated(self):
+        return not self.branch or not self.is_commit_id(self.branch)
+
     def export(self):
-        args = ["clone", "--recursive", "--depth", "1"]
+        is_commit = self.is_commit_id(self.branch)
+        args = ["clone", "--recursive"]
+        if not self.branch or not is_commit:
+            args += ["--depth", "1"]
             if self.branch:
-            args.extend(["--branch", self.branch])
+                args += ["--branch", self.branch]
-        args.extend([self.remote_url, self.src_dir])
+        args += [self.remote_url, self.src_dir]
+        assert self.run_cmd(args)
+        if is_commit:
+            return self.run_cmd(["reset", "--hard", self.branch])
+        return True
+
+    def update(self):
+        args = ["pull"]
         return self.run_cmd(args)
 
-    def get_latest_revision(self):
+    def get_current_revision(self):
-        return self.get_cmd_output(["rev-parse", "--short", "HEAD"],
+        return self.get_cmd_output(["rev-parse", "--short", "HEAD"])
-                                   cwd=self.src_dir).strip()
 
 
 class HgClient(VCSClientBase):
@@ -124,9 +147,12 @@ class HgClient(VCSClientBase):
         args.extend([self.remote_url, self.src_dir])
         return self.run_cmd(args)
 
-    def get_latest_revision(self):
+    def update(self):
-        return self.get_cmd_output(["identify", "--id"],
+        args = ["pull", "--update"]
-                                   cwd=self.src_dir).strip()
+        return self.run_cmd(args)
+
+    def get_current_revision(self):
+        return self.get_cmd_output(["identify", "--id"])
 
 
 class SvnClient(VCSClientBase):
@@ -134,12 +160,22 @@ class SvnClient(VCSClientBase):
     command = "svn"
 
     def export(self):
-        args = ["export", "--force"]
+        args = ["checkout"]
         if self.branch:
             args.extend(["--revision", self.branch])
         args.extend([self.remote_url, self.src_dir])
         return self.run_cmd(args)
 
-    def get_latest_revision(self):
+    def update(self):
-        return self.get_cmd_output(["info", "-r", "HEAD"],
+
-                                   cwd=self.src_dir).strip()
+        args = ["update"]
+        return self.run_cmd(args)
+
+    def get_current_revision(self):
+        output = self.get_cmd_output(["info", "--non-interactive",
+                                      "--trust-server-cert", "-r", "HEAD"])
+        for line in output.split("\n"):
+            line = line.strip()
+            if line.startswith("Revision:"):
+                return line.split(":", 1)[1].strip()
+        raise PlatformioException("Could not detect current SVN revision")

tests/commands/test_platform.py

@@ -13,15 +13,15 @@
 # limitations under the License.
 
 import json
+import os
+from os.path import join
 
-from platformio.commands.platform import \
+from platformio.commands import platform as cli_platform
-    platform_list as cmd_platform_list
+from platformio import exception, util
-from platformio.commands.platform import \
-    platform_search as cmd_platform_search
 
 
 def test_list_json_output(clirunner, validate_cliresult):
-    result = clirunner.invoke(cmd_platform_list, ["--json-output"])
+    result = clirunner.invoke(cli_platform.platform_list, ["--json-output"])
     validate_cliresult(result)
     list_result = json.loads(result.output)
     assert isinstance(list_result, list)
@@ -31,13 +31,13 @@ def test_list_json_output(clirunner, validate_cliresult):
 
 
 def test_list_raw_output(clirunner, validate_cliresult):
-    result = clirunner.invoke(cmd_platform_list)
+    result = clirunner.invoke(cli_platform.platform_list)
     validate_cliresult(result)
     assert "teensy" in result.output
 
 
 def test_search_json_output(clirunner, validate_cliresult):
-    result = clirunner.invoke(cmd_platform_search,
+    result = clirunner.invoke(cli_platform.platform_search,
                               ["arduino", "--json-output"])
     validate_cliresult(result)
     search_result = json.loads(result.output)
@@ -48,6 +48,79 @@ def test_search_json_output(clirunner, validate_cliresult):
 
 
 def test_search_raw_output(clirunner, validate_cliresult):
-    result = clirunner.invoke(cmd_platform_search, ["arduino"])
+    result = clirunner.invoke(cli_platform.platform_search, ["arduino"])
     validate_cliresult(result)
     assert "teensy" in result.output
+
+
+def test_install_uknown_from_registry(clirunner, validate_cliresult):
+    result = clirunner.invoke(cli_platform.platform_install,
+                              ["uknown-platform"])
+    assert result.exit_code == -1
+    assert isinstance(result.exception, exception.UnknownPackage)
+
+
+def test_install_uknown_version(clirunner, validate_cliresult):
+    result = clirunner.invoke(cli_platform.platform_install,
+                              ["atmelavr@99.99.99"])
+    assert result.exit_code == -1
+    assert isinstance(result.exception, exception.UndefinedPackageVersion)
+
+
+def test_complex(clirunner, validate_cliresult):
+    items = [
+        "teensy",
+        "https://github.com/platformio/platform-teensy/archive/develop.zip",
+        "https://github.com/platformio/platform-teensy.git",
+        "platformio/platform-teensy",
+    ]
+    for item in items:
+        with clirunner.isolated_filesystem():
+            os.environ["PLATFORMIO_HOME_DIR"] = os.getcwd()
+            try:
+                result = clirunner.invoke(cli_platform.platform_install,
+                                          [item])
+                validate_cliresult(result)
+                assert all([
+                    s in result.output
+                    for s in ("teensy", "Downloading", "Unpacking",
+                              "tool-scons")
+                ])
+
+                # show platform information
+                result = clirunner.invoke(cli_platform.platform_show,
+                                          ["teensy"])
+                validate_cliresult(result)
+                assert "teensy" in result.output
+
+                # list platforms
+                result = clirunner.invoke(cli_platform.platform_list,
+                                          ["--json-output"])
+                validate_cliresult(result)
+                list_result = json.loads(result.output)
+                assert isinstance(list_result, list)
+                assert len(list_result) == 1
+                assert list_result[0]["name"] == "teensy"
+                assert list_result[0]["packages"] == ["tool-scons"]
+
+                # try to install again
+                result = clirunner.invoke(cli_platform.platform_install,
+                                          ["teensy"])
+                validate_cliresult(result)
+                assert "is already installed" in result.output
+
+                # try to update
+                result = clirunner.invoke(cli_platform.platform_update)
+                validate_cliresult(result)
+                assert "teensy" in result.output
+                assert "Up-to-date" in result.output
+
+                # try to uninstall
+                result = clirunner.invoke(cli_platform.platform_uninstall,
+                                          ["teensy"])
+                validate_cliresult(result)
+                for folder in ("platforms", "packages"):
+                    assert len(os.listdir(join(util.get_home_dir(),
+                                               folder))) == 0
+            finally:
+                del os.environ["PLATFORMIO_HOME_DIR"]

tests/test_managers.py

@@ -0,0 +1,66 @@
+# Copyright 2014-present Ivan Kravets <me@ikravets.com>
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from platformio import util
+from platformio.managers.package import BasePkgManager
+
+
+def test_pkg_name_parser():
+    items = [
+        ["PkgName", ("PkgName", None, None)],
+        [("PkgName", "!=1.2.3,<2.0"), ("PkgName", "!=1.2.3,<2.0", None)],
+        ["PkgName@1.2.3", ("PkgName", "1.2.3", None)],
+        [("PkgName@1.2.3", "1.2.5"), ("PkgName@1.2.3", "1.2.5", None)],
+        ["id:13", ("id:13", None, None)],
+        ["id:13@~1.2.3", ("id:13", "~1.2.3", None)],
+        [util.get_home_dir(),
+         (".platformio", None, "file://" + util.get_home_dir())],
+        ["LocalName=" + util.get_home_dir(),
+         ("LocalName", None, "file://" + util.get_home_dir())],
+        ["https://github.com/user/package.git",
+         ("package", None, "git+https://github.com/user/package.git")],
+        ["https://github.com/user/package/archive/branch.zip",
+         ("branch", None,
+          "https://github.com/user/package/archive/branch.zip")],
+        ["https://github.com/user/package/archive/branch.tar.gz",
+         ("branch", None,
+          "https://github.com/user/package/archive/branch.tar.gz")],
+        ["https://developer.mbed.org/users/user/code/package/",
+         ("package", None,
+          "hg+https://developer.mbed.org/users/user/code/package/")],
+        ["https://github.com/user/package#v1.2.3",
+         ("package", None, "git+https://github.com/user/package#v1.2.3")],
+        ["https://github.com/user/package.git#branch",
+         ("package", None, "git+https://github.com/user/package.git#branch")],
+        ["PkgName=https://github.com/user/package.git#a13d344fg56",
+         ("PkgName", None,
+          "git+https://github.com/user/package.git#a13d344fg56")],
+        ["PkgName=user/package",
+         ("PkgName", None, "git+https://github.com/user/package")],
+        ["PkgName=user/package#master",
+         ("PkgName", None, "git+https://github.com/user/package#master")],
+        ["git+https://github.com/user/package",
+         ("package", None, "git+https://github.com/user/package")],
+        ["hg+https://example.com/user/package",
+         ("package", None, "hg+https://example.com/user/package")],
+        ["git@github.com:user/package.git",
+         ("package", None, "git@github.com:user/package.git")],
+        ["git@github.com:user/package.git#v1.2.0",
+         ("package", None, "git@github.com:user/package.git#v1.2.0")]
+    ]
+    for params, result in items:
+        if isinstance(params, tuple):
+            assert BasePkgManager.parse_pkg_name(*params) == result
+        else:
+            assert BasePkgManager.parse_pkg_name(params) == result