Merge branch 'feature/readme_checklinks' into 'master'

CI: add script for checking README links

Closes IDF-1846

See merge request espressif/esp-idf!9188

examples/bluetooth/bluedroid/ble/blufi/README.md

@@ -10,7 +10,7 @@ To test this demo, you need to prepare a mobile phone with blufi application ins
 
 Blufi is completely open source, here is the download link:
 
-* [blufi source code](https://github.com/espressif/esp-idf/tree/master/examples/bluetooth/blufi)
+* [blufi source code](https://github.com/espressif/esp-idf/tree/master/examples/bluetooth/bluedroid/ble/blufi)
 
 * [BluFi protocol](https://docs.espressif.com/projects/esp-idf/en/latest/api-guides/blufi.html?highlight=blufi#the-frame-formats-defined-in-blufi)
 

examples/bluetooth/bluedroid/classic_bt/bt_discovery/README.md

@@ -1,6 +1,5 @@
-================= =====
+| Supported Targets | ESP32 |
-Supported Targets ESP32
+| ----------------- | ----- |
-================= =====
 
 ESP-IDF BT-INQUIRY demo
 ======================
@@ -9,10 +8,10 @@ Demo of Classic Bluetooth Device and Service Discovery
 
 This is the demo for user to use ESP_APIs to perform inquiry to search for a target device and then performs service search via SDP.
 
-Options choose step:
+Options choose step:`
-    1. idf.py menuconfig.
+1. `idf.py menuconfig`
-    2. enter menuconfig "Component config", choose "Bluetooth"
+2. enter menuconfig `Component config`, choose `Bluetooth`
-    3. enter menu Bluetooth, choose "Classic Bluetooth" and do not choose "Release DRAM from Classic BT controller"
+3. enter menu Bluetooth, choose `Classic Bluetooth` and do not choose `Release DRAM from Classic BT controller`
-    4. choose your options.
+4. choose your options.
 
 After the program started, the device will start inquiry to search for a device with Major device type "PHONE" in the Class of Device Field. Then it will cancel the inquiry and started to perform service discovering on this remote device.

examples/bluetooth/bluedroid/classic_bt/bt_spp_acceptor/README.md

@@ -0,0 +1,19 @@
+| Supported Targets | ESP32 |
+| ----------------- | ----- |
+
+ESP-IDF BT-SPP-ACCEPTOR demo
+======================
+
+Demo of SPP acceptor role
+
+This is the demo for user to use ESP_APIs to create a SPP acceptor.
+
+Options choose step:
+1. `idf.py menuconfig`
+2. enter menuconfig `Component config`, choose `Bluetooth`
+3. enter menu Bluetooth, choose `Classic Bluetooth" and "SPP Profile`
+4. choose your options.
+
+Then set `SPP_SHOW_MODE` as `SPP_SHOW_DATA` or `SPP_SHOW_SPEED` in code(should be same with bt_spp_initator).
+
+After the program started, bt_spp_initator will connect it and send data.

examples/bluetooth/bluedroid/classic_bt/bt_spp_acceptor/README.rst

@@ -1,20 +0,0 @@
-================= =====
-Supported Targets ESP32
-================= =====
-
-ESP-IDF BT-SPP-ACCEPTOR demo
-======================
-
-Demo of SPP acceptor role
-
-This is the demo for user to use ESP_APIs to create a SPP acceptor.
-
-Options choose step:
-    1. idf.py menuconfig.
-    2. enter menuconfig "Component config", choose "Bluetooth"
-    3. enter menu Bluetooth, choose "Classic Bluetooth" and "SPP Profile"
-    4. choose your options.
-
-Then set SPP_SHOW_MODE as SPP_SHOW_DATA or SPP_SHOW_SPEED in code(should be same with bt_spp_initator).
-
-After the program started, bt_spp_initator will connect it and send data.

examples/bluetooth/bluedroid/classic_bt/bt_spp_initiator/README.md

@@ -0,0 +1,20 @@
+| Supported Targets | ESP32 |
+| ----------------- | ----- |
+
+
+ESP-IDF BT-SPP-INITATOR demo
+======================
+
+Demo of SPP initator role
+
+This is the demo for user to use ESP_APIs to create a SPP initator.
+
+Options choose step:
+1. `idf.py menuconfig`
+2. enter menuconfig `Component config`, choose `Bluetooth`
+3. enter menu Bluetooth, choose `Classic Bluetooth` and `SPP Profile`
+4. choose your options.
+
+Then set `SPP_SHOW_MODE` as `SPP_SHOW_DATA` or `SPP_SHOW_SPEED` in code(should be same with bt_spp_acceptor).
+
+After the program started, It will connect to bt_spp_acceptor and send data.

examples/bluetooth/bluedroid/classic_bt/bt_spp_initiator/README.rst

@@ -1,20 +0,0 @@
-================= =====
-Supported Targets ESP32
-================= =====
-
-ESP-IDF BT-SPP-INITATOR demo
-======================
-
-Demo of SPP initator role
-
-This is the demo for user to use ESP_APIs to create a SPP initator.
-
-Options choose step:
-    1. idf.py menuconfig.
-    2. enter menuconfig "Component config", choose "Bluetooth"
-    3. enter menu Bluetooth, choose "Classic Bluetooth" and "SPP Profile"
-    4. choose your options.
-
-Then set SPP_SHOW_MODE as SPP_SHOW_DATA or SPP_SHOW_SPEED in code(should be same with bt_spp_acceptor).
-
-After the program started, It will connect to bt_spp_acceptor and send data.

examples/bluetooth/bluedroid/classic_bt/bt_spp_vfs_acceptor/README.md

@@ -0,0 +1,17 @@
+| Supported Targets | ESP32 |
+| ----------------- | ----- |
+
+ESP-IDF BT-SPP-VFS-ACCEPTOR demo
+======================
+
+Demo of SPP acceptor role
+
+This is the demo for user to use ESP_APIs to create a SPP acceptor.
+
+Options choose step:
+1. `idf.py menuconfig`
+2. enter menuconfig `Component config`, choose `Bluetooth`
+3. enter menu Bluetooth, choose `Classic Bluetooth` and `SPP Profile`
+4. choose your options.
+
+After the program started, bt_spp_vfs_initator will connect it and send data.

examples/bluetooth/bluedroid/classic_bt/bt_spp_vfs_acceptor/README.rst

@@ -1,18 +0,0 @@
-================= =====
-Supported Targets ESP32
-================= =====
-
-ESP-IDF BT-SPP-VFS-ACCEPTOR demo
-======================
-
-Demo of SPP acceptor role
-
-This is the demo for user to use ESP_APIs to create a SPP acceptor.
-
-Options choose step:
-    1. idf.py menuconfig.
-    2. enter menuconfig "Component config", choose "Bluetooth"
-    3. enter menu Bluetooth, choose "Classic Bluetooth" and "SPP Profile"
-    4. choose your options.
-
-After the program started, bt_spp_vfs_initator will connect it and send data.

examples/bluetooth/bluedroid/classic_bt/bt_spp_vfs_initiator/README.md

@@ -0,0 +1,17 @@
+| Supported Targets | ESP32 |
+| ----------------- | ----- |
+
+ESP-IDF BT-SPP-VFS-INITATOR demo
+======================
+
+Demo of SPP initator role
+
+This is the demo for user to use ESP_APIs to create a SPP initator.
+
+Options choose step:
+1. `idf.py menuconfig`
+2. enter menuconfig `Component config`, choose `Bluetooth`
+3. enter menu Bluetooth, choose `Classic Bluetooth` and `SPP Profile`
+4. choose your options.
+
+After the program started, It will connect to bt_spp_vfs_acceptor and send data.

examples/bluetooth/bluedroid/classic_bt/bt_spp_vfs_initiator/README.rst

@@ -1,18 +0,0 @@
-================= =====
-Supported Targets ESP32
-================= =====
-
-ESP-IDF BT-SPP-VFS-INITATOR demo
-======================
-
-Demo of SPP initator role
-
-This is the demo for user to use ESP_APIs to create a SPP initator.
-
-Options choose step:
-    1. idf.py menuconfig.
-    2. enter menuconfig "Component config", choose "Bluetooth"
-    3. enter menu Bluetooth, choose "Classic Bluetooth" and "SPP Profile"
-    4. choose your options.
-
-After the program started, It will connect to bt_spp_vfs_acceptor and send data.

examples/bluetooth/bluedroid/coex/gattc_gatts_coex/README.md

@@ -8,10 +8,10 @@ This example demonstrates the coexistence of gattc and gatts.
 
 This example creates a GATT service and starts ADV. The ADV name is `ESP_GATTS_DEMO`, then waits to be connected. At the same time, a gatt client is created, the ADV name is `ESP_GATTS_DEMO`, the device is connected, and the data is exchanged. If the device is not found within 120 seconds, the example will stop scanning.
 
-ESP-IDF also allows users to create a GATT service via an attribute table, rather than add attributes one by one. And it is recommended for users to use. For more information about this method, please refer to [gatt_server_service_table_demo](../gatt_server_service_table).
+ESP-IDF also allows users to create a GATT service via an attribute table, rather than add attributes one by one. And it is recommended for users to use. For more information about this method, please refer to [gatt_server_service_table_demo](../../ble/gatt_server_service_table).
 
-To test this example, you can run the [gatt_client_demo](../gatt_client), which can scan for and connect to this example automatically, and run [gatt_server_demo](../gatt_server), Waiting to be connected. They will start exchanging data once the GATT client has enabled the notification function of the GATT server.
+To test this example, you can run the [gatt_client_demo](../../ble/gatt_client), which can scan for and connect to this example automatically, and run [gatt_server_demo](../../ble/gatt_server), Waiting to be connected. They will start exchanging data once the GATT client has enabled the notification function of the GATT server.
 
-Please check the [tutorial](tutorial/Gatt_Server_Example_Walkthrough.md) for more information about the gatts part of this example.
+Please check the [tutorial](../../ble/gatt_server/tutorial/Gatt_Server_Example_Walkthrough.md) for more information about the gatts part of this example.
-Please check the [tutorial](tutorial/Gatt_Client_Example_Walkthrough.md) for more information about the gattc part of this example.
+Please check the [tutorial](../../ble/gatt_client/tutorial/Gatt_Client_Example_Walkthrough.md) for more information about the gattc part of this example.
 

examples/protocols/pppos_client/README.md

@@ -14,7 +14,7 @@ When PPP connection has been established, the IP packet flow from application si
 ### Hardware Required
 
 To run this example, you need an ESP32 dev board (e.g. ESP32-WROVER Kit) or ESP32 core board (e.g. ESP32-DevKitC).
-For test purpose, you also need a cellular modem module. Here we take the [SIM800L](http://www.simcom.com/product/showproduct.php?lang=en&id=277) and [BG96](https://www.quectel.com/product/bg96.htm) as an example.
+For test purpose, you also need a cellular modem module. Here we take the [SIM800L](https://www.simcom.com/product/SIM800.html) and [BG96](https://www.quectel.com/product/bg96.htm) as an example.
 You can also try other modules as long as they embedded PPP protocol.
 
 **Note:** Since SIM800L only support **2G** which will **not** work in some countries. And also keep in mind that in some other countries it will stop working soon (many remaining 2G networks will be switched off in the next 2-3 years). So you should **check with your local providers for further details** if you try this example with any 2G modules.

examples/protocols/sockets/tcp_client_multi_net/README.md

@@ -130,4 +130,4 @@ or [Ethernet documentation](https://docs.espressif.com/projects/esp-idf/en/lates
 If using Ethernet for the first time, it is recommended to start with the [Ethernet example readme](../../../ethernet/basic/README.md), which contains instructions for connecting and configuring the PHY.
 Once Ethernet example obtains IP address successfully, proceed to this example.
 
-* When connecting using Wi-Fi, please refer to the WiFi examples in [examples/wifi/getting_started/](../wifi/getting_started).
+* When connecting using Wi-Fi, please refer to the WiFi examples in [examples/wifi/getting_started/](../../../wifi/getting_started).

examples/system/ota/otatool/README.md

@@ -1,6 +1,6 @@
 # OTA Tool Example
 
-This example demonstrates common operations the OTA tool [otatool.py](../../../components/app_update/otatool.py) allows the user to perform:
+This example demonstrates common operations the OTA tool [otatool.py](../../../../components/app_update/otatool.py) allows the user to perform:
 
 - reading, writing and erasing OTA partitions,
 - switching boot partitions, and

tools/ci/check_readme_links.py

@@ -0,0 +1,161 @@
+#!/usr/bin/env python
+#
+# Checks that all links in the readme markdown files are valid
+#
+# Copyright 2020 Espressif Systems (Shanghai) PTE LTD
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import os
+import re
+import os.path
+import urllib.request
+import urllib.error
+import concurrent.futures
+import argparse
+from pathlib import Path
+from collections import namedtuple, defaultdict
+
+EXCLUDE_DOCS_LIST = ['examples/peripherals/secure_element/atecc608_ecdsa/components/esp-cryptoauthlib/cryptoauthlib/**']
+
+# The apple apps links are not accessible from the company network for some reason
+EXCLUDE_URL_LIST = ['https://apps.apple.com/in/app/esp-ble-provisioning/id1473590141', 'https://apps.apple.com/in/app/esp-softap-provisioning/id1474040630']
+
+Link = namedtuple('Link', ['file', 'url'])
+
+
+class ReadmeLinkError(Exception):
+    def __init__(self, file, url):
+        self.file = file
+        self.url = url
+
+
+class RelativeLinkError(ReadmeLinkError):
+    def __str__(self):
+        return "Relative link error, file - {} not found, linked from {}".format(self.url, self.file)
+
+
+class UrlLinkError(ReadmeLinkError):
+    def __init__(self, file, url, error_code):
+        self.error_code = error_code
+        super().__init__(file, url)
+
+    def __str__(self):
+        files = [str(f) for f in self.file]
+        return "URL error, url - {} in files - {} is not accessible, request returned {}".format(self.url, ", ".join(files), self.error_code)
+
+
+# we do not want a failed test just due to bad network conditions, for non 404 errors we simply print a warning
+def check_url(url, files, timeout):
+    try:
+        with urllib.request.urlopen(url, timeout=timeout):
+            return
+    except urllib.error.HTTPError as e:
+        if e.code == 404:
+            raise UrlLinkError(files, url, str(e))
+        else:
+            print("Unable to access {}, err = {}".format(url, str(e)))
+    except Exception as e:
+        print("Unable to access {}, err = {}".format(url, str(e)))
+
+
+def check_web_links(web_links):
+
+    with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
+        errors = []
+        future_to_url = {executor.submit(check_url, url, files, timeout=30): (url, files) for url, files in web_links.items()}
+        for future in concurrent.futures.as_completed(future_to_url):
+            try:
+                future.result()
+            except UrlLinkError as e:
+                errors.append(e)
+
+        return errors
+
+
+def check_file_links(file_links):
+    errors = []
+
+    for link in file_links:
+        link_path = link.file.parent / link.url
+
+        if not Path.exists(link_path):
+            errors.append(RelativeLinkError(link.file, link.url))
+
+    print("Found {} errors with relative links".format(len(errors)))
+    return errors
+
+
+def get_md_links(folder):
+    MD_LINK_RE = r"\[.+?\]\((.+?)(#.+)?\)"
+
+    idf_path = Path(os.getenv('IDF_PATH'))
+    links = []
+
+    for path in (idf_path / folder).rglob('*.md'):
+        if any([path.relative_to(idf_path).match(exclude_doc) for exclude_doc in EXCLUDE_DOCS_LIST]):
+            print("{} - excluded".format(path))
+            continue
+
+        with path.open(encoding='utf8') as f:
+            content = f.read()
+
+        for url in re.findall(MD_LINK_RE, content):
+            link = Link(path, url[0].lstrip())
+            # Ignore "local" links
+            if not link.url.startswith('#'):
+                links.append(link)
+
+    return links
+
+
+def check_readme_links(args):
+
+    links = get_md_links('examples')
+    print("Found {} links".format(len(links)))
+
+    errors = []
+
+    web_links = defaultdict(list)
+    file_links = []
+
+    # Sort links into file and web links
+    for link in links:
+        if link.url.startswith('http'):
+                web_links[link.url].append(link.file)
+        else:
+            file_links.append(link)
+
+    for url in EXCLUDE_URL_LIST:
+        del web_links[url]
+
+    errors.extend(check_file_links(file_links))
+
+    if not args.skip_weburl:
+        errors.extend(check_web_links(web_links))
+
+    print("Found {} errors:".format(len(errors)))
+    for e in errors:
+        print(e)
+    if errors:
+        raise e
+
+
+if __name__ == '__main__':
+
+    parser = argparse.ArgumentParser(description='check_readme_links.py: Checks for dead links in example READMEs', prog='check_readme_links.py')
+    parser.add_argument("--skip-weburl", "-w", action='store_true', help="Skip checking of web URLs, only check links to local files")
+    args = parser.parse_args()
+
+    check_readme_links(args)

tools/ci/config/host-test.yml

@@ -312,3 +312,4 @@ test_mkdfu:
   script:
     - cd ${IDF_PATH}/tools/test_mkdfu
     - ${IDF_PATH}/tools/ci/multirun_with_pyenv.sh ./test_mkdfu.py
+

tools/ci/config/pre_check.yml

@@ -221,3 +221,12 @@ scan_tests:
     - python $CI_SCAN_TESTS_PY example_test -b make $EXAMPLE_TEST_DIR --exclude examples/build_system/idf_as_lib -c $TEST_CONFIG_FILE -o $EXAMPLE_TEST_OUTPUT_DIR
     - python $CI_SCAN_TESTS_PY example_test -b cmake $EXAMPLE_TEST_DIR --exclude examples/build_system/idf_as_lib -c $TEST_CONFIG_FILE -o $EXAMPLE_TEST_OUTPUT_DIR
     - python $CI_SCAN_TESTS_PY test_apps $TEST_APPS_TEST_DIR -c $TEST_CONFIG_FILE -o $TEST_APPS_OUTPUT_DIR
+
+check_readme_links:
+  extends: .check_job_template
+  tags: [ "amd64", "deploy", "internet" ]
+  allow_failure: true
+  variables:
+    PYTHON_VER: 3
+  script:
+    - python ${IDF_PATH}/tools/ci/check_readme_links.py

tools/ci/executable-list.txt

@@ -44,6 +44,7 @@ tools/ci/check_deprecated_kconfigs.py
 tools/ci/check_examples_cmake_make.sh
 tools/ci/check_examples_rom_header.sh
 tools/ci/check_idf_version.sh
+tools/ci/check_readme_links.py
 tools/ci/check_rom_apis.sh
 tools/ci/check_ut_cmake_make.sh
 tools/ci/checkout_project_ref.py