Merge branch 'feature/component_whole_archive' into 'master'

build system: add WHOLE_ARCHIVE component property, add link-time registration example

Closes IDFGH-7050 and IDFGH-5997

See merge request espressif/esp-idf!17642

docs/en/api-guides/build-system.rst

@@ -1342,7 +1342,8 @@ Set a specified *component*'s :ref:`component property<cmake-component-propertie
                          [EMBED_FILES file1 file2 ...]
                          [EMBED_TXTFILES file1 file2 ...]
                          [KCONFIG kconfig]
-                         [KCONFIG_PROJBUILD kconfig_projbuild])
+                         [KCONFIG_PROJBUILD kconfig_projbuild]
+                         [WHOLE_ARCHIVE])
 
 Register a component to the build system. Much like the ``project()`` CMake command, this should be called from the component's CMakeLists.txt directly (not through a function or macro) and is recommended to be called before any other command. Here are some guidelines on what commands can **not** be called before ``idf_component_register``:
 
@@ -1364,6 +1365,7 @@ The arguments for ``idf_component_register`` include:
   - REQUIRED_IDF_TARGETS - specify the only target the component supports
   - KCONFIG - override the default Kconfig file
   - KCONFIG_PROJBUILD - override the default Kconfig.projbuild file
+  - WHOLE_ARCHIVE - if specified, the component library is surrounded by ``-Wl,--whole-archive``, ``-Wl,--no-whole-archive`` when linked. This has the same effect as setting ``WHOLE_ARCHIVE`` component property.
 
 The following are used for :ref:`embedding data into the component<cmake_embed_data>`, and is considered as source files when determining if a component is config-only. This means that even if the component does not specify source files, a static library is still created internally for the component if it specifies either:
 
@@ -1399,6 +1401,7 @@ These are properties that describe a component. Values of component properties c
 - REQUIRED_IDF_TARGETS - list of targets the component supports; set from ``idf_component_register`` EMBED_TXTFILES argument
 - REQUIRES - list of public component dependencies; set from ``idf_component_register`` REQUIRES argument
 - SRCS - list of component source files; set from SRCS or SRC_DIRS/EXCLUDE_SRCS argument of ``idf_component_register``
+- WHOLE_ARCHIVE - if this property is set to ``TRUE`` (or any boolean "true" CMake value: 1, ``ON``, ``YES``, ``Y``), the component library is surrounded by ``-Wl,--whole-archive``, ``-Wl,--no-whole-archive`` when linked. This can be used to force the linker to include every object file into the executable, even if the object file doesn't resolve any references from the rest of the application. This is commonly used when a component contains plugins or modules which rely on link-time registration. This property is ``FALSE`` by default. It can be set to ``TRUE`` from the component CMakeLists.txt file.
 
 .. _cmake-file-globbing:
 

examples/build_system/cmake/plugins/CMakeLists.txt

@@ -0,0 +1,6 @@
+# The following three lines of boilerplate have to be in your project's
+# CMakeLists in this exact order for cmake to work correctly
+cmake_minimum_required(VERSION 3.5)
+
+include($ENV{IDF_PATH}/tools/cmake/project.cmake)
+project(plugins)

examples/build_system/cmake/plugins/README.md

@@ -0,0 +1,121 @@
+# Link Time Plugins Registration
+
+(See the README.md file in the upper level 'examples' directory for more information about examples.)
+
+This example demonstrates features of ESP-IDF build system related to link time registration of plugins. Link time registration of plugins is often used to add multiple implementations of a certain feature without having to make the application aware of all these implementations. With this approach, adding a new implementation is often as simple as adding a new source file or a new component. Aside from plugins, link time registration may be used for other purposes, such as automatic registration of test cases.
+
+# Overview of link time registration
+
+When using link time registration, there are typically two challenges: getting the plugin code linked into the application and enumerating the plugins at run time. The following sections explain these problems and the solutions to them.
+
+## Ensuring that the plugin code is included into the executable
+
+When GNU linker (ld) links a static library, it considers each object file in the library separately. The object file is ignored if it doesn't resolve any unresolved references known to the linker at that moment. With link-time plugin registration this is typically the case — the application doesn't explicitly reference any plugins, so the linker sees no reason to include the respective object files into the executable.
+
+Aside from adding an explicit reference from the application to the plugin object file, there are two common ways to resolve this issue:
+
+1. Link the object file of the plugin directly to the executable, and not via a static library.
+2. Instruct the linker to include every object file of a library into the executable, even those which don't resolve any references from the rest of the application. For GNU ld this can be achieved by surrounding the library on the linker command line with `-Wl,--whole-archive` and `-Wl,--no-whole-archive` flags.
+
+ESP-IDF build system implements the 2nd approach by providing a `WHOLE_ARCHIVE` component property. It can be set in component CMakeLists.txt in two ways. One is to add `WHOLE_ARCHIVE` option when calling `idf_component_register`:
+
+```cmake
+idf_component_register(SRCS file.c
+                       WHOLE_ARCHIVE)
+```
+
+Another is to call `idf_component_set_property` after registering the component:
+```cmake
+idf_component_set_property(${COMPONENT_NAME} WHOLE_ARCHIVE TRUE)
+```
+
+This will instruct the build system to surround the component library with whole-archive flags on the linker command line, ensuring that all object files from the library get included into the final application.
+
+Note that the linker also performs "garbage collection" at the end of the linking process, eliminating all functions and global variables which are not referenced anywhere. This is addressed in the current example using `__attribute__((constructor))` function attribute (for dynamic registration) and `KEEP()` linker fragment flag (for static registration).
+
+## Registering and enumerating the plugins
+
+To make use of the plugins linked into the application, the application must somehow enumerate them. There are 2 common ways to register and enumerate the plugins: dynamic and static. This example demonstrates both approaches.
+
+### Dynamic registration (or self-registration)
+
+With this approach, each plugin module has a function with `__attribute__((constructor))` attribute (in C) or a static global object with a non-trivial constructor (in C++). Startup code calls all constructor functions before the application entry point (`app_main`) is executed. Plugin constructor functions then register themselves by calling a function defined in the application. As an example, this registration function can, add structures describing the plugins into a linked list.
+
+### Static registration
+
+This approach relies on plugin description structures being collected into an array at link time.
+
+For each plugin, a structure describing the plugin (or a pointer to it) is placed into some special input section using `__attribute((section(".plugins_desc")))`. Using the linker script generator features in ESP-IDF, all entries from this input section can be gathered into a continuous array, surrounded by some symbols (e.g. `_plugins_array_start`, `_plugins_array_end`). At run time, the application casts the `&_plugins_array_start` pointer to the plugin description structure pointer and then iterates over structures collected from all plugins.
+
+## Example code overview
+
+This example contains 4 components:
+
+* `main` — Only calls two sample functions defined in `plugins` component.
+* `plugins` — The main part of the plugin system.
+
+   For dynamic registration, it provides an API which plugin components call to register themselves (`example_plugin_register`).
+
+   It also provides two sample functions for the application:
+   - `example_plugins_list`: prints the list of registered plugins. This function demonstrates static registration.
+   - `example_plugins_greet`: calls a function defined by each plugin with a given argument. This function demonstrates working with dynamically registered plugins.
+* `plugin_hello` and `plugin_nihao` — two almost identical components, each provides one plugin.
+
+  Note that multiple plugins may also be defined in the same component.
+
+Please refer to the comments in the example code for a more detailed description.
+
+## How to use example
+
+### Hardware Required
+
+This example runs on any ESP development board, no special hardware is required.
+
+### Build and Flash
+
+Build the project and flash it to the board, then run monitor tool to view serial output:
+
+```
+idf.py -p PORT flash monitor
+```
+
+(Replace PORT with the name of the serial port to use.)
+
+(To exit the serial monitor, type `Ctrl-]`.)
+
+See the Getting Started Guide for full steps to configure and use ESP-IDF to build projects.
+
+## Example Output
+
+```
+Nihao plugin performing self-registration...
+Successfully registered plugin 'Nihao'
+Hello plugin performing self-registration...
+Successfully registered plugin 'Hello'
+I (325) cpu_start: Starting scheduler on PRO CPU.
+I (0) cpu_start: Starting scheduler on APP CPU.
+List of plugins:
+- Plugin 'Hello', function greet=0x400d4f40
+0x400d4f40: plugin_hello_greet at /home/user/esp/esp-idf/examples/build_system/cmake/plugins/build/../components/plugin_hello/plugin_hello.c:14
+
+- Plugin 'Nihao', function greet=0x400d4f70
+0x400d4f70: plugin_nihao_greet at /home/user/esp/esp-idf/examples/build_system/cmake/plugins/build/../components/plugin_nihao/plugin_nihao.c:14
+
+Calling greet function of plugin 'Hello'...
+Hello, World!
+Done with greet function of plugin 'Hello'.
+Calling greet function of plugin 'Nihao'...
+你好 World!
+Done with greet function of plugin 'Nihao'.
+```
+
+## Troubleshooting
+
+When implementing the approaches described in this example, the following issues may occur:
+
+* Plugin self-registration function (constructor function) is never called. To troubleshoot this, check the application .map file — it is generated in the `build` directory of the project. Look for the object file where the constructor function is defined. If the object file and the constructor function are missing, it means that the object file was discarded. Double-check that the `WHOLE_ARCHIVE` property of the component is set correctly. Verify that on the linker command line, the component library is surrounded by `-Wl,--whole-archive`, `-Wl,--no-whole-archive`. To see the linker command line, build the project with verbose (-v) flag.
+* With static registration, the plugin description structure is missing from the link-time array. Same as in the case above, start by examining the map file.
+  - If the plugin object file is missing from the map file, double-check that the `WHOLE_ARCHIVE` property of the component is set correctly (see the instructions above).
+  - If the plugin object file is present, but the plugin description structure is missing, check that the linker fragment rule and `__attribute((section(...)))` use the same section name. Check that the linker fragment rule uses `KEEP()` flag.
+  - If the plugin description structure is in the map file but is not located inside the link-time array (is located in some other section), check the generated linker script found in the build directory (`build/esp-idf/esp_system/ld/sections.ld`). Check that the rules for placing the plugin description structure have correct precedence with respect to other rules in the linker script.
+

examples/build_system/cmake/plugins/components/plugin_hello/CMakeLists.txt

@@ -0,0 +1,3 @@
+idf_component_register(SRCS plugin_hello.c
+                       PRIV_REQUIRES plugins
+                       WHOLE_ARCHIVE)

examples/build_system/cmake/plugins/components/plugin_hello/plugin_hello.c

@@ -0,0 +1,45 @@
+/*
+ * SPDX-FileCopyrightText: 2022 Espressif Systems (Shanghai) CO LTD
+ *
+ * SPDX-License-Identifier: Unlicense OR CC0-1.0
+ */
+#include <stdio.h>
+#include <string.h>
+#include "plugins_api.h"
+
+/**
+ * This is an example function implemented by the plugin.
+ */
+static void plugin_hello_greet(const char* arg)
+{
+    if (arg == NULL) {
+        return;
+    }
+    printf("Hello, %s!\n", arg);
+}
+
+/* The code below demonstates both static and dynamic registration approaches. */
+
+/**
+ * Static registration of this plugin can be achieved by defining the plugin description
+ * structure and placing it into .plugins_desc section.
+ * The name of the section and its placement is determined by linker.lf file in 'plugins' component.
+ */
+static const example_plugin_desc_t __attribute__((section(".plugins_desc"),used)) PLUGIN = {
+    .name = "Hello",
+    .greet = &plugin_hello_greet
+};
+
+/**
+ * Dynamic registration of this plugin can be achieved by calling plugin registration function
+ * ('example_plugin_register') from a "constructor" function. Constructor function is called automatically
+ * during application startup.
+ */
+static void __attribute__((constructor)) plugin_hello_self_register(void)
+{
+    printf("Hello plugin performing self-registration...\n");
+    example_plugin_register(&(example_plugin_desc_t){
+        .name = "Hello",
+        .greet = &plugin_hello_greet
+    });
+}

examples/build_system/cmake/plugins/components/plugin_nihao/CMakeLists.txt

@@ -0,0 +1,5 @@
+idf_component_register(SRCS plugin_nihao.c
+                       PRIV_REQUIRES plugins)
+
+# This is equivalent to adding WHOLE_ARCHIVE option to the idf_component_register call above:
+idf_component_set_property(${COMPONENT_NAME} WHOLE_ARCHIVE TRUE)

examples/build_system/cmake/plugins/components/plugin_nihao/plugin_nihao.c

@@ -0,0 +1,45 @@
+/*
+ * SPDX-FileCopyrightText: 2022 Espressif Systems (Shanghai) CO LTD
+ *
+ * SPDX-License-Identifier: Unlicense OR CC0-1.0
+ */
+#include <stdio.h>
+#include <string.h>
+#include "plugins_api.h"
+
+/**
+ * This is an example function implemented by the plugin.
+ */
+static void plugin_nihao_greet(const char* arg)
+{
+    if (arg == NULL) {
+        return;
+    }
+    printf("你好 %s!\n", arg);
+}
+
+/* The code below demonstates both static and dynamic registration approaches. */
+
+/**
+ * Static registration of this plugin can be achieved by defining the plugin description
+ * structure and placing it into .plugins_desc section.
+ * The name of the section and its placement is determined by linker.lf file in 'plugins' component.
+ */
+static const example_plugin_desc_t __attribute__((section(".plugins_desc"),used)) PLUGIN = {
+    .name = "Nihao",
+    .greet = &plugin_nihao_greet
+};
+
+/**
+ * Dynamic registration of this plugin can be achieved by calling plugin registration function
+ * ('example_plugin_register') from a "constructor" function. Constructor function is called automatically
+ * during application startup.
+ */
+static void __attribute__((constructor)) plugin_nihao_self_register(void)
+{
+    printf("Nihao plugin performing self-registration...\n");
+    example_plugin_register(&(example_plugin_desc_t){
+        .name = "Nihao",
+        .greet = &plugin_nihao_greet
+    });
+}

examples/build_system/cmake/plugins/components/plugins/CMakeLists.txt

@@ -0,0 +1,3 @@
+idf_component_register(SRCS plugins.c
+                       INCLUDE_DIRS include
+                       LDFRAGMENTS linker.lf)

examples/build_system/cmake/plugins/components/plugins/include/plugins_api.h

@@ -0,0 +1,48 @@
+/*
+ * SPDX-FileCopyrightText: 2022 Espressif Systems (Shanghai) CO LTD
+ *
+ * SPDX-License-Identifier: Unlicense OR CC0-1.0
+ */
+#pragma once
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/* This stucture describes the plugin to the rest of the application */
+typedef struct {
+    /* A pointer to the plugin name */
+    const char* name;
+
+    /* A function which the plugin provides to the application.
+     * In this example, this function prints something to the console
+     * depending on the value of the argument 'arg'.
+     */
+    void (*greet)(const char* arg);
+} example_plugin_desc_t;
+
+/**
+ * @brief Register the plugin with the application
+ * This function is called from each plugin's "constructor" function.
+ * It adds the plugin to the list.
+ * @param plugin_desc  Pointer to the structure which describes the given plugin.
+ */
+void example_plugin_register(const example_plugin_desc_t* plugin_desc);
+
+/**
+ * @brief Print the list of registered plugins to the console.
+ * This function is called from the application.
+ */
+void example_plugins_list(void);
+
+/**
+ * @brief Invoke 'greet' function of each registered plugin with the given argument.
+ * This function is called from the application.
+ * @param arg  argument to pass to plugins' greet functions.
+ */
+void example_plugins_greet(const char* arg);
+
+
+#ifdef __cplusplus
+}
+#endif

examples/build_system/cmake/plugins/components/plugins/linker.lf

@@ -0,0 +1,13 @@
+[sections:plugins_desc]
+entries:
+    .plugins_desc
+
+[scheme:plugins_desc_default]
+entries:
+    plugins_desc -> flash_rodata
+
+[mapping:plugins_desc]
+archive: *
+entries:
+    * (plugins_desc_default);
+        plugins_desc -> flash_rodata KEEP() SORT(name) SURROUND(plugins_array)

examples/build_system/cmake/plugins/components/plugins/plugins.c

@@ -0,0 +1,71 @@
+/*
+ * SPDX-FileCopyrightText: 2022 Espressif Systems (Shanghai) CO LTD
+ *
+ * SPDX-License-Identifier: Unlicense OR CC0-1.0
+ */
+#include <stdio.h>
+#include <string.h>
+#include <sys/queue.h>
+#include "plugins_api.h"
+
+/**
+ * Demonstration of dynamic registration (self-registration):
+ *
+ * - example_plugin_register function is called from "constructor" functions of each plugin.
+ *   Information about the plugin is passed inside 'example_plugin_desc_t' structure.
+ *   This function adds each plugin description into linked list (s_plugins_list).
+ *
+ * - example_plugins_greet function iterates over the linked list.
+ */
+
+struct plugin_record {
+    example_plugin_desc_t plugin_desc;
+    LIST_ENTRY(plugin_record) list_entry;
+};
+
+static LIST_HEAD(plugins_list, plugin_record) s_plugins_list = LIST_HEAD_INITIALIZER(s_plugins_list);
+
+void example_plugin_register(const example_plugin_desc_t* plugin_desc)
+{
+    struct plugin_record *record = (struct plugin_record *) malloc(sizeof(struct plugin_record));
+    if (record == NULL) {
+        abort();
+    }
+    memcpy(&record->plugin_desc, plugin_desc, sizeof(*plugin_desc));
+
+    struct plugin_record *head = LIST_FIRST(&s_plugins_list);
+    if (head == NULL) {
+        LIST_INSERT_HEAD(&s_plugins_list, record, list_entry);
+    } else {
+        LIST_INSERT_BEFORE(head, record, list_entry);
+    }
+    printf("Successfully registered plugin '%s'\n", plugin_desc->name);
+}
+
+void example_plugins_greet(const char* arg)
+{
+    struct plugin_record *it;
+    LIST_FOREACH(it, &s_plugins_list, list_entry) {
+        printf("Calling greet function of plugin '%s'...\n", it->plugin_desc.name);
+        (*it->plugin_desc.greet)(arg);
+        printf("Done with greet function of plugin '%s'.\n", it->plugin_desc.name);
+    }
+}
+
+/**
+ * Demonstration of static registration.
+ * Symbols '_plugins_array_start' and '_plugins_array_end' mark the beginning and end
+ * of the array where 'example_plugin_desc_t' structures are placed by the linker.
+ * The names of these variables are determined by linker.lf in 'plugins' component,
+ * look for 'SURROUND(plugins_array)'.
+ */
+
+void example_plugins_list(void)
+{
+    printf("List of plugins:\n");
+    extern const example_plugin_desc_t _plugins_array_start;
+    extern const example_plugin_desc_t _plugins_array_end;
+    for (const example_plugin_desc_t* it = &_plugins_array_start; it != &_plugins_array_end; ++it) {
+        printf("- Plugin '%s', function greet=%p\n", it->name, it->greet);
+    }
+}

examples/build_system/cmake/plugins/main/CMakeLists.txt

@@ -0,0 +1,3 @@
+idf_component_register(SRCS "plugins_example_main.c"
+                    INCLUDE_DIRS "."
+                    PRIV_REQUIRES plugins)

examples/build_system/cmake/plugins/main/plugins_example_main.c

@@ -0,0 +1,13 @@
+/*
+ * SPDX-FileCopyrightText: 2022 Espressif Systems (Shanghai) CO LTD
+ *
+ * SPDX-License-Identifier: Unlicense OR CC0-1.0
+ */
+#include <stdio.h>
+#include "plugins_api.h"
+
+void app_main(void)
+{
+    example_plugins_list();
+    example_plugins_greet("World");
+}

examples/build_system/cmake/plugins/pytest_plugins.py

@@ -0,0 +1,32 @@
+# SPDX-FileCopyrightText: 2022 Espressif Systems (Shanghai) CO LTD
+# SPDX-License-Identifier: CC0-1.0
+
+import textwrap
+
+import pytest
+from pytest_embedded import Dut
+
+
+@pytest.mark.esp32
+@pytest.mark.esp32c3
+@pytest.mark.generic
+def test_plugins(dut: Dut) -> None:
+    log_text = textwrap.dedent(r"""
+        Nihao plugin performing self-registration...
+        Successfully registered plugin 'Nihao'
+        Hello plugin performing self-registration...
+        Successfully registered plugin 'Hello'
+        cpu_start: Starting scheduler
+        List of plugins:
+        - Plugin 'Hello'
+        - Plugin 'Nihao'
+        Calling greet function of plugin 'Hello'...
+        Hello, World!
+        Done with greet function of plugin 'Hello'.
+        Calling greet function of plugin 'Nihao'...
+        你好 World!
+        Done with greet function of plugin 'Nihao'.
+    """).strip('\n')
+
+    for line in log_text.split('\n'):
+        dut.expect_exact(line.encode('utf-8'))

tools/cmake/component.cmake

@@ -88,6 +88,8 @@ macro(__component_set_properties)
     __component_set_property(${component_target} EMBED_FILES "${__EMBED_FILES}")
     __component_set_property(${component_target} EMBED_TXTFILES "${__EMBED_TXTFILES}")
     __component_set_property(${component_target} REQUIRED_IDF_TARGETS "${__REQUIRED_IDF_TARGETS}")
+
+    __component_set_property(${component_target} WHOLE_ARCHIVE ${__WHOLE_ARCHIVE})
 endmacro()
 
 #
@@ -417,8 +419,9 @@ endfunction()
 # @param[in, optional] EMBED_TXTFILES (multivalue) list of text files to embed with the component
 # @param[in, optional] KCONFIG (single value) override the default Kconfig
 # @param[in, optional] KCONFIG_PROJBUILD (single value) override the default Kconfig
+# @param[in, optional] WHOLE_ARCHIVE (option) link the component as --whole-archive
 function(idf_component_register)
-    set(options)
+    set(options WHOLE_ARCHIVE)
     set(single_value KCONFIG KCONFIG_PROJBUILD)
     set(multi_value SRCS SRC_DIRS EXCLUDE_SRCS
                     INCLUDE_DIRS PRIV_INCLUDE_DIRS LDFRAGMENTS REQUIRES

tools/cmake/project.cmake

@@ -484,7 +484,18 @@ macro(project project_name)
     if(test_components)
         list(REMOVE_ITEM build_components ${test_components})
     endif()
-    target_link_libraries(${project_elf} ${build_components})
+
+    foreach(build_component ${build_components})
+        __component_get_target(build_component_target ${build_component})
+        __component_get_property(whole_archive ${build_component_target} WHOLE_ARCHIVE)
+        if(whole_archive)
+            message(STATUS "Component ${build_component} will be linked with -Wl,--whole-archive")
+            target_link_libraries(${project_elf} "-Wl,--whole-archive" ${build_component} "-Wl,--no-whole-archive")
+        else()
+            target_link_libraries(${project_elf} ${build_component})
+        endif()
+    endforeach()
+
 
     if(CMAKE_C_COMPILER_ID STREQUAL "GNU")
         set(mapfile "${CMAKE_BINARY_DIR}/${CMAKE_PROJECT_NAME}.map")

tools/cmake/scripts/component_get_requirements.cmake

@@ -66,7 +66,7 @@ macro(idf_component_mock)
 endmacro()
 
 macro(idf_component_register)
-    set(options)
+    set(options WHOLE_ARCHIVE)
     set(single_value KCONFIG KCONFIG_PROJBUILD)
     set(multi_value SRCS SRC_DIRS EXCLUDE_SRCS
                     INCLUDE_DIRS PRIV_INCLUDE_DIRS LDFRAGMENTS REQUIRES