docs(modem): Update docs and migration manual

Document the development mode

docs/esp_modem/en/README.rst

@@ -176,6 +176,41 @@ Component Kconfig
 Compile-time configuration is provided using menuconfig. Please check
 the description for the CMUX mode configuration options.
 
+Development Mode
+~~~~~~~~~~~~~~~~
+
+ESP-MODEM supports two different modes for handling AT command definitions:
+
+**Production Mode (Default)**
+    Uses pre-generated headers and sources with common AT commands from the ``command/`` directory.
+    This mode provides better IDE navigation and code completion, making it ideal for:
+
+    - Application development using existing AT commands
+    - Better code navigation and IntelliSense support
+    - Faster compilation times
+    - Stable, tested command implementations
+
+**Development Mode**
+    Uses in-place macro expansion with AT commands from the ``generate/`` directory.
+    This mode is designed for ESP-MODEM library developers who need to:
+
+    - Modify or add new AT command definitions in ``esp_modem_command_declare.inc``
+    - Work directly with the source definitions of AT commands
+    - Debug command implementations
+    - Contribute to the core ESP-MODEM library development
+
+.. note::
+    For adding support for new modem modules, you typically don't need development mode.
+    Instead, create a custom module class inheriting from ``GenericModule`` and add your
+    commands directly in your project's headers/sources. See the ``pppos_client`` example
+    for a demonstration of this approach.
+
+To enable development mode, set ``CONFIG_ESP_MODEM_ENABLE_DEVELOPMENT_MODE=y`` in your project configuration.
+
+.. note::
+    Development mode requires the C preprocessor to expand command definitions at compile time,
+    which may result in longer compilation times and larger binary sizes compared to production mode.
+
 Runtime configuration
 ~~~~~~~~~~~~~~~~~~~~~
 

docs/esp_modem/en/customization.rst

@@ -0,0 +1,202 @@
+Customization
+=============
+
+This chapter covers how to customize ESP-MODEM for your specific requirements by creating custom modules and adding new commands.
+
+Custom Module Development
+-------------------------
+
+For most customization needs, you don't need development mode. Instead, you can create custom modules that inherit from existing ESP-MODEM classes.
+
+Creating Custom Modules
+~~~~~~~~~~~~~~~~~~~~~~~
+
+The recommended approach for adding support for new modem modules or custom commands is to create a custom module class. This approach:
+
+- **Doesn't require development mode**
+- **Keeps your changes separate** from the core library
+- **Allows easy updates** of the ESP-MODEM library
+- **Provides full flexibility** for custom commands
+
+Basic Custom Module Example
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Here's a simple example of creating a custom module:
+
+.. code-block:: cpp
+
+    #include "cxx_include/esp_modem_api.hpp"
+    #include "cxx_include/esp_modem_command_library_utils.hpp"
+
+    class MyCustomModule: public GenericModule {
+        using GenericModule::GenericModule;
+
+    public:
+        // Add a new command
+        command_result get_custom_info(std::string &info) {
+            return esp_modem::dce_commands::generic_get_string(
+                dte.get(), "AT+CUSTOM?\r", info);
+        }
+
+        // Override an existing command
+        command_result get_signal_quality(int &rssi, int &ber) override {
+            // Custom implementation
+            return esp_modem::dce_commands::generic_get_string(
+                dte.get(), "AT+CSQ\r", rssi, ber);
+        }
+    };
+
+Using Custom Modules with C++ API
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+With the C++ API, you can use your custom module directly:
+
+.. code-block:: cpp
+
+    // Create DCE with custom module
+    auto dce = dce_factory::Factory::create_unique_dce_from<MyCustomModule>(
+        dce_config, std::move(dte), netif);
+
+    // Use custom commands
+    std::string info;
+    auto result = dce->get_custom_info(info);
+
+Using Custom Modules with C API
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To use custom modules with the C API, you need to:
+
+1. **Enable custom module support** in Kconfig:
+
+   .. code-block:: bash
+
+       idf.py menuconfig
+       # Navigate to: Component config → ESP-MODEM
+       # Enable: "Add support for custom module in C-API"
+
+2. **Create a custom module header** (e.g., ``custom_module.hpp``) in your main component
+
+3. **Implement the required functions**:
+
+   .. code-block:: cpp
+
+       // Create custom DCE function
+       DCE *esp_modem_create_custom_dce(
+           const esp_modem_dce_config_t *dce_config,
+           std::shared_ptr<DTE> dte,
+           esp_netif_t *netif) {
+           return dce_factory::Factory::create_unique_dce_from<MyCustomModule, DCE *>(
+               dce_config, std::move(dte), netif);
+       }
+
+       // Add C API wrappers for custom commands
+       extern "C" esp_err_t esp_modem_get_custom_info(esp_modem_dce_t *dce_wrap, char *info) {
+           if (dce_wrap == nullptr || dce_wrap->dce == nullptr) {
+               return ESP_ERR_INVALID_ARG;
+           }
+           std::string info_str{CONFIG_ESP_MODEM_C_API_STR_MAX};
+           auto ret = command_response_to_esp_err(
+               static_cast<MyCustomModule *>(dce_wrap->dce->get_module())->get_custom_info(info_str));
+           if (ret == ESP_OK && !info_str.empty()) {
+               strlcpy(info, info_str.c_str(), CONFIG_ESP_MODEM_C_API_STR_MAX);
+           }
+           return ret;
+       }
+
+4. **Use the custom commands** in your C code:
+
+   .. code-block:: c
+
+       char info[128];
+       esp_err_t ret = esp_modem_get_custom_info(dce, info);
+
+Complete Example
+~~~~~~~~~~~~~~~~
+
+See the ``examples/pppos_client`` example for a complete demonstration of custom module development. This example shows:
+
+- Creating a custom module that inherits from ``GenericModule``
+- Adding new commands (``get_time()``)
+- Overriding existing commands (``get_signal_quality()``)
+- Integration with both C++ and C APIs
+
+Available Base Classes
+----------------------
+
+You can inherit from several base classes depending on your needs:
+
+**GenericModule**
+    The most general implementation of a common modem. Use this when:
+    - Your modem supports most standard AT commands
+    - You need to add a few custom commands
+    - You want to override some existing commands
+
+**Specific Module Classes**
+    Inherit from existing module classes (e.g., ``SIM800``, ``BG96``, ``SIM7600``) when:
+    - Your modem is very similar to an existing one
+    - You only need minor modifications
+    - You want to leverage existing device-specific optimizations
+
+**ModuleIf**
+    Use this minimal interface when:
+    - You only need basic AT command functionality
+    - You don't need network interface features
+    - You want to implement a custom DTE without DCE
+
+Command Utilities
+-----------------
+
+ESP-MODEM provides utility functions to help implement custom commands:
+
+**Generic Command Helpers**
+    - ``generic_get_string()`` - Parse string responses
+    - ``generic_get_int()`` - Parse integer responses
+    - ``generic_set_string()`` - Send string commands
+    - ``generic_set_int()`` - Send integer commands
+
+**Response Parsing**
+    - ``get_number_from_string()`` - Extract numbers from responses
+    - ``get_string_from_response()`` - Extract strings from responses
+    - ``get_urc()`` - Handle unsolicited result codes
+
+Example Usage:
+.. code-block:: cpp
+
+    // Get a string value
+    command_result get_imei(std::string &imei) {
+        return esp_modem::dce_commands::generic_get_string(
+            dte.get(), "AT+CGSN\r", imei);
+    }
+
+    // Get an integer value
+    command_result get_signal_strength(int &rssi) {
+        return esp_modem::dce_commands::generic_get_int(
+            dte.get(), "AT+CSQ\r", rssi);
+    }
+
+Best Practices
+--------------
+
+**For Application Developers:**
+- Use production mode for better IDE support and faster builds
+- Create custom modules for new modem support
+- Inherit from ``GenericModule`` or other existing modules
+- Keep customizations in your project, not in the ESP-MODEM library
+
+**Module Design:**
+- Choose the most appropriate base class for your needs
+- Override only the commands you need to modify
+- Use the provided utility functions for common operations
+- Follow the existing command naming conventions
+
+**Testing:**
+- Test your custom module with real hardware
+- Verify compatibility with existing ESP-MODEM features
+- Test both C++ and C API usage if applicable
+- Consider edge cases and error handling
+
+**Documentation:**
+- Document your custom commands clearly
+- Provide usage examples
+- Explain any device-specific requirements
+- Note any limitations or known issues

docs/esp_modem/en/development.rst

@@ -0,0 +1,203 @@
+Development
+===========
+
+This chapter covers development mode, build system, and workflow for ESP-MODEM library developers.
+
+Development Mode
+----------------
+
+ESP-MODEM supports two different modes for handling AT command definitions, each optimized for different use cases.
+
+Production Mode (Default)
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Production mode uses pre-generated headers and sources from the ``command/`` directory. This mode is ideal for:
+
+- **Application development** using existing AT commands
+- **Better IDE navigation** and code completion
+- **Faster compilation times** with stable, tested implementations
+- **End users** who don't need to modify the core library
+
+Development Mode
+~~~~~~~~~~~~~~~~
+
+Development mode uses in-place macro expansion from the ``generate/`` directory. This mode is designed for:
+
+- **ESP-MODEM library developers** contributing to the core library
+- **Modifying core AT command definitions** in ``esp_modem_command_declare.inc``
+- **Debugging command implementations** at the source level
+- **Working directly with command source definitions**
+
+To enable development mode:
+
+.. code-block:: bash
+
+    idf.py menuconfig
+    # Navigate to: Component config → ESP-MODEM
+    # Enable: "Use development mode"
+
+Or set the configuration directly:
+
+.. code-block:: bash
+
+    idf.py -D CONFIG_ESP_MODEM_ENABLE_DEVELOPMENT_MODE=y build
+
+.. note::
+    Development mode requires C preprocessor expansion at compile time, which may result in longer compilation times and larger binary sizes.
+
+When to Use Development Mode
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+**Use Development Mode when:**
+- Contributing to the ESP-MODEM library itself
+- Modifying core AT command definitions in ``esp_modem_command_declare.inc``
+- Debugging issues in the command library
+- Adding new commands to the core library for all users
+
+**Use Production Mode when:**
+- Developing applications with ESP-MODEM
+- Adding support for new modem modules (see :ref:`customization`)
+- Creating custom commands for specific projects
+- General application development
+
+Build System
+------------
+
+The ESP-MODEM build system automatically handles the differences between production and development modes:
+
+- **Production mode**: Uses pre-generated sources from ``command/`` directory
+- **Development mode**: Uses macro expansion from ``generate/`` directory
+
+The build system selects the appropriate source directory based on the ``CONFIG_ESP_MODEM_ENABLE_DEVELOPMENT_MODE`` configuration.
+
+Directory Structure
+~~~~~~~~~~~~~~~~~~~
+
+ESP-MODEM uses two parallel directory structures:
+
+**``generate/`` Directory (Development Mode)**
+    Contains source files with macro definitions that get expanded at compile time:
+
+    - ``generate/include/esp_modem_command_declare.inc`` - Core AT command definitions
+    - ``generate/include/cxx_include/`` - C++ header templates
+    - ``generate/src/`` - Source file templates
+
+**``command/`` Directory (Production Mode)**
+    Contains pre-generated, expanded source files:
+
+    - ``command/include/esp_modem_api.h`` - Generated C API
+    - ``command/include/cxx_include/`` - Generated C++ headers
+    - ``command/src/`` - Generated source files
+
+Code Generation
+~~~~~~~~~~~~~~~
+
+ESP-MODEM uses a sophisticated code generation system to create the pre-generated sources. The ``generate.sh`` script:
+
+- Processes AT command definitions from ``esp_modem_command_declare.inc``
+- Uses C preprocessor metaprogramming to expand command prototypes
+- Generates both C and C++ API headers
+- Formats the generated code for consistency
+
+Generate Script Usage
+~~~~~~~~~~~~~~~~~~~~~
+
+The ``generate.sh`` script is located in ``components/esp_modem/scripts/`` and can be used to:
+
+**Generate all default files:**
+.. code-block:: bash
+
+    ./scripts/generate.sh
+
+**Generate specific files:**
+.. code-block:: bash
+
+    ./scripts/generate.sh generate/include/cxx_include/esp_modem_command_library.hpp
+
+**Generate files for documentation:**
+.. code-block:: bash
+
+    ./scripts/generate.sh ../../docs/esp_modem/generate/dce.rst
+
+The script automatically:
+- Determines the correct compiler (clang/clang++) based on file extension
+- Expands macros using C preprocessor
+- Formats generated code with astyle
+- Handles different file types (.hpp, .cpp, .h, .rst)
+
+Developer Workflow
+------------------
+
+Adding New AT Commands
+~~~~~~~~~~~~~~~~~~~~~~
+
+To add new AT commands to the core ESP-MODEM library:
+
+1. **Enable development mode** in your project
+2. **Edit** ``components/esp_modem/generate/include/esp_modem_command_declare.inc``
+3. **Add your command definition** using the ``ESP_MODEM_DECLARE_DCE_COMMAND`` macro:
+
+   .. code-block:: c
+
+       /**
+        * @brief Your new command description
+        * @param[in] param1 Description of parameter 1
+        * @param[out] param2 Description of parameter 2
+        * @return OK, FAIL or TIMEOUT
+        */
+       ESP_MODEM_DECLARE_DCE_COMMAND(your_new_command, command_result,
+                                     STR_IN(param1), INT_OUT(param2))
+
+4. **Test your changes** in development mode
+5. **Generate production files** using ``generate.sh``
+6. **Test in production mode** to ensure compatibility
+7. **Submit your changes** with appropriate tests
+
+Command Definition Macros
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+ESP-MODEM provides several macros for defining commands:
+
+- ``ESP_MODEM_DECLARE_DCE_COMMAND`` - Standard command declaration
+- ``STR_IN(param)`` - String input parameter
+- ``STR_OUT(param)`` - String output parameter
+- ``INT_IN(param)`` - Integer input parameter
+- ``INT_OUT(param)`` - Integer output parameter
+
+Testing and Validation
+~~~~~~~~~~~~~~~~~~~~~~
+
+When developing ESP-MODEM library changes:
+
+1. **Test in both modes** - Ensure your changes work in both development and production modes
+2. **Run existing tests** - Verify you don't break existing functionality
+3. **Test with multiple modules** - Ensure compatibility across different modem modules
+4. **Validate generated code** - Check that generated files are correct and properly formatted
+5. **Update documentation** - Add documentation for new commands or features
+
+CI/CD Integration
+~~~~~~~~~~~~~~~~~
+
+The ESP-MODEM project includes automated testing that:
+
+- **Validates generated files** - Ensures generated sources are up-to-date
+- **Tests both modes** - Runs tests in both development and production modes
+- **Checks formatting** - Validates code formatting and style
+- **Builds examples** - Ensures examples work with changes
+
+Best Practices
+--------------
+
+**For Library Developers:**
+- Use development mode when modifying core library files
+- Test changes in both production and development modes
+- Follow the existing code generation patterns
+- Update documentation when adding new commands
+- Include appropriate tests for new functionality
+
+**For Contributors:**
+- Submit changes that work in both modes
+- Include appropriate tests
+- Update relevant documentation
+- Consider backward compatibility
+- Follow the existing coding style and patterns

docs/esp_modem/en/index.rst

@@ -4,8 +4,11 @@ ESP-MODEM Programmers manual
 .. toctree::
 
    Brief intro <README>
+   Migration Guide <migration_guide>
    C interface <api_docs>
    C++ interface <cxx_api_docs>
    Advanced use cases <advanced_api>
+   Development <development>
+   Customization <customization>
    Internal design <internal_design>
    Internal implementation <internal_docs>

docs/esp_modem/en/migration_guide.rst

@@ -0,0 +1,64 @@
+Migration Guide
+===============
+
+ESP-MODEM v2.0 introduces production mode (default) with pre-generated sources for better IDE navigation. Previous behavior (development mode) requires explicit configuration.
+
+Breaking Changes
+----------------
+
+**Production Mode (Default)**
+- Uses pre-generated sources from ``command/`` directory
+- Better IDE navigation and code completion
+- Faster compilation
+
+**Development Mode (Optional)**
+- Uses macro expansion from ``generate/`` directory
+- Enable with ``CONFIG_ESP_MODEM_ENABLE_DEVELOPMENT_MODE=y``
+- Required for modifying core ESP-MODEM files
+
+Migration Steps
+---------------
+
+**Application Developers:**
+No changes required. Production mode is default.
+
+**Library Developers:**
+Enable development mode:
+
+.. code-block:: bash
+
+    idf.py -D CONFIG_ESP_MODEM_ENABLE_DEVELOPMENT_MODE=y build
+
+**Custom ``*.inc`` Files:**
+Use generation script:
+
+.. code-block:: bash
+
+    ./components/esp_modem/scripts/generate.sh your_file.inc
+
+**Build:**
+.. code-block:: bash
+
+    idf.py fullclean
+    idf.py build
+
+New Features (Coming Soon)
+--------------------------
+
+**Better URC Handling**
+- Enhanced unsolicited result code processing
+- Existing URC code remains compatible
+
+**AT-based Networking**
+- AT command networking examples now supports multiple connections
+
+Troubleshooting
+---------------
+
+**Build errors:** ``idf.py fullclean && idf.py build``
+
+**No IDE completion:** Use production mode (default)
+
+**Custom .inc files:** Use ``./components/esp_modem/scripts/generate.sh your_file.inc``
+
+**Modify core files:** Enable ``CONFIG_ESP_MODEM_ENABLE_DEVELOPMENT_MODE=y``