Firmware Over-The-Air (FOTA) Updates on ESP32

Firmware in embedded systems

Stored in non-volatile memory, typically in the device's flash, the device’s firmware persists after a reboot and can be overwritten. To effectively manage different firmware components, flash memory can be divided into multiple partitions. Each partition serves a distinct purpose, like the bootloader, application firmware, or specialized AI models for TinyML. This partitioning allows for the safe updating of a single component without affecting others, ensuring a firmware update process (for more information see article about Advanced Firmware Update).

Embedded devices often go beyond a single microcontroller unit (MCU) containing multiple processors, each assigned to a specific task. In addition to the main MCU, these devices may feature radio modems for wireless communication, smart sensors with local processing capabilities, and specialized components such as cameras or microphones for intelligent sensing. Each of these MCUs may have its own set of flash partitions, making them individually updatable.

Firmware Update Object in LwM2M

In the previous article, we’ve discussed the Firmware Update Object (/5) as defined in the LwM2M Object and Resource Registry.

In short, this object enables the management of the firmware update key functionalities such as installing firmware packages, updating firmware, and executing post-update actions. The process may involve rebooting the device, regardless of other factors like operating system architecture and the extent of software updates.

The primary goal is to empower LwM2M Clients to seamlessly acquire firmware images from any LwM2M Server using the specified object and resource structure.

For a better understanding of the update process status monitoring and robust error handling, in this post we will focus on two useful resources:

Resource ID 3: State - Indicates the current state for a firmware update. This value is set by the LwM2M Client.

• 0: Idle (before downloading or after successful updating)
• 1: Downloading (The data sequence is on the way)
• 2: Downloaded
• 3: Updating

Resource ID 5: Update Result - Contains the result of downloading or updating the firmware

• 0: Initial value. Once the updating process is initiated (Download /Update), this Resource MUST be reset to the Initial value.
• 1: Firmware updated successfully.
• 2: Not enough flash memory for the new firmware package.
• 3: Out of RAM during the downloading process.
• 4: Connection lost during the downloading process.
• 5: Integrity check failure for the newly downloaded package.
• 6: Unsupported package type.
• 7: Invalid URI.
• 8: Firmware update failed.
• 9: Unsupported protocol.

The firmware update resources the State and the Update Result are shown below:

NOTE:
The whole process is managed by the Client, the end user should not be concerned with it.

What is Bootloader and why do I need this in the Firmware Update over-the-air?

A bootloader as a program is responsible for loading and launching the operating system or firmware of a computer or embedded system. It is typically the initial program that runs when a device is powered on or restarted. The bootloader undertakes the essential configuration of internal modules within the ESP-IDF system, ensuring that fundamental settings are established for subsequent operations.

It is recommended to update to newer versions of ESP-IDF: when they are released. The OTA (over-the-air) update process can flash new apps in the field but cannot flash a new bootloader.

NOTE:
If testing an OTA update for an existing product in production, always test it using the same ESP-IDF bootloader binary that is deployed in production. For more information read Bootloader Compatibility

Firmware Update process for ESP32 device using ESP-IDF

Compiling and launching the LwM2M Client

To prepare a LwM2M Client side go through the following steps:

1. Install ESP-IDF and its dependencies on your computer. Please follow the instructions at https://docs.espressif.com/projects/esp-idf/en/v4.4.5/esp32/get-started/index.html up to and including the point where you call. $HOME/esp/esp-idf/export.sh

2. The project has been tested with ESP-IDF v4.4.5 but may work with other versions as well.

3. Open a command line interface and run git clone https://github.com/AVSystem/Anjay-esp32-client --recursive && cd Anjay-esp32-client

4. In the project directory run idf.py set-target esp32

5. Run idf.py menuconfig

6. Navigate to Component config/anjay-esp32-client:

• Select one of the supported boards or manually configure the board in the Board options menu
• Configure LwM2M client configuration in the Client options menu
• Configure Wi-Fi in the Connection configuration menu
• Navigate to Component config/Anjay library configuration to configure Anjay library and its dependencies (avs_commons and avs_coap)
• Press s on the keyboard to Save the configuration.

7. Run idf.py build to compile

8. Run idf.py flash to flash

9. The logs will be on the same /dev/ttyUSB<n> port that the above used for flashing, 115200 8N1

• You can use idf.py monitor to see logs on serial output from a connected device, or even more conveniently idf.py flash monitor as one command to see logs right after the device is flashed

Right now, the firmware update part is flashed on the device.

The Firmware Update module in Anjay

Anjay comes with a built-in Firmware Update module, which simplifies FOTA implementation for the user. Let’s dive into the code and discuss its most important fragments.

NOTE:
This part only describes functions that are in the code. The user doesn’t have to modify the code.

In our code, firmware update module installation will be taken by the function declared in firmware_update.h:

#ifndef FIRMWARE_UPDATE_H
#define FIRMWARE_UPDATE_H

#include <stdbool.h>
#include <anjay/anjay.h>

int fw_update_install(anjay_t *anjay);
bool fw_update_requested(void);
void fw_update_reboot(void);

#endif // FIRMWARE_UPDATE_H

In the main.c file the installation of the Firmware Update module takes place in two fragments:

#include <anjay/anjay.h>
#include <anjay/attr_storage.h>
#include <anjay/core.h>
#include <anjay/security.h>
#include <anjay/server.h>
#include "connect.h"
#include "default_config.h"
#include "firmware_update.h"
#include "lcd.h"
#include "main.h"
#include "objects/objects.h"
#include "sdkconfig.h"#include <anjay/anjay.h>
#include <anjay/attr_storage.h>
#include <anjay/core.h>
#include <anjay/security.h>
#include <anjay/server.h>
#include "connect.h"
#include "default_config.h"
#include "firmware_update.h"
#include "lcd.h"
#include "main.h"
#include "objects/objects.h"

static void anjay_init(void) {
 const anjay_configuration_t CONFIG = {
        .endpoint_name = ENDPOINT_NAME,
        .in_buffer_size = 4000,
        .out_buffer_size = 4000,
        .msg_cache_size = 4000
    };

 // Read necessary data for object install
    read_anjay_config();

    anjay = anjay_new(&CONFIG);
 if (!anjay) {
    avs_log(tutorial, ERROR, "Could not create Anjay object");
    return;
    }

 // Install Attribute storage and setup necessary objects
 if (setup_security_object(anjay) || setup_server_object(anjay)
            || fw_update_install(anjay)) {
    avs_log(tutorial, ERROR, "Failed to install core objects");
    return;
    }

 if (!(DEVICE_OBJ = device_object_create())
            || anjay_register_object(anjay, DEVICE_OBJ)) {
    avs_log(tutorial, ERROR, "Could not register Device object");
    return;
    }

 if ((LIGHT_CONTROL_OBJ = light_control_object_create())) {
    anjay_register_object(anjay, LIGHT_CONTROL_OBJ);
    }

 if ((PUSH_BUTTON_OBJ = push_button_object_create())) {
    anjay_register_object(anjay, PUSH_BUTTON_OBJ);
    }

#ifdef CONFIG_ANJAY_CLIENT_INTERFACE_ONBOARD_WIFI
 if ((WLAN_OBJ = wlan_object_create())) {
    anjay_register_object(anjay, WLAN_OBJ);
    }
#endif // CONFIG_ANJAY_CLIENT_INTERFACE_ONBOARD_WIFI
}

To implement I/O operations required to download the firmware a new globally defined structure with the entire shared state is used. This is stored in the firmware_update.c file:

#include <assert.h>
#include <stdatomic.h>
#include <stdbool.h>
#include <stddef.h>

#include <avsystem/commons/avs_log.h>

#include <anjay/anjay.h>
#include <anjay/fw_update.h>

#include <esp_ota_ops.h>
#include <esp_partition.h>
#include <esp_system.h>

#include "firmware_update.h"
#include "sdkconfig.h"

#if defined(CONFIG_ANJAY_CLIENT_CELLULAR_EVENT_LOOP) \
        && !defined(CONFIG_ANJAY_WITH_EVENT_LOOP)
#include "cellular_anjay_impl/cellular_event_loop.h"
#endif    // defined(CONFIG_ANJAY_CLIENT_CELLULAR_EVENT_LOOP)
         // && !defined(CONFIG_ANJAY_WITH_EVENT_LOOP)

static struct {
    anjay_t *anjay;
    esp_ota_handle_t update_handle;
    const esp_partition_t *update_partition;
    atomic_bool update_requested;
} fw_state;

The Firmware Update module consists of user-implemented callbacks of various sorts implemented in the firmware_update.c file:

• stream_open is called whenever a new firmware download is started by the Server. Its main responsibility is to prepare for receiving firmware chunks - e.g. by opening a file or getting flash storage ready, etc.

static int fw_stream_open(void *user_ptr,
 const char *package_uri,
 const struct anjay_etag *package_etag) {
    (void) user_ptr;
    (void) package_uri;
    (void) package_etag;

    assert(!fw_state.update_partition);

    fw_state.update_partition = esp_ota_get_next_update_partition(NULL);
    if (!fw_state.update_partition) {
        avs_log(fw_update, ERROR, "Cannot obtain update partition");
        return -1;
    }

    if (esp_ota_begin(fw_state.update_partition, OTA_SIZE_UNKNOWN,
                      &fw_state.update_handle)) {
        avs_log(fw_update, ERROR, "OTA begin failed");
        fw_state.update_partition = NULL;
        return -1;
    }
 return 0;
}

• stream_write is called whenever there is a next firmware chunk received, ready to be stored. Its responsibility is to append the chunk to the storage.

static int fw_stream_write(void *user_ptr, const void *data, size_t length) {
    (void) user_ptr;

    assert(fw_state.update_partition);

     int result = esp_ota_write(fw_state.update_handle, data, length);
     if (result) {
        avs_log(fw_update, ERROR, "OTA write failed");
        return result == ESP_ERR_OTA_VALIDATE_FAILED
             ? ANJAY_FW_UPDATE_ERR_UNSUPPORTED_PACKAGE_TYPE: -1;
        }
     return 0;
}

• stream_finish is called whenever the writing process is finished and the stored data can now be thought of as a complete firmware image. It may be a good moment here to verify if the entire firmware image is valid.

static int fw_stream_finish(void *user_ptr) {
    (void) user_ptr;

    assert(fw_state.update_partition);

     int result = esp_ota_end(fw_state.update_handle);
     if (result) {
        avs_log(fw_update, ERROR, "OTA end failed");
        fw_state.update_partition = NULL;
        return result == ESP_ERR_OTA_VALIDATE_FAILED
                   ? ANJAY_FW_UPDATE_ERR_INTEGRITY_FAILURE: -1;
        }
    return 0;
}

• reset is called whenever there is an error during firmware download, or if the Server decides to not pursue firmware update with downloaded firmware (e.g. because it was notified that firmware verification failed).

static void fw_reset(void *user_ptr) {
    (void) user_ptr;

 if (fw_state.update_partition) {
    esp_ota_abort(fw_state.update_handle);
    fw_state.update_partition = NULL;
    }
}

• perform_upgrade is called whenever the download is finished, the firmware is successfully verified on the Client and the Server decides to upgrade the device.

static int fw_perform_upgrade(void *user_ptr) {
    (void) user_ptr;

 int result = esp_ota_set_boot_partition(fw_state.update_partition);
 if (result) {
    fw_state.update_partition = NULL;
    return result == ESP_ERR_OTA_VALIDATE_FAILED
                   ? ANJAY_FW_UPDATE_ERR_INTEGRITY_FAILURE: -1;
    }

// ...

    atomic_store(&fw_state.update_requested, true);
    return 0;
}

static const anjay_fw_update_handlers_t HANDLERS = {
    .stream_open = fw_stream_open,
    .stream_write = fw_stream_write,
    .stream_finish = fw_stream_finish,
    .reset = fw_reset,
    .perform_upgrade = fw_perform_upgrade,
};

To install the module, we are going to use the fw_update_install() function which is called in the main.c file:

int fw_update_install(anjay_t *anjay) {
    anjay_fw_update_initial_state_t state = { 0 };
    const esp_partition_t *partition = esp_ota_get_running_partition();
    esp_ota_img_states_t partition_state;
    esp_ota_get_state_partition(partition, &partition_state);

    if (partition_state == ESP_OTA_IMG_UNDEFINED
            || partition_state == ESP_OTA_IMG_PENDING_VERIFY) {
    avs_log(fw_update, INFO, "First boot from partition with new firmware");
    esp_ota_mark_app_valid_cancel_rollback();
    state.result = ANJAY_FW_UPDATE_INITIAL_SUCCESS;
    }

    // make sure this module is installed for single Anjay instance only
    assert(!fw_state.anjay);
    fw_state.anjay = anjay;

    return anjay_fw_update_install(anjay, &HANDLERS, NULL, &state);
}

Connecting to the LwM2M Server

To connect to the Coiote IoT Device Management LwM2M Server, please register at https://eu.iot.avsystem.cloud/. The default Server URI (Kconfig option ANJAY_CLIENT_SERVER_URI) is set to EU Cloud Coiote DM instance.

NOTE:You may use any LwM2M Server compliant with LwM2M 1.0 TS. The server URI can be changed in the example configuration options (for this check the Compiling and launching the LwM2M Client step).

Go to the Device Center and click the Firmware Update tab

Click the Upload button and select the Basic Firmware Update type.

Into the Name field, type the file name or leave the name added automatically.

To perform an update, a binary file needs to be generated and uploaded to Coiote DM. To do so, build your project and upload the following binary to the Coiote DM:

$PROJECT_DIR/build/anjay-esp32-client.bin

NOTE:
In Coiote DM you have two options to upload the image: by uploading a new one or by choosing an already existing one from the resources.

In the next step, the Settings, there are two ways, defined by the LwM2M standard, how to deliver a new firmware package:

1. Pull method (recommended), which means sending the link to the device that the device can use to download the firmware image. Pull supports the following transport types:

• CoAP or CoAPs over UDP
• CoAP or CoAPs over TCP
• HTTP or HTTPs

2. Push method, which directly pushes the new firmware from the server onto the device. Push transmits the firmware over the same transport type as is used for device management, which is CoAPs over UDP by default.

NOTE:
We recommend using PULL download mode due to limitations imposed on other download modes.

Using the Pull method requires also choosing the proper image transport type. For constrained devices, it is recommended to use CoAP(s) for firmware downloads to avoid the need for additional protocol implementations.

NOTE:
Downloads using CoAP(s) over UDP tend to be slow due to the limitation of the maximum CoAP Block size of 1024 bytes and the required acknowledgments for each Block transfer.Choosing CoAP(s) over TCP or HTTP(s) usually results in faster download speeds. However, not every device supports these transport protocols.

By clicking the button Next and scheduling an update, the process will be in progress and it will start running through these four steps:

• The ESP32 device is prompted to prepare for the firmware update, initializing the update process.
• The LwM2M Client on the ESP32 board downloads the firmware and notifies the LwM2M Server, signaling that the firmware is ready for the update.
• The LwM2M Server triggers firmware update execution.
• The LwM2M Client on the ESP32 performs the firmware update after validating the integrity and authenticity of the new firmware which is done through a process called secure boot.
• The Client attempts to run the new firmware and reports the status to the Server. If the update goes successfully, the device transitions to running the new firmware; in case of errors, the device autonomously rolls back to the previous firmware version for stability.

This procedure is detailed in the LwM2M specifications (see LwM2M specifications).

Going to the data model and looking at the Firmware Update object you can see that the URI changed, showing the URL address of the Coiote, and the state also changed to 1 which is Downloading (The data sequence is on the way)

As the process is finished you can see in the firmware update tab that the process is finished with success and also the current firmware version is changed.

Going back to the data model you can see that the state changed to 0 which is Idle and the update result shows 1 which is Firmware updated successfully.

Conclusion

The practical demonstration with an ESP32 device clearly explains the firmware update process applicable to various IoT devices. During the whole process, the firmware update status can be monitored by reviewing the Resources State /5/0/3 and Update Results /5/0/5, helping identify and address problems promptly and ensuring the reliable and secure operation of IoT devices.

The importance of firmware updates over the air (FOTA) is growing as more resource-constrained IoT devices are deployed. As physical access can be difficult, FOTA provides a way to remotely fix bugs, patch security issues, and add features to connected devices. By making a clear process for remote firmware updates LwM2M outlines the easy and standardized process for every IoT device.

Useful links

Anjay documentation

Coiote documentation

Article about full-image FOTA

Article about multi-component FOTA updates

About Coiote IoT Device Management Platform