Gently communicating home using Matter

MatterWiFiOpenThreadnRF7002nRF5340nRF52840OTBR3D PrintingBluetoothOOBNFCEdge ImpulseOpenAIChatGPT

Motivation

People promise to themselves to live healthier and do their work early instead of at the last minute.

However, these promises are hard to keep since the rewards or consequences are delayed. Humans have no external control instance and are geared to spend the least effort.

In this project, I would like to tackle the problem of

• Getting into a bad posture while working in a home office (or working with computers over a long time in general).

And

• Getting homework done after school, which targets children at home.

How It Works

I want to use the home’s devices, like lights, to give feedback as a reminder to correct the posture. This shall work as an external control instance (consider adding negative rewards like shutting down the power of your TV instead of using lights), and it shall reduce the sensor overload and the need to unlock a potentially distracting smartphone. With time-spaced feedback and corrections, the correct behaviour will turn into a habit. Habits are automated actions you don’t need to think about and make decisions about actively, significantly reducing the effort to spend for a good posture.

So far, the idea, but how do you connect to the home to give feedback? Here, the Matter standard protocol helps out. Matter is a standard for smart homes designed to be interoperable between ecosystems among different manufacturers and brands with built-in security and reliability. Since all big players support it, we can use the phone or voice assistant of our choice to configure any automation of our home devices.

OK, we can interface with the home’s devices. Where to get the data from? Wearable devices measuring human physiology are commonly connected by Bluetooth Low Energy (BLE) to smartphone apps. Matter supports Thread or Wifi. To solve this issue, I will build a Matter Bridge that translates the BLE into the Matter network. The user can configure a Matter controller and decide what will happen. Compared to having the devices directly communicate, how the user connects the devices for feedback is up to his imagination. (Lights turn angry red until homework is started?)

To get an evaluation of the posture, I will build a device that uses an IMU, which measures the orientation and movement, and flex sensors, which measure to what degree they are bent, as input to an AI model using Edge Impulse.

A solution called Upright is on the market with the same goal (to train the user for a good posture). I see the difference in this approach as follows.

• Instead of notifying by vibration, the idea is to communicate gently - the feedback should be subtle enough not to disturb the concentration of your work.
• Feedback is freely configurable within the Home’s Matter network, not requiring a companion app on the smartphone.
• I will also use additional sensors to see if and how they improve the posture classification.

Commercial Matter bridges on the market usually connect the vendors' own devices to the Matter network. I am unaware of any bridge that can be configured to work with any BLE device. Overall, commercial Matter-enabled devices are still to be released in Japan.

When the contest started in September 2023, only one product was available in Japan, the SwitchBot Plug. Despite announcements (e.g. TP-Link in fall 2022 or Philipps Hue in January 2023), nothing is available one year later.

Update as of November: Matter arrived in Japan! Philipps Hue finally supports Matter, and the TP-Link products are now on sale. However, at this point, I have kept my project the same but will add the use of commercial devices in the following steps.

OK, with the above ideas, let’s start building a connection between wearable IoT devices and the smart home (innovation) to solve human health challenges (human-centred) based on the Matter protocol (Matter-ready smart application) with the newly released nRF7002 Development Kit.

Additionally, to cover the getting homework done use case, I will add a reminder app controlled by voice using OpenAI's whisper and completions API (ChatGPT) leveraging nRF7002's WiFi interface.

General Comments

This project was created between the contest announcement in August 2023 and the contest deadline in January 2024. This means that libraries and used SDKs have advanced in the meantime. Some steps might be slightly different - some workarounds might be not necessary anymore.

This project evaluated several technologies. I provide alternatives in case you do not have the exact same hardware. One of the next steps could be to select the best alternative and cut out unused parts. Still, I keep as much as possible, so that you can learn from my experiments.

Technical Description - High-level architecture

This is the technical overview of the whole project. Each part has a detailed description, including a complete circuit, software component, and sequence diagram.

Context of the Solution - Concepts of the Smart Home

The smart home consists of Devices that can be logically grouped into Rooms or Zones. For control, Automations are configured. Automations are repeatable actions that can be set up to run automatically on a specific input. Automations are made of three key components: Trigger (A device senses human presence, measures below a temperature threshold, daily at 8 o'clock), Condition (You are at home) and Action (open the door, turn on the heater). A Scene is a set of predefined settings for your devices. In most commercial systems, Assistants support setup and can trigger actions or advise proactively.

You can see that the power lies in automation. That is why I do not link the devices directly but let the user decide how he wants to use them for automation.

On the used boards

Feature-wise, all parts except the voice AI flow should run on any nRF52840 or nRF5340 boards (with some configuration adaptions). However, I recommend the nRF7002dk for the following reasons.

• Running Thread and Bluetooth in parallel has limitations, as described here Multiprotocol limitations in application development. In short, the application crashes if heavy BLE operations like scan or discovery run while the Matter stack is active on Thread. With the nRF7002dk, these limitations do not apply. Additionally, Matter over Wifi can run on the 5 GHz band, reducing interference with BLE, Thread and 2.4 GHz Wifi. On top of that, no OTBR is required, which saves a lot of trouble.

Update: There are no crashes in my case with the following addition in the board overlay.

/* Set IPC thread priority to the highest value to not collide with other threads. */
&ipc0 {
    zephyr,priority = <0 PRIO_COOP>;
};

• It is easy to integrate cloud-based services. This could be the Memfault module for easier debugging of issues (also when in the field)
• Wifi has a higher data throughput, which enables you to send extensive sensor data like audio or video data that cloud services could process.
• The Wi-Fi stack in Zephyr fully supports TCP (Thread not entirely), which makes it easy to connect, e.g. MQTT, to your application.

We will need the nRF7002dk for the homework use case as we connect to OpenAI's whisper and completions API (ChatGPT).

I started this project with a Switch Science nRF5340 MDBT53-1M board since I thought I could not use the nRF7002 Development Kit due to the missing technical regulations conformity certification for Japan. For experiments with the Bluetooth or Wi-Fi standard, there is a special procedure if the board is certified in other countries. I realized mid of November that the nRF7002dk declares conformity with radio equipment directive 2014/53/EU (RED) and has a CE mark. So I applied for this system. Details can be found at 技適未取得機器を用いた実験等の特例制度.

Static View

The following diagram shows how the project's components are deployed to the various boards.

The posture checker

• infers the quality of the posture with an ML (machine learning) model trained and deployed with Edge Impulse studio. Input data is the orientation of the neck measured by an IMU that fuses sensor data into an absolute orientation on the chip and flex sensors attached to the shoulders.
• provides the posture quality score by a Bluetooth Low Energy service secured with an out-of-band (UART) pairing procedure.
• can save sensor data on a local disk for later usage as training data.

The Matter bridge

• creates a dynamic Matter endpoint with clusters specified in a BLE service and attributes to a Matter cluster and attributes mapping configuration. The configuration maps the posture score to a current level on a level control cluster.
• handles (re)connection and subscription to the Posture Checker by BLE.
• implements Matter over Thread protocol.

The Matter Controller

• commissions the Matter devices into the Thread and Matter network.
• subscribes to the level cluster of the Bridge and sends the current level by a move to hue command to a colour control cluster on the Feedback light.

The NFC Reader

• receives the BLE OOB pairing data from the Posture checker via NFC Type 4 Tag.
• determines the presence of Felica cards to clear reminders.

The Open Thread Border Router (OTBR)

• connects the Thread network to other IP-based networks, such as Wi-Fi or Ethernet, so that the Matter Bridge and Feedback Light on the Thread network can communicate with the Matter Controller on the home's Ethernet.

The Open Thread Radio Co-Processor

• is the hardware interface for the OTBR.

The Feedback Light

• gives the user feedback by the colour of the light.

Dynamic Behavior

The following diagram illustrates the sequential aspects.

Architectural Constraints

• The solution shall be based on the Matter protocol and follow its standard to ensure interoperability with commercial Matter controllers and devices.
• The solution shall comply with the Make it Matter! contest guidelines, e.g. using specified development boards.
• The solution shall make the best use of the nRF Connect SDK.
• The solution shall abide by the Radio Law in Japan (devices should have technical regulations conformity certification or the special system for experiments can be applied).
• Comply with software licenses.
• Exclude expensive devices like commercial matter controllers.

Architectural Goals

• Reusability by injecting configuration and business logic into generic modules.
• Keep implementations simple by avoiding patterns where not needed or features that are not required.
• High level of security like Bluetooth out-of-band pairing to protect sensitive health data.
• The project shall be reproducible by the reader (reference alternatives).

Quality Requirements

• Requirements shall be defined, but no requirements tracing is required.
• No unit or integration tests are formally defined, but the above dynamic behaviour shall continuously work with an automatic recovery on a crash.
• No Matter certification is required, but best efforts shall be made to follow the Matter specification.
• No dedicated development process is required.
• Compiler warnings shall be fixed or deactivated line-based in the code with a rationale.

Risks

• Commercial devices are still ramping up and might not support all clusters as specified. Mitigation to use the CHIP tool.
• Commercial devices lack debugging features. Investigations of why something does not work are challenging. Mitigation to use the CHIP tool.
• The Nordic Matter Bridge Application has yet to be released (at project start). Mitigation to keep track of GitHub issues/merges, deeply analyse and test.

Videos

The purpose of the following videos is to demonstrate that the project is working. For detailed explanation, please read this story.

01 Introduction of the Parts

02 Posture Checker

03 Reminders

Below two screenshots of the live recognition. Unfortunately the values seemed to be unstable, likely due to different positions and orientations (?) than during training. The video above uses the recording of test data I took in November for a simulation mode. I didn't take a video back then.

Feedback Light

Summary

The home communicates by the coloured feedback light. It is moving on the colour circle from blue, meaning good, through green, yellow, red, and purple, meaning terrible. It is a Matter-enabled coloured WS2812B LED "Neopixel" based lighting using the colour control cluster. The front is a semi-transparent glass showing an owl on a tree I created at a glass workshop during a vacation trip. The case is 3D printed to hide the LED parts and to provide a stable stand.

Motivation

I decided to create my own device since no matter-enabled was available at the time of creation in Japan. Initially, I bought some used Philips Hue colour light bulbs that worked with the Apple Home app. However, despite many Matter-related articles on their website, their bridge does not support Matter one year later (remark: It is now supported). And since the Apple Home app cannot control the HSL colour-based colour cluster yet - it only shows the current colour - I decided to use the CHIP tool as a Matter controller. (I need to investigate if the lack of colour support is an issue with the project or the Home app.)

Parts Selection

• I selected the Adafruit Feather nRF52840 Express because it has a Neopixel built-in, which I used for testing before having the LED strip available. Also, this is one of the few boards with TELEC certification (R 018-180280).
• The LED light strip should be with a WS2812B controller. My selection is random - I bought it used for cheap. It's no name. Any WS2812B LED or Neopixel would do the job. A resistor, capacitor and level shifter are recommended. See Logic Level Shifters for Driving LED Strips
• I created the glass front in a workshop by selecting milky glass pieces and arranging decoration (an owl) on one half of them. The rest was taken care of by the workshop.
• The box is 3D printed. Since the Prusa Japan shop finally stocked the MK4 and distributed vouchers at the Tokyo Maker Faire, I used the chance to buy one. Initially, I thought of buying some fitting boxes. But now was a good chance to gather experience with 3D printing.
• Due to the power consumption of the LEDs, the best solution would be to use USB power. Using batteries is possible, too. (In the above image, I used Li-ion batteries. Just connect to BAT instead of USB 5V)

Hardware Assembly

I followed the Adafruit NeoPixel Überguide.

• I have only a few (14) LEDs, so the capacitor is sized down. I had a 100uF and 470uF available. I selected the physically smaller one.
• I had a 330 Ohm resistor.
• A level shifter is also recommended, but I do not use one. Too many cables. It means it is luck to work, but it usually works out.
• I connected the 5V to the USB power. With 14 LEDs, I can still use the USB power of my Mac while testing and reading logs over USB UART. Connect to BAT in case of a Li-ion battery.

The resistor is soldered inside the cable. You can see a small adaptor board (not in the parts list; it is this one SH connector DIP kit) where I can easily plug the LED strip out. It is a Stemma qt cable. The thin wires (24AWG) are rated enough for the 14 LEDs (measured 400mA peak for full white).

Designing and Printing the Box

This is my first 3D design. I searched printables.com for available models. Due to my size constraints, I decided to create my own model. The reference I based it on was Lithophane Lightbox by Desktop Inventions.

I selected TinkerCad to create the model. Just because it seemed simple and free. No deep consideration was made. Overall, the model was simple. However, it took longer than expected, and I learned some things.

Learnings

• I wanted 1.6 mm thick walls (4 lines with a 0.4 mm nozzle). It is less than the 1 mm unit in TinkerCad. I overlooked the option to set a 0.1 mm grid snapping. So I could enter fractional sizes, but not place the parts. Therefore, I switched to the Codeblocks, which also has the benefit of reproducibility.
• Well, it was not possible to download the code and share it here... This is a screenshot of the code. A share link can be created and you can see the project here link

• In Codeblocks, the pivot point is always the centre. If you change the size of the parts, you must move them half the length. It was pretty time-consuming.
• The code execution is always linear. I was not able to use the parameters well. Templates do not have parameters. All in all, it felt constrained - Constraints keep projects simple.
• Operations always work on the last block only. If it was a group, then on the whole object. It seems to matter where the code is located on the board.
• Ultimately, I was tired of managing all the numbers, so I switched to trial and error mode by changing one step and checking the outcome. Very tedious.

Next time I will use the interactive TinkerCad with 1 mm as the minimum unit.

I encountered the following issues with the print.

• (1st/top) There was no gap between the top and bottom. So it did not fit. But creating a gap on all sides (3rd) makes the top lose. → Find some middle way.
• (1st) The frame was too large, and the plate did not fit due to decorations near the frame. → Reduced the frame size.
• (2nd) Adjacent parts did not combine. I got one line of support material between objects floating, and the print was loose and easily breaking apart. → Always check for support parts in the G-Code preview before printing. Make sure that parts overlap before combining. - I did not find the root cause. It could be also a setting in the slicer, which I did not touch.
• (2nd) With faster print speed, the thin wall degraded in stability. → Use the quality print speed.
• The 4th on the bottom is the final good print.
• On the bottom are the initial screw holders. They also had a support material in between and did not stick. Additionally, I soldered pins to the board, and the board did not fit as the pins needed more space. The board is now placed outside.

Learnings of connecting the LED Strip

• My initial try ended up too bulky. When combining wires, I cut the wires too short and had no/less space for the shrinking tubes.

Software Part

Component Diagram

The Feedback Light Application is based on the Matter light example with the following additions or modifications.

• Instead of using an LED Widget to control the light by Matter, the LED Strip module interfaces WS2812B LEDs using the WS2812B I2S driver. LED Widgets are only used as status and factory reset LEDs.
• I added a colour control cluster with only mandatory items and HSL-based colour controls. Controls are implemented in the ZCL Callbacks.
• I augmented the App Events with colour control actions.
• Identify lets the WS2812B LEDs blink.
• I adapted the button and function handlers to work with only one button. Short press toggles OnOff and long press triggers factory reset. Further functionality is removed.
• The board overlays and configuration were adapted to run on an Adafruit Feather nRF52840 Express. Board files are already available in Zephyr.

Sequence Diagram

At startup, the Matter stack, buttons and LEDs are initialised. Then, a loop is entered to wait for events. Events can be triggered by user interaction via the buttons or by receiving Matter commands. Either way, I create an event and enqueue it to the event queue. The loop in the main thread takes the event and handles it, for example, by changing the light colour.

Challenges encountered

• The WS2812B SPI driver for nRF52840 is unreliable. A comment states,

At 4 MHz, 1 bit is 250 ns, so 3 bits is 750 ns. That's cutting it a bit close to the edge of the timing parameters, but it seems to work on AdaFruit LED rings.

• In my case, the colours are sometimes random, sometimes only applied to the first n LEDs, and occasionally random colours between the LEDs. I spent some time with level shifters and voltages, as I suspected these to be the root cause. Once I switched to the I2S drivers, everything worked 100% perfectly.
• The colour control cluster has quite a lot of attributes and commands. I made some code changes by mistake or some configuration mistake that the handling of the commands was not compiled in. I thought I needed to handle the commands, but after I found out, I deleted all my code again.
• Converting between RGB and other colour spaces seemed complex, but I found several libraries on GitHub and selected one.

Enabling Colour Control with the ZAP Tool

• To enable the colour cluster, it is required to load the existing.zap file into the ZAP tool and then enable and configure the cluster.
• Initially, I did not notice the Device option on Endpoint 1. I added the Color Cluster and enabled the minimum mandatory items according to the Matter cluster spec for Hue/Saturation control.
• I set FeatureMap, ColorMode and AdvancedColorMode to 0x0 to indicate only Hue/Saturation support.
• On my Home app, I could see the current colour but no controls for the colour. Maybe due to the wrong Device? I changed the Device to Matter Color Temperature Light. This enabled all attributes and commands required for a certified device. I deactivated Scenes and some other items in Color Control again, ignoring the warning that these were mandatory and generated. Still, I cannot control the colour. Maybe Hue/Saturation is not enough? I decided to postpone this until after the contest.

Device Configuration

Board Overlay

• Add the WS2812 node configured to use the i2s device with pin 1.08 (pin 5 of the board) as output to control 14 LEDs.

// Needed for LED_COLOR_ID_GREEN, LED_COLOR_ID_RED, LED_COLOR_ID_BLUE
#include <zephyr/dt-bindings/led/led.h>

 / {
    aliases {
        led-strip = &neopixel_led;
    };
};

// Only I2S_SDOUT in use
&pinctrl {
    i2s0_default_alt: i2s0_default_alt {
        group1 {
            psels = <NRF_PSEL(I2S_SCK_M, 0, 20)>,
                <NRF_PSEL(I2S_LRCK_M, 0, 19)>,
                <NRF_PSEL(I2S_SDOUT, 1, 8)>,
                <NRF_PSEL(I2S_SDIN, 0, 21)>;
        };
    };
};

i2s_led: &i2s0 {
    status = "okay";
    pinctrl-0 = <&i2s0_default_alt>;
    pinctrl-names = "default";
};

/ {
    neopixel_led: ws2812 {
        compatible = "worldsemi,ws2812-i2s";

        i2s-dev = < &i2s_led >;
        chain-length = <14>; /* arbitrary; change at will */
        color-mapping = 
        <LED_COLOR_ID_GREEN>,
        <LED_COLOR_ID_RED>,
        <LED_COLOR_ID_BLUE>;
        reset-delay = <120>;
    };
};

• Enable logging over USB

#include <zephyr/dt-bindings/led/led.h>

 / {
    chosen {
        zephyr,console = &usb_cdc_acm_uart;
        zephyr,shell-uart = &usb_cdc_acm_uart;
    };
};

zephyr_udc: &usbd {
    usb_cdc_acm_uart: cdc-acm-uart {
        compatible = "zephyr,cdc-acm-uart";
    };
};

• Disable unused peripherals

&adc {
  status = "disabled";
};
&i2c0 {
  status = "disabled";
};
&i2c1 {
  status = "disabled";
};
&pwm0 {
  status = "disabled";
};
&spi2 {
  status = "disabled";
};

Project Configuration

• Some configuration items were not correctly applied when they were set in the board specific configuration. Therefore I merged all configurations into one prj.conf for all projects.
• Enable the driver WS2812 with i2s

CONFIG_LED_STRIP=y
CONFIG_WS2812_STRIP=y
CONFIG_WS2812_STRIP_I2S=y
CONFIG_I2S=y

• Further configuration changes in comparison to the light bulb sample

# Disable factory data support. 
CONFIG_CHIP_FACTORY_DATA=n
CONFIG_CHIP_FACTORY_DATA_BUILD=n

# Disable NFC Commissioning
CONFIG_CHIP_NFC_COMMISSIONING=n

# Openthread (next 2 are default)
CONFIG_NET_L2_OPENTHREAD=y
CONFIG_OPENTHREAD_DEBUG=n

# Increase TX power 
CONFIG_OPENTHREAD_DEFAULT_TX_POWER=3

# Use the prebuild library (Set as default in KConfig too)
CONFIG_OPENTHREAD_NORDIC_LIBRARY_FTD=y
CONFIG_OPENTHREAD_FTD=y

# Disable shell and logging (to fit on a device with the Adafruit uf2 bootloader, e.g. XIAO BLE)
CONFIG_SHELL=n
CONFIG_CHIP_LIB_SHELL=n
CONFIG_LOG=n

• USB configuration for logging over USB CDC UART

CONFIG_USB_DEVICE_STACK=y
CONFIG_USB_DEVICE_PRODUCT="Feedback Light"
CONFIG_USB_CDC_ACM=y
CONFIG_UART_LINE_CTRL=y
CONFIG_USB_REQUEST_BUFFER_SIZE=2048

# Random number to avoid build warning
CONFIG_USB_DEVICE_VID=0x1234
CONFIG_USB_DEVICE_PID=0x5678

CONFIG_USB_DEVICE_MANUFACTURER="J"
CONFIG_USB_DEVICE_SN="MLB_000000000000"

CONFIG_USB_DEVICE_INITIALIZE_AT_BOOT=y

CONFIG_USB_COMPOSITE_DEVICE=n
# Enables more endpoints like 3 UARTs and USB Mass Storage (not used for this project)
CONFIG_USB_MAX_NUM_TRANSFERS=8

CONFIG_ISR_STACK_SIZE=4096

# Avoid lots of warning logs. It cannot log itself when not yet connected
CONFIG_USB_CDC_ACM_LOG_LEVEL_OFF=y

Selected Explanations of the Source Code

• The Matter stack handles the colour cluster commands, animates smooth transitions and reports the resulting value by the MatterPostAttributeChangeCallback callback. I create an event with the action LightingAction::SetHue and the new value. The isRelative attribute is not used anymore. It is there due to my misunderstanding that I need to implement commands myself.

void MatterPostAttributeChangeCallback(const app::ConcreteAttributePath &attributePath,
                                       uint8_t type, uint16_t size, uint8_t *value) {
  ClusterId clusterId = attributePath.mClusterId;
  AttributeId attributeId = attributePath.mAttributeId;

...
  } else if (clusterId == ColorControl::Id &&
             attributeId == ColorControl::Attributes::CurrentHue::Id) {
    ChipLogProgress(Zcl, "Cluster LevelControl: attribute CurrentHue set to %" PRIu8 "", *value);
    AppEvent event;
    event.Type = AppEventType::Lighting;
    event.LightingEvent.Action = LightingAction::SetHue;
    event.LightingEvent.Value = *value;
    event.LightingEvent.isRelative = false;
    AppTask::Instance().ZCLHandler(event);
...
  } else {
    ChipLogProgress(
        Zcl, "MatterPostAttributeChangeCallback(clusterId=%d, attributeId=%d) not handled!",
        clusterId, attributeId);
  }
}

• To set the LEDs to the last stored colour at startup, I implemented the emberAfColorControlClusterInitCallback function analogue to the emberAfOnOffClusterInitCallback of the sample.

void emberAfColorControlClusterInitCallback(EndpointId endpoint) {
  EmberAfStatus status;
  uint8_t storedHue;
  uint8_t storedSaturation;

  status = ColorControl::Attributes::CurrentHue::Get(endpoint, &storedHue);
  if (status != EMBER_ZCL_STATUS_SUCCESS) {
    status = ColorControl::Attributes::CurrentSaturation::Get(endpoint, &storedSaturation);
    if (status == EMBER_ZCL_STATUS_SUCCESS) {
      // Set the actual state to the cluster state that was last persisted
      AppEvent event;
      event.Type = AppEventType::Lighting;
      event.LightingEvent.Action = LightingAction::SetHueSaturation;
      event.LightingEvent.Value = storedHue;
      event.LightingEvent.Value2 = storedSaturation;
      event.LightingEvent.isRelative = false;
      AppTask::Instance().ZCLHandler(event);
      ChipLogProgress(Zcl, "emberAfOnOffClusterInitCallback: restore the hue %d and saturation %d",
                      storedHue, storedSaturation);
    }
  }
}

• I added the new actions to the LightingAction and attributes to the LightingEvent event.

enum class LightingAction : uint8_t {
  None = 0,
  SetLevel,
  SetEnable,
  SetHue,
  SetSaturation,
  SetHueSaturation,
  Blink,
};

struct AppEvent {
  union {
...
    struct {
      LightingAction Action;
      uint32_t Value;
      uint8_t Value2;
      bool isRelative;
    } LightingEvent;
...
};

• Events are then handled in the LightingActionEventHandler function.

void AppTask::LightingActionEventHandler(const AppEvent &event) {
...
  } else if (event.Type == AppEventType::Lighting) {
    switch (event.LightingEvent.Action) {
      case LightingAction::SetEnable:
        light.enable(event.LightingEvent.Value);
        break;
      case LightingAction::SetLevel:
        light.setLevel(event.LightingEvent.Value);
        break;
      case LightingAction::SetHue:
        light.setHue(event.LightingEvent.Value, event.LightingEvent.isRelative);
        break;
      case LightingAction::SetSaturation:
        light.setSaturation(event.LightingEvent.Value, event.LightingEvent.isRelative);
        break;
      case LightingAction::SetHueSaturation:
        light.setHue(event.LightingEvent.Value, event.LightingEvent.isRelative);
        light.setSaturation(event.LightingEvent.Value2, event.LightingEvent.isRelative);
        break;
      default:
        break;
    }
  }
}

• The logic for the LED light is implemented inside led_strip.cpp. It keeps the current state as HSL colour values. The current level of the level cluster is mapped to L. Values can be set as absolute or relative. In the relative case, the value overflows at max/min values. Whenever a value is set, it is applied immediately. For applying the values, conversion to RGB colour space is required. I used the following library from Alex Kuhl. I copied only the required function and the copyright/attribution comment to my code. alexkuhl/colorspace-conversion-library: An easy-to-use and understand C++ library for colorspace conversions. Currently works with RGB ←> HSL/HSV. Will probably add more in the future
• A word about the common parts of the Nordic Matter samples: I made them apparent by copying the parts I use into the project folder and modifying the CMake file. The only dependency is the LED Widget used for the system and factory reset LEDs.
• I adapted the button and function handlers to work with only one button. Short press toggles OnOff and long press triggers factory reset. Further functionality is removed. Only System LED and Factory reset LED are used. Not required parts like OTA or Wifi have been deleted.

Alternatives

• Front Glass: I recommend you search for "Lithophane 3D print" and print a picture you like. I have good experience with this page: Lithophane Makers.
• Board: Any nRF52840 board can be used. I added a board overlay for the Seeedstudio XIAO BLE. However, logging, shell, and debugging features must be removed with the default bootloader to save Flash.
• LEDs: Any WS2812B LED or Neopixel would do the job.

You can replace the whole part with a commercial Matter-based colour light.

Posture Checker

Summary

The posture checker evaluates the quality of a good posture by using flex sensors positioned on the shoulders and an absolute orientation sensor on the neck by applying machine learning algorithms (nowadays called AI) using the Edge Impulse framework. It can collect data to store it on a flash drive or send it live to train the model. The posture score is sent with Bluetooth Low Energy (BLE). Since it handles sensitive personal data, it applies anti-tracking measurements using privacy features (Resolvable Private Addresses). Furthermore, it requires the highest security level with out-of-band (OOB) pairing. Additionally, advertisement filtering is applied once bonded (pairing information is stored permanently).

Motivation

Since I do a lot of desk work and also use my computer during my free time, I need to take care of my health by maintaining a healthy posture. Focusing on work makes it easy to forget and fall back into unhealthy patterns. Therefore, I created a device that keeps checking and can inform me to take corrective actions.

Parts Selection

• I selected the XIAO BLE (Sense) because it is the smallest nRF52840-based board I know of that is also certified for usage in Japan. It also has 2MB of flash on the board that I used for logging measurement data for training. It has also solder pads for the NFC antenna and would have a li-ion charging circuit (both not used in the end)
• The BNO085 was selected since it has sensor fusion algorithms on the sensor. I don’t use the lsm4ds3tr on the sense board version. I wanted to try the BNO085 before, but it was hard to get with a long lead time. Chip shortages have been eased, and I finally got one. Reading the lsm4ds3tr is implemented for further evaluations with machine learning in the future.
• I assumed that additional measurements with a flex sensor could improve the prediction. There are only a few variations of flex sensors to choose from. I could get some used ones.
• I added an external ADC chip to interface with the flex sensors. I read that these external ADCs perform much better. I don’t feel confident in selecting a proper one. The ADS1015 was simply available and cheapest at an electronic store nearby. For my use case, it might not be necessary, though. The ADC impacts the implementation since the driver in Zephyr changes.

Hardware Assembly

The BNO085 and the ADS1015 are connected by i2c. A voltage 330-ohm divider connects the flex sensors to the ground. I used the stemma qt connectors for both. Since the sensors are connected to the body, they should be thin and lightweight, not too bulky. So, I soldered cables to the parts, not using pins.

Using NFC on the Seeedstudio XIAO BLE

This chapter is optional. It is for your reference if you want to use OOB with NFC. As I did not have an NFC polling device ready initially, I implemented OOB over UART first. There is also a configuration to switch OOB off.

Required steps to use the NFC feature

• It is required to configure the NFC pins for NFC with the UICR (Nordic documentation)
• It is required to recompile the (Adafruit_nRF52) bootloader (link) with the option USE_NFCT, which prevents setting the cmake option CONFIG_NFCT_PINS_AS_GPIOS, which disables the NFC pins for NFC usage on each reboot.
• The Adafruit nRF52 bootloader does not include the XIAO BLE board definition. I used 0hotpotman0's changes as a reference and applied them to the latest Adafruit nRF52 Bootloader version. See my git diff file below
• build with "make BOARD=xiao_nrf52840_ble_sense"
• Double reset the XIAO BLE board and copy the file _build/build-xiao_nrf52840_ble_sense/update-xiao_nrf52840_ble_sense_bootloader-0.7.0-32-g7210c39-dirty_nosd.uf2 to flash the bootloader.
• It is required to get an NFC Antenna and tune it with the correct capacitors (It won't work without these capacitors!). See the Nordic whitepaper nRF52832 NFC Antenna Tuning.
• I had a broken nRF52840-dk. So, I took the antenna and the required 300pf capacitors from there.
• UICR might be protected. In this case, erase using an SWD debugger. Erasing and verification is taken from Nordic Devzone 82798

# erase all
nrfjprog --eraseall

# Check the NFC-related UICR. It should return FFFFFFFF and not FFFFFFFE
nrfjprog --memrd 0x1000120C --w 32 --n 4

# Flash the Adafruit bootloader again after having erased it all.
nrfjprog -f nrf52 --verify --chiperase --program _build/build-xiao_nrf52840_ble_sense/xiao_nrf52840_ble_sense_bootloader-0.7.0-32-g7210c39-dirty_s140_7.3.0.hex --reset

Verification by flashing the sample NFC: Launch App with an iPhone.

I also posted this with more details on the Seeedstudio forum at https://forum.seeedstudio.com/t/xiao-ble-nfc-doesnt-work/264543/7.

Learnings

As with the Feedback Light, the first outcome was very bulky. The improvement was to connect the voltage dividers on the back of the board.

Electrical tape gets very sticky. I switched to Kapton tape.

Software Part

Architectural decisions

• Software components should be designed in a reusable fashion. The application shall inject application-specific logic, for example, by callback functions.
• All sensors should be accessible by a common interface. Therefore, the sensor aggregator receives multiple sensors and iterates over them to collect all values.
• Make only direct calls as the application flow is simple, meaning no event-driven architecture.

Component Diagram

Reading the sensor values goes through several abstraction layers. On the bottom is the vendor’s library, next to the class to interface the vendor’s library or, if there is a Zephyr driver, the Zephyr sensor interface. Next is a sensor abstraction implementing the SensorInterface. It defines only two functions: init() and getReading(struct posture_data). Posture_data is a struct of all sensor data and a timestamp.

To store and retrieve sensor values for training, I integrated multiple ways. The Edge Impulse data forwarder uses UART and BLE. The second way is to write binary data to a littlefs filesystem on the external flash. Littlefs because I read it is more robust against data corruption in case of sudden power-offs compared to FAT. However, it is supported out of the box for only some OS when making it available over USB Mass Storage.

• Why three ways? Initially, I thought to use BLE but switched because BLE needed to be initially connected after reboot and reconnected on each disconnect, or data was lost. The additional security measurements made it tedious, too. Writing to disk also needs less energy (essential when running on battery), and it is possible to download the data at once by a shell command that converts it to a CSV output.

Posture score calculation is done by the edge impulse wrapper that integrates inferring the score using machine learning.

Sending the posture score to the Matter Bridge is done by a Bluetooth service.

Sequence Diagram

The main module initialises all modules USB, persistence, sensors, Bluetooth and data forwarder. Then it reads all sensors every second, adds the datetime and distributes it to all parts.

Preparation of the Edge Impulse Model

Data Collection and Labelling

First, I tried recording 20 minutes in the internal storage. With the help of time stamps and a video, I went through the samples and labelled them with good and bad in a CSV sheet, which I imported to Edge Impulse. It took me some time to decide where the transition between good and bad was slightly ambiguous. At that time, I still recorded partially the Quaternion formatted data, which I did not understand. The data was not usable, and I deleted it again.

My second try was to use the BLE EI data forwarder. The nRF Edge Impulse App refused to connect with the error message "This device does not advertise the expected Edge Impulse Remote Management Service.". I did not find anything in the documentation. I was using the code as it is in the Machine Learning application. There, the data is sent by the Nordic UART Service. I found the source code with the required UUIDs here IOS-nRF-Edge-Impulse/Shared/Bluetooth/BluetoothManager/BluetoothManager.swift at master · NordicSemiconductor/IOS-nRF-Edge-Impulse. However, I decided to switch to using the UART EI data forwarder. The UART EI data forwarder worked quite well. The command line is interactive and easy to understand.

$ edge-impulse-data-forwarder
Edge Impulse data forwarder v1.21.1
Endpoints:
    Websocket: wss://remote-mgmt.edgeimpulse.com
    API:       https://studio.edgeimpulse.com
    Ingestion: https://ingestion.edgeimpulse.com

? **Which device do you want to connect to?** /dev/tty.usbmodem14606 (J)
[SER] Connecting to /dev/tty.usbmodem14606
[SER] Serial is connected (D0:63:37:E6:8B:09:01:5A)
[WS ] Connecting to wss://remote-mgmt.edgeimpulse.com
[WS ] Connected to wss://remote-mgmt.edgeimpulse.com

[SER] Detecting data frequency...
[SER] Detected data frequency: 2Hz
? **13 sensor axes detected (example values: [8634,1121,1661,1618,-84.52,57.47,-122,-0.47,1.84,9.84,-0.47,1.84,9.84]). What do you want to call them? Separate the names with ',':** time,flex1,flex2,flex3,roll,pitch,yaw,accelX,accelY,accelZ,magX,magY,magZ
? **What name do you want to give this device?** postureChecker
[WS ] Device "postureChecker" is now connected to project "posture-checker". To connect to another project, run `edge-impulse-data-forwarder --clean`.
[WS ] Go to https://studio.edgeimpulse.com/studio/276797/acquisition/training to build your machine learning model!
[WS ] Incoming sampling request {
  path: '/api/training/data',
  label: 'test',
  length: 10000,
  interval: 500,
  hmacKey: 'd29a653d3956e85c17d1bdf105c88f06',
  sensor: 'Sensor with 13 axes (time, flex1, flex2, flex3, roll, pitch, yaw, accelX, accelY, accelZ, magX, magY, magZ)'
}
[SER] Waiting 2 seconds...
[SER] Reading data from device...
[SER] Reading data from device OK (15 samples at 2Hz)
[SER] Uploading sample to https://ingestion.edgeimpulse.com/api/training/data...
[SER] Sampling finished

In Edge Impulse Studio at Data acquisition, I selected the device, entered the label good or bad and recorded 25 samples. 20% of the data is kept separate for testing.

Data Preprocessing and Pipeline

The next step is to set up the classification pipeline. The incoming data is (pre)processed in Windows, for example, the last 3 seconds. I flattened the data of each sensor type for preprocessing and put the result into a classifier using all input features and the labels good and bad as output features.

I chose average, minimum and maximum for the flex sensor and the BNO085 output and checked all boxes for the accelerator and magnetometer.

All features look well separated, which indicates that the input is easy to classify. So I expect the performance to be good.

Model Topology and Training

The next step is to decide on the model's topology (structure) and to train it. I left the defaults and increased the number of training cycles to 1000.

Model Testing

The above accuracy is calculated on the training data. Therefore, the next step is to test the model on unseen new data - the training data. The results look promising. The test data is skewed towards bad samples (probably because I had some test imports in it, which I deleted later on)

Model Deployment

The last step is to build and deploy the model. The downloaded zip folder is saved within the project. I used the Edge Impulse Wrapper module that takes care of the integration. I feed the data and read the results.

Device Configuration

Board Overlay

• I configured a littlefs filesystem on a partition on the flash memory. This filesystem shall be exposed as a USB Mass Storage Device.

/ {
 	// Configure the littlefs partition
    fstab {
        compatible = "zephyr,fstab";
        lfs1: lfs1 {
            compatible = "zephyr,fstab,littlefs";
            mount-point = "/lfs1";
            partition = <&lfs1_part>;
            read-size = <16>;
            prog-size = <16>;
            cache-size = <64>;
            lookahead-size = <32>;
            block-cycles = <512>;
        };
    };

	// Expose the littlefs partition as a USB Mass Storage Device
    msc_disk0 {
        compatible = "zephyr,flash-disk";
        partition = <&lfs1_part>;
        disk-name = "NAND";
        cache-size = <4096>;
    };
};

&p25q16h {
    partitions {
        compatible = "fixed-partitions";
        #address-cells = <1>;
        #size-cells = <1>;

		// Place the littlefs partition onto the external flash
        lfs1_part: partition@0 {
            label = "flash-storage";
            reg = <0x00000000 0x00200000>;
        };
    };
};

• The ads1015 device is connected to t2c1.

&i2c1 {
...
    ads1015: ads1015@48 {
        compatible = "ti,ads1015";
        reg = <0x48>;
        #io-channel-cells = <1>;
    };
};

• Configure pins for i2c.

&i2c1_default {
	group1 {
		psels = <NRF_PSEL(TWIM_SDA, 1, 15)>, <NRF_PSEL(TWIM_SCL, 1, 14)>;
	};
};

&i2c1_sleep {
	group1 {
		psels = <NRF_PSEL(TWIM_SDA, 1, 15)>, <NRF_PSEL(TWIM_SCL, 1, 14)>;
		low-power-enable;
	};
};

&i2c1 {
	compatible = "nordic,nrf-twi";
	status = "okay";

	pinctrl-0 = <&i2c1_default>;
	pinctrl-1 = <&i2c1_sleep>;
	pinctrl-names = "default", "sleep";

...
};

• Configuration of pins for UART is analogue.
• Logging over USB and disabling unused peripherals is the same as for the Feedback Light.

Project Configuration

Getting the configuration right took some time. There are several entries. In general, when configuring a feature, jump to the Kconfig files and read all possible configuration items to understand them and their dependencies.

Selected Explanations

• Mind your stack sizes. Start large to avoid losing time debugging crashes. Optimise when needed with the stack monitor feature. Since callbacks like for Bluetooth operate on the Bluetooth stack, it is good practice to schedule tasks for more extensive logic/calculations.
• I restricted the max BT connections (CONFIG_BT_MAX_CONN) to the minimum of expected connections, which is 1 for the bridge and 1 for the Memfault forwarder app.
• Maximise packet sizes for faster BT transfer of extensive data. Please look at the Bluetooth section in the configuration code to increase the packet size.

Selected Explanations of the Source Code

Handling Sensor Data

• Sensors are not accessed directly but through the sensor aggregator. It’s interface is init() and collectData(struct posture_data)

class SensorAggregator {
 public:
  SensorAggregator(SensorInterface* lsm, SensorInterface* bno, SensorInterface* flex);

  int collectData(struct posture_data* data);

 private:
  int init();
  std::vector<SensorInterface*> sensors;
};

• The sensor interface

class SensorInterface {
 public:
  virtual int init() = 0;
  virtual int getReading(struct posture_data*) = 0;
};

• There is a driver for the ADS1X1X available in Zephyr. However, it supports only reading one channel at a time. That is why I again configure the channel for each reading with the correct pin I want to read inside readPin.

int16_t SensorFlex::readPin(uint8_t pin) {
  int16_t buffer = -1;
  if (ads1015 != NULL) {
    const struct adc_channel_cfg ch_cfg = {
        .gain = ADC_GAIN_1,  // ADS1X1X_CONFIG_PGA_2048
        .reference = ADC_REF_INTERNAL,
        .acquisition_time = ADC_ACQ_TIME_DEFAULT,  // ADS1X1X_CONFIG_DR_128_1600
        .channel_id = 0,
        .differential = 0,      //Single-ended input, no differential
        .input_positive = pin,  // Activate single-ended input for pin
        .input_negative = 0     // Only relevant for differential input
    };

    const struct adc_sequence seq = {.options = NULL,
                                     .channels = BIT(0),
                                     .buffer = &buffer,
                                     .buffer_size = sizeof(buffer),
                                     .resolution = 11,
                                     .oversampling = 0,
                                     .calibrate = 0};

    // always reads ADS1X1X_CONFIG_MODE_SINGLE_SHOT
    adc_channel_setup(ads1015, &ch_cfg);
    adc_read(ads1015, &seq);
  }
  return buffer;
}

• There is no driver for BNO08x available in Zephyr. I decided to implement the driver given by the vendor in user space. Integrating the driver into Zephyr is quite complex. It also needs maintenance for each Zephyr upgrade. An out-of-tree driver would be nice, but it is even more effort. By having it in user space, I can copy it from project to project without worrying about losing the code by reinstalling an NCS (deleting old versions due to disk space).
• To interface the vendor’s library, it is required to implement the OS abstraction to use the i2c API and get the time in Zephyr. I more or less map the functions to the i2c_* functions in Zephyr. For the timer, I use k_uptime_get. However, there is more logic, so I looked at how Adafruit and Sparkfun implemented and took over the code. Sparkun’s library is also re-using the Adafruit code.

static int i2chal_write(sh2_Hal_t *self, uint8_t *pBuffer, unsigned len) {
  size_t i2c_buffer_max = I2C_MAX_BUFFER_SIZE;

  uint16_t write_size = MIN(i2c_buffer_max, len);
  if (i2c_write(i2c_dev, pBuffer, write_size, BNO08x_I2CADDR_DEFAULT)) {
    return 0;
  }

  return write_size;
}

static uint32_t hal_getTimeUs(sh2_Hal_t *self) {
  int64_t t = k_uptime_get() * 1000;
  return (uint32_t)t;
}

• The persistence module is inspired by the littlefs sample. I rewrote it to make it reusable across projects.

Bluetooth

• The overall structure is taken from the Nordic Academy Bluetooth Course. So, I skip explaining the basics here. Please take the course. The explanations are excellent.
• I created two custom services. One is timeService, which sets and gets the time as Unix time in seconds. The second, postureCheckerService, is for subscribing to the posture score, reading min/max values and sending a configuration (the last is ignored). Their behaviour on write is injected by callbacks from the main.c.
• The posture score has no permissions, meaning it cannot be read. It can only be subscribed to. The CCC requires an encrypted connection. This switches to LESC (low energy secure connection) connections. For some unknown reason, setting BT_GATT_PERM_READ_LESC does not change to LESC. No idea why.

BT_GATT_SERVICE_DEFINE(
    pcs_svc, BT_GATT_PRIMARY_SERVICE(BT_UUID_PCS),
...
    BT_GATT_CHARACTERISTIC(BT_UUID_PCS_SCORE_MEA, BT_GATT_CHRC_NOTIFY, BT_GATT_PERM_NONE, NULL,
                           NULL, &mea_value),
    BT_GATT_CCC(pcslc_ccc_cfg_changed, BT_GATT_PERM_READ_ENCRYPT | BT_GATT_PERM_WRITE_ENCRYPT),
...
    BT_GATT_CHARACTERISTIC(BT_UUID_PCS_CONFIG, BT_GATT_CHRC_WRITE, BT_GATT_PERM_WRITE_AUTHEN, NULL,
                           apply_config, NULL), 
);

• Before sending the value, I check the security flags for OOB and reject it if it is not set. I also added an automatic search for the attribute to avoid hardcoding the index of the attributes. It changes when changing the service definition.

int bt_pcs_send_score(struct bt_conn *conn, uint16_t score) {
  struct bt_gatt_notify_params params = {0};

  // Avoid magic numbers in sync with BT_GATT_SERVICE_DEFINE here
  // search for the attribute to be changed.
  // We know it should have the user_data of mea_value.
  for (int i = 0; i <= pcs_svc.attr_count; i++) {
    if (pcs_svc.attrs[i].user_data == &mea_value) {
      params.attr = &pcs_svc.attrs[i];
      break;
    }
  }
  params.data = &score;
  params.len = sizeof(score);
  params.func = NULL;

  mea_value = score;

  struct bt_conn_info info;
  bt_conn_get_info(conn, &info);

#if defined(CONFIG_APP_REQUIRE_OOB)
  if ((info.security.flags & BT_SECURITY_FLAG_OOB) == 0) {
    return -BT_ATT_ERR_AUTHENTICATION;
  }
#endif

  if (!conn) {
    return -ENODEV;
  } else if (bt_gatt_is_subscribed(conn, params.attr, BT_GATT_CCC_NOTIFY)) {
    LOG_INF("bt_pcs_send_score %d", score);
    return bt_gatt_notify_cb(conn, &params);
  } else {
    return -EINVAL;
  }
}

• I register various callbacks to print information about connection and authentication, remote pairing features, identity resolved, and physical parameter updates.
• For the OOB implementation, I took the shell implementation as an example: zephyr/subsys/bluetooth/shell/bt.c. As an exchange format, I took the format of how the shell prints the OOB info in print_le_oob(). The OOB keys are generated by calling bt_le_oob_get_local(). For other devices to connect with OOB, their OOB data and the bt_le_oob_set_sc_flag(true) are required.

struct bt_le_oob oob_local;

void local_oob_get(char *buffer, uint16_t size) {
  int err = bt_le_oob_get_local(BT_ID_DEFAULT, &oob_local);
...
}

• Why not use NFC for OOB? Why UART? The only supported module with drivers is the NFC Reader ST25R3911B Nucleo expansion board (X-NUCLEO-NFC05A1). This module is deprecated and not certified for usage in Japan (NFC devices don't need certification, but the user must prove it emits below a specific strength - I need more RF knowledge and have no expensive RF measurement equipment). The only module with an open driver, sample code and certification I found was a 10-year-old end-of-life FeliCa reader writer RC-S620S. It was hard to get... after some months, I really found a used one..., and I broke it by applying reverse polarity... end of the NFC story. You can think of a docking station for pairing now... not end of the story. I found out that it got the USB version which has the same prints and part number on the front. I figured out how to use it. See the chapter for the NFC Reader.
• Once bonded, advertising will be started with a filter of bonded devices until a shell command erases bonding.
• As there is no input capability, I confirm the passkey

EI Wrapper

Others (USB, UART, shell, EI data forwarder)

• The UART module sets up a receiving thread that reads line by line and tries to parse OOB data in the format "<bt address><space><address type><space><key><space><key>" (it is the same as the Bluetooth shell uses). After recognising this format, set_remote_oob is called, and local_oob_get is used to get the local OOB to send it back.
• The USB module is waiting for a serial console to be attached. The reason is that USB logging skips messages directly after startup. I used CoolTerm in an old version that did not have auto-reconnect. After updating it, coolTerm can not auto-reconnect. It also blinks the LED. I know it is rebooting.
• Shell commands for debugging are registered. They are get/set Unix time, start pairing without filter, remove bonds, set remote OOB data, delete the file from the filesystem, dump sensor data, reboot, etc.
• The EI Data Forwarder is mainly taken from the example. It works for UART and NUS (Nordic UART service for Bluetooth). I just adapted the formatting of the data format and left most of the code.

How to mount the littlefs

• Mac users must install the following software to enable user space file system mounts. Home - macFUSE. This would be the source code of the project osxfuse/osxfuse at support/osxfuse-3
• Check out the littlefs-project/littlefs-fuse: A FUSE wrapper that puts the littlefs in user-space project.
• Merge Added Mac OS X support by desertkun · Pull Request #19 · littlefs-project/littlefs-fuse. Essentially, this links against the installed libosxfuse. I needed to change override LFLAGS += -losxfuse to override LFLAGS += -losxfuse.1 and BLKGETSIZE64 to BLKGETSIZE to compile. Libosxfuse is found inside /usr/local/lib. Please check there how your file is called.
• To mount after attaching the device, assuming it is called diskX, where X is the highest number / latest attached device, use.

mkdir /Users/jens/lfs
sudo chmod a+rw /dev/disk6
./lfs -d --read_size=16 --prog_size=16 --block_size=4096 --block_count=512 --cache_size=64 --lookahead_size=32 /dev/disk6 /Users/jens/lfs

# The parameter must match the values in the board overlay file!
    fstab {
        ...
        lfs1: lfs1 {
            ...
            read-size = <16>;
            prog-size = <16>;
            cache-size = <64>;
            lookahead-size = <32>;
            ...
        };
    };

Where precisely do the block_size and block_count come from? It is printed out in the sample USB Mass Storage Sample Application — Zephyr Project documentation (nRF Connect SDK)

• I assume from CONFIG_NORDIC_QSPI_NOR_FLASH_LAYOUT_PAGE_SIZE=4096
• block_count can be calculated by disk size / block_size
• Unmount with umount /Users/jens/lfs

Memfault Integration

Memfault is a complete device monitoring solution that supports debugging and monitoring (and much more) of devices. If interested, look at Memfault integration — nRF Connect SDK 2.5.99 documentation and at this Introduction. I will not go into details here.

Alternatives

Flex sensors: Leaving them out will likely give similar posture scores.

BNO085: You can try to use the lsm4ds3tr and feed these into Edge Impulse.

XIAO BLE: You can use another nRF52840 board. If there is no external Flash, leave out writing the sensor data to Flash and use the EI Forwarder to collect training data.

Matter Bridge

Relation to the next chapter Bridge - Reminders Using Generative AI for Control: This chapter focuses on the bridge use case and the next on the reminders use case. The reminders use case was written in the extended timeline of the contest.

Summary

The bridge connects the Bluetooth-enabled sensors to the Matter home network, making them available for control with the Matter controller (Apple Home, Google Home, chip-tool, etc.). It can be easily deployed to any nRF52/53 board, but due to the concurrency of BT and Matter (OT/Wifi), usage of Wifi with the nRF7002 companion is recommended. The bridge is implemented from scratch with the architectural principles of generic reusability and the capability of handling privacy- and OOB security-enabled connections. To achieve reusability, application logic should be injected into the bridge by callbacks and runtime configuration like the BT to Matter mapping. Hence, configuring another BT device and another Matter cluster does not require code changes of the Bridge code—only the configuration in the App Task.

The print is a Swirly Mounting Grid for 0.1" PCBs byJMcK. The design is based on Adafruit Swirly Aluminum Mounting Grid for 0.1" Spaced PCBs. The purpose is to fix the parts and tidy up.

Please note that the Matter Bridge on the Switch Science board was last running for version 1. Please start with the Git commit for version 1. However, Some fixes are not merged, yet.

Motivation

With nRF Connect SDK version 2.5.0, Nordic offers a Matter bridge implementation (documentation). So why the implementation from scratch?

• When I started the implementation with version 2.4.x, the Matter bridge had yet to be released and was still under construction. This means at that time documentation was sparse, and I encountered several bugs that have been fixed around the same time I fixed them locally (I did not see the open merge requests earlier. I could even contribute with a tiny hint in the reviews.) Also, the implementation changed frequently.
• The Matter bridge does not support privacy (handling resolvable private addresses), so reconnects failed because the random address is saved, not resolved. I tried to add it, but the program flow differs for initial connect and reconnect. It was hard to find a small patch.
• Adding new sensors required significant code changes in the app (mainly adding new providers and matter devices). It looked like copying code (existing adapters, etc) with only minor adaptions (a different service UUID, etc.).

Addition: Several months have passed. Check out Nordic's Matter bridge and see if it fits your requirements.

Parts Selection

If you use the nRF7002dk, there is nothing else you need.

I started with the Switch Science nRF5340 MDBT53-1M board because the used Raytac module is, to my knowledge, the only one that is certified nRF5340 board for usage in Japan (R 201-210873). Since this board does not have an external flash, I selected the W25Q128JV sold by Adafruit because it is on a breakout board, and I can use regular pin headers.

Hardware Assembly

If you use the nRF7002dk, the uart1 default pins are (RX) P1.00 and (TX) P1.01. Please use level shifters if the other board does not run at 1.8V.

I only attached QSPI flash Adafruit QSPI DIP Breakout Board - W25Q128 (not required on Nordic DKs).

For the first Bluetooth pairing with the posture checker over UART OOB, connect the UART pins to the posture checker's UART pins. Alternatively, you can use NFC or the shell commands to transfer the OOB data printed out in the logs manually. (Or turn OOB off)

Software Part

Component Diagram

Bridge use case view - parts for the reminders use case greyed out.

OOB Exchange Manager sends the local OOB data and receives the remote OOB data in return.

Ble Device discovers attributes and subscribes or reads them.

Ble Connectivity Manager scans for and connects to devices. Returns a connection.

Matter Device handles the interface to Matter. Base class for Matter Device BLE and Matter Device Fixed.

Matter Device BLE uses the Ble Connectivity to connect to the device and the Ble Device to subscribe attributes.

Matter Device Fixed implements a dynamic Matter device for the Reminder App

Bridge Manager initialises and adds the dynamic cluster. It forwards requests to the correct Matter Device.

Nfc Uart handles the communication with the NFC reader.

Bridge Shell makes commands for debugging available, like removing the BT bond, setting remote OOB, restarting, etc.

App Task runs the main application that initialises the Matter stack and the bridge. It also handles user interaction with buttons or sets the status LED. It runs an event loop reacting to App Events.

Sequence Diagram

Please use the sequence diagram in combination with the descriptions below to navigate and understand the source code.

Device Configuration

Board Overlay (ssci_mdbt53_dev_board)

If you use the nRF7002dk, you can just skip this section.

• The vendor provides the board files for the ssci_mdbt53_dev_board. See usage guide Board Definition Files Download. The board files are an updated copy of the thingy53 files. There are some issues. First, it defines the timer2 in the netcore, which leads to the error below.

warning: NRF_802154_RADIO_DRIVER (defined at modules/hal_nordic/Kconfig:21, modules/hal_nordic/Kconfig:21) has direct dependencies (HAS_HW_NRF_RADIO_IEEE802154 && !y && HAS_NORDIC_DRIVERS) || (HAS_HW_NRF_RADIO_IEEE802154 && !y && HAS_NORDIC_DRIVERS && 0) with value n, but is currently being y-selected by the following symbols:
 - NRF_802154_SER_RADIO (defined at /opt/nordic/ncs/v2.4.0/modules/lib/matter/config/nrfconnect/chip-module/Kconfig.multiprotocol_rpmsg.defaults:75, modules/hal_nordic/Kconfig:157, modules/hal_nordic/Kconfig:157), with value y, direct dependencies y (value: y), and select condition HAS_HW_NRF_RADIO_IEEE802154 && !IEEE802154_NRF5 && HAS_NORDIC_DRIVERS (value: y)

Deleting the timers at child_image/multiprotocol_rpmsg/boards/ssci_mdbt53_dev_board_cpunet.overlay

/delete-node/ &timer0;
/delete-node/ &timer1;
/delete-node/ &timer2;

• Second, the board does not have any external flash like the mx25r64.

The configuration for mcuboot was not appropriately applied in the specified folder (added invalid syntax which did not break the build), so I copied it into my project under child_image/mcuboot/boards/ssci_mdbt53_dev_board_cpuapp.conf.

• I defined alternate pins for uart0 and used uart0 for logging and shell. Uart1 is used for OOB.
• I defined QSPI and external flash w25q128jv, which I connected to the board. QSPI uses dedicated pins that may not be exchanged according to the nRF5340 documentation. Please read the comments on the following code to see how I found the correct configuration. Also, read the descriptions at zephyr/dts/bindings/mtd/nordic, qspi-nor.yaml.

&qspi {
	status = "okay";
	pinctrl-0 = <&qspi_default>;
	pinctrl-1 = <&qspi_sleep>;
	pinctrl-names = "default", "sleep";
	w25q128jv: w25q128jv@0 {
		compatible = "nordic,qspi-nor";
		reg = < 0 >;
		sck-frequency = <8000000>;
		// Check the datasheet for the supported commands by the command id.
		writeoc = "pp4o";   // Quad data line SPI, PP4O (0x32)
		readoc = "read4io"; // Quad data line SPI, READ4IO (0xEB)
		// Property value must be one of NONE, S2B1v1, S1B6, S2B7, S2B1v4, S2B1v5, S2B1v6
		// The QE is register 2 bit 1
		// I found the appropriate values by running the sample
		//   zephyr/samples/drivers/jesd216. It will print out the values below.
		quad-enable-requirements = "S2B1v4";
		jedec-id = [ef 40 18];
		sfdp-bfp = [
			e5 20 f9 ff  ff ff ff 07  44 eb 08 6b  08 3b 42 bb
            fe ff ff ff  ff ff 00 00  ff ff 40 eb  0c 20 0f 52
            10 d8 00 00  36 02 a6 00  82 ea 14 c9  e9 63 76 33
            7a 75 7a 75  f7 a2 d5 5c  19 f7 4d ff  e9 30 f8 80
		];
		size = < DT_SIZE_M(16*8) >;
		// Check the datasheet to see if command DPD (0xB9) is supported.
		has-dpd;
		t-enter-dpd = < 3500 >;
		t-exit-dpd = < 3500 >;

		// https://developer.nordicsemi.com/nRF_Connect_SDK/doc/latest/nrf/scripts/partition_manager/partition_manager.html#ug-pm-static-providing
		// When you build a multi-image application using the Partition Manager, the device tree source flash partitions are ignored.
		//
		// Littlefs uses a partition called littlefs_storage.
		// Memfault uses memfault_storage.
		// Settings uses settings_storage.
		// See nrf/subsys/partition_manager/pm.yml.*
		
		partitions {
			compatible = "fixed-partitions";
			#address-cells = <1>;
			#size-cells = <1>;
			lfs1_part: partition@0 {
				label = "flash-storage";
				reg = <0x00000000 0x01000000>;
			};
		};
	};
};

• I deactivated unused peripherals.

Project Configuration

• Initially, I used configuration fragments to separate Bluetooth, Thread, and USB configurations. However, features impose different requirements on the same configuration, like CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE. If you accidentally overwrite to a smaller value in the fragments, it takes some time to discover why the application crashes. Therefore, fragments are not independent of each other, and since they are always active in my case, I merged them into one prj.conf. Furthermore, I had a situation where configuration in the board-dependent configuration file was not applied for some items.
• I increased some buffers and stack sizes.
• (Outdated with SDK 2.5.1 as there is no bootloader without OTA) The multiprotocol_rpmsg and its bootloader do not fit into the netcpu's flash memory when adding the BT central role. Therefore, I had to decrease the flash usage of the multiprotocol_rpmsg and its bootloader build multiprotocol_rpmsg_b0n as much as possible.
• I am adding the BT central role in both application and netcore.

CONFIG_BT_MAX_CONN=2
CONFIG_BT_CENTRAL=y

• The application defines two configurations.

CONFIG_BRIDGE_BT_RECOVERY_INTERVAL_MS=10000
CONFIG_BRIDGE_BT_RECOVERY_SCAN_TIMEOUT_MS=2000

• Please check the prj.conf and the comments inside the file for further configurations.

CMakeLists

CMakeLists is following the Matter samples with the following additions:

• Setting the minimal size overlay for the bootloader on the net core.

list(APPEND multiprotocol_rpmsg_b0n_OVERLAY_CONFIG overlay-minimal-size.conf)

• setting additional compile options for my application. Here, you can turn warnings into errors for your application code only. Due to flash constraints "-0g" has been removed again.

zephyr_library_compile_options(-Og -std=gnu++20 -Wno-volatile -Werror)

Challenges Encountered / Selected Explanations of the Source Code

• Using Thread and Bluetooth in parallel on nRF5340 lead to occasional crashes when doing a BT scan. See Multiprotocol limitations in application development. The issue was that I was not fully in control (and aware) of what and when the Matter used Thread. This issue was solved by adding the following to the board overlay.

/* Set IPC thread priority to the highest value to not collide with other threads. */
&ipc0 {
    zephyr,priority = <0 PRIO_COOP>;
};

• Using member functions of class instances as callbacks is impossible since the BT stack is written in c. Some callbacks, like the discovery complete callback, have user data (void* context) in their signature, which can take the instance as a pointer. Others, like the scan result callback and subscription callback, do not. The only solution I found is to maintain a mapping with information available in the callback, like the bt_conn, and the class instances, like BleDevice, in case of the discovery. In the case of the scan, I have one global struct scanConnectState scanState leveraging the fact that the system can only do one scan at a time.

// A mapping table
std::map<bt_conn *, BleDevice *> BleDevice::instances;

// A static function of BleDevice to get the right instance
  static BleDevice *Instance(bt_conn *conn) { return instances[conn]; }

// A static class function that can be used as a callback for BT Stack API
static void DiscoveryCompletedHandlerEntry(bt_gatt_dm *dm, void *context) {
  BleDevice *dev = BleDevice::Instance(bt_gatt_dm_conn_get(dm));
  dev->DiscoveryCompletedHandler(dm, context);
}

// The member function that handles the callback.
void BleDevice::DiscoveryCompletedHandler(bt_gatt_dm *dm, void *context) {
  LOG_INF("The GATT discovery completed");
...
}
struct scanConnectState {
    bool scanning;
    void *ctx;
    ScanCallback cb;
    struct bt_uuid *serviceUuid;
    struct bt_conn *conn;
    const bt_addr_le_t *addr;
  };

  // The purpose of this struct is 
  //   - to have the context, callback and service uuid during the (Dis)connected callbacks.
  //   - to react only to (dis)connects initiated by the bridge manager.
  struct scanConnectState scanState;

• Additionally, in the subscription callback, I do not have the information of the attributes UUID that send the update, only a handle. Therefore, I need to manually discover attributes in advance and build up a mapping table of attribute handles to uuid. There is no function in the BT stack to get the UUID by the handle. Additional complexity of the mapping table was that bt attributes could be of the type bt_uuid_16, bt_uuid_32 or bt_uuid_128.

std::map<uint16_t, struct bt_uuid *> mHandleToUuid;

void BleDevice::DiscoveryCompletedHandler(bt_gatt_dm *dm, void *context) {
  LOG_INF("The GATT discovery completed");
  bt_gatt_dm_data_print(dm);

  BleDevice *dev = BleDevice::Instance(bt_gatt_dm_conn_get(dm));

  const struct bt_gatt_dm_attr *attr = NULL;
  while (NULL != (attr = bt_gatt_dm_attr_next(dm, attr))) {
    if (attr->uuid->type == BT_UUID_TYPE_16) {
      auto *uuid = chip::Platform::New<bt_uuid_16>();
      *uuid = *(BT_UUID_16(attr->uuid));
      mHandleToUuid[attr->handle] = reinterpret_cast<bt_uuid *>(uuid);
    } else if (attr->uuid->type == BT_UUID_TYPE_32) {
      auto *uuid = chip::Platform::New<bt_uuid_32>();
      *uuid = *(BT_UUID_32(attr->uuid));
      mHandleToUuid[attr->handle] = reinterpret_cast<bt_uuid *>(uuid);
    } else if (attr->uuid->type == BT_UUID_TYPE_128) {
      auto *uuid = chip::Platform::New<bt_uuid_128>();
      *uuid = *(BT_UUID_128(attr->uuid));
      mHandleToUuid[attr->handle] = reinterpret_cast<bt_uuid *>(uuid);
    }
  }

  bt_gatt_dm_data_release(dm);
...

• There is no synchronous read function for reading attributes. So, I added a signal to the read function and called k_poll with a timeout to wait for the result.

void BleDevice::GATTReadCallback(bt_conn *conn, uint8_t att_err, bt_gatt_read_params *params,
                                 const void *data, uint16_t read_len) {
...
  k_poll_signal_raise(&mGattReadSignal, 1);
}

bool BleDevice::Read(struct bt_uuid *uuid, uint8_t *buffer, uint16_t maxReadLength) {
...
  k_poll_signal_init(&mGattReadSignal);
  k_poll_event_init(mGattWaitEvents, K_POLL_TYPE_SIGNAL, K_POLL_MODE_NOTIFY_ONLY, &mGattReadSignal);
  k_poll(mGattWaitEvents, ARRAY_SIZE(mGattWaitEvents), K_SECONDS(3));
  k_poll_signal_reset(&mGattReadSignal);
...

• I needed to make changes inside the Matter SDK to enable the privacy feature. The Matter SDK sets a static Bluetooth address. However, I need resolvable private addresses for the privacy feature to work. So, I prevent Matter from setting a random static address.

diff --git a/src/platform/Zephyr/BLEManagerImpl.cpp b/src/platform/Zephyr/BLEManagerImpl.cpp
index 8a7877aab3..97b7a07c75 100644
--- a/src/platform/Zephyr/BLEManagerImpl.cpp
+++ b/src/platform/Zephyr/BLEManagerImpl.cpp
@@ -113,6 +113,8 @@ CHIP_ERROR InitRandomStaticAddress()
     int error = 0;
     bt_addr_le_t addr;
 
+ return CHIP_NO_ERROR;
+ 
  // generating the address
  addr.type = BT_ADDR_LE_RANDOM;
  error     = sys_csrand_get(addr.a.val, sizeof(addr.a.val));

To implement a Matter bridge, the following steps are required as interactions with the Matter SDK (all done inside src/bridge/bridge_manager.cpp)

• Initialisation Register a dynamic endpoint using emberAfSetDynamicEndpoint. The parameters describe the Matter endpoint with its clusters, attributes and commands. (For the definitions of this app, see src/bridge/config.inc) The endpoint is described by app code. The ZAP tool is not required here.
• Matter → Device Communication Handle read and write requests from Matter for the dynamic endpoint by implementing emberAfExternalAttributeReadCallback and emberAfExternalAttributeWriteCallback.
• Device → Matter Communication Notify the Matter that an attribute value has changed by calling MatterReportingAttributeChangeCallback with the path to the attribute as a parameter.

I took the sample at modules/lib/matter/examples/bridge-app/linux/main.cpp for reference. It is a minimal bridge sample.

Bridge - Reminders Using Generative AI for Control

Summary

Reminders address the "Getting homework done after school, which targets children at home." use case. The core functionality is to manage reminders, which means adding, removing and checking them for the remaining time. Reminders can be added by voice via open.AI's Whisper and Completions (ChatGPT) API, by Shell or are hard-coded on init (only homework). They can be cleared by Felica cards on the NFC card reader (only homework), by voice or by Shell. The remaining time to the closed event is converted to a level value and reported to a Matter Level Control Cluster periodically. The level is expected to be converted to a colour like the posture feedback and sent to a Matter Color Light Bulb.

Feedback from the voice control is written on the display to check if the interpretation was correct since both Whisper and Completions do not deliver reliable results.

Reminders are persisted and audio data is buffered to a file on the filesystem.

Motivation

The original plan was to connect to a user's calendar like Google Calendar or the iOS Calendar. However, due to increasing security awareness and the resulting complexity (like short-lived authentication tokens or OAuth), it is quite hard to interface these services on microcontrollers. Furthermore, APIs are limited to the respective ecosystems.

Interfacing new technologies like generative AI seemed to be exciting to explore how to bring new features to resource-constrained microcontrollers. Therefore I decided to try these out.

Parts Selection

There are various kinds of microphone interfaces like analogue, pulse width modulation (PDM) or I2S. After some research to choose between PDM or I2S, I did not see a strong reason towards one of them. Therefore, I chose PDM because of its available driver and past experience. The PDM microphone works with 1.8V, too.

The display and level shifters are recycled parts from a previous project.

Hardware Assembly

I attached the microphone according to the pin assignment of the overlay. Note that the display runs 3.3V logic and is out of specification for the nRF7002dk. However the ST7789 controller works fine with 1.8V logic input. Note that the SI is not connected and therefore no input pin on the nRF7002dk side.

Software Part

Component Diagram

Reminders use case view - parts for the bridge use case greyed out.

RemindersApp is the entry point and encapsulates the reminders logic behind its interfaceThe Recorder transfers the audio data from the PDM interface to a file on QSPI flash by the Persistence which is taken over from Posture with minor changes. The recorder runs its own thread.RemindersApp uploads the file to OpenAI's Whisper API over Wi-Fi.RemindersApp uploads the transcribed text to OpenAI's Completions API.RemindersApp parses the received command in JSON format and calls the function of the Reminder.

Challenges encountered

• The microphone's sensitivity is very low. The gain is not configurable. Therefore I switched to nrfx drivers instead of Zephyr's dmic. I took the Thingy:53 firmware implementation as a reference.
• The microphone produces a crack after opening it. Therefore, I skip the first chunk of audio.
• 16 kHz raw audio needs large buffers. Therefore I reduced to 8 kHz.
• The smaller the chunks, the longer the filesystem processing takes. I settled with 1.5s chunks.
• OpenAI accepts only specific audio formats, no raw PCM audio. Therefore, I added a wave header at the beginning of the audio file. This was the simplest solution and does not require audio codecs.
• The SSL connection was very challenging. I figured out that a Baltimore CyberTrust Root Certificate Authority is the correct one by analyzing the certificate chain with a browser. Click on the lock icon, to see details and to download the certificate.

• I experimented a lot with MbedTLS configurations. I conclude that the important setting were CONFIG_MBEDTLS_RSA_C, CONFIG_MBEDTLS_SSL_SERVER_NAME_INDICATION and sufficient heap memory. Please find the resulting configuration in the Project Configuration section.
• Both Flash and RAM used 99% after tuning.

Compared to littlefs on the posture checker, there are some pitfalls, which are

• DISK_NAME must be one of "RAM", "NAND", "CF", "SD", "SD2", "USB", "USB2", "USB3"
• storage_dev is not used in the mount point struct, so "(uintptr_t) FIXED_PARTITION_ID(ffs1)" is used to open and initially erase the flash partition.
• The multi image build requires the definition of a static partition for the partition manager with extra parameter.
• Filenames are limited 8 characters unless FS_FATFS_LFN is enabled.
• Extra precaution is required to keep the parameters in sync between the partition manager, the device tree overlay and project configuration.

pm_static_nrf7002dk_nrf5340_cpuapp.yml

ffs1:
  address: 0x0
  size: 0x100000
  device: mx25r64
  region: external_flash
  affiliation:
  - disk
  extra_params:
    disk_cache_size: 0x1000
    disk_name: NAND
    disk_read_only: 0x0
    disk_sector_size: 0x200

Device Configuration

Board Overlay

• Add the PDM microphone

&pinctrl {
    pdm0_default_alt: pdm0_default_alt {
        group1 {
            psels = <NRF_PSEL(PDM_CLK, 1, 11)>,
                <NRF_PSEL(PDM_DIN, 1, 12)>;
        };
    };
};

dmic_dev: &pdm0 {
    status = "okay";
    pinctrl-0 = <&pdm0_default_alt>;
    pinctrl-names = "default";
    clock-source = "PCLK32M_HFXO";
};

• Add the FAT filesystem as USB Mass storage

/ {
    chosen {
        zephyr,flash-controller = &mx25r64;
        nordic,pm-ext-flash = &mx25r64;
    };

    msc_disk0 {
        compatible = "zephyr,flash-disk";
        partition = <&ffs1>;
        // disk-name must be one of "RAM","NAND","CF","SD","SD2","USB","USB2","USB3"
        disk-name = "NAND";
        cache-size = <4096>;
        // when changing sector-size, adapt CONFIG_FS_FATFS_MAX_SS or crash
        sector-size = <512>;
    };
};

&mx25r64 {
    partitions {
        compatible = "fixed-partitions";
        #address-cells = <1>;
        #size-cells = <1>;

        ffs1_part: partition@0 {
            label = "ffs1";
            reg = <0x00000000 0x00800000>;
        };
    };
};

• Add the Display configuration

// Definition of the SPI pins for the display
&spi3_default {
    group1 {
        psels = <NRF_PSEL(SPIM_SCK, 1, 15)>,
            <NRF_PSEL(SPIM_MISO, 1, 14)>,
            <NRF_PSEL(SPIM_MOSI, 1, 13)>;
    };
};

// Definition of the display
display_spi: &spi3 {
    status = "okay";
    cs-gpios = <&gpio0 27 GPIO_ACTIVE_LOW>;

    st7789v: st7789v@0 {
        compatible = "sitronix,st7789v";
        spi-max-frequency = <20000000>;
        reg = <0>;
        cmd-data-gpios = <&gpio1 2 GPIO_ACTIVE_LOW>;
// Use soft reset to save a pin
//		reset-gpios = <&gpio1 2 GPIO_ACTIVE_LOW>;
        width = <240>;
        height = <240>;
        x-offset = <0>;
        y-offset = <0>;
        vcom = <0x19>;
        gctrl = <0x35>;
        vrhs = <0x12>;
        vdvs = <0x20>;
        mdac = <0x00>;
        gamma = <0x01>;
        colmod = <0x05>;
        lcm = <0x2c>;
        porch-param = [0c 0c 00 33 33];
        cmd2en-param = [5a 69 02 01];
        pwctrl1-param = [a4 a1];
        pvgam-param = [D0 04 0D 11 13 2B 3F 54 4C 18 0D 0B 1F 23];
        nvgam-param = [D0 04 0C 11 13 2C 3F 44 51 2F 1F 1F 20 23];
        ram-param = [00 F0];
        rgb-param = [CD 08 14];
    };

};

// Disable i2c1 to free up pins used for the display
&i2c1 {
    status = "disabled";
};

Project Configuration

The configuration is quite large. I will focus on selected points. Please read the comments of the prj.conf file

• This is the final MbedTLS configuration.

############
# MbedTLS and security
############
CONFIG_MBEDTLS=y
CONFIG_NET_SOCKETS_SOCKOPT_TLS=y
CONFIG_MBEDTLS_TLS_LIBRARY=y
CONFIG_MBEDTLS_X509_LIBRARY=y

# Required to connect to open.ai
CONFIG_MBEDTLS_RSA_C=y
CONFIG_MBEDTLS_SSL_SERVER_NAME_INDICATION=y

# These sizes depend on the length of the certificate (chain)
# They are required to establish ssl connection with open.ai
CONFIG_MBEDTLS_ENABLE_HEAP=y
CONFIG_MBEDTLS_SSL_IN_CONTENT_LEN=3072
CONFIG_MBEDTLS_SSL_OUT_CONTENT_LEN=3072
CONFIG_MBEDTLS_HEAP_SIZE=32768

• To resolve the host names to IP addresses for the socket API, I configured the Google DNS server

CONFIG_DNS_RESOLVER=y
CONFIG_DNS_RESOLVER_MAX_SERVERS=1
CONFIG_DNS_SERVER_IP_ADDRESSES=y
CONFIG_DNS_NUM_CONCUR_QUERIES=1
CONFIG_DNS_SERVER1="8.8.8.8"

• Configuration for the filesystem

############
# Flash & Filesystem
############
CONFIG_DISK_ACCESS=y
CONFIG_FILE_SYSTEM=y
CONFIG_FAT_FILESYSTEM_ELM=y
CONFIG_FS_FATFS_LFN=y
CONFIG_FS_FATFS_LFN_MODE_STACK=y
# Already set above for BT
#CONFIG_FLASH=y
#CONFIG_FLASH_MAP=y
#CONFIG_FLASH_PAGE_LAYOUT=y

CONFIG_DISK_DRIVER_FLASH=y

CONFIG_SPI_NOR_FLASH_LAYOUT_PAGE_SIZE=4096

# When the PM is active additional settings are required.
# Finally the FFS1 is defined as static partition in pm_static_nrf7002dk_nrf5340_cpuapp.yml
CONFIG_PM_OVERRIDE_EXTERNAL_DRIVER_CHECK=y
CONFIG_PM_PARTITION_SIZE_FFS1=0x100000
CONFIG_PM_PARTITION_REGION_FFS1_EXTERNAL=y

• To add custom partitions to the partition manager, include the Kconfig templates in the Kconfig file

partition=FFS1
partition-size=0x100000
source "${ZEPHYR_BASE}/../nrf/subsys/partition_manager/Kconfig.template.partition_config"
source "${ZEPHYR_BASE}/../nrf/subsys/partition_manager/Kconfig.template.partition_region"

new configuration items will become available

CONFIG_PM_OVERRIDE_EXTERNAL_DRIVER_CHECK=y
CONFIG_PM_PARTITION_SIZE_FFS1=0x100000
CONFIG_PM_PARTITION_REGION_FFS1_EXTERNAL=y

Memory Optimization

To add several new components like audio and display, it was required to optimize the Flash and RAM usage. These are the options to save Flash with an indication of how much

• Switching off asserts with configuration CONFIG_ASSERT=n saves about 18 KB.
• Switching off Shell functionality with configuration CONFIG_SHELL=y saves about 18 KB. Partial savings can be achieved by CONFIG_CHIP_LIB_SHELL=n and CONFIG_BT_SHELL=n.
• Switching off logging CONFIG_LOG=n saves about 22 KB. Partial savings can be achieved by lowering log levels e.g. by CONFIG_LOG_DEFAULT_LEVEL=3 and disabling the possibility of raising log levels during runtime by CONFIG_LOG_RUNTIME_FILTERING=n
• Exclusion of WPA advanced features with configuration CONFIG_WPA_SUPP_ADVANCED_FEATURES=n saves about 25 KB.
• The nRF7002 firmware patches use 62 KB. By moving them to the filesystem on the external flash, the flash could be saved. Implementation to load the patches is required. See below.
• Since SDK version 2.5.1, the bootloader is not included anymore when OTA is off. This saves 28 KB.
• By shrinking the storage partition and removing the factory data partition 31 KB + 4 KB can be saved. Securing the flash will be off. See Partition layout.
• Removing the USB stack with CONFIG_USB_DEVICE_STACK=n saves 15 KB. This removes the USB Mass storage functionality.
• LVGL can be optimised by CONFIG_LV_CONF_MINIMAL=y. This switches every feature off. Adding each feature manually is necessary

To save RAM, the following options are available

• Minimise the stack sizes to the required minimum (be aware of crashes and keep some buffer!!!)
• Set an appropriate heap memories (CONFIG_HEAP_MEM_POOL_SIZE, CONFIG_MBEDTLS_HEAP_SIZE, CONFIG_CHIP_MALLOC_SYS_HEAP_SIZE). To check heap usage, activate CONFIG_SYS_HEAP_RUNTIME_STATS and check bridge/src/util.cpp for how to print the statistics.
• Optimize the network buffers according to Operating with a resource constrained host — nRF Connect SDK 2.5.99 documentation
• Optimize other buffers like lvgl caches (CONFIG_LV_Z_VDB_SIZE), logging (CONFIG_LOG_BUFFER_SIZE) and buffers of your application.
• I reuse the heap memory I need for the nRF7002 firmware patch for the recorder's buffer.

Finally, to identify further optimizations, run the Memory report in NRF CONNECT: ACTIONS.

See also the documentation at

• Memory footprint optimization — nRF Connect SDK 2.5.99 documentation
• Matter hardware and memory requirements
• Bootloader configuration in Matter

Selected Explanations of the Source Code

nRF7002 Firmware Patches

I converted the c array into a binary file and stored it on the filesystem instead of the internal flash. I modified zephyr_fw_load.c to read these files instead. This is the patch.

diff --git a/drivers/wifi/nrf700x/zephyr/src/zephyr_fw_load.c b/drivers/wifi/nrf700x/zephyr/src/zephyr_fw_load.c
index 079d48db9..462cce994 100644
--- a/drivers/wifi/nrf700x/zephyr/src/zephyr_fw_load.c
+++ b/drivers/wifi/nrf700x/zephyr/src/zephyr_fw_load.c
@@ -23,7 +23,15 @@
 #endif /* CONFIG_NRF_WIFI_PATCHES_EXT_FLASH */

 #include <zephyr_fmac_main.h>
-#include <rpu_fw_patches.h>
+//#include <rpu_fw_patches.h>
+
+typedef void (*fs_read_cb_t)(void *data, uint16_t len_read, void* ctx);
+
+extern int fs_init(bool eraseFlash);
+extern int fs_readFile(const char *path, void *buf, uint16_t len, fs_read_cb_t cb, void* ctx);
+extern size_t fs_getFileSize(const char *path);
+
+extern void print_sys_memory_stats(void);

 enum nrf_wifi_status nrf_wifi_fw_load(void *rpu_ctx)
 {
@@ -33,15 +41,37 @@ enum nrf_wifi_status nrf_wifi_fw_load(void *rpu_ctx)
        const struct device *flash_dev = DEVICE_DT_GET(DT_INST(0, nordic_qspi_nor));
 #endif /* CONFIG_NRF_WIFI_PATCHES_EXT_FLASH */

+       print_sys_memory_stats();
+
+       fs_init(false);
+
+       size_t imac_pri_buf_len = fs_getFileSize("imac_pri");
+       size_t imac_sec_buf_len = fs_getFileSize("imac_sec");
+       size_t umac_pri_buf_len = fs_getFileSize("umac_pri");
+       size_t umac_sec_buf_len = fs_getFileSize("umac_sec");
+       printf("%s: nrf_wifi_fmac_fw_load allocate sizes %d %d %d %d\n", __func__,
+               imac_pri_buf_len, imac_sec_buf_len, umac_pri_buf_len, umac_sec_buf_len);
+       void* imac_pri_buf = k_malloc(imac_pri_buf_len);
+       void* imac_sec_buf = k_malloc(imac_sec_buf_len);
+       void* umac_pri_buf = k_malloc(umac_pri_buf_len);
+       void* umac_sec_buf = k_malloc(umac_sec_buf_len);
+       int imac_pri_read = fs_readFile("imac_pri", imac_pri_buf, imac_pri_buf_len, NULL, NULL);
+       int imac_sec_read = fs_readFile("imac_sec", imac_sec_buf, imac_sec_buf_len, NULL, NULL);
+       int umac_pri_read = fs_readFile("umac_pri", umac_pri_buf, umac_pri_buf_len, NULL, NULL);
+       int umac_sec_read = fs_readFile("umac_sec", umac_sec_buf, umac_sec_buf_len, NULL, NULL);
+       printf("%s: nrf_wifi_fmac_fw_load read sizes %d %d %d %d\n", __func__,
+               imac_pri_read, imac_sec_read, umac_pri_read, umac_sec_read);
+       print_sys_memory_stats();
+
memset(&fw_info, 0, sizeof(fw_info));
-       fw_info.lmac_patch_pri.data = (void *) nrf_wifi_lmac_patch_pri_bimg;
-       fw_info.lmac_patch_pri.size = sizeof(nrf_wifi_lmac_patch_pri_bimg);
-       fw_info.lmac_patch_sec.data = (void *) nrf_wifi_lmac_patch_sec_bin;
-       fw_info.lmac_patch_sec.size = sizeof(nrf_wifi_lmac_patch_sec_bin);
-       fw_info.umac_patch_pri.data = (void *) nrf_wifi_umac_patch_pri_bimg;
-       fw_info.umac_patch_pri.size = sizeof(nrf_wifi_umac_patch_pri_bimg);
-       fw_info.umac_patch_sec.data = (void *) nrf_wifi_umac_patch_sec_bin;
-       fw_info.umac_patch_sec.size = sizeof(nrf_wifi_umac_patch_sec_bin);
+       fw_info.lmac_patch_pri.data = imac_pri_buf;
+       fw_info.lmac_patch_pri.size = (unsigned int) imac_pri_read;
+       fw_info.lmac_patch_sec.data = imac_sec_buf;
+       fw_info.lmac_patch_sec.size = (unsigned int) imac_sec_read;
+       fw_info.umac_patch_pri.data = umac_pri_buf;
+       fw_info.umac_patch_pri.size = (unsigned int) umac_pri_read;
+       fw_info.umac_patch_sec.data = umac_sec_buf;
+       fw_info.umac_patch_sec.size = (unsigned int) umac_sec_read;

#if defined(CONFIG_NRF_WIFI_PATCHES_EXT_FLASH) && defined(CONFIG_NORDIC_QSPI_NOR)
        nrf_qspi_nor_xip_enable(flash_dev, true);
@@ -58,5 +88,11 @@ enum nrf_wifi_status nrf_wifi_fw_load(void *rpu_ctx)
        nrf_qspi_nor_xip_enable(flash_dev, false);
#endif /* CONFIG_NRF_WIFI */

+       k_free(imac_pri_buf);
+       k_free(imac_sec_buf);
+       k_free(umac_pri_buf);
+       k_free(umac_sec_buf);
+       print_sys_memory_stats();
+
        return status;
}

Since the filesystem is not ready at this stage, I modified the initialization priorities of the flashdisk to POST_KERNEL 71, mmc_subsys to 71 and fat_fs to 72.

diff --git a/drivers/disk/flashdisk.c b/drivers/disk/flashdisk.c
index 3e43716c2b..3c11977211 100644
--- a/drivers/disk/flashdisk.c
+++ b/drivers/disk/flashdisk.c
@@ -560,4 +560,4 @@ static int disk_flash_init(void)
        return err;
}

-SYS_INIT(disk_flash_init, APPLICATION, CONFIG_KERNEL_INIT_PRIORITY_DEFAULT);
+SYS_INIT(disk_flash_init, POST_KERNEL, 71);
diff --git a/drivers/disk/mmc_subsys.c b/drivers/disk/mmc_subsys.c
index 8af8a01991..869d3e4b61 100644
--- a/drivers/disk/mmc_subsys.c
+++ b/drivers/disk/mmc_subsys.c
@@ -127,7 +127,7 @@ static int disk_mmc_init(const struct device *dev)
                        &mmc_data_##n,                                  \
                        &mmc_config_##n,                                        \
                        POST_KERNEL,                                            \
-                       CONFIG_SD_INIT_PRIORITY,                                \
+                       71,                             \
                        NULL);

DT_INST_FOREACH_STATUS_OKAY(DISK_ACCESS_MMC_INIT)
diff --git a/subsys/fs/fat_fs.c b/subsys/fs/fat_fs.c
index a87bc880e9..32bfae03f6 100644
--- a/subsys/fs/fat_fs.c
+++ b/subsys/fs/fat_fs.c
@@ -527,4 +527,5 @@ static int fatfs_init(void)
        return fs_register(FS_FATFS, &fatfs_fs);
 }

-SYS_INIT(fatfs_init, POST_KERNEL, 99);
+// Must be before CONFIG_WIFI_INIT_PRIORITY
+SYS_INIT(fatfs_init, POST_KERNEL, 72);

Reminder

The reminders module is encapsulated behind the interface defined in src/reminders/reminders_app.h.

• Initialize the reminders app with initRemindersApp(feedback_text_t f1, feedback_text_t f2). f1 and f2 are callbacks for the feedback of the transcription results and completion results with the signature typedef void (*feedback_text_t)(const char *data). Initialization gets the time from an NTP server using the date_time module. Then it loads the persisted reminders from the filesystem or creates a daily homework reminder at 18:30. Finally it logs all reminders.
• Start the recording for the voice recognition flow by startAiFlow() and stop the recording by stopRecording(). This is connected to the Button 2's onPress and onRelease.
• Handle Felica card detection by cardTriggerAction(const char *name). This clears the pending homework reminder.
• The remaining time for the next due is checked by checkdue()
• Timers can be added or removed by addReminder(const char *name, const char *dueDate, bool daily) and deleteReminder(const char *name) by the Shell. These functions are also used internally for the voice flow and Felica card detection.
• printReminders() logs all reminders for debugging by the Shell

Reminders are simple consisting of the following attributes

• name of max. 23 characters
• due date as UNIX timestamp (seconds since Jan 1st 1970 as int64_t)
• a flag if the reminder is daily recurring.
• Time is internally managed as UTC time, but the input/output time zone is JST time (UTC +9).
• Whenever a daily reminder is cleared, a new reminder 24h later is added. The only way to delete it is to update it as non-daily and then remove it.
• Input format for the reminder's due date is a string in the format "YYYY-MM-DD hh:mm"

Completions / Whisper

• Sending HTTP POST data in multipart/form-data format took some time to understand. It is required to define a boundary in the headers and get the newlines correct. I came up with the following template.

#define BOUNDARY "Your_Boundary_String"
#define NEWLINE "\r\n"
#define FILENAME "audio.wav"
...

static const char *headers[] = {
    "Authorization: Bearer " OPENAI_API_KEY NEWLINE,
    "Content-Type: multipart/form-data; boundary=" BOUNDARY NEWLINE,
    NULL
};

static const char* post_start =
    "--" BOUNDARY NEWLINE 
    "Content-Disposition: form-data; name=\"model\"" NEWLINE NEWLINE "whisper-1" NEWLINE 
    "--" BOUNDARY NEWLINE 
    "Content-Disposition: form-data; name=\"language\"" NEWLINE NEWLINE "en" NEWLINE 
    "--" BOUNDARY NEWLINE 
    "Content-Disposition: form-data; name=\"response_format\"" NEWLINE NEWLINE "text" NEWLINE 
    "--" BOUNDARY NEWLINE
    "Content-Disposition: form-data; name=\"file\"; filename=\"" FILENAME "\"" NEWLINE
    "Content-Type: audio/wav" NEWLINE NEWLINE;

// audio data in between here

static const char* post_end = NEWLINE "--" BOUNDARY "--" NEWLINE;

• The HTTP request using the HTTP client in Zephyr.

struct http_request req;
    memset(&req, 0, sizeof(req));

    req.method = HTTP_POST;
    req.url = OPENAI_API_AUDIO_TRANSCRIPTION_ENDPOINT;
    req.host = OPENAI_API_HOST;
    req.protocol = "HTTP/1.1";
    req.payload_cb = payload_cb;
    req.payload_len = strlen(post_start) + fs_getFileSize(path) + strlen(post_end);
    req.header_fields = headers;
    req.response = response_cb;
    req.recv_buf = recv_buf_ipv4;
    req.recv_buf_len = sizeof(recv_buf_ipv4);

    ret = http_client_req(sock4, &req, timeout, (void *)path);

with the payload callback where the file is read in chunks using another callback.

void fs_read_cb(void *data, uint16_t len_read, void *ctx) {
  int *sock = (int *)ctx;
  send(*sock, data, len_read, 0);
}

static int payload_cb(int sock, struct http_request *req, void *user_data) {
  static uint8_t buf[1600];
  uint16_t sent_bytes = 0;

  sent_bytes += send(sock, post_start, strlen(post_start), 0);
  sent_bytes += fs_readFile((const char *)user_data, buf, sizeof(buf), fs_read_cb, &sock);
  sent_bytes += send(sock, post_end, strlen(post_end), 0);

  LOG_INF("payload_cb: sent %d bytes.", sent_bytes);
  return sent_bytes;
}

I increased the HTTP_CONTENT_LEN_SIZE and HTTP_CONTENT_LEN_SIZE inside Zephyr code since there is no KConfig configuration.

diff --git a/subsys/net/lib/http/http_client.c b/subsys/net/lib/http/http_client.c
index 321e77d24a..276e8cd3df 100644
--- a/subsys/net/lib/http/http_client.c
+++ b/subsys/net/lib/http/http_client.c
@@ -26,8 +26,8 @@ LOG_MODULE_REGISTER(net_http, CONFIG_NET_HTTP_LOG_LEVEL);

 #include "net_private.h"
-#define HTTP_CONTENT_LEN_SIZE 11
-#define MAX_SEND_BUF_LEN 192
+#define HTTP_CONTENT_LEN_SIZE 22
+#define MAX_SEND_BUF_LEN 4096

 static int sendall(int sock, const void *buf, size_t len)
 {

• Utilization of the Socket API and integration of the CA certificate follows the sample zephyr/samples/net/sockets/http_client/src/main.c
• The completion API requires instructions in the form of a system prompt. I created the following prompt.

This is a reminder app.
It is your task to interpret the users' requests to add or delete reminders.

A reminder consists of a name and a due date.
The format of the due date is "YYYY-MM-DD hh:mm"

Now is "%s".

Examples of reminders:
{"name": "homework", "due": "2023-11-29 15:00", "daily": true}
{"name": "dinner", "due": "2023-11-29 18:00", "daily": false}
{"name": "go to bed", "due": "2023-11-28 21:30", "daily": true}

Both name and due are mandatory. Each reminder must have a name and a due date.

Please classify the request into one of "delete" or "add".
Then identify the parameters.

These are the current active reminders:
%s

Examples of requests with your responses:

Request: "I finished my homework."
Response: {"request": "delete", "parameter": {"name": "homework"}}

Request: "I finished dinner."
Response: {"request": "delete", "parameter": {"name": "dinner"}}

Request: "Delete the homework reminder."
Response: {"request": "delete", "parameter": {"name": "homework"}}

Request: "Add homework with a due date of 3 p.m."
Response: {"request": "add", "parameter": {"name": "homework", "due": "2023-11-29 15:00"}}

Request: "Add dinner at 6 p.m."
Response: {"request": "add", "parameter": {"name": "dinner", "due": "2023-11-29 18:00"}}

Request: "Add getting up at 6 a.m."
Response: {"request": "add", "parameter": {"name": "getting up", "due": "2023-11-29 06:00"}}

Request: "Can you add the meeting at 2:30 p.m.?"
Response: {"request": "add", "parameter": {"name": "meeting", "due": "2023-11-28 14:30"}}

Request: "Please delete the go to bed reminder."
Response: {"request": "delete", "parameter": {"name": "go to bed"}}


This is the request:

With the following replacements for "%s"

First
2023-11-28 18:54

Second
{"name": "homework", "due": "2023-11-29 15:00"}
{"name": "dinner", "due": "2023-11-29 18:00"}

The prompt follows the recommended pattern to explain the role, context and task. It also gives examples, which is known as Few-Shot Prompting. For details have a look at https://www.promptingguide.ai and the ChatGPT documentation.

It does not use the function calling API.

• Add a wave header to the raw PCM data. The header is fixed except for the field for the data length.

// http://soundfile.sapp.org/doc/WaveFormat/
struct wav_hdr {
  uint8_t RIFF;
  uint32_t ChunkSize;
  uint8_t WAVE;
  uint8_t fmt;
  uint32_t Subchunk1Size;
  uint16_t AudioFormat;
  uint16_t NumOfChan;
  uint32_t SamplesPerSec;
  uint32_t bytesPerSec;
  uint16_t blockAlign;
  uint16_t bitsPerSample;
  uint8_t Subchunk2ID;
  uint32_t Subchunk2Size;
};

const struct wav_hdr WAV_DEFAULTS = {
  {'R', 'I', 'F', 'F'},
  0,
  {'W', 'A', 'V', 'E'},
  {'f', 'm', 't', ' '},
  16,
  1,
  1,
  8000,
  8000 * 2,
  2,
  16,
  {'d', 'a', 't', 'a'},
  0
};

For this reason, I added a function to the persistence to overwrite a part of the file.

int fs_overwriteData(const char *path, void *data, uint16_t len, uint16_t offset) {
  if (!mInitialized) return -1;

  char fname[MAX_PATH_LEN];
  snprintf(fname, sizeof(fname), "%s/%s", mp.mnt_point, path);
  int rc, ret;

  struct fs_file_t file;
  fs_file_t_init(&file);

  rc = fs_open(&file, fname, FS_O_CREATE | FS_O_WRITE);
  if (rc < 0) {
    LOG_ERR("FAIL: open %s: %d", fname, rc);
    return rc;
  }

  rc = fs_seek(&file, offset, FS_SEEK_SET);
  if (rc < 0) {
    LOG_ERR("FAIL: seek %s: %d", fname, rc);
    goto out;
  }

  rc = fs_write(&file, data, len);
  if (rc < 0) {
    LOG_ERR("FAIL: write %s: %d", fname, rc);
    goto out;
  } else {
    LOG_INF("wrote %d bytes to %s", len, fname);
  }

out:
  ret = fs_close(&file);
  if (ret < 0) {
    LOG_ERR("FAIL: close %s: %d", fname, ret);
    return ret;
  }

  return (rc < 0 ? rc : 0);
}

• Implementation of the display is very simple. There are only 3 lables drawn on the screen. Since I do not have any input devices to poll, I only call lv_task_handler() when a redraw is required. Due to Flash size constraints, this is all for now.

int display_init(void)
{
  const struct device *display_dev;

  display_dev = DEVICE_DT_GET(DT_CHOSEN(zephyr_display));
  if (!device_is_ready(display_dev)) {
    LOG_ERR("Device not ready, aborting test");
    return 0;
  }
  
  label_build = lv_label_create(lv_scr_act());
  lv_label_set_text(label_build, __DATE__ " " __TIME__);
  lv_obj_align(label_build, LV_ALIGN_TOP_LEFT, 0, 0);
  ...
  lv_task_handler();
  display_blanking_off(display_dev);

  return 0;
}

• Since the NFC reader sends continuously detection events for the Felica cards I check for repeated message and only trigger another action after a timout of 5 seconds implemented by a delayed task.
• The reminder app is initialized 5 seconds after the kWiFiConnectivityChange event which is sent when WiFi is connected. I delay because on kWiFiConnectivityChange, DHCP did not assign an IP yet and network calls will fail.

Conclusion

There are several factors that can contribute to degrade the recognition quality, which is the quality of the pronunciation, the recording, noise, the transcription and the completion result. It is quite tricky and needs tuning and error handling. Due to the limiting flash, the visual UI is limited. I would like to try the nRF54H20 once available with its 2MB of Flash and 1MB of RAM. However it is still impressive that a microcontroller can run voice recognition, ChatGPT and a display besides the Matter stack. Cool.

NFC Reader

Summary

The NFC Poller "Card Reader" is responsible for receiving the BLE OOB Pairing data and for recognizing the Felica cards' presence that are used for clearing reminders. The Sony RC-S620U NFC Poller is a USB device and therefore needs a USB Host-capable device. The current choice is a Raspberry Pico that communicates with the Bridge by UART. The logic for setting up the NFC poller and for parsing the data is implemented on the Raspberry Pico using the Raspberry Pico C++ SDK.

Motivation

NFC/Felica reader devices that can be easily used with microcontrollers are rare in Japan - likely due to the radio regulations. The device widely used is the Sony RC-S620S, which was released in 2010 and is hard to get. If found used, then for 2-3x of the original price (7000-10000 JPY). I found one for 700 JPY sold as FRW-C0003 with the same prints RWUE-620US on it. However, it did not work as expected. Finally, I tore it apart to find a name inside. I got stuck with the letter U and finally found out that I have the USB version - U does not stand for UART. Initially, I refrained from trying to make this work as it seemed to be hard work and time-consuming due to missing drivers, freely available documentation, and lack of experience with USB devices, but after the deadline shifted and I saw that another participant used the Raspberry Pico as a USB-UART bridge, I gave it a try as I was interested in trying out a USB device.

Parts Selection

• I selected the RC-S620S (USB) version because it was radio-certified, is widely used in Japan and I could use other's experience as a starting point. However, I intended to select the UART version initially.
• I added a Raspberry Pico as the nRF5340 does not support USB host mode. (Looking forward to using an nRF54H20 when it is released).
• A flat band-to-pin adapter is required for the RC-S620S. Switch Science offers one FeliCa RC-S620S/RC-S730 ピッチ変換基板のセット(フラットケーブル付き)

Hardware connection

• UART RX and TX are connected by a level shifter to the bridge.
• Instead of using USB D+ and D-, which are only available by the USB port and test pads, I chose to use the Pico PIO USB library since USB Pins cannot be assigned to GPIO pins. Furthermore, the device is rated 3.3V and not 5V. I used a regular GND and not the test pad GND closest to USB. (Initially, I soldered the test pins, but the wire broke off.)
• The Raspberry Pico is powered by VSYS with 3.3V from the display output.

Learnings

It was tricky to find out which documentation to read and to figure out what the protocols in use were (Type 4 Tag, NDEF). It was very hard to make the device work. There were several datasheets available, each with a piece of information for the puzzle. Additionally, protocols are nested within each other.

The following pages helped me the most

• RC-S620/S Hands on by HMcircuit explains in detail how to use the RC-S620S, its protocol and also how to use it for NFC Type-A/B. He also reports that it is compatible with the PN532 User Manual (as Sony only released a "light" version manual publicly)
• The User Manual with the USB communication details is available on the FCC website FCC ID AK8RCS620U and the UART version manual with the communication protocol is explained here FCC ID AK8RCS620S
• Hacking an NFC ring explains NFC and its protocols like NDEF in detail
• I used the ndeflib source code to reverse engineer the NDEF record as I did not find a complete datasheet for it ndeflib/src/ndef/bluetooth.py
• and the file components/libraries/experimental_nfc/connection_handover/nfc_ble_pair_msg.c of nRF52_SDK has detailed comments on the OOB NDEF record structure
• An overview of the NFC Protocol Stack on Wikipedia Near-field communication
• Type 4 Tag Technical Specification

Communication protocol with RC-S620/S and NDEF OOB data

The communication sequence is command, ack, and response in the following format explained using the example of the reset command.

Reset Command
00      Fixed preamble
00 FF   Start of packet
03      Length of the data (D4 18 01)
FD      Checksum of the length, which is 0x0100 - len
D4      Fixed command code
18      Sub command "reset"
01      Argument for the reset command
13      Checksum which is 0x100 minus (the sum of all data between D4 and the checksum modulo 0x100)
00      Fixed postamble

Ack
00      Fixed preamble
00 FF   Start of packet
00 FF   ACK
00      Fixed postamble

Response
00      Fixed preamble
00 FF   Start of packet
02      Length of the data (D5 19)
EF      Checksum of the length, which is 0x0100 - len
D5      Fixed response code
19      Sub response "reset"
12      Checksum which is 0x100 minus (the sum of all data between D4 and the checksum modulo 0x100)
00      Fixed postamble

I use the following commands

• reset (0x18)
• GetFirmwareVersion (0x02) to check the device is reachable
• RFConfiguration (0x32) to configure timings or switch RF off
• InListPassiveTarget (0x4a) with parameters to detect Felica cards or NFC Type A cards. Both cannot be detected at the same time. Therefore I switch every second.
• InDataExchange (0x40) to read the OOB data (This is missing from the RC-S620/S "light" manual)

Luckily there was an example for the read command 0xb0 in the PN532 manual on page 131. With other resources, I identified the format as

D4     Fixed command code
40     Sub command
01     Detected card ID
00     ??
B0     Type 4 Tag Read command
00 00  page
F1     length

Following the result of reading the NFC tag created by the NFC pairing sample in the nRF Connect SDK. By converting this to ASCII, some strings like ‘application/vnd.bluetooth.ep.oob’ become visible.

00 00 FF F6 0A D5 41 00 00 B1 81 02 00 00 00 0D
48 73 15 C1 02 00 00 00 04 61 63 01 01 30 00 0A
20 00 00 00 52 01 61 70 70 6C 69 63 61 74 69 6F
6E 2F 76 6E 64 2E 62 6C 75 65 74 6F 6F 74 68 2E
6C 65 2E 6F 6F 62 30 08 1B 4C 0B BC 24 18 47 01
02 1C 00 11 10 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 11 22 60 0D 4B 8B 62 7C 84 1C 9D
FE 00 AB 5D 49 D5 3D 11 23 57 2C 27 0E 2C A8 C8
EF DD 02 64 81 0C 59 97 7D 03 19 00 00 02 01 04
08 09 50 6F 73 74 75 72 65 41 02 00 00 00 1A 54
70 10 13 75 72 6E 3A 6E 66 63 3A 73 6E 3A 68 61
6E 64 6F 76 65 72 00 2C 02 04 00 00 00 00 00 00

The data can be decoded as follows

00 00 FF        Preamble
F6              Length
0A              Length Checksum (F6 + 0A = 100)
D5              Response Command
41              Response for InDataExchange
00              Status code: No error
00              ??
B1              T4T Read response
----
81              NDEF record header - TNF + Flags: MB=1b ME=0b CF=0b SR=0b IL=0b TNF=001b (Well-Known)
02              NDEF record header - Record Type Length = 2 octets
00 00 00 0D     NDEF record header - Payload Length = 13 octets
                NDEF record header - ID Length missing since it is optional (IL=0b)
48 73           NDEF record header - Record(Payload) Type = ‘Hs’ (Handover Select Record)
                NDEF record header - Payload ID missing since it is optional (IL=0b)
15              NDEF record payload - Connection Handover specification version = 1.5 (0x15 = 0001 0101 = 1 5)
C1              NDEF record header - TNF + Flags: MB=1b ME=1b CF=0b SR=0b IL=0b TNF=001b (Well-Known)
02              NDEF record header - Record Type Length = 2 octets
00 00 00 04     NDEF record header - Payload Length = 4 octets
                NDEF record header - ID Length missing since it is optional (IL=0b)
61 63           NDEF record header - Record(Payload) Type = ‘ac’ (Alternative Carrier Record)
                NDEF record header - Payload ID missing since it is optional (IL=0b)
01              NDEF record payload - Carrier Power State = 1 (active)
01              NDEF record payload - Carrier Data Reference Length = 1 octet
30              NDEF record payload - Carrier Data Reference = ‘0’
00              NDEF record payload - Auxiliary Data Reference Count: 0
0A              NDEF record header - TNF + Flags: MB=0b ME=0b CF=0b SR=0b IL=1b TNF=010b (MIME media-type)
20              NDEF record header - Record Type Length = 32 octets
00 00 00 52     NDEF record header - Payload Length = 82 octets
01              NDEF record header - Payload ID Length = 1 octet
61 70 70 6C 69 63 61 74
69 6F 6E 2F 76 6E 64 2E
62 6C 75 65 74 6F 6F 74
68 2E 6C 65 2E 6F 6F 62   NDEF record header - Record(Payload) Type = ‘application/vnd.bluetooth.ep.oob’:
30              NDEF record header - Payload ID = ‘0’
08              Length 8
1B              LE Bluetooth Device Address
4C 0B BC 24 18 47   BLE Address in reverse order (network endianness).
01              Random Address
02              Length 2
1C              LE Role
00              Peripheral
11              Length 2
10              Security Manager TK Value
00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00
11              Length 17
22              LE Secure Connections Confirmation Value
60 0D 4B 8B 62 7C 84 1C
9D FE 00 AB 5D 49 D5 3D
11              Length 17
23              LE Secure Connections Random Value
57 2C 27 0E 2C A8 C8 EF
DD 02 64 81 0C 59 97 7D
03              Length 3
19              Appearance
00 00           ??
02              Length 2
01              Flags
04              ??
08              Length 8
09              Complete Local Name
50 6F 73 74 75 72 65        Posture
41              NDEF record header - TNF + Flags: MB=0b ME=1b CF=0b SR=0b IL=0b TNF=001b (Well-Known)
02              NDEF record header - Record Type Length = 2 octets
00 00 00 1A     NDEF record header - Payload Length = 26 octets
54 70           NDEF record header - Record(Payload) Type = ‘Tp’ (Service Parameter)
10              ?? Uri code??
13              Length 19
75 72 6E 3A 6E 66 63 3A
73 6E 3A 68 61 6E 64 6F
76 65 72        urn:nfc:sn:handover
00 2C 02        ??
----
04              Data checksum
00              Postamble

Since it is enough that one device has the other device’s OOB data, I read the data I need and do not send any response. The effort to re-implement the full exchange is high and not needed.

Software Part

I used the bare_api example of tinyusb as a starting point.

Component Diagram

• The Main initializes all modules and injects the dependencies as callbacks or variable pointers. Dependencies between modules are minimized. For example, the NFC module does not know that the transport is USB.
• The UART task communicates with the bridge by listening to commands. Responses are sent by printf.
• the NFC task has logic to send commands and parse the results. Depending on the commands the NFC module runs a simple switch-case state machine to send the appropriate commands and parse the response accordingly
• the USB module reads the USB descriptors upon device detection and saves the read and write handles as global variables (TinyUSB does not support user data in its callbacks). In the loop, the USB device is read for new data as USB devices cannot initiate to send data on its own.
• Logging can be switched off globally, which is required as the response is sent by printf commands and the bridge would receive the logs.

Interface with Bridge

• The communication with the Bridge is done by a simple custom serial protocol.
• Sending the character 0x01 configures the device, 0x02 detects Felica cards, 0x03 detects NFC type A cards and if detected reads the OOB data and formats it in the same format as in the Zephyr Bluetooth Shell, and 0x04 switches RF off.
• Possible responses are “ID:<NAME>” when one of the Felica cards is detected or OOB data. The Felica card IDs are hardcoded.

Challenges encountered

• The Main challenge was to figure out which protocols are used and how to encode and decode them.
• Zephyr OS does not have USB host device support for Raspberry Pico yet. Therefore I used the C++ SDK with a main loop bare metal approach.

Project CMake Configuration

• The project structure follows the projects in the SDK examples. Please check the official documentation at Raspberry Pi Documentation - The C/C++ SDK
• To include Pico PIO USB, clone the project (see clone_ext_dependencies.sh) and add it as a subdirectory in the main CMakeLists.txt. Then add the following to your project's CMakeLists.txt.

...
# Example source
target_sources(nfc_usb PUBLIC
...
# can use 'tinyusb_pico_pio_usb' library later when pico-sdk is updated
${PICO_TINYUSB_PATH}/src/portable/raspberrypi/pio_usb/dcd_pio_usb.c
${PICO_TINYUSB_PATH}/src/portable/raspberrypi/pio_usb/hcd_pio_usb.c
)
...
# use tinyusb implementation
target_compile_definitions(nfc_usb PRIVATE PIO_USB_USE_TINYUSB)
target_link_libraries(nfc_usb PUBLIC pico_stdlib pico_pio_usb tinyusb_host tinyusb_board hardware_uart)
...

Flash by copying the.uf2 file or by using a Pico Probe and the following command. Note that the "adapter speed 5000" was required to be added manually to work with another Raspberry Pico as a probe.

openocd -f interface/cmsis-dap.cfg -c "adapter speed 5000" -f target/rp2040.cfg -c "program build/nfc_usb/nfc_usb.elf verify reset exit"

Conclusion

The current implementation is minimal with hard-coded commands and OOB data parsing. Interfacing USB devices is easier than expected - only read descriptions and then write and read data to handles. Communicating with NFC devices was hard despite the standardized communication protocols. I would like to use Zephyr and its NFC libraries next time. So I am looking forward to using the nRF54H20 once released.

OpenThread Border Router

Matter for Thread devices need a Thread Border Router to communicate with other Matter devices running on IP-based networks like Thread over Wifi with the nRF7002. The chip-tool uses your computer's network, usually connected by Ethernet or Wi-Fi. The following is a quote from the documentation of the OTBR repository ( openthread/ot-br-posix at thread-reference-20230710) to explain what an OTBR does.

Per the Thread Specification, a Thread Border Router connects a Thread network to other IP-based networks, such as Wi-Fi or Ethernet. A Thread network requires a Border Router to link to other networks. A Thread Border Router minimally supports the following functions:* End-to-end IP connectivity via routing between Thread devices and other external IP networks* External Thread Commissioning (for example, a mobile phone) to authenticate and join a Thread device to a Thread networkOpenThread's implementation of a Border Router is called OpenThread Border Router (OTBR). OTBR is a Thread Certified Component on the Raspberry Pi 3B with a Nordic nRF52840 NCP.

Warning: When you read this, the repository might have been updated, and you might not need manual fixes, which I did or need to adapt. Of course, you can use the same old versions I used.

I follow this guide Thread Border Router - Bidirectional IPv6 Connectivity and DNS-Based Service Discovery | OpenThread and the referenced Step 4 of the Build a Thread network with nRF52840 boards and OpenThread codelab to build and flash a nRF52840 RCP device.

I confirmed the steps starting with a Raspberry Pi 4 running a fresh image of Raspberry Pi OS (64-bit) bullseye and bookworm.

Installation

Installing prerequisites on Linux

sudo apt-get install git gcc g++ pkg-config libssl-dev libdbus-1-dev libglib2.0-dev libavahi-client-dev ninja-build python3-venv python3-dev python3-pip unzip libgirepository1.0-dev libcairo2-dev libreadline-dev

Install some Raspberry Pi-specific dependencies.

sudo apt-get install pi-bluetooth avahi-utils

Get the repositories and fix the revisions to the release tag thread-reference-20230710

git clone https://github.com/openthread/ot-br-posix.git
cd ot-br-posix
git checkout 790dc77
git submodule update --init --recursive
cd third_party/openthread/repo/
git checkout 8bc2504
cd -

Call the bootstrap script with the build parameters. Most of them are defaults. I just analysed the scripts and made them apparent. Important are the following to adapt.

• Select the network interface you want to connect to. I used INFRA_IF_NAME=eth0
• Select how your dongle is connected. I used uart on /dev/ttyACM0. So add -DOTBR_RADIO_URL='spinel+hdlc+uart:///dev/ttyACM0'
• Select to include the web interface. It makes it easier to control. So add WEB_GUI=1
• Select Thread version 1.3.1 if you plan to commission multi-fabric to the Apple Home app (Important, 1.3.0 does not work!). So add -DOT_THREAD_VERSION=1.3.1

INFRA_IF_NAME=eth0 RELEASE=1 REFERENCE_DEVICE=1 BACKBONE_ROUTER=1 NETWORK_MANAGER=0 DHCPV6_PD=0 WEB_GUI=1 REST_API=1 BORDER_ROUTING=1 NAT64=1 DNS64=1 OTBR_OPTIONS="-DOT_THREAD_VERSION=1.3.1 -DOTBR_TREL=ON -DOTBR_NAT64=ON -DOT_DIAGNOSTIC=ON -DOT_FULL_LOGS=ON -DOT_PACKAGE_VERSION=8bc25042b -DOTBR_PACKAGE_VERSION=790dc77 -DOT_POSIX_CONFIG_RCP_BUS=UART -DOTBR_RADIO_URL='spinel+hdlc+uart:///dev/ttyACM0' -DOTBR_DUA_ROUTING=ON -DOT_DUA=ON -DOT_MLR=ON -DOTBR_DNSSD_DISCOVERY_PROXY=ON -DOTBR_SRP_ADVERTISING_PROXY=ON -DOT_BORDER_ROUTING=ON -DOT_SRP_CLIENT=ON -DOT_DNS_CLIENT=ON" ./script/bootstrap

Then, I had to make two fixes as it did not build out of the box. Please take a look at the comments. Then, I built using the setup script.

# comment out line 41 where it says die "dns64 is not tested under $PLATFORM." AND change the following ubuntu to debian
sed -i.bak -e '41d' script/_dns64
# Do this only on Raspbian OS
sed -i.bak "s/\"\$PLATFORM\" = ubuntu/\"\$PLATFORM\" = debian/" script/_dns64
# add the version check for 1.3.1
sed -i.bak "s/kThreadVersion13 = 4/kThreadVersion13 = 5/" src/ncp/ncp_openthread.cpp
sed -i.bak "s/version = \"1.3.0\"/version = \"1.3.1\"/" src/ncp/ncp_openthread.cpp
INFRA_IF_NAME=eth0 RELEASE=1 REFERENCE_DEVICE=1 BACKBONE_ROUTER=1 NETWORK_MANAGER=0 DHCPV6_PD=0 WEB_GUI=1 REST_API=1 BORDER_ROUTING=1 NAT64=1 DNS64=1 OTBR_OPTIONS="-DOT_THREAD_VERSION=1.3.1 -DOTBR_TREL=ON -DOTBR_NAT64=ON -DOT_DIAGNOSTIC=ON -DOT_FULL_LOGS=ON -DOT_PACKAGE_VERSION=8bc25042b -DOTBR_PACKAGE_VERSION=790dc77 -DOT_POSIX_CONFIG_RCP_BUS=UART -DOTBR_RADIO_URL='spinel+hdlc+uart:///dev/ttyACM0' -DOTBR_DUA_ROUTING=ON -DOT_DUA=ON -DOT_MLR=ON -DOTBR_DNSSD_DISCOVERY_PROXY=ON -DOTBR_SRP_ADVERTISING_PROXY=ON -DOT_BORDER_ROUTING=ON -DOT_SRP_CLIENT=ON -DOT_DNS_CLIENT=ON" ./script/setup

Note that on bookworm and Ubuntu 23, the npm installation fails since the sever registry.npmjs.org could not be reached by its IPv6 address. A temporary solution is to append the following line to /etc/hosts

104.16.20.35 registry.npmjs.org

Next, I build the firmware for the dongle. The dongle can be any nRF52840 board with a USB broken out. I used an MDBT50Q-RX because I had one around. It is an nRF52840 without any pins broken out... not very useful. So, it finally found some use. I also successfully used the Adafruit Feather that is now in the Feedback Light. You can use the Nordic dongle.

Download the repository and checkout the revision is compatible with the OTBR. Compatible means they both use the same revision of the openthread repo. For some reason (maybe incompatible versions), using the coprocessor sample nrf/samples/openthread/coprocessor did not work for me. The startup of the OTBR failed.

Do this on your development PC. It is an effort to get the gcc-arm-none-eabi running, and I am still determining the support for nrfjprog and nrfutil.

git clone https://github.com/openthread/ot-nrf528xx.git
cd ot-nrf528xx
git checkout 982244f
git submodule update --init --recursive

Build with the option to use UART over USB. According to the commit message of https://github.com/openthread/ot-nrf528xx/pull/606T, which is "submodule: bump openthread from 37fb770 to 8bc2504 #606", the openthread version should be 8bc2504, which is the same I use for ot-br.

./script/build nrf52840 USB_trans -DOT_SRP_SERVER=ON -DOT_ECDSA=ON -DOT_SERVICE=ON -DOT_DNSSD_SERVER=ON -DOT_SRP_CLIENT=ON

Then, flash it to the dongle.

cd build/bin
arm-none-eabi-objcopy -O ihex ot-rcp ot-rcp.hex
# Using a J-Link
nrfjprog -f nrf52 --verify --chiperase --program ot-rcp.hex --reset
-----------
# Using the Adafruit bootloader; Instructions from https://github.com/adafruit/Adafruit_nRF52_Bootloader
uf2conv.py ot-rcp.hex -c -f 0xADA52840
# Set the board in bootloader mode by pressing reset twice within 500ms and copy the uf2 file onto the board
-----------
# Using the nRF52840 Dongle bootloader
nrfutil pkg generate --hw-version 52 --sd-req=0x00 --application ot-rcp.hex --application-version 1 ot-rcp.zip
# Connect the nRF52840 Dongle to the USB port.
# Press the RESET button on the dongle to put it into the DFU mode. The LED on the dongle starts blinking red.
nrfutil dfu usb-serial -pkg ot-rcp.zip -p /dev/ttyACM0

Verification

Verify that the OTBR is up and running.

sudo service mdns status
sudo service otbr-agent status
sudo service otbr-web status
# to (re)start if not running
sudo service otbr-agent restart && sudo service otbr-web restart && sudo service mdns restart

Form a new network via the command line (you can also open the web GUI and press the FORM button)

# Form a Thread network
sudo ot-ctl dataset init new
sudo ot-ctl dataset commit active
sudo ot-ctl ifconfig up
sudo ot-ctl thread start
sudo ot-ctl state
sudo ot-ctl netdata show
sudo ot-ctl ipaddr

Get the active dataset as a hex-formatted string. You will need this for commissioning.

sudo ot-ctl dataset active -x
# gets a long number like 0e080000000000010000000300001835060004001fffe00208eacaf94459bd6f140708fdeab560bb1aa44e0510fee44d2c5a367ac50fcaf3152014e11e030f4f70656e5468726561642d646566300102def00410bc867631d6f4dd8eead1c66252af343c0c0402a0fff8

Troubleshooting

• My working configuration of /etc/sysctl.conf

sudo nano /etc/sysctl.conf
OpenThread configuration
net.core.optmem_max=65536
vm.swappiness=0
net.ipv6.conf.wlan0.accept_ra=2
net.ipv6.conf.wlan0.accept_ra_rt_info_max_plen=64
net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1

• I put it at the end of /etc/dhcpcd.conf

noipv6
noipv6rs

• USB 3.x devices cause heavy EMI

Any USB 3.x (most prominently, 5 Gbps specification, aka. USB 3.2 gen 1x1) device/connector can cause heavy EMI.

https://community.home-assistant.io/t/guide-for-zigbee-interference-avoidance-and-network-range-coverage-optimization/515752

• If you get the below mDNS error, you need to either restart the OTBR, use a new endpoint ID for the chip-tool or delete the chip-tool's caches.

[00:00:21.766,998] <inf> chip: [DL]advertising srp service: 4025AA9435623B68._matterc._udp
[00:00:22.005,706] <err> chip: [DL]SRP update error: domain name or RRset is duplicated

Chip-tool

Get the sources and use tag v1.1.0.0 (should match with the nRF Connect SDK 2.4.x and worked also after upgrading to 2.5.1). If you build on the main branch, the credentials/certificates differ, and commissioning fails. I made a dedicated checkout because building within the ncs tree failed for me with "Source file not found" for boringssl.

git clone https://github.com/project-chip/connectedhomeip.git
git checkout tags/v1.1.0.0 -b v1.1.0.0
git submodule update --init

Source at the top level

source scripts/activate.sh
# If this script says the environment is out of date, it can be updated by running
source scripts/bootstrap.sh

Build the chip-tool

# requires ZAP tool

# Option 1
# Download release binaries here https://github.com/project-chip/zap/releases/tag/v2023.05.04
export ZAP_DEVELOPMENT_PATH=/opt/zap

# Option 2
# Assuming all dependencies are installed
git clone https://github.com/project-chip/zap.git /opt/zap
cd /opt/zap
npm config set user 0 # only raspberry pi not mac
npm ci
export ZAP_DEVELOPMENT_PATH=/opt/zap

./scripts/examples/gn_build_example.sh examples/chip-tool out

Commission the device

./chip-tool pairing ble-thread 3 hex:0e08000000000001000035060004001fffe00708fdaeeae1f849d7a90c0402a0f7f8051000112233445566778899aabbccddeeff030e4f70656e54687265616444656d6f0410445f2b5ca6f2a93a55ce570a70efeecb000300000f0208111111112222222201021234 20202021 3840
./chip-tool identify identify 1000 3 1
./chip-tool identify identify 0 3 1

Commission to Homekit

./chip-tool pairing open-commissioning-window 3 1 300 1000 3840
> [1690947535961] [45126:2732731] [CTL] SetupQRCode: [MT:6FCJ1AFN00SODK0N320]
# Add a new device to Home using this QR code.
https://project-chip.github.io/connectedhomeip/qrcode.html?data=MT%3A6FCJ1AFN00SODK0N320

Additional Feedback on Commissioning with the Apple Home App and working on a Mac

If you work on a Mac, pairing Matter Accessory using the Open Source Matter Darwin chip-tool and iOS chip-tool will require installing the Bluetooth Central Matter Client Developer mode profile on MacOS or iOS/iPadOS. (from Testing with Apple Devices)

You cannot commission (with OTRB credentials + Matter fabric credentials) your device with the Home App using the Raspberry Pi OTBR because your Home App does not know the credentials and settings of your OTBR. You should be able to commission with the Home App using the Apple HomePod. (I don't own this device, I cannot try)

But if your device is already in the Thread network of your Raspberry Pi OTBR, then you can use the Home App to commission (with Matter fabric credentials) it to the Home App's Matter fabric. This can be achieved with the following commands using the chip-tool.

./chip-tool pairing ble-thread 3 hex:0e08000000000001000035060004001fffe00708fdaeeae1f849d7a90c0402a0f7f8051000112233445566778899aabbccddeeff030e4f70656e54687265616444656d6f0410445f2b5ca6f2a93a55ce570a70efeecb000300000f0208111111112222222201021234 20202021 3840
./chip-tool pairing open-commissioning-window 3 1 300 1000 3840

# Find the following line in the chip tool's output
# > [1690947535961] [45126:2732731] [CTL] SetupQRCode: [MT:6FCJ1AFN00SODK0N320]
# Add a new device to Home using this QR code.
# https://project-chip.github.io/connectedhomeip/qrcode.html?data=MT%3A6FCJ1AFN00SODK0N320

This is called the Multi-admin scenario. Please find more details in the Nordic documentation at Working with the CHIP Tool — Matter documentation (nRF Connect SDK) or this blog post Matter: Testing the nRF Connect platform with Apple, Google and Samsung ecosystems.

For me, commissioning with the Home App worked only on OTBRs built with OpenThread version 1.3.1.

Overall, the Home App does not tell you why it failed. It is required to make sense of the logs on the Apple device. Logs can be activated following these instructions Apple Training (But you don't want to go down that rabbit hole)

Example (if you try to commission Matter over Thread devices without Thread credentials)

[3151815363/2672715473] Failed to collect required credentials for accessory. Error: Error Domain=HAPErrorDomain Code=30 "(null)"
[HMMTRAccessoryServerBrowser] Failed to fetch Preferred Thread Credentials from owner with error Error Domain=ThreadCredentialsStore Code=9 "No preferred network found" UserInfo={NSLocalizedDescription=No preferred network found}
[1118926759/2672715473] Populating Thread credential collection error

Python scripting for Matter

There are two possibilities for Python scripting— the matter-repl of the connectedhomeip repository and the python-matter-server used in Home Assistant. I will present both. I prefer the python-matter-server as it works as a standalone script that is quite stable. I only have an issue with the subscribing and reading values of the Bridge. It works sometimes, but most of the time it cannot connect anymore while chip-tool works much better. I cannot tell why. The same commands work well on the feedback light.

For both, getting familiar with asyncio is very helpful.

matter-repl

Follow the documentation to build the matter-repl

Matter Python REPL

• Build the Python virtual environment. link
• Commands extracted from the above instructions.

# install matter-repl
scripts/build_python.sh -m platform -i separate
source ./out/python_env/bin/activate
sudo out/python_env/bin/chip-repl

# install local playground
pip3 install jupyterlab ipykernel
pip3 install jupyterlab-lsp
pip3 install python-lsp-server

cd /Users/jens/work/mcu_prj/prj/makeItMatter/connectedhomeip
source ./out/python_env/bin/activate
python -m ipykernel install --user
deactivate

jupyter-lab --no-browser --ip=192.168.10.4 --port=8888

Start the matter repl Jupyter Notebook. The following code connects the Posture Checker via the Matter Bridge to the Feedback Light. I subscribe to the changes of the LevelCluster provided by the Matter Bridge and send a command to change the Hue to the same value as the Feedback Light.

To start over, delete the JSON file given by the storage path.

# Clear any leftovers from the previous session
REALLY_CLEAR=False

if REALLY_CLEAR:
    import os, subprocess

    if os.path.isfile('repl.json'):
        os.remove('repl.json')

    # So that the all-clusters app won't boot with a stale prior state.
    # os.system('sudo rm -rf /tmp/chip_*')

Load the cluster configuration from the storagepath

# Load the cluster and settings of the controller
import chip.native
import pkgutil
module = pkgutil.get_loader('chip.ChipReplStartup')
%run {module.path} --storagepath repl.json

import chip.clusters as Clusters

Define variables for the endpoint IDs.

bridgeId = 3
lightId = 4

Commission the Matter Bride using the QR code from the re-opened commission window using the chip tool.

Commissioning directly with Thread credentials is also possible, but this frequently fails.

Please keep the Feedback Light off, as it uses the same default discriminator. It could be changed, but on Apple devices, these are blocked, and commissioning fails.

# retrieve the thread operational dataset by 'sudo ot-ctl dataset active -x'
if REALLY_CLEAR:
    devCtrl.CommissionThread(3840, 20202021, bridgeId, threadOperationalDataset=b'0e08000000000001000035060004001fffe00708fd80693996c824f50c0402a0f7f8051000112233445566778899aabbccddeeff030e4f70656e54687265616444656d6f0410445f2b5ca6f2a93a55ce570a70efeecb000300000f0208111111112222222201021234')
    # devCtrl.CommissionWithCode(setupPayload='MT:Y.K90AFN002J.F35E10',nodeid=bridgeId)

Commission the Feedback Light.

if REALLY_CLEAR:
    devCtrl.CommissionThread(3840, 20202021, lightId, threadOperationalDataset=b'0e08000000000001000035060004001fffe00708fd80693996c824f50c0402a0f7f8051000112233445566778899aabbccddeeff030e4f70656e54687265616444656d6f0410445f2b5ca6f2a93a55ce570a70efeecb000300000f0208111111112222222201021234')
    # devCtrl.CommissionWithCode(setupPayload='MT:6FCJ1AFN00KA0K7J020',nodeid=lightId)

Subscribe the values and pass them to the command setting the hue. This is very complex since Jupyter Notebook runs on asyncio, and the function for the command is async. Async functions cannot be called in the change callback. Additionally, asyncio works slightly differently in Jupyter Notebooks. The workaround is nest_asyncio. Both tasks hand over each other in the asyncio.sleep and queue.get functions. An additional issue was that the Matter Bridge missed implementing some attributes of the LevelCluster, which resulted in an error object that held the current value differently. This logic runs indefinitely until the Python kernel gets killed. Overall, this solution is very complex to forward a value between 2 Matter devices.

import asyncio
queue = asyncio.Queue()

import nest_asyncio
nest_asyncio.apply()

async def changeColors():
    print("changeColors")
    while True:
        levelValue = await queue.get()
        print(f"get {levelValue}")
        await devCtrl.SendCommand(4, 1, Clusters.ColorControl.Commands.MoveToHue(levelValue, 0, 1, 0, 0))

async def subscribe():
    print("subscribe")
    reportingTimingParams = (0, 2) # MinInterval = 0s, MaxInterval = 2s
    subscription = await devCtrl.ReadAttribute(bridgeId, [3, Clusters.LevelControl.Attributes.CurrentLevel], reportInterval=reportingTimingParams, returnClusterObject=True)
    print("subscribed")

    def OnAttributeChangeCb(path: 'TypedAttributePath', transaction: 'SubscriptionTransaction'):  
        # levelValue = transaction.GetAttribute(path)
        # Does not work due to missing attributes. Got a Decoding Error object with incomplete values. 
        # Therefore, get the error object's first value (TLVValue) since we know this is the current level.
        # Should be fixed in bridge version 2
        levelValue = transaction.GetAttributes().get(path.Path.EndpointId).get(path.ClusterType).TLVValue.get(0)
        print(f"put {levelValue}")
        queue.put_nowait(levelValue)

    subscription.SetAttributeUpdateCallback(callback=OnAttributeChangeCb)
    
    while True:
        await asyncio.sleep(0)

loop = asyncio.get_event_loop()
loop.create_task(changeColors())
loop.create_task(subscribe())

# Note that to stop subscriptions, a restart of the Jupyter kernel is required.
#subscription.Shutdown()

python-matter-server

Install client and server with pip. Make sure you have Python >3.10 or it will install a very old version. This comes with the chip stack pre-compiled on Linux OS.

To install Python on a Raspberry Pi

wget https://www.python.org/ftp/python/3.10.13/Python-3.10.13.tgz
tar zxf Python-3.10.13.tgz
cd Python-3.10.13/
./configure --enable-optimizations
make -j4
sudo make altinstall

To install the python-matter-server

python3.10 -m venv env
source env/bin/activate
pip install python-matter-server[server]==5.1.4

The server required the folder /data for some temporary files. Raspian OS does not have this folder - I assume this because it runs within a Docker container in the Home Assistant context, which has such a folder

sudo mkdir /data
sudo chmode go+w /data

First I start the server separately. Client and Server could also run within the same script. The below script is the full version with command line arguments and logging in place (as it is in the example script)

import argparse
import asyncio
import logging
import os
from pathlib import Path

from aiorun import run
import coloredlogs

from matter_server.server.server import MatterServer

logging.basicConfig(level=logging.DEBUG)
_LOGGER = logging.getLogger(__name__)

DEFAULT_VENDOR_ID = 0xFFF1
DEFAULT_FABRIC_ID = 1
DEFAULT_PORT = 5580
DEFAULT_STORAGE_PATH = os.path.join(Path.home(), ".matter_server")


# Get parsed passed in arguments.
parser = argparse.ArgumentParser(description="Matter Server.")
parser.add_argument(
    "--storage-path",
    type=str,
    default=DEFAULT_STORAGE_PATH,
    help=f"Storage path to keep persistent data, defaults to {DEFAULT_STORAGE_PATH}",
)
parser.add_argument(
    "--port",
    type=int,
    default=DEFAULT_PORT,
    help=f"TCP Port on which to run the Matter WebSockets Server, defaults to {DEFAULT_PORT}",
)
parser.add_argument(
    "--log-level",
    type=str,
    default="debug",
    help="Provide logging level. Example --log-level debug, default=info, possible=(critical, error, warning, info, debug)",
)

args = parser.parse_args()


if __name__ == "__main__":
    # configure logging
    logging.basicConfig(level=args.log_level.upper())
    coloredlogs.install(level=args.log_level.upper())

    # make sure storage path exists
    if not os.path.isdir(args.storage_path):
        os.mkdir(args.storage_path)

    # Init server
    server = MatterServer(
        args.storage_path, DEFAULT_VENDOR_ID, DEFAULT_FABRIC_ID, int(args.port)
    )

    async def run_matter():
        """Run the Matter server"""
        # start Matter Server
        await server.start()

    async def handle_stop(loop: asyncio.AbstractEventLoop):
        """Handle server stop."""
        await server.stop()

    # run the server
    run(run_matter(), shutdown_callback=handle_stop)

After the server finished startup, I have another script for commissioning. The following an excerpt.

async def commission(client: MatterClient, code: str):
        # set credentials
        await client.set_wifi_credentials(ssid="ReplaceMe", credentials="ReplaceMe")
        await client.set_thread_operational_dataset(dataset="ReplaceMe")
        # commission
        await client.commission_with_code(code)

    async def run_matter():
        """Run the Matter client."""

        # run the client
        url = f"http://127.0.0.1:{args.port}/ws"
        async with aiohttp.ClientSession() as session:
            async with MatterClient(url, session) as client:

                # start listening and continue when the client is initialized
                ready = asyncio.Event()
                asyncio.create_task(client.start_listening(init_ready=ready))
                await ready.wait()

                logging.info(f"Server Info: {client.server_info}")

                try:
                    await commission(client, args.code)
                except NodeCommissionFailed:
                    logging.error("Commissioning failed")

                # The only way I found to exit asyncio ...
                os._exit(0)

To receive the event that the client is initialized, an asyncio Event is used and awaited. This was not part of the example.

The client provides functions to query nodes, read, subscribe and send commands. The following script subscribes to the current level of the bridge, converts it to a hue value and sends a move to hue command to the feedback light.

import argparse
import asyncio
import logging
import os
from pathlib import Path

import aiohttp
from aiorun import run
import coloredlogs

from matter_server.client.client import MatterClient
from matter_server.common.helpers.util import create_attribute_path_from_attribute
from matter_server.common.models import EventType
from chip.clusters import Objects as Clusters

logging.basicConfig(level=logging.DEBUG)
_LOGGER = logging.getLogger(__name__)

DEFAULT_PORT = 5580
DEFAULT_URL = f"http://127.0.0.1:{DEFAULT_PORT}/ws"


# Get parsed passed in arguments.
parser = argparse.ArgumentParser(description="Matter Client - Forward Posture Score")
parser.add_argument(
    "--port",
    type=int,
    default=DEFAULT_PORT,
    help=f"TCP Port on which to run the Matter WebSockets Server, defaults to {DEFAULT_PORT}",
)
parser.add_argument(
    "--log-level",
    type=str,
    default="info",
    help="Provide logging level. Example --log-level debug, default=info, possible=(critical, error, warning, info, debug)",
)

args = parser.parse_args()


if __name__ == "__main__":
    # configure logging
    logging.basicConfig(level=args.log_level.upper())
    coloredlogs.install(level=args.log_level.upper())

    queue = asyncio.Queue()

    # Helper function since the client does not provide
    # the information on which endpoint a cluster is found.
    def findFirstEndpointForCluster(node, cluster):
        for endpoint in node.endpoints.values():
            if(endpoint.has_cluster(cluster)):
                return endpoint.endpoint_id
        return None        

    def handleEvent(eventType, value):
        if eventType ==EventType.ATTRIBUTE_UPDATED:
            queue.put_nowait(value)

    async def run_matter():
        # run the Matter client, conect to the Matter server.
        url = f"http://127.0.0.1:{args.port}/ws"
        async with aiohttp.ClientSession() as session:
            async with MatterClient(url, session) as client:

                # start listening and continue when the client is initialized
                ready = asyncio.Event()
                asyncio.create_task(client.start_listening(init_ready=ready))
                await ready.wait()

                logging.info(f"Server Info: {client.server_info}")

                # Subscribe to Matter server events
                client.subscribe_events(callback=handleEvent)

                bridge_node_id = None
                bridge_currentLevel_path = None
                feedbackLight_node_id = None
                feedbackLight_colorControl_endpoint = None

                for node in client.get_nodes():
                    print(f"Node ID={node.node_id}, name={node.name}, available={node.available}, device_info={node.device_info}, is_bridge_device={node.is_bridge_device}, endpoints={node.endpoints.values()}")
                    print(f"has cluster OnOff {node.has_cluster(cluster=Clusters.OnOff)}")
                    print(f"has cluster ColorControl {node.has_cluster(cluster=Clusters.ColorControl)}")
                    print(f"is bridge {node.is_bridge_device}")

                    # Since the Matter devices do not provide names (to be implemented) detect devices based on clusters and attributes.
                    if(node.is_bridge_device):
                        # This is the Bridge, subscribe to the current level of the level control cluster
                        bridge_node_id = node.node_id
                        bridge_currentLevel_path = create_attribute_path_from_attribute(
                            3,
                            Clusters.LevelControl.Attributes.CurrentLevel
                        )
                        
                        logging.info(f"Subscribe to path {bridge_currentLevel_path}")
                        await client.subscribe_attribute(
                            bridge_node_id,
                            bridge_currentLevel_path
                        )
                    if(node.has_cluster(cluster=Clusters.ColorControl)):
                        # This is the Feedback Light
                        feedbackLight_node_id = node.node_id
                        feedbackLight_colorControl_endpoint = findFirstEndpointForCluster(node=node, cluster=Clusters.ColorControl)
                
                # Handle the attribute changes
                while True:

                    def mapPostureScoreLevelToHue(level):
                        # Posture score range from 0 (good) to 255 (bad); 256 steps; stepping in positive direction
                        # Hue range from 166 (good) to 216 (bad); 205 steps; stepping in negative direction; roll over at 0 to 255
                        rangeMultiplicator = 205 / 256
                        hueGoodPoint = 166
                        hue = hueGoodPoint - (level * rangeMultiplicator)
                        if hue < 0:
                            hue = 255 - hue
                        return hue

                    levelValue = await queue.get()
                    hueValue = mapPostureScoreLevelToHue(levelValue)
                    logging.info(f"mapped level {levelValue} to hue {hueValue}")
                    try:
                        await client.send_device_command(
                            feedbackLight_node_id,
                            feedbackLight_colorControl_endpoint,
                            Clusters.ColorControl.Commands.MoveToHue(
                                hue=int(hueValue),
                                direction=1,
                                transitionTime=0,
                                optionsMask=0,
                                optionsOverride=0,
                            )
                        )
                    except:
                        logging.error(f"Sending move to hue command failed.")

    async def handle_stop(loop: asyncio.AbstractEventLoop):
        """Handle server stop."""
        logging.info("exit")
        # sys.exit() does not work
        os._exit(0)

    # run the server
    run(run_matter(), shutdown_callback=handle_stop)

Conclusion

I prefer python-matter-server. It is under active development, so I expect that it will be stabilized and further functionality added soon. Although documentation is few, it is good and reading the code to check the API is easy.

Next Steps

Some ideas on what could be done next.

• Further tuning of the posture score. Tuning of the Edge Impulse model and optimization of sensor usage.
• Implementation of further features of the Matter Bridge like multiple devices, or OTA.
• Implementation of an appealing visual and haptic user interface.
• Evaluate the combination with commercial Matter devices.
• Complete the requirements for all parts and introduce testing.

Concluding Words

Thank you for reading this long story to the end. I hope that you found a part useful and could learn something new. The main motivation was to share my learnings and give back something to the community.

This was quite an exciting project where I could try out several technologies like 3D Printing, Edge Impulse AI, BLE OOB, NFC, USB, Matter Protocol or working with LEDs for the first time.

Licenses & Used Libraries and Sources

Used Services

• Edge Impulse - The project is https://studio.edgeimpulse.com/public/276797/latest
• Memfault
• OpenAI

Source code based on or fromusers

• adafruit/Adafruit_BNO08x MIT license. Written by Bryan Siepert for Adafruit Industries
• ceva-dsp/sh2 Apache License, Version 2.0. Written by CEVA
• alexkuhl/colorspace-conversion-library. 3-clause BSD. Written by Alex Kuhl
• Samples in nRF Connect SDK. LicenseRef-Nordic-5-Clause. Written by Nordic Semiconductor
• Samples in Zephyr. Apache 2.0 license. Various Contributors. See link
• TinyUSB. MIT License. Various Contributors. See link
• python-matter-server. Apache 2.0 license. See link

3D Models

• Swirly Mounting Grid for 0.1" PCBs byJMcK. Creative Commons Public Domain. Design based on Adafruit Swirly Aluminum Mounting Grid for 0.1" Spaced PCBs.
• Recreation of Lithophane Lightbox by Desktop Inventions. Creative Commons Attribution.

SDK

• nRF Connect SDK. LicenseRef-Nordic-5-Clause. Written by Nordic Semiconductor
• pico SDK. BSD-3-Clause license. Written by Raspberry Pi Foundation

Documentation tooling

• StrictDoc /strictdoc-project/strictdoc Apache License 2.0
• Plantuml /Download Many license versions -> MIT License Version

Support at DiscordChannel

Thanks to the users who responded to my questions: Silvio, thareeq, Flaming Bandaid Box, raiderOne, tf, NordicSemi_Marte

Disclaimers

The posture checker is NOT a medical device and is not intended to be used as such or as an accessory to such nor diagnose or treat any conditions.