Keyword detection using Edge Impulse on RASynBoard

Intro

RASynBoard is a TinyML board that combines an on-board digital microphone, IMU, and WIFI/BLE with Syntiant's low-power NDP120 neural accelerator and Renesas' RA6M4 microcontroller.

This makes it perfect for always-on, low-power motion detection and acoustic detection.

The IO board includes GPIO and a debugger, making it easy to program, but the core board can also be detached and used on its own. It can even be powered with a LIPO battery for true TinyML applications.

This project will show how to run a model on the board using the following steps:

• Create a model using Edge Impulse
• Export the model as a.synpkg and place it on the IO board's microSD card
• Build an application for the RA6M4 in e2 studio to test the model. The app will flash the files from the microSD card into the Syntiant NDP120's RAM.
• Debug the application from Renesas e2 Studio

Software Installation

I used a Windows PC to program the board, but Linux is also supported.

1.) Download and install the Renesas e2Studio IDE (version 2022-10 or later)

• Download link for Windows version for RA family: https://www.renesas.com/us/en/software-tool/e2studio-information-ra-family

2) Download and install Renesas Flash Programmer (version 3.11.01, 5 Jan 2023)

• Download link for Windows version: https://www.renesas.com/us/en/document/swe/renesas-flash-programmer-v31101-windows

3) Download and install Tera Term (or equivalent) serial console application

4) Download and install the PL2303 software driver for Windows 10 (for USB-Serial cable) prolific_usb_serial_3.8.28.0(station-drivers).zip

Edge Impulse Setup

Note: there are many tutorials for how to train and deploy ML model via Edge Impulse. I'll be starting with a fully trained example model and focusing on the Syntiant related differences below.

• Login to Edge Impulse (or create a free account)
• You can find all public Edge Impulse projects for Syntiant here.
• Clone the official Syntiant-RC-Go-Stop Nicla project (select "clone this project" in the upper right-hand corner of the dashboard)

Data Collection

This project is meant to recognize the keywords stop and go. There are 2872 samples collected for each word. There's also a third class for background noise / other, with 3944 samples.

Generate Features

In the sidebar, you'll see that below "Create Impulse" the parameter selection and feature generation is now renamed "Syntiant" (normally this tab is named for the parameters being used to find features, for that type of model e.g. "MFCC" or "Image")

You'll need to make sure the NDP120 is selected here.

Train

The model architecture is very simple, with just three dense layers and three dropout layers:

So far, the variety of model architectures available to train the Syntiant board via Edge Impulse are somewhat limited. This is due to the need for fast, low-power inference, which requires extremely optimized models that fit the hardware architecture. Do you want speed, high throughput, or ease of porting your model? Pick two.

Accuracy

For keyword spotting models, high accuracy is required to avoid user frustration.

Edge Impulse makes it easy to see your validation accuracy (on the NN Classifier tab) and test accuracy (on the Model testing tab). They also provide a confusion matrix and data explorer to make it easy to visualize which samples the model got wrong.

The validation accuracy for the quantized model is 98%.

You can also see the unquantized model's validation accuracy by selecting "unoptimized (float32)" from the dropdown. At 97.9%, the unquantized validation accuracy is slightly lower than the quantized version, which points to possible overfitting. The test accuracy drops to 96%, which also suggests overfitting.

At this point, you might want to retrain the model with more data.

• One way is to use data augmentation. On the NN Classifier tab, select "Data augmentation." This will bring up a number of settings you can experiment with. After retraining with these settings, it brought my accuracy up to 96.04.

• You can also send your friends and relatives a QR code so they can record "stop" and "go" with their phone microphones and it will get added to the dataset. Go to the Data Aquisition tab and select "connect a device." Screenshot the QR code and spam everyone you know!

Deployment

On the deployment tab:

• Under Create Library, select "Syntiant NDP120 Library." This will bring up the "Configure posterior parameters" section.
• Select Find posterior parameters.

What are posterior parameters?

This is a post-processing algorithm that selects the tradeoff point between precision and recall (i.e., sensitivity and specificity). You can manually edit those parameters in JSON format or use the Find posterior parameters to search for the best values. It seems that these make a huge difference in the model's accuracy.

• Select "go" and "stop" as the classes (the z_openset class should be omitted except for testing purpose)
• Select "English radio (10 minutes)"
• Select Find Parameters.
• Once posterior parameters have been found, select Build.

Don't navigate away from this page while the project is building.

Once the project is built, it will automatically download as a zip file with the project files inside.

Drag the model to your microSD card

The RA6M4 demo application will attempt to load three.synpkg firmware files into the RAM memory of the NDP120 device:

• mcu_fw_120.synpkg
• dsp_firmware_noaec_ff.synpkg
• ei_model.synpkg

These need to be manually flashed onto the micro SD card, using an SD card adapter, then fitted back into the sdcard cage on back of RASynBoard.

• Unzip the zip file you found in the previous step.
• Locate the file ei_model.synpkg within the downloaded folder:

This is the model you trained in the previous step. Drag the ei_model.synpkg to your micro SD card.

The MCU and DSP files come from the demo project, so we'll access them in the next step.

Open the demo project in e2 Studio

You can find all of the RASynBoard related projects on Avnet's github: https://github.com/Avnet/RASynBoard_Projects

We'll be reusing a default application to test our model today.

• Download the rasyn_power_enable project from Avnet's github as a zip file

To import and build the project rasyn_power_enable please use the following steps:

• Open the e2-studio, select “File -> Import” to open the Import dialog
• Select the rasynboard_ew_demo zip file in the subsequent Import dialog
• After importing an existing project into your workspace, when using this for first time, you must open the (FSP) configuration and click on Generate Project Content, before attempting to build the project!
• Click OK on the dialog warning that a different FSP version will be used, after the FSP panel opens-up, click on Generate Project Content (in top right-hand corner) note: if you don't see it, you can find it by going to Project > Open FSP Configuration
• Now with the project name highlighted in Project Explorer left sidebar, click on the Build (hammer) icon on the toolbar to compile the project

Notes regarding Source Code in this Application

• The src folder contains the user source files for this custom application
• The ndp120 folder contains the Syntiant NDP120 SDK library files with APIs accessed in this project

Drag the MCU and DSP files to your SD card

Within ndp120/synpkg_files, find the files:

• mcu_fw_120.synpkg
• dsp_firmware.synpkg

Drag and drop these to your SD card. Eject the micro SD card from your PC and fit it onto the back of the IO Board.

Set up hardware for debug

You are now ready to test the model!

The IO Board and Core Board have three connectors that can be used to power RASynBoard, depending on the use case:

Remove the jumper (if fitted) that bridges MD and GND of the E2OB debugger MCU from the right-angle header at edge of the IO board.

• Connect RASynBoard to your PC via the USB-C connector on its IO board. This cable is used for both debug and power during development. RASynBoard should show-up as Renesas E2 Lite under Renesas USB Development Tools in your Windows Device Manager.
• For Printf debug output, use an external USB-Serial adapter cable as shown in the photo below, plugged into the PMOD connector.

Debug the program in Renesas E2 Studio

In E2 Studio, set up the following debug perspective:

• With the project name selected, open the Run menu, then select Debug Configurations…
• With the rasyn_power_enable Debug configuration selected in the left panel, open the Debugger tab, set the Debug hardware to E2 Lite (Arm), then select Connection Settings and make sure all settings match what is shown below…
• Once the Debug Configuration is correctly set, apply these settings, then launch Debug (Debug can be launched from within this form, or by using the Debug icon at top left of the IDE)
• If all succeeds, it will ask you to confirm perspective switch and you'll be switched to debugging mode.

Say "stop" and "go" several times. You should see the board's LED blink blue when it recognizes "stop."

You will also see the model's output in TeraTerm. Open TeraTerm or your serial program to view printf debug. Tera Term baud rate is 115200 8N1:

Note: if you get an error "config_clk_pll failed s: 1", simply eject the microSD card and then replace it and it should start up correctly.

Going Further

Flash the project using Renesas Flash Programmer

Now that you've built your project in E2 Studio, you can also flash the prebuilt binary file to the board using Renesas Flash Programmer.

• Download Renesas Flash Programmer here: https://www.renesas.com/rfp
• Download link for the Windows version is here.

You can find the prebuilt RA6M4 binary file in the debug folder of your project in E2 Studio:

To allow programming (and SWD debug) of the RA6M4 MCU via the E2OB debugger on the IO Board:

• Remove the 2mm shorting jumper from J3 on the IO Board
• Remove the 1.27mm shorting jumper from J5 on the Core Board (if there is one fitted)
• Connect USB cable from IO Board USB-C connector, to a USB-A port on the development PC

• Open Renesas Flash Programmer
• Under the Operation tab → Program File, click browse, then select the pre-built binary file (.SREC)
• Click the Start button to program the .srec file into the RA6M4 MCU flash memory

Once successfully programmed, powerdown RASynBoard by disconnecting the USB cable from the development computer.

• Fit the 2mm shorting jumper across pins 1 & 2 of J3 on the IO Board (to strap E2OB in Reset mode)
• Reconnect USB cable from USB-C on IO Board to the development computer or other +5V source
• Provided the uSD card has the expected three .synpkg files, the RGB-Blue LED should illuminate for about 3 seconds while these files are read from the SD card, and loaded into the RAM of the NDP12

There are many things you could do with a Stop / Go model. Maybe you could make use of the RASynBoard's PMOD connector to control a robot. You could also remove the core board and add a LIPO to be able to use it on its own.