Keyword Spotting with the SAME54 Curiosity Ultra

This tutorial will guide you through the process of setting up and running a pre-built keyword spotter on a SAME54 using a model developed with Edge Impulse. This tutorial includes the training data, trained model, and deployment code so you can get up and running quickly, but I’ll also be explaining how you can modify the project to develop your own keyword spotter.

Before You Start

Before we get started, you'll need to install and set up the required software as detailed in the steps below.

1. Install the MPLAB IPE tool (available with the MPLAB X installer tool) if you just want to flash the included HEX file. If you plan on compiling your own code, you can skip this step and use MPLAB X to program the board instead.

2. Install the MPLAB X IDE and XC32 compiler. These are required to load the keyword spotting project and to program the SAME54 board. You can use the default, free license for the XC32 compiler as we won't need any of the pro functionality here.

3. Create a free account with Edge Impulse if you haven’t already. We'll use this to process our sensor data and generate the keyword spotting classifier library. The Edge Impulse Studio is an entirely web-based UI so no need to download anything locally.

4. Install the Edge Impulse Command Line Interface (CLI). We'll need this to upload data to Edge Impulse Studio. If you'd prefer not to install this tool, you may use the upload data tool built into Edge Impulse Studio described here.

5. Finally, download the MPLAB X project from the GitHub repository linked to this guide which includes all the source code and configuration files required for this tutorial.

Configuring the Hardware

To enable audio collection for the SAME54, we first need to install the WM8904 daughterboard and configure the board’s jumpers appropriately. We’ll use Figure 2 (taken from the Users Guide) as a map for the different components on the board.

1. Connect the WM8904 daughterboard to the X32 audio interface (labeled 5 above) making sure to orient the board so that the 3.5mm audio jacks face the edge of the Curiosity board.

2. Set the jumpers on the audio codec board to match Figure 3; each jumper should be connected to the two left-most pins on the board when viewed from the perspective of the audio jacks.

3. Set the CLK SELECT jumper (labelled 10 above) so that it connects the MCLK and PA17 pins on the board as shown in Figure 4. This pin configuration lets the WM8904 act as the audio clock master to the SAME54’s I2S peripheral.

4. Connect your microphone to the audio daughterboard’s MIC IN jack.

5. Connect your PC’s USB connection to the Curiosity board’s EDBG micro USB connector (labeled 2 above).

Great! The hardware is now configured for the keyword spot demo project. If you have MPLAB X open, the Curiosity board should be automatically detected by MPLAB X.

For more detailed information about the Curiosity Ultra board including schematics, consult the user guide.

Keyword Spotter Firmware Overview

We’re now set up to run the pre-built keyword spotter firmware. If you'd like, go ahead and program your device with the included HEX file using the MPLAB IPE tool to follow along.

To present the keyword detection status, the firmware flashes the onboard LED1 and LED2 according to the table below. The LEDs can be located at the top left of the development board near the barrel connector; see Figure 5 for reference.

The LEDs will light up selectively based on which keyword is detected. The Program status LED states table shown below summarizes the behavior.

In addition, the firmware also prints the confidence scores for each class over the UART port. To read the UART port use a terminal emulator of your choice (e.g., PuTTY for Windows) with the following settings:

• Baudrate 115200
• Data bits 8
• Stop bits 1
• Parity None

For reference, an example of the serial port output is shown in Figure 7. Notice that the confidence scores are between 0 (no confidence) and 1 (certainty) and the class confidences roughly sum up to 1 (there is some arithmetic error that makes the sum slightly less than 1).

Going Further: Changing Keywords

If you don’t plan on making changes to the keyword spotting model, you can skip this section. If you do want to make changes or simply would like to explore the model development process, read ahead!

Okay, now you have a complete working example, but you might be wondering how you can modify the keyword spotter for your own needs. This section will cover the steps required to modify, re-train, and deploy the Edge Impulse project for a new task: recognizing the commands "up" and "down".

Cloning the Keyword Spotter Impulse

I’ve made my Edge Impulse project available publicly so that you can clone it and make any modifications you like. Public Edge Impulse projects are cloneable similar to how you might clone a git repository but instead of code, the cloned project will include all the labeled audio data, the neural network model, and all of the project’s configuration parameters. Follow the steps below to clone the project.

1. Open your browser and login to your Edge Impulse account.

2. Browse to the keywordspotter_demo project.

3. Click the Clone this project box at the top right of the screen.

4. Wait for cloning to complete (may take several minutes).

You should now have the project cloned to your account enabling you to make any changes you like.

Creating Your Dataset

Rather than collecting our own dataset, we'll be using the Google Speech Commands dataset (v0.02), and some background noise clips that are already included with the Edge Impulse keyword spotting dataset. A pre-built dataset for 'up' 'down' commands is included in the GitHub Releases page of the repository linked to this article, or you can build a dataset with different keywords using the instructions and helper scripts in the dataset subfolder of the GitHub repository here.

1. Download the keywordspotter_demo_updown_data.zip archive from the GitHub Releases page and extract it into a temporary folder. Alternatively, use the instructions provided in the dataset subfolder of the GitHub repository to create a dataset with different keywords.

2. Upload the dataset to your Edge Impulse keyword spotter project with the Edge Impulse Uploader tool. Note that we're letting Edge Impulse automatically select the training and test split of the data (this corresponds to the --category split option).

edge-impulse-uploader --clean
edge-impulse-uploader --label up --category split up/*.wav
edge-impulse-uploader --label down --category split down/*.wav
edge-impulse-uploader --label noise --category split noise/*.wav
edge-impulse-uploader --label unknown --category split unknown/*.wav

Feature Generation

With the data uploaded, we now need to perform the MFCC feature extraction step; this will generate the final form of the data to be used for training the model.

1. Click the MFCC item in the Edge Impulse Studio side menu.

2. On the MFCC page, browse to the Generate features tab and click the Generate features button.

Training Your Model

Now that the MFCC features for the input data have been generated, we can train the neural network.

Note that we are training "from scratch" here rather than using the Retrain model functionality; this is primarily done because we've changed the train and test data split (we don't want to evaluate performance with data we've already trained on), and secondarily because we are changing the class assignment of some of the data ("yes" "no" "unknown") so that transfer learning is unlikely to help us in this context.

Follow the steps below to train the model.

1. Click the NN classifier item in the Edge Impulse Studio side menu.

2. On the NN classifier page, click the Start training button to train the network.

The training should take several minutes. Once completed, Edge Impulse Studio will display a summary of the model's performance and size.

Deploying Your Impulse

When you’re satisfied with your trained model, follow the steps below to deploy the model and integrate it into your existing MPLAB X project:

1. Switch to the Deployment tab in Edge Impulse Studio and select the C/C++ deployment option.

2. Click the Build button that appears at the bottom of the page, leaving the default options selected.

3. Unzip the contents of the edge impulse ZIP file into your MPLAB X project’s src/ folder so that they overwrite the original Edge Impulse SDK files.

4. Rename all the.cc files from the Edge Impulse library to have a CPP suffix. To do this, open a terminal in the src/edge-impulse-sdk directory and run the command below that is appropriate to your OS:

Windows: ren *.cc *.cpp
Mac/Linux: find . -name "*.cc" -exec sh -c 'mv "$1" "${1%.cc}.cpp"' _ {} \;

If you're using the MPLAB X keyword spotter project from this post as a base you shouldn't have to do any modification of the project configuration to use the new code. However, there is a little code in main.cpp that is specific to using the "yes" "no" keywords which we modify in the next section.

Modify main.cpp

Last step; we just need to make a minor modification to the main program file to light the board LEDs when "up" and "down" are detected instead of "yes" and "no" (which are no longer defined as a class in the Edge Impulse code).

1. Open the keyword spotter MPLAB X project, and bring up the main.cpp source file.

2. Inside the main while loop look for the literal strings "yes" and "no" and replace them with "up" and "down" respectively. See Figure 12 for reference.

Flash the Firmware

Okay, modifications to the project are complete, so click Make and Program Device in the toolbar to program the device with the modified keyword spotter firmware. Once programming finishes, your board should respond to the keywords "up" "down" (and reject the keywords "yes" "no").

Final Remarks

That's it! You're now ready to start building your own keyword spotting application.

For more details about integrating your Impulse with an existing MPLAB X project, check out Microchip's Integrating the Edge Impulse Inferencing SDK article.

To learn more about Edge Impulse Studio, including tutorials for other machine learning applications, go to the Edge Impulse Docs.