See Sound in Real-Time Using Your Raspberry Pi Pico 🔊 👀

This guide was created on behalf of the Arm Software Developers team, follow us on Twitter: @ArmSoftwareDev and YouTube: Arm Software Developers for more resources!

Introduction

This guide will walk through creating a real-time audio spectrogram visualizer using a Raspberry Pi Pico board with an external digital microphone and TFT LCD display. It will allow you to see a real-time visual representation of your surrounding audio environment! You can see a preview of the project in action below.

We will use the same Adafruit PDM MEMS breakout board as used in the "Create a USB Microphone with the Raspberry Pi Pico" guide. However, instead of sending the digital audio data over USB to a PC, we will process it directly on the Raspberry Pi Pico using some common Digital Signal Processing (DSP) techniques. The audio signal will be transformed into something that can be displayed on an Adafruit 2.0" 320x240 Color IPS TFT Display with microSD Card Breakout in real-time using DSP techniques.

What is an Audio Spectrogram?

Audio spectrograms are used to visualize audio signals that are represented by amplitude over time into a format that shows how the frequencies contained in the signal change overtime.

In the image below, the left hand side shows the original audio signal and the right hand side shows the audio spectrogram representation of the audio signal.

You can see a direct correlation between the audio signal's amplitude and the associated frequencies in that signal via the color intensity in the spectrogram.

For a more detailed overview of DSP techniques used to create an Audio Spectrogram, checkout the "Fixed-point DSP for Data Scientists" deep dive guide on Towards Data Science.

Spectrogram's are also used Machine Learning (ML) based audio systems, the audio signal is transformed into a spectrogram, so that computer vision techniques can be used to classify the 2D "image representation" of audio signal. Real world examples of this include audio or speech recognition, and keyword spotting.

Processing Pipeline

To create the spectrogram and display it on the LCD display in real-time the following steps will be needed:

1. Collect N audio samples from the digital microphone.

2. Apply a Hanning Window to the collected audio samples.

3. Run an Real Fast Fourier Transform (RFFT) with the input of the previous step.

4. Calculate the magnitude of the RFFT.

5. Map each RFFT magnitude to a color value to display on the LCD display.

6. Display the new row on the LCD.

7. Scroll to the new row and repeat.

If we choose an RFFT size of 256, we will have 128 useable magnitudes outputs to display on the screen, since this is less than the 240 pixels in each row of the display, we can display each row twice to maximize the visual real estate of the display.

For a faster visual response time, we can collect 64 new audio samples from the microphone at a time (instead of waiting for 256 new samples) and combine them with the previous latest 192 ( = 256 - 64) samples for each cycle. With a sample rate of 16 kHz, we will have 64 / 16, 000 seconds to perform all our calculations and update the display. This results in 4 milliseconds per iteration.

We will use the Microphone Library for Pico to capture data from the digital microphone. Arm's CMSIS-DSP library will be used to process the audio data in real-time. CMSIS-DSP is optimized for Arm Cortex-M processors including the Arm Cortex-M0+, which the Raspberry Pi Pico's RP2040 microcontroller (MCU) is based on. The ST7789 Library for Pico will be used to drive the output of the ST7789 TFT display.

Hardware Setup

Solder male headers on to your Raspberry Pi Pico board, the Adafruit PDM MEMS Microphone Breakout board, and the 2" 320x240 Color IPS TFT Display with microSD Card Breakout board so they can be plugged into a breadboard. See The MagPi's "How to solder GPIO pin headers to Raspberry Pi Pico" guide for more details on soldering pin headers to the Raspberry Pi Pico board.

Once both items are soldered, place them on a breadboard and setup the wiring as follows:

Wiring setup in table format:

+---------+-------------------+     +---------+-------------------+
| PDM Mic | Raspberry Pi Pico |     | ST7789  | Raspberry Pi Pico |
|---------+-------------------|     |---------+-------------------|
|    3V   |        3V3        |     |   VIN   |        3V3        |
|---------+-------------------|     |---------+-------------------|
|    GND  |        GND        |     |   GND   |        GND        |
|---------+-------------------|     |---------+-------------------|
|    SEL  |        GND        |     |   SCK   |      GPIO18       |
|---------+-------------------|     |---------+-------------------|
|    DAT  |       GPIO2       |     |   MOSI  |      GPIO19       |
|---------+-------------------|     |---------+-------------------|
|    CLK  |       GPIO3       |     |   CS    |      GPIO17       |
+---------+-------------------+     |---------+-------------------|
                                    |   RST   |      GPIO21       |
                                    |---------+-------------------|
                                    |   D/C   |      GPIO20       |
                                    +---------+-------------------+

Your breadboard should look like this afterwards:

Setting up the Pico SDK development environment

You'll first need to setup you computer with Raspberry Pi's Pico SDK and required toolchains.

See the "Getting started with Raspberry Pi Pico" for more information.

Section 2.1 of the guide can be used for all Operating Systems, followed by the operating specific section:

• Linux: Section 2.2
• macOS: Section 9.1
• Windows: Section 9.2

Getting and compiling the pico-audio-spectrogram application

Make sure the PICO_SDK environment variable is set.

export PICO_SDK_PATH=/path/to/pico-sdk

In a terminal window, clone the git repository and change directories:

cd ~/ 

git clone --recurse-submodules https://github.com/ArmDeveloperEcosystem/audio-spectrogram-example-for-pico.git


cd audio-spectrogram-example-for-pico

Create a build directory and change directories to it:

mkdir build

cd build

Run cmake and make to compile:

cmake .. -DPICO_BOARD=pico

make

Hold down the BOOTSEL button on the board, while plugging the board into your computer with a USB cable.

Copy the audio_spectrogram.uf2 file to the mounted Raspberry Pi Pico boot ROM disk:

cp -a audio_spectrogram.uf2 /Volumes/RPI-RP2/.

Testing it out

You can now try out various sounds, including speaking different words to see how they look in real-time on the spectrogram.

Here is an example of what the speaking the word "yes" looks like on the display:

Similarly this is what the speaking the word "no" looks like on the display:

Examples for for various sounds from the "ESC-50: Dataset for Environmental Sound Classification" can be found below:

Conclusion

This guide covered how you can use a Raspberry Pi Pico board with an external digital microphone and TFT LCD to create a real-time audio spectrogram visualizer. The project uses the Microphone Library for Pico to capture 64 audio samples at a time from the microphone, then transforms the audio samples using Arm's CMSIS-DSP library into a spectrogram, which is then displayed on the TFT LCD display one row at a time using the ST7789 Library for Pico.

You can see the code for the project on the following GitHub repository: https://github.com/ArmDeveloperEcosystem/audio-spectrogram-example-for-pico

Learn More

Up-skill and get hands-on experience using tinyML at the upcoming Arm DevSummit, a 3-day virtual event between October 19–21. The event includes workshops on tinyML computer vision for real-world embedded devices and building large vocabulary voice control with Arm Cortex-M based MCUs. We hope to see you there!

https://devsummit.arm.com

Additional Raspberry Pi RP2040 resources

For more tutorials using the Raspberry Pi RP2040, check out these projects below:

• Create a USB Microphone with the Raspberry Pi Pico
• How to add LoRaWAN to the Raspberry Pi Pico
• Connect your Adafruit Feather RP2040 to The Things Network V3 using LoRaWAN
• Enable Ethernet connectivity on your Raspberry Pi Pico
• Low-power IoT intruder detector with the Raspberry Pi 4 and Pico