Location Identification using Sound

❓Problem

Many times in our application, we want to identify the environment of a user to automatically carry out certain tasks. For example, automatically switching on/off ANC (Automatic Noise Cancelling) in a headphone based on the user's environment. For such an application a GPS might be overkill and battery drainer.

Can we detect a user's environment without GPS?

💡A Solution

We, humans, are good at understanding our environment just by using auditory sensory skills. I can identify a multitude of sound sources and also guess an environment just by listening. In this project, I am trying to replicate the same behaviour using a TinyML board with a microphone running a machine-learning model.

⚙️ Hardware

This is an interesting problem and I decided to use equally interesting hardware in this project - Syntiant TinyML Board. This board is designed for voice, acoustic event detection and sensor machine learning application.

It is equipped with an ultra-low-power Syntiant NDP101 Neural Decision Processor, a SAMD21 Cortex-M0+ host processor, an onboard microphone and a 6-axis motion sensor on an SD card size package. This board is crazy small and perfect for my application.

Cheery on top is that Edge Impulse officially supports this board.

📈 Training Pipeline

As you might have guessed already, I am using Edge Impulse for preparing a TinyML model. Let's explore our training pipeline.

1. Data Collection

A good machine learning model starts with a good-quality dataset. I am using the ESC-50 datasets to prepare my own.

I have prepared a total of 7 classes denoting various locations by combining some of the classes from the ESC-50 dataset. The classes I have prepared are

• Airport (aeroplane and helicopter sound)
• Bathroom (brushing and toilet flush sound)
• Construction (jackhammer sound)
• Home (washing machine and vacuum cleaner sound)
• Road (siren, car horn and engine sound)
• Work (mouse click and keyboard typing sound)
• Anomaly (quiet environment sound)

I have used only the ESC-50 dataset to prepare a new dataset for this project. The sound samples contained within any of these classes are not the only representation of that class. They can be improved with more sounds from different sources.

The data acquisition tab of the edge impulse studio provides the facility to upload, visualise and edit the dataset. This tab provides obvious functions such as splitting into training and testing and a lot of advanced functions such as filtering, cropping, splitting data into multiple data and many more.

2. Impulse Design and Preprocessing

The 1st thing to do for training an ML model is impulse design. It can also be thought of as a pipeline design from preprocessing to training.

the dataset is of an audio type which is time-series data therefore impulse design tab automatically added the 1st block. Next, is the pre-processing block. Syntiant has prepared a pre-processing block called Audio Syntiant specifically for the NDP101 chip which is similar to Audio MFE but performs some additional processing. Next is the learning block - the classification (Keras) block is perfect for this use case. The last block shows the output features i.e. classes.

In the pre-processing block, start with the default parameters. If required then change the parameters to suit the need. In this project, default parameters worked perfectly so I just used them.

After adjusting the parameters, generate features.

3. Model Training

After pre-processing the dataset, it's time to train a model. In the NN Classifier tab adjust the training cycles, learning rate, etc. For the Syntiant TinyML board, the neural network architecture is fixed. I usually start with the default parameters and based on the performance of the network I adjust the parameter.

My model, trained with 1000 epochs and 0.0002 learning rate has 89.2% accuracy which is not bad. This tab also shows the confusion matrix which is one of the most useful tools to evaluate a model. This confusion matrix shows that the work class is the worst-performing class in our dataset.

At this exact moment, I realised that the work class contains sound samples of only keyboard and mouse operation, assuming people only work on computers. This is called bias, and I have inadvertently included bias in my machine-learning model. To correct my mistake I disabled the work class from the training and test sets using the filters provided by the data acquisition tab.

The pre-processing and NN classifier tab automatically adjusted itself for 6 classes after the dataset was modified.

4. Model Retraining

After updating the dataset, go to the Retrain model tab and click on the Train model.

This will run the pre-processing block and learning block in one click with the last know parameters.

After retraining the model with the same parameters, the training accuracy is 92.1%.

4. Model Testing

To test the model, jump over to the Model testing tab and click on classify all. It will automatically pre-process and perform inference using the last trained model. The testing performance of the model is 84.91%

Road class has the worst performance. Let's see why. Scroll to the sample which was classified incorrectly and click on the 3 dots and then click on the show classification

This will show the result of each window of that sample. In the raw data section, we can also play the audio corresponding to that window, giving us more insight into why the sample was incorrectly classified.

5. Deployment

To deploy the model to the target hardware (Syntiant TinyML in this case), go to the deployment tab. Syntiant TinyML board requires finding posterior parameters before building firmware. For other hardware, it is not required. Select the deployment option (library or firmware) and click build to generate output.

🧪 Live Testing

Let's test the model after deploying it on the Syntiant TinyML board.

🖨 3D Printable Case

To download a protective 3D printable case for the Syntiant TinyML board, please follow the below link

Case - https://discourse.syntiant.com/t/tinyml-board-3d-printable-case/70

📊 Edge Impulse Project

To clone this Edge Impulse project please follow the below link

Project - https://studio.edgeimpulse.com/public/150467/latest