Detecting worker accidents with sound classification

The International Labor Organization estimates that there are over 1 million work-related fatalities each year and millions of workers suffer from workplace accidents. However, even as technology advancements have improved worker safety in the industries, some accidents involving workers and machines remain undetected as they occur, leading to fatalities. This is because of the limitations in Machine Safety Systems. Safety sensors, controllers, switches and other machine accessories have been able to provide safety measures during accidents but some events remain undetected by these systems.

Some accidents which are difficult to be detected in industries includes :

• Falling Objects
• Objects strike or fall on employees
• Slips or Falls of employees
• Chemical burns or exposure
• Workers caught in moving machine parts

Detecting worker accidents with AI 🧠

Sound classification is one of the most widely used applications of Machine Learning. When in danger or scared, we humans respond with audible actions such as screaming, crying, or with words: “stop”, “help”. This alerts other people that we are in trouble and can also give them instructions such as stopping a machine, opening/closing a system. We can use sound classification to give hearing to machines and manufacturing setups so that they can be aware of the environment status.

I used Edge Impulse to develop a machine learning model which is capable of detecting accidents from workers screams and cries. This event can then be used to trigger safety measures such machine/actuator stop and sound alarms.

The Syntiant TinyML Board is a tiny development board with a microphone and accelerometer, USB host microcontroller and an always-on Neural Decision Processor™, featuring ultra low-power consumption, a fully connected neural network architecture, and fully supported by Edge Impulse.

You can find the public Edge Impulse project here: Acoustic Sensing of Worker Accidents. To add this project into your account projects, ensure you have an Edge Impulse account then click “Clone” at the top of the window. Next, go to deploying section of this tutorial to see how you can deploy the model to the Syntiant TinyML board.

Step 1: Preparing the dataset

I wanted to create a model that can recognize both key words and human sounds like cries and screams. For these, I have 4 classes in my model: stop, help, cry and scream. In addition to these classes, we also need another class that is not part of our 4 keywords. I labelled this class as “unknown” and it has sounds of people speaking, machines, vehicles, among others. Each class has 1 second of audio sounds.

In total, I have 31 minutes of data for training and 8 minutes of data for testing. For the unknown class, I used the public available Edge Impulse Key Spotting Dataset . From this dataset I used the “noise” audio files.

Step 2: Impulse Design

The Impulse design was very unique as I was targeting the Syntiant TinyML board. Under ‘Create Impulse’ I set the following configurations:

Window size is 968ms and window increase is 484ms; milliseconds(ms). For the processing block I selected Audio(Syntiant). Next, I set my learning block to Classification(Keras).

Afterwards, I used the processing block (Syntiant) to generate features from the dataset. I used the preset parameter values and they worked well. We can see the features from the dataset in the this 3D visualization. These are the Syntiant blocks that will be passed into the neural network.

The last step was to configure the neural network. Here, I set the number of training cycle as 80 with a learning rate of 0.0005. Edge Impulse automatically designs a default Neural Network architecture that works very well without requiring the parameters to be changed. However, if you wish to update some parameters, Data Augmentation can improve your model accuracy. Try adding noise, masking time and frequency bands and asses your model performance with each setting.

After training the model, I got an accuracy of 94% which is pretty good!

Step 3: Testing the model

When training the model, I used 80% of the data in the dataset. The remaining 20% is used to test the accuracy of the model in classifying unseen data. We need to verify that our model has not overfit by testing it on new data. If your model performs poorly, then it means that it overfit (crammed your dataset). This can be resolved by adding more dataset and/or reconfiguring the processing and learning blocks if needed. Increasing performance tricks can be found in this guide.

My model had a testing performance of 91% which is pretty good and acceptable!

From the testing results we can see new samples called “testing..” which were obtained from the environment and sent to Edge Impulse. The Expected Outcome column shows which class the collected samples belong to. In all cases, my model classifies the sounds correctly as seen in the Result column; it matches the Expected outcome column.

Step 4: Deploying to the Syntiant TinyML board

To deploy the model to the Syntiant Board, first click “Deployment”. Here, we will first deploy our model using the firmware build option. This will get audio data from the environment, and classify the data. When our audible events: cry, scream, help, stop, are detected the onboard RGB LED will turn on. When the unknown sounds are detected, the onboard RGB LED will be off. This runs locally on the board without requiring internet and runs with minimal power consumption.

Under “Build Firmware” select Syntiant TinyML.

Next, we need to configure posterior parameters. These are used to tune the precision and recall of our Neural Network activations, to minimize False Rejection Rate and False Activation Rate. More information on posterior parameters can be found here : Responding to your voice - Syntiant - RC Commands, in the section “Deploying to your device”.

Under “Configure posterior parameters” click “Find posterior parameters”. Check all classes apart from “unknown”, and for calibration dataset we use “No calibration (fastest)”. After setting the configurations, click “Find parameters”.

This will start a new task which we have to wait until it is finished.

When the job is completed, close the popup window and then click “Build” option to build the firmware. The firmware will be downloaded automatically when the build job completes. Once the firmware is downloaded, we first need to unzip it.

Connect a Syntiant TinyML board to your computer using a USB cable. Next, open the unzipped folder and run the flashing script based on your Operating System.

We can connect to the board’s firmware over Serial. To do this, open a terminal, select the COM Port of the Syntiant TinyML board with settings 115200 8-N-1 settings (in Arduino IDE, that is 115200 baud Carriage return).

Sounds such as “stop”, “help”, “aaagh!” or crying will turn the RGB LED to red.

For the “unknown” sounds, the RGB LED is off. While configuring the posterior parameters, the detected classes that we selected are the ones which trigger the RGB LED lighting.

Taking it one step further

We can use the Machine Learning model as a safety feature for actuators, machines or other operations involving people and machines.

To do this we can build custom firmware for our Syntiant TinyML board that turns a GPIO pin HIGH or LOW based on the detected event. The GPIO pin can then be connected to a controller that runs an actuator or a system. The controller can then turn off the actuator or process when a signal is sent by the Syntiant TinyML board.

I created a custom firmware (using Arduino 💯)that turns on GPIO 1 HIGH (3.3V) of the Syntiant TinyML Board whenever the alarming sounds are detected. GPIO 1 is next to the GND pin so we can easily use a 2-pin header to connect our TinyML board with another device.

Awesome! What’s next now? Checkout the custom firmware here and add intelligent sensing to your actuators and also home automation devices!

Intelligent sensing for 8bit LoRaWAN actuator

I leveraged my TinyML solution and used it to add more sensing to my LoRaWAN actuator. I connected the Syntiant TinyML board to an Atmega and SX1276 based development board called the WaziAct. This board is designed to play as a production LoRa actuator node with an onboard relay which I often use to actuate pumps, solenoids, and electrical devices. I programmed the board to read the pin status connected to the Syntiant TinyML board and when a signal is received it stops executing the main tasks. An alert is also sent to the gateway via LoRa while the main tasks remain halted. The Arduino code can be accessed here.

Below is a sneak peak of an indoor test. Now my “press a button” LoRaWAN actuations can run without causing harm such as turning on a faulty device, pouring water via solenoid/pump in unsafe conditions, and other accidental events!

Conclusion

We have seen how we can use sounds to train and deploy our ML solution easily and also run them locally on a development board. This, TinyML based intelligent sensing, is one of the many solutions that TinyML offers.

With Edge Impulse, developing ML models and deploying them has always been easy and fun. Syntiant TinyML board was chosen to deploy our model because it provides ultra-low power consumption, fully connected neural network architecture, has an onboard microphone, not forgetting its tiny size, and is also fully supported by Edge Impulse.