Hardware components | ||||||
 |
| × | 1 | |||
Software apps and online services | ||||||
 |
| |||||
 |
| |||||
 |
| |||||
With more and more of my colleagues having babys at home, and showing up completely overfatigued at work, I thought about how to make life a bit easier for them.
I could imagine, that monitoring a baby the whole night can be stressful and exhausting, so why not outsourcing this to AI - the answer to soo many problems these days :)
For this project, I decided to use MicroPython on PSOC™ 6. Why? Because I don't have a lot of time for these fun projects anymore and MicroPython is damn easy to use.
Next to the MicroPython port for PSOC™ 6, Infineon is offering a great tool to create and deploy AI models using embedded devices, such as the PSOC™ 6 AI Kit - DEEPCRAFT™ Studio.
Edge AI has a major advantage for this application compared to cloud-based solutions - the audio data does not leave the microcontroller and is not sent to any cloud, minimizing privacy concerns and removing dependency on an internet connection.
Now, that my colleagues are working on integrating DEEPCRAFT™ models into MicroPython (stay tuned - more soon!), getting this baby cry detection working was a matter of minutes!
It was so easy, that I decided to add a Home Assistant integration on top of it, so you can integrate the baby monitor properly in your smart home.
As in previous projects, I use Home Assistant MQTT Auto Discovery for this.
RequirementsIf you want to rebuild this project you need some basic setup first: A MQTT broker, a Home Assistant instance with the MQTT integration and of course a Wi-Fi network.
Here some useful links to get everything working:
Home Assistant installation: https://www.home-assistant.io/installation/
MQTT broker & integration: https://www.home-assistant.io/integrations/mqtt/
That's it! If you have Home Assistant, a MQTT broker and the MQTT integration running, you're ready to continue.
Now let's dive into the exciting world of edge AI and home automation, and learn how to create a baby cry detector that can make a real difference for new parents.
The TechnologyFor this project I use the PSOC™ 6 AI Kit, which has two Infineon IM72D128 microphones on board. For the baby cry detection I use one of DEEPCRAFT's "starter models", these are example models provided by DEEPCRAFT™ Studio for prototyping purposes.
For once, I use a modified MicroPython firmware for this project. At the moment, this is required in order to include the model in the MicroPython firmware and be able to import it into the application. Soon, it will be possible to put any DEEPCRAFT™-generated model on your PSOC™ 6 board running MicroPython without modifying the firmware, but we need a few more weeks of patience for that.
The Baby-Cry DetectionIn order to detect baby cries I am using a DEEPCRAFT™ Starter Model, which is trained specifically for detecting baby cries using audio input. In order to access the Starter Model, I downloaded DEEPCRAFT™ Studio and opened the "Baby Cry Detection Infineon" example project:
From this project I could generate the model files, which will be used to compile the custom MicroPython firmware for this project.
I am providing the custom MicroPython binary in the attachments of this article and the integration of DEEPCRAFT™ models in MicroPython will soon be possible without compiling custom binaries, that's why I do not describe the process here in detail. However, if you can't wait, have a look here.
Now, let's start by flashing the custom MicroPython firmware to our PSOC™ 6 AI Kit. Download the file "Custom MicroPython Firmware" from the attachments and navigate your command line to the folder containing the firmware file, make sure the downloaded file is called "baby-cry-mpy.hex" (or modify the command below) then execute these commands:
curl -s -L https://raw.githubusercontent.com/infineon/micropython/ports-psoc6-main/tools/psoc6/mpy-psoc6.py > mpy-psoc6.py
mpy-psoc6.py firmware-deploy -f baby-cry-mpy.hex -b CY8CKIT-062S2-AIMake sure to connect your board first. The output of the script should show "Firmware baby-cry-mpy.hex deployed successfully".
Now a custom firmware is running on your MicroPython board, indicated also by the welcome message, which you see once you run the Thonny IDE and connect to your MicroPython board (instructions here):
This custom firmware has integrated the Starter Model for the baby-cry detection, which we will use in the application code.
But first, let's look into setting up the onboard PDM microphones of the PSOC™ 6 AI Kit:
import machine
from machine import PDM_PCM
# Constants (adjust these according to your hardware and requirements)
SAMPLE_RATE_HZ = 16000 # Desired sample rate in Hz
AUDIO_BUFFER_SIZE = 512 # Size of the audio buffer (in samples)
AUDIO_BITS_PER_SAMPLE = 16 # Dynamic range in bits
MICROPHONE_GAIN = 12 # Microphone gain setting (best prediction observed at 12)
SOFTWARE_GAIN = 100.0 # Digital boost factor for input signal
# PDM_PCM configuration
clk_pin = "P10_4"
data_pin = "P10_5"
receive_buffer = array.array('h', [0] * AUDIO_BUFFER_SIZE)
# Set the required clock frequency
machine.freq(machine.AUDIO_PDM_24_576_000_HZ)
# Initialize audio
pdm_pcm = PDM_PCM(
0,
sck=clk_pin,
data=data_pin,
sample_rate=SAMPLE_RATE_HZ,
decimation_rate=64,
bits=PDM_PCM.BITS_16,
format=PDM_PCM.MONO_LEFT,
left_gain=MICROPHONE_GAIN,
right_gain=MICROPHONE_GAIN,
)
pdm_pcm.init()This was a lot of code at once. If you're interested in the details behind this, have a look at this Protip.
Ok, time to set up the AI model. We need to prepare a few variables for this and an output buffer which the model can use to write the results to:
IMAI_DATA_OUT_SYMBOLS = ["unlabelled", "baby_cry"]
# Initialize label scores and labels
label_scores = [0.0] * len(IMAI_DATA_OUT_SYMBOLS)
label_text = IMAI_DATA_OUT_SYMBOLS
output_buffer = array.array('f', [0.0] * len(IMAI_DATA_OUT_SYMBOLS))
# Initialize the model
model.init()The model is trained to accept normalized audio samples between -1 and 1, so we need to prepare a function to make sure that all samples are within this range:
# Function to normalize sample into range [-1, 1]
def sample_normalize(sample):
return sample / float(1 << (AUDIO_BITS_PER_SAMPLE - 1))Now that we have everything set up we can test the baby cry detection. For this, we write audio data from the microphones in the buffer, normalize it and pass it to the model, using the model.enqueue() function:
### AUDIO PROCESSING and BABY CRY DETECTION ###
# Read audio samples into buffer
num_bytes = pdm_pcm.readinto(receive_buffer) # Read audio samples into buffer.
num_samples = int(num_bytes / 2) # Each sample is 2 bytes
for i in range(num_samples):
# Get sample from buffer and amplify it
raw_sample = receive_buffer[i] * SOFTWARE_GAIN
# Normalize the sample to range [-1, 1]
normalized_sample = sample_normalize(raw_sample)
# Pass the sample to the model
model.enqueue([normalized_sample])The model is a time-series model, so it is also considering past values, even though we are only providing one sample at a time to the model. This is also the reason, why the model will not provide valid output directly from the start, but it will take a few seconds until the output is valid.
Let's check if there is valid output already and display it:
# Check if there is any model output to process
if model.dequeue(output_buffer) == 0: # IMAI_RET_SUCCESS
# We have valid data, display it
for idx, score in enumerate(output_buffer):
print(f"Label: {label_text[idx]:<10} Score(%): {score*100:.4f}")This prints the current scores for both available labels, "unlabelled" and "baby_cry", in percent. If we now want to test the baby cry detector, we can put the last two code fragments in a while(1) loop and will get output similar to:
Label: unlabelled Score(%): 100.0000
Label: baby_cry Score(%): 0.0000
Label: unlabelled Score(%): 91.1334
Label: baby_cry Score(%): 8.8666
Label: unlabelled Score(%): 0.0537
Label: baby_cry Score(%): 99.9463
Label: unlabelled Score(%): 0.4597
Label: baby_cry Score(%): 99.5403To test the classifier, I put some baby cries on my speakers - and they were properly detected!
Integration into Home AssistantThe classifier works, but without a proper integration into the smarthome it's not very helpful. I want to have a notification on my phone when baby cries are detected. Home Assistant is a great way to get this done!
I am a fan of the MQTT Auto Discovery feature of Home Assistant, because there is absolutely no additional configuration required on the Home Assistant instance (apart from the basics mentioned in the beginning of this article).
The code for Auto Discovery can become quite long with all the configuration.jsons, so we use the "uhome" module for MicroPython.
Let's start by connecting to WiFi and downloading the uhome and umqtt modules using mip (“mip installs packages”), the MicroPython version of Python's "pip":
### CONNECT TO WIFI ###
sta = network.WLAN(network.STA_IF)
sta.connect(wifi_secrets.ssid, wifi_secrets.psk)
### DOWNLOAD DEPENDENCIES ###
import mip
mip.install('umqtt.simple')
mip.install('umqtt.robust')
mip.install('github:ederjc/uhome/uhome/uhome.py')
import umqtt.robust
import uhomeNote: I am using "secret" files for passing WiFi credentials and other personal information (like the MQTT credentials later on). Feel free to replace these variables by hardcoded credentials in your code or make use of the attached "*_secrets.py" files and modify them.
Now we can make use of the uhome module, which helps us creating & handling the configuration messages for Home Assistant's MQTT Auto Discovery. If you want to learn more about Auto Discovery, have a look into this article, where I show how to manually set everything up without using the uhome module.
Entities & Devices in Home AssistantTo get started, we first need to understand the difference between entities and devices in Home Assistant. A device is usually a physical device, while an entity can be a sensor value, button, etc. of this physical device. Therefore, a device can have multiple entities. In this project, the device could be the PSOC™ 6 AI kit baby cry detector, while entities could be the result of the baby cry detection (detected/not detected) and diagnostic values, like the WiFi signal strength, current IP address, etc. You can read up more about Home Assistant's device registry here.
Setup with uhomeTo setup our device using the uhome module, we can simply create a new object of the device class:
device = uhome.Device('Baby-Cry Detector', manufacturer="Infineon", model="PSOC 6 AI Kit")This device needs to know how to communicate, so we need to create a MQTT client object using the umqtt module and pass this object to the device.connect() method:
mqttc = umqtt.robust.MQTTClient(device.id, mqtt_secrets.broker, user=mqtt_secrets.user, password=mqtt_secrets.password, keepalive=60)
device.connect(mqttc)Now we can create a "binary sensor" entity for the baby cry detector. This entity can have two states (detected/not detected), therefore it's a binary sensor:
baby_cry_entity = uhome.BinarySensor(device, 'Baby Crying')Note that I also have to pass the device object here.
To keep it simple I will not add any diagnostic entities (like signal strength, IP address, etc.) in this project. If you are interested in some examples how to do that have a look here.
Now that we have added the entity we need to "report" it to Home Assistant by sending a configuration message. In the uhome module this can be done by calling this command:
baby_cry_entity.discover()Yes, it's that easy! The device and entity already show up in Home Assistant now :)
As you can see, the entity does not have a state yet. Let's change that by reporting the state in the while(1) loop we set up earlier:
if output_buffer[1] > 0.5:
baby_cry_entity.publish("ON")
else:
baby_cry_entity.publish("OFF")This turns the binary sensor entity on, if the output of our model reports a probability larger than 50% for having detected a baby cry.
Now we need to tell Home Assistant to notify us when a baby cry is detected. For this, we create an automation which will trigger on the entity turning on and send a critical notification to the phone.
For this, we navigate to Settings -> Automations in the Home Assistant interface and create a new automation. The trigger should be set on the "Baby Crying" entity turning on:
And as action we want Home Assistant to send a critical iOS notification (this notification will also show up when the phone is muted or on DnD:
You might have recognized the delay after the notification. The purpose of this is to have a "cooldown time", to avoid getting spammed by notifications when the detector switches between on and off often in a short time span.
DemonstrationOk, let's give it a test! Since I have no crying baby here I take one of the many samples of crying babies available on the internet:
It works! After just a few seconds of baby crying the notification is showing up.
The HousingI designed a really simple housing for this application with two small holes for the board's microphones and of course a hole for the relevant USB-C connector. You can find it in the attachments of this article.
Now it's your turn! What will you build with DEEPCRAFT™ Studio and MicroPython?
Comments
Please log in or sign up to comment.