TinyML: Face Mask Detector using Wio Terminal and UnitV2

I've been recently playing around with TinyML stuff - specifically with Seeed Studio Wio Terminaland M5Stack UnitV2 development boards. Last time, I introduced TinyML: Face Mask Detector with M5Stack Core2 and UnitV2 project and wrote a program using M5Flow and TinyML. This time, I’ll replicate the same program in Python using M5Stack UnitV2, and Arduino for the Seeed Studio Wio Terminal in conjunction with the Telegram messaging app.

Initially, I thought it would be nice if the image data shot by Unit V could be transferred to the WIO Terminal, but the image data size can be quite bulky for the UART interface. UART transmission speed of 115200 bps (bits / sec) can take 16 seconds to transfer one image. Therefore, I decided to store images locally in the M5Stack UnitV2 and send images to messenger Telegramusing Telegram bot.

Project realization

This picture can better explain the overall architecture of project.

As shown in the figure above, it is a structural diagram of the entire system, and there are three main parts involved in coding: M5Stack UnitV2, Seeed Studio Wio Terminal and the Telegram messenger.

There are four main tasks performed here:

• Images are captured by a camera will be saved locally on the M5Stack UnitV2's microSD card.
• Data from M5Stack UnitV2 to Wio Terminal will be sent using Grove cable and communicates through the UART interface (TX, RX).
• Recognized results can be viewed visually from the Wio Terminal screen display.
• Push the for people who are not wearing a face mask, recognized results to the Telegram messenger through the Internet using Telegram API and Bot.

M5Stack UnitV2 - an independent AI camera for edge computing

The M5Stack Unit V2 is a Linux-powered AI camera module released in May 2021 that is the size of a thumb, but has advanced recognition functions.

At the moment, the AI ​​camera functions that you can try immediately are as follows.

• Object recognition & custom training
• Face recognition & detection, face recognition by training
• Automatic tracking of objects
• Shape recognition & matching & automatic area calculation
• Color detection & tracking
• Audio capture from microphone & frequency visualization
• QR code recognition & content reading

For more information, please visit the official website: UnitV2 Documentation.

Firmware download

The firmware update file is available from this link. You can perfrom it by following the firmware update procedure below.

• Download and unzip M5UnitV2RootfsRecoveryPackage-09072021.zip
• Copy M5UnitV2UpdPackage.img in the unzipped folder to the root of the microSD.
• Set the microSD in Unit V2 and connect it to the power supply while pressing the button at the top.
• Release the button when the LED turns red.
• The red LED lights up during the update. (It may takes 4 or 5 minutes.)

SSH connection to M5Stack UnitV2

Perform SSH connection from the terminal while in the M5Stack network.

Username: m5stackIP address: 10.254.239.1Password: 12345678

Start the terminal and execute the following command.

ssh m5stack @ 10.254.239.1

If the command is successful, you will be able to access M5Stack Unit V2 from the terminal.

You might see the below error messages:

sudo: error in /etc/sudo.conf, line 0 while loading plugin "sudoers_policy"
sudo: /usr/libexec/sudo/sudoers.so must be owned by uid 0
sudo: fatal error, unable to load plugins

Please execute following commands to get rid from the above issue.

chown root:root /usr/bin/sudo & chmod 4755 /usr/bin/sudo 
chown root:root /usr/libexec/sudo/sudoers.so 
chown root:root /etc/sudoersThen run

After the change, sudo can be used by the user who logged in to ssh.

Configuring Wi-Fi

Edit /etc/wpa_supplicant.conf

sudo vim /etc/wpa_supplicant.conf

Add the following contents. Enter your Wi-Fi SSID and password.

network={
    ssid="YOUR_SSID_NAME"
    psk="YOUR_PASSWORD"
}

Reboot M5Stack Unit V2. If the connection is successful, the Unit V2 will be able to access from the your local WI-FI network.

Please note that it is only in the 2.4GHz band. You may not be able to connect depending on the router settings. If you have internet connection, your ntp client will update the date and time automatically. Otherwise you have to use a RTC and get date and time on boot with hwclock-w.

Model Training

Through M5Stack's V-Training (Ai model training service) you can easily train the model.

For detailed operation steps, please refer to my previous tutorial - TinyML: Face Mask Detector with M5Stack Core2 and UnitV2.

Using the I2C Grove Port as UART on Wio Terminal

Serial communication is one of the important functions of Wio terminal, which can be used to display debugging information and realize data transfer between devices.

TheWio Terminaldevelopment board, from Seeed Studio, includes two grove ports. Basically, I2C port is used as UART.

The Arduino code is as follows.

#include <wiring_private.h>

static Uart Serial3(&sercom3, PIN_WIRE_SCL, PIN_WIRE_SDA, SERCOM_RX_PAD_1, UART_TX_PAD_0);

void setup() {
    Serial.begin(115200);
    while(!Serial);
    Serial3.begin(115200);
    pinPeripheral(PIN_WIRE_SCL, PIO_SERCOM_ALT);
    pinPeripheral(PIN_WIRE_SDA, PIO_SERCOM_ALT);
}

void loop() {
      if (Serial3.available()) 
    Serial.write(Serial3.read());

}


void SERCOM3_0_Handler()
{
  Serial3.IrqHandler();
}
void SERCOM3_1_Handler()
{
  Serial3.IrqHandler();
}
void SERCOM3_2_Handler()
{
  Serial3.IrqHandler();
}
void SERCOM3_3_Handler()
{
  Serial3.IrqHandler();
}

Next, start your browser from your computer and try accessing the following URLs.

• http://unitv2.py
• http://10.254.239.1/

This will launch the visual editor for AI development. Go to Object recognition tab.

Now open the Serial monitor on Arduino IDE and see the data coming. The data sent by UnitV2 is actually in json format, the style is as follows

{"num":1,"obj":[{"prob":0.415201932,"x":345,"y":102,"w":174,"h":211,"type":"without_mask"}],"running":"Object Recognition"}
{"num":1,"obj":[{"prob":0.481497407,"x":348,"y":104,"w":172,"h":209,"type":"without_mask"}],"running":"Object Recognition"}
{"num":1,"obj":[{"prob":0.44809258,"x":286,"y":8,"w":251,"h":311,"type":"without_mask"}],"running":"Object Recognition"}
{"num":1,"obj":[{"prob":0.500622392,"x":302,"y":10,"w":249,"h":309,"type":"without_mask"}],"running":"Object Recognition"}
{"num":1,"obj":[{"prob":0.489020079,"x":292,"y":1,"w":264,"h":312,"type":"without_mask"}],"running":"Object Recognition"}
{"num":1,"obj":[{"prob":0.462956905,"x":297,"y":2,"w":262,"h":310,"type":"without_mask"}],"running":"Object Recognition"}
{"num":1,"obj":[{"prob":0.463126242,"x":292,"y":0,"w":267,"h":311,"type":"without_mask"}],"running":"Object Recognition"}
{"num":1,"obj":[{"prob":0.498706013,"x":295,"y":2,"w":261,"h":308,"type":"without_mask"}],"running":"Object Recognition"}
{"num":1,"obj":[{"prob":0.432130277,"x":298,"y":0,"w":259,"h":304,"type":"without_mask"}],"running":"Object Recognition"}

M5Stack UnitV2 will transmit the data through the serial port at the same time during the identification process, we can receive the data through the serial port of the computer, and then analyze it.

Steps to configure Telegram Bot API with Python.

Let us build a Telegram Bot that receive images from the M5Stack UnitV2 of people who don't wear face masks in public. These bots can respond to messages, send images, video, audio and even implement new keyboard layouts and games. If your phone doesn’t have Telegram app then go to play store or Iphone’s app store to download.

• First you will need to open the BOTFATHER menu using this link or go to telegram app on your phone, or log in with telegram web and then search for bot father.
• The link will redirect you to your telegram.
• Now you will have a chat opened with the BotFather which you can start it right away by clicking start or writing "/start"
• To create a new Bot you will have to write in the chat "/newbot" and follow the steps to create a new bot.
• To activate your bot, please go to the link that has your bot's username and in that chat click "start" or write "/start"
• At the end of process, you will be given a token, something like 123456789:ABCdefGhIJKlmNoPQRsTUVwxyZ. This token represents the bot account. You are going to put this token onto python code from the M5Stack UnitV2 end.
• To find out the Chad-id run below command.

curl https://api.telegram.org/bot{Your_Tokern}/getUpdates

• Capturing the image using M5Stack UnitV2 and send it.

import requests

img = open("/media/sdcard/2022_03_09/Wio-Terminal-Wiki.jpg", 'rb')
TOKEN = 'YOUR-TELEGRAM_TOKEN'
CHAT_ID = 'YOUR_CHAT_ID'
url = ("https://api.telegram.org/bot"+ TOKEN + "/sendPhoto?chat_id="+CHAT_ID)
print(requests.post(url, files={'photo': img}))

<Response [200]> message indicates successfully sending of the image.

Aside from being a smartphone app, Telegram may also be used on a web browser.

Arduino Code from the Wio Terminal end

The cooperation of the two programs can realize a series of functions of identifying objects, saving to the SD card, image processing, information display, and uploading to Telegram messaging app.

The main tasks performed on the Wio Terminal are: accept the face mask recognition result from M5Stack UintV2 to Wio Terminal via UART and display it visually.

Install all the necessary libraries in the code, as shown below:

#include <wiring_private.h>
#include"TFT_eSPI.h"
#include "Seeed_FS.h"
#include"RawImage.h"
#include"Free_Fonts.h" 

static Uart Serial3(&sercom3, PIN_WIRE_SCL, PIN_WIRE_SDA, SERCOM_RX_PAD_1, UART_TX_PAD_0);
TFT_eSPI tft;

void setup() {
    Serial.begin(115200);
    Serial3.begin(115200);
      if (!SD.begin(SDCARD_SS_PIN, SDCARD_SPI))
  {
    Serial.println("initialization failed!");
    while (1);
  }
    pinPeripheral(PIN_WIRE_SCL, PIO_SERCOM_ALT);
    pinPeripheral(PIN_WIRE_SDA, PIO_SERCOM_ALT);
    tft.begin();
    tft.setRotation(3);
    tft.fillScreen(TFT_WHITE); 

}

void loop() {

while(Serial3.available()) {
    String str= Serial3.readString();// read the incoming data as string
    str.trim();
    Serial.println(str);
    if (str == "without_mask")
    {
       tft.fillScreen(TFT_WHITE); 
      //drawImage<uint16_t> ("no.bmp", 20, 0);  //prints string at (70,80)
      tft.setFreeFont(FF22); //select Free, Mono, Oblique, 12pt.
      tft.drawString("PLEASE WEAR A FASK MASK",20,110);//prints string at (70,110)   
    }
    else if  (str == "with_mask")
    { 
       tft.fillScreen(TFT_WHITE); 
      //drawImage<uint16_t> ("yes.bmp", 20, 0);
      tft.setFreeFont(FF22); //select Free, Mono, Oblique, 12pt.
       tft.drawString("OKAY",20,110);//prints string at (70,110)
    }
    else {
       tft.fillScreen(TFT_WHITE); 
       tft.setFreeFont(FF22); //select Free, Mono, Oblique, 12pt.
       tft.drawString("PLEASE WAIT",20,110);//prints string at (70,110)
      }
    delay(200);
  }
 
}


void SERCOM3_0_Handler()
{
  Serial3.IrqHandler();
}
void SERCOM3_1_Handler()
{
  Serial3.IrqHandler();
}
void SERCOM3_2_Handler()
{
  Serial3.IrqHandler();
}
void SERCOM3_3_Handler()
{
  Serial3.IrqHandler();
}

The code is quite simple.

Maximizing the Performance of the M5Stack Unit V2

M5Stack Unit V2 is based BuildrootEmbedded Linux distribution. It is a simple and efficient embedded Linux systems through cross-compilation. Embedded Linux is often less powerful than desktops in terms of CPU performance and memory capacity, making it difficult to compile your own programs. Since the Unit V2 development board is embedded with the Linux operating system, the computational resources required for the TinyML tasks is limited.

Click gear icon and select Switch to Jupyter notebook mode.

Open Jupyter notebook main page. Click "New" and select "Terminal"

To check the RAM and swap, run

free -m

Output

total        used        free      shared  buff/cache   available
Mem:        114         48          26        0          39        63
Swap:       0           0           0

Running Face mask recognition application will lead to an increased power consumption and will raise the temperature of your device. To prevent throttling keep an eye on the degrees of the CPU temperature and RAM usage.

Python code from the M5Stack Unit V2 end

Create your notebook, click the New drop-down button in the top-right and select Python 3. Remember to insert your bot's token into the code.

from json.decoder import JSONDecodeError
import requests
import subprocess
import json
import base64
import serial
import time
from datetime import datetime
from PIL import Image
import os
import io

TOKEN = 'YOUR_TELEGRAM_TOKEN'
CHAT_ID = 'YOUR_CHAT_ID'
uart_grove = serial.Serial('/dev/ttyS1', 115200, timeout=0.1)
reconizer = subprocess.Popen(['/home/m5stack/payload/bin/object_recognition', '/home/m5stack/payload/uploads/models/v2model_d60c398a896e07db'], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

reconizer.stdin.write("_{\"stream\":1}\r\n".encode('utf-8'))
reconizer.stdin.flush()
url = ("https://api.telegram.org/bot"+ TOKEN + "/sendPhoto?chat_id="+CHAT_ID)

img = b''
while 1:
    today = datetime.now()
    path = str(today.strftime("%Y_%m_%d") + "/")
    newpath="/media/sdcard/"+ path
    doc = json.loads(reconizer.stdout.readline().decode('utf-8'))
    #print(doc)
    if 'img' in doc:
        byte_data = base64.b64decode(doc["img"])
        img = bytes(byte_data)
        #print(img)
    elif 'num' in doc:
        for obj in doc['obj']:
            uart_grove.write(str(obj['type'] + '\n').encode('utf-8'))  #print(obj)
            if obj['type']=="without_mask":
                print('Please wear a face mask')
                if(os.path.exists(newpath)):
                    imagePath = (newpath + today.strftime("%Y_%m_%d"+"_"+ "%H_%M_%S") +".jpg")
                    img = Image.open(io.BytesIO(byte_data))
                    img.save(imagePath, 'jpeg')
                        
                else:
                    os.mkdir(newpath)
                    imagePath = (newpath + today.strftime("%Y_%m_%d"+"_"+ "%H_%M_%S") +".jpg")
                    img = Image.open(io.BytesIO(byte_data))
                    img.save(imagePath, 'jpeg')
                imgopen = open(imagePath, 'rb')        
                requests.post(url, files={'photo': imgopen})
                time.sleep(5)
            else:
                print('Okay')

With the Python program ready to go, it’s time to test everything out. The best way to launch the program is either via a terminal window or Jupiter notebook.

And it works like this:

Open Telegram on your phone, search for your bot using its name or username. And If someone does not wear a mask, it will be reported to the Telegram in real time.

At the end of this tutorial, you have learned how to integrate M5Stack UnitV2 with Telegram Bot. Using simple python code, we can handle the incoming images from M5Stack UnitV2 to the the Telegram Bot.

You can further expand this project! Go ahead!

Conclusion

Compared with Nvidia Jetson nano, M5Stack UnitV2 andSeeed Studio Wio Terminal are much more convenient and simpler. It can give people a quicker understanding of the entire AI edge computing and the entire process.

M5Stack UnitV2 is a very interesting device that allows you to immediately try various recognition functions using the browser of a personal computer or smartphone, but when you try to use it in combination with Seeed Studio Wio Terminal product, the text data of the recognition result can be seen from the Wio terminal display via UART and can be used for further processing, which depends on the user choice.

References

• UnitV2 recognition service
• Wio Terminal Grove UART
• How to send telegram messages with python tutorial
• M5Stack UnitV2 has arrived
• M5Stack UnitV2 - Out of Box Applications
• Tips for hacking M5Stack Unit V2