Pet detector with Seeed Studio Grove Vision AI V2

Pet Warning System

For a couple a weeks now I have a 5 months old dog pup. It's very energetic and always happy to wait for us when we arrive at home.

To enter my garage I have to go through a ramp downwards, in reverse - When I open the garage door, he will hear the garage opening and he will come running like there's no tomorrow.

I wanted a way to know if he is around the garage, so I can enter with the car safely.

I will use the Seeed Studio Grove Vision AI V2 to detect the dog pup (his name is Caju - cashew in English), it will send an alert to a Xiao ESP32S3 that I will have in the car, and it will turn on and off a red light connected to it.

The connection is done using MQTT.

Since I have Home Assistant with MQTT running (Mosquitto), I'm be using it to publish and subscribe a topic.

To power them (the detector and the warning light) I will use their battery capabilities (both have battery pads bellow) and they will also charge them when connected through the USB port.

Seeed Studio Grove Vision AI V2

The latest Grove Vision AI V2 is an awesome MCU.

We can use Machine Learning algorithms without having to train them ourselves.

It supports SenseCraft. Using SenseCraft AI we can deploy already trained modules directly onto the Grove Vision AI V2 and use it right away.

The model

Because I wanted to detect a dog, I browsed the available models and found one that would do perfectly. A model called Pet Detection, by SenseCraft AI.

I know the model has two classes - cat and dog - but it really doesn't matter. A pet is a pet and I won't be looking into that.

Deploying the model

To deploy the model, we just need to connect our board to our computer using a USB-C cable.

For this, I'm using Chromium with Linux, because my browser of choice - Firefox - does not work.

In the model page, press the deploy model button

It will lead us to a page where it shows the supported devices

Press the connect button

A new window will appear to confirm the board. Select it.

It will now flash the model onto the device. The messages on the screen will appear twice, so don't worry.

After flashing the model, you can immediately test it.

Here's an example with a photo of Caju.

Testing the model

Here's the model running having Caju as a model !

Arduino

The Grove Vision AI V2 is really awesome and it does a lot on its own, but to really unlock the potential of ML at the edge, we need to pair it with a XIAO.

First, let's add the library that will allow us to process the stream coming from the Grove Vision AI V2. You can find more information on the Seeed Studio Wiki.

On the Seeed Studio Wiki for the Grove Vision V2, press the button to download the necessary libraries.

On the Github page, press the code icon and select "Download ZIP".

Once you have the ZIP file downloaded, let's add it to the Arduino IDE libraries.

Open your Arduino IDE and head to Sketch > Include Library > Add.ZIP Library

Choose the location of the ZIP file and add it. After a while, you get a message of successfully installed library.

Now, we can open an example.

Go to examples > Seeed Arduino SSCMA > inference

To work with the example, like I've explained before, we need to pair the Grove Vision AI V2 with another XIAO. On my case, I'm going to use a Xiao ESP32-C3, but you can use a ESP32S3, for example.

After connecting the Xiao ESP32-C3, upload the code to it.

To upload the code to it, we need to be pressing the boot button and connect the USB cable. After that, we can upload the code.

It still not ready. We need to connect it to the Grove Vision AI V2. We can also do the previous step (uploading the code), with both connected.

Just connect the XIAO ESP32-C3 to the Grove Vision AI V2, with the USB port pointing in the same direction.

After uploading and connecting both, let's start detecting our pets.

In the Arduino IDE, open the serial monitor (Ctrl+Alt+M) and watch the output:

While not detecting anything, we just see some information, like the one above.

If we point the camera to a pet (dog or cat), we see the information has changed:

We now get a score. This is how confident our algorithm is that what's seeing is a dog or a cat.

MQTT

I'm going to be using the MQTT protocol to exchange messages between the MCUs.

It's a lightweight protocol, perfect for IoT (Internet of Things). It works by subscribing topics and publishing information to those topics, that the devices subscribing them will receive.

For the exchange of messages, we need a MQTT broker (a server). One of the most used is Eclipse Mosquitto.

Because I have an installation of Home Assistant, I already have a mosquitto broker running there, so I'm going to use it.

To debug and testing, you can use a MQTT client - it connects to the broker and you can subscribe and publish to topics. I like to use MQTT Explorer.

The code

Copy this code and upload it to the Xiao connected physically to the Grove Vision AI V2

#include <Seeed_Arduino_SSCMA.h>
#include <WiFi.h>
#include <PubSubClient.h>

#define DEBUG 0
//#define DEBUG 1

SSCMA AI;
const char* ssid = "<wifi_ssid>";
const char* password = "<wifi_password>";

// time after no detection warning is removed - in minutes
// if no pet detection during this time
// remove warning
const unsigned long warningTime = 1;

unsigned long controlTime = 0; //control variable
unsigned long startTime;

//set score minimal
// only with score above this we say it's detected
const int score = 70;

// MQTT stuff
const char *mqtt_broker = "<broker_ip_address>";
const char *mqtt_topic = "topic_to_publish";
const char *mqtt_username = "mosquitto_user";
const char *mqtt_password = "mosquitto_password";
const int mqtt_port = 1883;

WiFiClient espClient;
PubSubClient clientMqtt (mqtt_broker, mqtt_port, espClient);

// we define a function just to connect to MQTT
// because disconnect time
// we can't keep a connection opened for too long
void connectMQTT() {
    while (!clientMqtt.connected()) {
        String client_id = "Xiao-VisionV2";
        Serial.println("The client %s connects to MQTT broker!");
        if (clientMqtt.connect(client_id.c_str(), mqtt_username, mqtt_password)) {
            Serial.println("Connected to MQTT Broker");
        }
        else {
            Serial.print("failed with state: ");
            Serial.println(clientMqtt.state());
        }
    }
}

void setup() {
    AI.begin();
    Serial.begin(9600);
    // Connect to Wi-Fi
    WiFi.mode(WIFI_STA);
    Serial.println();
    Serial.print("Connecting to ");
    Serial.println(ssid);
    WiFi.begin(ssid, password);

    while (WiFi.status() != WL_CONNECTED) {
        Serial.print(".");
        delay(500);
    }
    Serial.println();
    Serial.print("IP Address: ");
    Serial.println(WiFi.localIP());
}

void loop() {
    //start counting time
    startTime = millis();
    if (!AI.invoke()) {
        if (DEBUG) {
            Serial.println("invoke success");
            Serial.print("perf: prepocess=");
            Serial.print(AI.perf().prepocess);
            Serial.print(", inference=");
            Serial.print(AI.perf().inference);
            Serial.print(", postpocess=");
            Serial.println(AI.perf().postprocess);
        }
        for (int i = 0; i < AI.boxes().size(); i++) {
            if (DEBUG) {
                Serial.print("Box[");
                Serial.print(i);
                Serial.print("] target=");
                Serial.print(AI.boxes()[i].target);
                Serial.print(", score=");
                Serial.print(AI.boxes()[i].score);
                Serial.print(", x=");
                Serial.print(AI.boxes()[i].x);
                Serial.print(", y=");
                Serial.print(AI.boxes()[i].y);
                Serial.print(", w=");
                Serial.print(AI.boxes()[i].w);
                Serial.print(", h=");
                Serial.println(AI.boxes()[i].h);
            }
            /*
            * Check if we are detecting a dog
            */
            if (AI.boxes()[i].score >= score) {
                connectMQTT();
                //publish mqtt message
                clientMqtt.publish(mqtt_topic, "1");
                controlTime = startTime; //reset control time
                clientMqtt.disconnect();
            }
        }
    }
    //check if since last detection have passed warningTime
    //so we can safely turn off the warning light
    if ((startTime - controlTime) >= (warningTime * 60000)) {
        connectMQTT();
        //turn the warning off
        Serial.println ("Disabling warning");
        clientMqtt.publish(mqtt_topic, "0");
        //disconnect from MQTT
        clientMqtt.disconnect();
    }
    sleep(2); //sleep 2 seconds
}

The Seeed Studio Wiki page for the Arduino functions explain them very well, so I'm not going through them here. I'm only going to explain my changes.

We're going to use the PubSubClient library for the MQTT and WiFi library for the WiFi.

We start by defining our WiFi network settings.

//WiFi settings
const char* ssid = "<your_ssid>";
const char* password = "<your_password>";

Next, we set a variable that contains the minimal score from which we assume we're detecting a Pet.

// set minimal score threshold
// only with score above this we say it's detected
const int score = 70;

The following variables are used to set the time by which the warning time is in effect.

The warningTime variable sets the time, in minutes, since the last detection of a Pet that we keep the warning status in effect. After that time is passed and no Pet is detected, we turn the warning led off.

The controlTime variable is reset every time a detection is successful, by setting it to the same time as the startTime variable.

The startTime variable is set in every loop()

// time after no detection warning is removed - in minutes
// if no pet detection during this time
// remove warning
const unsigned long warningTime = 1;
unsigned long controlTime = 0; //control variable
unsigned long startTime;

Next, the MQTT variables to connect to the broker. The broker, the topic to publish to (both publish and subscribe topics must be the same), user, password and port.

// MQTT stuff
const char *mqtt_broker = "<broker_ip_address>";
const char *mqtt_topic = "topic_to_publish";
const char *mqtt_username = "mosquitto_user";
const char *mqtt_password = "mosquitto_password";
const int mqtt_port = 1883;

Next, the variables to use the MQTT protocol with, from PubSubClient. We set the connection defining the broker and port.

WiFiClient espClient;
PubSubClient clientMqtt (mqtt_broker, mqtt_port, espClient);

Next we define a function to connect to MQTT. Because we can't keep a connection opened always, defining a function to this is better.

Every time we need to publish a topic, we connect to the broker using this function, publish the topic and then close the connection.

The client_id is a string that identifies our client. You can specify anything here.

void connectMQTT() {
    while (!clientMqtt.connected()) {
        String client_id = "Xiao-VisionV2";
        Serial.println("The client %s connects to MQTT broker!");
        if (clientMqtt.connect(client_id.c_str(), mqtt_username, mqtt_password)) {
            Serial.println("Connected to MQTT Broker");
        }
        else {
            Serial.print("failed with state: ");
            Serial.println(clientMqtt.state());
        }
    }
}

The setup function - that only runs once; when the MCU is turned on - will set some variables and establish the WiFi connection. It also initializes the AI module.

In the loop function, it repeatedly calls the invoke function to perform inference using the algorithms of Grove Vision AI V2.

if (!AI.invoke())

If success, it prints some metrics that I'm only printing if DEBUG is active.

Next, it goes on printing the bounding boxes that identify the locations and dimensions of the detected objects.

for (int i = 0; i < AI.boxes().size(); i++)

It's here, inside this for loop that we can see the scores of the detection.

I'm a bit stumped here because this value should have come from the classes - but the truth is, I get nothing from the classes - try the example code from the library and you'll see what I'm talking about. At least, for this model from SenseCraft.

Inside this for loop, if the score from the inference is the same or bigger than the one we've defined, we're detecting our Pet and let's throw a warning.

/*
* Check if we are detecting a pet
*/
if (AI.boxes()[i].score >= score) {
    connectMQTT();
    //publish mqtt message
    clientMqtt.publish(mqtt_topic, "1");
    controlTime = startTime; //reset control time
    clientMqtt.disconnect();
}

We open a connection to the MQTT broker and publish our topic, by setting it to "1".

Next, we reset the time of the warning and close the connection to the broker.

Now, we're going to check if since the last Pet detecting, the time elapsed is bigger or equal to the warningTime.

If it is, it's because we no longer detected our Pet and it's safe to assume he's no longer around and we can turn off the warning light.

//check if since last detection have passed warningTime
//so we can safely turn off the warning light
if ((startTime - controlTime) >= (warningTime * 60000)) {
    connectMQTT();
    //turn the warning off
    Serial.println ("Disabling warning");
    clientMqtt.publish(mqtt_topic, "0");
    //disconnect from MQTT
    clientMqtt.disconnect();
}

To calculate the time elapsed, every time we start a new loop, we store the current time.

If we detect our Pet, the controlTime is set to the same time as startTime. The controlTime variable starts at 0 when the program starts.

If we keep detecting our Pet, this variable gets set to the same value as startTime.

If we don't detect our Pet, startTime keeps resetting, but controlTime keeps the last time our Pet was detected.

When the time difference between the startTime and the controlTime is bigger than the warningTime (this is the time we've set that is safe to assume our Pet is no longer around), we turn the warning Led off by publishing 0 to our topic.

The warningTime is multiplied by 60.000 because we've set this variable in minutes. startTime and controlTime are in milliseconds. 1 minute is 60.000 milliseconds and we're converting it.

We then wait 2 seconds and start all over again.

I also have a version that does all this, and sends a message to a Telegram chat.

Here's the schematic for the detection system. The Xiao ESPS3-C3 is plugged in on the Grove Vision V2 Pins, but the power button and battery are wired to the battery pads bellow the ESP32S3-C3.

NOTE: In image there's a Grove Vision AI V1 because I couldn't find a V2 to use with Fritzing.

Building it - and before putting it in a 3D enclosure -

Warning Light

For the warning light, the setup is really easy. I paired a Xiao ESP32-S3 running MicroPython with a red LED.

Here's the wiring schematic. Nothing fancy.

Note: The Pin is GPIO9. In the image is a ESP32-C3 and the GPIO9 is not in the same place of the ESP32-S3, but the wiring is accurate for the ESP32-C3.

I've installed MicroPython following the instructions in the Seeed Studio Wiki. I've used the MicroPython version available there.

Next, using Thonny, a beginner friendly Python IDE that work's very well with MicroPython, I've uploaded the following code.

boot.py

The boot.py file is executed every time the MCU boots. I'm using it to connect to my WiFi network.

import network
# define connect to wifi
def do_connect():
    sta_if = network.WLAN(network.STA_IF)
    if not sta_if.isconnected():
        print ("Connecting to network...")
        sta_if.active(True)
        sta_if.connect('<sid>','<password>')
        while not sta_if.isconnected():
            pass
    print ("Network config: ",sta_if.ifconfig())


do_connect()

main.py

The main.py file is executed always, kinda like the loop() function from Arduino.

import ubinascii
import machine
import time
import _thread
from umqtt.simple import MQTTClient

# Led wiring Pin
led = machine.Pin(9, machine.Pin.OUT)

# this will control if we turn on or off the led
ledValue = 0

# control if we have a warning or not
warning = 0

# MQTT Broker
MQTT_BROKER = "broker_ip_address"
ClientID = ubinascii.hexlify(machine.unique_id())
topic = b"cajuWarning"
user = "mosquitto_user"
password = "mosquitto_password"

# callback function
def warningFunction(topic, msg):
    global warning
    print (f"Received message {msg} on topic {topic}")
    if (msg.decode() == "1"):
        warning = 1
    else:
        warning = 0
    print (f"Warning is: {warning}")

# Warning light
def warningLight():
    led.value(ledValue)
    
#connect
print (f"Connecting to MQTT :: {MQTT_BROKER}")
mqttClient = MQTTClient(ClientID, MQTT_BROKER, 1883, user, password)
mqttClient.set_callback(warningFunction)
mqttClient.connect()
print (f"Connected to :: {MQTT_BROKER}")
mqttClient.subscribe(topic)
while True:
    # wait for messages
    # this is non blocking - we must keep turning on and off the LED
    # there's another alternative that is wait_msg - it's blocking.
    # It will wait for a message and block until then
    mqttClient.check_msg()
    if (ledValue == 0 and warning == 1):
        ledValue = 1
    else:
        ledValue = 0
    warningLight()
    print (f"Warning: {warning}\t Led Value: {ledValue}")
    time.sleep(0.2)

The code

We start by declaring the led Pin.

# Led wiring Pin
led = machine.Pin(9, machine.Pin.OUT)

We start by declaring the state of the LED - 0, turned off.

Next, we need to control if we're on a warning state or not. Only on a warning state, we turning the LED on or off.

# this will control if we turn on or off the led
ledValue = 0
# control if we have a warning or not
warning = 0

Next, we have the MQTT definition settings.

The broker IP, user and password. Also, the topic we're going to subscribe - and see if we get a warning or not - pet is detected.

# MQTT Broker
MQTT_BROKER = "broker_ip_address"
ClientID = ubinascii.hexlify(machine.unique_id())
topic = b"cajuWarning"
user = "mosquitto_user"
password = "mosquitto_password"

The callback function will be called when a message is posted to the topic.

It's here that, if we get an value of 1 in the topic, a pet is detected and we enter a warning state, by changing the warning variable value to 1. If the topic value is 0, the warning state is terminated by changing the warning variable value to 0.

We declare the warning variable as global because it's the same variable declared in the top of the main.py and used in the main loop.

# callback function
def warningFunction(topic, msg):
    global warning
    # print (f"Received message {msg} on topic {topic}")
    if (msg.decode() == "1"):
        warning = 1
    else:
        warning = 0
    # print (f"Warning is: {warning}")

Next, we have declared the function that will turn on or off the LED, according with the ledValue variable.

# Warning light
def warningLight(ledValue):
    led.value(ledValue)

Now, we connect to the MQTT broker, define the callback function and subscribe to our topic.

#connect
print (f"Connecting to MQTT :: {MQTT_BROKER}")
mqttClient = MQTTClient(ClientID, MQTT_BROKER, 1883, user, password)
mqttClient.set_callback(warningFunction)
mqttClient.connect()
print (f"Connected to :: {MQTT_BROKER}")
mqttClient.subscribe(topic)

The main loop is where we check for new messages on the subscribed topic.

We start by checking for messages using check_msg. This is a non blocking function. It checks for new messages on the topic. If there is, the callback function is called.

After the callback is terminated, we check the value of the variables.

If we are at a warning state and the LED value is 0, we change it to 1 and call the function to turn the led light.

We wait 200ms.

Next iteration, the ledValue is still 1, so we enter the else: instructions - turn the ledValue to 0, turning off the led.

Again, wait 200ms.

Next iteration, the ledValue is 0, but we're still at a warning state, we turn the led on again and keep doing this until the warning state changes to 0.

It's this simple.

while True:
    # wait for messages
    # this is non blocking - we must keep turning on and off the LED
    # there's another alternative that is wait_msg - it's blocking.
    # It will wait for a message and block until then
    mqttClient.check_msg()
    if (ledValue == 0 and warning == 1):
        ledValue = 1
    else:
        ledValue = 0
    warningLight(ledValue)
    print (f"Warning: {warning}\t Led Value: {ledValue}")
    time.sleep(0.2)

When the main loop is running, here's the output

(Video)

Conclusion

This was fun to build. I will use it as soon as I build the 3D enclosures. I will also post them here.

The model is not very accurate. I don't know how it was build, but I had a lots of false positives. The was no pet in sight, but I was getting a warning. The opposite was also true - Caju was in sight, but the model didn't detect anything. I even lowered the score, but that increased the rate of false positives and didn't help with the false negatives.

Probably the best option is to train it myself (Roboflow has a lot of models already trained or a lot of datasets ready to use) and upload it to SenseCraft to deploy it.