A pet monitoring and alarm device based on Yolov8

Introduction

In this project, I have integrated the Seeed XIAO ESP32S3 Sense into the homeassistant platform via WIFI, turning it into a monitoring module that seamlessly integrates into the home internet system. Additionally, it supports automatic pet recognition beyond its basic monitoring function. We can deploy it in areas where we don't want our pets to enter, such as the kitchen or an open balcony. When the pet enters the designated area and is captured by the camera, homeassistant can promptly send push notifications to our mobile devices. It can also trigger other actions, such as playing sounds or activating toys, to attract the pet to leave the area.

Hardware Overview

The Seeed Studio XIAO Thumb-sized Development Board integrates a camera sensor, a digital microphone, and SD card support. With embedded ML computing capabilities and photography abilities, it becomes an excellent tool for intelligent voice and visual AI.

Features of the Seeed Studio XIAO Thumb-sized Development Board include:

• Powerful MCU Board: It features the ESP32S3 32-bit dual-core Xtensa processor chip, running at a high frequency of up to 240 MHz. It has multiple development ports and supports Arduino/MicroPython.
• Advanced Features (for Sense): It has a detachable OV2640 camera sensor with a resolution of 1600*1200, compatible with the OV5640 camera sensor. It also includes an additional digital microphone.
• Carefully Designed Power: It has lithium battery charging management capabilities and provides four power modes, including a deep sleep mode with power consumption as low as 14μA.
• More Possibilities for Memorable Experiences: It offers 8MB PSRAM and 8MB FLASH, with an SD card slot supporting external 32GB FAT memory.
• Outstanding RF Performance: It supports 2.4GHz Wi-Fi and BLE dual wireless communication. When connected to a U.FL antenna, it supports long-range communication of 100m+.
• Thumb-sized Compact Design: With dimensions of 21x17.5mm, it features the classic shape of the XIAO and is suitable for projects with limited space, such as wearable devices.

Description

The design of this project involves three main components. The first component is the IoT central platform, where I use Homeassistant deployed on a Raspberry Pi. The second component is the Seeed Studio XIAO Sense, and its code is written based on ESPhome. The ESPhome server is also set up on the Raspberry Pi.

The third component is AI image recognition, which is implemented using Yolov8. The code for this is deployed on the Raspberry Pi as well. The results of the recognition are communicated with Homeassistant through MQTT, so we also need to deploy an MQTT broker on the Raspberry Pi.

As you can see, multiple services are deployed on the Raspberry Pi in this project. To ensure future scalability and avoid conflicts between different service environments, all services are deployed using Docker.

How to do it

• Part 1, setting up Docker services on Raspberry Pi:

We need to install the latest 64-bit operating system on your Raspberry Pi, and then install Docker with the following cmds:

# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Add the repository to Apt sources:
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update

sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

After installation, we will create a new docker-compose.yml file and write the deployment information for Homeassistant, ESPhome, and Mosquitto.

version: '3'

services:
  homeassistant:
    image: ghcr.io/home-assistant/home-assistant:stable
    container_name: homeassistant
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /run/dbus:/run/dbus:ro
      - /home/hao/docker/homeassistant/config:/config
    privileged: true
    network_mode: host
    restart: unless-stopped
    
  esphome:
    image: ghcr.io/esphome/esphome
    container_name: esphome
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /home/hao/docker/esphome/config:/config
    environment:
      - USERNAME=admin
      - PASSWORD=admin
    privileged: true
    network_mode: host
    restart: unless-stopped

  mosquitto:
    image: eclipse-mosquitto
    container_name: mosquitto
    ports:
      - 1883:1883
      - 9001:9001
    volumes:
      - /home/hao/docker/mosquitto/mosquitto/config:/mosquitto/config
      - /home/hao/docker/mosquitto/mosquitto/data:/mosquitto/data
      - /home/hao/docker/mosquitto/mosquitto/log:/mosquitto/log
    restart: unless-stopped

The above file can be modified according to your specific needs in terms of the location of volumes.

After completing the setup, run "sudo docker-compose up -d" and wait for the services to finish running. At this point, all three services should be up and running.

However, there is an additional step required. We need to navigate to the "config" folder inside the volume directory of Mosquitto. If the folder is empty, manually create a "mosquitto.conf" file inside it. Then, use the "chown" and "chgrp" commands to change the user and group ownership of the file to 1883. Finally, insert the following content into this file

allow_anonymous true
listener 1883
# password_file /mosquitto/config/auth.pwd

After completing the setup, restart the Mosquitto container to start using it normally. If you wish to enable password protection, first use the following command to enter the container:

sudo docker exec -it mosquitto /bin/sh

After entering the container, use the following command to create a user named "admin". Follow the prompts to enter a password.

mosquitto_passwd -c "/mosquitto/config/auth.pwd" admin

Once done, modify the "mosquitto.conf" file by changing "allow_anonymous" to "false" and uncomment the last line.

• Part 2: Configuring ESPhome on Seeed XIAO ESP32S3 Sense

Find a PC that is on the same router as the Raspberry Pi. Open a browser and enter the IP address of the Raspberry Pi followed by port number 6052. This will take you to the ESPhome configuration interface.

In this interface, select "Add new device" at the bottom left corner, and then choose "Use ESPhome WEB".

Next, press and hold the BOOT button on the Seeed XIAO ESP32S3 Sense board, then connect it to the PC. In the pop-up page, click on "Connect" and select the corresponding port number for the Seeed XIAO ESP32S3 Sense. Click on "Connect" to establish the connection. Then, proceed to flash the initial firmware for first-time use.

After the process is complete, press the Reset button on the Seeed XIAO ESP32S3 Sense board once to go back to the ESPHome Dashboard interface. Click on "Add new device" in the bottom right corner, select "Continue" this time, and follow the guided steps to complete the setup. Once done, you will see a device appearing on the ESPHome Dashboard. Click on "Edit" and add the following code:

esphome:
  name: xiaos3-cam
  friendly_name: xiaos3_cam

  # PlatformIO build options
  platformio_options:
    build_flags: -DBOARD_HAS_PSRAM
    board_build.arduino.memory_type: qio_opi
    board_build.f_flash: 80000000L
    board_build.flash_mode: qio 

esp32:
  board: seeed_xiao_esp32s3
  variant: esp32s3
  framework:
    type: arduino
    version: latest
    platform_version: 6.5.0



# Enable logging
logger:

# Enable Home Assistant API
api:
  encryption:
    key: "xxxxxxxxxxxxxxxxxxx"

ota:
  password: !secret password

wifi:
  ssid: !secret wifi_ssid
  password: !secret wifi_password

  # Enable fallback hotspot (captive portal) in case wifi connection fails
  ap:
    ssid: "Xiaos3-Cam Fallback Hotspot"
    password: !secret password

captive_portal:

# Configuration for the status LED light
light:
  - platform: status_led
    id: light0
    name: "Xiao S3 State"
    pin:
      number: GPIO21
      inverted: true

# Configuration for the binary sensor (Boot Switch)
binary_sensor:    
  - platform: gpio
    pin: 
      number: GPIO2
      mode:
        input: true
        pullup: true
    name: Boot Switch
    internal: true
    on_press:
      # - voice_assistant.start:
      - light.turn_off: light0
    on_release:
      # - voice_assistant.stop:
      - light.turn_on: light0

# Configuration for the ESP32 Camera
esp32_camera:
  id: espcam
  name: My Camera
  external_clock:
    pin: GPIO10
    frequency: 20MHz
  i2c_pins:
    sda: GPIO40
    scl: GPIO39
  data_pins: [GPIO15, GPIO17, GPIO18, GPIO16, GPIO14, GPIO12, GPIO11, GPIO48]
  vsync_pin: GPIO38
  href_pin: GPIO47
  pixel_clock_pin: GPIO13
  resolution: 1280x1024
  
# Configuration for the ESP32 Camera Web Server
esp32_camera_web_server:
  - port: 8080
    mode: stream
  - port: 8081
    mode: snapshot

The API key mentioned in the code should be automatically generated when opening the file. You should replace it with the originally generated key.

After saving, click on "Secrets" in the top right corner and enter the following password information, modifying it according to your specific situation:

# Wi-Fi SSID and password
wifi_ssid: "ssid"
wifi_password: "password"
# Authentication
user: "admin"
password: "admin" 

After saving, click on "Install" and select the wireless installation method to proceed.

After the installation is complete, you can open the log to check if the camera is working properly. If everything is fine, the log will display the address of the camera.

Next, we will enter Homeassistant by typing the IP address of the Raspberry Pi followed by port number 8123. Follow the steps to create a user. Once the initialization is complete, click on "Configuration" and go to "Devices & Services". Add ESPhome and enter the IP address (without the port number) that you saw in the ESPhome Dashboard log, along with the API Key mentioned earlier. This will complete the configuration, and you will be able to see the added device in the Dashboard.

So far, we have successfully connected the Seeed XIAO ESP32S3 Sense to the Homeassistant platform, and it can now be used as a regular surveillance camera.

• Part 3: Configuring Yolov8 on Raspberry Pi and integrating it with Homeassistant for mobile alerts:

Before proceeding to the Python part, we need to configure MQTT. Python will communicate with Homeassistant through MQTT. Just like the previous steps, click on "Configuration", go to "Devices & Services" and choose MQTT.

You can simply fill in "localhost" for the configuration information. If you have set up authentication information, enter the corresponding username and password. If anonymous login is allowed, you can proceed to the next step without providing any credentials.

Next is the Python part. In the latest version of Raspbian, it is not recommended to directly install pip packages in the system's Python environment. Therefore, we need to create a virtual environment using Python venv and then install the required libraries.

pip install ultralytics
pip install ha-mqtt-discoverable

Then create a Python file and write the following code:

from ultralytics import YOLO
from ha_mqtt_discoverable import Settings
from ha_mqtt_discoverable.sensors import BinarySensor, BinarySensorInfo

CAM_URL = "http://xiaos3_ip:8080"
MQTT_USERNAME = "admin"
MQTT_PASSWORD = "admin"

mqtt_settings = Settings.MQTT(
    host="localhost", username=MQTT_USERNAME, password=MQTT_PASSWORD
)
sensor_info = BinarySensorInfo(name="dog", off_delay=3)
mysensor = BinarySensor(Settings(mqtt=mqtt_settings, entity=sensor_info))

model = YOLO("yolov8n.pt")
results = model.predict(CAM_URL, stream=True, show=False, conf=0.5)

# loop
for result in results:
    for box in result.boxes:
        class_id = result.names[box.cls[0].item()]
        cords = box.xyxy[0].tolist()
        cords = [round(x) for x in cords]
        confi = round(box.conf[0].item(), 2)
        print("Object type:", class_id)
        print("Coordinates:", cords)
        print("Probability:", confi)
        print("---")
        if class_id == "dog":
            mysensor.on() 

The top three lines of parameters need to be configured according to your actual situation. If the MQTT broker is set to allow anonymous access, MQTT_USERNAME and MQTT_PASSWORD should be set to None.

The code has two main parts: image recognition based on YOLOV8 and MQTT communication with Home Assistant.

For communication, in order to facilitate automatic discovery and configuration by Home Assistant, we are not using paho-mqtt to send MQTT directly. Instead, we are using a library called ha_mqtt_discoverable, which is based on paho-mqtt. This library enables automatic configuration in Home Assistant. We create a binary sensor using this library. When a pet is detected, the sensor is set to "on", and if the pet is not detected for more than 3 seconds, the sensor is set to "off".

The image recognition part involves retrieving frames from the stream of the Seeed XIAO ESP32S3 Sense and performing object recognition. If a pet is detected among the recognized objects, the binary sensor is turned on. The binary sensor will automatically turn off after 3 seconds. Therefore, if a pet is detected again within the 3-second time frame, the timer will be refreshed. The binary sensor will turn off if the pet is not detected for more than three seconds.

Now, run the code to test it. After running the code for a while (please be patient and wait a few minutes, then refresh the page), you should see a binary sensor named "dog" in the Home Assistant Dashboard. When you point the camera at a pet, you will see the sensor turn on. After the pet is not detected for three seconds, the sensor will turn off.

Finally, we need to configure automation to send push notifications to your phone when a pet is detected, ensuring that you are notified promptly.

Before that, we need to connect your phone to the Home Assistant system. You can search and download the Home Assistant app directly from the app store. Once downloaded, if your phone and Raspberry Pi are on the same network, the app should automatically discover the Home Assistant instance. Simply log in to the app.

Next, go into the automation and scenes configuration in Home Assistant. Click on the bottom right corner to create a new automation. Inside the automation, add the following content: When the binary_sensor.dog entity changes from off to on, trigger a push notification to your phone.

Once the configuration is set, we can test it by pointing the camera at the pet. You should immediately receive a push notification on your phone.

Demonstration

Currently, I have only added one automation, which sends a push notification to the owner when a pet is detected. Of course, there are more possibilities to make it even smarter. For example, you can connect other peripherals to the Home Assistant platform. When a pet is detected, these peripherals can play sounds or perform certain actions to attract the pet away from that area. I haven't demonstrated this further because the configuration steps are the same. I believe that after going through the above content, everyone can use their imagination to configure triggering events according to their own preferences.

I have also uploaded a video demonstration. Although it is in Chinese, you can still see how it works.

https://www.bilibili.com/video/BV16j421m7Fm

Some Thoughts

Although I don't have a pet, I have been researching how to make home automation smarter. This project, using the combination of Python and Home Assistant, has greatly expanded my thinking. By using MQTT as a bridge, we can practically integrate anything into Home Assistant, allowing all aspects of the home to be connected on a single platform and achieve true interoperability.