Kria Kitchen ML Lifeguard

Unattended cooking

Unattended cooking is one of the first three causes of fires in homes, especially for older people with memory problems or narcolepsy or very busy people. With the pandemic and working from home, the problems of distractions while cooking increase.

There are devices to solve this problem by connecting the device to the kitchen appliance, which has already caused fires in some cases due to poor installation. They also detect the presence of people using presence sensors that are not able to discriminate whether it is a person or a pet.

The Kitchen Lifeguard

The Kitchen ML Lifeguard is an intelligent system that monitors the kitchen using computer vision with Machine Learning. It is capable of detecting whether a person has been aware of kitchen cooking appliances and has not left them unattended for a long time.

How it works

The Kitchen ML Lifeguard is an intelligent device based in the Xilinx Kria KV260 that monitors the kitchen appliances through an array of thermal sensors capable of detecting if there are objects in the fire and if there are people watching. In the case in which during a programmable time high temperature is detected on the stove but not people, it will launch an audible alarm and a programmable notification to mobile, clock, mail, … to notify the user or family members or care services for the elderly.

The system is developed using the Xilinx Vitis AI development environment based int the Vitis-AI/demo at master · Xilinx/Vitis-AI (github.com). It makes easy to develop deep-learning inference applications by abstracting the intricacies of the underlying FPGA and ACAP.

Kitchen Lifeguard System Architecture

The unattended kitchen surveillance system is made up of two subsystems that communicate via TCP/IP

• The people detection system
• The thermal sensor

People detection system

The people detection system is in charge of monitoring if there are people attending the kitchen appliances while cooking is taking place.

The person detection system is made up of a Kria KV260 board and a USB camera. Installation and get started tutorial: Kria KV260 - 01- Starting with the Smart Camera Firmware

The people detection system uses the Vitis AI Library. The Vitis AI Library is a set of high-level libraries and APIs built for efficient AI inference with Deep-Learning Processor Unit (DPU). It is built based on the Vitis AI Runtime with unified APIs and provides easy-to-use interfaces for the AI model deployment on Xilinx platforms.

The Kria KV260 board is connected by Ethernet to a WIFI router.

The camera is a Logitech C170 USB webcam connected through USB.

Wireless thermal sensor alarm

The thermal sensor is composed of an ESP32 MCU and an IR Array MLX90640.

The ESP32 MCU is in charge of polling both the thermal sensor and the person detector. It is also in charge of launching the alarms in case of detecting a lack of attention to the kitchen appliances while the cooking is unattended. It also provides support for Wi-Fi connection to the sensor.

The communication between the wireless thermal sensor and the person detection sensor is carried out by means of a client-server communication. The people sensor acts as a server and the thermal sensor communicates with it via TCP/IP communication using simple response messaging.

Development setup

In addition to the starter kit you will need:

• Barrel Jack Power supply (12V, 3A)
• MicroSD card [16GB UHS-1]
• Micro-USB to USB-A cable
• USB Camera
• Ethernet cable
• HDMI cable to connect to a monitor

The Kria KV260 Vision AI Starter Kit is connected to a USB camera, a USB keyboard, a USB mouse and also connects to the ESP8266 MCU board via USB. During the development phase, the HDMI port has been connected with an hdmi to usb video capture adapter with my desktop computerfor video recording.

In the image you can see the system configuration during development.

Kria KV260 Hardware Setup

I have posted another blog posting my experiences setting up the system.

Kria KV260 - 01- Starting with the Smart Camera Firmware

The USB camera is a Logitech-Webcam-C170: Technical-Specifications

IPC- TCP server and People Detection process.

The TCP server and the Human Detector process work independently and communicate through a shared file.

The Human Detector process is the producer and the TCP Server the consumer.

...........writes  +-----------+  reads...........
H Detector-------->| data.dat  |<-------TCP Server
                   +-----------+

TCP Server at Kria KV260

The tcp server provides the date and time of the last time a person was detected.

The server responds via TCP on port 8000

When a client connects, it provides a string with the date and time in YYYY-MM-DD hh:mm:ss format.

TCP server response frame

2022-03-20 14:28:10,60.000000,2022-03-20 14:29:10

• Last time people detected: 2022-03-20 14:28:10
• Time since last detection: 60.0
• Actual Time: 2022-03-20 14:29:10

Setting up KV260 Firmware

You have to install the Vitis AI Runtime on your own image. The version of the board image should be 2021.2 or above.

Step 1: Installing a Board Image

A system images for the KV260 starter kit can be downloaded from here

Step 2: Flash OS Image to the SD card

You can use etcher (https://etcher.io/)

Install Vitis AI Library User Guide

Vitis AI Library User Guide (UG1354)

Clone repository

git clone --recurse-submodules https://github.com/javagoza/KriaKitchenLifeguard kriaklifeguard

Build

cd kriaklifeguard/
bash -x build.sh

Execute

To start both the person detection system and the tcp server with the following command

./sh klifeguard.sh

This will run both processes

#!/bin/bash
./klifeguard densebox_640_360 0 -t 8 &
./tcpserver &

X11 forwarding

From your Kria KV260 you can enable X11 forwarding with the following command (suppose in this example that the host machine IP address is 192.168.2.48) when logging in to the board using an SSH terminal because the video examples require a Linux windows system to work properly.

export DISPLAY=192.168.2.48:0.0

We are using Xming X Server from a Windows 10 system

Face Detection

The Kitchen ML Lifeguard uses the Face Detection library. Face Detection uses the DenseBox neural network to detect human faces. The input is a picture with the faces you want to detect and the output is a vector containing the information of each detection box.

vitis::ai::FaceDetect Base class for detecting the position of faces in the input image (cv::Mat). Input is an image (cv::Mat).

Output is a vector of position and score for faces in the input image.

Sample code:

auto image = cv::imread("sample_facedetect.jpg");
auto network = vitis::ai::FaceDetect::create(
               "densebox_640_360",
               true);
auto result = network->run(image);
for (const auto &r : result.rects) {
   auto score = r.score;
   auto x = r.x * image.cols;
   auto y = r.y * image.rows;
   auto width = r.width * image.cols;
   auto height = r.height * image.rows;
}

Checking the TCP server

You can use telnet to check the TCP Server

When a client connects, it provides a string with the date and time in YYYY-MM-DD hh:mm:ss format.

Kitchen Thermal Sensor

MAX TEMP 28.0
connecting to 192.168.2.95:8000
receiving from remote server
2022-03-20 14:28:10,60.000000,2022-03-20 14:29:10
closing connection
wait 5 sec...

Monitoring Web server

From a web browser we can connect to the web service that gives us the latest status detected by the system.

It gives us information about:

• the current time in the system,
• the last time a person was detected,
• the seconds that have passed since the last detection of a person
• and the current temperature that the system is detecting in the kitchen appliance that is monitoring.

Example: a person has been detected 2 seconds ago but the control temperature has not been reached. We are safe, there is no alarm.

Example: a person has been detected 2 seconds ago but the control temperature has not been reached. We are safe, there is no alarm.Example: a person has been detected 233 seconds ago and the control temperature of 70 C has been reached.

The buzzer is beeping as an alarm signal.

The Kitchen Lifeguard in action

• The USB camera is positioned looking at the possible people who are watching the cooking.
• The thermal sensor points towards the kitchen appliance to be monitored.

Project Files

You can access all the files needed to replicate the project in the following github repository:

https://github.com/javagoza/KriaKitchenLifeguard/

ESP32KlifeguardThermalSensor contains the Arduino sketch for the ESP32 board and the MLX90640 IR sensor array.

KriaKitchenLifeguard contains all the files needed to build the solution on the KV260 Vision AI Starter Kit.

Conclusion

The KV260 Vision AI Starter Kit is an excellent platform for prototyping accelerated machine learning, computer vision, and signal processing algorithms. Without the need for deep knowledge of hardware and thanks to The Vitis™ AI development environment, I have created a pure software solution in a few hours of development. Vitis AI libraries allow you to use the full potential of AI acceleration in Xilinx FPGA.