Thermal Face Detector

Introduction

In this project, I build a device which can be used to detect face from thermal camera readings and can accurately measure person temperature by extrapolating the facial skin temperature. If high temperature is detected, the person can use their smartphone to scan QR code (displayed at the screen) for further information. Also, a local helpline phone number is displayed where the person can contact for enquiry.

Main features

• Contactless
• Easy to use, reproduce and deploy at mass scale
• Low cost
• Open source
• User privacy
• Developing countries friendly
• Accuracy
• Speed

Device Selection

The balenaFin v1.0 kit is used with Raspberry PI Compute Module 3+.

The Gameduino 3 shield is used as a display. For the thermal camera Pimoroni MLX90640 sensor is used which has 32x24 pixels resolution.

The connection diagram is available at the Schematics Section.

Device Setup

To setup an application on the balenaFin using container we need to register and login into balenaCloud. After login we can create an application by clicking on the Create Application button which displays a form as below. We should choose Balena Fin (CM3) as Default device type and Starter as the Application type.

After clicking on the created application link and then clicking on the Add Device button we can fill in the Device information. We should choose Development as edition and Wifi+ Ethernet connection with ssid and password for home/work router.

We can now download the balenaOS by clicking the button at the bottom.

To flash the balenaOS to the balenaFin we can use balenaEcther which can be downloaded from here https://www.balena.io/etcher. We should connect a computer using a usb cable to balenaFIn and connect the power adapter.

After flashing is completed we should remove the usb cable and disconnect/reconnect the power adapter. After few minutes we should be able see the device in the balenaCloud dashboard.

The MLX90640 sensor uses I2C for getting thermal readings. To read the data fast we should update Define DT parameters as "i2c_arm=on", "spi=on", "audio=on", "i2c1_baudrate=1000000" in the Device Configuration.

Deployment

We can now deploy the application container to the device. We will enable local mode as shown below so that we can use the balena CLI which is a command line tool to interact with the balenaCloud to do rapid development and deployment. The balena CLI installation instructions can be found here https://github.com/balena-io/balena-cli/blob/master/INSTALL.md.

Below is the commands for the deployment.

$ balena login
$ git clone https://github.com/metanav/thermal_face_detection
$ cd thermal_face_detection
$ sudo balena scan
Reporting scan results
-
host: df0925a.local

$ balena push df0925a.local

All the prerequisites and dependencies are mentioned in the Dockerfile which can be found at the root directory of the repository cloned above. Everything is automatically installed and deployed to the target device.

Model selection for face detection

The deep learning models available in the public domain are mostly used for the images taken by photographic cameras. Usually they require high end computer with graphic card or neural network accelerator to perform at high frame rates. For this project I wanted to keep the cost low but still develop a device which can be used accurately at considerable speed. For this very purpose BlazeFace model is used which shows considerable accuracy in detecting faces in the low resolution thermal camera images. The model is used with the default trained weights without any transfer learning on the thermal images.

Application development

The main application has two parts. The first part is written in Python 3 which is mostly about getting data from the thermal camera and do the face detection and temperature measurement. The 2nd part is written in C which is mostly handling the Gameduino 3 TFT display for displaying thermal image and information. The code is available at the Github repository mentioned in the code section.

Inferencing on the device

The BlazeFace TensorFlow Lite model was downloaded from https://github.com/google/mediapipe. The TensorFlow Lite Runtime Python API is used to run inference on the device.

Demo

Face detection (normal temperature)

Face detection with face mask (normal temperature)

Face detection (high temperature)

The use cases for common benefit

It is an easy to use low-powered device which can be used safely since all operations are contactless and offline. This device does not use a photographic camera to detect the face instead it uses a thermal sensor to generate a thermal image and detect the face using a ready-to-use locally deployed deep learning model. There is no privacy threats using this device since it does not capture any real image of the user and does not send out any data to the cloud or any other location. The internet connection is only used for deployment and updates. This device can be installed in shopping malls, schools, and offices. Also, in the developing countries this device can be used in the less privileged areas where there is scarcity of electricity and internet. In those areas it can be operated using batteries and deployment can be done before installation.

Scope for improvement

The accuracy of the face detection can be improved further by transfer learning with thermal camera images. Currently the device only displays information in English language, it can be in multilingual with regional languages support. The helpline phone number and QR Code URL is hardcoded for my region but it can be based on the geographic location of the device and can be set using the Device environment variables in the balenaCloud.

All the instructions can be found at the Github repository mentioned in the code section. I would like to thank Hackster.io and all sponsors for this contest and I am thankful to Balena for providing the balenaFin development kit.