TouchFree v2: Contactless Temperature and Mask Checkup

Countries around the Globe are Reopening, living with the Novel Coronavirus is becoming the new way of life. But to Stop the Spread of the Virus we need to separate people having the Coronavirus from the Rest.

According to the CDC, fever is the leading symptom of the Coronavirus with up to 83% of Symptomatic Patients showing some signs of fever. Many Countries are making Temperature Checkups and Masks mandatory for Schools, Colleges, Offices, and other Workplaces.

Currently, Temperature Checkups are done manually using Contactless Thermometer. Manual Checkups can be Inefficient, Impractical (in places with a large footfall), and Risky.

To solve these problems, I have designed a Kiosk that automates the process of Temperature Checkup by using Facial Landmarking & Contactless IR Temperature Sensor and Mask Detection using Deep Learning Neural Network for less than $100.

The use of this Kiosk is not limited to Schools, Colleges, Offices, other Workplaces but can also be used at High-Risk Areas like Hospitals. This Device can also be used at Train Stations, Bus Stops, Airports, etc.

My approach for this project was to build a Streamlined Setup process such that anybody without any Prior Experience of Computer Vision or Deep Learning can use this. This is a fully functioning and ready to use Project. I have made this Project highly customizable by adding code files for each and every stand-alone part and the full version. Thus, you may use any of the parts of the project individually.

Thanks to Amazon AWS Credit, I was able to train my Deep Learning Model for Mask Detection with Good Compute Power using AWS EC2 Instance.

What's new in v2?

• All-new Web-based Dashboard Design compatible with almost all browsers
• Self Alignment Feature
• Reduced Cost for less than $100 ($150 with display)
• BalenaFin Support for large scale Deployments, use of Cellular Services to name a few.
• PC Support (Mac/Linux/Windows)
• Remote Access

Check Out v1 here- https://www.hackster.io/sakshambhutani2001/touchfree-automated-temperature-checkup-and-mask-detection-2cc337#things

Explanation

Firstly, the Tensorflow based Deep Learning Neural Network tries to detect if the person is wearing a Mask or not. The System has been made Robust by training it with many different examples to prevent False Positives.

Once, the System has detected the Mask it performs Facial Landmarking. The System is using DLIB Module for Facial Landmarking to find the best Spot on the Forehead of the person to take Temperature from.

Then by using PID Control System with Servo Motors, the system tries to align the Selected Spot on the Forehead with the Sensor OR if in self align mode the User can align himself based on the visual clues. Once aligned the system takes Temperature Reading using Contactless IR Temperature Sensor.

If the temperature is within normal Human Body Temperature range it allows the Person to Proceed and sends an email to the Admin with a Picture and other Details like Body Temperature, etc.

Software Setup

• PC installation is just a few lines follow guide on Github https://github.com/saksham2001/TouchFree-v2/blob/master/README.md#setup-for-pc-windowsmaclinux
• For Raspberry Pi just skip BalenaFin Setup, everything else remains the same.
• For BalenaFin skip Raspberry Pi Setup, everything else remains the same.

BalenaFin Setup

1. Download Balena Etcher for your Operating System from the Balena Etchers Download Page.

2. Download the Raspbian Image for Balena here.

3. Attach the SD card to your computer, Select the Latest Version of Raspberry Pi OS and click Flash to flash the OS to your Card.

4. Attach the SD card to the Raspberry Pi, connect it to a monitor, and Power it. Wait for a few minutes until the Raspberry Pi Boots up.

5. Connect your WiFi with your local WiFi Router by clicking the Wifi Icon Upper Right Corner of the Screen and entering your

Raspberry Pi Setup

1. Download Balena Etcher for your Operating System from the Balena Etchers Download Page.

2. Attach the SD card to your computer, Select the Latest Version of Raspberry Pi OS and click Flash to flash the OS to your Card.

3. Attach the SD card to the Raspberry Pi, connect it to a monitor, and Power it. Wait for a few minutes until the Raspberry Pi Boots up.

4. Connect your WiFi with your local WiFi Router by clicking the Wifi Icon Upper Right Corner of the Screen and entering your

Raspberry Pi Touch Screen Setup

Note- I have used a generic 3.5 inch Touch Screen. You can use any but the Official Raspberry Pi Touch Screen is Highly Recommended. If you choose to use a 3.5 inch Touch Screen you will have to solder a few Pins.

Follow this to Setup your official Raspberry Pi Touch Screen.

Follow this to Setup Generic 3.5 inch Touch Screen.

Raspberry Pi Camera Setup

Follow this to Setup Official Raspberry Pi Camera v1/ v2.

Enter the following commands in the Terminal one by one to install various Packages and Libraries.

Getting Pi Ready for Installations

Updating the Raspberry Pi OS

$ sudo apt-get update
$ sudo apt-get upgrade

Updating pip3

$ pip3 install --upgrade pip3

Expanding Swap Size and Changing Memory

Note- These need to be changed back after all the installations are Completed.

$ sudo nano /etc/dphys-swapfile

A new file will open. Navigate to CONF_SWAPSIZE=100 and change it to CONF_SWAPSIZE=1024. Then click Ctr + O, then Enter and then Ctr + X to save the changes.

Then run the following commands to get changes in effect.

$ sudo /etc/init.d/dphys-swapfile stop
$ sudo /etc/init.d/dphys-swapfile start

Then,

$ sudo raspi-config

and Navigate to Advanced Options => Memory Split and change from 128 => 16.

Then Reboot the Raspberry Pi.

Installing Requirements

$ sudo pip3 install numpy
$ sudo pip3 install scipy
$ sudo pip3 install scikit-image
$ sudo pip3 install RPi.GPIO
$ sudo pip3 install smbus2
$ sudo pip3 install PyMLX90614

Installing OpenCV

Installing Important Dependencies for OpenCV

Instaling OpenCV

$ sudo apt-get install python3-opencv

Installing TensorFlow

Note- The Model was trained on Tensorflow-2.2, if you are using any other versions of Tensorflow it may cause some problems.

$ sudo apt-get install -y libhdf5-dev libc-ares-dev libeigen3-dev gcc gfortran python-dev libgfortran5 \ libatlas3-base libatlas-base-dev libopenblas-dev libopenblas-base libblas-dev \ liblapack-dev cython libatlas-base-dev openmpi-bin libopenmpi-dev python3-dev
$ sudo pip3 install keras_applications==1.0.8 --no-deps
$ sudo pip3 install keras_preprocessing==1.1.0 --no-deps
$ sudo pip3 install h5py==2.9.0
$ sudo pip3 install pybind11
$ pip3 install -U --user six wheel mock
$ wget "https://raw.githubusercontent.com/PINTO0309/Tensorflow-bin/master/tensorflow-2.2.0-cp37-cp37m-linux_armv7l_download.sh"
$ chmod u+x tensorflow-2.2.0-cp37-cp37m-linux_armv7l_download.sh
$ ./tensorflow-2.2.0-cp37-cp37m-linux_armv7l_download.sh
$ sudo pip3 uninstall tensorflow
$ sudo -H pip3 install tensorflow-2.2.0-cp37-cp37m-linux_armv7l.whl

Check the Installations

$ python3
>>> import tensorflow
>>> tensorflow.__version__
2.2.0
>>> exit()

If you get no error in this step, it means you have successfully installed Tensorflow.

Installing DLIB

$ pip3 install dlib

Check the Installations

$ python3
>>> import dlib
>>> exit()

If you get no error in this step, it means you have successfully installed DLIB.

Changing Swap Size and Changing Memory back

$ sudo nano /etc/dphys-swapfile

A new file will open. Navigate to CONF_SWAPSIZE=1024 and change it to CONF_SWAPSIZE=100. Then click Ctr + O, then Enter and then Ctr + X to save the changes.

Then run the following commands to get changes in effect.

$ sudo /etc/init.d/dphys-swapfile stop
$ sudo /etc/init.d/dphys-swapfile start

Then,

$ sudo raspi-config

and Navigate to Advanced Options => Memory Split and change from 16 => 128.

Then Reboot the Raspberry Pi.

Download the Code

$ mkdir covid
$ cd covid
$ git clone https://github.com/saksham2001/TouchFree-v2

Environment Variables Setup

$ sudo nano /etc/profile

This will open a file, at the end of the file add the following lines if you want to get email alerts

export SEND_EMAIL="TRUE"
export EMAIL_ID="your-email-id@something.com"

Use your Email ID instead of "your-email-id@something.com"

export EMAIL_PWD="your-email-password"

Use your Email Password instead of "your-email-password"

export COUNTER_ID=0

Email Setup

To use the email Alert Service you will need a Gmail Account and you will need to Allow Less Secure Apps. Or, if you use 2 Factor Authentication then you might need to create an App Password.

Hardware Setup

Attach the Servos X and the Sensor Y as Given in the Circuit Diagram. All the Connections can be made as per the diagram. The Camera has to be connected to the Camera Connector and Raspberry Pi Display can be connected to the Display Connector. If you are using a Generic 3.5 inch Touch Screen, you can connect the screen to the required GPIOs. The GPIOs connecting to Servos will still be free, but you will need to solder to get 5v and SDA, SCL pins for the Sensor.

Note- Servo X is the Servo which moves Right-Left. Servo Y is the Servo which moves Up-Down.

Sensor Setup

After connecting the Sensor, run the following command

$ sudo su -c 'echo "Y" > /sys/module/i2c_bcm2708/parameters/combined'
$ sudo i2cdetect -y 1
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:          -- -- -- -- -- -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
50: -- -- -- -- -- -- -- -- -- -- 5a -- -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
70: -- -- -- -- -- -- -- --

Copy the port name (5a in this case)

$ cd ~/covid
$ nano app.py

Then change 5A in the given line

sensor = MLX90614(bus, address=0x5A)

Testing

$ cd ~
$ cd covid

Running the Complete Application with PID Alignment

$ python3 pi_app.py

Running the Complete Application with self Alignment

$ python3 pi_self_app.py

Remote Access (Optional)

$ sudo apt update 
$ sudo apt install snapd
$ sudo reboot

After Rebooting

sudo snap install ngrok

To Start Remote Access

ngrok HTTP 8050

Now a website address will be given on which you can view the dashboard after starting your python code as given earlier.

Enclosure and Camera-Sensor Enclosure

You can 3D Print the Enclosure using the files given below or else you can make Your own Makeshift Enclosure using Cardboard, Wood, etc.

I have 3D Printed this enclosure made by Stephen Wilson for my 3.5 inch Touch Screen Display.

If you are using the Official Raspberry Pi Display try finding Enclosures on Thingiverse. I liked this one a lot but have not tested it out.

For the Camera-Sensor Enclosure, I have custom made them so that the Camera and Sensor can align. If you want to change the enclosure, you might need to change the alignment point in the Code.

Note- These Cases are for Raspberry Pi 2/3 and will not work with Pi 4. For Pi 4 you will need to find a different Case. The files attached below for the Camera-Sensor Enclosure do not have any screw holes for the sensor (As the sensor comes in Many Shapes and Sizes). If you want to have Screw Holes feel free to edit these files.

Debugging

What should I do If the Mask Detection is not Working properly?

The Mask Detection Model has been trained with Synthetic Data in a very Limited Lighting Conditions and with a limited Face Mask Colours. If the model is not working well with your conditions you might need to retrain the model according to the instructions given below.

Note- This might be tricky if you get stuck at any step feel free to contact me!

1. Take Some Pictures on Your Raspberry Pi with mask and without a mask using the command-

$ raspistill -o image-name.jpg

Change the image-name every time you click an image.

2. Go to the Google Collab Notebook here and Duplicate it. Change the Runtime Type in Runtime => Change Runtime Type to GPU.

3. Then in your notebook Run the first 4 Cells. This will create two folders in files with mask and without mask. Add the images taken by you to these folders and run all other Cells.

4. Then a new model will be created as mask_detector.h5. Download this model and replace the old model in covid => models on your Raspberry Pi with it.

What should I do if the Temperature Checkup is not Working?

The Temperature Checkup requires proper lighting. Make sure you have proper lighting and that your Servos are working well.

Scope for Further Improvements

Currently, I am working on a BalenaFin specific version of this project using BalenaOS which allows for easy large scale deployments for businesses of various scales. Unfortunately, I did not get enough time to complete it before the deadline but I will soon publish it. In that version, I am using 2 docker containers over BalenaOS one with Balena Dash for Web-based Graphical User Interface and the other one for all the computation.