Forehead Facial Recognition System

Funding

Funding for this project was provided by the Cal Poly IEEE Student Branch.

System Overview

For this project, we chose to use a Raspberry Pi 4 Model B with 8 GB of RAM because image processing and temperature gathering can be somewhat computationally resource-intensive operations.

The camera that we used was the Raspberry Pi Camera V2, a natural choice given that we are also using the Raspberry Pi since it uses the CSI connector on the Pi. This allows faster communication speeds because the USB protocol overhead is eliminated.

The MLX90640 thermal camera was chosen due to its low cost (slightly inflated right now due to COVID-19), and accuracy of 1°C. We were initially looking at purchasing the PureThermal Mini Pro JST-SR (with FLIR Lepton 3.5) (https://www.digikey.com/en/products/detail/sparkfun-electronics/DEV-17544/13561756) since the FLIR Lepton 3.5 features an impressive 160x120 resolution (versus the MLX90640 that has a 32x24 resolution). We chose not to use the FLIR module due to the high cost ($375.00 as of 3/11/2021) and its accuracy of +/- 5°C. Additionally, the MLX90640 is simple to use with its I2C protocol.

The flowchart above outlines at a high level what the software is doing. The first step in the process is reading the camera stream waiting for a face to detect. After a face is detected, the forehead is identified and the face is classified. The forehead location is then passed to the thermal camera, which reports the temperature at the specified location.

After all the requisite information is gathered, the data is stored in a MariaDB database for easy retrieval. The information can then be shown on the embedded web display to the user.

Setting up the Pi

This guide assumes you are running Raspberry Pi OS V10 "Buster".

Required Software

• Docker: Docker is used to host the production server as well as the MariaDB database. Docker was chosen as it provides a virtualized, isolated platform for hosting individual services, or "containers". The allows the system to have greater modularity and scalability
• Python: The webserver and associated image processing scripts are all built into Python.

Setting up Docker

Start by updating and upgrading software on the Pi:

sudo apt update -y && sudo apt upgrade

Next, download the docker setup script:

curl -fsSL https://get.docker.com -o docker-setup.sh

Run the setup script:

sudo docker-setup.sh

At this point, you can delete docker-setup.sh if you wish.

Next, add the current user to the docker group:

sudo usermod -aG docker $USER

For the command to take effect, you must logout and log back in. This concludes the docker setup. To effectively use the provided repository, you will also need to install docker-compose.

sudo apt install -y libffi-dev libssl-dev python3 python3-pip
sudo pip3 -v install docker-compose

The following commands assume that you have already download the included repository and are in the root directory of the downloaded repository.

docker-compose allows docker to be configured and run with simple configuration scripts. For instance, the docker-compose.yml file provided in the repository is as follows:

services:
  frfts:
    container_name: frfts
    build:
      context: $DOCKERDIR/setup/docker/
      dockerfile: Dockerfile
      args:
        PUID: 500
        PGID: 500
        I2C_GID: 44
      env_file:
        - $DOCKERDIR/setup/docker/app/.env
      devices:
        - "/dev/i2c-1:/dev/i2c-1"
      ports:
        - "80:80"
      restart: unless-stopped
      depends_on:
        - mariadb
       networks:
         - frfts_net

  # MariaDB - MySQL Database
  mariadb:
    hostname: mariadb
    container_name: mariadb
    image: linuxserver/mariadb:latest
    restart: always
    networks:
      - frfts_net
    security_opt:
      - no-new-privileges:true
    ports:
      - "3306:3306"
    volumes:
      - $DOCKERDIR/database:/config
    environment:
      - PUID=$PUID
      - PGID=$PGID
      - MYSQL_ROOT_PASSWORD=$MYSQL_ROOT_PASSWORD
      - MYSQL_DATABASE=$MYSQL_DATABASE
      - MYSQL_USER=$MYSQL_USER
      - MYSQL_PASSWORD=$MYSQL_PASSWORD

To use the provided docker-compose.yml file, you need to add the specified variables to a .env file:

PYTHONPATH=${PYTHONPATH}:${PWD}

PUID=1001
PGID=984
MYSQL_DATABASE=
MYSQL_USER=
MYSQL_ROOT_PASSWORD=
MYSQL_PASSWORD=


TZ=""
USERDIR=""
DOCKERDIR=""

Fill in the variables to your preference, and save it in the same directory the docker-compose.yml file is in as .env. You will also need to provide another .env file in setup/docker/app/.env. This one should look like the following:

SQL_USER=
SQL_PASSWORD=
SQL_DATABASE=
SQL_HOST=

To build the included Dockerfile, you need to provide a .env file and some python whl files. The .env file provides variables to access the MariaDB database. The python whl files allow for quick installation of some packages that otherwise must be built from source. Please note, these wheel files are built for armv7l and will require the dependencies as outlined in the Dockerfile. These wheel files are provided in this writeup, and you can download them with the following command:

curl https://hacksterio.s3.amazonaws.com/uploads/attachments/1273036/wheels.zip --output setup/docker/wheels.zip
unzip setup/docker/wheels.zip
rm setup/docker/wheels.zip

To run the docker-compose.yml file, all you need to type is docker-compose up -d. Before you can run the docker-compose.yml file, you need to set up the docker network:

docker network create --gateway 192.168.15.1 --subnet 192.168.15.0/24 frfts_net

The provided Dockerfile runs the flask app in NGINX using uWSGI. The web application is exposed on port 80. You can port-forward this and expose it to the internet, but please be advised of the security implications of such a setup. In the setup that our team used, the webserver running on this Pi was forwarded to a global traefik reverse proxy system that also provides an SSL termination and protection using Keycloak so only authenticated users could access this (relatively unsecured) website. However, the configuration of such a system is outside the scope of this project.

Face Detection

This section and the next (Face Recogition) dive into the algorithms used for the project. Much of these sections is just for explanation and testing purposes and will differ from what is actually in the code in the GitHub. Their purpose is to give an overview of the crux of the algorithms at hand.

Installing the Modules (Optional)

In order to spin up a facial recognition system for this project, you must first interface with the Pi Camera module using Python. To do so, install the picamera[array] module. It's important to install the picamera[array] module rather than just picamera because it allows you to obtain an image as a NumPy array, a data type friendly to OpenCV applications.

sudo pip3 install picamera[array]

Speaking of NumPy, you will need that as well.

sudo pip3 install numpy

And finally, OpenCV. This part is a bit more involved because you have to build it from source rather than just a pip3 install. Note that this will take a while to install.

wget -O opencv.zip https://github.com/opencv/opencv/archive/3.4.3.zip
...
unzip opencv.zip                            
mv opencv-3.4.3 opencv                            
cd opencv                            
mkdir build                            
cd build
...
cmake -DCMAKE_BUILD_TYPE=RELEASE \                         
-DCMAKE_INSTALL_PREFIX=/usr/local \                            
-DINSTALL_PYTHON_EXAMPLES=ON \                            
-DINSTALL_C_EXAMPLES=OFF \                            
-DPYTHON_EXECUTABLE=/usr/bin/python3 \                            
-DBUILD_EXAMPLES=ON ..
...
make -j2
...
sudo make install                            
sudo ldconfig

This project processes images as arrays of RGB and grayscale pixel values. These are arrays are not Python lists, however. They are NumPy arrays and Mats (short for Matrices, a data type from OpenCV), which are the required data types for the facial detection and recognition algorithms.

Please note that the above section is optional and is solely for explanation's sake. All of the required modules are installed in the docker container and will be there when you clone the repo.

Interfacing with the Camera

from picamera.array import PiRGBArray
from picamera import PiCamera
import numpy as np

camera = PiCamera()
rawcapture = PiRGBArray(camera)                                             
camera.capture(rawcapture, format='bgr')
frame_array = rawcapture.array()

The code above initializes the camera as a PiCamera object, captures a frame in a BGR format, then converts the frame into a NumPy array called frame_array.

Haar Cascade

The method for face detection to use here is the Haar Face Cascade, built into OpenCV. To clarify, this is for detecting a face within a frame, not for actually classifying a face and assigning/matching it to a person. That comes later.

The Haar Cascade works by convolving the pixels of an image with Haar figures, seen in the image below. This is a way of detecting certain edges within an image, and, if the right combination of edges is found, it detects a face.

import time
import cv2

haar_face_cascade = cv2.CascadeClassifier('/usr/local/share/OpenCV/haarcascades\haarcascade_frontalface_default.xml')
gray = cv2.cvtColor(frame_array,cv2.BGR2GRAY)
faces = haar_face_cascade.detectMultiScale(gray, 1.1, 3)

(x,y,w,h) = faces[0]
cv2.rectangle(frame_array, (x,y), (x+w,y+h), (0,0,255), 3)

t = time.ctime(time.time())
cv2.putText(frame_array, t, (10,50), cv2.FONT_HERSHEY_SIMPLEX, 1, (0,0,255), 3)

First, load the cascade. The path to the .xml is '/usr/local/share/OpenCV/haarcascades\haarcascade_frontalface_default.xml' in this instance, but your mileage may vary, so make sure to double-check that you have the correct path. After cloning the repository, the path '/app/haarcascade_frontalface_default.xml' should work.

The Haar Cascade works best with a grayscale image, hence the conversion.

Finally, the function detectMultiScale function is the meat of the operation. In this case, it accepts a frame, a scale factor, and a parameter called MinNeighbors. The scale factor reduces the image size in order to be able to detect large faces (faces close to the camera), and MinNeighbors specifies how stringent the algorithm should be with its detection. Decreasing MinNeighbors means it's easier to detect a face, but there's a higher chance of false positives. Increasing it means fewer false positives, but comes at the expense of it being more difficult to detect a face. The scale factor and MinNeighbors were chosen here to be 1.1 and 3, respectively, by trial and error, but they can be tweaked to fit a specific implementation.

The output of the Haar Cascade is a list called faces. This list contains the upper left x coordinate, the upper left y coordinate, the width, and the height of a rectangle encapsulating all of the detected faces in the frame. Since there's only supposed to be one face in the frame, grab the 0th index and store it as coordinates. Then use cv2.rectangle to draw the face box.

The time of capture is printed in the top left corner of the frame using the time module and cv2's putText function.

Obtaining the Forehead Coordinates

To pinpoint the temperature measurement to a forehead, you must first find the coordinates of the forehead. Using the coordinates spit out of the Haar Cascade, obtain an estimate for the upper left (x,y) and lower right (x,y) of the forehead. Then you can draw a rectangle around it using cv2.rectangle.

start_x = x + w//4
start_y = y + h//8
end_x = x + 3*w//4
end_y = y + h//4

cv2.rectangle(frame_array, (start_x,start_y), (end_x,end_y), (0,0,255), 2)

At this point, the image is ready to be shown using the following code. The waitKey function displays the image for 10 ms or until a key is pressed, whichever comes first. To display the image indefinitely until a key is pressed, change the parameter from 10 to 0.

cv2.imshow("Image", frame_array)                                                  
cv2.waitKey(10)

Face Recognition

Caffe Deep Neural Network

If the Haar Cascade detects a face, the system then moves to classify it. The way to classify a face, like many biometrics, is with a print (here: faceprint). In this project, a faceprint is an array of length 1000 that is unique to a face.

The method for obtaining a faceprint is by using a deep neural network. A DNN can be trained to classify unique objects of a certain type within an image. To specify what type of object (such as a car, dog, person, etc.), the DNN must be trained; in this case, the DNN came pre-trained to classify a face.

The network was trained with a method called Triplet Loss, which essentially compares three faces at a time to give a basis for what matches and what does not.

There are different types of DNNs out there, but a good one to use here is the Caffe framework. It's developed by Berkeley Vision and is suited for edge/mobile applications. An example of another framework is TensorFlow, which is more suited for server applications.

To specify the DNN, OpenCV needs a .prototxt file and a .caffemodel file (you can download from these links, or get them from the GitHub). The prototxt file describes the network's architecture and is sourced from Berkeley Vision while the caffemodel file is the actual pre-trained network and comes from a study out of Cornell.

face_net = cv2.dnn.readNetFromCaffe('bvlc_googlenet.prototxt','bvlc_googlenet.caffemodel')                                                  
face_crop = frame_array[y:y+h, x:x+w, :]                                      
face_blob = cv2.dnn.blobFromImage(face_crop, 1, (224,224))                        
face_net.setInput(face_blob)                                                      
face_print = face_net.forward()

The DNN requires something called a blob for its input, which is essentially a wrapper for the face data in the required format. The blob is then sent through the net in the forward direction.

Matching Faces

Now that you have a faceprint, you need a way to compare it to other faceprints to find if it's a match. Rather than comparing the two arrays directly by subtraction, this project uses a "scoring" system. To get a score for how similar two faceprints are, you take the Euclidean Distance between them. This is done by first subtracting them, then taking the vector norm of the result using NumPy's linalg function. Note that the faceprint is an array within an array and looks like [[x0 x1 x2 ... x999]], so to access it use faceprint[0].

if np.linalg(face_print1[0] - face_print2[0]) < 0.2:
    print("it's a match")

If the Euclidean Distance is below a threshold, the faceprints match. The threshold here is 0.2, but that can be adjusted based on trial and error.

Temperature Measurement and Logging

Initiating and Setting up the Thermal Camera

To help with starting up the thermal camera, the following makersportal.com tutorial will be helpful. In particular, this project uses the "Real-Time Interpolation of MLX90640" section.

https://makersportal.com/blog/2020/6/8/high-resolution-thermal-camera-with-raspberry-pi-and-mlx90640

The thermal image displaying portion of the given code is omitted from our project as that is not useful to the user, but the accessing of data portion is used to get the temperature. However, image displaying is used in the calibration of the two cameras.

To get a rough calibration of the cameras, simultaneously run and display the images of both. Try to point both cameras in the same direction. Now use a hand or other warm object (something with a small, well-defined heating element would be best like a lighter) and wave it around within the frame of the raspberry pi camera. The frame of the raspberry pi camera is shown to be smaller than that of the thermal camera, so the object will exit the edge of the raspberry pi camera before it does so with the thermal camera. Move the object to all four edges of the raspberry pi camera and note on the graph of the thermal camera where the heat spot shows up on the thermal camera. These four points will be considered the "cutoff" points and will be used in the initialization of the thermal camera in the python script.

Other methods of calibration could be to move the thermal camera further back than the raspberry pi camera and then use the same lighter method to determine when the edges line up. If the two cameras line up to your satisfaction, you can set all of the cutoff points to 0. Please note that if the cameras are not lined up well enough, that the temperature of the forehead could be off.

The __init__ method within the thermal_camera class in the file match_temp.py uses the cutoff points to line up the cameras and make sure that the coordinates of the forehead given by the raspberry pi camera match the same location on the thermal camera. Below is an illustration of how this calibration works. The actual code can be found at the GitHub link.

Getting the Forehead Temperature

Once the face has been detected, and the box around the forehead identified, the temperature of the forehead can be determined. First, using the calibration, the coordinates from the forehead need to be converted into the coordinates for the thermal camera:

self.start_x = int(round(person.forehead[0]/self.scaled_width)) + self.left_cutoff
self.start_y = int(round(person.forehead[1]/self.scaled_height)) + self.top_cutoff
self.end_x = int(round(person.forehead[2]/self.scaled_width)) + self.left_cutoff
self.end_y = int(round(person.forehead[3]/self.scaled_height)) + self.top_cutoff

Now that we have the thermal camera coordinates, we need to grab the pixels from the camera, put them into an array, and then average the values in that array:

forehead_array = np.zeros([(self.end_y-self.start_y), (self.end_x - self.start_x)])
forehead_array = data_array[self.start_y:self.end_y+1, self.start_x:self.end_x+1]
temp = np.mean(forehead_array)
print('Average forehead temp: {0:2.1f}C ({1:2.1f}F)'.\
format(temp, ((9.0/5.0)*temp+32.0)))
person.temperature = temp

And that is all there is to getting the temperature of the forehead!

Adding to the Database

This project uses the MySQL MariaDB database to upload and store the necessary information including the person's user id, the faceprint as discussed earlier, the picture of the face, and finally the temperature and the time it was taken.

First, a connection needs to be made with the database, and a cursor to navigate the database needs to be established:

self.connection = mariadb.connect(user=os.getenv("SQL_USER"),\
        password=os.getenv("SQL_PASSWORD"),\
        database=os.getenv("SQL_DATABASE"), host=os.getenv("SQL_HOST"))
self.cursor = self.connection.cursor()

Now the tables to store the information need to be set up. This project uses two different tables, one for the Users and one for the Temperatures. The users table will have only one entry for each person and a unique face_id. This table also stores the faceprint array and the image of the face both as Binary Large Objects (BLOBs for short, different from the blob for the input of the DNN).

The temperature table can have many entries for a single user. The user id in the temperature table matches the face_id from the users table to indicate that a specific time/temperature pair belong to that user.

The following code sets these two tables up, and uses a "FOREIGN KEY" to connect the face_id in the Users table to the userId in the Temperature table:

self.cursor.execute("CREATE TABLE IF NOT EXISTS Users( face_id INT UNSIGNED AUTO_INCREMENT PRIMARY KEY,"
"face_print MEDIUMBLOB, frame MEDIUMBLOB) ")
self.cursor.execute("CREATE TABLE IF NOT EXISTS Temps( temp_num INT UNSIGNED AUTO_INCREMENT PRIMARY KEY,"
"userId INT UNSIGNED , time DATETIME, temp FLOAT,"
"FOREIGN KEY (userId) REFERENCES Users (face_id) ON DELETE CASCADE )")

Now the data can be stored in the tables. However, first, we must check that the user that just logged a temperature is not already in the table. To do this, we search through the Users table and compare the face print given by the facial recognition to the faceprints stored in the table already. As the faceprints are stored as BLOBs, they need to be converted back to arrays so that the two can be compared.

self.cursor.execute(f"SELECT face_print FROM Users WHERE face_id = {x};")
fp_string = self.cursor.fetchone()
fp_array = np.frombuffer(fp_string[0], np.uint8)
fp = cv2.imdecode(fp_array, cv2.IMREAD_GRAYSCALE)

# Compare the two face_prints
norm = np.linalg.norm(person.face_print - fp)
if norm < self.norm: # Found a matching face print
    break

When the norm of the difference between the two faceprints falls below the threshold, a match to a user that is already in the database has been made. Now the temperature simply needs to be stored in the Temperatures table.

Q4 = "INSERT INTO Temps (userId, time, temp) VALUES (%s,%s,%s)"
val4 = (same_face_id, formatted_date, person.temperature)
self.cursor.execute(Q4,val4)

If a match is not made, a new user is created with their own face_id, their face print and frame is stored, and then the temperature is stored in the Temperature table:

# Convert the frame image to a string
img_str = cv2.imencode('.jpg', person.frame)[1].tostring()
fp_str = cv2.imencode('.jpg', person.face_print)[1].tostring()

# Insert the frame into the table
Q1 = '''INSERT INTO Users (face_print, frame) VALUES (%s, %s)'''
val1 = (fp_str, img_str,)
self.cursor.execute(Q1,val1)

Once the changes to the database are executed, don't forget to commit the changes:

self.connection.commit()

Frontend System

For the frontend system, Flask, which is a simple and easy-to-use web framework that models after the REST style of API, provides a simple way to interface the already-existing database with a user. The following packages are the ones we used to import Flask and all of the necessary functions:

from flask import Flask, flask, request, url_for, saend_from_directory, Response, render_template, g

To initialize the Flask application the initialization line

app= Flask(__name__)

is called. This lets the Python interpreter know that the directives that we provide by typing

@app.route("<website path>")

where the <website path> is the directory after the project URL/IP address in your browser. This tells the Flask app the function to run to generate the correct page from before. For each of the three pages that the application has, there is a respective app.route(). Within these functions, the method

render_template("<template name>",<template data,>)

was used to create the pure html out of hand written templates. This is helpful because it creates an easy way of modifying each one of the webpages without having to directly modify the HTML.

For easy data visualization, use the python library matplotlib. A simple import statement on that library is all that is necessary to start. Matplotlib is typically used to generate image files of data visualizations, like:

The problem with this is that there is not an easy way for the library to write to something outside of a file. To solve this, the Python library, io, is used to create an in-memory file using io.BytesIO(). The final function that returns the image as a png, but as an in-memory buffer:

def createHistogram(times,temps):
    #create an in-memory file
    buf = io.BytesIO()
    #start creating the chart itself
    plt.figure(figsize=[10,8])
    plt.title("user temperature")
    plt.xlabel('Date',fontsize=15)
    plt.ylabel('Temperature (C)',fontsize=15)
    plt.plot(times,temps)
    #save the created plot to the virtual file
    plt.savefig(buf,bbox_inches='tight',format='png')
    #seek back to the beginning of the file
    buf.seek(0)
    return buf

This method of using an in-memory file made it quite simple to directly inject the image into the HTML document generated by Flask, and allowed a lighter-weight filesystem with lower latency, as the plot images never needed to be stored in a file. To inject the images into the HTML, the image was encoded into a jpg, then was encoded in base64 and placed directly into the HTML.

To retrieve data from the database at and after login, the same method that was detailed above is used, with the mysql_connector_python library and a series of mysql commands. For the face identification login, the user's image is polled and then a faceprint is generated by the DNN and compared against every other faceprint in the database.

query = ("SELECT face_print,face_id from Users;") # get all of the users
face_id = -1
cursor.execute(query)
for (face_print_remote,face_id_remote) in cursor.fetchall():
    fp_array = np.frombuffer(face_print_remote,np.uint8)
    fp = cv2.imdecode(fp_array,cv2.IMREAD_GRAYSCALE)
    norm = np.linalg.norm(face_print-fp)
    #error+="norm= "+str(norm)
    if norm<NORM_THRESHOLD:
        face_id = face_id_remote
        break

With the correct faceprint, the functions to generate the visual graph and to pull the frame of the user on the pi from the database is called and the HTML is generated, and then sent as a response.

For the live webcam, the frame of the camera was continually updated within the /video_feed directory of the service, and that live image was then embedded within the home page.

Conclusion

Thanks for your interest, and we hope you try out/have fun with the project!