HOME IoT: Real-Time Liveness Based Face Recognition

1.Introduction

Nowadays, Face recognition is heavily being used by the cell phone industries. Face recognition is used as the prominent biometric security because of its quickness to recognize faces. Because of its increasing demand, companies are investing hugely in this market to increase reliability and scalability. I am going to use one of the modern ways of face recognition with a spoofing detection (eye blink based) algorithm to increase security. My main goal is to keep the project economical as well as to provide better security. I will be explaining more about it later in this project.

we will have to finish off the setup required for this project. I will be using a virtual machine called Virtual Box as my Virtual private server. You can also use Amazon S3 or Digital Ocean Droplets as a cloud server and deploy your code there. But, all of these come for a price, so I will be using Virtual Box for this project.

2. Software Setup

• Virtual Box

First, we will have to install Virtual Box software on our computer. You can download it from here: https://www.virtualbox.org/wiki/Downloads

Then you will have to install the Ubuntu server image file. You can download it from here: http://releases.ubuntu.com/16.04/ubuntu-16.04.6-server-i386.iso

Once you open your Virtual box, you will have to set your RAM to a minimum of 4 GB and hard disk memory to a minimum of 20 GB. After doing this install the ubuntu image file and then remember to increase your monitor counter or the processor count to a minimum of 4.

(optional) After this, you have to upgrade your Ubuntu server to 18.04 as it provides lots of new features and is better for this project. Type following lines on the terminal:

sudo apt-get update && sudo apt-get upgrade
sudo do-release-upgrade

As we will be using OpenCV in our project, it will use lots of GUIs which cannot be provided by the ubuntu server and hence you will have to install Ubuntu desktop on your server. Yes, I am doing this way because with one command tasksel you can use any other lighter Desktop on your server as per your requirement.

Let’s get into our business. Type the following commands on your ubuntu server

sudo apt update && sudo apt upgrade
sudo add-apt-repository universe
sudo add-apt-repository multiverse
sudo apt install tasksel
sudo tasksel

Then, you will see a menu pane, where you have to select Ubuntu desktop and now all the packages will install and may take some time depending on your network. After this, you have to reboot your system.

sudo reboot

Finally, you will see your Ubuntu Desktop. Enter your login credentials and you are done.

• Bolt IoT

Bolt IoT is used as an alert system in this project. It also has its own cloud platform.

You have to register to use its cloud platform. Go to this.. www.cloud.boltiot.com

Once done with the registration you will receive a verification email and do the verification process then you are good to go.

Now, click on API in the menu and you will find your API key copy it somewhere and keep it safe and save the unique Bolt id that you will see on the main window adjacent to online /offline label and will start with BOLTxxxxxxx.

Now in the ubuntu terminal on your desktop that you created, you will have to type following commands to install Bolt packages to use the bolt device from a remote server. Which works on MQTT for communication with the server. You will have to install the pip package manager to install all python tools.

** Remember Python 3 is already installed and comes with Ubuntu 18.04

sudo apt-get -y update
sudo apt install python3-pip
sudo pip3 install boltiot

• OpenCV

Install OpenCV which will be used for face recognition. There are various ways of installing OpenCV and the one using cmake is quite tedious, which might lead to errors if not done correctly. It might be difficult for ones just starting out with all of these. The version of OpenCV has to be 3.6. So, I will be doing it in an easy way. Type following commands on the terminal:

sudo apt-get install python3-opencv
pip3 install opencv-python

** you can check whether OpenCV is installed or not by typing python3 on terminal and then type the following commands in the interpreter.

** while working on programs if additional libraries are asked you can easily install it by using pip3 command such as for dlib, and face_recognition library which will be used in the project

example:

pip3 install dlib

• IP Webcam

I am using an IP webcam app that is available on Google play store. It is a streaming server that works on your local network. You will get the network address when you will click on the ‘Start server’.

** you can then go to that address and see your video stream, then click on ‘browser‘ and click on a video and open in a new tab and then use that link afterward in the project to stream video in OpenCV.

(Alternatively)you can also use Raspi camera, but a person just having a laptop and mobile with interest in computer vision should be able to do this project and understand the intricacies. As the project's focus is that it should be easy for the consumer to deal with and should be easily accessible.

We have to remember that we have to keep the computer and mobile phone on the same network as we are using the mobile phone’s camera for the live stream.

• Mailgun

Mailgun(Email Automation) is used for sending mail to the owner from the remote server. So we can program it to process email with help of Mailgun API. Therefore, for the notification purpose with the help of python, we can send an email request.

You have to first register here: https://www.mailgun.com/

Then, you have to go to Sandbox Domain and add Recipients and then you have to click on ’invite new recipient’ and add your email id. Then you will get your new sandbox domain.

Here, you have to copy all the important credentials like Mailgun API key, sandbox URL, sender email should be ‘mailgun@your sandbox URL’ and keep all of these safe somewhere.

You can see this example for your reference, we will be using similarly:

def send_simple_message():
    return requests.post(
           "https://api.mailgun.net/v3/YOUR_DOMAIN_NAME/messages",
            auth=("api", "YOUR_API_KEY"),
            data={"from": "Excited User <mailgun@YOUR_DOMAIN_NAME>",
            "to": ["bar@example.com", "YOU@YOUR_DOMAIN_NAME"],
            "subject": "Hello",
            "text": "Testing some Mailgun awesomeness!"})

3. Bolt Hardware Schematic

We will now keep the hardware setup ready. We will be using buzzer as an alert module in our home system so that if an intruder tries to unlock then it will buzz. So to do it we have to connect our buzzer’s long pin (+ve) to digital 0 pin of Bolt module and another leg to ground(GND) pin. You have to set it up as per the image is shown below. The buzzer is an output device. We can connect a switch as shown below, only if u want to keep the buzzer off after intruder detection, but in the program, I have made the buzzer switch off once intruder gets out of the video frame.

4. Face recognition and anti-spoof detection with an alert system

Now, comes the main part where we will have to keep a good focus to understand face recognition as well as the liveness detection that we will be working on.

So, let's divide the project into three parts: face recognition, eye-blink- heuristic, IOT alert

Step1: Face recognition

You may start with face detection first. Where the job is done by OpenCV.

The program will give you some hands-on to understand how each frame is captured in live video and the face is detected. The haar cascade library is used

Here is the sample program:

The file should end with.py :

import numpy as np
import cv2
# multiple cascades: https://github.com/Itseez/opencv/tree/master/data/haarcascades
#https://github.com/Itseez/opencv/blob/master/data/haarcascades/haarcascade_frontalface_default.xml
face_cascade = cv2.CascadeClassifier('haarcascade_frontalface_default.xml')
cap = cv2.VideoCapture(‘http://192.168.0.0:8080/video’)
while true:
ret, img = cap.read()
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
faces = face_cascade.detectMultiScale(gray, 1.3, 5)
for (x,y,w,h) in faces:
cv2.rectangle(img,(x,y),(x+w,y+h),(255,0,0),2)
roi_gray = gray[y:y+h, x:x+w]
roi_color = img[y:y+h, x:x+w]
cv2.imshow('img',img)
k = cv2.waitKey(30) & 0xff # press ESC to exit video
if k == 27:
break
cap.release()
cv2.destroyAllWindows()

You will have to download the haar cascade frontal face library which is a classifier in.xml format. Usually, the classifier comes bundled with Opencv 3.6. You can also program for eyes, mouth, etc. You can code similarly and use different classifiers as per need.

**This is just for practice and understanding for beginners if you want to start directly with recognition you can skip all theabove face detection parts.

There are various ways of doing face recognition. The most common way of doing it is using the bundled OpenCV recognizers which trains the dataset and stores it in.yml file.

The recognizers are:

1.Eigenface

2.Fisherface

3.LBPH(Local Binary Patterns Histogram)

You can go here to understand more about it:

https://docs.opencv.org/2.4/modules/contrib/doc/facerec/facerec_tutorial.html#eigenfaces

It depends on the needs of the project to use which recognizer, as each has its own advantages and disadvantages. Always remember it is the dataset you train that gives you suitable accuracies. More the dataset more the accuracy. So, you may require 100s or 1000s of a dataset of one person which will become difficult if you have to train more than 1 owner.

Hence using these would lead to disadvantages, as there are advances in face recognition more work is going in deep learning as OpenCV was not introduced for this purpose. So, deep learning(Deep Neural Network or Conventional Neural Network) is the key to face recognition which will consider various intricacies of the face.

We will use the face_recognition library in this project which is based on dlib’ s state of the art face recognition(deep learning library). It says that the model provides an accuracy of 99.38 %. You can click over here to know more about it: https://pypi.org/project/face_recognition/

Only one image dataset is required per user and the recognition is quite accurate. The face_recognition works on RGB image, use the dataset accordingly.

Identify faces in pictures:

Recognize who appears in each photo.

You can go through this example of face recognition provided by Ageitgey which is fast and accurate used for real-time video.

The face_recognition library encodes the image dataset and then tries to match with the face in each frame of the live video stream. The face is encoded into a 128-d embedding vector. The face detection is done is through hog (Histogram of Gradient) model by default, there is even CNN (Convolutional Neural Network) model, but it takes a lot of time to process on CPU and is more accurate, but if there are large dataset then it is tedious, and hence it is always better to stick with the default model.

The face recognition will work like this if you try to implement the above example code.

But, you might notice that if you use your photo or printed picture or selfie then it will detect and recognize your face. So this library will recognize the face quickly but it will not be able to find the difference between a live person and a fake person. As we are dealing with a project of a face as a security feature it is important to solve this issue. We will proceed to the next step to learn about liveness detection.

Step2: Anti-spoofing algorithm: eye -blink based heuristic approach

There are various ways of dealing with a fake and an original live face. This comes under the category of thwart attack. So, the attacks can be in different ways such as print attacks, replay, and 3D mask attack and different approaches are used to solve this problem:

1. Optical flow algorithm based where Keras 3D convolution can be used or something that finds a difference between 3D and 2D faces.

2. The binary classification used to detect live faces using SVM to classify real or fake faces works mostly with selfies.

3. Heuristic-based eye blink or lip movement is used to differentiate between a live and a fake face.

We will be using heuristic-based eye blink detection with the help of a facial landmark technique. There are many ways, one of the ways is training an eye with a close and open dataset using the deep neural network, but then we will have to use 1000’s of such image dataset of the close and open eye and then implement on a live video stream with face recognition. you can read through this to gain some knowledge.

Hence, we have to solve this problem and to integrate it with a face recognition system that will detect the face quickly. We will have to detect the eye blinks for the liveness check using a facial landmark shape predictor. This is the way I am going to deal with this project. Of course, there will be some disadvantages such as someone showing a recorded video, but to eliminate most of the attacks you will have to integrate all of the above approaches and then the program will become quite complex, but if you are really interested and passionate you can start doing research on deep learning and machine learning. And guess what!!! You can build projects like Apple Face ID... So, coming back to our project as I said we will use be using facial landmarks to detect blinks of an eye. The facial landmarks are explained in Adrian’s blog

Detecting facial landmarks will localize the face in the 2D image and then will detect the key facial structure and plot the region of interest of the face in other similar images if they exist. The facial landmark library plots 68 points on the input image and through this, we can see the eye plots from points 37 -46.

The EAR(Eye-Aspect-Ratio) gives a constant value when the eye is open and falls rapidly when the eye gets closed. So this helps us to accurately determine the person’s eye blinks based on the vertical and horizontal eye landmark points. The shape predictor will be used for the ROI face. The two constants will be used in our program one being EAR threshold d will be 0.3 and the other will be the consecutive frames for the eye to be below the threshold which will be 3. The deep learning face detector will be used as I said earlier that the default is the’ hog’ model. So, the eye blinks will be detected in the consecutive frame of a live stream. You can see the number of eye blinks and the threshold on the top of the frame!!!

In the program, we will be counting the number of eye blinks and this blinks will be a determining factor between a real image and a fake image. Whenever a face is detected it will detect the blinks of a person and then will try to recognize the face using face_recognition if known face then it will authorize and display the results and if unknown face comes up the same thing applies if the eye blinks are detected then it will proceed to check the person is known and unknown. In this way, we deal with the liveness of a face.

Next, we will be moving to the final step of adding an IoT alert system when an intruder is detected.

Step3: IoT Intruder alert

This is an integral part of home security and Industrial security as IoT has revolutionized the world in terms of its need and ease that it provides to get the data securely and get the information when we might be in any part of the world.

we will have to create a python file on the ubuntu server which will have all the Bolt credentials like Bolt ID and API key and save it as aconfiguration file econf.py file. This will help to connect to the bolt device.

I am using a mobile camera that will stream live video on the host address on the port 8080, you can also use a Raspi camera or a computer webcam for this project. In the virtual machine, we will program and save the intruder image. This image will be sent to the user with the help of the Mailgun notifier, which will help in sending mail and alerting the owner. The Mailgun request program is done in python and all the credentials will be used, which I had said earlier to save.

When the stream starts and if anyone tries to use a fake photo or a selfie to unlock, then the blink detector will try to detect blinks.

Note: due to frame drops there would be some glitches in detecting blink sometimes, so it depends on quality of stream.

If the intruder is detected then the message will be sent. Then the server sends a message to the owner's Gmail account with an inline image using Mailgun API, I have not used the loop for multiple images of intruders you can do it if you want. Simultaneously, the buzzer will start to buzz.

5.CONCLUSION

Lots of research is going on deep learning to provide the most accurate face recognition and to eliminate thwart attacks. The project that I have made can be improved by working on algorithms and adding more solutions to overcome such attacks.