Face Tracking and Identification Using Walle-ng

Project Description

This project is focused on edge computing using a Raspberry Pi, Intel Movidius Compute Stick, and a Pi-cam to detect, track, and identify faces from a video stream in real-time. The application continuously monitors and adjusts the vertical and horizontal position of two servos based on the location of faces in a frame. If a face is detected, then the activity is logged and a notification SMS will be sent to the administrator using Amazon Web Services (AWS) API Gateway, Lambda, and Simple Notification Services (SNS). This project expands on the face recognition demofound in the OpenCV open model zoo.

Requirements

The total build cost less than $175 USD. The hardware can purchased from Amazon, Sparkfun, or Adafruit.

• Raspberry Pi 4B (4GB preferred)
• Movidius Compute Stick 2
• Micro HDMI Cable (RPI 4 only)
• Keyboard and Mouse
• Micro SD card (8GB or greater preferred)
• MakerFocus Raspberry Pi Pan/Tilt Hat
• MakerFocus Raspberry Pi Night Vision Camera
• Adafruit Flex Cable for Raspberry Pi Cam
• iUniker Raspberry Pi Cluster Case
• Raspberry Pi Charger (RPI 4)
• AWS Console Access
• Male to Female GPIO Ribbon Cable
• Drill

Assembly

Walle-ng can be assembled in a number of configurations, and it really depends on use case and user preference. For this install, two sections of the iUniker Raspberry Pi Cluster Case were used to house the Raspberry Pi and mount the Pan/Tilt hat. The Raspberry Pi was installed on top of the lower section of the case, and the fan was installed on the top (outside). It is recommended to install the heat sinks, connect and feed the Picam flex cable through opening in the top plastic piece, and place the female end of the GPIO ribbon cable on the Raspberry Pi GPIO pins prior to assembling the enclosure around the Raspberry Pi.

The most difficult part of assembly was attaching the pan/tilt hat to the uppermost section of the iUniker case. The top plastic enclosure piece (part that holds the pan/tilt hat) is the same piece that is used for the top of an iUniker enclosure (same as the middle piece). Once the pan/tilt hat is assembled using the provided instructions, place the hat on top of the iUniker and position it in such a way that you can mark the position of the 4 holes, located in the corners of the pan/tilt hat PCB, using a marker on the iUniker plastic. Once the holes are drilled, secure the pan/tilt hat to the plastic using the available hardware included with the iUniker kit. Notice in the image that I was only able to secure 3 out of the 4 holes to the plastic due to an opening in the iUniker plastic piece. Don't forget to attach the servo motor wiring to the bottom of the pan/tilt hat prior to securing it to the iUniker plastic.

The last part of assembly includes attaching the fan to the 5v and ground GPIO pins, plugging in the Intel Movidius Compute Stick into one of the USB3.0 ports, connecting the micro-HDMI, keyboard, and mouse (if they are intended to be used), and attaching the male end of the GPIO ribbon cable to the pan/tilt hat. The positive (red) wire of the fam should connect to GPIO2 or GPIO4, and the ground (black) wire should connect to GPIO6. A GPIO pinout for the RPI4 can be found in the references section.

Setting up Amazon Web Services

All the services used in this tutorial qualify for AWS Free Tierusage. If you do not have AWS console access, you will need to create an account on the AWS website. This section will provide details on setting up AWS SNS, API Gateway, and Lambda.

AWS Simple Notification Services (SNS)

SNS is a highly available, durable, secure, fully managed pub/sub messaging service that can be configured to send SMS or email messages to devices that subscribe to an SNS topic. For this project, one SNS topic will be created, and a mobile device will be configured as a subscriber to the topic. This is necessary to receive messages that are triggered from walle-ng events. In the AWS console, go to Simple Notification Services and perform the following actions:

Simple Notification Service (SNS) --> Topics --> Create topic

Set the following settings:

• Name: walle-sns-topic
• Display name: Alert
• Click --> Create topic

Once the topic is created, a view similar to the image below should be visible. Copy down the ARN associated with the walle-sns-topic. In this example, it is arn:aws:sns:us-east-1:646789677679:walle-ng-topic.

Create a subscription to the topic:

• Click --> Create subscription
• Topic ARN: select the walle-sns-topic (if not already selected)
• Protocol: SMS (or email if you wish)
• Endpoint: +15556667777 (Enter phone number, or email if email was selected)
• Click --> Create subscription

SNS should be set up to send SMS messages to a mobile device. It is recommended to publish a test message to verify setup. In order to publish a test message, perform the following actions:

• In the sidebar, select Topics --> walle-ng-topic
• Click --> Publish message
• Subject: Test
• Message Body: Test Test
• Click --> Publish message

AWS Lambda

AWS Lambda lets you run code without provisioning or managing servers. It is serverless, and will only run when a a POST request is made to API Gateway. Lambda will be responsible for parsing POST requests from walle-ng and publishing SNS messages. In AWS console, go to Lambda and perform the following actions:

Lambda --> Create Function --> Select "Author from scratch"

In the Basic information section, set the following actions:

• Function name: walle-ng-lambda
• Runtime : Python3.6
• Role Name : walle-ng-role
• Permissions : Basic Lambda, Amazon SNS publish policy
• Click --> Create Function

Once the Lambda function is created, scroll down to the Function code section and copy/paste the code below into the lambda_function.py script. Under TOPIC_ARN_SMS, change the value <sns_topic_arn> to the ARN you noted when you created the walle-ng-topic. For this example, <sns_topic_arn> would be changed to arn:aws:sns:us-east-1:646789677679:walle-ng-topic.

When done, click --> Save.

import boto3
import urllib.parse
import logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)

TOPIC_ARN_SMS = "<sns_topic_arn>"
SMS_SUBJ = "Alert"
SMS_MESG = "Observed unknown person(s)"

'''Setup SNS resource'''
def publish_sms_msg(topic_arn=TOPIC_ARN_SMS, sms_mesg=SMS_MESG, sms_subj=SMS_SUBJ):
    sns = boto3.resource('sns')
    publish_sms(sns, topic_arn, sms_mesg, sms_subj)

'''Send the SMS'''
def publish_sms(sns, topic_arn, sms_mesg, sms_subj):
    topic = sns.Topic(topic_arn)
    topic.publish(Message=sms_mesg, Subject=sms_subj)

'''Event handler'''
def lambda_handler(event, context):
    if event['httpMethod'] == 'POST':
        msg = event['body']
        msg = urllib.parse.parse_qs(msg)
        if msg['message']:
            alert = msg['message'][0]
            logger.info(alert)
            publish_sms_msg(sms_mesg=alert)
            return {
                'statusCode': 200,
                'headers': {
                    'Content-Type': 'application/json'
                },
                'body': 'OK'
            }
        else:
            return {
                'statusCode': 499,
                'headers': {
                    'Content-Type': 'application/json'
                },
                'body': 'Go Away!'
            }
    else:
        return {
            'statusCode': 499,
            'headers': {
                'Content-Type': 'application/json'
            },
            'body': 'Go Away!'
        }

The final output should look similar to the following:

In the next step, verify that the Lambda function is set up correctly by executing a test. Perform the following actions in the Lambda console:

• Click --> Test
• Check --> Create new Test
• Event name --> SMSTest
• Replace the JSON with the following and click --> create

{
    'httpMethod': 'POST',
    'body': 'message=Observed+test'
}

• Click --> Actions --> Publish new version
• Click --> Test

You should receive an SMS message to the phone number that was subscribed to the SNS topic. A successful message indicates that Lambda is configured properly to work with SNS.

AWS API Gateway

Amazon API Gateway is a fully managed service that makes it easy for developers to publish, maintain, monitor, secure, and operate custom APIs. API Gateway will be configured to accept POST requests containing messages from walle-ng. To add a layer of security to endpoint devices communicating with the API, an API key will be generated and walle-ng will be configured to use the API for each call. In AWS console, go to API Gateway and perform the following actions:

API Gateway --> Create API

Set the following:

• API name: walle-ng-api
• Description: Walle-ng API
• Click --> Create api

Select walle-ng-api --> Actions --> Create Resource

Set the following:

• Resource name: Walle API Gateway
• Resource path: walle-api-gateway
• Check --> Enable API Gateway CORS
• Click --> Create resource
• Click --> Actions --> Create method
• Select --> POST (when done, push the check mark button next to the drop down list)

In the POST setup window, set the following:

• Integration type: Lambda
• Check --> Use Lambda Proxy integration
• Lambda function: walle-ng-lambda
• Click --> Save
• Add Permission to Lambda Function click --> Ok

In the left sidebar, select API Keys

• Click --> API Keys --> Actions --> Create API Keys

Set the following:

• Name: walle-api-key
• Description: Walle API Key
• After the key is created, select the api key name --> Show (next to API Key). Write down the API key so it can be referenced in the future.

In the left sidebar, select APIs --> walle-ng-api

Set the following:

• Click --> POST --> Method Request
• Click --> API key required --> True --> Click the checkmark to save
• Click --> OPTIONS --> Method Request
• Click --> API key required --> True --> Click the checkmark to save
• Click --> Actions --> Enable CORS --> Enable CORS and replace existing CORS headers --> Yes, replace exsting values

Set the following values in Deploy API:

• Click --> Actions --> Deploy API
• Stage --> Stage
• Stage ame --> prod
• Click --> Deploy

Make note of the Invoke URL. This value will be used in the Walle client config. In this example, the Invoke URL is https://2szxrk2gh4.execute-api.us-east-1.amazonaws.com/prod. In order to generate the URL that POST request will be sent to, add walle-ng-api to the end of the Invoke URL. The final URL for this example would be https://2szxrk2gh4.execute-api.us-east-1.amazonaws.com/prod/walle-ng-api.

Before continuing to the Raspberry Pi installation section, make sure the endpoint URL and the API key area available in a notepad. During the walle-ng install, a script will ask for these values to make setup easier.

Raspberry Pi Installation

At this point, the goal is to successfully flash Raspbian OS to a micro SD card, enable SSH, and expand the filesystem. Once the system is running, project dependencies and tools will be installed to get a working "walle-ng" device.

Begin by downloading Raspbian Desktopto your computer. Create a bootable USB drive using Unetbootin,BalenaEtcher, or Rufusand your micro SD card. For help on this subject, refer to the references section at the bottom of this tutorial.

# Alternative download using bash. In a shell, run:
wget https://downloads.raspberrypi.org/raspbian/images/raspbian-2019-09-30/2019-09-26-raspbian-buster.zip
shasum -a 256 2019-09-26-raspbian-buster.zip
unzip 2019-09-26-raspbian-buster.zip

Example using Balena Etcher to flash Raspbian OS to a micro SD card:

Next, re-insert the micro SD card into the computer and add a empty file named 'ssh' onto the boot partition. In Linux, issue lsblk to show the available drives and mount points. Issue touch /media/anton/boot/ssh to create the empty file. Note: This path will be different so modify it to reflect the path of the mounted micro SD card on your host machine. Once the file is created, unmount the bootable drive and insert the micro SD card into the Raspberry Pi. Note: If you plan on using a keyboard, mouse, and external monitor, you can skip this step and enable SSH when you login to the Raspberry Pi.

# Show partitions (after inserting the micro SD card)
lsblk

# View files in the boot partition
ls -la /media/anton/boot

# Create an empty ssh file in the boot partition and verify
touch /media/anton/boot/ssh
ls -la /media/anton/boot

Apply power and SSH, or login via the GUI, to the the Raspberry Pi. If you need a quick way to find the IP address of your Raspberry Pi on your local LAN, use Nmap to locate it. The default password for new images is raspberry. It is recommended to change the default password to enhance the security of your Raspberry Pi.

# Locate Pi on the network
sudo nmap -sn 192.168.198.0/24 | grep Raspberry -B 2

# SSH to Pi using the password raspberry
ssh pi@192.168.198.167

# Change the root password
passwd

Example of how to use nmap and SSH to the Raspberry Pi:

After SSHing, or gaining a shell via desktop GUI, use the sudo raspi-config command to expand the file system, enable the camera, and enable I2C. The specific settings that should be set are:

• Change User Password --> Set a new Pi password when prompted
• Network Options --> N2 Wi-fi --> Set your SSID and Password
• Localisation Options --> Change Locale --> Select 'en_US.UTF-8 UTF-8'
• Localisation Options --> Change Timezone --> America --> New York
• Interfacing Options --> P1 Camera --> Enable
• Interfacing Options --> P4 SPI --> Enable
• Interfacing Options --> P5 I2C --> Enable
• Advanced Options --> A1 Expand Filesystem
• Finish --> Reboot now --> Yes

Once complete, re-login or SSH to the Pi.

Install walle-ng on Raspberry Pi

In a shell, clone the Walle-ng project to the Pi using git. If this is intended to be set up as a headless install, meaning Walle-ng will start on boot using the provided systemd service, it is recommended to add custom head shots of known users prior to running ./install.sh. If this is a normal install where the camera output will be viewed on the desktop after running python3 walle-ng.py in a terminal, then the images can be added after running the install script and post reboot. In the event there are download errors when running the install script, it may be necessary to fix the broken links by modifying the install script to point to the latest download URLs.

When running ./install.sh, a series of questions will be presented based on how the user intends on running walle-ng. The first question asks if this will be run in headless mode. By saying 'y' to this, it modifies the config.cfg file so that the video stream will not be shown on the Desktop. The follow-on question to this is do you want to create a systemd service to start walle-ng on boot. If this is also set to 'y', then a file called walled.service will be placed in /etc/systemd/system and enabled to start walle-ng.py on boot.

The next set of questions asks the user if they intend on setting up notifications using AWS. If the user says 'y' to this, the first question asks the user to input the endpoint URL. This is the URL that was saved earlier when setting up API Gateway, and for this example was https://2szxrk2gh4.execute-api.us-east-1.amazonaws.com/prod/walle-ng-api. The next question asks the user to enter their API Gateway API key.

# In the pi user context, clone the project to the Pi user's root directory
cd ~/
git clone https://github.com/af001/Walle-ng.git
chmod 755 install.sh
./install.sh

Running walle-ng on Raspberry Pi

Before running the application, users must place images in the dataset directory. It is recommended to place 1-10 images for each user in the dataset directory. Images should be cropped to only include the heads of users, and each image should be labeled sequentially. An example of this would be Anton-0.jpg, Anton-1.jpg, Jane-0.png, Jane-1.png, etc. The Python script that reads the files expects the format to be either.jpg or.png, and the name associated with the photos will be the name used when sending notifications. The Raspberry Pi can be used to take photos. For users that would like to use the Raspberry Pi cam, issue the following command in a terminal:

raspistill -o Anton-0.jpg
raspistill -o Anton-1.jpg

For headless installs that implement systemd, apply power and the walle-ng device should automatically start tracking faces and sending notifications. Things to consider if using headless are 1) make sure to have your images set in the dataset directory and 2) to validate the device is running ssh to the Pi and run sudo systemctl status walled. To start and stop the service, issue sudo systemctl stop walled or sudo systemctl start walled.

For installations that intend on using the Desktop GUI, then starting the application can be done by opening a terminal and executing cd Walle-ng && python3 wally-ng.py. To quit running the application, place focus on the window that contains the video and press the q button. Things to consider are 1) make sure to have your images set in the dataset directory and 2) you must login to the Pi Desktop in order to view the output of the video stream.

# Open a terminal on the Pi Desktop
cd ~/Walle-ng
python3 walle-ng.py

As an alternative to logging into the Pi desktop, you can use SSH on your host computer and run the following commands to view stream:

# SSH to the Pi with the -X option to view the steam on your host machine
ssh -X pi@192.168.198.167
cd ~/Walle-ng
python3 walle-ng.py

Customizing walle-ng

All settings for walle-ng are set in the config.cfg file located in the config directory. A more detailed explanation of the parameters are as follows:

[General]
# Input video device, default is 0
input_video    =0

# Output steam file. This writes the video stream to /tmp and can be read by other applications
output_video   =/tmp/output.mpeg

# Show the video stream in a seperate window on the desktop. Set to False for headless installs.
do_output      =True

# Auto-pause after each frame
do_timelapse   =False

# Crop the input stream to this width
crop_width     =0

# Crop the input stream to this height
crop_height    =0

# Output log file for debugging
log_file       =/home/pi/Walle-ng/walle-ng.log

[Faces]
# Path to headshots for performing face identification
# Images should be in the format of Name-0.jpg, Name-1.png
db_path        =/home/pi/Walle-ng/dataset

# Use Face Detection model to find faces on the face images, otherwise use full images.
do_detector    =False

[Model]
# Face detection model path
model_fd      =/home/pi/Walle-ng/models/face-detection-retail-0005.xml

# Landmark regression model path
model_ld      =/home/pi/Walle-ng/models/landmarks-regression-retail-0009.xml

# Face reidentification model path
model_rd      =/home/pi/Walle-ng/models/face-reidentification-retail-0095.xml

[Inference]
# Inference device, can be [CPU,GPU,FPGA,MYRIAD,HETERO]
# To use Movidius, set to MYRIAD
device_fd      =MYRIAD
device_lm      =MYRIAD
device_rd      =MYRIAD

# Probability threshold for face detections
thresh_fd      =0.6

# Cosine distance threshold between two vectors for face identification
thresh_rd      =0.3

# Scaling ratio for bboxes passed to face recognition
scale_ratio    =1.15

# Show verbose debugging to logs
do_verbose     =False

# Show stats
do_stats       =False

# Allow growing the face database, in addition allow dumping new faces on disk. In that
# case the user will be asked if he wants to add a specific image to the images gallery.
# The user should specify the name for the image in the open window and press `Enter`.
# If it's not, then press `Escape`. The user may add new images for the same person by
# setting the same name in the open window.
do_grow        =False

[AWS]
# AWS API Gateway endpoint URL. This should point to the POST request URL
notify_url     =<api_gateway_endpoint_url>

# AWS API Gateway API Key
api_key        =<api_gateway_api_key>

# Enable/Disable notifications, True = Enabled
do_notify      =True

Documentation Link: https://af001.github.io/Walle-ng-Documentation/intro.html