AI Facial Recognition WiFi Lock

Introduction:

iPhone X Face ID hits people with surprise, people starting to realize through AI deep learning their face is much more unique and accurate than their fingerprint.

We've built a platform that shows how the AI on the Edge works on Ultra96 and Intel Movidius NCS and Tensorflow Inception Facenet, using all the default camera that came with the development kit. This project can be extended to using facial recognition to unlock deadbolts, record entries, turn on different light themes, and many others.

For the first part of the guide, we used Android to open the lock via WiFi. If you have already completed the Netduino WiFi Lock, you can skip straight to Step 5. In which we will be using AI on the Edge to unlock the deadbolts.

TensorFlow Facenet

We are using Facenet based on https://github.com/davidsandberg/facenet along with Movidius' SDK. The Xilinx Ultra96 FPGA will be perfect to handle the OpenCV and we will be inference the facial recognition through NCS.

Facenet is not a classifier that is trained to classify a face as belonging to a particular individual that it was trained on. Instead it is trained to find and quantify landmarks on faces in general. By comparing the face landmark quantification values (network inference output) on two images, it is possible to determine how likely the two faces are of the same person. This allows us to create a simple and easy way to do facial recognition over a single image. Movidius will be used to inference the AI in real time on the edge devices.

Step 1: Gather Materials

We will need following items:

• Ultra96 Board
• Netduino 3 WiFi
• Seeed Grove Base Shield
• Seeed Relay
• Deadbolt
• Camera
• A box for enclosure

Step 2: Set Up Netduino 3 WiFi

Netduino is a C# and .NET based IoT board that has a lot of features, this way we can create the entire project based on C#.

Install Visual Studio from https://visualstudio.microsoft.com/vs/mac/, In this case I am using Mac. Afterwards go to Visual Studio Community menu -> Extension -> Gallery, and search for "MicroFramework" to install Micro Framework.

Follow this page's instruction on update to the latest firmware

http://developer.wildernesslabs.co/Netduino/About/Updating_Firmware/#mac

After installing latest firmware, we need to setup the WiFi network via Netduino Deploy so the device is connected to the internet, We will use static IP here so we can call the sockets using our phone later

Step 3: Wire the Hardware

Because the deadbolt requires 12v and regular output of Netduino would only give 5v, We would need to tap into Netduino's power source by solder 2 wires on top like below, this would allow us to turn the deadbolt on and off without additional power source.

We can then tap the output of the 12v from the device to connect directly to relay, one to deadbolt, so relay would receive 12v postive from the board itself and negative to the relay. When all said and done it should look like following

Step 4: Set Up Netduino Webserver

We can now use following code to connect to the computer around. When it works we can move onto next step, we've created a Netduino Webserver for other devices to connect via WiFi network. The following code setup the socket, and when receiving "ON" it will turn on LED as well as the relay, which allow us to open the lock.

using Microsoft.SPOT.Hardware;
using Microsoft.SPOT.Net.NetworkInformation;
using SecretLabs.NETMF.Hardware;
using SecretLabs.NETMF.Hardware.Netduino;
namespace Lock
{
public class Program
{
// Main method is called once as opposed to a loop method called over and over again
public static void Main()
{
Thread.Sleep(5000);
App app = new App();
app.Run();
OutputPort led = new OutputPort(Pins.ONBOARD_LED, false);
OutputPort relay = new OutputPort(Pins.GPIO_PIN_D5, false);
int port = 80;
Socket listenerSocket = new Socket(AddressFamily.InterNetwork,
SocketType.Stream,
ProtocolType.Tcp);
IPEndPoint listenerEndPoint = new IPEndPoint(IPAddress.Any, port);
Debug.Print("setting up socket");
// bind to the listening socket
listenerSocket.Bind(listenerEndPoint);
// and start listening for incoming connections
listenerSocket.Listen(1);
Debug.Print("listening");
while (true)
{
Debug.Print(".");
// wait for a client to connect
Socket clientSocket = listenerSocket.Accept();
// wait for data to arrive
bool dataReady = clientSocket.Poll(5000000, SelectMode.SelectRead);
// if dataReady is true and there are bytes available to read,
// then you have a good connection.
if (dataReady && clientSocket.Available > 0)
{
byte[] buffer = new byte[clientSocket.Available];
clientSocket.Receive(buffer);
string request =
new string(System.Text.Encoding.UTF8.GetChars(buffer));
Debug.Print(request);
if (request.IndexOf("ON") >= 0)
{
led.Write(true);
relay.Write(true);
Thread.Sleep(5000);
led.Write(false);
relay.Write(false);
}
string statusText = "Lock is " + (led.Read() ? "ON" : "OFF") + ".";
// return a message to the client letting it
// know if the LED is now on or off.
string response = statusText ;
clientSocket.Send(System.Text.Encoding.UTF8.GetBytes(response));
}
// important: close the client socket
clientSocket.Close();
}
}
}
public class App
{
NetworkInterface[] _interfaces;
public string NetDuinoIPAddress { get; set; }
public bool IsRunning { get; set; }
public void Run()
{
//this.IsRunning = true;
bool goodToGo = InitializeNetwork();
this.IsRunning = false;
}
protected bool InitializeNetwork()
{
if (Microsoft.SPOT.Hardware.SystemInfo.SystemID.SKU == 3)
{
Debug.Print("Wireless tests run only on Device");
return false;
}
Debug.Print("Getting all the network interfaces.");
_interfaces = NetworkInterface.GetAllNetworkInterfaces();
// debug output
ListNetworkInterfaces();
// loop through each network interface
foreach (var net in _interfaces)
{
// debug out
ListNetworkInfo(net);
switch (net.NetworkInterfaceType)
{
case (NetworkInterfaceType.Ethernet):
Debug.Print("Found Ethernet Interface");
break;
case (NetworkInterfaceType.Wireless80211):
Debug.Print("Found 802.11 WiFi Interface");
break;
case (NetworkInterfaceType.Unknown):
Debug.Print("Found Unknown Interface");
break;
}
// check for an IP address, try to get one if it's empty
return CheckIPAddress(net);
}
// if we got here, should be false.
return false;
}
public void MakeWebRequest(string url)
{
var httpWebRequest = (HttpWebRequest)WebRequest.Create(url);
httpWebRequest.Method = "GET";
httpWebRequest.Timeout = 1000;
httpWebRequest.KeepAlive = false;
httpWebRequest.GetResponse();
/*
           using (var streamReader = new StreamReader(httpResponse.GetResponseStream()))
           {
               var result = streamReader.ReadToEnd();
               Debug.Print("this is what we got from " + url + ": " + result);
           }*/
}
protected bool CheckIPAddress(NetworkInterface net)
{
int timeout = 10000; // timeout, in milliseconds to wait for an IP. 10,000 = 10 seconds
// check to see if the IP address is empty (0.0.0.0). IPAddress.Any is 0.0.0.0.
if (net.IPAddress == IPAddress.Any.ToString())
{
Debug.Print("No IP Address");
if (net.IsDhcpEnabled)
{
Debug.Print("DHCP is enabled, attempting to get an IP Address");
// ask for an IP address from DHCP [note this is a static, not sure which network interface it would act on]
int sleepInterval = 10;
int maxIntervalCount = timeout / sleepInterval;
int count = 0;
while (IPAddress.GetDefaultLocalAddress() == IPAddress.Any && count < maxIntervalCount)
{
Debug.Print("Sleep while obtaining an IP");
Thread.Sleep(10);
count++;
};
// if we got here, we either timed out or got an address, so let's find out.
if (net.IPAddress == IPAddress.Any.ToString())
{
Debug.Print("Failed to get an IP Address in the alotted time.");
return false;
}
Debug.Print("Got IP Address: " + net.IPAddress.ToString());
return true;
//NOTE: this does not work, even though it's on the actual network device. [shrug]
// try to renew the DHCP lease and get a new IP Address
//net.RenewDhcpLease ();
//while (net.IPAddress == "0.0.0.0") {
//    Thread.Sleep (10);
//}
}
else
{
Debug.Print("DHCP is not enabled, and no IP address is configured, bailing out.");
return false;
}
}
else
{
Debug.Print("Already had IP Address: " + net.IPAddress.ToString());
return true;
}
}
protected void ListNetworkInterfaces()
{
foreach (var net in _interfaces)
{
switch (net.NetworkInterfaceType)
{
case (NetworkInterfaceType.Ethernet):
Debug.Print("Found Ethernet Interface");
break;
case (NetworkInterfaceType.Wireless80211):
Debug.Print("Found 802.11 WiFi Interface");
break;
case (NetworkInterfaceType.Unknown):
Debug.Print("Found Unknown Interface");
break;
}
}
}
protected void ListNetworkInfo(NetworkInterface net)
{
Debug.Print("MAC Address: " + BytesToHexString(net.PhysicalAddress));
Debug.Print("DHCP enabled: " + net.IsDhcpEnabled.ToString());
Debug.Print("Dynamic DNS enabled: " + net.IsDynamicDnsEnabled.ToString());
Debug.Print("IP Address: " + net.IPAddress.ToString());
Debug.Print("Subnet Mask: " + net.SubnetMask.ToString());
Debug.Print("Gateway: " + net.GatewayAddress.ToString());
if (net is Wireless80211)
{
var wifi = net as Wireless80211;
Debug.Print("SSID:" + wifi.Ssid.ToString());
}
this.NetDuinoIPAddress = net.IPAddress.ToString();
}
private static string BytesToHexString(byte[] bytes)
{
string hexString = string.Empty;
// Create a character array for hexadecimal conversion.
const string hexChars = "0123456789ABCDEF";
// Loop through the bytes.
for (byte b = 0; b < bytes.Length; b++)
{
if (b > 0)
hexString += "-";
// Grab the top 4 bits and append the hex equivalent to the return string.        
hexString += hexChars[bytes[b] >> 4];
// Mask off the upper 4 bits to get the rest of it.
hexString += hexChars[bytes[b] & 0x0F];
}
return hexString;
}
public string getIPAddress()
{
return this.NetDuinoIPAddress;
}
}
}

After uploading you'd see the image below like, with internet connected. Once deployed on Netduino we can use following command on terminal

echo "ON" | nc 192.168.1.90 80 

Netduino Webserver Unlocking Deadbolt as shown below

Step 5: Installing PYNQ linux

Ultra96 is pretty new, but the support group was kind enough to get the base Ubuntu running, which is a great deal since this allows me to build different platforms off ultra96. The compiled version of debian can be downloaded fromhttps://fileserver.linaro.org/owncloud/index.php/s/jTt3MYSuwtLuf9d After that we can use tools like ether to load it onto the mini-sd card.

Upon booting, we first have to fix some errors by removing corrupted repos.

sudo rm -r /var/lib/apt/lists/*

This allows us to install all the packages as needed to use the platform.

Step 6: Install Movidius NCSDK

So to get AI and Computer Vision to work, we can utilize the Movidius NCS which have set of tools to get our project running. There isn't a walkthrough for Ultra96 so we've basically referenced this through https://movidius.github.io/blog/ncs-apps-on-rpi/

First we have to install dependencies, this part does not come with PYNQ, so we will install them to make sure everything works.

apt-get install libgstreamer1.0-0 gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly gstreamer1.0-libav gstreamer1.0-doc gstreamer1.0-tools libgstreamer-plugins-base1.0-dev
apt-get install libgtk-3-dev
apt-get install -y libprotobuf-dev libleveldb-dev libsnappy-dev
apt-get install -y  libopencv-dev libhdf5-serial-dev 
apt-get install -y protobuf-compiler byacc libgflags-dev 
apt-get install -y libgoogle-glog-dev liblmdb-dev libxslt-dev 

Next we can install NCSDK SDK, which contains API that can connect applications to the NCS. Since NCSDK was not built for Ultra96, we can make following modifications for the workaround with markjay4k's version

cd /home/xilinx/
mkdir -p workspace
cd workspace
git clone https://github.com/markjay4k/ncsdk-aarch64.git
cd ncsdk/api/src
make
make install 

This work around should get us the NCSDK API working with Ultra96 and PYNQ

Next we will try to get NcAppZoo and Hello World running for Neural Computing Stick, we need to repeat the old python 3.6 on pynq

Step 7: AI on the Edge with Ultra96 and NCS

There are a lot of ways we can make the trigger, since this is an AI based application I will make an AI based trigger. In this guide we will be using a SSD neural net that is pre-trained and working with Caffe, and the detection will be garbage. So we'd also learn how to utilize other neural network with little bit of work.

In this step we, previously we've trained the through caffe model, we have to compile the graph on another machine as we've only installed API and not toolkit as they are not being built for aarch64. However, since the API works, we can simply build it on another machine and transfer the graph file over to the Ultra96. The FPGA can handle all the CV2, and the Movidius NCS.

Install Movidius NCS with Facenet

cd ~/workspace
git clone https://github.com/movidius/ncappzoo.git
cd ncappzoo
make
cd tensorflow
make

Zip the entire folder and copy it onto ultra96

Step 8: Build Facial Recognition Validation

The Movidius App Zoo should be able to get the entire software stack for NCS and OpenCV, next is we can create our own folder and run based on the video_face_matcher example. We only need to change is validation image under validated_images as facenet can train based on this. You can see the full code on github repo.

Using the following code, we should be able to access the WiFi point same as WiFi Arduino lock with the cell phone. How we are doing it is specifically making sure that we have validated the image over 10 frames to avoid false positive.

#! /usr/bin/env python3
# Copyright(c) 2017 Intel Corporation. 
# License: MIT See LICENSE file in root directory.
from mvnc import mvncapi as mvnc
import numpy
import cv2
import sys
import os
import requests
EXAMPLES_BASE_DIR='../../'
IMAGES_DIR = './'
VALIDATED_IMAGES_DIR = IMAGES_DIR + 'validated_images/'
validated_image_filename = VALIDATED_IMAGES_DIR + 'valid.jpg'
GRAPH_FILENAME = "facenet_celeb_ncs.graph"
# name of the opencv window
CV_WINDOW_NAME = "FaceNet"
CAMERA_INDEX = 0
REQUEST_CAMERA_WIDTH = 640
REQUEST_CAMERA_HEIGHT = 480
# the same face will return 0.0
# different faces return higher numbers
# this is NOT between 0.0 and 1.0
FACE_MATCH_THRESHOLD = 1.0
# Run an inference on the passed image
# image_to_classify is the image on which an inference will be performed
#    upon successful return this image will be overlayed with boxes
#    and labels identifying the found objects within the image.
# ssd_mobilenet_graph is the Graph object from the NCAPI which will
#    be used to peform the inference.
def run_inference(image_to_classify, facenet_graph):
   # get a resized version of the image that is the dimensions
   # SSD Mobile net expects
   resized_image = preprocess_image(image_to_classify)
   # ***************************************************************
   # Send the image to the NCS
   # ***************************************************************
   facenet_graph.LoadTensor(resized_image.astype(numpy.float16), None)
   # ***************************************************************
   # Get the result from the NCS
   # ***************************************************************
   output, userobj = facenet_graph.GetResult()
   return output
# overlays the boxes and labels onto the display image.
# display_image is the image on which to overlay to
# image info is a text string to overlay onto the image.
# matching is a Boolean specifying if the image was a match.
# returns None
framecount = 0
def overlay_on_image(display_image, image_info, matching):
   global framecount
   rect_width = 10
   offset = int(rect_width/2)
   if (image_info != None):
       cv2.putText(display_image, image_info, (30, 30), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (255, 0, 0), 1)
   if (matching):
       framecount += 1
           # match, green rectangle
       cv2.rectangle(display_image, (0+offset, 0+offset),
                     (display_image.shape[1]-offset-1, display_image.shape[0]-offset-1),
                     (0, 255, 0), 10)
       if framecount >= 20:
           url = "http://192.168.21.210"
           foobar ="ON"
           try:
               requests.post(url, data="ON")
           except:
               print('success')
           framecount=0
   else:
       # not a match, red rectangle
       framecount = 0
       cv2.rectangle(display_image, (0+offset, 0+offset),
                     (display_image.shape[1]-offset-1, display_image.shape[0]-offset-1),
                     (0, 0, 255), 10)
# whiten an image
def whiten_image(source_image):
   source_mean = numpy.mean(source_image)
   source_standard_deviation = numpy.std(source_image)
   std_adjusted = numpy.maximum(source_standard_deviation, 1.0 / numpy.sqrt(source_image.size))
   whitened_image = numpy.multiply(numpy.subtract(source_image, source_mean), 1 / std_adjusted)
   return whitened_image
# create a preprocessed image from the source image that matches the
# network expectations and return it
def preprocess_image(src):
   # scale the image
   NETWORK_WIDTH = 160
   NETWORK_HEIGHT = 160
   preprocessed_image = cv2.resize(src, (NETWORK_WIDTH, NETWORK_HEIGHT))
   #convert to RGB
   preprocessed_image = cv2.cvtColor(preprocessed_image, cv2.COLOR_BGR2RGB)
   #whiten
   preprocessed_image = whiten_image(preprocessed_image)
   # return the preprocessed image
   return preprocessed_image
# determine if two images are of matching faces based on the
# the network output for both images.
def face_match(face1_output, face2_output):
   if (len(face1_output) != len(face2_output)):
       print('length mismatch in face_match')
       return False
   total_diff = 0
   for output_index in range(0, len(face1_output)):
       this_diff = numpy.square(face1_output[output_index] - face2_output[output_index])
       total_diff += this_diff
   print('Total Difference is: ' + str(total_diff))
   if (total_diff < FACE_MATCH_THRESHOLD):
       # the total difference between the two is under the threshold so
       # the faces match.
       return True
   # differences between faces was over the threshold above so
   # they didn't match.
   return False
# handles key presses
# raw_key is the return value from cv2.waitkey
# returns False if program should end, or True if should continue
def handle_keys(raw_key):
   ascii_code = raw_key & 0xFF
   if ((ascii_code == ord('q')) or (ascii_code == ord('Q'))):
       return False
   return True
# start the opencv webcam streaming and pass each frame
# from the camera to the facenet network for an inference
# Continue looping until the result of the camera frame inference
# matches the valid face output and then return.
# valid_output is inference result for the valid image
# validated image filename is the name of the valid image file
# graph is the ncsdk Graph object initialized with the facenet graph file
#   which we will run the inference on.
# returns None
def run_camera(valid_output, validated_image_filename, graph):
   camera_device = cv2.VideoCapture(CAMERA_INDEX)
   camera_device.set(cv2.CAP_PROP_FRAME_WIDTH, REQUEST_CAMERA_WIDTH)
   camera_device.set(cv2.CAP_PROP_FRAME_HEIGHT, REQUEST_CAMERA_HEIGHT)
   actual_camera_width = camera_device.get(cv2.CAP_PROP_FRAME_WIDTH)
   actual_camera_height = camera_device.get(cv2.CAP_PROP_FRAME_HEIGHT)
   print ('actual camera resolution: ' + str(actual_camera_width) + ' x ' + str(actual_camera_height))
   if ((camera_device == None) or (not camera_device.isOpened())):
       print ('Could not open camera.  Make sure it is plugged in.')
       print ('Also, if you installed python opencv via pip or pip3 you')
       print ('need to uninstall it and install from source with -D WITH_V4L=ON')
       print ('Use the provided script: install-opencv-from_source.sh')
       return
   frame_count = 0
   cv2.namedWindow(CV_WINDOW_NAME)
   found_match = False
   while True :
       # Read image from camera,
       ret_val, vid_image = camera_device.read()
       if (not ret_val):
           print("No image from camera, exiting")
           break
       frame_count += 1
       frame_name = 'camera frame ' + str(frame_count)
       # run a single inference on the image and overwrite the
       # boxes and labels
       test_output = run_inference(vid_image, graph)
       if (face_match(valid_output, test_output)):
           print('PASS!  File ' + frame_name + ' matches ' + validated_image_filename)
           found_match = True
       else:
           found_match = False
           print('FAIL!  File ' + frame_name + ' does not match ' + validated_image_filename)
       overlay_on_image(vid_image, frame_name, found_match)
       # check if the window is visible, this means the user hasn't closed
       # the window via the X button
       prop_val = cv2.getWindowProperty(CV_WINDOW_NAME, cv2.WND_PROP_ASPECT_RATIO)
       if (prop_val < 0.0):
           print('window closed')
           break
       # display the results and wait for user to hit a key
       cv2.imshow(CV_WINDOW_NAME, vid_image)
       raw_key = cv2.waitKey(1)
       if (raw_key != -1):
           if (handle_keys(raw_key) == False):
               print('user pressed Q')
               break
   if (found_match):
       cv2.imshow(CV_WINDOW_NAME, vid_image)
       cv2.waitKey(0)
# This function is called from the entry point to do
# all the work of the program
def main():
   # Get a list of ALL the sticks that are plugged in
   # we need at least one
   devices = mvnc.EnumerateDevices()
   if len(devices) == 0:
       print('No NCS devices found')
       quit()
   # Pick the first stick to run the network
   device = mvnc.Device(devices[0])
   # Open the NCS
   device.OpenDevice()
   # The graph file that was created with the ncsdk compiler
   graph_file_name = GRAPH_FILENAME
   # read in the graph file to memory buffer
   with open(graph_file_name, mode='rb') as f:
       graph_in_memory = f.read()
   # create the NCAPI graph instance from the memory buffer containing the graph file.
   graph = device.AllocateGraph(graph_in_memory)
   validated_image = cv2.imread(validated_image_filename)
   valid_output = run_inference(validated_image, graph)
   run_camera(valid_output, validated_image_filename, graph)
   # Clean up the graph and the device
   graph.DeallocateGraph()
   device.CloseDevice()
# main entry point for program. we'll call main() to do what needs to be done.
if __name__ == "__main__":
   sys.exit(main())

When we recognize wrong face we can display the red borders around, to indicate the wrong person being detected.

When the right person is being detected we can display green border around, once they pass the 10 frames we can use that to unlock the deadbolt

When everything is deployed you should see

Now we can see a full demo of how Ultra96 with AI on the Edge opening up the deadbolt via Wifi

Here is the full demo