Published September 15, 2024 © Apache-2.0

Smart Lock using Face Recognition

This project presents a Smart Lock that uses ML to detect Authorized faces. This ML Model runs on the Edge with no need for backend server.

IntermediateShowcase (no instructions)Over 1 day271

Things used in this project

Hardware components

RT-Thread Vision Board

DC Lock

Software apps and online services

RT-Thread IoT OS

Edge Impulse Studio

OpenMV

Story

Introduction

With latest advancement in technology it became feasible to run Machine learning (ML) Model on the Edge, without the need to send the data to the backend to be processed on big servers. On the edge we can deploy the Model not on Single Board Computer (SBC) but on low cost Microcontroller.

Scope

In this Project we demonstrate how to use a ML model to Recognize faces and control a DC Lock, this all taking place on the Edge.

How Project Work

Using "Edge Impulse" we built a ML Model that can recognize specific face. We run this model on the Vision Board to open the lock if the board recognized the correct face.

RT-Thread Vision Board

The Vision Board is based on Renesas RA8D1 microcontroller equipped with ARM Cortex-M85 running at 480 MHz. The Vision board can be programed using MicroPython using OpenMV IDE.

RT-Thread Vision Board

Vision Board : Pinout

Vision Board and MicroPython

The Vision board cab be programed with MicroPython using OpenMV IDE. To be able to user the MicroPython interpreter you have to flash the board with the example provided by RT-Thread. You can find this example in RT-Thread Studio, just compile and flash it without any modification.

RT-Thread Studio example

Now you will be able to use the board as OpenMV board, and program it using MicroPython in the OpenMV IDE

Vision Board as OpenMV board

Control GPIO pins in MicroPython

The Current Board firmware that enable the board to run MicroPython only allow controlling the RGB LED, so in order to control a GPIO pins we need to modify RT-Thread Example to add another Pin. For simplicity I configured the Pin that connect to the DC-Lock as a fourth Pin, as the DC lock is controlled by a HIGH-LOW signal. Code for this change can be found the code section.

Machine Learning Pipeline

The process of developing Machine Learning application requires multiple steps that need to be performed, the following Diagram shows the mandatory steps need to have working machine learning application.

Machine Learning Pipeline

Data Collection : The process of developing machine learning application always starts with collecting the data that will be used to teach the model about the problem we need to solve. For example if we are building a model to detect spam emails, we need to have a huge email samples some of them are normal emails while the others are spam emails.
Training : This is the process of teaching the model using the data in the previous step, for simplicity we can say we are giving the model examples and let him learn from these samples.
Deployment : This is the process of installing our model, and this is the final step before being able to use the model, weather to deploy it on a powerful server or and edge device that run on a battery.
Testing : Starting from here we can use our ML application, but before the first use we need to try it to know the level of accuracy we have achieved with this Model.

Build Dataset

Our dataset in the application will consist of images, with two labels (Categories). The first set is images with the valid face that will open the lock, and the other set of images will contain faces that aren't consider valid.

For the first set of data with the valid face we have automated the process by writing a python script that will run on a PC to capture images and crop the image to have the face without much information this script has speed up the process very much. (Script can be found in the code section).

For the second set of data we have used random faced that we got from Kaggle LINK .

Train the ML Model

With Edge Impulse we can use image data we have prepared in the previous step to train out model. After Training the model we see training Accuracy exceeds 84%.

Training Performance

Test the Model (Optional)

With the Help of Edge Impulse we can test our model virtually without the need to use the actual Hardware. For sure the result can differ from the testing result on the actual hardware, but it gives us an indication about the model performance and whether we need adjust our model and retrain out model to increase accuracy.

Virtual testing result

Deploy the ML Model

With support of Edge Impulse we can generate the model as OpenMV Library, this will generate 3 files :

.tflite file, which represents the model itself
.txt file, which contains all the labels used by the model
.py file, which is a sample code the could be used directly in the OpenMV IDE, this code uses the model to recognize valid faces.

Both.tflite and.txt need to copied to the SD card in the RT-Vision Board, and.py file will be run from OpenMV IDE

Generate the model in EdgeImpulse

Testing the model on Real Target

After running the Python script attached in the code section, we can see that board can recognize valid persons with good accuracy, I've set in the code to open the lock if the confidence above 60%. In some case with good light condition the confidence reaches more than 90%.

Actual testing

Code

"""
    Project  Name : Smart Lock
    Auther : Mohamed Bedair
    Description : This project recognize valid faces and open DC Lock when detected
   
"""

import sensor, image, time, os, tf, uos, gc
from machine import LED
DC_LOCK = LED("DC_LOCK")

sensor.reset()                         # Reset and initialize the sensor.
sensor.set_pixformat(sensor.RGB565)    # Set pixel format to RGB565 (or GRAYSCALE)
sensor.set_framesize(sensor.QVGA)      # Set frame size to QVGA (320x240)
sensor.set_windowing((240, 240))       # Set 240x240 window.
sensor.skip_frames(time=2000)          # Let the camera adjust.

net = None
labels = None

try:
    # load the model, alloc the model file on the heap if we have at least 64K free after loading
    net = tf.load("trained.tflite", load_to_fb=uos.stat('trained.tflite')[6] > (gc.mem_free() - (64*1024)))
except Exception as e:
    print(e)
    raise Exception('Failed to load "trained.tflite", did you copy the .tflite and labels.txt file onto the mass-storage device? (' + str(e) + ')')

try:
    labels = [line.rstrip('\n') for line in open("labels.txt")]
except Exception as e:
    raise Exception('Failed to load "labels.txt", did you copy the .tflite and labels.txt file onto the mass-storage device? (' + str(e) + ')')

clock = time.clock()
while(True):
    clock.tick()

    img = sensor.snapshot()

    # default settings just do one detection... change them to search the image...
    for obj in net.classify(img, min_scale=1.0, scale_mul=0.8, x_overlap=0.5, y_overlap=0.5):
        print("**********\nPredictions at [x=%d,y=%d,w=%d,h=%d]" % obj.rect())
        img.draw_rectangle(obj.rect())
        # This combines the labels and confidence values into a list of tuples
        predictions_list = list(zip(labels, obj.output()))

        for i in range(len(predictions_list)):
            print("%s = %f" % (predictions_list[i][0], predictions_list[i][1]))

    #print(clock.fps(), "fps")
    if predictions_list[1][1] > 0.6:
        print("Valid Face Detected")
        DC_LOCK.on()
        time.sleep_ms(5000)
        DC_LOCK.off()

import cv2 as cv
import time
import os

# Image directory
Dataset_Dir = r'D:\workspace\Contests\RT-Thread Design Contest\Dataset'

counter = 0

face_classifier = cv.CascadeClassifier(
    cv.data.haarcascades + "haarcascade_frontalface_default.xml"
)


def Detect_And_Save_Faces(frame):
    global counter
    gray_image = cv.cvtColor(frame, cv.COLOR_BGR2GRAY)
    faces = face_classifier.detectMultiScale(gray_image, 1.1, 5, minSize=(40, 40))
    for (x, y, w, h) in faces:
        crop_img = frame[x-30:y + h + 40 + 20, y-50:x + w + 20]
        filename = "Img_" + str(counter) + ".jpg"
        cv.imwrite(os.path.join(Dataset_Dir , filename), crop_img)
        counter += 1

        cv.rectangle(frame, (x-30, y-50), (x + w + 20 , y + h + 40), (0, 255, 0), 4)

# Create video captuer instance and configure the camera options 
cap = cv.VideoCapture(cv.CAP_DSHOW)
cap.set(cv.CAP_PROP_FRAME_WIDTH, 500)
cap.set(cv.CAP_PROP_FRAME_HEIGHT, 500)
cap.set(cv.CAP_PROP_FPS, 60)


while True:
    ret, frame = cap.read()

    if not ret:
        print("Invaild Input")
        break
    
    Detect_And_Save_Faces(frame) 

    cv.imshow("Window", frame)

    time.sleep(0.5)

    key = cv.waitKey(1)
    if key == ord('q'):
        break


cap.release()
cv.destroyAllWindows()

Credits

Mohamed Ali Bedair

7 projects • 4 followers

Engineer/Maker

Smart Lock using Face Recognition

Things used in this project

Hardware components

Software apps and online services

Story

Introduction

Scope

How Project Work

RT-Thread Vision Board

Vision Board and MicroPython

Control GPIO pins in MicroPython

Machine Learning Pipeline

Build Dataset

Train the ML Model

Test the Model (Optional)

Deploy the ML Model

Testing the model on Real Target

Schematics

System Block Diagram

Code

Machine Learning Inferencing

Build Dataset

Define the Pin that Connects the DC_Lock

Add DC_Lock pin to the list of pins we have

Credits

Mohamed Ali Bedair

Comments

Embed the widget on your own site

Smart Lock using Face Recognition

Smart Lock using Face Recognition

Things used in this project

Hardware components

Software apps and online services

Story

Introduction

Scope

How Project Work

RT-Thread Vision Board

Vision Board and MicroPython

Control GPIO pins in MicroPython

Machine Learning Pipeline

Build Dataset

Train the ML Model

Test the Model (Optional)

Deploy the ML Model

Testing the model on Real Target

Schematics

System Block Diagram

Code

Machine Learning Inferencing

Build Dataset

Define the Pin that Connects the DC_Lock

Add DC_Lock pin to the list of pins we have

Credits

Mohamed Ali Bedair

Comments

Related channels and tags