Smart Door Opener with 3D Facial Authentication

exec summary

With the introduction of the iPhone X in September 2017, Apple introduced authentication with biometric facial features as a reliable technology in our everyday lives. With this project, we want to bring this process as a door opener from the smartphone to the smart home. Suitable for one or two-family houses.

checklist

• time required: a weekend
• cost: 500 USD plus IP interface for the door intercom

material required

• Intel RealSense ID F455 camera
• Raspberry PI 4B >= 4 GB RAM
• Geeek Pi Raspberry Pi 4 Armor Case
• 16 GB or higher microSD memory card e.g. Samsung EVO Select 64 GB microSD
• Meanwell IRM-60-5ST 5V 10A power supply
• sensor exterior wall light Severina by Lindby as casing
• RGB LED 64x32 P2, 5 matrix module 160mm x 80mm e.g. Adafruit 5036
• E18-D80NK IR retro-reflective sensor + 10 KΩ pull-up resistor
• 40-pin GPIO ribbon cable or 10x male/female jumper cables, 20x female/female jumper cables
• M3 brass spacer bolts 4x 40mm, 4x15mm, 2x 10mm
• 8x M3 nuts for fastening the spacer bolts to the rear panel of the housing
• 4x M2.5 screws 12mm for attaching the Raspi to the rear panel of the housing
• ¼ inch screw for attaching the camera on the bottom of the case
• Cable grommet for the USB cable on the underside of the housing
• Shrink tubing to insulate the low-voltage power terminal
• an IP-based interface for the front door buzzer, e.g. a Siedle gateway (German brand house intercom)
• Optional: Adafruit RGB Matrix Bonnet with 40 Pin Pitch Stacking Header - to increase the distance of the matrix bonnet to the board so that it protrudes over the armor case
• Optional: PIR sensor HC-SR 501 as an alternative presence detector and replacement for the E18-D80NK retro-reflective sensor

required tools

• drill, screwdriver, soldering iron

Next big thing

When Apple introduced the iPhone X in September 2017, replacing the fingerprint sensor with Face ID was the "next big thing". A "dot projector" projects thousands of invisible dots onto the user's face and uses an infrared camera to create a three-dimensional point cloud from this pattern, which is then compared with the saved profile using AI inference.

📷 Advertising motif RealSense ID (image courtesy Intel)

Early 2021, Intel picked up this principle with the announcement of the RealSense ID F455 camera and offers makers an interesting product for integration into self-made solutions. The camera scores with extensive documentation, an open source SDK for Linux, Windows and Android as well as the languages C, C++, C# and Python. In contrast to Apple, it allows the authentication of several people, whose profiles are either stored centrally on a server or in a database on the camera itself. The "false acceptance rate" - i.e. the number of people who were wrongly admitted - is given by Intel as well as by Apple at 1:1 million. The entire authentication process of the camera takes less than a second. The RealSense ID F455 is therefore the perfect candidate for opening our front door via 3D face recognition with a low error rate and in a short time.

Functionality

The smart door opener has replaced the outdated entrance lighting and is therefore housed in the casing of one. The system is only supplied with main voltage and communicates via the in-house WIFI. It must therefore be ensured that the location of the admission control has an appropriate WIFI reception.

Since the camera itself does not have a presence detector, it is triggered by an infrared reflex light barrier. The person asking for admission wipes their hand past the light barrier or comes a little closer with their face. A faint red glow from the infrared illuminator of the upside-down camera at the bottom of the case can be seen briefly. The camera now projects invisible infrared points with a wavelength of 850 nm onto the face, records them with two full HD cameras integrated on the side and uses them to form a three-dimensional point cloud, which it compares with the stored face profiles via AI inference. The camera needs less than a second for this authentication process and transfers the result - in the positive case the name of the recognized person - to the Raspberry Pi computer via USB cable. Since we only use one camera, these profiles are stored locally on the camera. For authentication with multiple accesses, the camera offers a server mode in which the access profiles are stored on a central server and can thus be used by multiple cameras. If the authentication is positive, the name of the recognized person is shown on the LED matrix display for a few seconds and the door is opened via MQTT command via WIFI. When idle, the LED matrix panel shows the time, the day of the week and the current date. You can find a demonstration of how it works via YouTube video here:

Hardware Setup

A Raspberry 4B with Raspberry PI OS aka Raspian is used as the host computer. We decided on an outdoor light in a stainless steel housing for the housing, which gives off an inconspicuous image when placed in front of the front door. In addition to the Raspi, it contains a 5V power supply with 50 watts of power, a retro-reflective sensor and an Adafruit 5036 LED matrix display with a resolution of 64x32 RGB LEDs. This is attached to the base plate of the housing using 4 spacer bolts, each with an added length of 55 mm (40 mm + 15 mm). The matrix module has a thickness of 15 mm, resulting in a distance of 70 mm between the base plate and the frosted acrylic glass. To ensure that the LED matrix content does not become too blurred due to the satin finish, it is important that the LED matrix module is in direct contact with the acrylic glass. Before the structure is integrated into the dismantled lamp housing, the components are wired as a prototype and the functionality of the individual components and the system is tested.

Wiring

There are 2 options for connecting the RGB matrix display to the Raspberry PI:

• Adafruit RGB Matrix Hat/Bonnet: This plug-in board offers tidy cabling via the HUB75 connector, which is included with the LED matrix module. In order to avoid that the plug-in board does not get in the way of the Raspi Armor Case, it must be raised using a 40-pin pitch stacking header. This creates the disadvantage of a higher design of approx. 6 cm. Since the Raspi is installed in the lower part of the housing opposite the sensor, this is OK for using the PIR sensor HC-SR 501, but leads to a collision when assembling the E18-D80NK IR light barrier with a housing length of 5.5 cm of the lamp housing. Therefore, when using the E18-D80NK light barrier, we recommend either the discrete wiring shown below or swapping places - power supply unit in the lower part of the housing, Raspi in the upper part! The Raspi is powered via the jack plug or the screw terminals on the Adafruit Bonnet. In addition, the Adafruit Bonnet uses other GPIO Ports for communication, so in Adafruit Bonnet case we use GPIO 19 for the presence sensor. A jumper cable for the sensor output must therefore be soldered onto the bonnet.

Discrete wiring with 40-pin GPIO ribbon cable or female-to-female jumper cable

The discrete wiring using jumper or ribbon cables is a little more fiddly but just as functional. The presence sensor (PIR or photoelectric sensor) is used in the case of discrete wiring to connect to GPIO Pin 5. The Meanwell power supply has a 4-pin screw terminal at the low-voltage output, which supplies power to both the RGB matrix via the power cable included in the scope of delivery and the Raspi via GPIO pin 2 for 5V and GPIO pin 39 for GND. We do not recommend using the jumper cables for powering the Raspi, but rather using a thicker cable with higher cross-section and snappier terminals. To do this, we rededicated 2 redundant cable clamps from the LED Matrix power supply, pulled them out with a small screwdriver and covered them with shrink tubing so that there is no risk of short circuits on GPIO pin 2.

Repurposing of two 5V power cables from the LED matrix module to supply the Raspis

The instructions for the discrete wiring of the LED matrix module are shown in detail here: https://github.com/hzeller/rpi-rgb-led-matrix/blob/master/wiring.md

The camera is attached upside down on the underside of the housing. A USB-C cable, which is included with the camera, is fed into the housing through a hole to be drilled behind the camera and plugged into one of the 4 USB-A ports on the Raspberry Pi.

Presence sensor

The frontal housing hole, which was reserved for the lens of the PIR sensor, is filled with the reflex infrared light barrier by screwing it on the inside and outside with the two enclosed plastic nuts. The sensor is supplied with a voltage of 5V via the matrix bonnet. The brown cable is routed to the terminal labeled "5V Out" - if the Adafruit Matrix Bonnet is used.

With discrete wiring, there are several ways to obtain 5V voltage from the GPIO pins. E.g. +5V on pin 4 and GND - blue cable - on pin 34. The sensor output - black cable - is connected to pin 29 or logical GPIO port 5.

If you don't have a wall opposite or a similar object that reflects infrared light, you can use a PIR sensor, e.g. the HC-SR 501, instead of the light barrier. In our case, this registered many false positive events and thus frequently triggered the camera, since the LED matrix module flickers invisibly to the human eye and irritated the PIR sensor through an opposite wall.

Case installation

The housing of the Severina by Lindby sensor-controlled outdoor wall light is made for the RGB LED matrix with dimensions of 160 mm x 80 mm and offers just enough space to house all components. To mount it, you have to drill a few holes in the rear panel.

The camera is screwed to the lower stainless steel sheet from the inside of the housing with a ¼ inch screw. In order to prevent the valuable camera from simply twisting off, it is secured with two spacer bolts of 10 mm in length.

Software installation

For software installation we assume a freshly installed Raspberry Pi OS aka Raspian, which is connected to the Internet via the in-house WIFI. We also recommend turning on the ssh and vnc services for remote access via the Raspberry Pi system configuration. Instructions for installing Raspian can be found here, for example (German source): https://www.heise.de/tipps-tricks/Raspberry-Pi-einrichten-so-klappt-s-4169757.html

The program "dooropenerF455" is based on two software modules: First, the RealSense ID SDK from Intel for controlling the 3D camera and second, the RPI LED Matrix Module from Henner Zeller. We use an MQTT interface to open the door and therefore install Mosquitto as an MQTT broker:

# installation of smartdoorF455
# bring Raspi up2date and get additional software developement tools
sudo apt update -y
sudo apt upgrade -y
# install essential development tooling
sudo apt install cmake build-essential -y
# mosquitto is the MQTT broker
sudo apt install mosquitto mosquitto-dev libmosquittopp-dev mosquitto-clients -y
# libgraphicsmagick++-dev is required to build rpi-rgb-led-matrix utilities
sudo apt install libgraphicsmagick++-dev libwebp-dev -y
# you may want to encrypt network communication
sudo apt install libssl-dev libcurl4-openssl-dev -y
# let's build the RealSense ID SDK by Intel
cd ~
git clone https://github.com/IntelRealSense/RealSenseID.git
cd RealSenseID
mkdir build
cd build
cmake .. 
make -j4
# let's build the rpi-rgb-led-matrix library by Henner Zeller
cd ~
git clone https://github.com/hzeller/rpi-rgb-led-matrix.git
cd rpi-rgb-led-matrix/
make -C examples-api-use
# let's build the smartdoorF455 application
cd ~
git clone https://github.com/joergwall/smartdoorF455.git
cd smartdoorF455
mkdir build
cd build
cmake ..
make -j4

The compiled C++ binary executable "smartdoorF455" or C pendant "smartdoorF455_c" should now exist in the ~/smartdoorF455/bin directory. The source codes can be found in ~/smartdoorF455/cpp or ~/smartdoorF455/c. The program is started via shell script with sudo:

cd ~/smartdoorF455/bin
sudo ./run_smartdoorF455.sh

or who prefers the variant in programming language C:

sudo ./run_smartdoorF455_c.sh

and provides the following output depending on the user name and time/date:

/home/pi/log created
smartdoorF455 started successfully
watch /home/pi/log/20211216_092446_smartdoorF455.log for errors

Even if the camera has not yet rehearsed a face, the time, the day of the week and the date should now be visible. If this is not the case, please go to the chapter Troubleshooting. To abort the program, please use the martial command:

sudo killall smartdoorF455

Teach faces for authentication

In order to bring the face of authorized users into the camera, we use a tool with a command line interface. If the device /dev/ttyACM0 is missing, use /dev/ttyACM1 instead. The parameters currently stored in the camera and a selection menu now appear. The rotation parameter can be set to 0 in the "s" menu or upside down to 180 depending on whether the camera is positioned upside down - i.e. depending on whether the camera is screwed upside down on the housing or upright, e.g. on the included mini tripod. The menu item "e" offers training with local profile storage on the camera. The face should be about 30 to 50 cm away from the camera. The procedure then looks like this:

# enroll user for authentication
cd ~/RealsenseID/build/bin
sudo ./rsid-cli /dev/ttyACM0
Connected to device

Authentication settings::
 * Rotation: 0 Degrees
 * Security: High
 * Algo flow Mode: All
 * Face policy : Single
 * Dump Mode: CroppedFace
 * Matcher Confidence Level : High
Please select an option:

  'e' to enroll.
  'a' to authenticate.
  'd' to delete all users.
  'c' to capture images from device.
  's' to set authentication settings.
  'g' to query authentication settings.
  'u' to query ids of users.
  'n' to query number of users.
  'b' to save device's database before standby.
  'v' to view additional information.
  'x' to ping the device.
  'q' to quit.
  
server mode options:
  'E' to enroll with faceprints.
  'A' to authenticate with faceprints.
  'U' to list enrolled users
  'D' to delete all users.
> e
User id to enroll: Julia
Connected to device
  *** Hint Success
  *** Hint Success

This way the facial profiles of all authorized persons are learned. When assigning a name, note that the RGB LED matrix module can only display a small number of letters. With the font we use, these are only 5 characters. If necessary, please use an abbreviation or reduce the size of the BDF font for the abbreviation in line 99 of the source code, so that up to 8 characters can be displayed in one lin

#define FONT_NAME FONT_PATH "4x6.bdf"

Sourcecode

The source code under ~/smartdoorF455/cpp or ~/smartdoorF455/c must be adapted to the circumstances. Is MQTT used or is an Adafruit Bonnet used for wiring the LED RGB matrix module? This is to be adjusted in the source code using appropriate #define constant definitions. If, for example, the Adafruit Bonnet is used, GPIO 19 is used instead of GPIO 5 for the infrared reflex light barrier, since the RGB matrix is controlled via other GPIO pins:

/* START SYSTEM CONFIGURATION SECTION */
#define MOSQUITTO_IN_USE /* comment this line, if MQTT is not used to open door */
#define ADAFRUIT_BONNET_IN_USE /* comment this line, if you have direct cable wiring from Raspi to LED Matrix */
#define STDOUT_ADDTL_INFO  /* provides additional information on stdout e.g. prints date/time when movement sensor triggers camera */
/* END   SYSTEM CONFIGURATION SECTION */

Open Sesame

In order to be able to unlock the front door with the solution presented here, the door intercom must provide an IP interface. If you want to tackle a replica of this project, you should first analyze the current status of the bell system in detail and try to find ways to add an IP interface.

We need to cope with a Siedle bus-based door intercom and therefore use a gateway from Oskar Neumann, which translates MQTT commands into the Siedle bus via WIFI, however this is no longer available on the market. Siedle offers the Smart Gateway SG-150 as an IP interface, which, however, at more than 600 USD for self-construction projects represents a hefty chunk. Other alternatives for a Siedle IP interface that we have not checked are third-party gateways such as from Hamares or the smart door opener from SMS Guard, which also provides an MQTT interface.

Anyone who is already using an IP-based door intercom system is off the hook and can adapt the source code between the C/C++ code marked "TRIGGER DOOR OPENER START" and "TRIGGER DOOR OPENER END" to trigger the door opener. If this trigger is an http request, for example, it can be inserted into the ~/smartdoorF455/c/main.c or ~/smartdoorF455/cpp/main.cpp file using a Linux system call:

/* TRIGGER DOOR OPENER START - ADAPT THIS CODE according to your interface to 
   your door buzzer */
/* exemplary adaptation of the C/C++ code */
/* Assumption: the door opener can be opened with the http request */
/* -------  http://192.168.178.27:8083/fhem?cmd=set%20Siedle%20open  */
/* Make sure the curl module is installed in Raspian: */
/* sudo apt install curl */
/* Since no MQTT protocol is used for the door opener */
/* remove those enclosed between "#ifdef MOSQUITTO_IN_USE" and "#endif" */
/* lines. */

system("curl http://192.168.178.27:8083/fhem?cmd=set%20Siedle%20open");

/* TRIGGER DOOR OPENER END */

Mosquitto MQTT Broker

If MQTT is used for communication with the door opener, the following configuration file must be created for the configuration of the MQTT broker on the Raspi:

sudo nano /etc/mosquitto/conf.d/mymosquitto.conf
# add following lines 
listener 1883
listener 1884
allow_anonymous true

After that, restart the broker‚:

sudo systemctl restart mosquitto

A mosquitto client lets us eavesdrop on the exchanged messages by the broker for debugging purpose

mosquitto_sub -d -t "#"

In c't 6/2018 page 164 (German Source) Jan Mahn explains in detail how MQTT works. In particular, the protection of MQTT via encryption is shown there, which we have omitted here for reasons of complexity. The solution presented here is therefore as secure or insecure as the domestic WIFI. This reading is recommended for all makers if they decide to use MQTT as a protocol.

Camera security

Intel offers a secure communication mode in the RealSense ID SDK. This mode enables the camera to be paired with the host system and encrypted communication. This eliminates the possibility of uninvited guests gaining unauthorized access via another RealSense ID camera - quickly plugged in via USB-C - with other face profiles. For reasons of complexity, we have not used this mode here. Makers can find instructions on how to enable this mode here:

https://github.com/IntelRealSense/RealSenseID#secure-communication

Documentation

In addition to studiying the source code, it is worth taking a closer look at the documentation for the libraries used

https://github.com/IntelRealSense/RealSenseID

https://github.com/hzeller/rpi-rgb-led-matrix

and the underlying project containing this English plus German instructions:

https://github.com/joergwall/smartdoorF455

The data sheet Intel RealSense ID F455 is worth analysing. A good description for E18-D80NK IR Reflexionslichtschranke we've found in the Online Store Roboter-Bausatz.de (German Source). In particular, the setting of the detection distance and the use of the pull-up resistor are explained in more detail here. Information on their alternative with the PIR sensor HC-SR 501 is also available [there](https://www. Roboter-bausatz.de/p/hc-sr501-pir-infrarot-sensor-modul).

Optimization

In order to remove one of the four processor cores from the care of the process scheduler in order to leave it to our application for exclusive use, the following file must be adapted:

sudo nano /boot/cmdline.txt

„isolcpus=3“ append to the end of the line so that it looks something like this:

console=serial0,115200 console=tty1 root=PARTUUID=e0d8ecc0-02 rootfstype=ext4 fsck.repair=yes rootwait quiet splash plymouth.ignore-serial-consoles isolcpus=3

This step takes effect after restarting the computer and is intended to prevent any flickering of the LED matrix display. The processor affinity of our program is assigned to the released CPU at the end of the run_smartdoor_F455.sh start script with the taskset command.

End of Life of Intel RealSenseID F455 camera :-(

Although Intel only announced the RealSense ID product line as a solution for facial biometric authentication in January 2021, it was already discontinued with effect from February 28, 2022. We are currently not aware of an alternative that combines the three-dimensional scan with artificial intelligence for face recognition. So if you want to get hold of such a camera, you have to hurry... This Intel website points out this fact: https://www.intelrealsense.com/facial-authentication/

Troubleshooting

When replicating this smart home door opener consisting of a transformer, LED matrix, Raspi and, if necessary, matrix bonnet, small problems can occur here and there, of which we would like to address the known ones here. First to hardware topics:

• power supply

The Raspi reacts sensitively when the supply voltage drops below the critical value of 4.63 V. The red LED starts flashing or goes completely dark. The Meanwell transformer supplies a stable voltage, but as shown in the Chapter Wiing, care should be taken to ensure that the power supply has a sufficient cable cross-section and non-slip terminals. If in doubt, you should measure the voltage at the GPIO pins 2/4 and 6 with a multimeter and, if necessary, provide amplification, e.g. with an additional cable.

• Distorted image on the LED matrix display

If the LED Matrix Bonnet is used and increased via a 40 pin pitch stacking header, we know of shaky candidates. In particular, a pitch stacking header that came with the armor case caused us problems. Relevant electronics mail order companies offer pitch stacking headers with gold-plated contacts for slim money, which have served us well.

The software also occasionally bitches. First, we check the current log file for informative error messages. To do this, go to the log directory, look for the most recent log file and output it as follows:

cd ~/log
ls -la
more ./20211216_092446_smartdoorF455.log

• „Couldn't load fontfiles“

To eliminate this error, the constant FONT_PATH in the C and C++ code must be adjusted to the absolute path where the bdf fonts can be found. Depending on the user name used, this is e.g.: "/home/pi/pi-rgb-led-matrix/fonts/". Correct the corresponding line in the source code in ~/smartdoorF455/cpp/main.cpp and ~/smartdoorF455/c/main.c and recompile by starting the make command in the build directory:

cd ~/smartdoorF455/build
make

• "on_result: Error"

We wave our hand in front of the presence sensor, but there's no faint red glow from the camera's infrared illuminator - the camera just won't initiate the authentication process. After restarting the Raspberry Pi computer, it sometimes happens that the camera does not initialize correctly when the application is first started. Exit the program, start it again:

sudo killall smartdoorF455
cd ~/smartdoorF455/bin
sudo ./run_smartdoorF455.sh

• "initInitialise: Can't lock /var/run/pigpio.pid cannot initialize GPIO"

Is another process running that is blocking the GPIO? Restarting the program also helps with this error message, as shown above.

• LED matrix display remains dark

No error message in the log file, the LED matrix module is wired correctly, but the display still remains dark? Then the incorrect setting of the ADAFRUIT_BONNET_IN_USE constant could be the cause. For discrete wiring in C and C++ code, comment them out as follows

/* #define ADAFRUIT_BONNET_IN_USE */

We wave our hand in front of the presence sensor, but there is no faint red glow from the camera's infrared illuminator - the camera will wilund recompile (see above).

In our experience, once the software is up and running and provides the service it is intended for, the system is stable for many days and weeks. Even grandmother now leaves her key at home and relies on face control for entry.

Changes and enhancements

• No RGB matrix module

Alternatively, other, simpler structures in other housings are possible: If you want to do without the RGB LED matrix module in the replica, you can use a multicolored RGB light-emitting diode, for example, to display the authentication status. This eliminates the need for the powerful 5V / 10 A transformer and the usual USB-C power supply of the Raspi is used. The project can thus be significantly slimmed down and can be accommodated in smaller housing dimensions.

• Ubuntu instead of Raspberry Pi OS

Raspberry Pi OS is very robust and makes optimal use of the hardware, but Intel's RealSense ID SDK has limited support on it. As an alternative to Raspian, we have successfully tested Ubuntu Linux 20.4. Ubuntu becomes interesting when extended functions of the RealSense ID software are to be used, such as access to screenshots of the camera in order to send them via Telegram Messenger via bot. If you want to follow this path and learn more about the RealSense ID SDK, we recommend flashing a separate SD card for this task with Ubuntu Linux.

• Pixel art to cheer up the neighbors

If you want to play funny animations on the LED matrix display, we recommend the example programs under ~/rpi-rgb-led-matrix/utils. Here you can, for example, conjure up gif animations on the LED matrix display with the following command lines:

# spinning Super Mario 
cd ~/rpi-rgb-led-matrix/utils/
make
wget https://media1.giphy.com/media/QxZEtFE02ofY00gJ71/giphy.gif 
sudo ./led-image-viewer --led-rows=32 --led-cols=64 --led-brightness=90 --led-pixel-mapper "Rotate:270" --led-rgb-sequence=RBG --led-no-hardware-pulse ./giphy.gif

or When wired to the Adafruit RGB LED Matrix Bonnet

sudo ./led-image-viewer --led-rows=32 --led-cols=64 --led-brightness=90 --led-pixel-mapper Rotate:270 --led-rgb-sequence=RBG --led-gpio-mapping=adafruit-hat --led-no-hardware-pulse ./giphy.gif

This article offers a more in-depth look at the LED matrix module options Pixelart mit Pi im Make Magazin 5/2021 by Daniel Bachfeld (German Source).

Conclusion

User authentication with three-dimensional biometric facial recognition is a new field that opens up exciting application scenarios for the maker community. It is a great pity that Intel is burying this new business field just 8 months after the announcement. Nevertheless, we were enthusiastic about this 3D camera and have been doing their job reliably and without complaint for weeks as a door opener (or, to put it correctly, as a door release, because you still have to open the front door yourself). Anyone considering a replica must hurry to get hold of an Intel RealSense ID F455 camera.