How to Design and Build A Face Tracking Video Camera

In our last blog 3 Edge-AI Metaverse Interaction Solutions for Potential Killer Applications using OpenNCC we introduced an OpenNCC based Voice Tracking & Face Tracking Camera. It is a customization solution for one of our customers, which is a good application device for video meetings and interactive gaming.

In today’s blog, we’d like to share the idea of how to design different working modes on the software side to provide an efficient edge-ai computing camera and apply a useful strategy to provide a stable and smooth user experience. In the end, we will introduce how to build this kind of product with a face tracking feature based on OpenNCC’s computing camera module/AI camera module and its 4K sensor module.

Here we go.

Key Features in Software Designing:

• 1 Face Library
• 3 Modes: Scanning Mode, Tracking Mode, and Stand-by Mode
• 3 Algorithms: Face Detection, Face Matching, and Face Tracking
• 1 Stand-by Timeout Mechanism

Full Progress Design Diagram:

Details about each Function:

1. Face Library

Before running the device, save a Targeted Face image to the Face Library. Currently, OpenNCC only supports tracking one face in one OpenNCC Device.

2. Scanning Mode

When powered on, the OpenNCC Device enters Scanning Mode by default.

In Scanning Mode, every scanned frame will be scaled from 4K resolution(the original sensor resolution) to 1080P for full FOV(Field of View) preview. In the preview frame, a Face Detection Algorithm is running to detect whether there are any faces in the frame. Once a face is detected, it will be matched with the one saved in the Face Library.

3. Tracking Mode

Once the face is matched successfully, the OpenNCC Device enters Tracking Mode.

In Tracking Mode, a Tracking Area will be displayed on the full screen, which is 1080P resolution that cut from the original 4k based on the Tracked Face as a center. The original full FOV frames will be scaled down and superimposed on the lower right corner of the screen. This strategy features the Tracked Face to give the audience the best preview experience.

Meanwhile, in Tracking Mode, the Tracking Algorithm keeps calculating whether the Tracked Face is still in the Tracking Area in real-time.

4. Stand-by Mode

Once the Tracked Face leaves the Tracking Area, the OpenNCC Device enters Stand-by Mode, and Stand-by Timeout Mechanism is activated.

In Stand-by Mode, the Tracking Area and the full FOV will be displayed the same as they are in Tracking Mode. Meanwhile, the Face Detection Algorithm and Face Matching Algorithm will be restarted to search faces in full FOV and match them with the Targeted Face saved in the Face Library in real-time. According to the rules set by Stand-by Timeout Mechanism, there are three different cases that happen in this Mode:

If Time ≤ Time Limitation

1>. If the Tracked Face is found, matched, and returns back to the Tracking Area, then the OpenNCC Device returns to Tracking Mode;

2>. If the Tracked Face is found and matched, but it does not return to the Tracking Area, recenter the Targeted Face and modify the Tracking Area to fit it, then the OpenNCC Device returns to Tracking Mode.

If Time ＞ Time Limitation

3>. OpenNCC Device enters Scanning Mode.

Here Time Limitation is a set interval, for example, 3 seconds. The intent of the Stand-by Timeout Mechanism is to prevent frequent screen-switch when Tracked Face is at the edge of the Tracking Area, thus providing a relatively stable preview experience. This strategy is helpful especially when the Tracked Face moves back and forth at the edges of the Tracking Area.

Hardware Solution: As mentioned in the beginning, this is a customization case for one of our OpenNCC customers. On our side, we mainly focused on the camera parts. To make it AI-on-Device, all three algorithms are deployed inside the OpenNCC computing board which is based on Intel Movidius VPU Myriad X. For the sensor module, we use an 8MP module that can provide high-quality video images.

Readers who have known OpenNCC for a while know that OpenNCC is open-source and you can find its source code on our GitHub repository. Though firmware for this product is not open-source as it is a customization project, developers can still develop similar features based on OpenNCC hardware components, open-source we offered on GitHub, and free AI models offered on Intel OpenVINO Model Zoo.

Recommended SKU to try out this application: OpenNCC UB4.