Virtual Gamepad with Vision AI

1.Project Plan

For people with limited mobility, it is difficult to perform certain actions such as controlling a game joystick, which requires flexible fingers, or using a VR controller, which requires flexible limbs. Control is a problem that needs to be addressed for people with limited mobility to participate in sports.
For the game control scenario, combined with the practical application of current machine vision technology, a solution is proposed: using a camera to collect head motion images, obtaining the head position in real time through an intelligent facial recognition model, analyzing and judging the motion, and interpreting the head position into keystrokes such as up, down, left, and right. The recognition results are used to control the gamepad through Bluetooth virtual handle technology.
In this project, image signals are collected using a camera, the Grove Vision AI 2 loads the Face Detection model, and action discrimination is performed through the ESP32 module. Control information is output according to standards through the Bluetooth module interface of the Esp32.

2.Part Sourcing

1）Grove Vision AI 2

It is an MCU-based vision AI module powered by Arm Cortex-M55 & Ethos-U55. It supports TensorFlow and PyTorch frameworks and is compatible with Arduino IDE. With the SenseCraft AI algorithm platform, trained ML models can be deployed to the sensor without the need for coding. It features a standard CSI interface, an onboard digital microphone and an SD card slot, making it highly suitable for various embedded AI vision projects.

2) OV5647 Fov Camera

This camera module benefits from Fisheye Lens to achieve 62 Field of View, utilizing the OV5647 sensor to reach 2592 x 1944 active array size image show, supporting Raspberry Pi 3B+4B.

3）ESP32 module

ESP-32S Wifi Bluetooth combo module is ultra high performance and ultra low-power consumption Wi-Fi and Bluetooth combo wireless platform based on ESPRESSIF ESP32 chipset. ESP-32S integrates dual-core processor, 448 KByte ROM,520 KByte SRAM,16 KByte SRAM in RTC, 802.11 b/g/n/e/I Wi-Fi, Bluetooth v4.2 BR/EDR & BLE, clocks & Times, abundant peripheral Interfaces and sercurity mechanism.

3.Building

1）step1: Select Model and Deploy

Luckly, we can find many models trained in SenseCraft AI Model Assistant. Face Detection is a Swift-YOLO model trained on the face detection dataset. This face detection can be used on Grove Vision AI for recognizing faces in images and is useful in a variety of scenarios, including areas such as security and surveillance, smart homes, business analytics, public transportation, and image processing applications. The model enables the detection, recognition and counting of faces to support security management, smart device operation, business operation optimization and image processing.

Connect Vision AI 2 board to SenseCraft AI Model website and download the model.

2) step2:Connect Analysis module

Connect the Grove Vision AI (WE2) module to the default I2C interface of your Arduino board( Esp32 module) using the 4-Pin Cable. Make sure each wire is connected to the correct pin.

• SCL -> SCL (Grove Vision AI WE2)
• SDA -> SDA (Grove Vision AI WE2)
• VCC -> VCC (Grove Vision AI WE2, 3.3V)
• GND -> GND (Grove Vision AI WE2)

esp32 module will receive Grove Vision 2 board message through I2C protocol， then analysis face movement and map to direction.

4.Code and Algorithm

1）Receive Grove Vision2 message， Parse，Analysis and send through bluetooth.

First, install depended library:

• Seeed_Arduino_SSCMA v1.0.0
• ArduinoJson v7.1.0
• ESP32-BLE-Gamepad v0.5.5

As first time setup gamepad, we should caliberate the center point of gamepad, so first time enter loop will get center x,y values, then if the face box's x exceed center x beyond value theta 10, we will signal DPAD_RIGHT. we will compare the box[0] position with center point to determine the DPAD direction. Each loop delay 0.5 seconds to refresh gamepad status.

#include <Arduino.h>
#include <Seeed_Arduino_SSCMA.h>
#include <BleGamepad.h>
BleGamepad bleGamepad;
SSCMA AI;
int CenterX = 110;
int CenterY = 100;
int theta = 10;
bool first = true;
void setup()
{
    Serial.begin(9600);
    Serial.println("Starting BLE gamepad App!");
    //initial and setup Grove Vision 2
    AI.begin();
    //initial gamepad
    bleGamepad.begin();
}
void loop()
{
    // invoke once, no filter , get image
    if (!AI.invoke(1, false, true))
    {
        //first time, caliberate the center point
        if(first && AI.boxes().size()>1)
        {
            CenterX = AI.boxes()[0].x;
            CenterY = AI.boxes()[0].x;
            first = false;
        }
        //filter multiface detect, only select first.
        if (AI.boxes().size()>1)
        {
            if(AI.boxes()[0].x < CenterX - theta)
            {
                bleGamepad.setHat1(DPAD_LEFT);
            }
            if( AI.boxes()[0].x > CenterX + theta)
            {
                bleGamepad.setHat1(DPAD_RIGHT);
            }
            if( AI.boxes()[0].y > CenterY + theta)
            {
                bleGamepad.setHat1(DPAD_DOWN);
            }
            if( AI.boxes()[0].y < CenterY - theta)
            {
                bleGamepad.setHat1(DPAD_UP);
            }
        }
    }
    //interval 0.5 seconds.
    delay(500);
}