kennedy saine Banda
Published © Apache-2.0

FaceController

Reimagine gaming as we know it. Forget clunky controllers and static interfaces. Your face is now the ultimate game controller.

AdvancedShowcase (no instructions)5 hours317
FaceController

Things used in this project

Hardware components

web camera
It identifies and tracks 468 distinct points (landmarks) on a person's face in real-time, creating a highly detailed 3D mesh model.
×1
Internet Computer
Real-time FaceMesh calculations: Running FaceMesh requires significant processing power, especially if capturing high-resolution video or tracking multiple faces simultaneously. This explains why powerful computers are often used for demanding applications like AR, animation, and research.
×1

Software apps and online services

OpenCV
OpenCV – Open Source Computer Vision Library OpenCV
OpenCV (Open Source Computer Vision Library) is a powerful and versatile library for real-time computer vision and image processing. It's open-source, freely available,
cvzone
This is a Python package called which simplifies image processing and AI functionalities using libraries like OpenCV and Mediapipe. It provides modules for common computer vision tasks like: Finding contours Measuring distances and angles Tracking objects Drawing shapes and text on images Applying filters and effects
ctypes
Ctypes is a foreign function library for Python. It helps Python programs interact with code written in other languages, primarily C and C++.
off road racing game
A specific video game called "Off-Road Super Racing" is a 3D truck racing game released in 2010, featuring championship mode and single races with customizable trucks and realistic physics....but you can use any game for testing purposes

Story

Read more

Code

FaceMesh Tracking

Python
The script is designed to interact with the user's face through a webcam feed, particularly focusing on facial landmarks like the lips and head tilt.
from cvzone.FaceMeshModule import FaceMeshDetector
import cv2 as cv 
import keyinput
import cvzone

# Initialize FaceMeshDetector
detector = FaceMeshDetector()

# Open the webcam
cap = cv.VideoCapture(0)

# Main loop to capture video frames
while True:
    # Read a frame from the webcam
    success, frame = cap.read()

    # Check if the frame was successfully captured
    if not success:
        break

    # Detect face and get face mesh in the frame
    img, faces = detector.findFaceMesh(frame, True)

    # Check if faces are detected
    if faces:
        for face in faces:
            # Get specific points for the lips
            upperLip = face[13]
            lowerLip = face[14]

            # Calculate the vertical distance between upper and lower lip
            leftEyeVerticalDistance, info = detector.findDistance(upperLip, lowerLip)

            # Debugging: print the vertical distance
            # print(leftEyeVerticalDistance)

            # Perform actions based on the vertical distance
            if leftEyeVerticalDistance > 15:
                # Accelerate
                cvzone.putTextRect(frame, "accelerate", (15, 450))
                keyinput.release_key('s')
                keyinput.release_key('a')
                keyinput.release_key('d')
                keyinput.press_key('w')
            else:
                # Brake
                cvzone.putTextRect(frame, "braking", (15, 450))
                keyinput.release_key('a')
                keyinput.release_key('d')
                keyinput.release_key('w')
                keyinput.press_key('s')

            # Check for head tilt to determine steering direction
            if face[94][0] > 300:
                cvzone.putTextRect(frame, "Turn Left", (350, 450))
                keyinput.release_key('s')
                keyinput.release_key('d')
                keyinput.press_key('a')
            else:
                cvzone.putTextRect(frame, "Turn Right", (350, 450))
                keyinput.release_key('s')
                keyinput.release_key('a')
                keyinput.press_key('d')

    # Display the processed frame
    cv.imshow("steering", frame)

    # Check for the 'q' key to quit the application
    if cv.waitKey(1) & 0xff == ord("q"):
        break  # Exit the loop and close the application

ctypes

Python
The provided Python code uses the ctypes library to interact with the Windows API (user32.dll) and simulate keyboard input. The purpose of this code is to define functions (press_key and release_key) that can be used to programmatically simulate the pressing and releasing of keys.
import ctypes

# Virtual key codes for specific keys
keys = {
    "w": 0x11,
    "a": 0x1E,
    "s": 0x1F,
    "d": 0x20,
}

# Pointer to unsigned long for extra information
PUL = ctypes.POINTER(ctypes.c_ulong)

# Structure for keyboard input
class KeyBdInput(ctypes.Structure):
    _fields_ = [("wVk", ctypes.c_ushort),
                ("wScan", ctypes.c_ushort),
                ("dwFlags", ctypes.c_ulong),
                ("time", ctypes.c_ulong),
                ("dwExtraInfo", PUL)]

# Structure for hardware input
class HardwareInput(ctypes.Structure):
    _fields_ = [("uMsg", ctypes.c_ulong),
                ("wParamL", ctypes.c_short),
                ("wParamH", ctypes.c_ushort)]

# Structure for mouse input
class MouseInput(ctypes.Structure):
    _fields_ = [("dx", ctypes.c_long),
                ("dy", ctypes.c_long),
                ("mouseData", ctypes.c_ulong),
                ("dwFlags", ctypes.c_ulong),
                ("time", ctypes.c_ulong),
                ("dwExtraInfo", PUL)]

# Union of input types
class Input_I(ctypes.Union):
    _fields_ = [("ki", KeyBdInput),
                ("mi", MouseInput),
                ("hi", HardwareInput)]

# Structure for generic input
class Input(ctypes.Structure):
    _fields_ = [("type", ctypes.c_ulong),
                ("ii", Input_I)]

# Function to simulate pressing a key
def press_key(key):
    extra = ctypes.c_ulong(0)
    ii_ = Input_I()
    ii_.ki = KeyBdInput(0, keys[key], 0x0008, 0, ctypes.pointer(extra))
    x = Input(ctypes.c_ulong(1), ii_)
    ctypes.windll.user32.SendInput(1, ctypes.pointer(x), ctypes.sizeof(x))

# Function to simulate releasing a key
def release_key(key):
    extra = ctypes.c_ulong(0)
    ii_ = Input_I()
    # 0x0008: KEYEVENTF_SCANCODE (Flag to specify the key is a hardware scan code)
    # 0x0002: KEYEVENTF_KEYUP (Flag to specify key release)
    ii_.ki = KeyBdInput(0, keys[key], 0x0008 | 0x0002, 0, ctypes.pointer(extra))
    x = Input(ctypes.c_ulong(1), ii_)
    ctypes.windll.user32.SendInput(1, ctypes.pointer(x), ctypes.sizeof(x))

Credits

kennedy saine Banda
5 projects • 8 followers
Innovator blending AI, ML, and embedded systems to craft smart solutions. Expertise in biomedical engineering, CV, welding & IoT.
Contact
Thanks to Shahariar.

Comments

Please log in or sign up to comment.