Inclusive Gaming - Gaming with head movements and mic input

Introduction

Hey! Im an 18 year old student from Latvia. I made this project for the AMD Pervasive AI Developer Contest. Here's the original idea proposal :

What problem are you going to solve?

Many individuals with physical disabilities, particularly those who lack full control of their hands, face significant limitations in accessing and enjoying gaming experiences. Existing solutions often require specialized hardware or complex setups, hindering accessibility and affordability.

What are you going to build to solve this problem?

I propose an innovative AI-powered system that utilizes webcam input to recognize and interpret head and face movements, translating these gestures into game actions. This solution will empower disabled gamers to experience the joy of gaming without compromising their mobility or financial resources.

So what did I come up with?

I had quite a bit of errors, but at the end I managed to make an working prototype.

I also made an simple interface so that the user can easily interact with the code and understand its functioning. When you run the program, it shows this welcome screen. Here you need to choose your main webcamera port. Usually it's the first one. Once selected, you have to press "Next".

That will bring you to this screen, where you need to choose one of theese 3 preset games. You can choose to play either Tetris (With the webpage "Tetr.io" or desktop app), Among Us or Rocket League.

Once selected, you will get this instruction window. It will explain, how to control the preset game - how to use the voice commands and head tracking. When you press "Next", you will hear an TTS voice that will confirm the start of the countdown and then you wi have 10 secconds to open the game and get ready! Once the timer has ended there will be an gentle sound and the algorithm will start.

You can also change the lenght of the timer manually, by changing the number for the variable on the line 19. It will automatically update the countdown and the instructions.

Before running the code

Before running the Unc_Main.py file, you have to install python and then install few libraries, by opening cmd and running (each line seperatly) :

python -m pip install pysimplegui
pip install opencv-python
pip install teachable-machine
pip install pygame
pip install pynput
pip install pyttsx3
pip install pyaudio speech_recognition

First time, when running the app, you will be ask to make an account for pysimplegui. Make it and log in. After that run the app again and it will start, as expected.

How is the amd W7900 pro used?

The amd w7900 pro boosts the ai performance, because the code uses the amd ROCm software. It makes the AI model able to run loclay, on the gpu, making it run smoothly. By using the AMD W7900 pro gpu, you will also get an excellent gaming quality, so that you can enjoy gameplay, while the code is running in the background.

Thank you AMD for sending me the W7900 pro GPU. This is just the beggining. I already have a lot of project ideas to make with this gpu! :)