Game Control using Hand Gesture Recognition

Project Description

In this project, we’ll create a simple game and control the player movements using gesture detection. Two gestures are taken into consideration which are used to control the player movements and are captured by the gesture detection system. A player moves the basket by using gestures to capture randomly generated apples on the screen. Each capture earns 1 point.We use Socket programming to send the data from the detection code to the Wiznet EVB PICO using ethernet. The game is a time constrained one and once the set time is elapsed the game will stop the detection automatically and display the high score. The gesture detection and data transmission is done by opencv and python.

Things used in this project

Hardware

• Wiznet EVB PICO W5100S
• Jumper wires
• SSD1306 OLED Display
• Bread Board
• Web Cam

Software

• Arduino IDE
• Python
• Libraries Included Arduino : Adafruit Graphics Library, Ethernet and EthernetUDP Library
• Libraries Included Python : Sockets Library, OpenCV, Mediapipe

Story

The first thing that comes to our mind listening to the words Game Control is the Joystick. By integrating AI as a control system for the gaming environment we can enable immersive gaming without the requirement for electronics. In this project We have made an AI based controller for a 2D game. This enables fine tuning of the movements. AI is used to capture gestures and use these gestures to control the movement of the player. All the game screens and score board are displayed using an SSD1306 OLED display.

Methodology

The webcam will continually capture frames, it will then be analyzed for the presence of gesture and each gesture is numbered. We consider two gestures and this gesture will essentially move the controllable player gestures to and fro the Y axis.A time is also set when the capturing starts. Using gesture controls a player will be able to catch the randomly generated apples on screen. If a collision is detected between basket and apple a score is incremented. Once the time out reaches 60 seconds a “STOP” control signal is sent and the capture is stopped, displaying the “Game Over” screen and the Score is displayed.

FlowExplanation

The flow starts with the video capture. Each frame is analyzed for gestures using hand landmarks. If a gesture is detected it is analyzed to see if it's a “HI” gesture or a “FIST” gesture. All the gestures are given a particular number and this number is transmitted via socket connection. If “HI” gesture number 5 is transmitted, a closed “FIST” gesture number 0 is transmitted. The transmission is done in real time. On receiving the coordinates in the Wiznet EVB PICO W5100s, it checks for the presence of 5 or 0. If 5 is received the player basket is moved downward and if 0 is received downwards. The game is displayed on an OLED SSD1306 display.

Circuit diagram Explanation

VCC, GND[Pin 36, Pin 38] are connected to the VCC and GRD of the SSD1306 OLED Display. The SDA pin of the OLED Display is connected to the GPIO4 pin [Pin 6] which acts as the data pin. The SCL pin of the OLED Display is connected to the GPIO5 pin [Pin 7] which acts as the clock pin. A USB is attached for powering the Wiznet EVB PICO W5100S and the OLED Display. An Ethernet is attached to establish socket Ethernet UDP Communication. The display used is an i2c display which uses 4 pins and will operate in 3v range.SDA pin is used for data and SCL is the clock signal to synchronize the display.

How to integrate AI ?

In this project OpenCV and Mediapipe are used for gesture detection. We use gestures in this project for the player basket movements. The gesture detection code will identify the landmarks of each finger and analyze the position of fingers. The “HI'' gesture will move the player basket along the Y axis downward and “CLOSED FIST” will move the player baskets along the Y downwards. The communication between the Wiznet EVB PICO W5100s board and the AI is established by socket UDP connection. On detecting the gestures, each gesture is given a certain number and this number is passed to the socket connection. The game stop control signal “STOP” is also passed through this socket connection.

Arduino Code Explanation

All the libraries and variables initialize in line no [13-36]. This initializes the ethernet libraries, the SSD1306 display and the socket binding address and the port. Line number [41-63]. Start the ethernet connection and initialize the screen with zeros. The ethstart() function enables the ethernet port and sets the IP address variable. This will be available for 10000 seconds / 10 seconds. This IP address has to be copied and typed to the python console.Void loop starts at line number [66-75] which repeatedly calls the main_menu() function and updates the display continuously presenting the changes to the screen. The main_menu() function line number [202-214] which has 5 functions. read_input() function line number [98-121] will read the control signals via signals and set the current_input variable. After the read_input() function call action() function is called. The will sets the player’s baskets Y coordinate essentially moving it up and down to catch the incoming apples. The movement is done in real time. move_basket function will display the paddles as per the coordinates. enemy_fire() function line number [163] will fire the apples randomly from a y coordinate. Speed of the apples is increased as it moves along the X coordinate. When a collation happens when the baskets X and the apples X coordinate happens, a score is added and will also display a “HIT” message on the screen. Once it reaches the set X coordinate. The bullet will reset its position to a random Y coordinate. In line number [110] a control signal condition is enabled for the Game Over screen. This is done by receiving the “STOP” signal after 1 minute of gameplay. This will trigger the activation of the game_over() function line number [81]. Which will print a “Game Over” screen along with the score.

The Gameplay

The game is a single player game in which the player has to capture as many apples as they can in under 60 seconds. The apples are randomly generated anywhere on the screen and will move fast every millisecond. The player has to move the basket by using gestures. Each capture of apples will reward 1 score. The basket can be moved only up or down. The player can use the “CLOSED FIST” Gesture to move the basket down and “HI” gesture to move the basket up. By the end of 60 seconds the capture will stop and a game over screen will appear projecting the score.

Youtube:

References

● Arduino I2C SSD1306 OLED screen tutorial - The EECS Blog

● Arduino with Python LESSON 17: Transferring Data over Ethernet UDP - YouTube