Rock-Paper-Scissors Game

Introduction:

Our goal was to create a two-player game where users could play against a computer, but with a twist—replacing the traditional 2D screen with a physical robot. What better game to choose than the classic Rock-Paper-Scissors we all grew up playing? Imagine wanting to play but not having friends around—how cool would it be to face off against a mechanical hand? That’s exactly what we’ve built: a robotic hand that can play Rock-Paper-Scissors with you anytime, for as long as you want. You’ll wear a sensor-equipped glove that automatically tracks your gestures, scores the game, and determines the winner!

Related products, works and inspiration:

Our idea began with the goal of detecting and replicating complex movements of human hands. Our inspiration drew heavily from projects like Tatum Robotics, which demonstrated the exciting possibilities of robotics in assisting users to learn sign language.

While the applications in sign language learning were fascinating, we wanted to take this concept further and build a fun and entertaining experience. This led us to thinking of different gesture-based games where users could directly interact with a robotic system—what better choice than the timeless game of Rock-Paper-Scissors?

Summary of Milestone 1:

In milestone 1, we explored two primary ideas:

ASL learning using Wearable Gloves: A wearable device designed to assist users in learning American Sign Language (ASL). The glove detects gestures and provides real-time haptic feedback to help users correct their hand and finger positioning, combining gesture recognition and feedback for an intuitive learning experience.

Rock-Paper-Scissors Game: A system where two players wear gloves that detect hand gestures (rock, paper, or scissors) and monitor the game in real-time. One player is a human, while the second can be either a human or a computer (a mechanical hand). The system automatically tracks scores for seamless play.

We leaned towards the ASL learning using Wearable Gloves idea and began making progress on this idea.

Summary of Milestone 2:

In milestone 2, we focused on building the hardware and software for gesture recognition. We integrated flex sensors and touch sensors to measure finger bending and touch detections respectively. With these sensor measurements, we were able to detect simple gestures like numbers in ASL. We completed the circuit and code to set the foundation for testing simple gestures, which can be worked on for recognizing more complex gestures later.

Summary of Milestone 3:

After successfully developing gesture recognition, we shifted our focus to the next phase: gesture correction. Our initial plan was to use Electrical Muscle Stimulation (EMS) to provide feedback and automatically move the fingers into the correct position when the user performed a gesture incorrectly. The envisioned workflow involved a user selecting a word to learn, performing the gesture, and receiving feedback based on whether it was correct or not. If it was incorrect, the EMS would force the fingers to move to the correct position.

However, we faced challenges with the EMS device and circuits, which weren't precise enough for controlling individual fingers.

We hence pivoted to our second idea: building Rock-Paper-Scissors game. We had our sensor-equipped glove for the human player, ready from our work done in the previous milestones.

The next step was to create the mechanical hand for the robot player. We 3D printed a mechanical hand with three joints and four segments per finger for flexibility. To control the movement of each finger of the mechanical hand, we would normally need one servo motor per finger, but that would require five motors in total, which is inefficient and would make the design more complex. Instead, we took advantage of the fact that, in the Rock-Paper-Scissors game, certain fingers often move together. Specifically, the index and middle fingers typically move together as a pair, and the ring and little fingers move together as another pair. We decided to use three servo motors to control three sets of fingers:

• Thumb: Controlled by one servo motor.
• Index and Middle Fingers: Controlled together using a second servo motor.
• Ring and Little Fingers: Controlled together using a third servo motor.

Each servo motor pulls a thread wrapped around a spool, and this thread is connected to the fingers through a mechanical system. The pulling of the thread causes the fingers to bend or straighten depending on the motor's movement. To ensure that the fingers return to their open position after being bent, we used elastic bands. These bands were passed through the back of the fingers, and their tension helps pull the fingers back to the open hand position when the servo motors stop pulling the threads.

End-to-End System Overview

Our system connects the microcontroller and web app over WiFi, enabling real-time communication between the game interface and the mechanical hand. We use the ESPAsyncWebServer.h library to facilitate asynchronous communication between the ESP32 microcontroller and the web app. The microcontroller is powered by the laptop, providing a stable power source for the entire setup.

When the user selects Start Game on the web app (which can be accessed via a mobile phone or laptop), the game begins, and players compete until one of them scores 3 points. Each round has a 5-second countdown. During this time, the code randomly selects a gesture for the robot, while the player also prepares for their gesture. The system immediately recognizes the human player’s gesture and compares it with the robot’s gesture, determining the winner of that round. The game continues until one player wins 3 rounds, at which point the overall winner is declared.

Hardware Components

We use servo motors that operate within a range of 0 to 180 degrees. In this setup, 180 degrees represents a fully curled position of the fingers, while 0 degrees corresponds to the fully straight position. As soon as the robot’s gesture is randomly selected, the servo motors rotate to generate the desired gesture on the mechanical hand, right before the countdown ends. The combination of motors, spools, threads, and elastic bands allows the mechanical hand to perform the gestures smoothly and in real time.

For controlling the robot's hand, the servo motors are set to specific angles based on the gesture. The code follows this logic:

• For Rock, all three sets of servo motors (corresponding to the thumb, index-middle, and ring-little fingers) are set to 180 degrees to simulate a closed fist.
• For Paper, all servo motors are set to 0 degrees, resulting in an open hand.
• For Scissors, the thumb and ring-little fingers are curled (180 degrees), while the index-middle fingers are open (0 degrees).

For the human hand glove, we attach five touch sensors at the tips of the fingers. The touch sensor data is mapped to specific gestures as follows:

• For Rock, all five touch sensors (on the thumb, index, middle, ring, and little fingers) will register as TRUE, indicating a closed fist.
• For Scissors, only the touch sensors on the little, ring, and thumb fingers will be TRUE, corresponding to a gesture with the index and middle fingers extended.
• For Paper, none of the touch sensors will register as TRUE, corresponding to an open hand gesture.

The touch sensors and servo motors are efficiently connected to the same microcontroller, with all components placed on a single breadboard circuit. This system ensures that the game progresses without delay, with the robot and human player responding instantly.