Amazing Control Computer Using Hand Motion And Arduino

In this project, we are going to learn how to build gesture-controlled laptops or computers. It is based on using the combination of Arduino and Python.

Instead of using a keyboard, mouse or joystick, we can use our hand gestures to control certain functions of a computer like to play/pause a video, move left/right in a photo slideshow, scroll up/down in a web page and many more. This is why I decided to control VLC Media Player as a hand gesture project.

The idea behind the project is quite easy by using two Ultrasonic Sensors (HC-SR04) with Arduino. We will place the two sensors on the top of a laptop screen and calculate the distance between the hand and the sensor. Counting on the information from Arduino that is sent to Python through the serial port, this information will then be read by Python which is running on the computer in order to perform certain actions.

Hardware components:

• Arduino

• Two Ultrasonic Sensors (HC-SR04)

• Some wires

Software apps:

• Arduino IDE

• Python IDLE

• PySerial library (We will use to communicate with serial ports).

• PyAutogui library (We will use to perform actions).

So I assume that you have already installed Python and PySerial library and have successfully done some basic projects. If not, don’t worry; I advise you to follow my previous tutorial (Programming Arduino Using Python). Also if you want to get acquainted with the Ultrasonic Sensor you can take a look (here).

Step 1: Watch the Video for More Details

Step 2: Hand Gestures Purpose

The following are the 5 commands hand gestures that I’ve programmed for demonstration purposes.

• First hand gesture: It allows us to 'Play/Pause' VLC by placing the two hands in front of the right/left Ultrasonic Sensor at a particular far distance.

• Second gesture: It allows us to 'Rewind' the video by placing a hand in front of the left sensor at a particular far distance.

• Third gesture: It allows us to 'Forward' the video by placing a hand in front of the right sensor at a particular far distance.

• Forth gesture: It allows us to 'Increase Volume' of the video by placing a hand in front of the left sensor at a particular far distance and moving away from the Sensor.

• Fifth gesture: It allows us to 'Decrease Volume' of the video by placing a hand in front of the left sensor at a particular far distance and get near to the sensor.

Step 3: Schematic

The schematic is quite simple, you should just follow the instructions below. So each sensor has 4 pins:

• Vcc - this pin is connected to 5V+.

• Trig - you need to define this pin in your program.

• Echo - this pin is the same as Trig, you also need to define it.

• GND - this pin is connected to ground.

Step 4: Programming Python

1. Install Python IDLE

You can skip this step if you have installed the Python IDLE already in your computer. If yes, then go to step 2, or else look to the following instructions.

• Go to the python website and download it (here).

• Once you have done, you move on to installation by keeping the directory in which the python is getting installed by default.

NOTE: Even if your computer is operating on 64-bit, you can use 32-bit Python itself due to the lack of compatibility with Arduino libraries.

2. Install PySerial library

PySerial is a Python API module which is used to read and write serial data to Arduino or any other Microcontroller. To install on Windows, simply visit PySerial's Download Page and following the steps below:

• Download the PySerial from the link above.

• Install it by keeping the setting as the default You should be sure that Pyserial worked correctly. To do this; you type in:

import serial

If you haven't confronted any error, you're good. Otherwise I advise you to check your installation and Python IDLE extension.

3. Install PyAutogui library

The purpose of PyAutoGUI is to provide a cross-platform Python module for GUI automation for human beings. The API is designed to be as simple as possible with sensible defaults. Follow the below steps to install pyautogui for Windows. (To follow these steps you should already have installed Python IDLE.)

If you are using other platforms, I advise you to take a look here.

• Open Windows Command

• Type in the following command

cd C:\Python27

• Then you type:

python –m pip install –upgrade pip

• The last command is:

python –m pip install pyautogui

NOTE: To be sure that pyautogui worked correctly just type in:

import pyautogui

If you haven't confronted any error, you're good.

Step 5: Arduino Code

To initiate a connection with the Arduino from Python, we first have to figure out which COM Port the Arduino is on. This task is simply made by the Ardunio programming environment as I noted in the picture above.

const int trigger1 = 2; //Trigger pin of 1st Sesnor
const int echo1 = 3; //Echo pin of 1st Sesnor
const int trigger2 = 4; //Trigger pin of 2nd Sesnor
const int echo2 = 5;//Echo pin of 2nd Sesnor
long time_taken;
int dist,distL,distR;
void setup() {
Serial.begin(9600); 
  
pinMode(trigger1, OUTPUT); 
pinMode(echo1, INPUT); 
pinMode(trigger2, OUTPUT); 
pinMode(echo2, INPUT); 
}
/*###Function to calculate distance###*/
void calculate_distance(int trigger, int echo)
{
digitalWrite(trigger, LOW);
delayMicroseconds(2);
digitalWrite(trigger, HIGH);
delayMicroseconds(10);
digitalWrite(trigger, LOW);
time_taken = pulseIn(echo, HIGH);
dist= time_taken*0.034/2;
if (dist>60)
dist = 60;
}
void loop() { //infinite loopy
calculate_distance(trigger1,echo1);
distL =dist; //get distance of left sensor
calculate_distance(trigger2,echo2);
distR =dist; //get distance of right sensor
//Pause Modes -Hold
if ((distL >40 && distR>40) && (distL <60 && distR<60)) //Detect both hands
{Serial.println("Play/Pause"); delay (500);}
calculate_distance(trigger1,echo1);
distL =dist;
calculate_distance(trigger2,echo2);
distR =dist;
//Control Modes
//Lock Left - Control Mode
if (distL>=13 && distL<=17)
{
  delay(100); //Hand Hold Time
  calculate_distance(trigger1,echo1);
  distL =dist;
  if (distL>=13 && distL<=17)
  {
    Serial.println("Left Locked");
    while(distL<=40)
    {
      calculate_distance(trigger1,echo1);
      distL =dist;
      if (distL<10) //Hand pushed in 
      {Serial.println ("Volume Increased"); delay (300);}
      if (distL>20) //Hand pulled out
      {Serial.println ("Volume Decreased"); delay (300);}
    }
  }
}
//Lock Right - Control Mode
if (distR>=13 && distR<=17)
{
  delay(100); //Hand Hold Time
  calculate_distance(trigger2,echo2);
  distR =dist;
  if (distR>=13 && distR<=17)
  {
    Serial.println("Right Locked");
    while(distR<=40)
    {
      calculate_distance(trigger2,echo2);
      distR =dist;
      if (distR<10) //Right hand pushed in
      {Serial.println ("Rewind"); delay (300);}
      if (distR>20) //Right hand pulled out
      {Serial.println ("Forward"); delay (300);}
  }
}
}
delay(200);
}

If you read the Arduino code, you will observe 5 commands which control certain keyboard functions in order to achieve the required task.

arduino_pythogui.ino Download

Step 6: Python Code

First up, we need a simple program to get the Python sending data over the serial port.

import serial #Serial imported for Serial communication
import time #Required to use delay functions
import pyautogui   #Required to to perform actions
ArduinoSerial = serial.Serial('com15',9600) #Create Serial port object called arduinoSerialData
time.sleep(2) #wait for 2 seconds for the communication to get established
while 1:
    incoming = str (ArduinoSerial.readline()) #read the serial data and print it as line
    print incoming
    
    if 'Play/Pause' in incoming:
        pyautogui.typewrite(['space'], 0.2)
    if 'Rewind' in incoming:
        pyautogui.hotkey('ctrl', 'left')  
    if 'Forward' in incoming:
        pyautogui.hotkey('ctrl', 'right') 
    if 'Volume Incresaed' in incoming:
        pyautogui.hotkey('ctrl', 'down')
        
    if 'Volume Decreased' in incoming:
        pyautogui.hotkey('ctrl', 'up')
    incoming = "";