In this tutorial, we will interface two HC-SR04 ultrasonic sensors and Surilli WiFi to control your computer or laptop by hand gesture. We can control few functions of web browser like switching between tabs, scrolling up and down in web pages, shift between tasks (applications), play or pause a video and increase or decrease the volume (in VLC Player) with the help of hand gestures.
STEP 1: Downloading PythonThe first step is to download the Python IDE from the official website. The installation procedure and steps mentioned in this project are relevant only for Windows 7 Operating System (might be similar to Windows 10). The setup procedure for Python on other operating systems like MAC OS and Linux will be different.The version of Python used in this tutorial is Python 2.7.14.
STEP 2: Installing Python on Your ComputerAfter downloading the installation file, proceed with installation by double clicking it. Select “Install for all users” and click on next. Do not change the default installation directory, which is “C:\Python27\” and click on next.
Next, select the “Add python.exe to path” option in the Customize Python 2.7.14 area. This will automatically add the Python directory to the Windows System Path. If you skip this step, you can add the directory to the path manually. Proceed and finish the installation.
Python is successfully installed on your computer. If you have set the system path (either manually or while installing path), you can check if the Python is successfully installed or not.
For this, run the Command Prompt with Administrator rights and just enter “python”. If python is successfully installed on your computer, you will get a response corresponding to the Python Version and type of installation.
At this point, you can start using the Python IDE by searching for “Python” in the Start Menu. You will get IDLE (Python GUI) option. IDLE stands for Integrated Development and Learning Environment, a GUI based Python IDE.
Selecting it will open the Python Shell, which lets you use the Python interpreter in an interactive mode i.e. you can type commands and see the results instantaneously.
Getting pySerial
In order to access the Serial Port of the Computer through Python, we are going to need a library called pySerial. pySerial is a Python Serial Port Extension that provides access to the Serial Port for Python running on different Operating Systems like Windows, Linux, Mac OS (OSX), etc.
In order to download the pySerial,use the attached link.
This link will download pySerial-3.4.
Extract the contents of the file to any folder like “C:\pyserial-3.4”.
NOTE: Location is important, as we have to navigate to this location and install the pySerial.
Now, run the Windows Command Prompt with Administrator privileges and navigate to the folder where pySerial is extracted to.
In order to install pySerial, type the following command in the command prompt and hit enter.
Python setup.py install
After successful installation of pySerial, you can check if it is integrated to Python or not by entering the following command in the Python IDLE.
Import serial
If everything goes well, you will not get any error.
Components Required- Surilli WiFi
- 2 ultrasonic sensors
- USB cable (for connecting WiFi with Laptop or PC)
- Connecting wires
- A laptop with internet connection
The design of the circuit is very simple, but the setup of the components is very important. The Trigger and Echo Pins of the first Ultrasonic Sensor (that is placed on the left of the screen) are connected to Pins 12 and 16 of the Surilli WiFi. For the second Ultrasonic Sensor, the Trigger and Echo Pins are connected to Pins 4 and 5 of the Surilli GSM.
Now, coming to the placement of the Sensors, place both the Ultrasonic Sensors on top of the Laptop screen, one at the left end and the other at right. You can use double sided tape to hold the sensors onto the screen.
Coming to Surilli WiFi, place it on the back of the laptop screen. Connect the wires from Surilli WiFi to Trigger and Echo Pins of the individual sensors. Now, we are ready for programming the Arduino.
Programming Your Arduino IDE to Detect Gestures:The important part of this project is to write a program for Arduino IDE such that it converts the distances measured by both the sensors into the appropriate commands for controlling certain actions.The hand gestures in front of the Ultrasonic sensors can be calibrated so that they can perform five different tasks on your computer. Before taking a look at the gestures, let us first see the tasks that we can accomplish.
- Switch to Next Tab in a Web Browser
- Switch to Next Tab in a Web Browser
- Scroll Down in a Web Page
- Scroll Up in a Web Page
- Switch between two Tasks (Chrome and VLC Player)
- Play/Pause Video in VLC Player
- Increase Volume
- Decrease Volume
The following are the 5 different hand gestures or actions that I’ve programmed for demonstration purpose.
Gesture 1: Place your hand in front of the Right Ultrasonic Sensor at a distance (between 15 CM to 35 CM) for a small duration and move your hand away from the sensor. This gesture will Scroll Down the Web Page or Decrease the Volume.
Gesture 2: Place your hand in front of the Right Ultrasonic Sensor at a distance (between 15 CM to 35 CM) for a small duration and move your hand towards the sensor. This gesture will Scroll up the Web Page or Increase the Volume.
Gesture 3: Swipe your hand in front of the Right Ultrasonic Sensor. This gesture will move to the Next Tab.
Gesture 4: Swipe your hand in front of the Left Ultrasonic Sensor. This gesture will move to the Previous Tab or Play/Pause the Video.
Gesture 5: Swipe your hand across both the sensors (Left Sensor first). This action will Switch between Tasks.
Set Up Arduino IDE for Surilli:Make sure you have selected the right port, board and processor for the Surilli as shown in the picture below and it is programmable (compile and upload “Blink” from File>Examples>Digital>Blink onto your Surilli to check if everything is working fine).
The CircuitryThe circuitry is very simple. It's mostly the programming. Follow the figure below to set up your hardware.
Now you have completed setting up your hardware and Arduino IDE. Copy and paste the Arduino sketch given below into your Arduino IDE and hit upload. After it is uploaded, the results can be viewed on the Serial Monitor in Arduino IDE.
Arduino Code:
const int trigPin1 = 12; // the number of the trigger output pin ( sensor 1 )
const int echoPin1 = 16; // the number of the echo input pin ( sensor 1 )
const int trigPin2 = 4; // the number of the trigger output pin ( sensor 2 )
const int echoPin2 = 5; // the number of the echo input pin ( sensor 2 )
////////////////////////////////// variables used for distance calculation
long duration;
int distance1, distance2;
float r;
unsigned long temp=0;
int temp1=0;
int l=0;
////////////////////////////////
void find_distance (void);
// this function returns the value in cm.
/*we should not trigger the both ultrasonic sensor at the same time.
it might cause error result due to the intraction of the both soundswaves.*/
void find_distance (void)
{
digitalWrite(trigPin1, LOW);
delayMicroseconds(2);
digitalWrite(trigPin1, HIGH);
delayMicroseconds(10);
digitalWrite(trigPin1, LOW);
duration = pulseIn(echoPin1, HIGH, 5000);// here this pulsein function wont wait more then 5000us for the ultrasonic sound to came back. (due to this it wont measure more than 60cm)
// it helps this project to use the gesture control in the defined space.
// so that, it will return zero if distance greater then 60m. ( it helps usually if we remove our hands infront of the sensors ).
r = 3.4 * duration / 2; // calculation to get the measurement in cm using the time returned by the pulsein function.
distance1 = r / 100.00;
/////////////////////////////////////////upper part for left sensor and lower part for right sensor
digitalWrite(trigPin2, LOW);
delayMicroseconds(2);
digitalWrite(trigPin2, HIGH);
delayMicroseconds(10);
digitalWrite(trigPin2, LOW);
duration = pulseIn(echoPin2, HIGH, 5000);
r = 3.4 * duration / 2;
distance2 = r / 100.00;
delay(100);
}
void setup()
{
Serial.begin(9600);
pinMode(trigPin1, OUTPUT); // initialize the trigger and echo pins of both the sensor as input and output:
pinMode(echoPin1, INPUT);
pinMode(trigPin2, OUTPUT);
pinMode(echoPin2, INPUT);
delay (1000);
}
void loop()
{
find_distance(); // this function will stores the current distance measured by the ultrasonic sensor in the global variable "distance1 and distance2"
// no matter what, the program has to call this "find_distance" function continuously to get the distance value at all time.
if(distance2<=35 && distance2>=15) // once if we placed our hands in front of the right sensor in the range between 15 to 35cm this condition becomes true.
{
temp=millis(); // store the current time in the variable temp. (" millis " Returns the number of milliseconds since the Arduino board began running the current program )
while(millis()<=(temp+300)) // this loop measures the distance for another 300 milliseconds. ( it helps to find the difference between the swipe and stay of our hand in front of the right sensor )
find_distance();
if(distance2<=35 && distance2>=15) // this condition will true if we place our hand in front of the right sensor for more then 300 milli seconds.
{
temp=distance2; // store the current position of our hand in the variable temp.
while(distance2<=50 || distance2==0) // this loop will run untill we removes our hand in front of the right sensor.
{
find_distance(); // call this function continuously to get the live data.
if((temp+6)<distance2) // this condition becomes true if we moves our hand away from the right sensor (**but in front of it ). here " temp+6 " is for calibration.
{
Serial.println("down"); // send "down" serially.
}
else if((temp-6)>distance2) // this condition becomes true if we moves our hand closer to the right sensor.
{
Serial.println("up"); // send "up" serially.
}
}
}
else // this condition becomes true, if we only swipe in front of the right sensor .
{
Serial.println("next"); // send "next" serially.
}
}
else if(distance1<=35 && distance1>=15) // once if we placed our hands in front of the left sensor in the range between 15 to 35cm this condition becomes true.
{
temp=millis();
while(millis()<=(temp+300))
{
find_distance();
if(distance2<=35 && distance2>=15) // if our hand detects in the right sensor before 300 milli seconds this condition becomes true. ( usually it happens if we swipe our hand from left to right sensor )
{
Serial.println("change"); // send "change" serially.
l=1; // store 1 in variable l. ( it avoids the program to enter into the upcoming if condition )
break; // break the loop.
}
}
if(l==0) // this condition will become true, only if we swipe our hand in front of left sensor.
{
Serial.println("previous"); // send "previous" serially.
while(distance1<=35 && distance1>=15) // this loop will rotate untill we removes our hand infront of the left sensor. this will avoid not to enter this if condition again.
find_distance();
}
l=0; // make l=0 for the next round.
}
}
Python Programming for the ProjectWriting Python Program for Arduino based Hand Gesture Control is very simple. You just need to read the Serial data from Arduino and invoke certain keyboard key presses. In order to achieve this, you have to install a special Python Module called PyAutoGUI.
Installing PyAutoGUIThe following steps will guide you through the installation of PyAutoGUI on Windows Computers. The module PyAutoGUI will help you to programmatically control the mouse and keyboard.
With the help of PyAutoGUI, we can write a Python Program to mimic the actions of mouse like left click, right click, scroll, etc. and keyboard like key press, enter text, multiple key press, etc. without physically doing them. Let us install PyAutoGUI.
If you observe in the Arduino Code, the gesture mentioned above have been converted into 5 Commands that are sent to the Serial Port. Using these 5 commands, you can write a Python Program to control certain Keyboard Functions in order to achieve the required task.
Open Command Prompt with Administrator privileges and change to the directory where you have installed Python (in my case, it is C:\Python27).
If you have installed the latest version of Python, then pip (a tool for installing packages in Python) will already be installed. To check if pip is installed or not, type the following command.
pip -V
You should upgrade to the latest package of pip using the following command. If pip is already in its latest version, then ignore this step.
python -m pip install -U pip
Or
python -m pip install –upgrade pip
After upgrading pip, you can proceed to install PyAutoGUI. In order to install PyAutoGUI, type the following command.
python -m pip install pyautogui
Or
pip install pyautogui
If everything goes well till now, you can proceed to write the Python Code. If you observe the Arduino Code given above, the Arduino sends out five different texts or commands through Serial Port upon detecting appropriate hand gestures. These commands are
- Next
- Previous
- Down
- Up
- Change
Using these commands along with few functions in PyAutoGUI (like hotkey, scroll, keyDown, press and keyUp), you can write a simple Python Code that will execute the following tasks of keyboard and mouse.
- Data = “next” – – > Action = Ctrl+PgDn
- Data = “previous” – – > Action = Ctrl+PgUp
- Data = “down” – – > Action = Down Arrow
- Data = “up” – – > Action = Up Arrow
- Data = “change” – – > Action = Alt+Tab
- Keypress = Up Arrow – – > Action = Increase Volume
- Keypress = Down Arrow – – > Action = Decrease Volume
- Keypress = Ctrl+PgUp – – > Action = Play/Pause
Python Code:
import serial # add Serial library for serial communication
import pyautogui # add pyautogui library for programmatically controlling the mouse and keyboard.
Arduino_Serial = serial.Serial('com15',9600) # Initialize serial and Create Serial port object called Arduino_Serial
while 1:
incoming_data = str (Arduino_Serial.readline()) # read the serial data and print it as line
print (incoming_data) # print the incoming Serial data
if 'next' in incoming_data: # if incoming data is 'next'
pyautogui.hotkey('ctrl', 'pgdn') # perform "ctrl+pgdn" operation which moves to the next tab
if 'previous' in incoming_data: # if incoming data is 'previous'
pyautogui.hotkey('ctrl', 'pgup') # perform "ctrl+pgup" operation which moves to the previous tab
if 'down' in incoming_data: # if incoming data is 'down'
#pyautogui.press('down') # performs "down arrow" operation which scrolls down the page
pyautogui.scroll(-100)
if 'up' in incoming_data: # if incoming data is 'up'
#pyautogui.press('up') # performs "up arrow" operation which scrolls up the page
pyautogui.scroll(100)
if 'change' in incoming_data: # if incoming data is 'change'
pyautogui.keyDown('alt') # performs "alt+tab" operation which switches the tab
pyautogui.press('tab')
pyautogui.keyUp('alt')
incoming_data = ""; # clears the data
Play with the program to see how it reacts to different values and logic.
If you make something fun and interesting, do share it with our community.
That’s all for now. If you have any queries, visit surilli.io or contact our support. Stay connected with the Surilli family for more amazing stuff. :-)
Comments
Please log in or sign up to comment.