TactileSight

Introduction

Hey! I am an 18-year-old student from Latvia. I made this project for the Build2Gether V2 Contest - indoor category. Here's the original idea proposal :

What problem are you going to solve?

Problem Identified: Difficulty with Object Identification and Manipulation for Visually Impaired Individuals Underlying Causes: Limited Sensory Input: Visual impairment significantly restricts access to visual information about an object's shape, size, location, and orientation. This lack of information makes it challenging for visually impaired individuals to identify and interact with objects in their environment. Reliance on Touch: People with visual impairments rely heavily on touch to gather information about objects. However, touch alone can be insufficient for accurate identification, especially for unfamiliar objects. Difficulty with Distance Perception: Visually impaired individuals often have difficulty gauging the distance between themselves and an object. This can lead to problems with grasping objects accurately.

What are you going to build to solve this problem?

proposed solution is a glove equipped with an AI-powered camera, distance sensor, and vibration feedback. This glove would assist visually impaired individuals with object identification and manipulation in indoor environments. When pointed at an object, the camera and AI would identify it (e.g., cup, bottle...) and transmit an audio voice telling what the object is through earphones. Additionally, the distance sensor would trigger a vibration in the glove, with intensity increasing as the user gets closer, aiding in grasping the object at the right distance. This combination of audio and haptic feedback would address the limitations of touch-only interaction, promoting independence and reducing reliance on assistance for everyday tasks.

So what did I come up with?

The project turned out almost as written in the proposed solution, with some improvements from the contest masters.

I made a "smart glove" that has haptic motors, light, adistance sensor, and a camera. When you want to grab an object, just ask it "What's that?" and using image recognition, it will understand what is that object and to your Bluetooth headphones answer - "That is (+ object name).". If the scene is too dark, it will automatically turn on the light and that will help the image recognition to find out what is that object. When you find what are you searching for, for example, a cup, you just start pushing your palm closer and using the sensor it will 1) change the less (red, yellow, green), as you get closer and 2) the haptics motors will get stronger, as you get closer.

Demo video

Instructions

Okay, let's get started! To make things easier, I made a small custom PCB. It has few led's, that indicate how close the hand is to an object. Because the device splits into 2 parts, the PCB also has connectors, so you can easily disconnect the "brains" from the glove.

PCB and assembly

So I made this PCB. You can find it at the bottom of this page.

To get it printed, go to pcbway.com and press quote now -> quick-order pcb, upload my gerber file, and set the settings, as below :

If you want, you can change the PCB color. I made a small mistake while ordering mine, so I had to solder on the other side. If you use these settings, then it will be as it should :)

Now that you have your PCB, we can start assembling everything. First of all, you need to find a glove, that you are willing to use for this project. I chose a generic builders glove. You can find something more fancy.

To fix the components in place, I found a needle and some string and sewed them. Here's where I sew the haptic feedback mini-vibration motors :

Here's how you should sew and solder the haptic motors. Leave the cable about this long, so you will be able to connect them to the "motherboard". Now sew and wire the distance sensor, like this :

And the last thing to add to the glove is a USB camera. I found some old one and striped it down so there's only the PCB and it takes less space.

Keep in mind, that the camera should be facing the front of the glove, and the distance sensor should be a little tilted to the center so that it will see the objects getting close.

Now that everything is in place, you have to solder the "motherboard". I used 3 color LEDs so that it is easier to see. The normal is red, in sight is yellow, and green is close.

If you have never soldered, here's an tutorial, to help you get started :

Get the components and solder it like this (all the resistors should be about 220 ohm !) :

When it's done, sew it to the top of the glove with the 2 4-pin connectors facing you. Now solder the connectors to the wires you prepared, like this :

Now solder the other side (2 4-pin) and make a about 6 cm long wire to the unihiker. I kept the unihiker sewed on another wristband on the arm. That way it is way more comfortable to wear. Then you can just plug it into a power bank. So, wire the other side like this :

Here is the port location for the unihiker :

One more thing! As the contest masters pointed out, there can be some bad environmental conditions. Maybe it's too dark, so I also made use of a light. You can use any 3.3v or 5v light. Attach it to the front of the glove. Then the sorter leg should be soldered to the unihikers GND pin and for the longer - use a 220ohm resistor and connect it to pin P8. The light will automatically turn on when the image recognition algorithm is engaged. Now, everything should be ready for the software part.

Coding

Get your unihiker and connect it to your PC with the USB cable. I used VS code by sshing into the board. Here an get started guide : https://www.unihiker.com/wiki/get-started

Connect your Bluetooth headphones to the unihiker, so that you don't need to worry about a lot of wires dangling.

Then just upload and run the code, I wrote, which you can find below on my GitHub page.

Before running the code! Remember, to install all the necessary libraries for the code to work :

pip install pinpong
pip install speech_recognition
pip install opencv-python
pip install numpy
pip install tensorflow
pip install pyttsx3

You can also set it so that the script always runs when it starts up!

Enjoy!!! :))