Assistive technology for blind person

Problem:- Blind person use stick to identify roads for potholes and road drainage. weather it is open or not. A stick does the job for that. But what about Physical obstruction or obstruction that are in 3 dimensions like car, moving car, Traffic lights of roads and the information provided by the traffic lights on the road, how far are the 3d objects. These are some of the information necessary for a blind person while he is walking on road. Until someone helps them in crossing the road. Underlying causes:- traditional technology only uses stick that help a blind person to identify and detect roads. But not 3 dimensional object that are in front of them and how far are those 3 dimensional objects are and what those objects are.

Why did you decide to make it?

To participate in Build2gether competition. And to build assistive technology for the disabled. So that it can be used by them.

Working

For this project we have used esp32 s3 sense development board an internet router and an i3 laptop with 8 gb ram. We need high ram to run computer vision program in it for object detection and other processing.

First we flash micro python with cam driver in the esp32 s3 sense board. link to the micro python driver is given bellow.

https://wiki.seeedstudio.com/XIAO_ESP32S3_Micropython/

The process to flash the micro python driver is given bellow.

https://www.hackster.io/manojroy123/flashing-micro-python-interpreter-on-esp32-s3-sense-ecbbf3 

After flashing the driver, test it on thonny IDE. To setup follow the steps bellow.

Setting up micro python for Thonny IDE:-

Step 1:-

Open Thonny IDE you will see Thonny window as shown bellow.

Step 2:-

Connect Esp32 s3 sense to you computer usb port.

Step 3:-

Go to tools and click on options as shown bellow

Step 4:-

A window will appear as shown bellow on that window click 'interpreter' tab on it select 'MicroPython (ESP32)' as an interpreter and on port select 'Espressif Device @ COM5'

Than click Ok

On the right hand corner you will find the following text 'MicroPython (ESP32). Espressif Device @ COM5' and a '>>>' on shell terminal.

Step 5:-

For testing write the following command on Shell terminal 'print("Hello World")' and you will get out put 'Hello World'

You have completed setting up micropython for Thonny.

Setting up camera driver and video streaming for ESP32 s3 sense:-

For setting up camera driver and start video streaming for the link given bellow.

https://www.hackster.io/manojroy123/esp32-s3-sense-video-streaming-in-micro-python-0d1486 

The above video streaming is black and white. Please use the following steps to change the black and white video streaming to colour video streaming.

1) Open 'streaming_server.py' from esp32 s3 sense

2)Search for 'camera.speffect(2)' on the file. The camera is set to black and white mode.

3)Change the value of 'camera.speffect(2)' to 'camera.speffect(0)' which will change the setting of camera to color.

Main program

purpose:- We want to build a product that can do the following things that will assist a blind person to achieve some of objectives.

1) Scan for Physical obstruction

2)obstruction that are moving

3)detecting and judging traffic signal

4)detecting signs

5)How far are those obstruction and its position.

Not only this we even want to communicate this data to the user using voice.

How did we attain the objective:- We have attain the following objective by building a computer vision program and using a Yolo machine learning model to detect those object, it's location, distance and movement and communicate it to the user via text to speech convertor.

Note:- Yolo is a pertained machine learning model with predefined objects in it.

Video of Working product:-

Video of product definition and details:-

What the product is

Video of how to use that product

Python code file link given bellow:-

https://drive.google.com/file/d/1yk5YvkOoo1NMFS7lD64SMWKpVeMlNhAD/view?usp=drive_link

Links for Yolo files:-

coco.names

https://drive.google.com/file/d/1iNCmZIc3UFBaQCEv5S5uh4zfMFlR2puJ/view?usp=drive_link

yolov3.cfg

https://drive.google.com/file/d/1-1FPPOV6h_XMnOGCoHJPhP2pdNHdmMdx/view?usp=drive_link

yolov3.weights

https://drive.google.com/file/d/1_Bp4-_36EPYaaqSPEv4oSEoVLTkf6Nnd/view?usp=drive_link