Be my Eyes!!

New innovations every day in the field of sensing and wireless communication, blind people have been able to attain a great degree of independency in sedentary settings. However, they still have to rely on an overseer or a walking stick to leave the security of their homes. This is highly risky as a walking stick though effective in figuring out path on a pavement or static objects in front, it is powerless with regard to warning one against an approaching vehicle on the street. This is just an example of how in secure our blind counterparts really are. No solution exists currently that can give persons with such disabilities complete confidence to venture out of their homes without worrying about safety.

To solve the shortcomings of available offerings, I propose to make a cap that can inform the wearer about the surroundings and give warnings in time. The cap would also be equipped with vibration motors for haptic feedback so that one can be informed of directions while on the road. Machine learning would be used to optimise the path the wearer would take to reach a particular destination. The module would also link with the person`s smartphone over WiFi for integration with Google Maps for directions. When an object comes in front of the wearer, the vibration motors would give haptic feedback for the person to stop. Similar instructions would be given to cross the road by identifying traffic lights and zebra crossings. Image classifiers of various objects on the road would identify objects like doors and pavements and inform the wearer about the surroundings through a speech API on an Android application (something like Google assistant)

The camera data would be analysed with a Deep Neural Network based Image Classifier that would identify mundane objects seen on the streets and send that information over Wifi to the wearer`s smartphone where an Android application gives audio information about the surroundings. While walking on the path, three vibration motors mounted on the front, left and right of the cap, operated through relays that in turn are switched by IO pins on the Ultra96V2 board would give haptic feedback to the wearer about when to stop, move left or right. The Android application would also link with Google Maps and use it to optimise the path the wearer must take to reach the destination such that least crossings and traffic is detected. While crossing a road, an IO pin operates a piezoelectric buzzer that warns people in the vicinity.

Various image classifier models from the Xilinx Model Zoo would be used like those for identifying vehicles, humans, postures and traffic symbols. For other objects like pavements or crossings trained datasets from Caffe would be used to finally deploy teh models on the programmable logic using Vitis

The AI connect

Vitis would be used to analyse computationally taxing parts of the machine learning functions to be used like clustering. The functions would then be accelarated on the programmable logic of the Ultra96V2 board. Programmable Logic would help accelarate parts of the code that otherwise would lead to thread blocks on a conventional system.

Deep neural networks(DNN) would be used to classify roadside objects from the video feed and for the shortest path determination. FOr some common models like the human or road sign detection stock models from the Xilinx Model ZOo would be used. For other models, custom models would be trained(some of them are available in public domain)

Update: Have been tryig to add APIs for the Husky Lens camera for one month now. However, the detection models are not working as perfectly as desired

Update #2

Roadtested my solution to not much success. Pedestrians are detected but the model is simply not reliable enough for detecting moving objects. I do not know where the problem is for I have given my best