Nethra - Smart Cane

Story and Problem

People with visual impairments face significant challenges in adapting to outdoor activities due to various safety, transportation, and navigation issues. Despite the availability of affordable technologies like smart canes and improved transportation options, safety remains a critical concern. This problem is exacerbated in environments that are less structured or lack consistent accessibility features. The key challenges include:

1. Obstacle Detection and Avoidance:

• Uneven Pavements and Potholes: Navigating uneven pavements, potholes, and other ground-level obstacles is a major challenge. These hazards can cause trips, falls, and injuries, particularly in areas where sidewalks are poorly maintained or nonexistent.
• Obstacles in the Walking Path: Rocks, debris, and other unexpected obstacles can be difficult to detect and avoid, especially in areas without tactile paving or clear markers for visually impaired individuals.

2. Navigating and Avoiding Crowded Places:

• Crowded Areas like Parks and Markets: Visually impaired individuals often find it difficult to navigate through crowded places such as parks, markets, and busy streets. The unpredictability of people’s movements in such areas can make it challenging to maintain a safe path without bumping into others or getting lost.
• Auditory Overload in Crowded Spaces: The noise in crowded places can make it difficult for visually impaired individuals to rely on auditory cues, leading to confusion and disorientation.

3. Interactions with Animals:

• Street Animals: In regions like India, it is common to encounter stray dogs and other animals sleeping on the streets. Stepping on or startling these animals unknowingly can lead to potentially dangerous situations, including bites or falls.
• Animal Excrement: Navigating around or avoiding animal excrement on streets is another challenge, which can lead to health and hygiene issues if encountered unknowingly.

4. Urban Safety Hazards:

• Disorganized and Dangerous Streets: In many urban areas, especially in developing countries, streets are poorly organized and may lack clear demarcations for pedestrians. The absence of well-maintained sidewalks, proper street lighting, and accessible pedestrian crossings creates a hazardous environment for visually impaired individuals.
• Traffic and Speeding Vehicles: The lack of pedestrian-friendly infrastructure and the prevalence of speeding vehicles further exacerbate the risks, making it difficult for visually impaired individuals to cross streets safely or even walk along the side of the road.

In summary, the adaptation of visually impaired individuals to outdoor activities is hindered by a combination of environmental, infrastructural, and societal factors. While some technologies offer solutions for certain aspects of navigation, the overall safety and accessibility of outdoor environments remain a significant issue that needs to be addressed comprehensively.

Solution

The proposed solution involves developing a compact, camera-based image-processing device designed to assist visually impaired individuals with navigation and obstacle detection. This device will be mounted on a cane and will utilize the Grove Vision AI module in conjunction with a Xiao microcontroller to analyze visual data in real-time. The goal is to create a system that can identify and alert the user to various objects and situations that might pose a risk or require attention, such as rocks, potholes, crowds, and nearby dogs.

Grove Vision AI Module:

The Grove Vision AI module is an edge AI device capable of performing on-device image processing and object detection. It will serve as the primary sensor, capturing visual data from the environment and analyzing it using pre-trained models.

The module was trained to recognize specific objects and scenarios that are common obstacles for visually impaired individuals, such as uneven surfaces (potholes, rocks), crowded areas, and animals like dogs.

Xiao ESP32S3

The Xiao microcontroller serves as the processing unit, interfacing with the Grove Vision AI module and managing the overall system operation. It processes the data received from the AI module and determines the appropriate feedback to provide to the user. The Xiao’s compact size and low power consumption make it ideal for creating a portable, battery-powered device that can be easily attached to a cane. Currently, the Xiao uses a vibration motor for haptic feedback. Once an object is detected and classified, the system will provide haptic feedback to the user through small, strategically placed vibration motors on the device. The haptic feedback pattern varies with respect to the object detected.

Assembly & Usage

Connecting to a CSI interface camera ​

Connecting to a CSI interface camera ​

Once you have the Grove Vision AI V2 and any supported camera ready to go, then you can connect them via the CSI connection cable. When connecting, be sure to pay attention to the direction of the row of pins and don't plug them in backwards. When looking down at the connector, the blue plastic should be facing up.

Now attach the Xiao ESP32 to the blue AI vision module as shown below:

Then, attach the antenna to the Xiao's antenna connector. It will be a little hard to connect the antenna so don't try to attach the antenna straight down onto the connector. Instead, angle the antenna, then push down and wiggle it on.

The 3D files for the enclosure are provided below for you to download and print them. The setup can be fixed in the enclosure. The enclosure has a provision to fix with the cane.

The SenseCraft Web-Toolkit

SenseCraft Web Toolkit is a user-friendly tool built into the SSCMA platform that simplifies the process of deploying AI models. It allows you to effortlessly transfer models to different devices without any coding required. The model was trained as per the wiki provided by Seeedstudio. The model was trained with a minimal dataset to test the initial feasibility.

When a model is running in the Grove vision AI, the output looks as below:

Based on the threshold value and the target, the vibration pattern was programmed differently.

//Task 1 to read from Grove AI module.
  xTaskCreatePinnedToCore(
                    Task1code,   
                    "Task1",     
                    10000,       
                    NULL,        
                    1,           
                    &Task1,      
                    0);                           
  delay(500); 

//Task 2 to control the vibration motor.
  xTaskCreatePinnedToCore(
                    Task2code,   
                    "Task2",     
                    10000,       
                    NULL,        
                    1,           
                    &Task2,     
                    1);

The ESP uses multitasking for this particular function. Initially, the ESP reads from the Grove Vision AI module and then based on the target, the vibration pattern varies as shown below in the code:

if(aiTarget == 0) // Nothing detected
{
digitalWrite(motorPin, LOW);          
}
if(aiTarget == 1)
{
for(int i=0;i>3;i++) //Persons detected - Vibrate thrice
{
digitalWrite(motorPin, HIGH);          
delay(200);
digitalWrite(motorPin, LOW);          
delay(200);  
}
}
if(aiTarget == 2) //Dog detected - Vibrate 5 times
{
for(int i=0;i>5;i++)
{
digitalWrite(motorPin, HIGH);          
delay(200);
digitalWrite(motorPin, LOW);          
delay(200);  
}}
if(aiTarget == 3) //Pothole detected - Vibrate 5 times
{
digitalWrite(motorPin, HIGH);          
delay(2000);  
digitalWrite(motorPin, LOW);          
}

Further Work:

The project entails extensive field testing, particularly in the outdoors, and includes multiple iterations throughout its development. Below listed are some of the planned experiments & iterations to be performed.

• Add a battery management system to support the device. Currently, the device uses a power bank connected to the Xiao.
• Use extensive data to train the model and test the model to make sure it is fool proof.
• Developing advanced algorithms to locate the objects based on the YOLO boxes results (Explained further below).
• Design an IP-rated enclosure to avoid rain and dust.

Using the YOLO boxes to navigate:

Object detection is a computer vision technique that pinpoints and identifies specific objects within images or videos. Unlike simply categorizing an entire picture, it precisely locates objects and outlines them with bounding boxes. This technology is essential for applications like self-driving cars, security systems, and augmented reality that require a detailed understanding of visual scenes.

As shown in the image above, a box provides accurate information to locate the object in a frame. Given the context that the device is mounted on a stick/cane, a complex algorithm can be developed to navigate properly using auditory feedback of the location.

Feel free to contact me for further questions.

Happy Making