Sense Staff

⭐ Sense Staff

The Sense Staff exists as an alerting mechanism to help vision impaired users navigate through unknown environments. At its core it's a depth sensor connected via a Unihiker from DFRobot to provide detection of obstacles. Upon detection of these issues it informs the user via a bluetooth headset allowing them to take heed and adjust their direction. The associated screen displays the depth information allowing the user to potentially reference it for further information if they are able to see at a close range but the sound element allows any user of various levels of impairment to be aware.

🦸‍♀️ Thank You Project Sponsors

Thank you to DFRobot for providing the Unihiker to the contest participants. It's a powerful single board computer that runs a flavor of linux and has libraries allowing it to access the GPIO available. It provides connection points for i2c, has a microphone, buzzer, and so on. It's really a nice board and worked well for my use case. I originally was going to have to use a much weaker depth detection camera but with the help of this device it made it doable for high resolution detection of objects.

⏲ Background

Originally I had thought to make a machine learning model to detect obstacles in such a way that the user could identify what they were from a limited subset. This would have been a neat demo but the contest masters informed me of a real need to detect problems like wires hanging low in their environment that a limited subset may not include. I opted to go with an Oak-D Lite depth sensing camera as I felt I could integrate it with the DFRobot Unihiker to get enough information to respond accordingly.

💻 Hardware

For this project I utilized an Oak-D Lite depth camera hooked up via a bendable camera mount to the staff or cane used by the user. In this way any sized cane can be utilized without needing to worry about attachment issues. From there the camera is connected to the Unihiker which has been sewn onto a wristband. A battery pack powers the Unihiker allowing it to run on the go. Additional battery packs can be swapped out as needed for longer use time.

✨ Code

This project uses the RGB depth output from the Oak-D Lite, has an alerting mechanism for detected objects, and is set to trigger for a range of depth. I decided to use the depth image that was colorized to detect objects which I then alert on sending a notification over bluetooth via audio.

To get the code setup you'll need to copy it and the wav file over to your Unihiker and then run it via the Run Programs section.

Setting Up Bluetooth

Unihiker's guide has some information on setting up bluetooth.

The short of the guide though is repeated below:

bluetoothctl
default-agent
power on
scan on 
trust xx:xx:xx:xx:xx:xx (Device ID)
pair xx:xx:xx:xx:xx:xx  (Device ID)
connect xx:xx:xx:xx:xx:xx (Device ID)
scan off

With this you'll find the bluetooth device, trust, and connect to it. Once it was connected I immediately began hearing audio on the playing of sound files via my headphones.

Setting Up Oak-D Lite

To setup the Oak-D Lite software you'll need to ssh into your device. Once in the terminal you can git clone the repository and install its dependencies. It is suggested that you use a venv for this but in my case I just installed it directly with no issues.

git clone --recursive https://github.com/luxonis/depthai.git
cd depthai
echo 'SUBSYSTEM=="usb", ATTRS{idVendor}=="03e7", MODE="0666"' | sudo tee /etc/udev/rules.d/80-movidius.rules
sudo udevadm control --reload-rules 
sudo udevadm trigger
python3 install_requirements.py

Once that finishes you're ready to use the device.

Setting Up Code

I utilized Visual Studio's SSH connection to a remote to login to my Unihiker and proceeded to setup the script from there. The getting started guide for Unihiker has more information on the process. You can find the entire code for this project in the attachments. Once it has been installed you should be able to trigger it via the "Run Programs" menu from the Unihiker.

🎉 Usage

The device sends an audio alert when it detects an obstruction in your path. In this way you can walk around with the depth camera scanning the surroundings for issues and sending messages whenever there's a problem in sight. It uses a background thread for the audio to prevent slowness with the rendering. In addition the screen is used for display to assist those who may be able to see closer up to get a better idea of the nature of the obstruction. The thresholds for detecting items can be modified via the code but it defaults in such a way that even wires are detected.

😅 Challenges

As mentioned earlier I originally was thinking of using an Maixsense A010. I didn't realize how low the resolution would be initially so had started down the path in earnest until I later opted for the Oak-D Lite to better meet the feedback from the contest masters.

I was also going to use the disparity data from the Oak-D Lite for detecting objects but was seeing too many false positives from background noise. I noticed the IR based cameras from them are a lot smoother so regret not buying one of those for additional clarity but the RGB color based object detection for alerts has worked well so wasn't fully needed for this project. A clear improvement for the project could be using an active stereo camera (a bit more expensive) to further improve detection and reduce false positives from the detection noise.