Published September 5, 2024 © CC BY-NC-SA

Sense Staff

Unihiker powered depth sensing staff accessory for detecting obstructions and alerting over bluetooth

BeginnerFull instructions provided6 hours278

Lucky Draw for TWO Submissions

Build2gether 2.0 — Inclusive Innovation Challenge

Things used in this project

Hardware components

DFRobot UNIHIKER - IoT Python Programming Single Board Computer with Touchscreen

Oak-D Lite

Lenovo X5 Bone Conduction Headphones

Bendable Phone Tripod

Not the exact one I used but close enough. The one I used I've had for some time / no manufacturer listed.

Wristband

Story

⭐ Sense Staff

The Sense Staff exists as an alerting mechanism to help vision impaired users navigate through unknown environments. At its core it's a depth sensor connected via a Unihiker from DFRobot to provide detection of obstacles. Upon detection of these issues it informs the user via a bluetooth headset allowing them to take heed and adjust their direction. The associated screen displays the depth information allowing the user to potentially reference it for further information if they are able to see at a close range but the sound element allows any user of various levels of impairment to be aware.

1 / 3 • SenseStaff Detecting

🦸‍♀️ Thank You Project Sponsors

Thank you to DFRobot for providing the Unihiker to the contest participants. It's a powerful single board computer that runs a flavor of linux and has libraries allowing it to access the GPIO available. It provides connection points for i2c, has a microphone, buzzer, and so on. It's really a nice board and worked well for my use case. I originally was going to have to use a much weaker depth detection camera but with the help of this device it made it doable for high resolution detection of objects.

⏲ Background

Originally I had thought to make a machine learning model to detect obstacles in such a way that the user could identify what they were from a limited subset. This would have been a neat demo but the contest masters informed me of a real need to detect problems like wires hanging low in their environment that a limited subset may not include. I opted to go with an Oak-D Lite depth sensing camera as I felt I could integrate it with the DFRobot Unihiker to get enough information to respond accordingly.

💻 Hardware

For this project I utilized an Oak-D Lite depth camera hooked up via a bendable camera mount to the staff or cane used by the user. In this way any sized cane can be utilized without needing to worry about attachment issues. From there the camera is connected to the Unihiker which has been sewn onto a wristband. A battery pack powers the Unihiker allowing it to run on the go. Additional battery packs can be swapped out as needed for longer use time.

✨ Code

This project uses the RGB depth output from the Oak-D Lite, has an alerting mechanism for detected objects, and is set to trigger for a range of depth. I decided to use the depth image that was colorized to detect objects which I then alert on sending a notification over bluetooth via audio.

To get the code setup you'll need to copy it and the wav file over to your Unihiker and then run it via the Run Programs section.

Setting Up Bluetooth

Unihiker's guide has some information on setting up bluetooth.

The short of the guide though is repeated below:

bluetoothctl
default-agent
power on
scan on 
trust xx:xx:xx:xx:xx:xx (Device ID)
pair xx:xx:xx:xx:xx:xx  (Device ID)
connect xx:xx:xx:xx:xx:xx (Device ID)
scan off

With this you'll find the bluetooth device, trust, and connect to it. Once it was connected I immediately began hearing audio on the playing of sound files via my headphones.

Setting Up Oak-D Lite

To setup the Oak-D Lite software you'll need to ssh into your device. Once in the terminal you can git clone the repository and install its dependencies. It is suggested that you use a venv for this but in my case I just installed it directly with no issues.

git clone --recursive https://github.com/luxonis/depthai.git
cd depthai
echo 'SUBSYSTEM=="usb", ATTRS{idVendor}=="03e7", MODE="0666"' | sudo tee /etc/udev/rules.d/80-movidius.rules
sudo udevadm control --reload-rules 
sudo udevadm trigger
python3 install_requirements.py

Once that finishes you're ready to use the device.

Setting Up Code

I utilized Visual Studio's SSH connection to a remote to login to my Unihiker and proceeded to setup the script from there. The getting started guide for Unihiker has more information on the process. You can find the entire code for this project in the attachments. Once it has been installed you should be able to trigger it via the "Run Programs" menu from the Unihiker.

🎉 Usage

The device sends an audio alert when it detects an obstruction in your path. In this way you can walk around with the depth camera scanning the surroundings for issues and sending messages whenever there's a problem in sight. It uses a background thread for the audio to prevent slowness with the rendering. In addition the screen is used for display to assist those who may be able to see closer up to get a better idea of the nature of the obstruction. The thresholds for detecting items can be modified via the code but it defaults in such a way that even wires are detected.

1 / 4

😅 Challenges

As mentioned earlier I originally was thinking of using an Maixsense A010. I didn't realize how low the resolution would be initially so had started down the path in earnest until I later opted for the Oak-D Lite to better meet the feedback from the contest masters.

1 / 2 • Print of an earlier Maixsense A010 holding staff topper (intended to use zip ties to hold to the staff)

I was also going to use the disparity data from the Oak-D Lite for detecting objects but was seeing too many false positives from background noise. I noticed the IR based cameras from them are a lot smoother so regret not buying one of those for additional clarity but the RGB color based object detection for alerts has worked well so wasn't fully needed for this project. A clear improvement for the project could be using an active stereo camera (a bit more expensive) to further improve detection and reduce false positives from the detection noise.

Image Credit: https://docs.luxonis.com/hardware/platform/features/depth/

Code

SenseStaff.py

import cv2
import numpy as np
import depthai as dai
import threading
import datetime
from unihiker import Audio

# Weights to use when blending depth/rgb image
rgbWeight = 0.4
depthWeight = 0.6

fps = 30
monoResolution = dai.MonoCameraProperties.SensorResolution.THE_720_P

audio = Audio()
audio_playing = False
audio_lock = threading.Lock()

def send_alert(area, height, color_intensity):
    global audio_playing
    global audio_lock

    timestamp = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")
    print(f"[{timestamp}] Alert: Object detected in {area} area, height: {height}, color intensity: {color_intensity}")

    if not audio_playing and area == "front":
        threading.Thread(target=play_audio, daemon=True).start()

def play_audio():
    global audio_playing
    global audio_lock

    with audio_lock:
        if not audio_playing:
            audio_playing = True
            audio.play('obstruction.wav')
            audio_playing = False

def process_rgb_image(frame, color_threshold, min_height, screen_width):
    gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
    mask = cv2.inRange(gray, color_threshold, 255)
    vertical_sum = np.sum(mask, axis=1)
    detected_height = np.count_nonzero(vertical_sum)

    if detected_height >= min_height:
        left_sum = np.sum(mask[:, :screen_width//3])
        front_sum = np.sum(mask[:, screen_width//3:(2*screen_width)//3])
        right_sum = np.sum(mask[:, (2*screen_width)//3:])

        if max(left_sum, front_sum, right_sum) == front_sum:
            area = "front"
        else:
            area = "other"

        avg_intensity = np.mean(gray[mask > 0])
        send_alert(area, detected_height, avg_intensity)

    cv2.imshow("Mask", mask)

pipeline = dai.Pipeline()
device = dai.Device()
queueNames = []

camRgb = pipeline.create(dai.node.ColorCamera)
left = pipeline.create(dai.node.MonoCamera)
right = pipeline.create(dai.node.MonoCamera)
stereo = pipeline.create(dai.node.StereoDepth)

disparityOut = pipeline.create(dai.node.XLinkOut)
disparityOut.setStreamName("disp")
queueNames.append("disp")

rgbCamSocket = dai.CameraBoardSocket.CAM_A

camRgb.setBoardSocket(rgbCamSocket)
camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_720_P)
camRgb.setFps(30)

try:
    calibData = device.readCalibration()
    lensPosition = calibData.getLensPosition(rgbCamSocket)
    if lensPosition:
        camRgb.initialControl.setManualFocus(lensPosition)
except:
    raise

left.setResolution(dai.MonoCameraProperties.SensorResolution.THE_720_P)
left.setBoardSocket(dai.CameraBoardSocket.LEFT)
left.setFps(30)

right.setResolution(dai.MonoCameraProperties.SensorResolution.THE_720_P)
right.setBoardSocket(dai.CameraBoardSocket.RIGHT)
right.setFps(30)

stereo.setDefaultProfilePreset(dai.node.StereoDepth.PresetMode.HIGH_DENSITY)
stereo.setLeftRightCheck(True)
stereo.setExtendedDisparity(True)
stereo.setSubpixel(False)
stereo.setDepthAlign(rgbCamSocket)

left.out.link(stereo.left)
right.out.link(stereo.right)
stereo.disparity.link(disparityOut.input)

with device:
    device.startPipeline(pipeline)

    frameDisp = None

    depthWindowName = "depth"
    cv2.namedWindow(depthWindowName, cv2.WINDOW_NORMAL)
    cv2.setWindowProperty(depthWindowName, cv2.WND_PROP_FULLSCREEN, cv2.WINDOW_FULLSCREEN)

    screen_width = 240  
    screen_height = 320
    color_threshold = 200
    min_height = 5

    while True:
        latestPacket = None

        queueEvents = device.getQueueEvents(("disp",))
        if len(queueEvents) > 0:
            packets = device.getOutputQueue("disp").tryGetAll()
            if len(packets) > 0:
                latestPacket = packets[-1]

        if latestPacket is not None:
            frameDisp = latestPacket.getFrame()
            maxDisparity = stereo.initialConfig.getMaxDisparity()
            frameDisp = (frameDisp * 255. / maxDisparity).astype(np.uint8)
            frameDisp = cv2.applyColorMap(frameDisp, cv2.COLORMAP_HOT)
            frameDisp = np.ascontiguousarray(frameDisp)
            frameDisp = cv2.rotate(frameDisp, cv2.ROTATE_90_CLOCKWISE)
            frameDisp = cv2.resize(frameDisp, (screen_width, screen_height))
            process_rgb_image(frameDisp, color_threshold, min_height, screen_width)
            cv2.imshow(depthWindowName, frameDisp)

        if cv2.getWindowProperty(depthWindowName, cv2.WND_PROP_VISIBLE) < 1:
            break
        cv2.waitKey(1)