Published September 5, 2024

Navigation Assistance for Visually Impared

I have friends who struggle to see their phone, let alone maps. This device is a quick and bight direction indicator.

IntermediateFull instructions provided5 hours171

Honorable Mention Visual Impairments

Build2gether 2.0 — Inclusive Innovation Challenge

Navigation Assistance for Visually Impared

Things used in this project

Hardware components

M5Stack ATOM Matrix ESP32 Development Kit

Used as a watch for a navigation aid

DFRobot UNIHIKER - IoT Python Programming Single Board Computer with Touchscreen

Simple python scripts to test the functionality of the navigation aids

Nordic Semiconductor nRF52 Development Kit

Used to develop a larger version of the Visual Aid

Software apps and online services

MicroPython

Hand tools and fabrication machines

Soldering iron (generic)

Solder Wire, Lead Free

Mini Side Cutter, 120mm Length with 25mm Jaw Capacity

3D Printer (generic)

Story

Audio Submission

Since this is submission is for those visually impared, I recorded an audio version of my submission.

As suggested in my feedback from the Contest Masters of the Build2gether 2.0 — Inclusive Innovation Challenge, I spent several days researching what solutions were currently on the market and found:

Apple - Their watch shows detailed navigation now, but it would be hard to read for those visually impaired.

On a side note I thought I would look at if the device could be used to aid those in wheelchairs, but there was all theory about routes based on changes to elevation, but nothing had been done. There was an app for wheelchair accessible entrances, but that did not have navigation.

Google - Same as apple, turn by turn navigation, but only a little arrow on the watch. Their maps did have a wheelchair accessibility option though, but nothing about helping those with sight impairments.

I decided to start with Google Maps navigation because they were at least starting to include accessibility data to their maps. This was my first time using Google API, but I managed to get it working on the DFRobot Unihiker and while I could extract directions, it would not actually do live navigation and accessibility options are stored in the places API, which is completely different.

1 / 4

The DFRobot Unihiker is great for developing python code. It is simple to use and works straight out of the box! Examples included to start and lots of connectivity types.

I decided to make an App to do the navigation, unfortunately App development is not something I have done before and I decided I would not be able to commit the hours needed to get to a point I wanted. This led me to the decision to concentrate on the hardware.

I have made a device with the M5Stack Atom Matrix, which displays a direction based on a message sent to it, via Bluetooth.

My initial display was of an arrow with a programmable colour, tailored to be the easiest to be seen by the user. After testing this, it was decided that it was not very effective and the display had to be animated. This would make the navigation more easily visible. I made several different animations and a moving bar was the easiest and quickest to read.

I also allowed for the colour and brightness to be changed to make it easily visible based on the user preference.

The animation speed can be adjusted based on personal preference.

BLE Serial Communication

My initial testing and experimentation with BLE Serial was done with the Nordic Semiconductor's nRF52840 DK because ultimately that is the chipset I would use for this device. It has low power requirements and is a dedicate BLE chip.

Usage

Connect via a Bluetooth UART connection and the following commands can be sent to the device:

Animations

arrow - Standard Arrow

shortArrow - Arrow without a tail

line - single scrolling line

2Lines - 2 scrolling lines

Navigation

FORWARD - Forward Animation

BACK - Behind Animation

LEFT - Turn Left Animation

RIGHT - Turn Right Animation

Settings

C R# G# B# - Substitute R#, G#, B# for 0-255, to set the Red Green and Blue Brightness

D # - Substitute # with a number to set the delay in ms between frames of the animation

Code is written in micropython to allow for easy update and modification.

Conclusion

Unfortunately at this stage the technology does not allow for this device to be used, but I hope in the not too distant future, Google or Apple could add in the ability to communicate with bluetooth devices such as mine. Only other alternative would be to write a whole new App, with accessibility in mind.

Future Versions

I would love to build my ultimate goal of a battery powered version, using the Nordic nrf52 versions. Low power and dedicated BLE! Just look at how small the module is!

Of cause a larger version isn't off the cards either.

Code

# This example demonstrates a peripheral implementing the Nordic UART Service (NUS).

# This example demonstrates the low-level bluetooth module. For most
# applications, we recommend using the higher-level aioble library which takes
# care of all IRQ handling and connection management. See
# https://github.com/micropython/micropython-lib/tree/master/micropython/bluetooth/aioble

import atom
import time
import bluetooth
from ble_advertising import advertising_payload

from micropython import const

COLOUR_OFF = (0, 0, 0)
COLOUR = (200, 200, 0)
delay = 100

dispMatrix = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

a = atom.Matrix()
a.set_pixels_color(*COLOUR_OFF)
dispType = "2lines"
i = 0
j = 0

def set_screen(dm):
    i = 0
    for n in dm:
        if n == 0:
            a.set_pixel_color(i, 0, 0, 0)
        else:
            a.set_pixel_color(i, *COLOUR)
        i += 1

def dispMatrix(i, j):
    DispMatrix = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
    for x in range(5):
        for y in range(5):
            if x == i:
                DispMatrix[x*5+y] = 1
            elif y == j:
                DispMatrix[x*5+y] = 1
            else:
                DispMatrix[x*5+y] = 0
    return DispMatrix

def rotMatrix(m, q):
    if q == 1:
        for o in range(5):
            m[(o*5)-5] = 0
    elif q == 5:
        for o in range(5):
            m[o] = 0
    for a in range(q):
        m.insert(len(m) - 1, m.pop(0))
    return m

def revMatrix(m, q):
    if q == 1:
        for o in range(5):
            m[(o*5)-1] = 0
    elif q == 5:
        for o in range(5):
            m[24-o] = 0
    for a in range(q):
        m.insert(0, m.pop(len(m) - 1))
    return m


def goForward():
    global i
    global dispMatrix
    dispMatrix = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
    if dispType == "arrow":
        dispMatrix = [0, 0, 1, 0, 0, 0, 1, 1, 1, 0, 1, 0, 1, 0, 1, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0]
    elif dispType == "shortArrow":
        dispMatrix = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 1, 0, 0, 0, 1]
    elif dispType == "2lines":
        dispMatrix = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
    else:
        dispMatrix = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1]
    for i in range(5):
        # dispMatrix = dispMatrix [5: ] + dispMatrix[ :5]
        set_screen(dispMatrix)
        dispMatrix = rotMatrix(dispMatrix, 5)
        time.sleep_ms(delay)

def goBack():
    global i
    global dispMatrix
    dispMatrix = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
    if dispType == "arrow":
        dispMatrix = [0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1, 0, 1, 0, 1, 1, 1, 0, 0, 0, 1, 0, 0]
    elif dispType == "shortArrow":
        dispMatrix = [1, 0, 0, 0, 1, 0, 1, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
    elif dispType == "2lines":
        dispMatrix = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
    else:
        dispMatrix = [1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
    for i in range(5):
        set_screen(dispMatrix)
        dispMatrix = revMatrix(dispMatrix, 5)
        time.sleep_ms(delay)

def goLeft():
    global i
    global dispMatrix
    dispMatrix = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
    if dispType == "arrow":
        dispMatrix = [0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 1, 1, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0]
    elif dispType == "shortArrow":
        dispMatrix = [0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1]
    elif dispType == "2lines":
        dispMatrix = [0, 0, 0, 1, 1, 0, 0, 0, 1, 1, 0, 0, 0, 1, 1, 0, 0, 0, 1, 1, 0, 0, 0, 1, 1]
    else:
        dispMatrix = [0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1]
    for i in range(5):
        set_screen(dispMatrix)
        dispMatrix = rotMatrix(dispMatrix, 1)
        # dispMatrix.insert(len(dispMatrix) - 1, dispMatrix.pop(0))
        time.sleep_ms(delay)

def goRight():
    global i
    global dispMatrix
    dispMatrix = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
    if dispType == "arrow":
        dispMatrix = [0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 1, 1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0]
    elif dispType == "shortArrow":
        dispMatrix = [1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0]
    elif dispType == "2lines":
        dispMatrix = [1, 1, 0, 0, 0, 1, 1, 0, 0, 0, 1, 1, 0, 0, 0, 1, 1, 0, 0, 0, 1, 1, 0, 0, 0]
    else:
        dispMatrix = [1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0]
    for i in range(5):
        set_screen(dispMatrix)
        dispMatrix = revMatrix(dispMatrix, 1)
        # dispMatrix.insert(0, dispMatrix.pop(len(dispMatrix) - 1))
        time.sleep_ms(delay)

def update_display(new_val):
    global dispType
    # Clear the display.
    a.set_pixels_color(*COLOUR_OFF)

    global COLOUR
    if new_val == "FORWARD":
        goForward()
    elif new_val == "BACK":
        goBack()
    elif new_val == "LEFT":
        goLeft()
    elif new_val == "RIGHT":
        goRight()
    elif new_val == "arrow" or new_val == "shortArrow" or new_val == "2lines" or new_val == "lines":
        dispType = new_val
    elif new_val[0] == 'C':
        nc = list(COLOUR)
        i = 0
        for c in tuple(new_val.split(" ")):
            if c != 'C':
                print("c: ", c)
                nc[i] = int(c)
                i += 1
        COLOUR = tuple(nc)
    elif new_val[0] == 'D':
        print("delay: ", new_val[1: ], "ms")
        global delay
        delay = int(new_val[1: ])

    a.set_pixels_color(*COLOUR_OFF)


_IRQ_CENTRAL_CONNECT = const(1)
_IRQ_CENTRAL_DISCONNECT = const(2)
_IRQ_GATTS_WRITE = const(3)

_FLAG_WRITE = const(0x0008)
_FLAG_NOTIFY = const(0x0010)

_UART_UUID = bluetooth.UUID("6E400001-B5A3-F393-E0A9-E50E24DCCA9E")
_UART_TX = (
    bluetooth.UUID("6E400003-B5A3-F393-E0A9-E50E24DCCA9E"),
    _FLAG_NOTIFY,
)
_UART_RX = (
    bluetooth.UUID("6E400002-B5A3-F393-E0A9-E50E24DCCA9E"),
    _FLAG_WRITE,
)
_UART_SERVICE = (
    _UART_UUID,
    (_UART_TX, _UART_RX),
)

# org.bluetooth.characteristic.gap.appearance.xml
_ADV_APPEARANCE_GENERIC_COMPUTER = const(128)


class BLEUART:
    def __init__(self, ble, name="direction-assist", rxbuf=100):
        self._ble = ble
        self._ble.active(True)
        self._ble.irq(self._irq)
        ((self._tx_handle, self._rx_handle),) = self._ble.gatts_register_services((_UART_SERVICE,))
        # Increase the size of the rx buffer and enable append mode.
        self._ble.gatts_set_buffer(self._rx_handle, rxbuf, True)
        self._connections = set()
        self._rx_buffer = bytearray()
        self._handler = None
        # Optionally add services=[_UART_UUID], but this is likely to make the payload too large.
        self._payload = advertising_payload(name=name, appearance=_ADV_APPEARANCE_GENERIC_COMPUTER)
        self._advertise()

    def irq(self, handler):
        self._handler = handler

    def _irq(self, event, data):
        # Track connections so we can send notifications.
        if event == _IRQ_CENTRAL_CONNECT:
            conn_handle, _, _ = data
            self._connections.add(conn_handle)
        elif event == _IRQ_CENTRAL_DISCONNECT:
            conn_handle, _, _ = data
            if conn_handle in self._connections:
                self._connections.remove(conn_handle)
            # Start advertising again to allow a new connection.
            self._advertise()
        elif event == _IRQ_GATTS_WRITE:
            conn_handle, value_handle = data
            if conn_handle in self._connections and value_handle == self._rx_handle:
                self._rx_buffer += self._ble.gatts_read(self._rx_handle)
                if self._handler:
                    self._handler()

    def any(self):
        return len(self._rx_buffer)

    def read(self, sz=None):
        if not sz:
            sz = len(self._rx_buffer)
        result = self._rx_buffer[0:sz]
        self._rx_buffer = self._rx_buffer[sz:]
        return result

    def write(self, data):
        for conn_handle in self._connections:
            self._ble.gatts_notify(conn_handle, self._tx_handle, data)

    def close(self):
        for conn_handle in self._connections:
            self._ble.gap_disconnect(conn_handle)
        self._connections.clear()

    def _advertise(self, interval_us=500000):
        self._ble.gap_advertise(interval_us, adv_data=self._payload)


def nav():

    ble = bluetooth.BLE()
    uart = BLEUART(ble)

    def on_rx():
        direct = uart.read().decode().strip()
        update_display(direct)
        print("rx: ", direct)

    uart.irq(handler=on_rx)
    try:
        while True:
            # uart.write(uart.read().decode().strip() + "\n")
            time.sleep_ms(1000)
    except KeyboardInterrupt:
        pass

    uart.close()


if __name__ == "__main__":
    nav()

# Helpers for generating BLE advertising payloads.

# A more fully-featured (and easier to use) version of this is implemented in
# aioble. This code is provided just as a basic example. See
# https://github.com/micropython/micropython-lib/tree/master/micropython/bluetooth/aioble

from micropython import const
import struct
import bluetooth

# Advertising payloads are repeated packets of the following form:
#   1 byte data length (N + 1)
#   1 byte type (see constants below)
#   N bytes type-specific data

_ADV_TYPE_FLAGS = const(0x01)
_ADV_TYPE_NAME = const(0x09)
_ADV_TYPE_UUID16_COMPLETE = const(0x3)
_ADV_TYPE_UUID32_COMPLETE = const(0x5)
_ADV_TYPE_UUID128_COMPLETE = const(0x7)
_ADV_TYPE_UUID16_MORE = const(0x2)
_ADV_TYPE_UUID32_MORE = const(0x4)
_ADV_TYPE_UUID128_MORE = const(0x6)
_ADV_TYPE_APPEARANCE = const(0x19)

_ADV_MAX_PAYLOAD = const(31)


# Generate a payload to be passed to gap_advertise(adv_data=...).
def advertising_payload(limited_disc=False, br_edr=False, name=None, services=None, appearance=0):
    payload = bytearray()

    def _append(adv_type, value):
        nonlocal payload
        payload += struct.pack("BB", len(value) + 1, adv_type) + value

    _append(
        _ADV_TYPE_FLAGS,
        struct.pack("B", (0x01 if limited_disc else 0x02) + (0x18 if br_edr else 0x04)),
    )

    if name:
        _append(_ADV_TYPE_NAME, name)

    if services:
        for uuid in services:
            b = bytes(uuid)
            if len(b) == 2:
                _append(_ADV_TYPE_UUID16_COMPLETE, b)
            elif len(b) == 4:
                _append(_ADV_TYPE_UUID32_COMPLETE, b)
            elif len(b) == 16:
                _append(_ADV_TYPE_UUID128_COMPLETE, b)

    # See org.bluetooth.characteristic.gap.appearance.xml
    if appearance:
        _append(_ADV_TYPE_APPEARANCE, struct.pack("<h", appearance))

    if len(payload) > _ADV_MAX_PAYLOAD:
        raise ValueError("advertising payload too large")

    return payload


def decode_field(payload, adv_type):
    i = 0
    result = []
    while i + 1 < len(payload):
        if payload[i + 1] == adv_type:
            result.append(payload[i + 2 : i + payload[i] + 1])
        i += 1 + payload[i]
    return result


def decode_name(payload):
    n = decode_field(payload, _ADV_TYPE_NAME)
    return str(n[0], "utf-8") if n else ""


def decode_services(payload):
    services = []
    for u in decode_field(payload, _ADV_TYPE_UUID16_COMPLETE):
        services.append(bluetooth.UUID(struct.unpack("<h", u)[0]))
    for u in decode_field(payload, _ADV_TYPE_UUID32_COMPLETE):
        services.append(bluetooth.UUID(struct.unpack("<d", u)[0]))
    for u in decode_field(payload, _ADV_TYPE_UUID128_COMPLETE):
        services.append(bluetooth.UUID(u))
    return services


def demo():
    payload = advertising_payload(
        name="micropython",
        services=[bluetooth.UUID(0x181A), bluetooth.UUID("6E400001-B5A3-F393-E0A9-E50E24DCCA9E")],
    )
    print(payload)
    print(decode_name(payload))
    print(decode_services(payload))


if __name__ == "__main__":
    demo()

Credits

David Barrett

4 projects • 3 followers

I have a Bachelor of Technology (Computer Technology) from Latrobe University and I keep myself up to date with the latest technology.

Comments

Awards

Honorable Mention Visual Impairments

Build2gether 2.0 — Inclusive Innovation Challenge

Navigation Assistance for Visually Impared