Face/head module and sensor
Face obstacle detection
Abdomen/Lower modules and sensors
All Modules
Mounts
Dynamic Vision Assist Object Recognition and Optical Character Recognition
Blindfolded navigation demo
Conclusion

Published September 5, 2024

Dynamic Vision Assist

The Dynamic Vision Assist is a wearable that is designed to allow a user with low to no vision navigate trails, parks, and hiking paths.

AdvancedFull instructions providedOver 6 days336

GRAND PRIZE VISUAL IMPAIRMENTS

Build2gether 2.0 — Inclusive Innovation Challenge

Things used in this project

Hardware components

Raspberry Pi 5

Runs Debian with DVA scripts.

OpenCV AI Kit: OAK-D

For Object Recognition and Optical Character Recognition

Blues Notecard (Cellular)

To monitor GPS coordinates of wearer and push them to the cloud.

M5Stack M5StickC ESP32-PICO Mini IoT Development Board

To control vibration motors for haptic feedback and to manage data from ToF sensors

M5Stack Core2 ESP32 IoT Development Kit

To receive tactile feedback from the m5stack mechanical key buttons and send requests respective DVA API endpoints determined by 1 of 3 mechanical keys

M5Stack Mechanical Key Button Unit

To communicate with the M5stack Core2 for the purpose of switching from object recognition (yolov4), OCR (tesseract), and to mute the NNs.

Blues Notecarrier A

To support the Blues Notecard Cellular

M5Stack Unbuckled Grove Cable 1m/2m/50cm/20cm/10cm

To connect ToF sensors to the M5stack StickC modules as well as to connect the mechanical keys to the M5stack Core2.

M5Stack HAT-Vibrator

To vibrate with varying intensities depending on the ToF sensors.

GL Inet 300M Smart Router

To connect the Raspberry pi 5 to the M5stack Core2 tactile NN selector.

Portable power bank

To power the Raspberry pi 5 and the Oak-D camera

Velcro Cable Ties

Cable management

Crossbody Bag

Waterproof bag to house Dynamic Vision Assist components.

Magnetic Ball Joints

To adjust ToF sensors that are magnetically attached to the Dynamic Vision Assist or DVA housing.

Magnet Mount Kit

Qaekie Bone Conducting Headphones

4 port usb hub

m5stack Magnetic USBc charger

Solar Panel, 2.5 W

Software apps and online services

Raspberry Pi Raspbian

Blues Notehub.io

IFTTT Maker service

Hand tools and fabrication machines

3D Printer (generic)

Ender 3 pro.

Story

Dynamic Vision Assist

The Dynamic Vision Assist is a wearable tool for individuals who have visual impairments and wish to navigate outdoor areas like parks, hiking trails, or walking paths with little to no assistance.

Summary

The Dynamic Vision Assist wearable core components consist of three M5stickCplus esp32 modules, an Oak-D ai camera, A Raspberry Pi 5, an M5stack Core2, and Blues Notehub Cellular Card.

unpacked wearable

It uses a combination of Haptic feedback based on distance of objects/obstacles near the wearer and neural networks to identify common objects. All distance data is translated to haptic feedback of varying intensities through the use of vibration motors. The Neural network classification data is transmitted to the user's ear using text to speech functionality and open-ear bone conducting headphones.

tof sensor with haptic feedback

bone conducting bluetooth headphoneses

These StickC modules gather data from infrared time of flight laser sensors.

tof sensor

These sensors are mounted on a cross shoulder bag/pouch.

tof sensors magnetically mounted on housing/bag

These modules translate distance data into haptic feedback via vibration hats. (vibration motor modules)

The face obstacle detection module and sensor is worn on upper strap of the Dynamic Vision Assist is angled upwards in order to detect any obstacles that would interfere with the wearer's head or face.

The stick modules are mounted on the cross body bag/housing for the Dynamic Vision Assist as outlined in the above photo. The two ground obstacle detection modules are worn below the face/head obstacle detection module. They are pointed angled towards the ground in a cross over configuration so that each module detects a line of sight in a cone from the user's center area.

The stick modules and the vibration units are mounted on the back of the DVA bag/housing.

StickC mounted on back of DVA wearable

DVA haptic feedback modules in action

A baseline ground distance calibration can be triggered by a long button push of button A (labeled m5) on each module.

The stick module sounds two beeps, the first lower then a higher frequency when this calibration is set. When the calibration is cleared by double pressing the A button, the tones are played in reverse.

Setting distance to ground by long pressing button A and double pressing button

This sets the baseline distance to the ground so that if the distance increases by more than 2 inches, the lower body vibration motors return a rapid on off vibration pattern alerting the user that there are steps or a steep decline ahead.

decline/edge detection and vibration pattern

To return a battery percentage on each of the three modules, the B button is pressed and the vibration motor on any of the modules returns a number of quick pulses in tenths to denote battery percentage.

b button

4 vibrational pulses denote that the battery life is at approximately 40 percent

If the button is pressed and the battery percentage is under 10 percent, the module will not vibrate but will beep.

Face/head module and sensor

1. Vibrates with full duty if an object is within 3-4 feet of a user's face or head area

Face obstacle detection

2. The sensor must be angled properly so that at 3-4 feet, the vibration alarm initiates if an obstacle is at face/head level

preferred sensor angle

Abdomen/Lower modules and sensors

1. Vibrates at different intensities when an obstacle is an front of the wearer on either hip or side of the body. This allows the wearer to determine which side of their body the obstacle is on.

Obstacle detection

2. Once the sensors are adjusted properly on the body, a long press of the A button on both/either sensor sets the baseline distance to the ground. This allows the wearer to detect decline/steps with a series of repetitive pulses

3. If the button A is pressed, the vibration motor returns battery percentages via vibration pulses in tenths

All Modules

1. Powered on by pressing the side button on the lower left hand button of the module. Power on is followed by two fast vibration pulses.

Power on

2. Powered off by long pressing the side button on the lower left hand button of the module.

3. The M5stickC module is easily connected to its respective time of flight sensor via a grove cable on port A

4. Easy charger attachment via m5stsck magnetic charging cable

port A grove connection, magnetic charging.

Mounts

Uses Magnetic Adjustable ball joints. Fully adjustable and customizable to the user's needs.

1. Face/Head sensor mount

2. Lower body abdomen sensors.

GPS Tracking. Included in the DVA wearable is a Blues Notecarrier along with a Blues Note cellular card. This module is configured to push GPS coordinates to the Blues Hub every 3 minutes. This allows an emergency contact to access last known GPS coordinates if the wearer encounters an issue. I have Successfully tested web-hooks with both IFTTT and Notehub to push GPS coordinates to an emergency contact. The user accessing Notehub has instant access to Google Maps with the wearer's pinpointed coordinates.

And last but not certainlynotleast.......

Dynamic Vision Assist Object Recognition and Optical Character Recognition

Raspberry pi 5 running dva api and neural networks. Uses an inexpensive power bank

The Dynamic Vision Assist leverages a Luxonis Oak-d pro AI camera and the Depth-AI Model Zoo by way of a Raspberry pi 5. A wearer is able to switch between Yolov4 and Tesseract models to learn more about their environment through Object and Optical Character Recognition.

1. WiFi Access point and Tactile neural network selector

M5stack Core2 neural network selector

Glinet portable access point.

Both the M5core2 tactile controller and the Raspberry pi 5 must be connected to the same GL inet network. Additionally, a user must set the correct Raspberry pi API endpoint ip address in the M5stack Micropython script I have included.

2. Object recognition mode.

The dynamic vision assist boots into object recognition mode and begins describing surrounding objects. This information is processed through text to speech and is then sent to the bone conducting headphones that the wearer is equipped with. This allows the user to receive critical information without losing hearing acuity.

Person Detection

DVA detecting a dog.

3. OCR mode

Once the user has decided they want to try OCR mode, they must push the middle mechanical key on the tactile controller

When the user presses the center key, an API request is sent to the DVA API that is running on the Raspberry Pi 5. This instructs the script to switch to Tesseract and begin optical character recognition.

OCR in action

This functionality allows the wearer to read signs and important trail information when hiking or navigating parks.

4. If the wearer chooses to mute neural networks/text to speech altogether, they can push the third and final button to stop neural network activity and mute the volume in the headphones.

5. When the user is ready to return to Object recognition, they will push the key on the opposite side and a unique tone will play while the green led illuminates.

Note:

Each key plays a unique tone when pushed and has a corresponding color denoted by a lit led for low vision users when the button has selected a specific neural network

Red=mute, Green=Object Recognition, White=OCR

Renewable Energy Charging

A solar panel can easily be detached in the case the user needs to charge any of the batteries or headphones needed for this wearable. This allows a user to recharge in a remote environment despite battery life being 8+ hours as it currently stands.

Charging headphones with solar panel

And now for the moment we have all been waiting for...

Intro

Discovering a tree

Walking through center park with wearable

Navigation between trees and onto path with inclines.

Conclusion

The dynamic vision assist is an effective solution to trail and park navigation. By tightening my scope to trail/park navigation, I was able to hone in on the needs that a wearer may have in this environment. Through the use of AI, microcontrollers, perseverance, and innovation, I believe the Dynamic Vision Assist is a boon for low to no vision users. Thanks to all the Judges and participants for their hard work. Together we can change the future.

Code

#!/bin/bash
# Log output to a file for debugging
exec &> /home/pi/start_ai_cam.log

# Pause for 5 seconds to ensure the environment is ready
sleep  5

# Print current date and time to the log
echo "Script started at: $(date)"

# Activate the virtual environment
source /home/pi/Desktop/luxonis/envDepthAI/bin/activate

# Run the Python script
python /home/pi/Desktop/autorun_ai_cam_script/DVA_API.py

# Print script completion to the log
echo "Script completed at: $(date)"

from m5stack import *
from m5ui import *
from uiflow import *
import time
import i2c_bus
from easyIO import *
import unit
import hat


setScreenColor(0x111111)
tof_0 = unit.get(unit.TOF, unit.PORTA)

hat_vibrator_0 = hat.get(hat.VIBRATOR)

distance = None
calibrate = None
batt = None



label1 = M5TextBox(7, 99, "label1", lcd.FONT_DejaVu40, 0xFFFFFF, rotate=0)


# Describe this function...
def hip_distance():
  global distance, calibrate, batt, i2c0
  distance = tof_0.distance
  if calibrate == None:
    if distance==distance < 400:
      hat_vibrator_0.set_duty(100)
    elif distance==distance < 800:
      hat_vibrator_0.set_duty(80)
    elif distance==distance < 1200:
      hat_vibrator_0.set_duty(60)
    elif distance==distance < 1600:
      hat_vibrator_0.set_duty(40)
    elif distance==distance < 1800:
      hat_vibrator_0.set_duty(30)
    elif distance==distance < 2000:
      hat_vibrator_0.set_duty(20)
    else:
      hat_vibrator_0.turn_off()
  elif calibrate != None:
    if distance==distance >= calibrate + 101:
      hat_vibrator_0.set_duty(100)
      wait_ms(100)
      hat_vibrator_0.set_duty(0)
      wait_ms(50)
    elif distance==distance < 400:
      hat_vibrator_0.set_duty(100)
    elif distance==distance < 800:
      hat_vibrator_0.set_duty(80)
    elif distance==distance < 1200:
      hat_vibrator_0.set_duty(60)
    elif distance==distance < 1600:
      hat_vibrator_0.set_duty(40)
    elif distance==distance < 1800:
      hat_vibrator_0.set_duty(30)
    elif distance==distance < 2000:
      hat_vibrator_0.set_duty(20)
    else:
      hat_vibrator_0.turn_off()

# Describe this function...
def one_pulse():
  global distance, calibrate, batt, i2c0
  hat_vibrator_0.set_duty(50)
  wait_ms(100)
  hat_vibrator_0.turn_off()
  wait_ms(50)

# Describe this function...
def two_pulse():
  global distance, calibrate, batt, i2c0
  hat_vibrator_0.set_duty(50)
  wait_ms(100)
  hat_vibrator_0.turn_off()
  wait_ms(50)
  hat_vibrator_0.set_duty(50)
  wait_ms(100)
  hat_vibrator_0.turn_off()
  wait_ms(50)


def buttonA_pressFor():
  global distance, calibrate, batt, i2c0
  hat_vibrator_0.turn_off()
  label1.setColor(0xcc33cc)
  calibrate = distance
  axp.setLcdBrightness(100)
  speaker.tone(800, 200)
  speaker.tone(1200, 200)
  axp.setLcdBrightness(0)
  pass
btnA.pressFor(0.8, buttonA_pressFor)

def buttonB_wasPressed():
  global distance, calibrate, batt, i2c0
  hat_vibrator_0.turn_off()
  if batt==batt < 10:
    hat_vibrator_0.turn_off()
    speaker.setVolume(25)
    speaker.tone(900, 200)
  elif batt==batt < 20:
    one_pulse()
  elif batt==batt < 30:
    two_pulse()
  elif batt==batt < 40:
    one_pulse()
    two_pulse()
  elif batt==batt < 50:
    two_pulse()
    two_pulse()
  elif batt==batt < 60:
    two_pulse()
    two_pulse()
    one_pulse()
  elif batt==batt < 70:
    two_pulse()
    two_pulse()
    two_pulse()
  elif batt==batt < 80:
    two_pulse()
    two_pulse()
    two_pulse()
    one_pulse()
  elif batt==batt < 90:
    two_pulse()
    two_pulse()
    two_pulse()
    two_pulse()
  elif batt==batt < 95:
    two_pulse()
    two_pulse()
    two_pulse()
    two_pulse()
    one_pulse()
  elif batt==batt <= 100:
    two_pulse()
    two_pulse()
    two_pulse()
    two_pulse()
    two_pulse()
  else:
    hat_vibrator_0.turn_off()
  pass
btnB.wasPressed(buttonB_wasPressed)

def buttonA_wasDoublePress():
  global distance, calibrate, batt, i2c0
  hat_vibrator_0.turn_off()
  label1.setColor(0xcc33cc)
  calibrate = None
  label1.setText('clear')
  axp.setLcdBrightness(100)
  speaker.tone(1200, 200)
  speaker.tone(800, 200)
  axp.setLcdBrightness(0)
  pass
btnA.wasDoublePress(buttonA_wasDoublePress)


speaker.setVolume(50)
hat_vibrator_0.set_duty(0)
axp.setLcdBrightness(0)
i2c0 = i2c_bus.easyI2C(i2c_bus.PORTA, 0x00, freq=400000)
label1.setColor(0x000000)
hat_vibrator_0.set_duty(100)
wait_ms(100)
hat_vibrator_0.set_duty(0)
wait_ms(50)
hat_vibrator_0.set_duty(100)
wait_ms(100)
hat_vibrator_0.set_duty(0)
while True:
  batt = map_value((axp.getBatVoltage()), 3.7, 4.1, 0, 100)
  hip_distance()
  label1.setText(str(calibrate))
  wait_ms(2)

# curl -X POST http://localhost:5000/switch_endpoint -H "Content-Type: application/json" -d '{"endpoint": "OR_OA"}'
# curl -X POST http://localhost:5000/switch_endpoint -H "Content-Type: application/json" -d '{"endpoint": "OCR"}'
# curl -X POST http://localhost:5000/switch_endpoint -H "Content-Type: application/json" -d '{"endpoint": "IDLE"}'


import os
import cv2
import depthai as dai
import numpy as np
import requests
import pytesseract
from flask import Flask, request, jsonify
from threading import Thread
from gtts import gTTS
import pygame
import time

# Start with object detection instead of OCR
current_endpoint = "OR_OA"

# Model path
model_path = "/home/pi/Desktop/luxonis/depthai-python/examples/models/yolo-v4-tiny-tf_openvino_2021.4_6shave.blob"

# Class labels for Tiny YOLOv4
class_labels = [
    "person", "bicycle", "car", "motorcycle", "airplane", "bus", "train", "truck", "boat", "traffic light",
    "fire hydrant", "stop sign", "parking meter", "bench", "bird", "cat", "dog", "horse", "sheep", "cow",
    "elephant", "bear", "zebra", "giraffe", "backpack", "umbrella", "handbag", "tie", "suitcase", "frisbee",
    "skis", "snowboard", "sports ball", "kite", "baseball bat", "baseball glove", "skateboard", "surfboard",
    "tennis racket", "bottle", "wine glass", "cup", "fork", "knife", "spoon", "bowl", "banana", "apple",
    "sandwich", "orange", "broccoli", "carrot", "hot dog", "pizza", "donut", "cake", "chair", "couch",
    "potted plant", "bed", "dining table", "toilet", "tv", "laptop", "mouse", "remote", "keyboard",
    "cell phone", "microwave", "oven", "toaster", "sink", "refrigerator", "book", "clock", "vase",
    "scissors", "teddy bear", "hair drier", "toothbrush"
]

# Initialize Flask app
app = Flask(__name__)

# Dictionary to keep track of the last time each label was spoken
last_spoken_time = {label: 0 for label in class_labels}

@app.route('/switch_endpoint', methods=['POST'])
def switch_endpoint():
    global current_endpoint
    new_endpoint = request.json.get('endpoint')
    if new_endpoint in ["OCR", "OR_OA", "IDLE"]:
        current_endpoint = new_endpoint
        return jsonify({"status": "success", "current_endpoint": current_endpoint})
    else:
        return jsonify({"status": "error", "message": "Invalid endpoint"}), 400

@app.route('/ocr', methods=['POST'])
def handle_ocr():
    data = request.json
    text = data.get("ocr_results", "")
    if text.strip():
        print(f"OCR Results: {text}")  # Logging OCR results for debugging
        send_to_gtts(text)
    return jsonify({"status": "success"})

@app.route('/or_oa', methods=['POST'])
def handle_or_oa():
    data = request.json
    results = data.get("or_oa_results", [])
    if results:
        print(f"OR_OA Results: {results}")  # Logging OR_OA results for debugging
        labels = [class_labels[result['label']] for result in results if result['label'] < len(class_labels) and result['label'] is not None]
        current_time = time.time()
        for label in labels:
            if current_time - last_spoken_time[label] > 5:
                print(f"Detected: {label}")  # Logging detected labels for debugging
                send_to_gtts(label)
                last_spoken_time[label] = current_time
    return jsonify({"status": "success"})

def send_to_gtts(text):
    tts = gTTS(text)
    tts.save("response.mp3")
    pygame.mixer.init()
    pygame.mixer.music.load("response.mp3")
    pygame.mixer.music.play()
    while pygame.mixer.music.get_busy():
        pygame.time.Clock().tick(10)
    os.remove("response.mp3")

# Initialize pipeline with both OCR and Object Detection
def create_pipeline():
    pipeline = dai.Pipeline()

    # Create and configure color camera node
    cam_rgb = pipeline.create(dai.node.ColorCamera)
    cam_rgb.setPreviewSize(416, 416)
    cam_rgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)
    cam_rgb.setInterleaved(False)
    cam_rgb.setBoardSocket(dai.CameraBoardSocket.CAM_A)
    cam_rgb.setColorOrder(dai.ColorCameraProperties.ColorOrder.BGR)
    
    # Adjust camera settings for better brightness and enable auto-exposure
    control = dai.CameraControl()
    control.setAutoExposureEnable()
    control.setAutoWhiteBalanceMode(dai.CameraControl.AutoWhiteBalanceMode.AUTO)
    control.setAutoExposureCompensation(2)  # Increase exposure compensation
    cam_rgb.initialControl.setManualExposure(20000, 800)
    cam_rgb.initialControl.setAutoWhiteBalanceMode(dai.CameraControl.AutoWhiteBalanceMode.WARM_FLUORESCENT)

    # Create XLink output for video
    xout_video = pipeline.create(dai.node.XLinkOut)
    xout_video.setStreamName("video")
    cam_rgb.preview.link(xout_video.input)

    # Create ImageManip node to resize the image
    manip = pipeline.create(dai.node.ImageManip)
    manip.initialConfig.setResize(416, 416)
    manip.initialConfig.setFrameType(dai.RawImgFrame.Type.BGR888p)
    cam_rgb.preview.link(manip.inputImage)

    # Initialize object detection
    object_detection_nn = pipeline.create(dai.node.YoloDetectionNetwork)
    object_detection_nn.setBlobPath(model_path)
    object_detection_nn.setConfidenceThreshold(0.5)
    object_detection_nn.setNumClasses(80)  # Adjust based on your model
    object_detection_nn.setCoordinateSize(4)
    
    # Add appropriate anchors
    anchors = [
        10, 14, 23, 27, 37, 58, 81, 82, 135, 169, 344, 319
    ]
    object_detection_nn.setAnchors(anchors)
    
    object_detection_nn.setAnchorMasks({"side26": [1, 2, 3], "side13": [3, 4, 5]})
    object_detection_nn.setIouThreshold(0.5)
    
    manip.out.link(object_detection_nn.input)

    xout_nn = pipeline.create(dai.node.XLinkOut)
    xout_nn.setStreamName("detections")
    object_detection_nn.out.link(xout_nn.input)
    
    return pipeline

# Function to handle OCR
def perform_ocr(frame):
    return pytesseract.image_to_string(frame)

# Function to perform object recognition and obstacle avoidance
def perform_object_recognition_and_obstacle_avoidance(detections):
    detection_results = detections.detections
    return [{"label": det.label, "confidence": det.confidence, "bbox": (det.xmin, det.ymin, det.xmax, det.ymax)} for det in detection_results]

# Main processing function
def main():
    pipeline = create_pipeline()

    with dai.Device(pipeline) as device:
        video = device.getOutputQueue(name="video", maxSize=4, blocking=False)
        detections = device.getOutputQueue(name="detections", maxSize=4, blocking=False)
        
        while True:
            video_frame = video.get()
            frame = video_frame.getCvFrame()

            if current_endpoint == "OCR":
                ocr_results = perform_ocr(frame)
                if ocr_results.strip():  # Only send non-empty OCR results
                    requests.post('http://localhost:5000/ocr', json={"ocr_results": ocr_results})
            
            elif current_endpoint == "OR_OA":
                dets = detections.get()
                or_oa_results = perform_object_recognition_and_obstacle_avoidance(dets)
                if or_oa_results:
                    requests.post('http://localhost:5000/or_oa', json={"or_oa_results": or_oa_results})
                    # Read detected labels
                    labels = [class_labels[result['label']] for result in or_oa_results if result['label'] < len(class_labels) and result['label'] is not None]
                    current_time = time.time()
                    for label in labels:
                        if current_time - last_spoken_time[label] > 5:
                            print(f"Detected: {label}")  # Logging detected labels for debugging
                            send_to_gtts(label)
                            last_spoken_time[label] = current_time

                    # Draw bounding boxes and labels on the frame
                    for detection in or_oa_results:
                        bbox = detection['bbox']
                        label = class_labels[detection['label']]
                        confidence = detection['confidence']
                        cv2.rectangle(frame, (int(bbox[0] * frame.shape[1]), int(bbox[1] * frame.shape[0])), 
                                      (int(bbox[2] * frame.shape[1]), int(bbox[3] * frame.shape[0])), (0, 255, 0), 2)
                        cv2.putText(frame, f"{label}: {confidence:.2f}", (int(bbox[0] * frame.shape[1]), int(bbox[1] * frame.shape[0]) - 10), 
                                    cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0), 2)

            elif current_endpoint == "IDLE":
                # If in IDLE mode, do nothing
                pass

            # Display the frame
            cv2.imshow("Video", frame)

            if cv2.waitKey(1) == ord('q'):
                break

            # Add a small delay to reduce the frequency of requests
            time.sleep(0.1)  # Reduced delay to improve responsiveness

    cv2.destroyAllWindows()

if __name__ == "__main__":
    # Start the Flask server in a separate thread
    flask_thread = Thread(target=lambda: app.run(host='0.0.0.0', port=5000, debug=True, use_reloader=False))
    flask_thread.start()

    # Run the main pipeline function
    main()

from m5stack import *
from m5stack_ui import *
from uiflow import *
import wifiCfg
import time
import urequests
import unit


screen = M5Screen()
screen.clean_screen()
screen.set_screen_bg_color(0x000000)
key_0 = unit.get(unit.KEY, unit.PORTA)
key_1 = unit.get(unit.KEY, unit.PORTB)
key_2 = unit.get(unit.KEY, unit.PORTC)






label0 = M5Label('Dynamic', x=75, y=20, color=0xff02ec, font=FONT_MONT_38, parent=None)
label1 = M5Label('Vision', x=103, y=95, color=0xff02ec, font=FONT_MONT_38, parent=None)
label2 = M5Label('Assist', x=106, y=160, color=0xff02ec, font=FONT_MONT_38, parent=None)


wifiCfg.doConnect('YOUR SSID', 'YOUR PASSWORD')
key_0.set_color(0xff0000, 1)
key_1.set_color(0x33ff33, 1)
key_2.set_color(0xffffff, 1)
screen.set_screen_brightness(30)
wait(3)
label0.set_hidden(True)
label1.set_hidden(True)
label2.set_hidden(True)
while True:
  if (key_0.get_switch_status()) == 0:
    try:
      req = urequests.request(method='POST', url='http://192.168.1.241:5000/switch_endpoint',json={'endpoint':'IDLE'}, headers={'Content-Type':'application/json'})
      gc.collect()
      req.close()
    except:
      pass
    screen.set_screen_bg_color(0xff0000)
    label0.set_text_color(0x000000)
    key_0.set_color(0xff0000, 100)
    key_1.set_color(0x33ff33, 1)
    key_2.set_color(0xffffff, 1)
    speaker.playTone(523, 1/4, volume=4)
    speaker.playTone(392, 1/2, volume=4)
  if (key_1.get_switch_status()) == 0:
    try:
      req = urequests.request(method='POST', url='http://192.168.1.241:5000/switch_endpoint',json={'endpoint':'OR_OA'}, headers={'Content-Type':'application/json'})
      gc.collect()
      req.close()
    except:
      pass
    screen.set_screen_bg_color(0x33ff33)
    label0.set_text_color(0x33ff33)
    key_1.set_color(0x33ff33, 100)
    key_2.set_color(0xff0000, 1)
    key_0.set_color(0xffffff, 1)
    speaker.playTone(440, 1/2, volume=4)
  if (key_2.get_switch_status()) == 0:
    try:
      req = urequests.request(method='POST', url='http://192.168.1.241:5000/switch_endpoint',json={'endpoint':'OCR'}, headers={'Content-Type':'application/json'})
      gc.collect()
      req.close()
    except:
      pass
    screen.set_screen_bg_color(0xffffff)
    label0.set_text_color(0xffffff)
    key_2.set_color(0xffffff, 100)
    key_1.set_color(0x33ff33, 1)
    key_0.set_color(0xff0000, 1)
    speaker.playTone(587, 1/2, volume=4)
  wait_ms(2)

from m5stack import *
from m5ui import *
from uiflow import *
import i2c_bus
import time
from easyIO import *
import unit
import hat


setScreenColor(0x111111)
tof4m_0 = unit.get(unit.TOF4M, unit.PORTA)

hat_vibrator_0 = hat.get(hat.VIBRATOR)

distance = None
batt = None



label1 = M5TextBox(13, 39, "Dynamic", lcd.FONT_DejaVu24, 0xe600ff, rotate=0)
label2 = M5TextBox(29, 134, "Assist", lcd.FONT_DejaVu24, 0x9900ff, rotate=0)
label3 = M5TextBox(46, 186, "1.0", lcd.FONT_DejaVu24, 0x0041ff, rotate=0)
label0 = M5TextBox(29, 86, "Vision", lcd.FONT_DejaVu24, 0x4d00ff, rotate=0)


# Describe this function...
def head_distance():
  global distance, batt, i2c0
  distance = tof4m_0.get_single_distance_value
  if distance==distance < 1000:
    hat_vibrator_0.set_duty(100)
  else:
    hat_vibrator_0.turn_off()

# Describe this function...
def one_pulse():
  global distance, batt, i2c0
  hat_vibrator_0.set_duty(50)
  wait_ms(100)
  hat_vibrator_0.turn_off()
  wait_ms(50)

# Describe this function...
def two_pulse():
  global distance, batt, i2c0
  hat_vibrator_0.set_duty(50)
  wait_ms(100)
  hat_vibrator_0.turn_off()
  wait_ms(50)
  hat_vibrator_0.set_duty(50)
  wait_ms(100)
  hat_vibrator_0.turn_off()
  wait_ms(50)


def buttonA_wasPressed():
  global distance, batt, i2c0
  hat_vibrator_0.turn_off()
  if batt==batt < 10:
    hat_vibrator_0.turn_off()
    speaker.setVolume(25)
    speaker.tone(900, 200)
  elif batt==batt < 20:
    one_pulse()
  elif batt==batt < 30:
    two_pulse()
  elif batt==batt < 40:
    one_pulse()
    two_pulse()
  elif batt==batt < 50:
    two_pulse()
    two_pulse()
  elif batt==batt < 60:
    two_pulse()
    two_pulse()
    one_pulse()
  elif batt==batt < 70:
    two_pulse()
    two_pulse()
    two_pulse()
  elif batt==batt < 80:
    two_pulse()
    two_pulse()
    two_pulse()
    one_pulse()
  elif batt==batt < 90:
    two_pulse()
    two_pulse()
    two_pulse()
    two_pulse()
  elif batt==batt < 95:
    two_pulse()
    two_pulse()
    two_pulse()
    two_pulse()
    one_pulse()
  elif batt==batt <= 100:
    two_pulse()
    two_pulse()
    two_pulse()
    two_pulse()
    two_pulse()
  else:
    hat_vibrator_0.turn_off()
  pass
btnA.wasPressed(buttonA_wasPressed)


axp.setLcdBrightness(0)
i2c0 = i2c_bus.easyI2C(i2c_bus.PORTA, 0x00, freq=400000)
hat_vibrator_0.turn_off()
axp.setLcdBrightness(50)
wait(3)
axp.setLcdBrightness(0)
hat_vibrator_0.set_duty(100)
wait_ms(100)
hat_vibrator_0.set_duty(0)
wait_ms(50)
hat_vibrator_0.set_duty(100)
wait_ms(100)
hat_vibrator_0.set_duty(0)
while True:
  batt = map_value((axp.getBatVoltage()), 3.7, 4.1, 0, 100)
  head_distance()
  wait_ms(2)

Credits

Colonel Panic

2 projects • 5 followers

Heckr

Thanks to Friedelino and Caban.

Comments

Awards

GRAND PRIZE VISUAL IMPAIRMENTS

Build2gether 2.0 — Inclusive Innovation Challenge

Dynamic Vision Assist

Things used in this project

Hardware components

Software apps and online services

Hand tools and fabrication machines

Story

Face/head module and sensor

Face obstacle detection

Abdomen/Lower modules and sensors

All Modules

Mounts

Dynamic Vision Assist Object Recognition and Optical Character Recognition

Blindfolded navigation demo

Conclusion

Schematics

Stick C Wiring

core2

m5 core2 wires

StickC Wiring

20240904_181104_GBAMYpk7gM.jpg

project without housing

notehub

core2

magnetic charger

Code

DVA_API startup bash script.

Hip haptic feedback unit

Raspberry pi DVA API

DVA API tactile controller

Head haptic feedback unit

Credits

Colonel Panic

Comments

Awards

Embed the widget on your own site

Dynamic Vision Assist

Dynamic Vision Assist

Things used in this project

Hardware components

Software apps and online services

Hand tools and fabrication machines

Story

Face/head module and sensor

Face obstacle detection

Abdomen/Lower modules and sensors

All Modules

Mounts

Dynamic Vision Assist Object Recognition and Optical Character Recognition

Blindfolded navigation demo

Conclusion

Schematics

Stick C Wiring

core2

m5 core2 wires

StickC Wiring

20240904_181104_GBAMYpk7gM.jpg

project without housing

notehub

core2

magnetic charger

Code

DVA_API startup bash script.

Hip haptic feedback unit

Raspberry pi DVA API

DVA API tactile controller

Head haptic feedback unit

Credits

Colonel Panic

Comments

Awards

Related channels and tags