Created September 4, 2024

Home Assist v0.1

Smart voice assistant with capability to locate everyday objects through voice and auditory feedback for visually impaired individuals

Honorable Mention Visual Impairments

Build2gether 2.0 — Inclusive Innovation Challenge

Things used in this project

Hardware components

M5Stack ATOMS3 Lite

DFRobot UNIHIKER - IoT Python Programming Single Board Computer with Touchscreen

Seeed Studio XIAO ESP32S3 Sense

Buzzer

Software apps and online services

Microsoft VS Code

Espressif esp-idf

Jupyter Notebook

MQTT

Story

1. Introduction

This project aims to assist visually impaired individuals in locating everyday objects like keys using auditory feedback. The system comprises of a Unihiker Linux board and an ESP32-based microcontroller (M5Stack Atom S3 Lite or Seeed Studio Xiao S3) with buzzer interfaced onboard. The Unihiker board is equipped with a touchscreen which runs a Python script that listens to voice commands via the Google Speech Recognition API and sends corresponding messages to an MQTT broker using paho-mqtt library. This triggers periodic buzzer on ESP32 controller which aids in locating the object for visually impaired persons. Upon locating the target object, user can press button to reset the buzzer. Unihiker also provides built-in GUI library which is used to provide a visual interface. However, system can be fully operated through voice commands and auditory feedback. Below is the demonstration of the project.

Demo Video

2. System Overview

The system architecture involves several components working together:

User Interface (UI): A touchscreen on the Unihiker board for basic feedback, combined with voice input and built-in audio.
Speech Recognition: The Unihiker board listens to user commands and converts spoken words into text using the Google Speech Recognition API.
MQTT Communication: The system communicates through MQTT protocol, sending commands to an open MQTT broker (MQTT Cool) that relays them to the subscribed ESP32 microcontroller. The command is encoded and sent in json format.
Auditory Feedback: Upon receiving a valid command, the ESP32 decodes json packet and triggers a buzzer if command matches keyword associated with device. This helps the user locate their items. A physical button on the ESP32 device allows users to stop the buzzer once the object is located.

System Diagram:

system overview

3. Components Used

3.1 Hardware Components

Unihiker Board:

A Linux-based board that supports Python scripting and includes a touchscreen for user interaction.
Runs speech recognition using the Google Speech Recognition API.
Acts as the publisher in the MQTT protocol.

ESP32 Microcontroller (M5Stack Atom S3 Lite or Seeed Studio Xiao S3):

Compact and powerful microcontroller boards equipped with Wi-Fi and Bluetooth, used for receiving commands from the MQTT broker.
Controls a buzzer that provides auditory feedback.
Includes a button to reset the buzzer after locating the object.

Buzzer:

A simple auditory output device that beeps when activated by the ESP32 microcontroller. It is directly connected to ESP32 board.

3.2 Software Components

Python (on Unihiker):

Used to run the primary application script that listens for user commands, processes them, and sends messages to the MQTT broker. Unihiker can run jupyter notebooks. All code for assistant is contained inside jupyter notebook.

Google Speech Recognition API:

Transforms voice input into text, enabling the system to recognize commands such as "Find keys" or "Locate phone."

MQTT Broker (MQTT Cool):

Acts as a middleman to relay messages from the Unihiker board to the ESP32-based devices. The broker is hosted on the public MQTT Cool platform. Note:- It is a public MQTT broker and should only be used for testing purposes as it is not secure at all and all the topics can be accessed by anyone on internet.

ESP-IDF Toolchain:

It is native IoT development toolchain provided by Espressif for their ESP32 series of microcontroller boards. It provides a FreeRTOS based environment in C / C++ for embedded development. All functionality for ESP32 boards is implemented in esp-idf. MQTT broker is subscribed at ESP32 end and it listens for incoming data stream, decode JSON string to extract commands and on successful match produces a buzzer output.

4. Construction and Assembly

4.1 Unihiker Board Setup

Initializing the Board:

Connect the board with PC through USB cable provided and wait for the board to boot.
After booting, you'll see UNIHIKER logo on screen of the board. Now open browser in your PC and enter following url: http://10.1.2.3
After loading the website, go to Network Settings tab and enter credentials to connect to your network.
Now, go to Service Toggle tab and then enable Jupyter Notebook and after that click on open page.
Now, upload jupyter notebook and assets in their respective directories. This should look like this:

jupyter notebook directory structure

Now open jupyter notebook.
Install all necessary Python dependencies such as paho-mqtt for MQTT communication and speech_recognition for voice processing.
After resolving all dependencies, run voice home assist main script. You should see this UI:

Home Assist mainscreen

4.2 MQTT Broker Setup

MQTT Cool Broker Setup:

Go to https://testclient-cloud.mqtt.cool/ and setup a broker.
Subscribe to the topic /test/qos0

4.3 ESP32 Setup

ATOMS3 Lite or XIAO S3:

Interface buzzer with your controller according to following schematic:

m5stack atoms3 lite wiring diagram

Adjust to same pins if you are using seedstudio XIAO S3. The pins used are 38 and Gnd.
Go to https://docs.espressif.com/projects/esp-idf/en/stable/esp32/get-started/index.html and install and setup esp-idf extension in Vscode following instructions on website.
Clone code from github in a suitable location.
Connect ESP32 device using USB to your PC.
In IDF terminal run command: idf.py menuconfig and configure connection configuration according to your WiFi and save.
Run idf.py build flash command in terminal and wait for IDF to flash the device.

Python Script:

Write the Python code to handle speech recognition and MQTT publishing. The script continuously listens for user commands, processes them, and sends the appropriate MQTT messages.
Set up the touchscreen UI to provide basic feedback (e.g., "Listening..." or "Command received").
Python Script:
Write the Python code to handle speech recognition and MQTT publishing. The script continuously listens for user commands, processes them, and sends the appropriate MQTT messages.
Set up the touchscreen UI to provide basic feedback (e.g., "Listening..." or "Command received").

MQTT Broker Connection:

Configure the Unihiker to connect to the public MQTT broker (MQTT Cool). Ensure that it publishes messages under a specific topic (e.g., /object-locator/find).
MQTT Broker Connection:
Configure the Unihiker to connect to the public MQTT broker (MQTT Cool). Ensure that it publishes messages under a specific topic (e.g., /object-locator/find).

5. System Functionality

Setup and power-up all the devices following above steps.
Say "HELLO" infront of UNIHIKER. It will go in listening mode with a buzzer beep and screen will change to following:

Home Assist Listening

Now, say a command. In our case, say "find keys". It will also show recognised command on screen and if command is valid it will send it to ESP32 and it will start beeping.
After finding ESP32 device, press pushbutton on device to reset buzzer.

Conclusion:

As we saw in demonstration above, Home Assist is an effective hand-off way to locate common household items for visually impaired individuals through auditory feedback. In future, it may be extended to other use cases too such as direction finding e.t.c.

Product Refrences:

Code (ESP32):

Note: I wasnt able to push code to github or make refactoring changes in time therefore i am providing ESP32 code here as Google Drive link. In future, I'll post Github link with refactored code if i get some time to do it. The code for UNIHIKER is given in usual code section.

https://drive.google.com/drive/folders/1eVpQvLbFc9TO9Z1OhR-GdyeB7psScPkC?usp=drive_link

Home Assist UNIHIKER jupyter notebook code

import time
import json
import speech_recognition as sr
from pinpong.board import Board, Pin
from pinpong.extension.unihiker import *
from unihiker import GUI
import paho.mqtt.client as mqtt

# MQTT configuration
broker = 'broker.mqtt.cool'  # MQTT broker address
port = 1883  # MQTT broker port (usually 1883 for non-SSL)
topic = '/test/qos0'  # Replace with your desired topic

def create_json_string(data_string):
    # Create the dictionary with the given data string
    data = {
        'cmd': data_string
    }

    # Convert the dictionary to a JSON string
    json_str = json.dumps(data)
    return json_str

# Global client instance
client = None

def on_connect(client, userdata, flags, rc):
    print(f'Connected with result code {rc}')
    # Publish the JSON data once connected
    global json_data
    client.publish(topic, json_data)
    print(f'Published: {json_data} to topic: {topic}')

def start_mqtt():
    global client
    # Create a new MQTT client instance
    client = mqtt.Client()

    # Assign the callback function
    client.on_connect = on_connect

    # Connect to the MQTT broker
    client.connect(broker, port, 60)

    # Start the loop to process network traffic and dispatch callbacks
    client.loop_start()

def stop_mqtt():
    global client
    if client is not None:
        client.loop_stop()
        client.disconnect()

def send_mqtt_data(mqtt_data):
    global json_data
    json_data = create_json_string(mqtt_data)
    start_mqtt()
    time.sleep(2)
    stop_mqtt()
    
# Initialize the UNIHIKER
Board().begin()

# Instantiate the GUI class
gui = GUI()

def home_screen():
    #clear screen
    gui.fill_rect(x=0,y=0,w=240,h=320,color="white")
    # Load and show assistant image
    img_image = gui.draw_image(x=120, y=100, w=120, h=100, image='upload/bot.png', origin='center')
    #info_text = gui.draw_text(x=50, y=160, text='Home Assist v0.1')
    gui.draw_text(x=25, y=170, text='Home Assist v0.1', font_size=17, color="green")
    gui.draw_text(x=25 - 1, y=170, text='Home Assist v0.1', font_size=17, color="green")
    gui.draw_text(x=25 + 1, y=170, text='Home Assist v0.1', font_size=17, color="green")
    gui.draw_text(x=40, y=220, w=180, text='Say HELLO to wake device and then say       something...', font_size=11, color="grey")
    
def listening_screen():
    #clear screen
    gui.fill_rect(x=0,y=0,w=240,h=320,color="white")
    img_image = gui.draw_image(x=120, y=100, w=120, h=100, image='upload/assistant.png', origin='center')
    gui.draw_text(x = 120,y=190,text="Listening.....", font_size=18, color="black", angle=0, origin="center")
    gui.draw_text(x = 120 + 1,y=190,text="Listening.....", font_size=18, color="black", angle=0, origin="center")
    gui.draw_text(x = 120 - 1,y=190,text="Listening.....", font_size=18, color="black", angle=0, origin="center")
    #info_text = gui.draw_text(x=80, y=180, text='Listening.....')

def show_cmd_onscreen(text):
    gui.draw_text(x=80, y=220, text="' " + text + " '", color="green")
    gui.draw_text(x=80 + 1, y=220, text="' " + text + " '", color="green")
    gui.draw_text(x=80 - 1, y=220, text="' " + text + " '", color="green")
    
# Load and show assistant image
#img_image = gui.draw_image(x=120, y=100, w=120, h=100, image='upload/bot.png', origin='center')
#img_image = gui.draw_image(x=120, y=100, w=120, h=100, image='upload/assistant.png', origin='center')
#info_text = gui.draw_text(x=50, y=160, text='Home Assist v0.1')

# Initialize recognizer
recognizer = sr.Recognizer()

# Define the wake word
WAKE_WORD = "hello"
# Function to check if the wake word is in the text
def contains_wake_word(text):
    return WAKE_WORD in text.lower()

# Function to listen and process audio
def listen_and_recognize():
    home_screen()
    wake_up_state = False
    with sr.Microphone() as source:
        print("Listening for the wake word...")
        while True:
            # Adjust for ambient noise and listen
            recognizer.adjust_for_ambient_noise(source)
            audio = recognizer.listen(source)
            
            try:
                # Recognize speech using Google Web Speech API
                text = recognizer.recognize_google(audio)
                print(f"Detected: {text}")
                
                # Check if the wake word is detected
                if contains_wake_word(text):
                    buzzer.pitch(300, 1)
                    print(f"Wake word '{WAKE_WORD}' detected. Listening for commands...")
                    listening_screen()
                    wake_up_state = True
                else:
                    if (wake_up_state == True):
                        print(f"already woke up detected a command")
                        wake_up_state = False
                        print(f"command: {text}")
                        if text == "find keys":
                            print("found keys")
                            send_mqtt_data(text)
                            buzzer.pitch(200, 1)
                        else:
                            print("something else")
                            buzzer.pitch(200, 1)
                        show_cmd_onscreen(text)
                        #info_text = gui.draw_text(x=80, y=220, text="'" + text + "'")
                        time.sleep(3)
                        home_screen()
                            
                    else:
                        print(f"Listening for wakeword...")
                        home_screen()
                    
            except sr.UnknownValueError:
                # Handle the case where speech is not recognized
                pass
            except sr.RequestError as e:
                # Handle request errors
                print(f"Request error: {e}")

if __name__ == "__main__":
    listen_and_recognize()

Credits

Yahya Khalid

2 projects • 1 follower

Comments

Awards

Honorable Mention Visual Impairments

Build2gether 2.0 — Inclusive Innovation Challenge

Embed the widget on your own site

Home Assist v0.1

Home Assist v0.1

Things used in this project

Hardware components

Software apps and online services

Story

1. Introduction

2. System Overview

3. Components Used

3.1 Hardware Components

3.2 Software Components

4. Construction and Assembly

4.1 Unihiker Board Setup

4.2 MQTT Broker Setup

4.3 ESP32 Setup

5. System Functionality

Conclusion:

Product Refrences:

Code (ESP32):

Code

Home Assist UNIHIKER jupyter notebook code

Credits

Yahya Khalid

Comments

Awards