Created September 3, 2024

Smart Wearable for Real-Time Object Recognition and Auditory

A wearable device using Nordic nRF52840 and Grove AI Kit for real-time object recognition and audio feedback

Things used in this project

Hardware components

Nordic Semiconductor nRF52840 Multi-Protocol SoC

Seeed Studio Grove starter kit plus for Intel Edison

Angel Sensor M1

Bluetooth speaker

DFRobot USB Charger for 7.4V LiPo Battery DFR0564

Story

To begin with the challenge of object location and identification in indoor environments using some wearable assistive devices for visually impaired persons, a project can be done by integrating certain components. Now, for the example, I will utilize two of the components:
Nordic Semiconductor nRF52840: This is a powerful microcontroller with Bluetooth and is very well suited for wearables, as it is power-friendly while being quite powerful at the same time.

Seeed Studio Grove AI Kit: This includes a camera module along with an AI processing unit that is crucial for real-time object recognition and description.

Project: Wearable Object Recognition and Description System

Objective: Design a wearable device that identifies an object by implementing computer vision with Bluetooth technology to describe it audibly in real time. Components:

nRF52840 Development Kit by Nordic Semiconductor

Function: It shall act as a central unit for data reception and will implement Bluetooth functionality to send the data.

Grove AI Kit by Seeed Studio

Function: This kit will capture images through its camera module and recognize objects with its AI processing capabilities.

Steps Involved in the Project
Setting Up and Integration:
The camera module within the Grove AI Kit is interfaced with an AI processing unit. This AI Kit will be used to capture the picture and identify the objects using its pre-trained models.

Microcontroller Integration: The Grove AI Kit will need to be interfaced with the Nordic Semiconductor nRF52840 using UART or any other suitable communication protocol. A microcontroller acts in data transfer between AI Kit and the user's auditory feedback mechanism.
Object Recognition:
Training AI/Pre-trained Model: Employ already-trained object detection models in the Grove AI Kit for the identification process. These models can effortlessly identify objects that are in common indoor surroundings, such as doors, furniture, and signs.

Real-time Processing: Using the AI Kit for real-time image processing that can detect the identification of an object in real time as the user moves around the environment.

Communication and Feedback:
Data Transmission: Implement Bluetooth transmission using the Nordic Semiconductor nRF52840 on the AI Kit to send object recognition data either to a smartphone or a dedicated feedback system.

Auditory Feedback: Design an application or interface on the smartphone receiving the object data through Bluetooth and convert the information to audible descriptions. This can be achieved by means of text-to-speech technology in an application that describes identified objects to the user.
Wearable Design:
Housing and Mounting: Wearable casing or harness to securely attach the Grove AI Kit and the Nordic Semiconductor nRF52840; the device should be lightweight and designed for comfort over long periods.

Power Management: Power management shall ensure that the battery life is extended to a practical level for everyday applications.

Testing and Calibration:

Field Testing: The device shall be tested in a variety of indoor settings for accurate object recognition and reliable feedback.

User Testing: Implement testing with visually impaired users to gain feedback regarding object recognition algorithms and overall experience. Improvement and Iteration: Improve the Models: Enhance object identification using training of custom models, if necessary.

Expand Capabilities: Add other functionalities such as obstacle detection or navigation assistance as required by users. Summary:
The next project will combine the Nordic Semiconductor nRF52840 Bluetooth with the Grove AI Kit for object recognition into one wearable assistive device for visually impaired people. You will be integrating the components of a system that gives real-time object description to increase the level of independence and confidence in users who navigate indoor spaces.
Of course! Now, let's go into detail with this project, even down to some example code that will better help you understand how you might create this as a wearable assistive device for object recognition and description. For this project, we will be making use of the Nordic Semiconductor nRF52840, along with the Seeed Studio Grove AI Kit.
Project: Wearable Object Recognition and Description System

Overview
This project is about designing a wearable assistive device for the visually impaired to recognize and describe objects in real time using computer vision. It will be developed on a Nordic Semiconductor nRF52840 microcontroller integrated with a Seeed Studio Grove AI Kit, featuring a camera and an AI processing unit. The device will capture images of objects, recognize those objects, and give audio feedback via Bluetooth communication.
Required Components

Nordic Semiconductor nRF52840 Development Kit
Seeed Studio Grove AI Kit - with Camera Module Bluetooth Speaker/Smartphone for providing Audio Feedback Power Supply-including but not limited to a Battery Pack Wearable Enclosure-for instance, a small pouch or a vest Connecting Cables and Connectors System Design Sketch Image Capturing and Processing: The camera module of the Grove AI Kit captures an image of the ambient environment. The image is then processed in the AI processing unit to detect and identify objects. Data Transfer: The Nordic Semiconductor nRF52840 microcontroller receives the object data, detected and identified via UART from the Grove AI Kit.

That data is then transmitted over Bluetooth to either a smartphone or Bluetooth speaker for auditory feedback.
Auditory Feedback:
Real-time voice descriptions of the objects identified are given through a smartphone application or via a Bluetooth speaker.
Hardware Connections
Grove AI Kit with Nordic Semiconductor nRF52840:
Connect UART TX/RX pin of the AI Kit with the corresponding TX/RX on the nRF52840.
VCC and GND pins of the AI Kit with the nRF52840.
Software and Code
Configuration of Grove AI Kit:
Ensure the correct firmware and model (object recognition) are loaded onto the Grove AI Kit. For this example, we'll assume it is pre-set up and ready to use.

Nordic Semiconductor nRF52840 Code:

The nRF52840 will handle the UART communication with the Grove AI Kit, as well as the Bluetooth transmit. Here is a simplified example using the Nordic SDK(nRF5 SDK) and the nRF Connect SDK.

For this, create a basic mobile application that is capable of receiving BLE data and then further processing on it for generating voice. In a nutshell, the steps that this app will perform will be:

Establish the connection with nRF52840 via Bluetooth.

Receive data from the object.

Use TTS to transform object data into spoken words.

If developing an application, consider leveraging libraries like react-native-ble-manager for React Native and CoreBluetooth for iOS to abstract much of the Bluetooth logic.
Testing and Calibration

Field Testing: The device should be tested in an indoor environment for its object recognition and response.

Further tuning of the AI model or software based on test results.

User Feedback: Discussions with users who are visually impaired to obtain input regarding the usability and effectiveness of the device.

Further refinement of the device based on this feedback to offer a better user experience. Summary
The project will combine the Nordic Semiconductor nRF52840 with the Seeed Studio Grove AI Kit, an object recognition wearable device for the visually impaired. The nRF52840 handles Bluetooth communication, while object recognition is handled in real-time by the Groove AI Kit. The auditory feedback of the system is provided either through a smartphone or a Bluetooth speaker that enables users with enhanced independence and confidence in indoor mobility.
By doing this, the one-stop service for visually impaired persons' challenges will be practical and effective. The tip of modern technology needs to be applied to improve their daily life.

Code

c code

#include <stdio.h>
#include "nrf.h"
#include "nrf_gpio.h"
#include "nrf_uart.h"
#include "nrf_ble_qwr.h"
#include "ble_advertising.h"
#include "ble_conn_params.h"
#include "ble_nus.h"
#include "app_timer.h"

#define UART_TX_PIN   6
#define UART_RX_PIN   8
#define UART_BAUD_RATE NRF_UART_BAUDRATE_115200

static void uart_event_handler(nrf_uart_event_t event_type)
{
    // Handle UART events (e.g., data received)
}

static void uart_init(void)
{
    nrf_uart_configure(NRF_UART0, NRF_UART_PARITY_EXCLUDED, UART_BAUD_RATE);
    nrf_uart_pins_set(NRF_UART0, UART_TX_PIN, UART_RX_PIN, NRF_UART_PIN_DISCONNECTED, NRF_UART_PIN_DISCONNECTED);
    nrf_uart_event_clear(NRF_UART0, NRF_UART_EVENT_RXDRDY);
    nrf_uart_int_enable(NRF_UART0, NRF_UART_INT_MASK_RXDRDY);
    nrf_uart_task_trigger(NRF_UART0, NRF_UART_TASK_STARTRX);
    nrf_uart_task_trigger(NRF_UART0, NRF_UART_TASK_STARTTX);
}

static void ble_nus_data_handler(const uint8_t *data, uint16_t length)
{
    // Handle received data from BLE NUS
    // Process object recognition results
}

int main(void)
{
    uart_init();

    // Initialize BLE stack and services
    ble_nus_init();
    ble_advertising_start();

    while (true)
    {
        // Main loop
        __WFE(); // Wait for event
    }
}

Credits

1NH21CE030 Udaya Bhaskar reddy

1 project • 1 follower

Smart Wearable for Real-Time Object Recognition and Auditory

Things used in this project

Hardware components

Story

Schematics

Images

images

Code

c code

Credits

1NH21CE030 Udaya Bhaskar reddy

Comments

Embed the widget on your own site

Smart Wearable for Real-Time Object Recognition and Auditory

Smart Wearable for Real-Time Object Recognition and Auditory

Things used in this project

Hardware components

Story

Schematics

Images

images

Code

c code

Credits

1NH21CE030 Udaya Bhaskar reddy

Comments