Home Asistant with Voice Control

1.project plan

When people with visual impairments are engaged in indoor activities, using various household appliances is a necessary daily activity, such as switching lights, turning on and off televisions, opening and closing doors, etc. However, many household appliances are designed according to the situation of able-bodied people without considering the difficulties faced by people with limited mobility due to visual impairment. After comprehensively considering several basic daily activity scenarios, we have designed a basic solution: using voice recognition to receive user commands and directly or indirectly control various household appliances through analyzing commands. Therefore, the project requires three modules: speech command recognition module, speech command analysis module, and speech command instruction module. The voice command recognition module uses the tflite model provided by Grove Vision AI 2 and voice input hardware, which can be customized with corresponding command words. The voice command analysis module analyzes the voice commands. The electrical appliances that can be directly controlled will be directly controlled, while those that cannot be directly controlled will be indirectly controlled through mechanisms such as matter and devices to transmit commands to them.

2.Part Sourcing

1）Grove Vision AI 2

It is an MCU-based vision AI module powered by Arm Cortex-M55 & Ethos-U55. It supports TensorFlow and PyTorch frameworks and is compatible with Arduino IDE. With the SenseCraft AI algorithm platform, trained ML models can be deployed to the sensor without the need for coding. It features a standard CSI interface, an onboard digital microphone and an SD card slot, making it highly suitable for various embedded AI vision projects.

2) Xiao ESP32 module

Seeed Studio XIAO Series are diminutive development boards, sharing a similar hardware structure, where the size is literally thumb-sized. The code name "XIAO" here represents its half feature "Tiny", and the other half will be "Puissant". Seeed Studio XIAO ESP32S3 Sense integrates camera sensor, digital microphone and SD card supporting. Combining embedded ML computing power and photography capability, this development board can be your great tool to get started with intelligent voice and vision AI.

3) nRF52840 DK

The nRF52840 DK is a versatile single-board development kit for Bluetooth Low Energy, Bluetooth mesh, Thread, Zigbee, 802.15.4, ANT and 2.4 GHz proprietary applications on the nRF52840 SoC. It is the recommended Nordic development kit for Amazon Sidewalk. It also supports development on the nRF52811 SoC.

The nRF52840 DK can also be used for Matter over Thread where Thread is used for transport and Bluetooth LE for commissioning. Matter devices based on Thread are required to feature Bluetooth LE concurrently to enable adding new devices to a network.

3.Building

1）step1: Select Model and Deploy

Edge Impulse website can build datasets, train models, and optimize libraries to run directly on device; from the smallest microcontrollers to gateways with the latest neural accelerators. we can build AI model with this site.

• Create a project: Once you're logged in, create your first project. Give it a name that reflects your project's goal, whether it's recognizing sounds, detecting objects, or something entirely unique.

• Collect/import data:To teach your device, you need data. Edge Impulse provides user-friendly tools for collecting data from your sensors, such as recording audio, capturing images, or reading sensor values. We recommend using a hardware target from this list or your smartphone to start collecting data when you begin with Edge Impulse.You can also import existing datasets or clone a public project to get familiar with the platform.

• Label your data:Organize your data by labeling it. For example, if you're working on sound recognition, label audio clips with descriptions like "dog barking" or "car horn." You can label your data as you collect it or add labels later, our data explorer is also particularly useful to understand your data.
• Pre-process your data and train your model: This is where the magic happens. Edge Impulse offers an intuitive model training process through processing blocks and learning blocks. You don't need to write complex code; the platform guides you through feature extraction, model creation, and training.
• Run the inference on a device: After training your model, you can easily export your model to run in a web browser or on your smartphone, but you can also run it on a wide variety of edge devices, whether it's a Raspberry Pi, Arduino, or other compatible hardware. We also provide ready-to-flash binaries for all the officially supported hardware targets. You don't even need to write embedded code to test your model on real devices!

2) step2:Connect Analysis module

Connect the Grove Vision AI (WE2) module to the default I2C interface of your Arduino board(Xiao Esp32 module) using the 4-Pin Cable. Make sure each wire is connected to the correct pin.

• SCL -> SCL (Grove Vision AI WE2)
• SDA -> SDA (Grove Vision AI WE2)
• VCC -> VCC (Grove Vision AI WE2, 3.3V)
• GND -> GND (Grove Vision AI WE2)

Xiao esp32 module will receive Grove Vision 2 board message through I2C protocol， then create analysis report and send it to TTS module realtime.

3) step3:Connect nRF52830 module

Connect nRF52830 module from Xiao esp32module UART，then control action can be forward to household appliance components through Matter protocol.

4.Code and Algorithm

1) data collection and upload

there we can find the source app to record voice through Grove Vision AI 2: https://github.com/HimaxWiseEyePlus/Seeed_Grove_Vision_AI_Module_V2/tree/main/EPII_CM55M_APP_S/app/scenario_app/kws_pdm_record

we can see tutor how to compile the app and update and flash firmware: https://wiki.seeedstudio.com/grove_vision_ai_v2_himax_sdk/

2) arduino code for analysis model result and action

First, install depended library:

• Seeed_Arduino_SSCMA v1.0.0
• ArduinoJson v7.1.0

#include <Arduino.h>
#include <Seeed_Arduino_SSCMA.h>
#define LIGHT_ON 1
#define LIGHT_OFF 2
#define THRESHOLD 0.7
SSCMA AI;
void turnon_light(){
    Serial1.write(LIGHT_ON);
}
void turnoff_light(){
    Serial1.write(LIGHT_OFF);
}
void setup()
{
    //initial and setup Grove Vision 2
    AI.begin();
    //initial serial debug monitor.
    Serial.begin(9600);
    //initial serial1 for forward control to nRF52830
    Serial1.begin(9600);
    //message application started.
    Serial.print("Voice Control App start!");
}
void loop()
{
    String analysis = "";
    // invoke once, no filter , get image
    if (!AI.invoke(1, false, true))
    {
        for (int i = 0; i < AI.classes().size(); i++)
        {
            if (AI.classes()[i].score > THRESHOLD)
            {
                if (AI.classes()[i].target == LIGHT_ON)
                    turnon_light();
                if (AI.classes()[i].target == LIGHT_OFF)
                    turnoff_light();
            }
        }
    }
    //interval 3 seconds.
    delay(1000);
}

3) nRF52830 dk forward matter code，matter protocol is complexity，we can reference tutor and example in nRF DK document and sources.

#include "LightSwitch.h"
#include "AppEvent.h"
#include "BindingHandler.h"

#include <app/server/Server.h>
#include <app/util/binding-table.h>
#include <controller/InvokeInteraction.h>

#include <app/clusters/switch-server/switch-server.h>

#include <app-common/zap-generated/attributes/Accessors.h>

using namespace chip;
using namespace chip::app;
using namespace chip::app::Clusters;

void LightSwitch::Init(chip::EndpointId aLightDimmerSwitchEndpoint, chip::EndpointId aLightGenericSwitchEndpointId)
{
    BindingHandler::GetInstance().Init();
    mLightSwitchEndpoint          = aLightDimmerSwitchEndpoint;
    mLightGenericSwitchEndpointId = aLightGenericSwitchEndpointId;
}

void LightSwitch::InitiateActionSwitch(Action mAction)
{
    BindingHandler::BindingData * data = Platform::New<BindingHandler::BindingData>();
    if (data)
    {
        data->EndpointId = mLightSwitchEndpoint;
        data->ClusterId  = Clusters::OnOff::Id;
        switch (mAction)
        {
        case Action::Toggle:
            data->CommandId = Clusters::OnOff::Commands::Toggle::Id;
            break;
        case Action::On:
            data->CommandId = Clusters::OnOff::Commands::On::Id;
            break;
        case Action::Off:
            data->CommandId = Clusters::OnOff::Commands::Off::Id;
            break;
        default:
            Platform::Delete(data);
            return;
        }
        data->IsGroup = BindingHandler::GetInstance().IsGroupBound();
        DeviceLayer::PlatformMgr().ScheduleWork(BindingHandler::SwitchWorkerHandler, reinterpret_cast<intptr_t>(data));
    }
}

void LightSwitch::DimmerChangeBrightness()
{
    static uint16_t sBrightness;
    BindingHandler::BindingData * data = Platform::New<BindingHandler::BindingData>();
    if (data)
    {
        data->EndpointId = mLightSwitchEndpoint;
        data->CommandId  = Clusters::LevelControl::Commands::MoveToLevel::Id;
        data->ClusterId  = Clusters::LevelControl::Id;
        // add to brightness 3 to approximate 1% step of brightness after each call dimmer change.
        sBrightness += kOnePercentBrightnessApproximation;
        if (sBrightness > kMaximumBrightness)
        {
            sBrightness = 0;
        }
        data->Value   = (uint8_t) sBrightness;
        data->IsGroup = BindingHandler::GetInstance().IsGroupBound();
        DeviceLayer::PlatformMgr().ScheduleWork(BindingHandler::SwitchWorkerHandler, reinterpret_cast<intptr_t>(data));
    }
}

void LightSwitch::GenericSwitchInitialPress()
{
    DeviceLayer::SystemLayer().ScheduleLambda([this] {
        // Press moves Position from 0 (idle) to 1 (press)
        uint8_t newPosition = 1;

        Clusters::Switch::Attributes::CurrentPosition::Set(mLightGenericSwitchEndpointId, newPosition);
        // InitialPress event takes newPosition as event data
        Clusters::SwitchServer::Instance().OnInitialPress(mLightGenericSwitchEndpointId, newPosition);
    });
}

void LightSwitch::GenericSwitchReleasePress()
{
    DeviceLayer::SystemLayer().ScheduleLambda([this] {
        // Release moves Position from 1 (press) to 0 (idle)
        uint8_t previousPosition = 1;
        uint8_t newPosition      = 0;

        Clusters::Switch::Attributes::CurrentPosition::Set(mLightGenericSwitchEndpointId, newPosition);
        // ShortRelease event takes previousPosition as event data
        Clusters::SwitchServer::Instance().OnShortRelease(mLightGenericSwitchEndpointId, previousPosition);
    });
}