Edge AI voice controlled Smart Plug hack, no Cloud or App

Introduction

Smart plugs are overall these days. They usually come with a matching Mobile App which can be downloaded from an App Store. They have WiFi / BLE capability and the App helps to control them. Optionally they can also have possibilities for integration with Home Assistant like Alexa. This also makes it possible to control the plugs using voice control with the help of Alexa or similar cloud services.

All that said, this still requires a roundtrip to Cloud services and an access to a Wireless network to communicate. How about having the voice control brought to the Edge and make it local to the Smart Plug itself ?

There are many make and model of Smart plugs available. Some of them make use of a communication module like CB2S (Tuya) or ESP02S. These modules have WiFi / BLE radio on them and makes communication and control possible.

The idea is to open the smart plug housing and replace the communication module with one of our own which has the capability for local voice control.

So let's get to it!

But first some back story...

Back story (Optional)

Quite a few years ago, when I was still young :) I saw a Bengali TV movie on a cable channel. Unusual of it's time, it was a Sci-Fi drama where a blind human lady was living with an Android. I do not recall most of the story except for a specific scene when she was lying on the bed and kept on uttering in finite periodic sequence, two words(in Bengali language of course)... Light...Dark...

The ceiling light in the room responded to her accordingly and kept itself switching on and off as she continued for a while.

This project, is a tribute to that beautiful story as I try to replicate the voice operated light switch.

Caution

Smart plugs work on AC mains power and they convert the high voltage into low level DC voltage to the necessity of the controller circuit. It is very dangerous and can be life threatening to work with AC mains circuit if you do not have proper knowledge or take necessary precaution. Any incorrect wiring can cause short circuit and life threatening shock hazard for the hacker or the user. Hence a word of caution, if you proceed with the implementation of this project, do this at your own risk. As the author and creator of this project I do not take any responsibility of any mishaps.

Ground work

The center piece of this project is a custom made module called 'Orthosie' which is powered by NINA-B306 RF module and has a tiny PDM microphone. The module needs to be flashed with the right firmware and finally programmed with a suitable speech recognition Machine Learning model which works integrated with the GPIO of the NINA module to send the right actuating signal to the plug relay.

Details of this module is added as in the Code section of this tutorial. The Github repository details the construction and schematics of this module including pinout

First steps

We can't hack a smart plug unless we have one. So I went to my local grocery store and got one -

We have to open this up in order to work on the internal circuit.

A word of caution,again! Do not connect the plug to live AC mains power and try to work on this. The circuit must be enclosed and safely housed before turning on the power. Working with exposedAC mains circuit can be life threatening! If you do not understand the dangers associated with this implementation, do not attempt to work on this.You should only attempt to make this project at your own risk.

With the aforementioned caution in mind we can attempt to pry open the plug. I used a Magnum knife to pry open the enclosure. This can be also dangerous working with a sharp knife.

After few jab and pull with the knife the side starts to open -

Repeating the same action on the remaining 3 sides gets the job done and the top is loose now -

Let's take the circuit out and take a look inside -

Let the desoldering begin

Once we have the circuit out of the box, it's time to desolder the existing controller module board. Using flux is essential as always to let the solder melt away -

Using the soldering iron and a solder wick wire the solder can easily be absorbed into the metal mesh and the joints cleaned -

Once done completely the module will just fall off -

Building Orthosie

Well, we start with the PCB which is made by a PCB manufacturer (Recommended best approach unless you have a 6 layer precision PCB manufacturing solution at home) and we make that sit still (I call it 'Hold the Board' or 'Hoboard' :) ) -

Make it further immovable by setting other unused boards next to in and apply solder paste -

It only needs a few parts as mentioned in the schematics -

And we place the components on top of the paste and let it bake! Now this step can be achieved several ways depending on the equipment available (Reflow Oven, Hot air soldering station, Soldering plate etc.)

Once competed te top side is ready -

Now we focus on the bottom, which only requires 1 component, the PDM mic -

For this, I used hot air soldering -

That concludes assembling the Orthosie hardware module.

Let's get down to business

Now that the plug hardware is ready to be fitted with Orthosie module, we need to prepare the module first. Here is a pinout of the module including all its interfacing pins -

As mentioned before, the module is powered by NINA-B306 RF module from uBlox. In its original form as purchased, the RF module does not have any firmware in it. The module contains a Nordic semiconductor chip NRF52840 which is the same as "Arduino Nano BLE Sense" board.

Our first job with Orthosie would be to load the Arduino Nano BLE Sense firmware on it. This can be done with the help of the Software Debug (SWD) interface pins on the module. We would need a Segger J-Link debugger unit and Segger J-Flash software to get this done.

For the mechanical interface with the SWD pins, I used a contraption of pogo pin clip as shown below. One on the left is the SWD interface (3x2 pin connector). The one on the right is USB Serial interface (4x1 pin connector). Pin distance is 2.54 mm.

For the Flashing of the "Arduino Nano BLE Sense" firmware only the SWD interface is needed. While using the USB interface the SWD interface pogo connector clip is only used to power the chip.

Firmware install

This step requires a Segger JLink Software Debugger. I used the hex file from the Arduino Library folder for "Arduino Nano BLE Sense". JFlash from the Segger toolset is used to flash the firmware on the blank NINA-B306 module. This requires few configuration on the JFlash project. I used configurations for NRF52840 chip -

And finally the programming config -

Once the 'bootloader.hex' file is dragged and dropped on the JFlash window and the hardware contraption connected to Orthosie module, Production Programming can be chosen (This is pure cheeky) -

Once completed a popup confirms the flashing is done -

Program Ready

Once the module is flashed with the firmware, the device shows up as a "Arduino Nano 33 BLE" device on Arduino IDE. Beyond this point, with both the pogo connector clips in place, Orthosie module is ready to be programmed using Arduino IDE just like any other "Arduino Nano BLE Sense".

Below is a pin map of the module compared with NINA-B306, Arduino Nano BLE and ESP-02S.

Note. Arduino Nano BLE Pins are the same as the Orthosie module

The Orthosie module has a PDM Mic fitted on the front of the board, which will be used to listen-in while we run a preprogrammed voice recognition sketch with logical control to send appropriate signals through the interfacing pins of the smart plug interface.

Building a voice recognition ML model

Arduino documentation provides details instructions on how to build a machine learning model for voice recognition using Arduino Nano BLE Sense. I decided to exploit the same for this project. Documentation link is below -

Edge Impulse tutorial with Arduino Nano BLE Sense

The instructions say to generate your own voice samples for specific commands. I started this journey but gave it up after creating around 144 voice samples. Typical data set requires a minimum of per command 100 samples. Using over 1000 samples per command yields a better result. Then again it will never be the best because Machine Learning models are nothing but a mathematical guess work on the input data.

To get a fast start on the model I used the Tensorflow speech command dataset which has a limited number of speech command samples in.wav format. The data set link is below -

Download link to sample speech data set (very large)

Of the entire dataset I decided to focus on some random words for the command such as - 'marvin', 'visual', 'zero' and 'on'. Below is the result of the generated model (using Edge Impulse studio) with a test performance report using the confusion matrix -

Once the model is built we can then export this as Arduino Library from the Deployment section of Edge Impulse studio -

I took the liberty of making this model myself and putting this on Github along with a worked example for smart plug interfacing. The link to the Github repo is mentioned in the Attachments section.

Understanding the software

Now that all the pieces of the puzzle are sorted out, we can now get down on preparing the sketch that will run on Orthosie module and help control the Smart plug using voice control.

The sample sketch constructs the following architecture -

This means that there are two different ways to interact with the Smart plug, after it has been hacked with a preprogrammed Orthosie module.

The sketch runs on the following flow algorithm -

The device listens for a wake word. Upon recognising the wake word, the devices allows a window of 5 seconds to listen to a command word and upon receiving the commands it acts accordingly and updated the GPIO interface with the right configuration to trigger the relay and optionally LEDs on the smart plug.

Once a BLE scanner device makes a connection with the module the flow goes into a BLE sub-flow and continues until the scanner disconnects. The BLE interface then has the option to send commands to turn the relays on or off. Once disconnected, control is returned to the main flow where the device continue to listen to the wake word again.

In the sample sketch, I configured the wake word to be 'Marvin'. The command words supported by this ML model are 'Visual' and 'zero' for Turning On and Off respectively. Why such weird choice of the word you ask ? Well, those were the most unrelated sample commands in the training dataset I could find.

Let's focus on the sketch

The sketch has is sample implementation and extension of the built library from Edge Impulse and can be found here -

Link to example sketch

The primary task is the speech recognition of the sounds and identify if the wake word ('marvin') is mentioned.

if ((result.classification[ix].label == "marvin") && result.classification[ix].value > 0.9) {
    Serial.println("Somebody activated Marvin!");
    marvinCall = HIGH;
    previousMillis = currentMillis;    
}

Once that is true a flag is set to true for a finite period during which it would listen for the command words ('visual' or 'zero') -

else if (marvinCall && (result.classification[ix].label == "visual") && result.classification[ix].value > 0.8) {
    turnPlugOn();
    Serial.println("Smart plug on");
} else if (marvinCall && (result.classification[ix].label == "zero") && resul    t.classification[ix].value > 0.8) {
    turnPlugOff();
    Serial.println("Smart plug off");
}

The BLE connection loop runs within the main loop and waits for an incoming BLE connection from a BLE client. This being a prototype only, I did not program any hard security measures. Meaning any client can connect. Once the sketch runs the device advertises itself as 'Orthosie' for BLE connection and the Smart Plug Service.

// set advertised local name and service UUID:
BLE.setLocalName("Orthosie");
BLE.setAdvertisedService(smartplugService);
// add the characteristic to the service
smartplugService.addCharacteristic(smartplugCharacteristic);
// add service
BLE.addService(smartplugService);
// set the initial value for the characteristic:
smartplugCharacteristic.writeValue(0);
// start advertising
BLE.advertise();
Serial.println("BLE Smart Plug");

Once a client device (A smartphone BLE App like nRFConnect or LightBlue) connects the sketch goes into a subloop where it expects a value to be overwritten for the BLE Characteristic. The value being as simple as 0x1 to turn the plug on and 0x0 to turn off.

while (central.connected()) {
// if the remote device wrote to the characteristic,
// use the value to control the LED:
    if (smartplugCharacteristic.written()) {
        switch (smartplugCharacteristic.value()) {   // any value other than 0
        case 00:
            Serial.println("Smart Plug OFF");
            turnPlugOff();
        break;
        case 01:
            Serial.println("Smart Plug ON");
            turnPlugOn();
        break;
        default:
            Serial.println(F("Smart Plug OFF"));
            turnPlugOff();
        break;
        }
    }
}

How to program ?

As mentioned earlier, Orthosie can be programmed with the help of Arduino IDE. So download the library from Github and configure the IDE to program the sketch. But we need to use the same pogo contraption as shown before to connect and program the device. Required connections are as shown here for the touch points on the front of the module -

The sketch takes for the first time about 5 - 10 minutes to compile on my computer after which Arduino IDE would be able to upload the sketch it successfully.

Upload completed

The Arduino IDE can help test/debug and here is sequence of test cases -

1. Test Wake Word

2. Test turning the plug on using voice

3. Test turning the plug off using voice

For the BLE it is somewhat similar but we need the BLE client to make connection to the device -

1. Connect with the device

2. Turn the plug on by setting the characteristics value to 0x1

3. Turn the plug off by setting the characteristics value to 0x0

And we are ready to move to the next phase!

Solder the module on plug board

Let's put the board on the smart plug.

Here is a ready view of the installed module on the smart plug. Depending on the make and model the orientation of the module can vary.

Let's test

Here is a link to the video of voice command test on the hacked Smart Plug -

Testing the BLE controls

And finally a video of the test of the controls over BLE

I used LightBlue App to test the connection. This is very rudimentary as it accepts two Hex values (0x0 to turn off and 0x1 to turn on) to control the Smart Plug.

That's all folks...Oh wait!

In the Makers world there is always something to improve and nothing is at its final step. The voice recognition model can always be further tuned and enhanced. Depending on the casing of the plug the voice control can be distance sensitive. I was not particularly happy with the range of the BLE (3 meters max line of sight.) rendering a close proximity operation.

If you are making this ? I wish you success and don't forget to have fun making it as I did all along the way. Until next time...'Marvin'...'Zero'.