Surangani - Voice Enabled Energy Monitoring

Introduction

Voice controlled smart homes are the future! Why use screens and buttons when you can just TELL your fan to turn on? The Snips AI voice platform allows you to do that and more, and all on the edge!

This project allows you to use the power of your voice to control and monitor the energy consumption of the devices in your home.

What is Snips?

The Snips AI Voice Platform is a decentralized, private-by-design voice assistant. If you've used services like Amazon Alexa and Google Home, you might know that voice is processed in the cloud. However, Snips processes the user’s voice directly on-device. Anyone can easily build CUSTOM voice assistants in a matter of minutes using their easy to use web interface. You can provide your own training data, and then install the assistant on your hardware and run it completely on the edge.

This video will give you a quick intro to Snips.

Hardware

Excited to start making your own custom voice assistant? Then I suggest you get one of the Voice development kits from Seeed

Link : https://www.seeedstudio.com/snips.html

However, I already had a Raspberry Pi 3 B+ lying around, and it was too late for me to order one of the ReSpeaker 2-Mics Pi HATs (the Raspberry Pi itself doesn't have any hardware to handle microphones that come with a 3.5mm jack)

So I ended up using a pair of Jabra ENC060 headphones, which comes with a USB adapter. This however meant that I had to wear the headphones every time when issuing voice commands. So if you can, get the Pi HAT.

In order to monitor the power usage, I purchased a Modbus enabled energy meter from ebay. This allowed me to measure parameters such as AC Voltage, Current, Power and Energy using modbus commands from the Raspberry Pi.

You will also need a RS485 to TTL converter, a Relay Module and some wires.

Installing Snips

Note - A more detailed set of instructions can be found at : https://docs.snips.ai/getting-started/quick-start-raspberry-pi

Prior to installing snips, you will have to install Raspbian on your Raspberry Pi. You can follow the instructions on this link to install Raspbian on your board.

While Raspbian is being set up, be sure to install Git and NodeJS on your computer. You will need the Node Package Manger (NPM) to install the Sam Command Line Interface. Sam allows you to setup snips on your raspberry pi using the terminal of your own pc. However, you will need a remote access tool like puTTY later on. (if you're using Windows)

Follow the instructions on this link to setup Node and the NPM on your windows PC. (For Linux users : https://tecadmin.net/install-latest-nodejs-npm-on-ubuntu/)Once that's done, you can use NPM to install Sam on your computer.

npm install -g snips-sam

Make sure that both your PC and Raspberry Pi is connected to the same WiFi network, and run

sam devices

to view the list of available devices. However, my raspberry pi didn't show up when I tried this, so I ended up using the IP address of my Pi to connect to it.

Executing the following command will install all the repositories required to run Snips, and the voice platform itself on your Pi. Once it's done, don't forget to login to your snips console account using

sam login

Since I used my own headset, I had to setup audio using the command - sam setup audio.

You can use the following commands to see if everything works correctly.

Once everything is setup, we can move on to the Snips Console to setup our assistant.

Creating a Custom Assistant

This video explains how to create custom assistants using the Snips console.

The following video shows the custom assistant I made for this project.

This app is published in the Snips store -> Link to App

Setting Up a Personal Hotword

Instead of using 'Hey Snips' to wake up my assistant, I wanted to use 'Surangani' (It's a name of a woman in a popular Sri Lankan song :D )

I used this guide to set up my personal hotword :https://github.com/snipsco/snips-record-personal-hotword

You will need puTTy in order to SSH into your RPi.

Then use this command to start recording your hotword. And provide a name for the generated file as well.

sudo systemctl stop snips-audio-server; python script_recording.py your_hotword_name

The following image shows the hotword being recorded and trained on. Take note of the generated model.

Use the commands in the guide to test out your model, and if you're happy, you can update your assistant configuration.

Move your model to the snips directory using this command,

sudo mkdir -p /etc/snips/; sudo mv <path_to_your_model> /etc/snips/

You will have to edit the the /etc/snips.toml file using vim or nano.

Deploying Assistants

If you setup everything correctly, you can now deploy your assistant to your raspberry pi using 'sam install assistant'. I initially deployed my assistant with the Greetings app, and ensured that the hotword works.

If everything works, you can deploy your custom assistant on to the device.

Hardware and Wiring

I first used a standard plug base and fixed the relay module and current transformer to measure the current. That way I could plug any device into this special plug base without being exposed to open connectors. Mains voltage is dangerous, so take extra care.

Make sure to tighten down everything really well. Use crimp connectors if you can find some.

I had enough room inside the sun box to fit my relay module as well. Neat!

Now we have a portable, safe setup that we can connect to mains voltage!

I also printed some parts to make my setup look more like the Voice interaction Kit. (3D Printable parts can be found in the github repo along with the wiring diagram )

Writing the Action Code

In order for the Raspberry Pi to actually do something, such as telling you the weather forecast, we will have to write some handler code. This is code which is executed when a certain event happens on the Snips platform, such as when an intent has been detected.

For this I used one of the python templates provided by snips.Link : https://docs.snips.ai/articles/platform/create-an-app/python-template

Out of these, I chose https://github.com/snipsco/snips-app-template-py. You can download this to your Raspberry Pi using Git and begin writing your handler code in action-app_template.py

Let's go through each component of the action code I wrote. ( I changed the file name to surangani-app.py)

from snipsTools import SnipsConfigParserfrom hermes_python.hermes 
import Hermesfrom hermes_python.ontology 
import *import ioCONFIG_INI = "config.ini"

These import statements come with the template. You will have to install hermes-python on your raspberry pi. On your terminal, type

sudo pip2 install python-hermes

(I used python2.7 which is why I used pip2 to install the required package)

The template also provides a config file, which I did not use in this implementation. (You can use it when deploying this on multiple satellites etc.)

##MODBUS 
import minimalmodbus
instrument = minimalmodbus.Instrument('/dev/ttyS0', 1, mode = 'rtu') # port name, slave address - Use multiple objects to talk with multiple power meters
instrument.serial.baudrate = 9600

These are the required parameters for the modbus communication. I used the minimalmodbus package which you can install using,

sudo pip2 install minimalmodbus

The parameters for the modbus communication are specified in the datasheet of the energy meter. I am using the UART0 pins of the Raspberry Pi, the slave address is selected as 1 and the baud rate is set to 9600.

from threading import Thread
import RPi.GPIO as gpio
gpio.setmode(gpio.BCM)
gpio.setup(4, gpio.OUT)
gpio.output(4, gpio.HIGH)
  
MQTT_IP_ADDR = "localhost"
MQTT_PORT = 1883
MQTT_ADDR = "{}:{}".format(MQTT_IP_ADDR, str(MQTT_PORT))
availableDevice = ('fan')

Next we import the RPi.GPIO package since we want to use GPIO4 as an output to control the relay. We're also using the Python Thread library to monitor the current while we check other parameters (more on this later). The MQTT parameters are also set up. ( I did not change these since the MQTT broker is hosted in my RPi itself. If you're using more satellite kits, you might have to change the MQTT_IP_ADDR to that of the main device hosting the broker.

The availableDevice list must be edited if more devices must be added. Currently I have only one device, which is my fan.

Take a good look at surangani-app.py, you will notice that withing the Class Surangani_app, there are several functions. Take note of master_intent_callback(self, hermes, intent_message). This function sees if any intents have been triggered, and calls the function that satisfies that intent.

# --> Master callback function, triggered everytime an intent is recognized
   def master_intent_callback(self,hermes, intent_message):
       coming_intent = intent_message.intent.intent_name
       if coming_intent == '&3a8gr9ZbNyA4GVLEvw1nreV9Z3jMl1zn5dYBXWw2:getVoltage':
           self.getVolts_callback(hermes, intent_message)
       elif coming_intent == '&3a8gr9ZbNyA4GVLEvw1nreV9Z3jMl1zn5dYBXWw2:turnOn':
           self.turnOn_callback(hermes, intent_message)
       elif coming_intent == '&3a8gr9ZbNyA4GVLEvw1nreV9Z3jMl1zn5dYBXWw2:turnOff':
           self.turnOff_callback(hermes, intent_message)
       elif coming_intent == '&3a8gr9ZbNyA4GVLEvw1nreV9Z3jMl1zn5dYBXWw2:getCurrent':
           self.getCurrent_callback(hermes, intent_message)
       elif coming_intent == '&3a8gr9ZbNyA4GVLEvw1nreV9Z3jMl1zn5dYBXWw2:getPower':
           self.getPower_callback(hermes, intent_message)
       elif coming_intent == '&3a8gr9ZbNyA4GVLEvw1nreV9Z3jMl1zn5dYBXWw2:getEnergy':
           self.getEnergy_callback(hermes, intent_message)
       elif coming_intent == '&3a8gr9ZbNyA4GVLEvw1nreV9Z3jMl1zn5dYBXWw2:setMaxCurrent':
           self.setMaxCurrent_callback(hermes, intent_message)

Here you can see the intents and their callback functions.

Let's look at one of the callbacks. The getVolts_callback is used to find the voltage received by the connected device.

# --> Sub callback function, one per intent
   def getVolts_callback(self, hermes, intent_message):
       # terminate the session first if not continue
       hermes.publish_end_session(intent_message.session_id, "")
       # action code goes here...
       print '[Received] intent: {}'.format(intent_message.intent.intent_name)
       ##These are printed out on the console for debugging purposes.
       ##We find out the device by using the deviceName slot. This is done by the following lines of code.
       device = None
       if intent_message.slots:
           device = intent_message.slots.deviceName.first().value
           # check if valid
           if device.encode("utf-8") not in availableDevice:
               device = None
       if device is None:
           reply = "No device specified"
       else:
           reply = "Voltage received by "+str(device) +" is " + str(self.get_volts()) + " Volts" ##The get_volts function is called to handle the modbus transaction
       hermes.publish_start_session_notification(intent_message.site_id, reply,"Surangani_app") ##The result is spoken by the assistant
       hermes.publish_end_session(intent_message.session_id, reply) 

The get_volts function is explained in brief below.

def get_volts(self):
       try:
           voltage = instrument.read_register(0x0000, numberOfDecimals=1, functioncode=4, signed=False) ##Voltage is stored in address 0x0000
           return voltage
       except ValueError:
           return self.get_voltage() ##sometimes the modbus communication fails, if so try again.
       except IOError:
           return 0 ##if there is no response from device, return 0

Similarly, there are separate callbacks and modbus handler functions for all 7 intents. If you go through the surangani-app.py, you'll see how each intent is handled.

In order to continuously monitor the current, a thread is created that keeps checking the current to see if it has exceeded the given threshold. If so, the device is switched off. (Check monitorCurrent and setMaxCurrent functions)

Surangani In Action

After the action code is complete, it can be run using

python surangani-app.py

Note : If you're using my code as is, replace action-app_template.py with surangani-app.py in my github repo.

If you want it to start when the RPi boots then check out this link : https://www.dexterindustries.com/howto/run-a-program-on-your-raspberry-pi-at-startup/

The video below shows the basic capabilities of Surangani.

This next video shows how current can be monitored and limited using voice commands. You can see that initially the fan draws around 180mA at low speeds. Then the assistant is told to set the maximum current to 250mA, and the speed of the fan is increased, which in turn increases the amperage. Once this passes 250mA, the relay is triggered. You can see the Raspberry Pi continuously polling the energy meter in order to read the current draw.

Conclusions

The Snips AI voice platform is a great tool to get your voice application up and running quickly. More than anything, it is really user friendly, which allows you to create complex voice queries in a matter of seconds. They have a whole bunch of pre trained apps which are perfect for smart homes, but if you want to create your own custom app its really easy to do so. And the other plus is that everything runs on the Raspberry Pi itself, so your smart home doesn't become dumb as soon as the internet goes out!