Published April 16, 2019 © GPL3+

Accio - Searching for Things Using Voice

Losing things is a part of life. Accio makes sure you always get back what you lose, via voice.

AdvancedFull instructions provided15 hours1,269

Accio - Searching for Things Using Voice

Things used in this project

Hardware components

Raspberry Pi Zero Wireless

Raspberry Pi 3 Model B+

Espressif ESP8266 ESP-01

Buzzer

SparkFun FTDI Basic Breakout - 3.3V

Seeed Studio ReSpeaker 2-Mics Pi HAT

Software apps and online services

Hand tools and fabrication machines

Soldering iron (generic)

Story

Marker hardware for Accio, consisting of ESP-8266, Buzzers and LED

"Accio" happened, basically because it was essential for the smooth functioning of my daily routines. Every morning, while getting ready for work, locating my bike keys and wallet would be a struggle. Yes, everything could have been solved if I could just decide to keep my keys at the same place all the time, but where's the fun in that..

Hence I began looking for technologies I could use to solve my problem. The first set of ideas involved BLE markers and tags, but it felt like an overkill. GPS never was a feasible option, since I was sure that I'll always bring back my bike keys home anyway; the problem was definitely local in nature. The ideal solution seemed to be Wi-Fi based, since it would meet my localization requirement. Power consumption would be an issue, but I decided to go along with it to see how it would work out. And so I chose an ESP-8266 for Accio, since I was slightly familiar with a NodeMCU, which is based on the same Espressif chip used by the ESP-8266, but with an FTDI chip on board, so as to enable communications with the computer directly as a COM port. The ESP-8266 module would save space, but requires an extra FTDI USB to Serial converter module to program or send and receive data.

FTDI USB to Serial converter

ESP-8266 module

Now, since all those parts were taken care of, the biggest question was how to trigger the ESP-8266 modules. Buttons were an option; press this button to locate marker A, and press that button to locate marker B. Simple and effective. But I was obsessed with voice. Calling out for marker A, and marker A responding; that was the best solution. Voice would have a nice natural feel to it. And so, I decided to try out Snips.ai, with which all processing could be done locally at the edge(and so need of Internet all the time), and could be run on Raspberry Pi as well.

However, I never realized how easy getting started with Snips.ai would be. Setup was super smooth and easy, with sufficient documentation available for all necessary parts. I had decided to use a Base station(which is a Raspberry Pi 3B+, running all the NLU and hotword algorithms), and a Satellite Kit(Raspberry Pi W, which houses the audio capture microphones, sending audio data to the Base station to be decoded). One Base station could have more than one Satellite station, all linked to each other.

Satellite Kit of Snips.ai, with a 2-mic Respeaker array docked onto R-Pi W

What I basically wanted to implement was, on saying "Accio keys" to the system, the marker named "keys" is supposed to catch my attention, letting me know where I kept it last. This needed two separate parts :

The hardware stuff, with the ESP-8266, buzzers, and other related electronics I had to incorporate into the PCB, and
The software stuff, which involved the Snips.ai code, MQTT channels and setting up of communication protocols, as well as the microcontroller program for the ESP-8266.

To give a rough overview of what happens in the system from start to finish,

Upon uttering the "hotword", for eg, Jarvis, the system must wake up, realizing that I am about to ask it to track one marker.
When I say "Accio keys", the system understands that I'm asking for my "keys", and so it pings the ESP-8266 module attached to that particular marker over an MQTT channel.
The ESP-8266 with the identity "keys" understands that it is being called, and so it lights up LEDs and turns on buzzers to let me know where I can find it

Operation of Accio

Marker boards I designed

All associated codes and CAD files(for the PCB I made for this purpose) have been attached. The cost split-up for one marker board would be:

ESP-8266 : 7 USD
Buzzers : 1.5 USD
Regulators, capacitors etc : 3 USD

Hence, less than 10 USD in total, is the approx cost to build one such marker.

The Snips app I made for Accio is here, under the name "Accio". This same assistant can be used while starting to experiment with Snips. I have used the keywords "alpha", "beta" and "gamma" as the items, which means that the app will give an alert over the relevant mqtt channel whenever it hears one of the above mentioned items in the continuous speech recordings. The program "index.js" (see the attachments) listens to this particular channel, and decodes the output of the Snips app for the ESP-8266 on the hardware markers.

The KiCAD project for the marker board schematic and PCB Layout is at https://github.com/swaglord42/Accio/tree/latest

The complete operation is shown in this video.

Video description for Accio

For more details on working with ESP-8266, follow this link.

I would suggest using only the Raspberry Pi 3B+ for working with Snips, ie, with only the base station, which would run the entire Snips assistant. However, if you decide to use the Satellite configuration(like I did for this project), there is a good setup description for Satellites here.

Schematics

Code

#include <ESP8266WiFi.h>
#include <PubSubClient.h>


// Update these with values suitable for your network.
const char* ssid = "WiFi_name_here";
const char* password = "WiFi_passwd_here";
const char* mqtt_server = "IP_address_of_the_MQTT_master(raspberry pi 3b+ in this case)";
#define mqtt_port 1883
#define MQTT_USER ""
#define MQTT_PASSWORD ""
#define MQTT_SERIAL_PUBLISH_CH "hermes/"
#define MQTT_SERIAL_RECEIVER_CH "hermes/asr/test"
//    "hermes/asr/test" topic has defined by me in index.js file

WiFiClient wifiClient;

PubSubClient client(wifiClient);

void setup_wifi() {

    delay(10);
    
    // We start by connecting to a WiFi network
      
    Serial.println();
    Serial.print("Connecting to ");
    Serial.println(ssid);
    WiFi.begin(ssid, password);
    while (WiFi.status() != WL_CONNECTED) {
      delay(500);
      Serial.print(".");
    }
    randomSeed(micros());
    Serial.println("");
    Serial.println("WiFi connected");
    Serial.println("IP address: ");
    Serial.println(WiFi.localIP());
}

void reconnect() {

  // Loop until reconnected
  
  while (!client.connected()) {
    
    Serial.print("Attempting MQTT connection...");
    // Create a random client ID
    String clientId = "NodeMCUClient-";
    clientId += String(random(0xffff), HEX);
    // Attempt to connect
    
    if (client.connect(clientId.c_str(),MQTT_USER,MQTT_PASSWORD)) {
      
      Serial.println("connected");
      //Once connected, publish an announcement...
      client.publish("/ic/presence/nm/", "hello world");
      // ... and resubscribe
      client.subscribe(MQTT_SERIAL_RECEIVER_CH);
    } else {
      
      Serial.print("failed, rc=");
      Serial.print(client.state());
      Serial.println(" try again in 5 seconds");
      // Wait 5 seconds before retrying
      delay(5000);
    
    }
  }
}


void callback(char* topic, byte *payload, unsigned int length) {
   
   //this is where we add the action code
   //check payload pointer to get data sent from the base station R-Pi over mqtt
   //comment the necessary sections as per requirement  
    
    if(*payload=='1')
    {
      Serial.println("got alpha"); 
      digitalWrite(2,HIGH);
      delay(500);
      digitalWrite(2,LOW);
      delay(500);
      digitalWrite(2,HIGH);
      delay(500);
      digitalWrite(2,LOW);
      delay(500);
      digitalWrite(2,HIGH);
      delay(500);
      digitalWrite(2,LOW);
      delay(500);
      
    }
    else if(*payload=='2')
    {
      Serial.println("got beta");
      digitalWrite(2,HIGH);
      delay(300);
      digitalWrite(2,LOW);
      delay(600);
      digitalWrite(2,HIGH);
      delay(1000);
      digitalWrite(2,LOW);
      delay(600);
      digitalWrite(2,HIGH);
      delay(300);
      digitalWrite(2,LOW);
      delay(300);
      
    }
    else
    {
      Serial.println("got gamma");
      digitalWrite(2,HIGH);
      delay(100);
      digitalWrite(2,LOW);
      delay(100);
      digitalWrite(2,HIGH);
      delay(1000);
      digitalWrite(2,LOW);
      delay(1000);
      digitalWrite(2,HIGH);
      delay(100);
      digitalWrite(2,LOW);
      delay(100);
      
    }
    Serial.println();
}

void setup() {

  pinMode(2, OUTPUT);
  Serial.begin(115200);
  Serial.setTimeout(500);// Set time out for 
  setup_wifi();
  client.setServer(mqtt_server, mqtt_port);
  client.setCallback(callback);
  reconnect();

}


void publishSerialData(char *serialData){
  
  if (!client.connected()) {
    reconnect();
  }
  client.publish(MQTT_SERIAL_PUBLISH_CH, serialData);
}
void loop() {

   client.loop();

     if (Serial.available() > 0) {
     char bfr[101];
     memset(bfr,0, 101);
     Serial.readBytesUntil( '\n',bfr,100);
     Serial.print(bfr);
  }

}

var mqtt = require('mqtt');

var hostname = "mqtt://(ip_addr_of_base_station_r_pi_here)";
var client  = mqtt.connect(hostname);

client.on('connect', function () {
    console.log("[Snips Log] Connected to MQTT broker " + hostname);
    client.subscribe('hermes/#');
});

client.on('message', function (topic, message) {
    if (topic === "hermes/asr/startListening") {
        onListeningStateChanged(true);
    } else if (topic === "hermes/asr/stopListening") {
        onListeningStateChanged(false);
    } else if (topic.match(/hermes\/hotword\/.+\/detected/g) !== null) {
        onHotwordDetected()
    } else if (topic.match(/hermes\/intent\/.+/g) !== null) {
        onIntentDetected(JSON.parse(message));
    }
});

function onIntentDetected(intent) {
  
  //This function is the action code. I have set the keywords as alpha, beta and gamma for the 3 markers. When they are identified, this code sends a ping over the mqtt channel "hermes/asr/test"
    console.log("[Snips Log] Intent detected: " + JSON.stringify(intent));
    if(JSON.stringify(intent).match(/alpha/g)!== null) {
	console.log("alpha detected");
        client.publish("hermes/asr/test","1");
   }
    else if(JSON.stringify(intent).match(/beta/g) !== null) {
	client.publish("hermes/asr/test","2");
        console.log("beta detected");
   }
    else if(JSON.stringify(intent).match(/gamma/g) !== null) {
        client.publish("hermes/asr/test","3");
        console.log("gamma detected");
   }
}

function onHotwordDetected() {
    console.log("[Snips Log] Hotword detected");
}

function onListeningStateChanged(listening) {
    console.log("[Snips Log] " + (listening ? "Start" : "Stop") + " listening");
}