"Hey Mycroft, Turn on the Lamp"

Introduction & Motivation

Voice and home automation make a natural combination. Yesterday's dream of coming home and telling your home what to do is today's reality.

In order to experiment with home automation, I decided to modify an older project of mine, BLElectric Light 101, and make it a voice controlled IoT device with Mycroft/Picroft.

The Overall Design

I've "recycled" the 3D printed Edison style electric bulb from Chuck Hellyebuck's Filament Friday. A neopixel ring enclosed in the bulb provides the light and color effects. In the original project, I used an Arduino 101 to control the ring using its' on board BLE capability.

In this project, I've replaced the Arduino 101 & BLE with an ESP8266 based board the SparkFun ESP8266 Thing - Dev Board, and turned it into a web server for the bulb. This allows the bulb, or thing, to be placed on my home Wifi network.

Using the python HTTP requests library within my MyCroft skill gives us: Voice IoT!

I also decided to hook up a stepper motor to the board. Why? Well because I could! But seriously, from the voice commands I use to actuate it, you can probably guess where I am going with this project in the future. . .

Please note that this project could just be done with the Adafruit Feather HUZZAH with ESP8266 WiFi as well.

Getting Started with an 8266 Based Board

The ESP8266 is a WiFi-enabled microcontroller and it can be programmed through the Arduino IDE. Boards based on this chip from adafruit and Sparkfun are inexpensive and have a large number of libraries, tutorials and code examples to learn from. Their key advantages are the ability to easily program them with the Arduino IDE and WiFi capability. The small size of these boards makes them very easy to imbed into objects and connect easily over WiFi. A relative disadvantage of the board is the small number of GPIO and ADC pins, but as you can see from my project, Huzzah! Color Me With PubNub!, there are ways to overcome this.

Overall, the ease of use, massive community support and sharing and WiFi make this a nearly ideal IoT microcontroller for Makers. The up and coming ESP32 boards look to make any concerns about pin numbers a things of the past!

In order to get started with the board on the Arduino, I point you to both Sparkfun and adafruit web site. Both guides are useful and the recommended setup seem to work for either board:

• ESP8266 Thing Development Board Hookup Guide

• Adafruit Feather HUZZAH ESP8266

Both tutorials cover the installation of the drivers and libraries necessary to use the boards along with the Arduino IDE.

After installing the ESP8266 libraries, you should look through the WiFi example sketches. I used these examples to come up with my own web server code, which is described below.

The Web Server

A web server is a program that listens on a port for http commands and responds to them by executing code along specified routes( A route is code to handle specific http commands - this will be apparent later). The ESP8266 provides several libraries and numerous code examples to make designing your own web server very easy. Don't be intimidated, your web server sketch will follow the same structure of an Arduino IDE program. It will have a setup() and loop() function, along with some functions, etc.

Our web server will have 4 routes. One to handle the root, one to turn the bulb on and off, one to handle light colors and a final one to control our stepper motor. You can think of each of these routes as a function or block of code to execute on demand. In a future refactoring, the bulb related routes will be changed to one route, with 2 parameters, but more on that later. For now I'll go line-by-line on how to set up the code for a web server on your ESP8266 board.

Just like any program written in the Arduino IDE we will write our web server as a sketch to upload onto our board. In order to set up our server, we need to import a few standard libraries:

#include <ESP8266WiFi.h> 
#include <WiFiClient.h> 
#include <ESP8266WebServer.h> 
#include <ESP8266mDNS.h> 

In order to connect our web server to our WiFi network we will have to provide our network id and password and create a web server object to listen on port 80:

const char* ssid = "********"; 
const char* password = "**********"; 
ESP8266WebServer server(80); 

If you follow the code, you'll see a list of functions to handle the routes I mentioned above. However, we'll jump into the setup() function of the sketch first. As we are writing our first web server, it's we will need to write messages to the serial port. This will allow us to see the IP address of the board we are using, which will be essential for us to send commands to the board!

Serial.begin(115200); 
 WiFi.begin(ssid, password); 
 Serial.println(""); 
 // Wait for connection 
 while (WiFi.status() != WL_CONNECTED) { 
   delay(500); 
   Serial.print("."); 
 } 
 Serial.println(""); 
 Serial.print("Connected to "); 
 Serial.println(ssid); 
 Serial.print("IP address: "); 
 Serial.println(WiFi.localIP()); 
 if (MDNS.begin("esp8266")) { 
   Serial.println("MDNS responder started"); 
 } 

Once you see the IP address in the serial output, take note of it. It will be used to communicate with the board in our MyCroft skill. It will be something like 172.16.254.1 and in order to connect to the board you will issue an http command such as:

http://172.16.254.1/stepper 

The IP is the host address of server and '/stepper' is a resource path or route that we specify in our code. In fact the following lines specify each of our paths and the response each server should execute. To understand this, read it like you are the web server:

On receiving the the path "/stepper" from the http request, http://172.16.254.1/stepper, I will then execute the code in the function called handleStepper

server.on("/", handleRoot); 
server.on("/lamp", handleCommand); 
server.on("/color", handleColor); 
server.on("/stepper",handleStepper); 

The stepper handling code is very easy to understand(I apologize for using the delay function) and therefore not very interesting.

So I will cover the handleCommand() code which will help you to not only learn how to create and handle a resource path, but also have to have parameters and values sent to your function! The code for handleCommand is defined in the function void handleCommand() and it must have the exact name specified in the parameter to passed to the server.on() function.

void handleCommand() 
{ 
 String message = "done"; 
 int commandValue; 
 ...

We create a String variable called message to pass back to the client making the request. You can modify this string along the way to pass any message you wish back to your client. A client could be your web browser, a program you write, or in our case the Mycroft/Picroft system.

We then create an integer variable called commandValue. Our client will pass in a parameter value. In this case the parameter value will be a 1 or 0 depending on whether the client wishes to turn the lamp on or off. In a browser window this would look something like:

http://176.16.254.1/lamp?cmd=1 
or
http://176.16.254.1/lamp?cmd=0

Remember we created the /lamp resource path to handle this? When this resource path is followed, the function called to handle it has access to the parameters and values sent by using the server.arg() function. in our case there is just one called cmd:

commandValue = (server.arg("cmd")).toInt(); 

First, of course, we check to make sure the client sent in a parameter, and if not used our message variable to send an error message back to the client. There are much better ways to handle this, but since we are creating the client and the commands it will send, this is good enough for now. The cmd parameter value is received as a String type. In order to use it effectively in the very elegant switch/case control structure, we cast it to an integer type with the toInt() function of the String class. We can then execute code corresponding to turning the lamp on or off:

switch(commandValue) 
{ 
  case 0: 
   currentColor = 0; 
   setLampColor(); 
   break; 
  case 1: 
   currentColor = 1; 
   setLampColor(); 
   break; 
  default: 
   message = "choose from 0, 1 for lamp off/on"; 
   break;      
} 

Two important points here. First if the client should send in a nonsensical cmd parameter value, we will send an error message back to the client. Second, we use a state variable called currentColor to set and keep track of the current lamp color. In the case of turning the lamp on or off, this would be white or black respectively. Using this approach we can reuse setLampColor() for both on/off and specific color setting. The variable currentColor acts not only as the state variable of the bulb, but also is the index into an array of neopixel color objects.

If you followed all that, you can design a web server of your own to embed in any 8266 project and make a connected thing!

Clients

Now that the web server is up and running we can work on developing a client in python.

As we saw above, we can use a web browser as our client, perhaps even writing our own custom HTML/Javascrip/CSS page to interact with it. We could also write our own python script to do the same. As long as the computer running it has access to your network you are in business. You can then test much of the functionality you would like to include in your voice skill prior to implementing the skill in Mycroft.

If you are comfortable with wring Mycroft skills, you don't have to write a script to test the client functionality outside of the of Mycroft. The bulk of the action in a Mycroft skill occurs in the skill's __init__.py file. You can implement the python code write in this file and debug from there. It's up to you!

Requests & The IoT Demo Skill

If you haven't created your own skill yet, I recommend taking a look these resources:

• Creating a skill

• Geeked Out Solutions

• MycroftAI/mycroft-skills

• btotharye/mycroft-homeassistant

After looking at these, go ahead and get your own "Hello World" skill up and running! The following is not a comprehensive tutorial on skill design and implementation, just the highlights and some insights.

It all starts with this:

import requests

If you can do it with python on your raspberry pi, you can control it with your voice under Mycroft!

The key to interacting with our web server with python is the Requests: HTTP for Humans library. Nothing describes the library better than their own description:

Requests is an elegant and simple HTTP library for Python, built for human beings.

And it is designed and built for humans to use! We will use this library to communicate with our IoT devices under voice control. The library will allow us to simply replicate the http commands discussed above within our python intent code.

This IoT Demo skill will be composed of 3 intents:

def handle_lamp_command_intent(self, message):
def handle_lamp_color_intent(self, message):
def handle_feeder_intent(self,message):

Each one of these intents will correspond to one of the routes on our server(except for the root route). It doesn't have to be this way and perhaps this is not the most efficient design, but it works and I'll stick with it for now.

Registering the intent in the initialize() function:

lamp_color_intent = IntentBuilder("LampColorIntent").require("LampColorKeyword").require("ColorName").build()
	self.register_intent(lamp_color_intent, self.handle_lamp_color_intent)

It is important to note that this intent will use the LampColorKeyword.voc file to match the utterance with this specific intent. The file looks like this:

change
make

This allows is to say something like, "Hey Mycroft, make the color pink." This is not really ideal, but I had some difficulty it creating the proper diction for this intent and getting it to recognize phrases like this. As I learn more about intents and their relation to the vocab or .voc files, it seems to me that these are intended to be composed of key words that make to utterances and not the complete utterance. I think this could cause some concern for collision of overlapping keywords. I have found and continue to find the following resources key in my continued understanding of intents, keywords, etc:

• btotharye/mycroft-homeassistant

• What is Adapt?

As we are changing the color of the lamp, we need to be able to respond to a color requested by the user. This is done with the use of regex. The file, lampactions.rx looks like this:

(turn|switch) (?P<Action>on|off) (?P<DeviceName>.*)
(change|make) (?P<Action>color|color to) (?P<ColorName>.*)

If btoharye looks at this, he'll notice some similarity to his home assistant work!

The second line is our focus. This regex will allow us to pass the color named by the user under the variable name ColorName on the message bus. The match to this is made by matching the words change or make. I am not using the Action variable at this time.

def handle_lamp_color_intent(self, message):	
	lamp_color = message.data.get("ColorName")
	LOGGER.info("Lamp Color: " + lamp_color)
	if self.color_map.has_key(lamp_color):
		self.speak_dialog("lamp.color",{"color": lamp_color})
		color_index = self.color_map[lamp_color]
		r = requests.get('http://ip_here/color?color='+str(color_index))
	else:
		self.speak_dialog("lamp.color.error",{"color": lamp_color})

We can create the function handler and use the parameter message to grab the the named color from the users' utterance:

 def handle_lamp_color_intent(self, message):	
	lamp_color = message.data.get("ColorName")

The LOGGER object prints out these statements to the /var/log/mycroft-skills.log. Another way to debug is to have Mycroft speak with the self.speak_dialog() function. Prior to this, I was writing the python skill functionality seperately and them bringing that code into the __init__.py file. However, getting comfortable with debugging "within" Mycroft itself will save you some time and I recommend moving towards this approach.

LOGGER.info("Lamp Color: " + lamp_color)

Prior to all of this, we created a python map called color_map:

def __init__(self):
      . . .
    self.color_map = {'black': 0, 'white': 1, 'blue': 2, 'green': 3, 'orange': 4, 'red': 5, 'purple': 8, 'yellow': 9, 'pink': 10}

This approach allows us to take the ColorName utterance and use it as a key to access a corresponding color value to send to the ESP8266 Dev Thing. As you can see from this code in the sketch, the color key values and color indexes match.

// From the sketch file running on the ESP8266 board
void setStandardColors() 
{ 
 black   = neoRing.Color(0,0,0); //used for 'off' state 
 white   = neoRing.Color(255,255,255); 
 blue    = neoRing.Color(0,0,255); 
 green   = neoRing.Color(255,0,0); 
 orange  = neoRing.Color(140,255,0); 
 red     = neoRing.Color(0,255,0); 
 red_orange  = neoRing.Color(69,255,0); 
 sky_blue    = neoRing.Color(206,135,235); 
 purple      = neoRing.Color(0,255,255); 
 yellow      = neoRing.Color(255,255,0); 
 pink        = neoRing.Color(0,255,180); 
 standardColors[0] = black; 
 standardColors[1] = white; 
 standardColors[2] = blue; 
 standardColors[3] = green; 
 standardColors[4] = orange; 
 standardColors[5] = red; 
 standardColors[6] = red_orange; 
 standardColors[7] = sky_blue; 
 standardColors[8] = purple; 
 standardColors[9] = yellow; 
 standardColors[10] = pink; 
} 

This makes sending the color message easy, especially as we are using the request library!

if self.color_map.has_key(lamp_color):
	self.speak_dialog("lamp.color",{"color": lamp_color})
	color_index = self.color_map[lamp_color]
	r = requests.get('http://ip_here/color?color='+str(color_index))

Where ever you see ip_here, replaced this with your boards IP address. Future versions will include this in a settings.json file.

In the case where there is no key matching an available color, we report this using the error dialog in the lamp.color.error.dialog file. We can help the user by repeating the requested color, with some valid options(for more on this, see below).

sorry {{color}} not available maybe try red green or blue
sorry {{color}} not available maybe try orange yellow or purple
{{color}} is not an option you can try red green or blue instead
{{color}} is not an option you can try purple orange or yellow
sorry could you try purple blue or orange instead of {{color}}
sorry could you try red yellow or green instead of {{color}}

Voice Design

There are so many colors! How do you know which to try and how can we avoid cognitive overload while providing helpful suggestions . . .such are the challenges of voice interfaces.

When Mycroft does not recognize the color requested, such as "chartreuse", it replies with a list of 3 alternatives. The three alternatives are a nice way of not only letting the user know some valid options, they also steer the user towards basic color requests. A comprehensive of list of all the color options available would be too much to handle and take up too much time.

sorry {{color}} not available maybe try red green or blue
sorry {{color}} not available maybe try orange yellow or purple
... 

As I reflect on good voice design I dislike my phrases here: " ..maybe try"? Shouldn't the device know what colors it knows? Lame, I know, I'll get rid of this . . .

Additionally these alternatives are "hard" coded in the .dialog file. A future refactoring would perhaps benefit from proving 3 random colors from those available. Another option, that would give a greater sense of intelligence and responsiveness would be to have Mycroft provide 2 or 3 color options closest in hue to the requested option.

Currently multi-turn dialogues are not directly supported by Mycroft:

Does Mycroft support multiturn/conversational skills?

This is a work in progress. In the current mainline there is the ability to tell Mycroft to listen immediately after saying anything. Inside a skill you would invoke this with self.speak("utterance", expect_response=True) ... see https://github.com/MycroftAI/mycroft-core/blob/dev/mycroft/skills/core.py#L3262)

That just allows the user to talk back to Mycroft without having to say "Hey Mycroft".

For real conversational interaction, there is some ongoing work to implement a converse() method. That is in a forthcoming pull request: https://github.com/MycroftAI/mycroft-core/pull/9253

This will allow a Skill to have a preview of utterances, before invoking the Adapt intent parser. Only recently used skills receive the converse() notification and it is passed to them in order of use. So you could code in a skill that does something like this:

@intent_handler(IntentBuilder().require('alarm').require('cancel').build()) def handle_cancel_alarm(): self.speak("are you sure you want to cancel?", expect_response=True) def converse(self, utterances, lang="en-us"): if utterances == "yes": # do whatever self.speak("Alarm canceled")

That is obviously simple-minded, but with this code the following interaction would work:

User: Hey Mycroft, cancel the alarm please Mycroft: Are you sure you want to cancel? User: Yes Mycroft: Alarm canceled

There will be more tools to make building and managing conversations easier in the future, but this is the foundation.

Here, where we have an unknown color requested, a multi-turn approach would be ideal from a user experience perspective.

The 3D Print

You can read about the 3D print and how to put the components together in my write up, BLElectric Light 101. I want to restate here that the Lulzbot mini was able to print the bulb as a single unit without supports. Try it, it works and makes your post-print finishing much, much easier!

The Design

Demos

Conclusions

Hope you enjoy this project . . .more on the way!