Create Custom Actions for Google Assistant

A Google Home Mini recently made it into the collection of gadgets I use on a daily basis for casting music, setting countdown timers, reminders, recipes, etc. However the capabilities of the software behind it, Google Assistant, expand way beyond that. We saw how it could, one day soon, be making calls and booking appointments for you during the Google IO 18 keynote. Additionally, it is expected to be the catalyst for driver experience, as brilliantly demonstrated by Volvo Cars in the promotional video of their next generation infotainment system. It runs native Android just like your smartphone! There is so much potential in this technology I had to try it out myself.

As it often happens, I had a solution sitting in my living room and was looking for a suitable problem. First thought was to create a connected home with various low power sensors transmitting various data such as temperature, humidity, movement, camera feed etc. This project is still in the plans but since it is rather extensive, there is little motivation to kickstart it. Little did I know, the ideal use case for Google Assistant was under my nose or specifically next to my door. A project that shows me every day whether I am running late for the bus, the VasttraPi!

This Raspberry Pi gadget should the upcoming bus departures for the nearby station and displays them on a screen letting me know whether I should hurry during my morning preparation routine. However, lazy as I am, it would be awesome if I could get hold of the bus departures while laying in my sofa. Moreover, I face the same problem at work. Since I do not like waiting for the tram I have to check an app on my phone. Well not anymore! Now I can just say: "Hey Google, ask my personal bus stop about departures from work."

Google Assistant, by default, cannot inform you (in Gothenburg, Sweden) about bus or tram departures directly. It would be great if I could ask "When is the next bus leaving?" The best Google Assistant can do, at the moment, is tell when you should leave to reach a specific destination in time. However this is not good enough for me as I need more details to plan ahead.

In order to customize my interaction with Google Assistant, I first tried IFTTT. This is a really simple to set up service and allows you to control your connected "things". However it has a huge drawback: The communication is one-way. With IFTTT (for now) you can only trigger an operation on your server but not read back the response. This was a showstopper for me as I would not be able to read back the upcoming departures. And now you know why a custom Action had to be created. To get an overview of what an Action as well as the complementary concepts are, please check out this 5 minute video.

Long story short, the fundamental idea is that your "action" will be invoked based on a set of phrases, e.g. "Talk to my personal bus stop." Next, the verbal or written input will be processed by Google and if found relevant, it shall trigger a corresponding "intent" which eventually results in a "fulfillment." In my case, the input which would trigger the intent would be a location. If I mention that I am at a certain location, then whatever I say as well as the keywords that triggered the intent are sent over to my web server. The server will process the key terms and produce a response which fulfills the user's intent. Specifically, if the supplied location is "home" or "work" then the server makes a request to the Västtrafik API and formulates a readable/talkable list of bus departures. The Action has been working well so far, mainly when at work daily and when at home in the afternoons or weekends.

Two obstacles have been encountered. First, I cannot use it in the mornings since it will wake up my girlfriend! But this is what I have the VasttraPi for. Second, "My Personal Bus Stop" is not extendable, as it will only tell me the departures from home and work. Fetching departures from arbitrary stations is not difficult in principle, but it will either require the user to pronounce the name of the station or for Google Assistant to determine the station of interest based on the user's location. The former is difficult to accomplish, since Google Assistant is really bad at Swedish names and they get interpreted wrong. The latter is more feasible, however it will greatly increase the complexity of the project.

Specifically, assuming the GPS location is accurate enough, I have my doubts as to whether the nearest station is actually the one the user is typically interested in. Especially in areas with many stations in the vicinity, the Action would have to present the user with the different available options making the interaction slower. Ultimately, as my girlfriend and I are the primary users of this Action, such a feature would provide little value to us therefore the "cost" to implement it cannot be justified for the time being.

It is very easy to make your own Action which will communicate with your own server. You can go through the official tutorial (strongly suggested) and/or check out these step by step instructions on Instructables. Interested just in the Flask server code? Well, you shall find it on GitHub!