Here is a tutorial to show how to simply connect google assistant to Amazon AWS through DialogFlow.
The goal is to is make a program which is able to manage and response a user voice request.
But first, what is DialogFlow?
Dialogflow is a Google-owned developer of human–computer interaction technologies based on natural language conversations.
The following image shows the Flow of conversation within DialogFlow
The component we are going to use here are:
User: We, Machines!
Text / Voice : The user interacts with an app like smart watch / google home.
Agents : Help convert user request into actionable data.
Intent: Support or the service that the user wants from the agent. Intent is configured by the developers. Intent determines the action by the code.
Fulfillment: This is a web-hook. This part of the conversation lets you pass on the request from your app to an external source and get response and pass it back to the user. Setting up a web-hook allows you to pass information from a matched intent into a web service and get a result from it.
Note
No programming experience is required as this series is mostly going to be using GUI(Graphical User Interface) of Google’s dialogflow
DialogFlow - Walkthrough
Setting up DialogFlow account
- Go to : https://dialogflow.com/
- Create an account with a gmail account, and “agree” to the terms & conditions.
We are gonna build an agent that helps us with workflow monitor e.g, check if we have tires in the inventory
Build the Agent
Agents translate user requests to actionable data i.e. intents. It’s essentially a module within dialog flow which incorporates Natural Language Processing to understand what the user meant and to figure out what “action” has to be carried out
Explore the settings page…the ML settings gives us the type of machine learning we want to use for this agent.
In this project we will use Use the Hybrid (Rule-based and ML) mode.
We will define the threshold value for the confidence score at 0.3, In simple terms, if the match is less than 30% percent then the inbuilt “fallback intent” (sorry, I don’t understand etc) will be triggered.
Think of using Intent as a mapping what a user says and what your software should execute. Intents are configured by developers and used to determine the action taken by the code.
Phrases you can expect from the user that will trigger the intent.
Action & Parameters
The inputs you might need from the user to take an action on the user request. For eg: In tires intent, we might need the Size, etc to finish the action. We have to set entities first for it to be mapped here but go with the flow for now.
Response
The responses of the agent in this project will be sent to the Google Assistant integration.
Entities are used to extract parameter values from user queries. So when the user says “Book me a flight ” we usually ask for the city, date etc.these are called entities.
here our Entities will be called tire which will be the size of the tire 16, 18 inch etc
Fulfillment
Our Amazon AWS web service will receive a POST request from Dialogflow in the form of the response to a user query matched by intents with webhook enabled.
In the next part we will explain how to configure AWS in order to receive dialogflow POST requests and be able to respond.
AWS - Walkthrough
Create an AWS user and get in the AWS console.
DynamoDB
First let's open DynamoDB console and let's create a db table called "orders" where we will store all the orders requests for certain tire size. We will insert the primary key and the sort key.
Create an AWS IoT Rule
Next, we should open the IoT console and link an MQTT topic to a certain rule which will trigger some actions. So we click on the "Act" tab, and we create a rule.
Set rule query statement
We define the rule query statement which will identify to which mqtt topic we are linked. We want to listen on outOfStock messages and insert in the DB a new order.
Rule options
By creating the rule we can choose which kind of actions the rule can triggers, there are a lot of options.
Configure “insert in DynamoDB” action
We will choose the action to insert the payload of the message received in a DynamoDB table which we already created.
NOTE: In order to automatically insert an item in a dynamoDB table through AWS IOT rule, both primary and sort key need to be defined on the destination table.
Iot Role
For every type of action AWS is asking to assign a particular role to that action in order to give the permission to one service of AWS to access another service. In this case we create a new role which will be allowed to access DynamoDB from AWS IoT.
Create a Lambda function
Now that we created an MQTT topic and an action we need to actually publish to that topic. We will do so by using an AWS Lambda Function. Our function will be called by google assistant, our google action will send us a JSON request asking for a certain tire size.
So next we need to open the Lambda Console and we create a lambda function from scratch, choosing as a language Python.
We need to fill the basic information..
Allow Lambda to access AWS IoT
In order for the lambda function to access DynamoDB or AWS IoT it should has a particular role. We need to go to the IAM Console (Identity Management Console). We will see that a role for our function was already created in the list. We click on that role and we will attach an inline policy by pasting this JSON which will give to our function the permission to publish on an IOT topic.
By refreshing our lambda page we can see that on the "output" side of the function we now have AWS Iot as a possible "output".
Allow Lambda to access DynamoDB
Still on the IAM Console on our function role we will then add the micro-services policy which will give us the DynamoDB access permission.
And again by refreshing the lambda page we see that DynamoDB was enabled
Write the Lambda function
We now start by writing our function. We parse the JSON request looking for the tire size requested, we access the db in the "stock" table (which we created earlier with some items, at the end of the post you can see a snapshot) to see if a tire with that size exist. If it exist we will respond to google with the model name and a picture of it.
Otherwise we will answer to Google Assistant by saying that we couldn't find anything and we will publish an outOfStock event by requesting a certain size and a certain quantity.
Here's the full code:
import json
import boto3
from boto3.dynamodb.conditions import Key, Attr
from botocore.exceptions import ClientError
def lambda_handler(event, context):
#Parse your JSON message and get out all your attributes
tire_size = event["queryResult"]["parameters"]["tires"]
tire_size = tire_size[:2]
print("Tire size requested:", tire_size)
#Accessing DB to check for existence
dynamodb = boto3.client('dynamodb')
db_response = dynamodb.get_item(TableName='stock', Key={'inches':{'N': str(tire_size)}})
try:
#Found an item corresponding to the size requested
item = db_response['Item']
print("Found model:", item["model"]["S"])
model = item["model"]["S"]
#Preparing the response for Google Dialogflow
response = { "payload":
{
"google": {
"expectUserResponse": "true",
"richResponse": {
"items": [
{
"simpleResponse": {
"textToSpeech": "Here's what I got Gianluca"
}
},
{
"basicCard": {
"title": str(tire_size) +"inches tires - "+ model,
"image": {
"url": "https://di-uploads-pod4.dealerinspire.com/mercedesbenzbrampton/uploads/2017/05/mercedes-benz-tires.jpg",
"accessibilityText": "Cool tires"
},
"imageDisplayOptions": "WHITE"
}
}
]
}
}
}
}
except KeyError as e:
#Couldn'find a correct model for the size
print("Couldn't find anything")
#Preparing the response for Google Dialogflow
quantity = 50
response = { "payload":
{
"google": {
"expectUserResponse": "true",
"richResponse": {
"items": [
{
"simpleResponse": {
"textToSpeech": "The item you requested is out of stock, I placed an order for you"
}
},
{
"basicCard": {
"title": str(quantity)+ " "+ str(tire_size) +"inches tires were ordered",
"image": {
"url": "https://di-uploads-pod4.dealerinspire.com/mercedesbenzbrampton/uploads/2017/05/mercedes-benz-tires.jpg",
"accessibilityText": "Cool tires"
},
"imageDisplayOptions": "WHITE"
}
}
]
}
}
}
}
#notifying the order to aws-iot
client = boto3.client('iot-data', region_name='us-east-2')
client.publish(
topic='iot/outOfStock',
qos=0,
payload=json.dumps({"inches":int(tire_size), "quantity":quantity})
)
return response
REST API creation
In our last step we need to give Google Dialogflow a link to which send its requests. Google is able to only send POST requests so we open the API Gateway console of AWS and we create our API, choosing the protocol and a name.
API Resource creation
Then we click on "Actions" and "Create new resource" which we will call "tire"
Create API method
Then again on "Action" we click on create Method and we will select "POST" method, we will then select the integration type which in our case is a Lambda Function. We need to insert our Lambda function name and when we click OK it will ask the confirmation to give the API the access to that function.
Deploy API
In order to make it accessible from Google we need to deploy our API. So click again on Action "Deploy API" and create a new stage name. Now by clicking on the "Stages" section of your API (on the left menu), you can find the stage name you created and by clicking on it you will get the URL to which external services can send their POST requests.
API triggers Lambda Function
By doing these steps we can refresh again our lambda function page and we will see that automatically the API Gateway was added as a trigger for the lambda function.
Subscribe to MQTT & Test
Now we can finally test all of our environment. We can go on the AWS-IOT console in section "Test", subscribe to the topic "outOfStock" where we will publish with the lambda function. We send a google voice command which will ask for a tire size which is not in the DB and here we will see that a message was published asking for an order.
Results
As a result in our DynamoDB "order" table we can see the that the item was inserted.
The Stock Table
This is our "stock" table
Comments
Please log in or sign up to comment.