Alexa Copilot is a voice controlled copilot for FlightGear flight simulator. The aim is to fully automate the tasks typically performed by copilot and reduce pilot's workload so that he/she can focus on flying. It is an attempt to go one step further with Amazon Alexa and use it for voice-automating systems that are normally controlled manually. (View this text in correct formatting on our Wiki page)
Alexa Copilot is constructed by combining :
- A custom Alexa Skill Set to handle voice commands and map them to intents
- A flask-ask based to app to handle Alexa requests
- A Telnet based Copilot that issues commands to FlightGear based on Alexa requests
- A HiveMQ based MQTT server for communication between flask-ask app and Telnet Copilot
This section contains step-by-step instructions on how to install Alexa Copilot, to test its features and set up the required things for development. The instructions for Mac OS X but all tools/libraries used are Linux compatible.
1 - Installing FlightGear- Follow the instructions on this page to download and install FlightGear simulator on your system
- Once installed, we are going to run FlightGear from the command line. For Mac, the installation path needs to be added to the
$PATH
variable. This can be done in~/.bash_profile
file :export PATH=$PATH:/Applications/FlightGear.app/Contents/MacOS/
- Now run FlightGear using below arguments. This code is also present in runme_fg file :
fgfs --telnet=5401 --timeofday={noon} --airport=SSFF --runway=02
- The
--telnet=5401
starts a Telnet server listening on the port 5401. This is what we will be using to communicate with FlightGear
- We are going to setup a local MQTT server using HiveMQ. Signup to download a trial version of HiveMQ for the purpose of development from this page and download HiveMQ
- Once downloaded go to the
bin
directory and start the server by executing./run.sh
- Leave the server running as long as you plan on using Alexa Copilot
- Install Paho MQTT Python client which will allow use to communicate using MQTT protocol. Use
pip install paho-mqtt
.
In order to test Alexa without a Echo/Tap/Dot device, we are going to install a local version of Alexa app.
- Follow this tutorial to setup the app on your PC.
- Once the installation and configuration is done, we need to run :
- The Node.js based
{INSTALL_PATH}/samples/companionService
usingnpm start
- The
{INSTALL_PATH}/samples/javaClient
bymvn exec:exec
. Before running this, you might need to add<argument>-Djna.library.path=/Applications/VLC.app/Contents/MacOS/lib/</argument>
to{INSTALL_PATH}/samples/javaclient/pom.xml
. This enables the app to find VLC libraries required for recording and playing back sound
The skill we have developed is still under the Certification process and so can't be used directly. This section will tell you how to install our custom Alexa Skill on your Amazon account so that you can edit and test it. The Amazon Webpage gives more information about Understanding Custom Kills.
4a - Building Intent Model on Amazon- Go to the Alexa Section in the developer portal.
- From the developer portal console page, click Apps & Services, then Alexa.
- In the Alexa Skills Kit box, click Get Started. This displays a list of your existing Alexa skills.
- Click the Add a New Skill button.
- Select the Language you want to add from the drop-down list. You can add additional languages after completing the initial configuration. For the Skill Type, select Custom Interaction Model.
- Enter the Name and Invocation Name.
- In the Interaction model :
- Under Intent Schema, copy and paste the contents of intent_schema.json file. We have designed the schema by trying to map various control options in the simulator to Intents in Alexa. For e.g.
EngineIntent
will consist of actions related to engines.
- Under Custom Slot Types, copy and paste contents of [custom_slot_types.txt]. These slots contain various values that simulator controls can take. For e.g.
throttle
can be set to any value contained inLIST_OF_THROTTLE_VALS
- Under Utterances copy the contents of utterances.txt. Utterances are simply the things a pilot can say to Alexa Copilot in order to control various things on the aircraft. If the utterances are significantly different from each other then the Intent also has to be made different. Thus
AutoStartIntent
needs to be a separate intent and can't be included inEngineIntent
- Save the Intent model.
We try to map controls on the pilot's control panel to Intents. Example of EngineIntent
is shown below. Here we have modeled 3 possible controllable options that pilot can use with the engine - throttle, mixture and prime
.
{
"intent": "EngineIntent",
"slots": [
{
"name": "THROTTLE",
"type": "LIST_OF _THROTTLE_VALS"
},
{
"name": "MIXTURE",
"type": "LIST_OF_MIXTURE_VALS"
},
{
"name": "PRIME",
"type": "LIST_OF_PRIME_VALS"
}
]
},
Each control comes with a set of values which the pilot can specify while giving the command or utterance. Thus when the pilot gives the command to throttle engine to ten percent
, the value ten
will get captured by Alexa.
EngineIntent throttle the engine to {THROTTLE} percent
EngineIntent push the mixture to {MIXTURE} percent
EngineIntent prime the engine {PRIME} times
After all the flight controls are mapped to intents, the VUI diagram would be something similar to below :
In order to handle requests, we will be building flask-ask based app. For the purpose of testing we will be hosting it locally using ngrok and then provide the URL to Amazon. A video tutorial by the creator of flask-ask can be found here.
- Download and unzip ngrok. Start the server by executing
./ngrok http 5000
. This will open a screen similar to below :
- Copy the
https
URL and go to the Configuration section on the Alexa Skill Develop page on Amazon
- Under Endpoint, select HTTPS and paste the
ngrok
URL
- Go to the SSL section and select the second options as below :
Now the custom skill is correctly configured to reroute Alexa requests to our local computer
- Keep the server running to prevent the HTTP URL from changing. Under the free version of
ngrok
, the URL changes everytime it is re-launched
- Go to a new terminal and in the alexa_custom_skill directory execute the skill by
python alexa_copilot_skill.py
. The skill handler is designed to :
- Receive requests coming from Alexa through the HTTP server and send back responses. You can test this by going to Test section on the Amazon Skill Developer page. Enter one of the utterances such as
throttle the engine to ten percent
and output will be like below :
"request": {
"type": "IntentRequest",
"requestId": "EdwRequestId.72244989-70ff-4729-a09c-75d4aa97447f",
"locale": "en-US",
"timestamp": "2016-12-18T15:46:44Z",
"intent": {
"name": "EngineIntent",
"slots": {
"MIXTURE": {
"name": "MIXTURE"
},
"PRIME": {
"name": "PRIME"
},
"THROTTLE": {
"name": "THROTTLE",
"value": "10"
}
}
}
As we can see from the request
object, Alexa is able to resolve the utterance into THROTTLE
type of EngineIntent
schema with a value
of 10.
- Start HiveMQ on your local machine
- Start javaClient and companionService from the Alexa Voice app configured in Section 3
- Start the ngrok server to tunnel HTTP requests from Alexa
- Start FlightGear using the
runme_fg
script
- Start the Telnet Copilot
- Press the Listen button on the javaClient GUI and talk to Alexa as done in this demo video :
Comments