Alexa Copilot

Alexa Copilot is a voice controlled copilot for FlightGear flight simulator. The aim is to fully automate the tasks typically performed by copilot and reduce pilot's workload so that he/she can focus on flying. It is an attempt to go one step further with Amazon Alexa and use it for voice-automating systems that are normally controlled manually. (View this text in correct formatting on our Wiki page)

Alexa Copilot is constructed by combining :

• A custom Alexa Skill Set to handle voice commands and map them to intents

• A flask-ask based to app to handle Alexa requests

• A Telnet based Copilot that issues commands to FlightGear based on Alexa requests

• A HiveMQ based MQTT server for communication between flask-ask app and Telnet Copilot

Setup Instructions

This section contains step-by-step instructions on how to install Alexa Copilot, to test its features and set up the required things for development. The instructions for Mac OS X but all tools/libraries used are Linux compatible.

1 - Installing FlightGear

• Follow the instructions on this page to download and install FlightGear simulator on your system

• Once installed, we are going to run FlightGear from the command line. For Mac, the installation path needs to be added to the $PATH variable. This can be done in ~/.bash_profile file : export PATH=$PATH:/Applications/FlightGear.app/Contents/MacOS/

• Now run FlightGear using below arguments. This code is also present in runme_fg file :

• fgfs --telnet=5401 --timeofday={noon} --airport=SSFF --runway=02

• The --telnet=5401 starts a Telnet server listening on the port 5401. This is what we will be using to communicate with FlightGear

2 - Installing MQTT

• We are going to setup a local MQTT server using HiveMQ. Signup to download a trial version of HiveMQ for the purpose of development from this page and download HiveMQ

• Once downloaded go to the bin directory and start the server by executing ./run.sh

• Leave the server running as long as you plan on using Alexa Copilot

• Install Paho MQTT Python client which will allow use to communicate using MQTT protocol. Use pip install paho-mqtt .

3 - Installing Alexa Voice App

In order to test Alexa without a Echo/Tap/Dot device, we are going to install a local version of Alexa app.

• Follow this tutorial to setup the app on your PC.

• Once the installation and configuration is done, we need to run :

• The Node.js based {INSTALL_PATH}/samples/companionService using npm start

• The {INSTALL_PATH}/samples/javaClient by mvn exec:exec . Before running this, you might need to add <argument>-Djna.library.path=/Applications/VLC.app/Contents/MacOS/lib/</argument> to {INSTALL_PATH}/samples/javaclient/pom.xml . This enables the app to find VLC libraries required for recording and playing back sound

4 - Setting up the Custom Alexa Skill

The skill we have developed is still under the Certification process and so can't be used directly. This section will tell you how to install our custom Alexa Skill on your Amazon account so that you can edit and test it. The Amazon Webpage gives more information about Understanding Custom Kills.

4a - Building Intent Model on Amazon

• Go to the Alexa Section in the developer portal.

• From the developer portal console page, click Apps & Services, then Alexa.

• In the Alexa Skills Kit box, click Get Started. This displays a list of your existing Alexa skills.

• Click the Add a New Skill button.

• Select the Language you want to add from the drop-down list. You can add additional languages after completing the initial configuration. For the Skill Type, select Custom Interaction Model.

• Enter the Name and Invocation Name.

• In the Interaction model :

• Under Intent Schema, copy and paste the contents of intent_schema.json file. We have designed the schema by trying to map various control options in the simulator to Intents in Alexa. For e.g. EngineIntent will consist of actions related to engines.

• Under Custom Slot Types, copy and paste contents of [custom_slot_types.txt]. These slots contain various values that simulator controls can take. For e.g. throttle can be set to any value contained in LIST_OF_THROTTLE_VALS

• Under Utterances copy the contents of utterances.txt. Utterances are simply the things a pilot can say to Alexa Copilot in order to control various things on the aircraft. If the utterances are significantly different from each other then the Intent also has to be made different. Thus AutoStartIntent needs to be a separate intent and can't be included in EngineIntent

• Save the Intent model.

Intent Modeling Strategy

We try to map controls on the pilot's control panel to Intents. Example of EngineIntent is shown below. Here we have modeled 3 possible controllable options that pilot can use with the engine - throttle, mixture and prime .

{ 
     "intent": "EngineIntent",  
     "slots": [  
       { 
         "name": "THROTTLE", 
         "type": "LIST_OF _THROTTLE_VALS" 
       }, 
       { 
         "name": "MIXTURE", 
         "type": "LIST_OF_MIXTURE_VALS" 
       }, 
       { 
         "name": "PRIME", 
         "type": "LIST_OF_PRIME_VALS" 
       } 
     ] 
}, 

Each control comes with a set of values which the pilot can specify while giving the command or utterance. Thus when the pilot gives the command to throttle engine to ten percent , the value ten will get captured by Alexa.

EngineIntent throttle the engine to {THROTTLE} percent 
EngineIntent push the mixture to {MIXTURE} percent 
EngineIntent prime the engine {PRIME} times 

After all the flight controls are mapped to intents, the VUI diagram would be something similar to below :

4b - Handling Alexa Requests

In order to handle requests, we will be building flask-ask based app. For the purpose of testing we will be hosting it locally using ngrok and then provide the URL to Amazon. A video tutorial by the creator of flask-ask can be found here.

• Download and unzip ngrok. Start the server by executing ./ngrok http 5000 . This will open a screen similar to below :

• Copy the https URL and go to the Configuration section on the Alexa Skill Develop page on Amazon

• Under Endpoint, select HTTPS and paste the ngrok URL

• Go to the SSL section and select the second options as below :

Now the custom skill is correctly configured to reroute Alexa requests to our local computer

• Keep the server running to prevent the HTTP URL from changing. Under the free version of ngrok , the URL changes everytime it is re-launched

• Go to a new terminal and in the alexa_custom_skill directory execute the skill by python alexa_copilot_skill.py . The skill handler is designed to :

• Receive requests coming from Alexa through the HTTP server and send back responses. You can test this by going to Test section on the Amazon Skill Developer page. Enter one of the utterances such as throttle the engine to ten percent and output will be like below :

"request": { 
   "type": "IntentRequest", 
   "requestId": "EdwRequestId.72244989-70ff-4729-a09c-75d4aa97447f", 
   "locale": "en-US", 
   "timestamp": "2016-12-18T15:46:44Z", 
   "intent": { 
	      "name": "EngineIntent", 
	      "slots": { 
	        "MIXTURE": { 
	          "name": "MIXTURE" 
	        }, 
	        "PRIME": { 
	          "name": "PRIME" 
	        }, 
	        "THROTTLE": { 
	          "name": "THROTTLE", 
	          "value": "10" 
	        } 
	      } 
   } 

As we can see from the request object, Alexa is able to resolve the utterance into THROTTLE type of EngineIntent schema with a value of 10.

Running Alexa Copilot

• Start HiveMQ on your local machine

• Start javaClient and companionService from the Alexa Voice app configured in Section 3

• Start the ngrok server to tunnel HTTP requests from Alexa

• Start FlightGear using the runme_fg script

• Start the Telnet Copilot

• Press the Listen button on the javaClient GUI and talk to Alexa as done in this demo video :