LEGO MINDSTORMS Voice Challenge: Mission 3

Welcome to the world of LEGO MINDSTORMS EV3 and voice with Alexa! These instructions are part of a series created to show you how to connect your EV3 Brick to Alexa, and build custom voice-based interactions that incorporate EV3 motors and sensors. With this knowledge, you can build and submit your own creation to the LEGO MINDSTORMS Voice Challenge – Powered by Alexa. Don’t be shy bringing in other LEGO elements — let your imagination run wild!

Here is what is included in this series:

• Setup: Get your EV3 development environment setup.
• Mission 1: Get your EV3 Brick connected to a compatible Echo device, and reacting to the wake word.
• Mission 2: Build EV3RSTORM's legs, and react to music played from your Echo device.
• Mission 3: Add arms and a cannon to EV3RSTORM, and react to voice commands using an Alexa skill you create. (you are here)
• Mission 4: Give EV3RSTORM eyes (IR Sensor), and make your Alexa skill react anytime an intruder is detected.

Helpful resources

Throughout these missions, several resources will be referenced along the way. These references will come in handy as you learn more about the different ways you can create your own LEGO MINDSTORMS EV3 creation that works with Alexa:

• Alexa Gadgets Toolkit Overview
• Alexa Skills Kit Documentation
• EV3 Python API Documentation
• EV3Dev Python Robot Examples

Mission 3: Make EV3RSTORM move using voice

At this point, you should have completed the Setup Guide, Mission 1, and Mission 2 to get your EV3 Brick connected to your Echo device and reacting to Alexa. You're on a roll!

In this mission, you will explore how to create custom interactions between your EV3 Brick and an Alexa skill. An Alexa skill is like an app for Alexa, and is built using code stored in the cloud. You will:

• Give EV3RSTORM arms, and a cannon
• Create an Alexa skill to control EV3RSTORM
• Make EV3RSTORM move, change speed, and react to custom commands triggered by your voice

In the end, EV3RSTORM can react to an Alexa skill like this:

The code for this mission can be found in the /alexa-gadgets-mindstorms/mission-03 folder in your VS Code workspace. Let’s walk through the steps to complete this mission!

Build EV3RSTORM

For this mission, you will need to have followed the EV3RSTORM build instructions up to page 92. This will give EV3RSTORM the ability to react to movement and other commands. You will also need to attach the cannon (build instructions are on pages 106-116). The build should look like this:

Create your Alexa skill

With the Amazon Developer account you used in Mission 1 handy, you can create your Alexa skill. This skill will react to voice commands and direct the way your robot should move, using commands like, “move forward”, “move back”, “sound the alarm”, and more!

While the code for your gadget is written in Python 3, the code for your skill is written in Node.js in order to take advantage of the Alexa-hosted option for your Skill.

Note: At the time of writing, the Python option for Alexa-Hosted skills was not available, but this method will work. We will work to add a guide for Python-based skills in the future. In the meantime, you can reference the Color Cycler Alexa skill for the Raspberry Pi sample to get started.

Let’s walk through the steps of creating your Skill:

1. Sign in to developer.amazon.com.

2. In the top header, hover over Alexa, and click on Alexa Skills Kit.

3. In the upper-right of the screen, click on Create Skill.

4. Enter a Skill Name, maybe “MINDSTORMS”. The name you give your Skill will also be the way you open the Skill. For example, “Alexa, open mindstorms”, or, “Alexa, tell mindstorms to go forward”. You can modify this later.

5. Select your default language. Make sure the language select matches the language used on your Echo device.

6. Select Custom for the “Choose a model to add to your skill” option.

7. Select Alexa-Hosted for the “Choose a method to host your skill's backend resources” option.

Note: At the time of writing, the Python option for Alexa-Hosted skills was not available, but this method will work. We will work to add a guide for Python-based skills in the future. In the meantime, you can reference the Color Cycler Alexa skill for the Raspberry Pi sample to get started.

8. Click Create skill in the upper-right.

9. Once you click Create skill, you will see a modal while your skill is being prepared to be customized.

Enable the Custom Interface Controller

The Custom Interface Controller allows you to send custom commands to your EV3Brick (Custom Directives), and receive events from your EV3 Brick (Custom Events). For this mission, you will need to enable the Custom Interface Controller, so you can use Custom Directives:

1. Click on Interfaces in the left navigation of the Alexa Developer Console.

2. Toggle Custom Interface Controller to ON.

3. Scroll to the top of the page and click Save Interfaces

That’s it! With Custom Interface Controller toggled on, you can write code that sends custom directives to your EV3 Brick and program how you want it to react. Learn more about Custom Interfaces.

Define the Skill Interaction Model

The Skill Interaction Model defines how you can speak to your skill, and what kind of commands it can expect to respond to. The interaction model includes intents, slots, sample utterances that you define, and program against in your skill’s code. Learn more about the Skill Interaction Model.

1. In the Alexa Developer Console, under Interaction Model, click on JSON Editor.

2. In the /alexa-gadgets-mindstorms/mission-03 folder, you will see a folder called skill-nodejs. Within that folder, there is a model.json file. Copy the interaction model JSON from that file, and paste it into the editor, or drag and drop the JSON file onto the drop zone to upload it.

After pasting the JSON into the Alexa skill JSON Editor, click Save Model, and then Build Model presented at the top of the console interface. It may take some time for the model to build, so be patient. In the meantime, let’s explore the Intents and Slots in the JSON file.

An intent represents an action that fulfills a user's spoken request. Intents can optionally have arguments called slots. For example, in the JSON you will see a MoveIntent with slots for Direction, which maps to a list of words that represent the different directions you can say, and a Duration slot that maps to a predefined AMAZON.NUMBER integer value. The combination of the intent and slots will allow to write code that reacts to:

“Alexa, tell mindstorms to move forward 5 seconds.”

Other intents in the JSON file include SetSpeedIntent, that accepts a Speed slot mapped to AMAZON.NUMBER, which is an unbounded integer. You will need to perform a range check and bound it from 1 to 100 inclusive, in the skill code. The SetCommandIntent with a Command slot that maps to a list of commands defined within CommandType. These intents and slots allow you to write code that reacts to:

“Alexa, tell mindstorms to set speed to 50 percent.”

“Alexa, tell mindstorms to fire the cannon.”

Take a moment to browse through these intents and their slots — key things to note are:

• Intent names (MoveIntent, SetSpeedIntent, SetCommandIntent)
• Slot name (Direction, Duration, Speed, Command)
• Slot values ("forward", "backward", "1", "2", "100")

With your Intents and Slots defined, you’re ready to start implementing the Skill code.

Implementing the Skill Logic

There’s a lot to learn about creating skills, but for the purpose of this mission, we’ll guide you through using the Alexa-Hosted skill option you selected earlier, and share additional resources at the end. With an Alexa-Hosted skill, you can start writing code for your skill directly in the Alexa Developer Console:

1. Click on Code in the top navigation bar of the Alexa Developer Console.

2. In VS Code, open the index.js file in the mission-03/skill-nodejs/lambda folder.

3. Copy the code in the index.js file into the index.js file in the Alexa Developer Console Code Editor.

4. Copy the contents of the package.json and util.js files to the respective files in the Alexa Skill Code Editor.

5. Create a new file by clicking the New File icon in the upper-left of the Code Editor, and fill in the path and file name as /lambda/common.js

6. With the common.js file created, make sure the file is open, and then copy the code in the common.js file from the mission-03/skill-nodejs/ folder in VS Code to the common.js file in the Alexa Skill Code Editor.

Now, let’s take a look at the core elements of the index.js file. The snippet below contains all of the handlers used by this skill. The last four handlers are lifted from the skill template — no modifications needed.

exports.handler = Alexa.SkillBuilders.custom()
    .addRequestHandlers(
        LaunchRequestHandler,
        SetSpeedIntentHandler,
        MoveIntentHandler,
        SetCommandIntentHandler,
        Common.HelpIntentHandler,
        Common.CancelAndStopIntentHandler,
        Common.SessionEndedRequestHandler,
        Common.IntentReflectorHandler, // make sure IntentReflectorHandler is last so it doesn't override your custom intent handlers
    )
    .addRequestInterceptors(Common.RequestInterceptor)
    .addErrorHandlers(
        Common.ErrorHandler,
    )
    .lambda();

There are also four unique handlers that have been added:

LaunchRequestHandler

This handler is invoked when you launch the skill by saying, “Alexa, open mindstorms”. When the skill is launched, the following occurs:The endpointID of the MindstormsGadget is obtained using:

let request = handlerInput.requestEnvelope;
let { apiEndpoint, apiAccessToken } = request.context.System;
let apiResponse = await Util.getConnectedEndpoints(apiEndpoint, apiAccessToken);
if ((apiResponse.endpoints || []).length === 0) {
    return handlerInput.responseBuilder
        .speak("I couldn't find an EV3 Brick connected to this Echo device. Please check to make sure your EV3 Brick is connected, and try again.")
        .getResponse();
}

If there is no EV3 Brick connected to the Echo device, the skill replies with a message. Otherwise, the endpointID is stored as a session attribute. Session attributes allow you to retain data during the skill session:

let endpointId = apiResponse.endpoints[0].endpointId || [];
Util.putSessionAttribute(handlerInput, 'endpointId', endpointId);

Once the endpointID is stored, the skill continues by welcoming the person speaking to the Echo device:

return handlerInput.responseBuilder
    .speak("Welcome, you can start issuing move commands")
    .reprompt("Awaiting commands")
    .getResponse();

SetSpeedIntentHandler

This handler is invoked when you say, “Alexa, tell mindstorms to set speed to 50 percent”, or similar. When the skill is invoked using this utterance, the following code obtains the speed value from the slot and bounds it between 1-100, and stores that value for reference in a session attribute.

let speed = Alexa.getSlotValue(handlerInput.requestEnvelope, 'Speed');
speed = Math.max(1, Math.min(100, parseInt(speed)));
Util.putSessionAttribute(handlerInput, 'speed', speed);

MoveIntentHandler

This handler is invoked when you say, “Alexa, tell mindstorms to move forward 5 seconds”, or other similar supported utterance. When the skill is invoked using this utterance, the following code gets the values from the direction, duration and speed slots to construct a command that will be sent to the EV3 Brick to make EV3RSTORM move. If a value is not available for the speed slot, the command falls back to the speed value stored as as a session attribute.

const request = handlerInput.requestEnvelope;
const direction = Alexa.getSlotValue(request, 'Direction');

// Duration is optional, use default if not available
const duration = Alexa.getSlotValue(request, 'Duration') || "2";

// Get data from session attribute
const attributesManager = handlerInput.attributesManager;
const speed = attributesManager.getSessionAttributes().speed || "50";
const endpointId = attributesManager.getSessionAttributes().endpointId || [];

Once all the parameters are available, a control custom directive is constructed to send to the MindstormsGadget. This custom directive is then parsed on the EV3 Brick using code we’ll review later.

const directive = Util.build(endpointId, 'Custom.Mindstorms.Gadget', 'Control',
    {
        type: 'move',
        direction: direction,
        duration: duration,
        speed: speed
    });
return handlerInput.responseBuilder
    .speak(speechOutput).reprompt("awaiting command")
    .addDirective(directive).getResponse();

SetCommandIntentHandler

This handler is invoked when you say, “Alexa, tell mindstorms to activate sentry mode”, or other similar supported utterance. When the skill is invoked using this utterance, the following code gets the values from the command slot to construct a command that will be sent to the EV3 Brick to make EV3RSTORM react to a special command.

let command = Alexa.getSlotValue(handlerInput.requestEnvelope, 'Command');
let directive = Util.build(endpointId, 'Custom.Mindstorms.Gadget', 'Control',
{
    type: 'command',
    command: command,
    speed: speed
});
return handlerInput.responseBuilder
    .speak(`command ${command} activated`).reprompt("awaiting command")
    .addDirective(directive).getResponse();
}

With this skill logic in place, you can test the skill, but you will find that you receive a response indicating that your EV3 Brick has not been detected:

1. Click Deploy in the upper-right of the Alexa Skill Code Editor. Wait for the deployment process to complete.

2. In the top navigation bar of the Alexa Developer Console, click on Test.

3. You can choose to enable the microphone on your computer or not.

4. Switch the testing from Off to Development using the dropdown under the navigation bar.

5. In the panel on the left, you can either type “open mindstorms”, or say that utterance when you press the microphone button (you will have opted to enable the microphone on your computer for this to work).

6. You should get a response along the lines of, “I couldn't find an EV3 Brick connected to this Echo device. Please check to make sure your EV3 Brick is connected, and try again”, which you specified in the LaunchRequestHandler.

Your skill is working! Let’s complete the other side of the experience by reviewing and running the EV3 code to get EV3RSTORM to react to your skill.

Registration and capabilities

With your skill created, it’s time to review and run the code that can react to the Alexa skill you created. Using the same VS Code workspace from Mission 2, open the mission-03 folder in the alexa-gadgets-mindstorms folder. You should see a familiar series of files: an INI file, and Python file. This will be a common pattern for all the missions you will be working through.

As with the other missions, the first thing you need to do is add your Amazon ID and Alexa Gadget Secret for your registered gadget, and add a new capability to your gadget specified in the initialization file. You can access the code you need to run on the EV3 Brick at /alexa-gadgets-mindstorms/mission-03. Within this folder, open the mission-03.ini file, which should look like:

[GadgetSettings]
amazonId = YOUR_GADGET_AMAZON_ID
alexaGadgetSecret = YOUR_GADGET_SECRET

[GadgetCapabilities]
Custom.Mindstorms.Gadget = 1.0

The above should look familiar. Replace YOUR_GADGET_AMAZON_ID and YOUR_GADGET_SECRET with the ID and Secret tied to the Alexa Gadget you registered in the Amazon Developer Console.

With those changes, your EV3 Brick will be able to connect to your Echo device as an Alexa Gadget, and you’ll see a different capability has been specified. The Custom.Mindstorms.Gadget should look familiar from the skill code you were working with when the commands to send to your EV3 Brick were being constructed. Custom Interfaces are different because YOU get to define what data is sent to and from your gadget, rather than Alexa. The naming convention for adding a Custom Interface is as follows:

Custom.<namespace> = <version>

This can be broken down as:

• Custom. is constant prefix that is always appended to the namespace
• namespace is defined by you. It can be any arbitrary string. In this example, you are using Mindstorms.Gadget as the namespace of our interface. Note that namespace is not case sensitive
• version is set to 1.0 since this is the currently supported version of your Custom Interface. It can’t, and shouldn’t need to, be changed.

With the Custom Interface added to your EV3 Brick’s Gadget Capabilities, you can respond to custom directives sent to your EV3 Brick sent via the Alexa skill you created.

Handling the Custom Directive from the Alexa skill

From the Alexa Skill, you are sending a custom directive — specifically the custom directive with the Custom.Mindstorms.Gadget namespace and Control name. In order for EV3RSTORM to react, you need to define what happens when data is received via the Custom.Mindstorms.Gadget interface you defined. Within the mission-03 folder, open the mission-03.py file.

Similar to handling the Alexa wake word and the music tempo directives, handling this directive requires extending the AlexaGadget class and defining a callback method. Within the MindstormsGadget class, a callback method is defined using a convention to match the capability you added to the.ini file:

on_custom_namespace_name(self, directive)

This method definition breaks down as:

• on_custom is a prefix indicating that this is custom callback method
• namespace is your defined custom interface namespace
• name is your defined custom interface name
• directive is the custom directive with the corresponding namepace and name

The formatting of the method definition is derived from the dot syntax of the Custom Interface, coupled with the custom interface name, and dots are converted to dashes, as well as lowercase characters. So, for the custom interface that maps directly to what was specified in the.ini file, the method looks like:

on_custom_mindstorms_gadget_control(self, directive)

You can learn more about custom directives in the documentation, but the MindstormsGadget class for this mission looks like this:

from agt import AlexaGadget

class MindstormsGadget(AlexaGadget):

    def __init__(self):
        super().__init__()

    def on_custom_mindstorms_gadget_control(self, directive):
        """
        Handles the Custom.Mindstorms.Gadget control directive.
        :param directive: the custom directive with the matching namespace and name
        """
        payload = json.loads(directive.payload.decode("utf-8"))
        print("Control payload: {}".format(payload))
        control_type = payload["type"]

        if control_type == "move":
            """
            Expected params: [direction, duration, speed]
            """
            self._move(payload["direction"], int(payload["duration"]), int(payload["speed"]))

if __name__ == '__main__':
    MindstormsGadget().main()

This receives the custom directive, which you can then parse and use the parameters to make EV3RSTORM respond in a variety of ways.

Implementing the move commands

With the MindstormsGadget configured to receive commands from your skill, you can add the logic to process them. First, let's tackle the move commands. Within mission-03.py, you will use ev3dev2.motor API to control the way EV3RSTORM will move. The movement relies upon MoveTank and other classes in the ev3dev API, which you can refer to in the ev3dev Python documentation. The code looks like this:

from ev3dev2.motor import OUTPUT_A, OUTPUT_B, OUTPUT_C, MoveTank, SpeedPercent, MediumMotor

class MindstormsGadget(AlexaGadget):
    def __init__(self):
        """
        Performs Alexa Gadget initialization routines and ev3dev resource allocation.
        """
        super().__init__()
        """
        ev3dev initialization
        """
        self.leds = Leds()
        self.sound = Sound()
        self.drive = MoveTank(OUTPUT_B, OUTPUT_C)
        self.weapon = MediumMotor(OUTPUT_A)

    def _move(self, direction, duration: int, speed: int, is_blocking=False):
        """
        Handles move commands from the directive.
        Right and left movement can under or over turn depending on the surface type.
        :param direction: the move direction
        :param duration: the duration in seconds
        :param speed: the speed percentage as an integer
        :param is_blocking: if set, motor run until duration expired before accepting another command
        """
        print("Move command: ({}, {}, {}, {})".format(direction, speed, duration, is_blocking))
        if direction in Direction.FORWARD.value:
            self.drive.on_for_seconds(SpeedPercent(speed), SpeedPercent(speed), duration, block=is_blocking)

        if direction in Direction.BACKWARD.value:
            self.drive.on_for_seconds(SpeedPercent(-speed), SpeedPercent(-speed), duration, block=is_blocking)

        if direction in (Direction.RIGHT.value + Direction.LEFT.value):
            self._turn(direction, speed)
            self.drive.on_for_seconds(SpeedPercent(speed), SpeedPercent(speed), duration, block=is_blocking)

        if direction in Direction.STOP.value:
            self.drive.off()
            self.patrol_mode = False

The Python code above sets up the motors connected to the EV3 Brick, and tells EV3RSTORM how to move when it receives commands from the Alexa skill, including:

• How the motors move based on a direction: forward, backward, right, left, and brake
• How the fast the motors will move based on the speed you set with your voice, or what’s stored in the session attribute you stored on the skill-side
• How long the motors should run based on the duration you specified with your voice

You’ll notice how what’s specified in the EV3 Brick Python code maps to the Alexa skill you created earlier.

Implementing other control types

The code above handles the move control type. Our skill interaction model also has command control types. This control type is a one to one mapping. For example, the command fire cannon from the skill maps directly to the fire cannon functionality in the gadget:

def on_custom_mindstorms_gadget_control(self, directive):
        payload = json.loads(directive.payload.decode("utf-8"))
        print("Control payload: {}".format(payload))
        control_type = payload["type"]
        if control_type == "move":
            """
            Expected params: [direction, duration, speed]
            """
            self._move(payload["direction"], int(payload["duration"]), int(payload["speed"]))

        if control_type == "command":
            """
            Expected params: [command]
            """
            self._activate(payload["command"])

    def _activate(self, command, speed=50):
        """
        Handles preset commands.
        :param command: the preset command
        :param speed: the speed if applicable
        """
        print("Activate command: ({}, {})".format(command, speed))
        if command == "circle":
            self.drive.on_for_seconds(SpeedPercent(int(speed)), SpeedPercent(5), 12, block=False)

        if command == "cannon":
            self.weapon.on_for_rotations(SpeedPercent(100), 3)

This code parses the custom directive received from the skill, and maps it to code that defines how EV3RSTORM should react to different commands, including driving in a circle, or firing a cannon.

Test EV3RSTORM with the Alexa skill

Now that you’ve created your skill, and reviewed the Python code that will cause EV3RSTORM to react to that skill, you’re ready to test out the experience. You will need to run the Python code before starting the skill:

1. Make sure VS Code is connected to your EV3 Brick. You should see a green dot next to your EV3 Brick’s name in the EV3 DEVICE BROWSER.

2. Copy the missions folder in your computer workspace to your EV3 Brick. Click on the Send workspace to device button next to the EV3DEV DEVICE BROWSER text that shows up when you hover over the text.

When you click this button, you will see the status of the files being copied in the bottom-right of VS Code.

3. Once the files have copied over to your EV3 Brick, you can run the Python code by navigating to the Python file in the EV3 DEVICE BROWSER , right-click on the Python file, and select Run.

4. Once the program starts, you should see a prompt in the debug console when your EV3 Brick connects to your Echo device.

You will also see a message displayed on your EV3 Brick's screen.

Now comes the fun part! Make some room for EV3RSTORM, and try the following commands to see how EV3RSTORM reacts:

“Alexa, open mindstorms”

Alexa should respond acknowledging the skill session is active and listening for commands. The skill must be in session in order for the other following commands to work.

“Move forward 2 seconds”

Alexa should acknowledge the command, and EV3RSTORM should move forward.

“Set speed to 100 percent”

This will make EV3RSTORM move faster — try running the move command again to see.

“Move in a circle”or "Fire the cannon"

EV3RSTORM should move in a circle, or fire the cannon.

You should also see messages printed to the debug console that look like this:

EV3RSTORM has learned some new tricks that work with an Alexa skill! You can always stop the program by pressing Control + C.

Note: If Alexa responds with something along the lines of, "I don't understand", check and make sure the language that your skill is using matches the language used on your Echo device. e.g. EN-CA, versus EN-US.

Note: If you run into connectivity issues, you can try unpairing your EV3 Brick from your Echo device by following the Unpairing your EV3 Brick instructions in Mission 1. You may also want to try forgetting your EV3 Brick from the Bluetooth settings of your Echo device before pairing again.

Other things to explore

Congratulations, you have completed Mission 3! In this mission, you developed a voice interaction model and integrated it into an Alexa-Hosted skill. Then, you leveraged the Alexa Gadgets Toolkit Custom Interface Controller to send commands to the MindstormsGadget. Finally, you used the MoveTank from the ev3dev2.motor API to react to those commands and make EV3RSTORM move.

To expand on what you did, you could try changing EV3RSTORM’s movements, adding a new command, or bringing in wake word or tempo from previous missions. Check out the Alexa Gadgets Toolkit documentation or explore other capabilities of ev3dev by referring to the documentation.

What to do next

In Mission 4, you’ll create an Alexa skill that that reacts to when the proximity sensor on EV3RSTORM detects an intruder!

Go to Mission 4