This project has been created to be displayed at Lego World Utrecht 2019 (Netherlands). Each year adult LEGO fans are displaying their lego creations in Utrecht to show the capabilities of Lego Mindstorms. Hence, one of the leading edge technologies of voice control could not be missed at this event this year.
The key idea for this project was to build a fun and interactive game. After a few days of testing and observing how people interact with Alexa the following game was created:
The game consist out of a Lego Mindstorms Robot that can be commanded by Alexa through a field to get to the correct Color.
During the whole event around 100.000 people got the chance to play with Alexa and see what the capabilities of Lego Mindstorms are.
Next, the game is explained in detail. After this, the interaction with Alexa and Lego Mindstorms is enlightened. Finally, instructions to make your own Alexa Color Game are given.
A robot in a quest for the right colorA schematic overview of the game is depicted below. The game consist of the following elements:
- a field consisting out of black lines connecting colored dots. The field can be of any shape or form. The colors that are used are restricted to the colors that can be identified by the EV3 color sensor.
- In the left bottom corner, the LEGO Mindstorms robot is visible (in iconic Alexa blue). The robot can move along the black lines to different colored dots. The main components of the robot are: two motors, color sensors, and EV3 brick.
- An Alexa device capable of communication with the EV3 brick.
The Color Game is played as follows:
- At the start of the game, Alexa tells the user to which color the robot wants to go.
- Using voice commands the user should bring the robot to the desired color.
- The commands: Forward, Left and Right can be used.
- The game is won when the robot reaches the desired color. At this moment Alexa will play upbeat music.
- The game is lost when the robot is going to move over the edge of the field. At this moment Alexa will play sad music.
An Important element of the field are the black lines connecting the colored dots. Using these lines it is possible to program the Mindstorms EV3 robot to move directly from one colored dot to the other. This allows to complete a color game with about 4-6 commands, which makes the game very easy to play. By increasing the size of the field the game automatically becomes harder.
Besides this game, many other games can be created using this project by simple extensions of the code. Some examples:
- Collect the most point: A game with the goal to score as many points by collecting colors. Especially fun when using a very big field and assigning different amount of points to each color.
- Collectevery color: A game where the goal is to collect all colors with as few commands as possible, or as fast as possible.
To keep this project to the point only the basic color game is enlightened. You are challenged to extend the given code to other games.
Use voice control to playThis game combines the power of Lego Mindstorms and Amazon Alexa:
- A LEGO Mindstorms robot can be commanded to perform tasks. By itself, user interaction with a LEGO Mindstorms robot can only be achieved through a remote or buttons. Feedback to the user can be giventhrough a small screen or small speaker.
- The voice commands of Alexa are used to interact with a physical gameBy itself Alexa is already capable of playing some games. However, these games do not use any physical platform. Hence, the user can not have any interaction with the physical world.
By combining these two powerful elements an intuitive game is created. Voice controlling a LEGO Mindstorms robot brings a whole new dimension to the experience. Moreover, Alexa makes it possible to play upbeat sounds when winning and sad sounds when doing something wrong, inherently leading to a very intuitive and playful interaction, as described below.
Interaction between: Alexa;the user; the Mindstorms EV3; and the environment
The interaction between all elements during a game is explained in detail below.
Start of the game:
- The user can start a game by telling Alexa:'Alexa, open color game'
- Alexa will respond to the user with: 'Hello, can you bring me to [color]'The color is chosen randomly, to keep the game unpredictable
- Alexa will send the chosen color to the Mindstorms EV3 such that it can detect if the desired color is reached
During the game:
- During the game the user can issue the following commands to Alexa:'Move Forward' 'Turn Left' 'Turn Right'(variations of these commands will also be recognized)
- If Alexa recognizes one of the commands the command is send to the Mindstorms EV3
- The Mindstorms EV3 will use the color sensors to detect the environment in order to perform commands:- the black lines can be followed by the Mindstorms EV3 to get to a new colored dot. - the color of the dots can be detected by the Mindstorms EV3- the edge of the field can be detected by the Mindstorms EV3
- When finishing a command the Mindstorms EV3 is on a colored dot and sends this color to Alexa
- Alexa will respond to the user with: 'I am on [detected color by EV3], what should I do next'
- If the user issues a command not recognized by Alexa, Alexa will politely inform the user with the sentence: 'can you repeat that, you can only use the commands Forward, Left and right'By saying this, the user will exactly know what to do.
End of the game:
- If the Mindstorms EV3 reaches the color chosen by Alexa:- Alexa will play upbeat music and say 'Thank you' to inform the user she is happy- The Mindstorms EV3 robot will do a little dance to inform the user he is happy
- If the Mindstorms EV3 detects from the environment that it is leaving the field:- The Mindstorms EV3 robot will stop immediately. - Alexa will play sad music and tell the user 'Game over, please put the robot back on a colored dot and restart'
The user can restart the game at any point by telling Alexa: 'Restart color game'.
Examples of conversations between Alexa and the user can be seen in the video above.
Create your own Alexa Color GameTo make your own Alexa Color Game you need to do the following
1) Build the field
2) Build the Mindstorms EV3 Robot
3) Program the Mindstorms EV3 Robot
4) Program the interaction with Alexa
1)Building the Field
The field can be made out of any material: printed on paper, painted on the floor, using tape on carton boxes, completely out of LEGO as in the video above, you name it...
It should consist of the following elements:
- Lines (approx 1.6 cm width)
- Colored dots (minimum 3 cm x 3 cm)
- Border of the field
The colors of each of the elements should identifiable by the Mindstorms EV3 color sensor.
An example of a field is given below and can simply be printed on paper. But you are encouraged to make your own unique field!
2)Buildingthe Mindstorms EV3 Robot
The robot designed for this project is build in the iconic Alexa blue color. But the basic tasks of this robot can be performed using a simple base robot. You can make your own base robot and decorate however you like.
All the main components of the base robot are in the LEGO Mindstorms EV3 sets (retail 31313 or education 45544). This makes it possible for everyone with one of these sets to make the Alexa Color game.
The base robot should be capable of the following:
- Turn around own its own axes
- Follow a line
- Detect colors
This can be achieved with the following LEGO elements:
- Mindstorms EV3 brick
- 2 Motors and 2 wheels
- Color sensor(s)
- Lego connecting everything together
The design of the robot used in the video has 2 color sensors. With simple code adjustments a robot with a single color sensor can also be used. Moreover, adding color sensors in the back would make it possible to move backwards.
The following figures are detailed photos of the base robot. From these photos it is possible to reproduce the design of this base robot. Or you can get inspiration on how to build a base robot using your own LEGO.
3)Programming the Mindstorsm EV3 robot
The complete program for the robot can be found in the attachment. The most important code snippets are explained below.
Note that, when choosing for a slightly different design of the robot and/or field some parameters should be slightly altered. For instance the speed and rotation time.
The following code snippet is used when the command forward is used. The actions performed are:
- Move of the current square (Drive until both color sensors do not see red yellow or green)
- Follow black line until next square is detected
- Move a little bit forward to get the center of robot onto the center of the colored dot
- When the action is completed a message is sent to Alexa with the current color
def Forward(self):
speed = 360/1.5 # deg/sec
dt = 100 # milliseconds
stop_action = "coast"
previous_color = False #For robustness a color must be observed two samples in a row
#First move outside square
while self.lcs.value() in Colors.GOAL and self.rcs.value() in Colors.GOAL:
self.lm.run_timed(time_sp=dt,speed_sp=speed,stop_action=stop_action)
self.rm.run_timed(time_sp=dt,speed_sp=speed,stop_action=stop_action)
#Follow line until next square
while not ((self.lcs.value() in Colors.GOAL or self.rcs.value() in Colors.GOAL) and previous_color) and not self.lcs.value() in Colors.OUTSIDE and not self.rcs.value() in Colors.OUTSIDE:
if self.lcs.value() in Colors.FIELD and self.rcs.value() in Colors.LINE:
self.lm.run_timed(time_sp=dt,speed_sp=speed*2,stop_action=stop_action)
self.rm.run_timed(time_sp=dt,speed_sp=speed*1,stop_action=stop_action)
elif self.rcs.value() in Colors.FIELD and self.lcs.value() in Colors.LINE:
self.lm.run_timed(time_sp=dt,speed_sp=speed*1,stop_action=stop_action)
self.rm.run_timed(time_sp=dt,speed_sp=speed*2,stop_action=stop_action)
else:
self.lm.run_timed(time_sp=dt,speed_sp=speed*1,stop_action=stop_action)
self.rm.run_timed(time_sp=dt,speed_sp=speed*1,stop_action=stop_action)
if self.lcs.value() in Colors.GOAL or self.rcs.value() in Colors.GOAL:
previous_color = True
else:
if previous_color == True: #False color observation
previous_color = False
sleep(dt / 1000)
#Move Onto Middle square
if self.lcs.value() in Colors.OUTSIDE or self.rcs.value() in Colors.OUTSIDE: # If outside field observed
self._send_event(EventName.ARRIVED, {'color': 6, 'previous_move': 0})
elif (self.lcs.value() in Colors.GOAL or self.rcs.value() in Colors.GOAL):
# Set color of square
color = 0
if self.lcs.value() in [3,4,5,7]:
color = self.lcs.value()
elif self.rcs.value() in [3,4,5,7]:
color = self.rcs.value()
# Move straight until sensors of square
while self.lcs.value() in Colors.GOAL or self.rcs.value() in Colors.GOAL:
self.lm.run_timed(time_sp=dt,speed_sp=speed,stop_action=stop_action)
self.rm.run_timed(time_sp=dt,speed_sp=speed,stop_action=stop_action)
sleep(dt / 1000)
# Move a tiny bit more forward to get center of the robot on the center of the square
stop_action="brake"
dt=300
self.lm.run_timed(time_sp=dt,speed_sp=speed,stop_action=stop_action)
self.rm.run_timed(time_sp=dt,speed_sp=speed,stop_action=stop_action)
sleep(dt/1000)
# Square Reached
print("Square Reached. Sending event to skill")
self._send_event(EventName.ARRIVED, {'color': color, 'previous_move': 0}) #Send Color Square
self.current_color=color
# If square color is Goal Color: do a dance
if color == self.goal_color:
self.Dance()
The following code snippet is used to turn left or right. The actions performed are:
- Turn in given direction (True = Right, False = Left)
- When the action is completed a message is sent to Alexa with the current color
def Turn(self,direction = True):
speed = 360/3
dt = 1500
stop_action= "brake"
if direction==True: # Turn Right
self.lm.run_timed(time_sp=dt,speed_sp=-speed,stop_action=stop_action)
self.rm.run_timed(time_sp=dt,speed_sp=speed,stop_action=stop_action)
else: # Turn Left
self.lm.run_timed(time_sp=dt,speed_sp=speed,stop_action=stop_action)
self.rm.run_timed(time_sp=dt,speed_sp=-speed,stop_action=stop_action)
sleep(dt/1000)
print("Turn Complete. Sending event to skill")
self._send_event(EventName.ARRIVED, {'color': self.current_color, 'previous_move': 1}) #Send message to indicate turn is complete
The following code snippet is used when the robot needs to dance. The actions performed are:
- Turn a few times left and right.
def Dance(self):
speed = 360/2
dt = 250
stop_action= "brake"
# Turn back and forth a few times
for i in range(1,5,1):
self.lm.run_timed(time_sp=dt,speed_sp=((-1)**i)*speed,stop_action=stop_action)
self.rm.run_timed(time_sp=dt,speed_sp=-((-1)**i)*speed,stop_action=stop_action)
sleep(dt/1000)
4)Programming the interaction with Alexa
The code that takes care of the communication between Alexa and the Mindstorms EV3 is based on the mission provided by the Voice Challenge. Hence, it is recommend to complete these missions before programming the Alexa Color Game.
A key learning moment in programming the interaction between the Mindstorms EV3 and Alexa is dealing with the following:
Actions performed by the EV3 take some time. Hence, there is some time between commands where the user is waiting for the robot. Unfortunately, Alexa is a bit impatient and closes the communication quite quick when the user does not issue new commands. To solve this, the Mindstorms EV3 sends a command to the Alexa after completing each task, this allows to issue new commands without communication time-outs with the user.
The full code can be found attached to this project. The most important code snippets are described below.
Code snippet to choose random goal color:
- Selecting a random number out of 3, 4 and 5, representing green yellow and red, respectively.
- Set speech output corresponding to goal color
- Make directive with goal color to send to Mindstorms EV3
// Define goal color (random selection out of 3 4 5)
const goal_color = Math.floor(Math.random() * 3)+3;
Util.putSessionAttribute(handlerInput, 'goal_color', goal_color);
let speechOutput = "There is no goal color defined"
if (goal_color === 3){
speechOutput = "Hello, can you bring me to green";
} else if (goal_color === 4){
speechOutput = "Hello, can you bring me to yellow";
} else if (goal_color === 5){
speechOutput = "Hello, can you bring me to red";
}
// Make directive that sends goal color to EV3
let directive = Util.build(endpointId, NAMESPACE, NAME_CONTROL,
{
type: 'goal_color',
goal_color: goal_color
});
return handlerInput.responseBuilder
.speak(speechOutput)
.addDirective(directive)
.addDirective(Util.buildStartEventHandler(token,60000, {}))
.withShouldEndSession(false)
.reprompt("What should I do next?")
.getResponse();
Code snippet: Event handler receiving messages from Mindstorms EV3
- On event received with name 'Arrived' Alexa has received a message from the Mindstorms EV3 indicating that it finished a command (Forward, Left, Right)
- The payload: - the command it finished- the color of the square at which the robot is currently
- If the current color is equal to the goal color Alexa will play upbeat music
- If the current color is equal to the color of the edge Alexa will play sad music
- If the color is one of the other colors Alexa will tell the user at which color the robot is standing and will ask what to do next.
let speechOutput;
if (name === 'Arrived') {
const attributesManager = handlerInput.attributesManager;
const goal_color = attributesManager.getSessionAttributes().goal_color || 1;
let color = parseInt(payload.color);
let previous_move = parseInt(payload.previous_move)
speechOutput = "Color not recognized" + previous_move;
if (color===5){
if (goal_color === 5 && previous_move === 0){
speechOutput = WIN_MUSIC + "Thank you, I reached red"
} else {
if (previous_move === 0){
speechOutput = "I am on red, what should I do next?"
} else {
speechOutput = "What should I do next?"
}
}
} else if (color === 4){
if (goal_color === 4 && previous_move === 0){
speechOutput = WIN_MUSIC + "Thank you, I reached yellow"
} else {
if (previous_move === 0){
speechOutput = "I am on yellow, what should I do next?"
} else {
speechOutput = "What should I do next?"
}
}
} else if (color === 3){
if (goal_color === 3 && previous_move === 0){
speechOutput = WIN_MUSIC + "Thank you, I reached green"
} else {
if (previous_move === 0){
speechOutput = "I am on green, what should I do next?"
} else {
speechOutput = "What should I do next?"
}
}
} else if (color === 6){
speechOutput = GAMEOVER_MUSIC + "Game over. Please put me back on a colored dot and restart"
}
return handlerInput.responseBuilder
.speak(speechOutput, "REPLACE_ALL")
.reprompt("What should I do next?")
.getResponse();
}
SummaryBy combining the power of Lego Mindstorms and Amazon Alexa an intuitive and interactive game is created. Voice controlling a LEGO Mindstorms robot brings a whole new dimension to the experience of commanding a robot. The physical robot brings a new level of interaction to playing games with Alexa. The design of the robot and the field allows for a simple and fast game experience. The robot that performs the game only needs 2 motors and a color sensor and can therefore be build out of the standard Lego Mindstorms EV3 sets.
Comments