GRIPP3R Trash and Recycle Bot

Introduction:

As the world is becoming more and more aware of sustainability, recycling has been implemented across the world city by city. With help of robotics, we can even make that process even easier. In this article we will build a prototype that let robot take care of garbage for us.

Step 1: Building the GRIPP3R

GRIPP3R from MindStorm EV3 seems to be the most ideal robot to handle recycle sorting task, as we can build out the robot and use it to grip things and throw things. The robot takes about 2-3 hours to build using lego MindStorm Set, but really only first 68 pages of the instruction book is necessary, as everything else is really cosmetic. Full instruction can be seen at https://www.lego.com/cdn/cs/set/assets/blt3cf20feda579587c/31313_GRIPP3R_2016.pdf

Once Robot is fully built, we first need to test some basics to make sure it functions correctly. We can download MindStorms Home at https://www.lego.com/en-us/themes/mindstorms/downloads and play around with basic functions. Our first test is to see whether what we are trying to do is even possible, below is the little simple program that we wrote to grab the stuff, turn around and walk away.

When this is being run you should see the image below, the wiring is kinda needed for computer to run the program, we won't need the wiring on our final product. Now that this is done we can move onto next step.

We'd Gyro sensor that does not come with the kit (it's included in core education kit if you have that one). We can do it without but it's fairly inaccurate to time the 180 degree turn without it.

We can place at the center of the robot where it has the spot

Once this part is done we can move onto step 2.

Step 2: Getting dev environment ready

Alexa gadget has written entire setup guide on how to connect Alexa with the Mindstorm, you can easily visit https://www.hackster.io/alexagadgets for more information. We will only highlight the major part in this step, it is highly recommended you go through there to get the environment setup.

Setup GRIPP3R Side

In this step we setup the lego mindstorms environment with visual studio Code and EV3DEV enviroment. There are few things to understand specifically. The EV3DEV image can be downloaded from

https://www.ev3dev.org/downloads/

You can use Etcher to format the SD card, this can be downloaded via

https://www.techspot.com/downloads/6931-etcher.html

The usb port and sd card is located on the left of the brick if it's faced down, or left side of the robot if it's facing you. This took a little time to figure out.

We can now insert the sd card and bootup EV3, we can setup wifi network from the ev3dev menu.

This would allow us to connect Visual Studio Code directly to ev3 instead of using usb wires. Once connected remember the ip as we might need it later, make sure your computer is on the same network as the ev3 brick.

Setup on Mac/PC side

And We'd need Microsoft VS Code which can be downloaded at https://code.visualstudio.com/. You can open alexa-gadgets-mindstorms folder to install ev3dev-browser extension as described in Enviroment Setup Guide by AlexaGadget

Once you click show recommendation it should show up on the left side as below.

We can start a new workspace called gripp3r helper, and have a few empty files or just download my code form the source. Now we will do the GRIPP3R side, we can connect to wifi instead of doing USB programming. Upon clicking click here to connect with my device ev3dev should show up, if not you can enter the ip manually via "I Don't see my device"

Click the download button on the browser to test if it works. If it works everything should be loaded into the ev3 brick and we are all set for environment setup.

Since we are going to use bluetooth to control the unit, let's enable the bluetooth here as well.

Step 3: Getting Alexa Voice Service ready

This step closely follows the alexagadget's mission 1, I'd recommend going through that guide to get your brick ready to respond to Alexa's calling. We will summarize through.

We first need amazon developer account https://developer.amazon.com. Sign up for a developer account if you don't already have one. And we need to go create a product in Alexa Voice Service.

Just go keep things simple, let's create a product using same as Mission 1 from alexagadget.

Name: MINDSTORMS EV3

• Name: MINDSTORMS EV3

Product ID: EV3_01

• Product ID: EV3_01

Product type: Alexa Gadget

• Product type: Alexa Gadget

Product category: Animatronic or Figure

• Product category: Animatronic or Figure

Product description: Whatever you like

• Product description: Whatever you like

Skip upload image

• Skip upload image

Do you intend to distribute this product commercially?: No

• Do you intend to distribute this product commercially?: No

Is this a children’s product or is it otherwise directed to children younger than 13 years old?: No

• Is this a children’s product or is it otherwise directed to children younger than 13 years old?: No

Once product is created we will have the app id and app secret.

We will use the alexa mission-01 as a test bed, an to do so we just change the id and secret with your own, please make sure there are no quotes needed.

[GadgetSettings]
amazonId = YOUR_GADGET_AMAZON_ID
alexaGadgetSecret = YOUR_GADGET_SECRET

[GadgetCapabilities]
Alexa.Gadget.StateListener = 1.0 - wakeword

[GadgetSettings]
amazonId = YOUR_GADGET_AMAZON_ID
alexaGadgetSecret = YOUR_GADGET_SECRET

[GadgetCapabilities]
Alexa.Gadget.StateListener = 1.0 - wakeword

We can upload the code like the previous step, full explaination of the code is located on mission 01 itself.

To execute the code, we will do something a bit more different than alexa gadget, it takes a little time to get these code executed and sometimes we are waiting hanging without knowing the reason. We can do it manually via shell terminal itself.

After that we can execute our files just like any other linux.

At this point we can activate alexa wake word with the GRIPP3R itself.

Step 4: Building Alexa app

This step closely summarize the alexagadget's mission 3 and and mission 4, as we need can build out alexa app for basic functions.

We need to create a new skill under alexa console https://developer.amazon.com/alexa/console/ask, for consistency sake let's name it mindstorms just like the missions.

We will use custom and Alexa hosted backend through node js, this is just to keep consistency if you are following the missions, and to make it easier for writing different programs

We can upload JSON file from mission 4 and build it from there. just do not fire any cannon as the GRIPP3R motor isn't suppose to handle that. Upon getting the json copied and pasted click build model to continue.

This gives us enough to do the following, we will get to these later.

Intent names (MoveIntent, SetSpeedIntent, SetCommandIntent)

• Intent names (MoveIntent, SetSpeedIntent, SetCommandIntent)

Slot name (Direction, Duration, Speed, Command)

• Slot name (Direction, Duration, Speed, Command)

Slot values ("forward", "backward", "1", "2", "100")

• Slot values ("forward", "backward", "1", "2", "100")

Upon these are built, we will upload all of mission 4's file into lambda in alexa Code screen. Detail code explanation can be seen from here.

After that we can go to test mode to turn the Skill testing is enabled for "Development"

As for code, you can either upload the code base, or following code from mission 4 for testing right now.

#!/usr/bin/env python3
# Copyright 2019 Amazon.com, Inc. or its affiliates.  All Rights Reserved.
#
# You may not use this file except in compliance with the terms and conditions
# set forth in the accompanying LICENSE.TXT file.
#
# THESE MATERIALS ARE PROVIDED ON AN "AS IS" BASIS. AMAZON SPECIFICALLY DISCLAIMS, WITH
# RESPECT TO THESE MATERIALS, ALL WARRANTIES, EXPRESS, IMPLIED, OR STATUTORY, INCLUDING
# THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NON-INFRINGEMENT.
import os
import sys
import time
import logging
import json
import random
import threading
from enum import Enum
from agt import AlexaGadget
from ev3dev2.led import Leds
from ev3dev2.sound import Sound
from ev3dev2.motor import OUTPUT_A, OUTPUT_B, OUTPUT_C, MoveTank, SpeedPercent, MediumMotor
from ev3dev2.sensor.lego import InfraredSensor
# Set the logging level to INFO to see messages from AlexaGadget
logging.basicConfig(level=logging.INFO, stream=sys.stdout, format='%(message)s')
logging.getLogger().addHandler(logging.StreamHandler(sys.stderr))
logger = logging.getLogger(__name__)
class Direction(Enum):
"""
The list of directional commands and their variations.
These variations correspond to the skill slot values.
"""
FORWARD = ['forward', 'forwards', 'go forward']
BACKWARD = ['back', 'backward', 'backwards', 'go backward']
LEFT = ['left', 'go left']
RIGHT = ['right', 'go right']
STOP = ['stop', 'brake', 'halt']
class Command(Enum):
"""
The list of preset commands and their invocation variation.
These variations correspond to the skill slot values.
"""
MOVE_CIRCLE = ['circle', 'move around']
MOVE_SQUARE = ['square']
SENTRY = ['guard', 'guard mode', 'sentry', 'sentry mode']
PATROL = ['patrol', 'patrol mode']
FIRE_ONE = ['cannon', '1 shot', 'one shot']
FIRE_ALL = ['all shots', 'all shot']
class EventName(Enum):
"""
The list of custom event name sent from this gadget
"""
SENTRY = "Sentry"
PROXIMITY = "Proximity"
SPEECH = "Speech"
class MindstormsGadget(AlexaGadget):
"""
A Mindstorms gadget that can perform bi-directional interaction with an Alexa skill.
"""
def __init__(self):
"""
Performs Alexa Gadget initialization routines and ev3dev resource allocation.
"""
super().__init__()
# Robot state
self.sentry_mode = False
self.patrol_mode = False
# Connect two large motors on output ports B and C
self.drive = MoveTank(OUTPUT_B, OUTPUT_C)
self.weapon = MediumMotor(OUTPUT_A)
self.sound = Sound()
self.leds = Leds()
self.ir = InfraredSensor()
# Start threads
threading.Thread(target=self._patrol_thread, daemon=True).start()
threading.Thread(target=self._proximity_thread, daemon=True).start()
def on_connected(self, device_addr):
"""
Gadget connected to the paired Echo device.
:param device_addr: the address of the device we connected to
"""
self.leds.set_color("LEFT", "GREEN")
self.leds.set_color("RIGHT", "GREEN")
logger.info("{} connected to Echo device".format(self.friendly_name))
def on_disconnected(self, device_addr):
"""
Gadget disconnected from the paired Echo device.
:param device_addr: the address of the device we disconnected from
"""
self.leds.set_color("LEFT", "BLACK")
self.leds.set_color("RIGHT", "BLACK")
logger.info("{} disconnected from Echo device".format(self.friendly_name))
def on_custom_mindstorms_gadget_control(self, directive):
"""
Handles the Custom.Mindstorms.Gadget control directive.
:param directive: the custom directive with the matching namespace and name
"""
try:
payload = json.loads(directive.payload.decode("utf-8"))
print("Control payload: {}".format(payload), file=sys.stderr)
control_type = payload["type"]
if control_type == "move":
# Expected params: [direction, duration, speed]
self._move(payload["direction"], int(payload["duration"]), int(payload["speed"]))
if control_type == "command":
# Expected params: [command]
self._activate(payload["command"])
except KeyError:
print("Missing expected parameters: {}".format(directive), file=sys.stderr)
def _move(self, direction, duration: int, speed: int, is_blocking=False):
"""
Handles move commands from the directive.
Right and left movement can under or over turn depending on the surface type.
:param direction: the move direction
:param duration: the duration in seconds
:param speed: the speed percentage as an integer
:param is_blocking: if set, motor run until duration expired before accepting another command
"""
print("Move command: ({}, {}, {}, {})".format(direction, speed, duration, is_blocking), file=sys.stderr)
if direction in Direction.FORWARD.value:
self.drive.on_for_seconds(SpeedPercent(speed), SpeedPercent(speed), duration, block=is_blocking)
if direction in Direction.BACKWARD.value:
self.drive.on_for_seconds(SpeedPercent(-speed), SpeedPercent(-speed), duration, block=is_blocking)
if direction in (Direction.RIGHT.value + Direction.LEFT.value):
self._turn(direction, speed)
self.drive.on_for_seconds(SpeedPercent(speed), SpeedPercent(speed), duration, block=is_blocking)
if direction in Direction.STOP.value:
self.drive.off()
self.patrol_mode = False
def _activate(self, command, speed=50):
"""
Handles preset commands.
:param command: the preset command
:param speed: the speed if applicable
"""
print("Activate command: ({}, {})".format(command, speed), file=sys.stderr)
if command in Command.MOVE_CIRCLE.value:
self.drive.on_for_seconds(SpeedPercent(int(speed)), SpeedPercent(5), 12)
if command in Command.MOVE_SQUARE.value:
for i in range(4):
self._move("right", 2, speed, is_blocking=True)
if command in Command.PATROL.value:
# Set patrol mode to resume patrol thread processing
self.patrol_mode = True
if command in Command.SENTRY.value:
self.sentry_mode = True
self._send_event(EventName.SPEECH, {'speechOut': "Sentry mode activated"})
# Perform Shuffle posture
self.drive.on_for_seconds(SpeedPercent(80), SpeedPercent(-80), 0.2)
time.sleep(0.3)
self.drive.on_for_seconds(SpeedPercent(-40), SpeedPercent(40), 0.2)
self.leds.set_color("LEFT", "YELLOW", 1)
self.leds.set_color("RIGHT", "YELLOW", 1)
if command in Command.FIRE_ONE.value:
print("Fire one", file=sys.stderr)
self.weapon.on_for_rotations(SpeedPercent(100), 3)
self._send_event(EventName.SENTRY, {'fire': 1})
self.sentry_mode = False
print("Sent sentry event - 1 shot, alarm reset", file=sys.stderr)
self.leds.set_color("LEFT", "GREEN", 1)
self.leds.set_color("RIGHT", "GREEN", 1)
if command in Command.FIRE_ALL.value:
print("Fire all", file=sys.stderr)
self.weapon.on_for_rotations(SpeedPercent(100), 10)
self._send_event(EventName.SENTRY, {'fire': 3})
self.sentry_mode = False
print("sent sentry event - 3 shots, alarm reset", file=sys.stderr)
self.leds.set_color("LEFT", "GREEN", 1)
self.leds.set_color("RIGHT", "GREEN", 1)
def _turn(self, direction, speed):
"""
Turns based on the specified direction and speed.
Calibrated for hard smooth surface.
:param direction: the turn direction
:param speed: the turn speed
"""
if direction in Direction.LEFT.value:
self.drive.on_for_seconds(SpeedPercent(0), SpeedPercent(speed), 2)
if direction in Direction.RIGHT.value:
self.drive.on_for_seconds(SpeedPercent(speed), SpeedPercent(0), 2)
def _send_event(self, name: EventName, payload):
"""
Sends a custom event to trigger a sentry action.
:param name: the name of the custom event
:param payload: the sentry JSON payload
"""
self.send_custom_event('Custom.Mindstorms.Gadget', name.value, payload)
def _proximity_thread(self):
"""
Monitors the distance between the robot and an obstacle when sentry mode is activated.
If the minimum distance is breached, send a custom event to trigger action on
the Alexa skill.
"""
count = 0
while True:
while self.sentry_mode:
distance = self.ir.proximity
print("Proximity: {}".format(distance), file=sys.stderr)
count = count + 1 if distance < 10 else 0
if count > 3:
print("Proximity breached. Sending event to skill", file=sys.stderr)
self.leds.set_color("LEFT", "RED", 1)
self.leds.set_color("RIGHT", "RED", 1)
self._send_event(EventName.PROXIMITY, {'distance': distance})
self.sentry_mode = False
time.sleep(0.2)
time.sleep(1)
def _patrol_thread(self):
"""
Performs random movement when patrol mode is activated.
"""
while True:
while self.patrol_mode:
print("Patrol mode activated randomly picks a path", file=sys.stderr)
direction = random.choice(list(Direction))
duration = random.randint(1, 5)
speed = random.randint(1, 4) * 25
while direction == Direction.STOP:
direction = random.choice(list(Direction))
# direction: all except stop, duration: 1-5s, speed: 25, 50, 75, 100
self._move(direction.value[0], duration, speed)
time.sleep(duration)
time.sleep(1)
if __name__ == '__main__':
gadget = MindstormsGadget()
# Set LCD font and turn off blinking LEDs
os.system('setfont Lat7-Terminus12x6')
gadget.leds.set_color("LEFT", "BLACK")
gadget.leds.set_color("RIGHT", "BLACK")
# Startup sequence
gadget.sound.play_song((('C4', 'e'), ('D4', 'e'), ('E5', 'q')))
gadget.leds.set_color("LEFT", "GREEN")
gadget.leds.set_color("RIGHT", "GREEN")
# Gadget main entry point
gadget.main()
# Shutdown sequence
gadget.sound.play_song((('E5', 'e'), ('C4', 'e')))
gadget.leds.set_color("LEFT", "BLACK")
gadget.leds.set_color("RIGHT", "BLACK")

And the ini file

[GadgetSettings]
amazonId = amazonid
alexaGadgetSecret = secret
[GadgetCapabilities]
Custom.Mindstorms.Gadget = 1.0

[GadgetSettings]
amazonId = amazonid
alexaGadgetSecret = secret
[GadgetCapabilities]
Custom.Mindstorms.Gadget = 1.0

This should give us the ability to control the robot. We will run through the same shell as below. When done, you should be able to control the robot.

Step 5: Setup Gripper Bot Alexa side, TrashIntent, RecycleIntent

Now it's time to write the robot logic. In order to complete this project we need robot to do 2 things, bringing the item to garbage/trash, and bringing the item to recycle.

In this step we will be changing entire project, so the previous missions are not very useful now. We can use the json file below. One of the important change is "invocationName": "gripper bot" so we can call up gripper bot to help us in Alexa.

Model is now changed as following

{
"interactionModel": {
"languageModel": {
"invocationName": "gripper bot",
"intents": [
{
"name": "AMAZON.CancelIntent",
"samples": []
},
{
"name": "AMAZON.HelpIntent",
"samples": []
},
{
"name": "AMAZON.StopIntent",
"samples": []
},
{
"name": "RecycleIntent",
"slots": [],
"samples": [
"Recycle this",
"Take this to recycle"
]
},
{
"name": "TrashIntent",
"slots": [],
"samples": [
"trash this",
"take it to trash",
"garbage",
"take this to garbge",
"take this to the rubbish bin"
]
},
{
"name": "AMAZON.NavigateHomeIntent",
"samples": []
}
],
"types": []
}
}
}

As for index.js in lamdba, we are adding additional 3 Intent based on mission-4, in this case we will be passing either "trash" or "recycle" command back to the robot.

Full code can be seen on the code side in github.

Step 6: Writing the GRIPP3R Helper Robot Logic

We'd Gyro sensor that does not come with the kit (it's included in core education kit if you have that one). We can do it without but it's fairly inaccurate to time the 180 degree turn without it.

from ev3dev2.sensor.lego import GyroSensor
self.gyro = GyroSensor()

There are 3 major code reviews here, it's all very straight forward. First is turning, we use another thread handling turning, by setting it's flag

self.isTurning = False
self.isRecycleTurning = False

After the flag is set we can go to _proximity_thread to calculate it's angle.

def _proximity_thread(self):
"""
Monitors the distance between the robot and an obstacle when sentry mode is activated.
If the minimum distance is breached, send a custom event to trigger action on
the Alexa skill.
"""
while True:
#distance = self.ir.proximity
angle = self.gyro.angle
#print("Proximity: {}".format(distance), file=sys.stderr)
if self.isTurning == True:
print("angle: {}".format(angle), file=sys.stderr)
self.leds.set_color("LEFT", "YELLOW", 1)
self.leds.set_color("RIGHT", "YELLOW", 1)
if abs(angle) >= 180 or abs(angle) <= -180:
self.isTurning = False
self._move(Direction.STOP.value[0], 0, 0,)
self.gyro.reset()
self.gyro.mode = 'GYRO-RATE'
self.gyro.mode = 'GYRO-ANG'
self.leds.set_color("LEFT", "GREEN", 1)
self.leds.set_color("RIGHT", "GREEN", 1)
elif self.isRecycleTurning == True:
if abs(angle) >= 90 or abs(angle) <= -90:
self.isTurning = False
self._move(Direction.STOP.value[0], 0, 0,)
self.gyro.reset()
self.gyro.mode = 'GYRO-RATE'
self.gyro.mode = 'GYRO-ANG'
self.leds.set_color("LEFT", "GREEN", 1)
self.leds.set_color("RIGHT", "GREEN", 1)
time.sleep(0.1)

In the meanwhile in the mean thread we can use

while not self.isRecycleTurning:
time.sleep(0.1)

to wait until turning is done. In the meanwhile we also did make a quick spin first then calibrate rest of the angle slowly. This is how we make turns.

Second part is TrashIntent, as we receive trash command the robot is to grab the item, turn 180 degrees, walk to the trash bin and drop the item.

def _trash(self, duration=10, speed=50):
#grip the stuff
self.grip.on_for_rotations(SpeedPercent(100), 1.2)
self.leds.set_color("LEFT", "GREEN", 1)
self.leds.set_color("RIGHT", "GREEN", 1)
#first turn 180 degrees
self.leds.set_color("LEFT", "GREEN", 1)
self.leds.set_color("RIGHT", "GREEN", 1)
self.isRecycleTurning = True
self.gyro.reset()
self.gyro.mode = 'GYRO-RATE'
self.gyro.mode = 'GYRO-ANG'
self.isTurning = True
self.drive.on_for_seconds(SpeedPercent(100), SpeedPercent(-100), 1.2)
self.drive.on_for_seconds(SpeedPercent(3), SpeedPercent(-3), 40)
while not self.isRecycleTurning:
time.sleep(0.1)
#drive to the recycle bin
self.drive.on_for_seconds(SpeedPercent(44), SpeedPercent(44), 2.4)
#drop the item
while not self.touch.is_pressed:
self.grip.on_for_degrees(SpeedPercent(10), -90)
self.drive.on_for_seconds(SpeedPercent(-44), SpeedPercent(-44), 2.4)
#now turn 180 degrees so it can walk back
self.leds.set_color("LEFT", "GREEN", 1)
self.leds.set_color("RIGHT", "GREEN", 1)
self.isRecycleTurning = True
self.gyro.reset()
self.gyro.mode = 'GYRO-RATE'
self.gyro.mode = 'GYRO-ANG'
self.isTurning = True
self.drive.on_for_seconds(SpeedPercent(100), SpeedPercent(-100), 1.2)
self.drive.on_for_seconds(SpeedPercent(3), SpeedPercent(-3), 40)

In recycle command, we turn the robot 90 degrees to walk to trash bin and drop it, then walk back and turn 90 degree reverse to get back to original position.

def _recycle(self, duration=10, speed=50):
#grab the item
self.grip.on_for_rotations(SpeedPercent(100), 1.2)
self.leds.set_color("LEFT", "GREEN", 1)
self.leds.set_color("RIGHT", "GREEN", 1)
#first turn 90 degrees
self.leds.set_color("LEFT", "GREEN", 1)
self.leds.set_color("RIGHT", "GREEN", 1)
self.isRecycleTurning = True
self.gyro.reset()
self.gyro.mode = 'GYRO-RATE'
self.gyro.mode = 'GYRO-ANG'
self.isRecycleTurning = True
self.drive.on_for_seconds(SpeedPercent(-100), SpeedPercent(100), 0.45)
self.drive.on_for_seconds(SpeedPercent(-3), SpeedPercent(3), 40)
while not self.isRecycleTurning:
time.sleep(0.1)
#drive to the recycle bin
self.drive.on_for_seconds(SpeedPercent(50), SpeedPercent(50), 3)
#drop the item
while not self.touch.is_pressed:
self.grip.on_for_degrees(SpeedPercent(10), -90)
self.drive.on_for_seconds(SpeedPercent(-50), SpeedPercent(-50), 3)
#now turn 90 degrees so it can walk back
self.leds.set_color("LEFT", "GREEN", 1)
self.leds.set_color("RIGHT", "GREEN", 1)
self.isRecycleTurning = True
self.gyro.reset()
self.gyro.mode = 'GYRO-RATE'
self.gyro.mode = 'GYRO-ANG'
self.isRecycleTurning = True
self.drive.on_for_seconds(SpeedPercent(100), SpeedPercent(-100), 0.45)
self.drive.on_for_seconds(SpeedPercent(3), SpeedPercent(-3), 40)

Step 7: Demo!

Now after all the hard work, we can see the demo in action! We have a working prototype that allows robot to throw trash and recycle for us!