ALBERT Lab

Background

Space. The Final Frontier. As NASA prepares to head back to the moon, the ISS has been speeding across the sky at 17, 000 mph for the last twenty years, performing untold amounts of scientific research that benefit us right here on earth. One of the key themes of this research has been studying the behavior of life in microgravity, both at the human level and the microscopic scale. As humanity dreams of settling on other planets, we must first rigorously understand how biological organisms operate in microgravity. Will some pathogens, relatively harmless on earth, become dangerous in microgravity? Can artificial organs be grown and manufactured in weightless environments? Just how exactly does space change the function of cells? What medical advances are possible to save lives here on earth?

Space experiments are expensive. A single astronaut's time could be worth $60, 000 to $80, 000 per hour. Many astronauts are not scientists themselves, but engineers and pilots. They receive extensive training and instructional procedures for experiments, but basic science is often not their native discipline.

In addition, future missions to Mars may find themselves too far from Earth for real-time interaction with mission control to help run or tend to experiments. The crew must be self-supporting and have the help they need to operate the experiments if this is the case.

Problem Statement

Due to the complexity of running and monitoring experiments in microgravity in Low Earth Orbit (LEO) and during long-duration interplanetary flight, astronauts need an intuitive method of interacting with experiments that abstracts away the need for niche knowledge of the specific scientific field and facilitates simple interaction with minimal training.

Solution Overview

To solve this solution, I have prototyped ALBERT, or the Alexa-controlled LEGO Biological ExpeRimenT. ALBERT demonstrates the effectiveness of a voice-controlled experiment interface that would allow astronauts to intuitively interact with systems in the absence of continuous communication with mission control and without the need for extensive training.

In its current form, ALBERT is capable of reacting to voice commands to create new bacterial growth experiments, selecting, swabbing, storing, and monitoring an agar plate. Astronauts need only insert a swab and say, "Alexa, use this swab." To check an experiment, the crew member can simply ask, "Alexa, check plate 1." Given that ALBERT is made from LEGO materials, it does experiment some variability in success, however, it was repeatedly able to create and store plates.

Importance of Intuitive Voice Control

ALBERT's intuitive voice control interface is critical to its primary goal -- simplifying the interface to complex experiments to enable virtually untrained crew members to quickly and adeptly create and monitor scientific research. Extensions of the current work will allow dynamic voice interactions with multiple experiments, integration of image processing and datalogging, and critical streamlining of the entire experiment process from launch to landing.

Why Alexa?

Alexa is the perfect voice-input method for ALBERT. By categorizing commands via intents, Alexa outperforms a simple voice recognition system by its ability to understand the purpose behind an utterance. Additionally, Alexa's ability to integrate with so many other tools and systems opens the door for its use as a unified natural language interface to complex systems such as orbiting laboratories or spacecraft with intuitive interactions.

Applications

Like many other technologies developed for space exploration, ALBERT's impact can extend far beyond the ISS or interplanetary vehicles. Potential other applications include:

• Automating laboratories on Earth (like this)
• Demonstrating the value of integrating laboratories with voice assistants like Alexa for a unified human-laboratory or crew-vehicle interface
• Stimulating the development of fully automated experiment monitoring tools that could leverage new deep learning and AI techniques

Design Problems

While building the ALBERT prototype, several key design problems were encountered. They are briefly summarized here along with the chosen solution.

Rigid Frame

A key concern at the start of the project was ensuring the careful positioning of the necessary sections (workstation, robotic arm, sterile plate storage, and live experiment storage). Misalignment among these components can negatively affect the reliability and repeatability of the procedures.

To mitigate this, all the key models were secured to an aluminum TETRIX frame.

Reliable Plate Positioning

LEGO gear trains often experience a large amount of play, making very precision positioning very difficult. However, the plates must be consistently placed on the workstation in a tight area to enable the arm to lift and replace the lids reliably. Additionally, the vertical storage racks have little room for horizontal position errors.

By placing axles as guides at the base of the workstation platter, I could ensure that the plates would be guided down into the platter even in the presence of horizontal manipulator error. To ensure the plates could be consistently vertically stored, angled guides were included in the rack design.

Swabbing the Plate

To create an experiment plate, the swab cannot simply touch the agar, but must be swiped over some amount of surface area.

Rather than require a second robotic arm to maneuver the swab over the surface of the agar, I elected to directly spin the dish to swab an arc across its surface. Simply by attaching a motor with rubber tire to the workstation allows simple rotating of the dish without the need for a complex assembly.

Controlling Two Subsystems

To limit complexity, ALBERT is divided into two primary subsystems, the robotic arm and the workstation. ALBERT's primary controller, the EV3 Intelligent Brick, must be able to communicate with the workstation to trigger swab and monitoring operations.

The ev3dev operating system enables the EV3 to remote control the NXT over a USB connection using the nxt-python library.

Two-Position Gripper/Wrist

To store plates vertically and place them horizontally in the workstation requires two arm positions, horizontal and vertical.

A wrist joint enables this rotation, but care must be taken to align the wrist rotation axis with the center of the end effector. Misalignment will cause the horizontal and vertical gripper orientations to be slightly offset from each other, causing issues when placing the plate or removing the lid.

Repeatable Base Positions

The arm must be able to repeatably position itself at the sterile rack, workstation, and storage rack. Originally, this was done with dead reckoning, but for improved performance the EV3 color sensor is used to detect colored stripes as the arm base rotates.

Lifting the Dish Lid

Lifting the dish lid is hard. The tolerance is only about +/- 0.25 cm in my estimate, which is tough to achieve with LEGO gearing. Although ALBERT was able to lift the lid by gripping the edge several times, it was not consistent. THe lid would often pivot in the gripper, preventing it from being replaced on the dish, or slide out of the gripper entirely.

The solution was to slightly modify the dish lid to add a "handle" that the gripper could easily hold on to. This handle was made of two bevel gears and was hot-glued to the dish lid.

Software Architecture

The software stack is divided into two major sections: the Alexa-hosted Node.js skill and the Python Alexa Gadget code on the EV3.

Alexa Skill

The Alexa skill code can be browsed in the skill directory. Several key intents are defined and implemented, following the pattern of the sample missions for the LEGO MINDSTORMS Alexa Challenge.

• MakePlateIntent - This intent sends the "make plate" directive to the EV3 gadget
• CheckPlateIntent - This intent sends the "check plate" directive to the EV3 gadget along with the number of the plate to check. In most cases, this will be one, but it one be extended to devices with more than one storage rack in the future.
• GadgetEventHandler - This handler catches the events returned back from the EV3 gadget to report when a plate has been made or the status of a previously made plate.

EV3 Python Gadget Code

The gadget code is designed to receive gadget commands from the cloud-based Alexa skill and activate the ALBERT. This includes a class to encapsulate ALBERT's capability in albert.py and a gadget class based off the sample code, written in albert_gadget.py. The gadget code closely mirrors the example missions, which were used as a starting point. The ALBERT control code makes extensive use of keypoints defined for the manipulator (arm) to concisely encode the sequence of motions required for each task. These tasks can then be triggered from the gadget code. The primary gadget event handler is straightforward, processing the Alexa directives and generating events based on the tasks.

def on_custom_mindstorms_gadget_control(self, directive):
    try:
        payload = json.loads(directive.payload.decode("utf-8"))
        print("Control payload: {}".format(payload), file=sys.stderr)
        control_type = payload["type"]
        if control_type == "make_plate":
            # make the plate
            self.albert.make_plate()
            self.plate_counter += 1
            self.send_custom_event('Custom.Mindstorms.Gadget', 'plate_finished', {
                'plate_number': self.plate_counter
            })
        elif control_type == "check_plate":
        # check the plate
        # will need plate number
        number = int(payload["plate_number"])
        if number > self.plate_counter:
            self.send_custom_event('Custom.Mindstorms.Gadget', 'plate_status', {
                'plate_number': -1
            })
        else:
            refl = self.albert.check_plate()
            self.send_custom_event('Custom.Mindstorms.Gadget', 'plate_status', {
                'plate_number': number,
                'reflectivity': round(refl, 2)
            })
    except KeyError:
        print("Missing expected parameters: {}".format(directive), file=sys.stderr)

Here is a snippet of the code in albert.py used to align the base using the color sensor. Notice the final rotation at the end to move the sensor from the edge of the colored tape to the center. This ensures that the base stops at the correct position when rotating from either direction.

def rotate_base(self, color):
    '''
    Rotate from one color indicator to another.
    Color order is:
    YELLOW <--> BLUE   <--> RED
    STORE  <--> STATION <--> STERILE
    '''
    current_color = self.color_sensor.color
    if current_color == color:
        return
    direction = 1
    if (current_color == STATION_COLOR and color == STERILE_COLOR) or current_color == STORE_COLOR:
        direction = -1
    self.arm_base.on(SPEEDS[0]*direction, block=False)
    while self.color_sensor.color != color:
        pass
    self.arm_base.stop()
    self.arm_base.on_for_rotations(SPEEDS[0], direction*STRIPE_BIAS)

Along with the functions to move the shoulder and wrist to key points, this allows us to write the procedure functions like:

def get_plate(self, from_storage=False, upside_down=False):
    '''
    Sequence to get a plate and place it in the workstation.
    Post-conditions
    Gripper: WIDE
    Arm:     DOWN
    Base:    STATION
    '''
    src = STORE_COLOR if from_storage else STERILE_COLOR
    self.move_to_keypoint(KP_UP_HORZ)
    self.rotate_base(src)
    self.set_gripper(GRIP_NARROW)
    self.move_to_keypoint(KP_DOWN_HORZ)
    self.set_gripper(GRIP_CLOSED)
    self.move_to_keypoint(KP_UP_HORZ)
    self.rotate_base(STATION_COLOR)
    dest_up = KP_UP_VERT_INVERT if upside_down else KP_UP_VERT
    dest_down = KP_DOWN_VERT_INVERT if upside_down else KP_DOWN_VERT
    self.move_to_keypoint(dest_up)
    self.move_to_keypoint(dest_down)
    self.set_gripper(GRIP_WIDE)
    self.move_to_keypoint(KP_DOWN_HORZ)

These can be further called to perform the complete task:

def make_plate(self):
    ''' Sequence to make a plate. '''
    self.get_plate()
    self.lift_lid()
    self.swab_plate()
    self.lower_lid()
    self.store_plate()

Some of the supporting functions support additional parameters that are used when checking a plate:

def check_plate(self):
    ''' Sequence to check plate. '''
    self.get_plate(from_storage=True, upside_down=True)
    self.move_to_keypoint(KP_UP_HORZ)
    refl = self.station.check_status()
    self.move_to_keypoint(KP_DOWN_HORZ)
    self.store_plate(is_upside_down=True)
    return refl

The workstation.py file leverages the nxt-python library to remote control the NXT over a USB connection. First, we can setup the workstation connection:

self.brick = nxt.locator.find_one_brick(method=nxt.locator.Method(bluetooth=False))
self.color = Color20(self.brick, PORT_4)
self.tilt_motor = Motor(self.brick, PORT_B)
self.rotate_motor = Motor(self.brick, PORT_C)

The swab and check plate operations are simple, moving the head into position and rotating the dish if necessary. The swab function looks like this:

def swab(self):
    self.tilt_motor.turn(TILT_MOTOR_POWER, TILT_DEGREES)
    self.rotate_motor.turn(ROTATE_MOTOR_POWER,PLATE_ROTATION_DEGREES)
    self.tilt_motor.turn(-TILT_MOTOR_POWER, TILT_DEGREES)

All the code is encapsulated in classes to facilitate maintenance.

Build Your Own

All the CAD models were created using Studio. Step-by-step build instructions are omitted for brevity and since your build of ALBERT will likely have a different form factor or experiment design.

1. Build the Arm

The arm CAD model is attached for details on building the arm.

2. Build the Workstation

Build the workstation using the attached workstation.io file.

3. Build the TETRIX Base and Plate Holders

Build two plate holders (mirror images) using plate_holder.io.

To create the platform with colored stripes, I used a section of foam-core board. Align the arm with each of the sterile rack, workstation, and storage rack, and place a strip of red, blue, and yellow colored tape under the sensor, respectively.

4. Setup the Software Environment

1. Download ev3dev from the download page.

2. Flash the image to an SD card using Etcher.

3. Connect the EV3 brick to your computer with a USB A-to-Mini-B cable and enable USB internet sharing.

4. On your computer, install Visual Studio Code from Microsoft.

5. Once installed, open the extensions panel (Ctrl+Shift+X). Then search for and install the ev3dev-browser extension.

6. On your computer, download the source code from the GitHub repo. If you are unfamiliar with git, just download the code as a zip file using the button at the top right of this page. Advanced users can clone the repository.

7. In order to communicate with the NXT workstation, nxt-python must be installed on the EV3 (not your computer). To do this, you will need to run a few commands in the EV3 brick terminal.

• To access this terminal, expand the "ev3dev Device Browser" group at the bottom of the Explorer panel (Ctrl+Shift+E) and connect to your brick by clicking "Click here to connect a device."
• Once connected, right click the ev3dev device and select "Open SSH Terminal."
• Run the following commands in the terminal (you can ignore the lines starting with a "#"; those are comments)

# install pip3 (python3 package manager)
sudo apt install python3-pip

# install pyusb for USB connection to NXT
sudo pip3 install pyusb

# install nxt-python (python3 beta)
wget https://github.com/ev3dev/nxt-python/archive/ev3dev-stretch.zip
unzip ev3dev-stretch
cd nxt-python-ev3dev-stretch
python3 setup.py install # you may need to prepend `sudo`

5. Setup the Alexa Gadget

(These steps are adapted from the LEGO MINDSTORMS Voice Challenge Mission 1 example)

1. Create an Amazon Developer account if you do not yet have one

2. Go to the Alexa Voice Service page and click the "PRODUCTS" button.

3. Click the "CREATE PRODUCT" button at the top right of the page.

4. Name the product "MINDSTORMS EV3" with ID of "EV3_01". The product type can be "Alexa Gadget" with a category of "Animatronic or Figure". Fill out a description, choose "No" for intending commercial distribution, and "No" for if it is a child product. Finishing clicking through the product creation.

5. Click on your new device in the list of devices in the console.

6. On the product's page, take note of the Amazon ID and Alexa Gadget Secret codes. You will need them later.

6. Code the Alexa Skill

(These steps are adapted from the LEGO MINDSTORMS Voice Challenge Mission 3 example.)

1. Go to the Alexa Developer Console and click the blue "Create Skill" button on the right hand side of the page.

2. Name the skill "ALBERT" and use the default selected "Custom" model. At the bottom of the page, choose "Alexa-Hosted (Node.js)". Finish clicking through the skill creation pages.

3. Enable the custom interface controller by selecting "Interfaces" in the left-hand menu, and turning on "Custom Interface Controller." Save the updates using the button at the top of the page.

4. Next, setup the interaction model by clicking the "JSON Editor" item in the left-hand menu (under the "Interaction Model" heading) and drag and drop the model.json file from the skill folder in the ALBERT source code. Save and build the model using the buttons at the top of the page.

5. Click the "Code" tab at the very top of the screen. Create the following files and copy and paste the contents from the corresponding ALBERT source files in the skill/lambda directory.

• common.js - Common intent handling, such as for help or cancel actions (from Mission 3)
• index.js - Contains the main intent and event handlers
• package.json - Package information
• util.js - Utility code (from Mission 3)

6. Click "Save" and then "Deploy" to activate your skill!

7. Create a albert_gadget.ini file in the gadget directory with the following contents:

[GadgetSettings]
amazonId = YOUR_GADGET_AMAZON_ID
alexaGadgetSecret = YOUR_GADGET_SECRET
[GadgetCapabilities]
Custom.Mindstorms.Gadget = 1.0

The ID and secret are the values from step 5.

7. Downloading and Running

Almost there! In the VS Code Explorer panel (Ctrl+Shift+E) find the ev3dev device at the bottom of the screen. Click the download button near the left edge of the ev3dev Device Browser header.

If you don't see this button, make sure you are connected to your EV3 brick

Once your files have been copied over, find the albert/gadget/albert_gadget.py file, right-click it, and select "Run."

If this is your first time running, you will need to pair the EV3 with your Alexa device over Bluetooth. You can do this through the Alexa app. Make sure that Bluetooth is turned on for the EV3.

If you run into issues with Bluetooth being unavailable on the EV3, see this Github issue. TL;DR: Run these lines on your EV3 terminal (same terminal from which you installed nxt-python)

sudo systemctl mask systemd-rfkill.service
sudo systemctl mask systemd-rfkill.socket

and then restart your EV3.

About Me

Hey! I'm Matthew, and I'm studying machine learning for my Ph.D. at the University of Kentucky and first worked with LEGO MINDSTORMS through the NXT over 10 years ago. Get to know me more on my blog, https://mruss.dev.

---

These instructions are current as of 2019-12-25.