Published May 21, 2021 © GPL3+

Ava v2 - NLP, 3D Depth Vision, Visual SLAM, 12 Neural Nets

This bot combines 3D depth vision, SLAM, NLP, Kinematics,12 machine learning models, 12 sonars, and 22 actuators into a single social robot.

ExpertWork in progressOver 83 days1,394

Ava v2 - NLP, 3D Depth Vision, Visual SLAM, 12 Neural Nets

Things used in this project

Hardware components

8GB Mele Windows Stick PC

Orbbec Astra S Depth Camera

Intel T265 Tracking Camera

Seeed Studio Grove - Infrared Temperature Sensor Array (AMG8833)

Arduino Mega 2560

SSC-32u Servo Controller

Sabertooth 2x5 Motor Controller

ServoCity Gearboxes and Servos

Actuonix Linear Actuators

12V Planetary Gear Motors

Devantech Sonars

Software apps and online services

Spacy

Used for basic NLP features like chunking, pos tagging, named entity recognition, etc.

OpenCV – Open Source Computer Vision Library OpenCV

For basic machine vision tasks

TensorFlow

For processing off-the-shelf pre-trained machine learning models (10)

Python

The bot runs python locally and on a server using flask.

Microsoft - Visual Studio 2019

The bot runs a lot of C# code on the LattePanda.

Android Studio

I use android for a voice remote.

YOLO Darknet

This model is used to recognize 80 different types of objects.

NLP Transformer Machine Learning Models

These models (5) are used for question answering, sentiment detection, masking, etc.

Microsoft DialoGPT Model

This model is used for conversational responses.

WolframAlpha API

This API is used for encyclopedia type knowledge in structured JSON format.

Wikipedia API

This API is used for encyclopedia type knowledge in text format.

ConceptNet API

Hand tools and fabrication machines

3D Printer (generic)

Story

Videos

This one is Ava playing a game of "touch the nearest object" to demonstrate her using kinematics (and depth camera) to figure out how to use her core and arms to touch various objects in various hard to reach positions.

This is Ava playing some simple card games to demonstrate object recognition, kinematics for pointing at cards, and general turn taking and other contextual game playing skills.

Waypoint navigation to perform a room inspection.

Some robot yoga and miscellaneous tasks...

First video with this bot...

Major Sensors

Orbbec Astra S Image and Depth Cam

Intel T265 Tracking Cam (Pose, SLAM, IMU, 2 Fisheyes)

2 Thermal Cameras

Compass

12 Sonars

Microphone

Machine Learning Models

For vision, this bot uses YOLO, AlexNet, Age/Gender, and Emotion Detection Models For verbal, this bot uses a variety of transformer models for questions answering, sentiment, masking, speech to text, etc.

Images

Usain Bolt Pose

This was taken after I installed the Devantech Sonars.

You can see the brake lights here. They also give turn signals.

The fairings are off, this was one my first reaches for the ground.

The endo-skeleton. Most of the metal parts are from ServoCity, but I found custom steel 12mm rods (for the ab slider)online that are hollow and lighter.

Here you can see the Latte Panda and USB hub. The movidius stick is in blue. There is a plastic fairing that covers all this.

You can see the metal endoskeleton here. Later I replaced the hip servos with actuators.

I originally used MTS sprockets made by Lynxmotion but then replaced them with custom 3D printed ones here.

The entire torso can raise up 4 inches. By altering the elevation on the left vs. right, the bot can lean side to side.

This pic was after I replaced the gearbox servos that were in the hips...with linear actuators instead.

The legs can move independently at each hip. By moving both in unison, the bot can lean forward or back.

The ears are made to carry 4 sensors apiece...a thermal cam, sonar, mic, and sharp distance sensor.

Depth cam on top, tracking cam with the 2 fisheyes is below it. The tracking cam uses Visual SLAM to predict the pose.