OmniFetch - OmniBot AMR and LeRobot AI

Overview

Large facilities often deploy expensive and burdensome AMR robots that are not able to grab and transport objects on the platform itself. Typically, a mobile robot has a nominal "docking variance" of up to 2-4 inches. This means that without the enhancement of AI, direct playback or "dead reckoning" of a robot arm's joint movements is not a feasible approach to mobile object manipulation. This project leverages cutting-edge, sleek, and affordable technologies to create a hands-free, remotely operated robot assistant capable of picking up and transporting critical or hazardous objects in unpredictable environments.

By combining GyroPalm gesture control with the LeRobot, powered by Seeed Studio's reComputer and NVIDIA, we’ve created a solution that tackles latency issues common in remote control systems while improving accessibility for those with disabilities as well as industrial users who need to handle hazardous materials.

Our project integrates a LeRobot SO-ARM-100 robotic arm mounted on an OmniBot V2 mobile base platform. The OmniBot platform allows the robot to navigate efficiently through typical indoor environments such as homes, labs, and industrial facilities. Our solution has the ability to dynamically pick up objects, even if displaced or rotated, making it an ideal solution for handling hazardous materials, assisting disabled users with medication retrieval, or serving as a tabletop butler in busy work environments.

This project represents a step forward in assistive robotics, particularly for scenarios where fine motor skills may not be reliable, and latency in teleoperation is a limiting factor.

Robot Base Platform

The foundation of our system is the OmniBot V2, a versatile mobile robot platform by GyroPalm to handle complex indoor navigation tasks. The robot is constructed with a sturdy 2020 aluminum extrusion frame and compact yet powerful chassis, making it highly modular and adaptable for mounting additional hardware.

Some key features of the OmniBot V2 platform include:

360° Lidar Navigation: Utilizing Slamtec’s 2D Lidar for robust mapping and autonomous navigation in environments with obstacles.

Compact Design: The platform’s 30” tall structure allows it to function at tabletop height, ideal for retrieving objects from shelves, tables, or countertops. The robot features a max payload capability of 250 lbs.

Maneuverability: With a footprint small enough to pass through standard 28” US doorways, the robot can access tight spaces with ease.

IR-Guided Docking Station: For automatic battery charging and improved power management.

We designed the robot to serve as a stable, elevated platform for the LeRobot SO-ARM-100, ensuring the arm operates efficiently at an optimal height that matches most table surfaces and benchtops.

Gesture Control with GyroPalm

Our project integrates GyroPalm to allow users hands-free control via custom gestures. This enables users to remotely summon the robot, direct it to specific locations, and initiate automated pickup routines — all with intuitive hand motions. The GyroPalm Developer kit (DK2) features GyroPalm Studio, a cloud-based IDE that allows developers to customize the on-screen interface and custom gestures.

Key benefits of GyroPalm integration include:

Intuitive Control: Users can easily train custom gestures, making the system adaptable to individual needs.

Latency Tolerance: Even with intermittent wireless connectivity, GyroPalm’s intelligent command queuing ensures the robot continues performing tasks without losing progress.

Accessibility: Users with limited mobility or dexterity can operate the robot effortlessly, expanding its potential for assistive roles. In past events, users have found it incredibly easy to gain the skills to drive a robot using GyroPalm in minutes.

The Problem with Traditional Technology

Typical remote-controlled robots face several challenges when deployed in real-world environments:

Latency in Teleoperation: Remote control over Wi-Fi or cellular networks often suffers from delays, making precise actions difficult. Should the connection drop, the remote robot typically is stranded or must engage recovery behavior. It will no longer be able to continue with desired teleoperation.

Human Dependency on Fine Motor Skills: Traditional systems rely heavily on precise control, which can be challenging for individuals with disabilities or during tasks requiring delicate handling. Teleoperation is typical a 1-to-1 input to output system. By augmenting certain maneuvers with AI policies, this can reduce users' cognitive workload when controlling a robot remotely.

Object Displacement: Static automation systems struggle to adjust when objects are shifted or rotated unexpectedly.

Limited Storage Capacity: A robot that lacks dynamic object handling may encounter problems when its tray is already filled or when objects are positioned awkwardly.

Advantages of Our Solution

Our solution effectively addresses these limitations with the following innovations:

Dynamic Object Pickup: Using advanced motion planning and data-driven inference, our robot can locate and manipulate objects — even if they’ve been displaced or rotated — improving reliability in real-world scenarios.

Compact & Efficient Design: The robot’s slim profile (28” width) allows it to pass through standard doorways, while its 30” height aligns with typical table and workbench heights for seamless object retrieval.

Multi-Role Functionality: In addition to object retrieval, the robot’s flat platform design allows it to double as a tabletop butler for serving drinks, medications, or tools.

Improved Accessibility: Users with disabilities can effortlessly summon and control the robot with simple GyroPalm gestures, bypassing the need for intricate joystick control.

Servo Motor Calibration with LeRobot SO-ARM-100

The LeRobot SO-ARM-100 offers impressive precision, but achieving optimal performance requires careful calibration and mounting of both USB webcams. We performed comprehensive servo motor calibration to ensure the arm could dynamically detect and grasp objects even if misaligned. Key calibration steps included:

Joint Calibration: Each servo was calibrated to achieve accurate angular positioning.

Payload Balancing: We adjusted torque limits to handle different payload weights, ensuring the robot can pick up both lightweight and moderately heavy objects.

Grip Precision: The gripper was optimized to securely grasp objects of varying shapes and sizes.

Leader-Follower Robot Teleoperation

To enable intuitive remote control, we implemented a leader-follower mirroring system between two LeRobot arms. The leader arm (operated manually) mirrors its movements to the follower arm on the OmniBot platform, allowing users to remotely demonstrate desired object manipulation techniques.

Key highlights of our teleop system:

Real-Time Mirroring: The follower arm replicates the leader arm’s movement with minimal latency.

Adaptive Learning: By recording joint positions and movement patterns, we improved the system’s ability to handle new object orientations.

Data Collection for ACT Policy (Accelerated Compliant Teleoperation)

We captured detailed motion data from the LeRobot SO-ARM-100 using:

Dual-Camera Setup: Recording two angles simultaneously allowed us to extract precise joint positions.

Pose Tracking: Each frame included detailed joint angle data to improve model accuracy.

This data was crucial for training our robot’s ACT model to perform dynamic object pickup reliably.

ACT Model Training

Our ACT model was trained using collected motion data to predict optimal arm trajectories. The model enables the robot to:

Predict Object Locations: Even if objects are displaced or misaligned.

Adapt to Unexpected Situations: The model’s reinforcement learning process ensures the robot refines its handling skills through iterative practice.

Improve Stability: The trained model corrects minor positioning errors to reduce the risk of objects falling during pickup or transport.

Automation Workflow

Inference is triggered seamlessly by either a:

Custom GyroPalm Gesture: Users can initiate a pre-trained behavior to pick up specific objects.

Voice Command: The system is designed to support voice activation as an alternative input.

Button Trigger: For direct control via a simple interface.

Ultimate Workflow: From Gesture to Delivery

The complete workflow unites all these features into a seamless user experience:

Gesture Activation: Using a GyroPalm gesture, the user signals the robot to start.

Navigation to Workbench: The OmniBot V2 autonomously navigates to the designated workstation.

Object Pickup: The LeRobot arm dynamically adjusts to grasp objects — even if displaced or misaligned — and securely places them in a designated bin or tray.

Return Delivery: The robot navigates back to the user or designated drop-off point.

Summon the Robot in Real-Time: Users can also control the robot’s navigation live using GyroPalm’s gesture driving feature.

This workflow ensures a powerful, reliable, and accessible solution for users in industrial, medical, or home environments.

Conclusion

Our project combines innovative gesture control, intelligent motion planning, and compact robot design to deliver a practical, real-world solution for critical object retrieval and delivery. Whether for assisting disabled individuals, transporting hazardous materials, or streamlining workplace efficiency, this system offers a versatile and powerful solution.

We hope our project inspires further advancements in accessible robotics, leveraging hands-free control and adaptive learning to create smarter, more responsive machines.