AgileX achieves two-arm collaborative tasks based on Aloha

Showcase: AgileX achieves two-arm collaborative tasks based on Mobile Aloha

Mobile Aloha is a whole-body remote operation data collection system developed by Zipeng Fu, Tony Z. Zhao, and Chelsea Finn from Stanford University. link.

Based on Mobile Aloha, AgileX developed Cobot Magic, which can achieve the complete code of Mobile Aloha, with higher configurations and lower costs, and is equipped with larger-load robotic arms and high-computing power industrial computers. For more details about Cobot Magic please check the AgileX website.

In previous projects, AgileX successfully implemented the entire process of data collection, data replay, model inference and reproduction for the single-arm gripping task based on Mobile Aloha. Now AgileX collects more and more complex data and achieves the entire reasoning process of the dual-arm long sequence multi-target gripping task.

Task description

The task can be described as:First, stretch out the right arm, pick up the black block from the table, and then place the black block on the horizontally placed box in the center of the table. Then place the right robotic arm back, and at the same time stretch out the left robotic arm, pick up the red block from the table, and place it on the central box on the desktop. Finally, put the left arm back in its original place.

Compared with the previous tasks, the difficulty of this task has been upgraded: from the original short sequence to a long sequence task. Upgrading from a single robotic arm mission to a dual robotic arm mission. The target of clamping also changes from single to multiple.

Data Collection

In this task, the Orbbec DaBai camera is used to collect at a frequency of 30HZ, and contains 50 sets of acquisition data, and each set of data is collected at a fixed step size. Camera data contains color images, depth images, and point cloud information. The data collection platform is equipped with 2 master arms and 2 follower/puppet arms.

Data Replay

The data replay script loads and reads the collected joint data and reproduces it as it is.

Inference

Aloha is implemented based on the ACT (Action Chunking with Transformers) algorithm model. See the figure below for the specific model:

Perform model training and inference. Pick up the black block correctly with your right arm and place it on the middle box of the table. The left arm also successfully picked up and placed the red block.

The black and red blocks were manually placed back from the central box on the table. The robotic arm still recognized that the object on the box had been removed and successfully completed the task again: the right and left arms respectively picked up the black and red blocks and placed them back on the central box on the table.

Generalization

Add interference to the original data set task actions to test the generalization ability of the model.After the left arm finished a whole task and went back to the initial place, the red black on the central box was then manually removed and placed back on the table. The right arm detected that the black block didn’t change its position, so no picking action was performed. The left arm found that the red bolck was removed, so picked it onto the central box.

Summary

In this case, AgileX successfully achieved the entire process of data collection, model training and inference, and generalization ability verification for a two-arm multi-target grasping task based on Mobile Aloha. AgileX will continue to collect more scenes and more complex tasks, so please stay tuned.

About AgileX

Established in 2016, AgileX Robotics is a leading manufacturer of mobile robot platforms and a provider of unmanned system solutions. The company specializes in independently developed multi-mode wheeled and tracked wire-controlled chassis technology and has obtained multiple international certifications. AgileX Robotics offers users self-developed innovative application solutions such as autonomous driving, mobile grasping, and navigation positioning, helping users in various industries achieve automation. Additionally, AgileX Robotics has introduced research and education software and hardware products related to machine learning, embodied intelligence, and visual algorithms. The company works closely with research and educational institutions to promote robotics technology teaching and innovation.