DeXtreme Gives Low-Cost Robot Hands Human-Like Dexterity Using Rapid Simulation Training

Researchers at the University of Toronto, the University of Southern California, NVIDIA, and Snap have come up with a way to make robot limbs considerably more dextrous — by better transferring what the machine learns in simulation to the real world, using an approach dubbed DeXtreme.

"While it has been possible to teach robots with legs how to walk for some time, robots with hands have generally proven to be much more tricky to control," explain Gavriel State and Ankur Handa, co-authors of the DeXtreme paper, in a joint statement. "A hand with fingers has more joints that must move in specific coordinated ways to accomplish a given task. Traditional robotics control methods with precise grasps and motions are incapable of the kind of generalized fine motor control skills that humans take for granted."

A relatively low-cost approach to the problem is to train the robot control system in simulation, but previous efforts have found that what works well in the simulator doesn't necessarily transfer to a physical robot operating in the real world. That's where DeXtreme comes in, the researchers explain.

"DeXtreme policies are trained to adapt to a wide range of conditions in simulation. Consequently, our vision-based policies significantly outperform the best vision policies in the literature on the same reorientation task and are competitive with policies that are given privileged state information via motion capture systems," the team writes. "Our work reaffirms the possibilities of sim-to-real transfer for dexterous manipulation in diverse kinds of hardware and simulator setups."

To prove the concept, the team chose a relatively low-cost hand-like gripper system: the Allegro Hand, available at a cost some one-tenth that of its competition but featuring only four fingers to a human hand's five and no functional wrist.

Three off-the-shelf visible-light cameras are used to track the hand and the object to be manipulated from different angles, and its control system is driven using a model trained entirely in-simulation using NVIDIA'S Isaac Gym fed with synthetic data generated by the Omniverse Replicator.

"Training can happen on a single Omniverse OVX server," State and Handa explain. "Training a good policy takes about 32 hours on this system, equivalent to 42 years of a single robot’s experience in the real world. Not needing a separate CPU cluster for simulation means a 10–200x reduction in computing costs for training at current cloud rental rates. Because we can use Isaac Gym to train the model, training time and cost can be dramatically reduced."

More information on the project is available on the DeXtreme website, along with a link to a preprint of the paper on Cornell's arXiv server.