From Real to Sim and Back Again

There are hundreds of books and simple, guaranteed-to-work seven-step plans offering to teach you just about any skill that you want to learn. But very frequently what we find is that the most effective way to learn a new skill is through plain old practice. There is a lot of wisdom in the old saying: if at first you don't succeed, try, try again. And as engineers have found in recent years, the same basic methods also apply to robots.

While robots naturally excel at fixed, repetitive tasks like those found in, for example, an assembly line, they struggle mightily when they find themselves in an unstructured and dynamic environment like what is found in the typical home. That is the main reason why the few practical domestic robots we have today are limited to relatively simple tasks, like vacuuming the floor, to this day.

Learning by example offers the promise of teaching robots to carry out much more complex tasks in our homes. However, existing approaches have significant flaws that are keeping us far away from the goal of building a general-purpose domestic robot. One option is called imitation learning and involves training algorithms to perform a task using data collected as human experts demonstrate it. The problem is that this approach requires an impractically large number of demonstrations to be robust, and it does not learn anything it was not explicitly shown, like how to recover from an unexpected event.

Reinforcement learning attempts to build a more robust system by allowing the robot to self-collect data through trial and error, improving along the way. This approach has seen many successes, but the number of trials required to learn complex tasks can simply be unreasonably large for a physical robot to carry out. A team led by researchers at MIT had the idea to combine aspects of both of these approaches to build more robust robot control systems that do not have impractical training requirements.

Called RialTo, the system takes a unique real-to-sim-to-real approach. To teach it a new skill, a user first uses their smartphone camera to capture images of the environment in which the robot will be working. These images are then converted into a three-dimensional simulated environment. A relatively small number of real-world expert demonstrations are then supplied and also transferred to the virtual environment. At that point, reinforcement learning can be carried out entirely in the simulation, where the robot can move millions of times faster than in the real world, and does not need to be concerned about creating unsafe situations. Finally, a teacher-student distillation process is utilized to transfer the model from the virtual world to the real world of sensors and physical robots.

The team put RialTo to the test in eight different manipulation tasks that included placing dishes in a drying rack and stacking books. When compared with other existing methods, the new technique was found to improve the average task success rate by 67 percent. Moreover, RialTo was also shown to be very robust to unexpected situations. For example, when using a robot arm to pick up a cup, it was able to recover and still complete the task even if the team repeatedly moved the cup to intentionally make it difficult.

Many additional demonstrations, and also source code, can be found on the project’s website.