Watch and Learn

It is a wonder that anyone ever got anything done before the internet came to be. What kind of a DIY demigod just happens to know how to do any conceivable home repair or car maintenance job off the top of their head? Without firing up a web browser and watching a how-to video or three, most of us would not know where to even begin. But with just a little bit of instruction, we can generally get by pretty well, if not become a MacGyver.

That is one of the many ways in which we differ from robots. Today’s robot control algorithms typically require massive numbers of demonstrations to learn from before they become even somewhat competent. If we learned like that, we would have to watch every home repair video on YouTube before we could change a light bulb. Needless to say, that is horribly inefficient and better methods are needed before the dream of general-purpose domestic robots becomes a practical reality.

A new framework proposed by researchers at Cornell University may be just what we need to help robots learn in a more human-like way. Called RHyME (Retrieval for Hybrid Imitation under Mismatched Execution), their system enables robots to learn a new task by watching just a single how-to video.

Traditionally, training robots to perform everyday tasks has required paired demonstrations where both a human and robot perform the same task. This method is difficult to scale and fragile when the human and robot move differently. A robot might simply fail if the human in the video performs the action in a more fluid or complex manner than the robot can replicate.

RHyME addresses this challenge by rethinking how robots interpret human demonstrations. Instead of expecting a perfect match between a human’s and a robot’s movements, RHyME uses a sophisticated matching system rooted in artificial intelligence. The key idea is to focus not on exact visual similarity but on semantic similarity — the meaning and intent behind each part of the task.

To do this, RHyME relies on a concept called optimal transport, a mathematical technique that helps align sequences of actions in a way that captures the overall structure of a task. Rather than comparing each frame of a human video directly to a robot’s frame, the system looks at entire sequences and finds the most meaningful correspondences. It is a bit like comparing two different ways to make a sandwich. One person might start with the meat, another with the condiments, but the end goal is the same. RHyME finds those deeper connections.

Once the system has interpreted the human video, it retrieves and assembles short clips from its database of robot experiences that align with each segment of the task. These snippets act as training examples, effectively creating a custom playbook for the robot to follow, even if it has never seen that exact task before.

This imitation approach allows the robot to perform complex, multi-step tasks with only 30 minutes of robot-specific training data, which is a dramatic reduction compared to conventional methods. In both simulations and real-world tests, robots using RHyME achieved more than double the success rate on new tasks compared to previous approaches.

By enabling robots to learn the way we do, we are rapidly moving toward the development of more intelligent, capable, and versatile machines. As the technology matures, the idea of robots handling real-world tasks with just a small amount of guidance may finally become a reality.