Natural Language Feedback Lets You Correct Robots' Planning Failures with a Simple Sentence
Spoken corrections, like "stay away from the mustard," prove a great way to quickly fix errors in planning for robotic systems.
A team of researchers from the Massachusetts Institute of Technology (MIT), the Universities of Utah and Washington, and NVIDIA has put forward an approach to correct robots when they're doing something wrong — by simply telling them what the problem is.
"When humans design cost or goal specifications for robots, they often produce specifications that are ambiguous, under-specified, or beyond planners’ ability to solve," the team explains. "In these cases, corrections provide a valuable tool for human-in-the-loop robot control."
"Corrections might take the form of new goal specifications, new constraints (e.g. to avoid specific objects), or hints for planning algorithms (e.g. to visit specific waypoints)," the researchers continue. "Existing correction methods (e.g. using a joystick or direct manipulation of an end effector) require full teleoperation or real-time interaction."
A better approach: Just telling the robot what it's doing wrong, in what the researchers team "natural language feedback" and which can be as simple as telling a robot vacuum cleaner "don't go into the bathroom" or "go to the far end of the corridor where you missed a bit."
The team's work involves mapping natural language sentences, like "don't squeeze the banana," into transformations of cost functions — which, in turn, provide a way to correct the robot's goals where planning has failed. In testing, the approach certainly seems to work: In scenarios where the planner, STORM, was known to have failed, the robot's behavior could be corrected with an 81 per cent success rate with a single feedback sentence rising to 91 per cent with a second.
"Our method, the researchers claim, "makes it possible to compose multiple constraints and generalizes to unseen scenes, objects, and sentences in simulated environments and real-world environments."
The team's work was presented at Robotics Science and Systems XVIII, with more information available on the project website and a preprint published to Cornell's arXiv server.
