By Your Command

The development of robots that can understand and follow spoken instructions has a lot of potential in a variety of areas, and holds the promise of greatly improving human-machine interaction. One promising application is in the field of home automation, where voice-controlled robots can assist with tasks such as controlling smart appliances, adjusting room temperatures, or managing household security systems. By understanding natural language commands, these robots can seamlessly integrate into domestic settings, providing convenience and efficiency for users, especially those with mobility constraints or busy lifestyles.

In industrial and manufacturing environments, robots equipped with natural language processing capabilities can streamline production processes and enhance operational efficiency. Workers can issue verbal commands to robots for tasks such as material handling, assembly line operations, or quality control inspections. This integration of spoken language instructions enables a more intuitive and flexible human-robot collaboration, optimizing productivity and facilitating the execution of complex manufacturing operations.

Although these robots have immense potential, the complexity of natural language comprehension remains a significant challenge. Interpreting the nuances and context of human speech requires sophisticated natural language processing algorithms that can discern the semantic meaning behind verbal commands. Additionally, it is critical for robots to integrate environmental perception in order to understand how spoken language relates to their surroundings. This requires advanced sensor technologies and perception capabilities to interpret the physical context and spatial relationships, enabling robots to execute tasks accurately and in line with human expectations.

Researchers at MIT CSAIL wanted to give robots the human-like ability to understand natural language and leverage that information to interact with real-world environments. Toward that goal, they developed a system called F3RM (Feature Fields for Robotic Manipulation). F3RM is capable of interpreting open-ended language prompts, then using three-dimensional features inferred from two-dimensional images, in conjunction with a vision foundation model, to locate targets and understand how to interact with them. In this work, a six degrees of freedom robotic arm was used to demonstrate the F3RM system.

The system first captures a set of 50 images, from a variety of angles, of the environment surrounding the robot. These two-dimensional images are used to build a neural radiance field that provides a 360-degree three-dimensional representation of the area. Next, the CLIP visual foundation model, which was trained on hundreds of millions of images, is leveraged to create a feature field that adds geometry and semantic information to the model of the environment.

The final step of the process involves training F3RM on data from a few examples of how the robot can perform a particular interaction. With that knowledge, the robot is ready to deal with a user request. After the object of interest in the request is located in three-dimensional space, F3RM will search for a grasp that is most likely to succeed in grasping the object. The algorithm also scores each potential solution to make sure that the option that is most relevant to the user prompt is selected. After verifying that no collisions will be caused, the plan is executed.

The team showed that F3RM is capable of locating unknown objects that were not a part of its training set, and also that the system can operate at many levels of linguistic detail, easily distinguishing between, for example, a cup full of juice and a cup of coffee. However, as it is presently designed, F3RM is far too slow for real-time interactions. The process of capturing the images and calculating the three-dimensional feature map take several minutes. It is hoped that a new method can be developed that will be capable of performing at the same level with just a few images.