Spoon Feeding Knowledge to Robots

If you need to fill a job that requires little more than repetition of the same physical task over and over again, a robot might be the ideal candidate for it. Not only are robots fast and efficient in these scenarios, but they also never get tired or call in sick. But while machines are masters of simple, preprogrammed motions, they are only just beginning to learn how to work and adapt in the sort of unstructured environments that we find outside the confines of a production line or warehouse.

This is a big reason why we do not have much help from robots with domestic tasks, for example. Sure, they are pretty good at vacuuming the floor, but — for starters, at least — that requires little more than driving around in a pattern while doing some basic obstacle avoidance. When it comes to something more complex, like making dinner, the job gets exponentially harder. There are dozens of things that have to happen to make a meal, and each of these tasks is difficult enough to bring a robot to its knees.

Researchers at National Yang Ming Chiao Tung University and NVIDIA teamed up to tackle one of the tasks that stand in the way of building an eventual robotic chef. They have developed what they call the SCOoping robot learNing framEwork, which forms the not-at-all-labored acronym SCONE. This framework deals with the deceptively difficult task of scooping different types of food. This seems like an easy enough thing to do because us humans are so adaptable and versatile, but robots struggle with the many properties of food, such as its deformability, fragility, fluidity, and granularity.

SCONE operates in two key stages: Interacting and Manipulation. In the interacting stage, the robot directly engages with the food to collect dynamic sensory data. This interaction provides important real-time insights into the food's physical properties — such as texture and firmness — through feedback from the robot’s sensors. Unlike methods that rely solely on visual input or predefined actions, SCONE actively perceives food characteristics by physically engaging with the target, allowing it to adjust to the unpredictable nature of different food types.

Data from the interactions is processed through an interactive encoder and a state retrieval module. The first module encodes the observations from the robot’s interaction with the food, capturing broader, high-level information about the food’s global properties. The second module focuses on more detailed, local state information at each step of interaction. It records finer details, such as how a specific part of the food responds to the scooping action, enabling the robot to make precise adjustments during the task.

These two streams of information are fused into a task-related embedding. This embedding effectively summarizes the relevant food properties and state information needed for the robot to execute its scooping policy. The framework operates in a closed-loop system, meaning it continuously integrates the robot’s current observations with the task-related embedding to guide its actions in real time. This feedback loop allows the robot to adjust its strategy during each moment of interaction, enhancing its adaptability to different types of food.

SCONE's ability to generalize across different food types was demonstrated in real-world tests, where it achieved a 71 percent success rate in scooping six previously unobserved food categories across three levels of difficulty. These tests confirmed SCONE's superior adaptability compared to static methods, as it minimized spillage and food damage, key challenges in robotic food manipulation.

SCONE may be just one piece of the puzzle, but without it, the picture might never be complete.