AI Could Do That Blindfolded

Picking up an object and rotating it in our hands is such a simple task that we often forget how complex the many coordinated movements are. Our brains and bodies work together seamlessly to complete this task, which involves many different muscles and joints, and a wealth of sensory information. As we reach for the object, our brains process visual cues and spatial information to calculate the precise position and orientation of our hand. Simultaneously, our muscles receive signals from the brain to contract and relax in a synchronized manner, ensuring the proper alignment of our fingers and the perfect amount of force needed to hold the object securely. The intricate network of tendons and ligaments ensures that our fingers move smoothly and accurately, allowing us to maintain a firm grasp while effortlessly rotating the object.

This seemingly effortless action is a testament to the remarkable integration of our sensory, motor, and cognitive systems, making it a challenging feat to replicate in robotic hands, which often require complex algorithms and advanced sensor technology to even approach the level of finesse and adaptability exhibited by our hands. Not only are these robotic systems very expensive, which puts them out of the reach of many researchers, but they are also so complex as to be very difficult to control. And despite their advanced hardware and control systems, they are easily tripped up by simple object manipulations that a toddler could handle with their eyes closed.

Engineers at the Hong Kong University of Science and Technology and University of California San Diego took on the challenge of building a robotic hand that can perform complex object manipulations, but without expensive or otherwise impractical sensing systems. Initially focusing on the goal of rotating an arbitrary object in the hand without any visual information, they built a system that relies on an array of simple, inexpensive touch sensors and machine learning algorithms.

The team constructed a four-fingered robotic hand with simple, binary touch sensors that only indicate if contact with an object is being made or not. They do not provide nearly as much information as the high-resolution touch sensors commonly found in such a system, but they cost far less at about $12 each. And because of that low cost, many more of them can be practically installed — in this case, sixteen touch sensors were installed, with three on each finger, and four more on the palm. As such, low-resolution data can be collected that spans a large region of the hand. That more broad sensing coverage also allows the researchers to eliminate the need for visual information.

A reinforcement learning algorithm was developed to control the hand’s motion planning system. In order to rapidly collect data to train the algorithm, a simulated environment was created in which the hand interacted with a wide variety of objects of various shapes and attempted to rotate them. Using only the information from the binary touch sensors, the algorithm learned how to rotate virtually any object that came into contact with the hand.

After training the model, it was deployed to a real-world robotic hand. In a series of trials, objects with shapes that were not a part of the training dataset were placed in the hand. It was discovered that this system enabled the hand to rotate these objects without getting stuck or losing its grip. These objects ranged from a jar of peanut butter to a tomato and a rubber duck.

At present, the researchers are looking to enhance the utility of their innovation by extending the functionality to include more complex tasks, like catching, throwing, and even juggling. They are also considering trying out a more dense array of sensors as they work towards creating more intelligent and capable robot hands.