Getting a Grasp on Object Identification

Scientists and engineers have been making incredible strides in the field of robotics, with new technologies constantly emerging to improve the capabilities of machines. One area of robotics that has proven particularly challenging, however, is the development of robotic hands that can replicate the complex capabilities of the human hand.

Despite advances in robotics, artificial hands still struggle to replicate the range of motion, strength, and precision of the human hand. But perhaps the most difficult aspect to replicate is the sense of touch.

Touch is incredibly complex and multifaceted. The human hand is capable of sensing pressure, temperature, texture, and even pain. Additionally, the human hand is capable of differentiating between objects based on their shape, size, weight, and other physical properties.

It is this latter capability, that of distinguishing between objects, that a team of researchers at MIT has been working to improve. They have created a robotic hand with three fingers consisting of a soft silicone skin and a rigid, 3D printed skeleton. Using specialized sensors and intelligent algorithms, this artificial hand was given the ability to identify objects that it grasped a single time.

Existing solutions typically tackle this problem through one of two approaches — with high-resolution finger tip sensors or with larger low-resolution sensors that cover the length of the fingers. While many of these solutions have proven themselves capable of accurate object recognition, they require that the object be in full contact with the fingertips, or be regrasped multiple times.

The newly proposed solution takes a different approach by placing a pair of special sensors, called GelSight sensors, underneath the skin of each finger. Each sensor consists of a camera and three LEDs of different colors. When an object is grasped by the hand, the LEDs illuminate the inner portion of the silicone skin, which highlights contours made in the skin by its many points of contact with the object. Between the two cameras in each finger, the entire length of it can be imaged.

To make sense of this data, a ResNet-50 architecture neural network was trained to translate sensor readings into a prediction of the object that the hand is holding. When testing this model against the validation dataset, an average classification accuracy rate of 94.1% was observed. Further live tests were conducted in which the robot grasped one of three different objects. In these experiments, the model made the correct prediction in 80% of cases.

Aside from the tactile sensing capabilities of the hand, the physical construction also confers some useful properties on the device. By including both a rigid skeleton and a soft skin-like layer on the surface, much like a real hand, it is able to pick up heavy items, like a power drill, while still maintaining the ability to pick up a soft, deformable object like a plastic water bottle. Being able to perform a wide variety of tasks is essential for any general-purpose robotic hand.

Moving forward, the team plans to explore adding a thumb-like joint to the fingers, and also adding full fingertip sensing. They believe that this will make the device more useful for sensing and dexterous manipulation tasks. Additionally, they are considering the possibility of adding markers that will allow the GelSight sensors to track slip, shear, and torsional forces, which would be expected to make object detection more accurate.