Robots Get in Touch with Reality

Imagine that you could only use your fingertips to pick things up for a day. That may not sound like so much of a problem when working with a computer, but working with tools, driving, and any number of other normal activities would be somewhere between awkward and impossible under these conditions. But this is exactly the situation that most robots are in. Due to technical complexities or cost, robotic manipulators typically only have sensors embedded in their fingertips, which leaves them at a major disadvantage in terms of dexterity.

In contrast, we are capable of sensing the shape of objects, and the force with which they are being grasped, along the entire length of our fingers. This remarkable ability, known as tactile sensing, allows us to perform intricate tasks with precision and adapt to varying circumstances. It is what allows us to gently pick up a fragile egg without breaking it, firmly grip a heavy tool, or discern the texture of a surface just by touching it.

The development of GelSight sensors in recent years has promised to give robotic manipulators more human-like abilities. These devices leverage low-cost cameras embedded within a soft elastomer skin to measure tactile information with a high degree of precision. As the elastomer is deformed by a grasped object, the camera captures the changes to the inner surface of the skin, and that data is in turn leveraged to calculate contact areas and object shapes. But unfortunately, due to the focal distances of the camera lenses and their viewing angles, GelSight sensors can only capture data from small areas, leaving their range limited to fingertip sensing.

Thanks to the work of a pair of researchers at MIT, GelSight sensors can now behave in a more human-like way. Some clever optical hacks have made it possible for a single camera to measure fine tactile sensing information from the full length of a robotic finger. These new sensors, called GelSight Svelte, were demonstrated in a robotic hand that showed itself to have amazing versatility and dexterity, grasping large, heavy objects like a human hand, or manipulating small objects to perform delicate work with the fingertips.

In order to position the camera far enough away from the surface of the skin, the researchers leveraged a pair of mirrors. One of the mirrors is flat and positioned across from the camera at an angle, such that it can reflect the light that falls on it. Another long, curved mirror is positioned behind the sensor. The combination of these two mirrors serve to reflect and refract light in just such a way that the entire surface of the finger is visible to the camera. This was no easy task — software had to be designed to determine the proper alignment and curvature of the mirrors to get a complete view of the interior of a finger’s full length.

Additionally, a plastic backbone was built into the robotic hand to provide extra information about forces being applied to the fingers. It can even determine if complex factors, like twisting forces, are being felt by the fingers. This information, in conjunction with imaging data from the GelSight Svelte sensor is fed into a machine learning algorithm to assist in interpreting it.

The three-fingered hand that was built with their technology was shown to be able to perform a wide range of tasks that involved pinch grasps, lateral pinch grasps, and power grasps. In each case, the full area of each finger was available for sensing. The incredible sensitivity of the GelSight Svelte sensors was also demonstrated as it clearly identified a number of objects, like screws, as they were pressed into the surfaces of the fingers.

The sensors already provide a great deal of versatility and dexterity, but moving forward the team wants to enhance those abilities further. They intend to, for example, add additional joints of the fingers so that they can bend in more human-like ways.