A Deep Dive Into Reinforcement Learning

Uncrewed Underwater Vehicles (UUVs) are emerging as invaluable tools for a variety of applications due to their versatility and ability to operate in challenging underwater environments. These autonomous or remotely operated vehicles play a crucial role in tasks ranging from scientific research to defense and industry applications. One of the primary advantages of UUVs lies in their capability to access areas that are difficult or dangerous for humans to reach, providing an efficient and cost-effective means of exploration and data collection beneath the ocean's surface.

However, controlling UUVs presents a number of unique challenges, particularly with signal propagation in water. While radio waves are easily transmitted through air, in water these signals face increased attenuation, making remote control and communication with UUVs more challenging. This limitation poses obstacles in terms of real-time control and data transfer, especially at greater depths. Furthermore, traditional means of autonomous control, such as GPS navigation, are hindered by the fact that water blocks GPS signals, rendering them ineffective for underwater vehicles.

Furthermore, the low visibility conditions commonly found in natural bodies of water, such as oceans and lakes, pose an additional hurdle for UUVs. Murky waters limit the effectiveness of cameras and other optical sensors, making it difficult for these vehicles to navigate and perform tasks that rely on visual information.

A team led by researchers at Flinders University in Australia is very interested in putting UUVs to work, cleaning up the hulls of ships where bio-organisms like to hang out. The films that these organisms create are known to introduce invasive species around the world, and also increase drag on ships, which decreases their fuel efficiency. But, for the aforementioned reasons, this is a deceptively difficult job.

As a step towards more capable autonomous UUVs, the researchers developed a control system using deep reinforcement learning methods, with some non-traditional tweaks that allowed it to learn more, and faster. Many efforts have been underway to improve these algorithms, but in this case, the team focused their attention on memory buffers. As a reinforcement learning system learns through trial and error, the actions taken, and the observed results, are stored in this memory.

This information is used to update the model’s weights to help it improve over time. But the information taken from this memory is normally sampled at random. That is not how humans learn, rather the researchers noted that we tend to look back on more recent experiences, and in particular, at experiences that resulted in a beneficial outcome. Accordingly, they tweaked their control system such that it would give more weight to recent items in the memory buffer, especially those that achieved a large positive outcome.

In a series of trials, it was found that this method allowed models to be trained more quickly than when using traditional methods. Accordingly, energy efficiency was also enhanced during the training process. Both of these factors are crucial for UUVs, because for effective operation, the vehicles often need to be retrained after they are put into service. Since these submersibles are very expensive, they need to become competent in their roles very quickly so they do not wind up damaged or lost. And of course energy is in limited supply onboard UUVs, so minimizing the amount used in training the control system is quite important.

To date, the researchers have primarily tested their methods in simulated environments. In the near future, they intend to try the system out on real UUVs in the ocean. They hope that effort will lead to a new class of autonomous underwater vehicles that will benefit industry and the environment alike.