AI and Security Cameras Predict the Weather

Widely distributed networks of sensors are more practical than ever to deploy as a result of the plummeting costs of electronic components. However, the costs and challenges associated with physically deploying and maintaining the devices can still be quite substantial. These factors are hindering many important applications that could benefit from large sensor networks — especially those involving environmental monitoring.

If you think about it, we already have a huge installed base of sensors all over the world. They may not be measuring all of the things that we want them to measure, but perhaps they could be co-opted for other purposes. A group led by researchers at the Nanjing Normal University realized that the security cameras that are nearly ubiquitous in today’s world might be a great target for repurposing. Aside from video, these cameras also typically capture audio, and the researchers believed that this could present an opportunity to collect data for their own area of interest — namely, rainfall intensity estimation.

Using the video stream for this purpose might seem like the obvious first choice, however that would require dealing with variations in lighting, shadows, and so on, that would complicate matters greatly. Since using audio would sidestep these issues, and reduce the computational complexity of the problem as well, the team decided to train a deep learning model that could estimate rainfall intensity based solely on the sound of falling rain hitting the ground.

As a first step, they created a publicly available audio dataset called the Surveillance Audio Rainfall Intensity Dataset (SARID). The samples were collected during six different real-world rainfall events, and were split into 12,066 pieces. Each piece is labeled to indicate the ground truth value for rainfall intensity, and also what the environmental conditions looked like. It was noted, for example, the type of surface that the rain was falling on, what background noises were present, and other metrics like temperature, humidity, and wind were also recorded in the annotation.

Several analysis pipelines were evaluated to help understand which would provide the best performance for the task. It was found that using Mel-Frequency Cepstral Coefficients to extract features from the audio samples, before forwarding those features into a Transformer-based machine learning model produced the best rainfall intensity estimates. When compared to ground truth measurements, this pipeline achieved a root mean absolute error of 0.88 mm h-1 and a coefficient of correlation of 0.765.

The current dataset is somewhat limited in that it does not compare how different surfaces (e.g., asphalt, grass) affect the audio captured during the same rainfall event. Including this type of data could serve to enhance the accuracy of the system. Furthermore, a number of meteorological metrics were not utilized by the model — this also presents an opportunity for future improvement. Looking ahead, the researchers plan to address these issues. They are also considering the possibility of developing a multimodal model that incorporates the video stream from security cameras to see if that might make the estimates even better.