Sound Advice on Seeing the Unseen

As a general rule, when you need to measure some phenomenon with extremely high levels of precision, the instruments that are required for the job will balloon in cost. This can easily price the typical hobbyist out of exploring ideas that they are interested in, and not only is that a bummer, but it can also serve to stifle innovation. But as instrument costs climb into the tens or hundreds of thousands of dollars, it simply becomes impossible for the vast majority of people to get their hands on the tools they need.

As hardware hackers, we should always be asking ourselves if there is another option when an apparent roadblock is standing in our path. That is what YouTuber Benn Jordan explored in a recent video that aims to reveal to us the vast amount of information that is hidden in simple audio and video files captured by inexpensive hardware. In particular, Jordan was thinking about professional motion amplification systems that are frequently used to predict and diagnose problems with machinery. Commercial systems (useful ones, anyway) sell for tens of thousands of dollars and go up from there.

So Jordan considered how a far less expensive motion amplification system could be developed for jobs like locating electronics that are generating unwanted noise or measuring an individual’s heartbeat from a video. It occurred to him that even a 4K camera on a modern smartphone captures over eight million pixels per frame — that is a lot of data! From there, one only needs to detect changes in pixels over time to amplify motion.

Jordan also wanted to build an acoustic camera that can locate the source of a sound. Toward that goal, he hooked a 1080p camera up to a Raspberry Pi 5, then paired it with a 16-channel USB microphone. Feeding this data into Acoular, a Python-based framework for acoustic beamforming, made it possible to locate exactly where a sound originated from in a video frame. It was demonstrated, for example, that the system could pick out which chicken in a group was making noise. It could also locate the positions of distant hawks that might be interested in snacking on those chickens.

Vibration detection was taken on in another project. A surprisingly simple approach was pursued. By blending a video file over itself, with a slight time difference between frames, vibrations can be revealed. This hack proved to be useful in locating an area that was vibrating in the acoustic rafters of Jordan’s filming studio when certain frequencies of audio are played. And with a little bit more filtering on the software side to eliminate irrelevant movements, this setup was even capable of measuring an individual’s heartbeat using only a standard video stream.

Through the remainder of the video, Jordan showed how inexpensive motion amplification, Schlieren imaging (which makes the atmosphere visible), and thermal imaging of sound can be achieved. All of these hacks may not necessarily be useful, but they will undoubtedly make you reconsider what is possible with the instruments that you already have in your home electronics lab. What hidden information might you be able to reveal that could save you bundles of cash or help you to better explore your ideas?