This Dynamic Piece of Art Responds to Its Picture Being Taken

The concept

When thinking of various forms of art, the most common are typically paintings or sculptures that hardly move, let alone react to the presence of someone standing in front. Inspired by Peter Vogel and Walter Giers, Estonian artist Tauno Erik's latest project involves adding embedded machine learning into a very interesting circuit that can play tunes from a Soviet-era doorbell board when a person is detected taking pictures of it.

Electronic components

To accomplish this goal, Erik began by selecting which components he needed in order to build the piece. Controlling everything is done with a single Arduino Nano 33 BLE Sense due to its suite of onboard sensors that can sense motion. Because of the need for object detection, he included an OV7670 camera module that takes a picture of the immediate area and transmits it back to the microcontroller over the SPI bus. The previously mentioned doorbell assembly is activated by the Nano 33 BLE Sense and can, in turn, play a variety of sounds through a connected speaker. And last of all, a simple LED sculpture was added for giving the viewer some extra feedback.

Training an object detection model

Most of the time spent building this project was dedicated to the machine learning model since Erik had to collect nearly 700 images in total and label each one as either unknown, human, or cell phone. One problem he encountered was that the model expected image sizes of 96x96 pixels and in RGB888 format. So to solve it, two functions were created. The first converts the incoming YCbCr422 format into RGB88 while the second resizes the frame buffer with a bilinear interpolation algorithm. Once trained, the model achieved an adequate accuracy of 70.5%, mostly due to how similar pictures with phones looked to those without one.

Responding to photography

As seen in his demonstration video here on YouTube, the artwork begins with silence and waits until a person is detected by the APDS9960's proximity sensors to activate an LED. Furthermore, actively taking a picture will cause the doorbell circuit to get toggled on where it can then blare a series of tones. More information can be found here on Erik's website.