Christopher Moravec's Interactive Artwork Eavesdrops on Your Conversations, Altering to Match Topics
A chain of machine learning models, including OpenAI's Whisper and GPT-4, power this automatically-updating picture frame.
Self-described "maker of things" Christopher Moravec has turned the myth of people's smartphones and voice assistants constantly listening in to their private conversions into reality — in order to automatically generate topical artwork.
"The WhisperFrame listens to conversations in our living room and then generates art based on those conversations," Moravec writes of his project. "[It] generates a new image after every five minutes of active conversation. When there hasn’t been any talking, it will revert to showing randomly selected images generated in the past."
The core concept of the project, which has an always-on microphone recording snippets of nearby conversation, brings up the pervasive but always-unproven myth of companies using smartphones and voice assistants to monitor nearby conversations for topics which could be data-mined and monetized. This time around, though, the very-real conversational recordings are being mined for thematic content which can be fed to a generative artificial intelligence (AI) system to create artificial art.
The recordings are made in 15-20 second loops, then submitted to OpenAI's Whisper application programming interface (API) for automatic transcription into text. When five minutes has elapsed, these extracts are fed to OpenAI's GPT-4 large language model (LLM) with the prompt to extract one key topic and turn it into a prompt for an image-generating model — which is, in turn, fed to Stable Diffusion, the resulting picture downloaded, and the display updated.
"It’s a bit self-fulfilling in that as people talk about the image it drew, it becomes more likely that it tries to illustrate that one again, as the topic is more likely to be selected by GPT-4," Moravec admits. "But it’s still awesome! I even created a second one for my office that generates images during meetings! It might even be a new way to make meeting notes, a list of images representing the meeting instead of action items. It probably won’t catch on, though!"
The full write-up is available on Moravec's website; all generated images are available to browse on a dedicated site.