Black Box Camera Uses AI to Recreate Scenes

You do not see objective reality; you perceive a world distorted by your own brain. That's the reason that optical illusions work and why you can go about your day without being overwhelmed by sensory information. Cameras, on the other hand, do capture objective reality — at least as much as they can from a single vantage point with lens distortion. But what if a camera worked more like the human brain and recorded a subjective retelling of reality? That's what Jasper van Loenen answered when he built the Black Box Camera.

This might just be the most interesting camera we're ever come across. Like a Polaroid, the user can snap a photo and get an instant print. But the printed photo won't be an exact match of the scene. Instead, it will be a visual interpretation of the scene. This is very similar to how your brain perceives your surroundings, or how you retell stories in your mind and the stories change slightly with each retelling.

This works by using OpenAI's DALL-E 2 to simulate the human brain's subjective influence. The camera takes a normal photo, then passes it to a machine learning model that describes what it sees in that photo with a lot of detail. The camera passes that description on to DALL-E 2 and asks it to generate a new image that matches. Finally, the camera prints out the AI-generated image. The printed photo often looks a lot like the original, but is never identical. There are always differences between the reality and the interpretation — sometimes subtle and sometimes quite major — just like a person's perception of the world.

Particulars about the hardware are missing from the project page, but we can make inferences based on van Loenen's other work. We can deduce that the 3D-printed camera enclosure contains a Raspberry Pi single-board computer and a Fujifilm Instax Mini Link smartphone photo printer. The Raspberry Pi connects to Wi-Fi to send prompts to DALL-E 2, and connects to the photo printer via Bluetooth. The camera sensor is likely either a USB webcam or a Raspberry Pi Camera module. The viewfinder is analog: a mirror reflects the scene upwards towards the user.

The Black Box Camera is a really creative use of modern technology, but also a very astute reflection of the human experience and great reminder that we can never fully trust what we see.