Erik Oaks' Drew Gives OpenAI's DALL-E 2 a Physical Outlet for Its Artificial Creativity

Maker Eric Oaks has given a text-to-image deep learning model a way to express itself in the real world, building a pen plotter robot linked to OpenAI's DALL-E 2 — and it can even play Pictionary with you.

"[It's] a robot that can draw anything," Oaks explains of his creation. "If I tell this robot to draw an elephant riding a bicycle on Mars, it will draw it. If I tell this robot to draw a robot drawing a robot, it will draw it. [It's built] using parts I found around my home, such as rails, a pulley system, and stepper motors, [and] an Arduino to controller the stepper motors to make everything move."

At its heart, Oaks' creation is a simple two-dimensional XY plotter. Given a pen to hold and a piece of paper on which to draw, it will happily sketch out any vector graphic. Where it drifts from the norm is being able to make up its own artwork — by farming the creative part of the work out to OpenAI's DALL-E 2.

Released in April last year as a successor to 2021's DALL-E, DALL-E 2 is a deep learning model capable of creating images from a text input using OpenAI's Generative Pre-trained Transformer (GPT). Given any prompt, DALL-E 2 will have a crack at illustrating it — and it's these illustrations that are being fed into Oaks' drawing robot.

The imagery fed to the robot — which Oaks named "Drew" — comes from DALL-E 2, with an external tool handling the conversion from bitmap to vector and reformatting to GRBL to drive the plotter. "Another feature I implemented in Drew was a one-player Pictionary mode where you can ask Drew to draw something random," Oaks adds. "The script will call ChatGPT [OpenAI's text chat model] to come up with something for Drew to draw."

Oaks isn't the only one investigating how artificial intelligence projects like DALL-E can be given an outlet in the real world. Carnegie Mellon University's Robotics Institute has been working on a similar project, dubbed FRIDA, in which an off-the-shelf commercial robot arm is tied into a text-to-image model and given a brush and some pints to bring them to life.

More details on Oaks' project are available in his YouTube video; FRIDA, meanwhile, is detailed in a preprint available on Cornell University's arXiv server.