Turn Doodles of Instruments Into Actual Music

Developed by Andreas Refsgaard and Gene Kogan at this year’s Nabi AI Hackathon in Seoul, Doodle Tunes is a project that turns drawings of instruments into actual music through machine learning.

As you can see in the video below, a camera looks at your paper, detects instruments that you have drawn (such as a saxophone, a piano or a drum), and begins playing electronic music with those instruments.

Doodle Tunes was created with openFrameworks, which uses computer vision (OpenCV) and convolutional neural networks (ofxCcv) to analyze the images. OSC messages are then sent to Ableton Live, triggering the various instrumental clips.

You can read more on this awesome project here, and find its software on GitHub.