AI Puts Creative Storytelling in Your Hands

Ah, the nightly bedtime routine — wide-eyed children brimming with excitement for storytime with their parents. And parents that are spent from the long day, with no creativity left in them to invent yet another story about talking forest animals and their narrow escape from the forces of evil. But not too evil of course, no one wants a midnight visit to talk about nightmares, after all. Ah, the pressure! Who do these kids think they are, anyway, Dr. Seuss?

But the show must go on! So, with the heavy weight of their children’s expectations on their shoulders, they embark on another storytelling adventure. The pressure mounts as they desperately try to concoct a tale that lives up to their children's vivid imaginations. But nothing is coming to mind! There will be a revolt! A refusal to go to sleep! All is lost.

Anyone who has experienced this feeling during their childrens’ nighttime rituals will be able to really appreciate the interesting storytelling device created by Aria Xiying Bao and Yubo Zhao at MIT’s School of Architecture and Planning. Their interactive projector, called the Narratron, allows users to make some shadow puppets, then it writes an entire story involving the shadow puppet characters that they created. The fully narrated story is projected on the wall to create what they call a phygital storytelling experience.

Designed in the style of an old reel-to-reel projector, the front of the Narratron has an on-device camera that captures the hand puppet shapes being projected on the wall. The images are fed into trained image classifiers that were built with the Teachable Machine platform to determine what the users were trying to portray. These then serve as the main characters in the story.

These characters are then inserted into a children’s story with the help of a GPT-3.5 large language model that never gets tired or runs into a mental roadblock. This model combines plotlines, dialogues, and descriptive elements to create an endless number of stories.

To generate visual elements to accompany the story, the Stable Diffusion algorithm was leveraged. It was trained on a set of images painted in a childlike style with the help of the Replicate framework. This helps to make the stories more engaging, and also ensures that the imagery will be age-appropriate. With all of the other pieces in place, a speech synthesizer was utilized to tell the stories audibly.

The Narratron itself is made from commercial-grade SLS-printed nylon and is powered by a microcontroller. The speaker, projector, and all of the other components are inside the case to give the device a minimalist appearance. A hand crank on the side of the Narratron is a great touch — it serves only to advance to the next chapter of a story, which a button could just as easily do, but turning the crank really gives one the feeling that they are involved in making the story. Perhaps that will give some solace to the parent that feels guilty for handing their duties over to a machine.

Another interesting feature of the Narratron is that the users can make more shadow puppets at any time to change the plot of the story. New characters can be introduced in this way, and it further reinforces that the story is being co-created by the users and the machine.

While the Narratron is a proof of concept device at this point, it has some serious appeal. It is easy to envision such a device on the shelves of a toy store in the near future. I personally know at least one parent that would love to get their hands on a Narratron.