Local LLMs and Python Power Chris Fenton's Grasso, the Yard Robot Built From "a Bunch of Garbage"

Electrical engineer Chris Fenton has built a yard robot driven by compact on-device large language model (LLM) technology — and built from "a bunch of garbage" in homage to the 1980s and 1990s sci-fi aesthetic.

"I, like many other people on the planet, have been following along with the recent development of large language models (LLMs) like ChatGPT and friends, and I thought it seemed like a good time to try something fun. I’ve always liked the idea of ‘independent’ robots – think Bender from Futurama, not some sad robot always talking about its creator, or questioning its existence.

"I got a Lego Mindstorms kit for Christmas about 25 years ago, and the first thing I did was build a robot ‘hamster’ that just hung out and wandered around a little pen in my room. There’s also been a ton of great sci-fi written from the perspective of robots that’s come out recently (go to your library and get something by Anne Leckie or Becky Chambers!)."

The product of these musings is Grasso, a yard robot driven by "a kind of Python 'madlib' wrapped around two LLMs," one of which can handle image inputs and the other of which is text only. Grasso's Python brain begins by capturing an image from an integrated webcam and submitting it to the multi-modal LLM to generate a text description; this is then used to inform a prompt that is submitted to a text-only LLM to generate the robot's next action.

"I wanted Grasso to be entirely ‘local’ (it’s ultimately meant to live off of solar power in my yard, after all!), which puts a lot of limitations on what I can get away with," Fenton explains. "A 4k token context limit (and the underpowered CPU running things) means I needed to get creative. A core part of Grasso is that it’s ‘stateful’ – the prompt incorporates both its most recent two actions, as well as a bank of 6 ‘core memories’ that Grasso can choose to update."

Grasso's body, meanwhile, is based on the "Trash Robot" aesthetic. "It seems like these were a staple of 1980s and 1990s TV and movies," Fenton explains, "walk into a junkyard, slap together a bunch of garbage in a montage full of of sparks and motivating rock music, plug in a 'CPU' somehow and *bam* Domo Arigato!" Fenton's build, then, houses its electronics in an upcycled plastic bucket, on top of which is a toaster "torso" with an HDMI display. Above these, on a plank of wood, is a head made from an upturned watering can, a webcam — with added googly-eye — and a mini-umbrella, while the robot's arms are built from leftover pipe lagging.

Inside the bucket is a compact computer based on an Intel Processor N100 and 16GB of DDR5 memory, which runs the Llava-v1.6-mistral-7B multi-modal and Mistral-7B on-device. The motors are controlled via an Arduino microcontroller with motor driver, and the display and webcam connect over HDMI and USB respectively. There's also a surprise, in the form of Grasso's voice: an Aicom Accent SA Text-to-Speech synthesizer, built in the 1980s and powered by a Zilog Z80.

"This was a find from the junk pile at NYCResistor and completely lacked documentation," Fenton explains of the speech synthesizer, "but after some Internet Sleuthing, I was able to track down the creator of it on Facebook and he was able to dig up 35 year old documentation from a box in his garage somewhere! The Hawking-esque voice is really quite amazing."

The project is documented in full on Fenton's website, along with a copy of the Python script powering the robot — provided under an unspecified open source license.