Free the AIs!

Generative artificial intelligence (AI) has made a big splash this past year, with text to image generators and coding assistants coming into the spotlight in a big way. But the large language models (LLMs) that power chatbots like OpenAI’s ChatGPT and Google’s Bard have arguably been the most impactful of these technologies to date.

These sophisticated models have not only revolutionized natural language understanding and generation but have also found applications in diverse fields such as content creation, customer support, and even mental health chatbots. The ability of LLMs to understand context, generate human-like responses, and adapt to various conversational styles has elevated them to become indispensable tools for businesses and individuals alike.

But these tools have not been without their controversy. The computing horsepower required to run these models has been massive, which has put these tools in the hands of a few large corporations. And that has sparked worries about the ways that these models could be manipulated — what they will say, and what they will be prevented from saying. Moreover, it raises many questions about who is reading the chat transcripts, and the purposes that this information might be used for. And aside from these big issues, there are also more mundane concerns, like the lack of availability to the chatbots due to high usage levels or connectivity issues.

Open source solutions, like Meta AI’s LLaMA model, have appeared on the scene to address these issues and allow individuals to run their own LLMs without a datacenter or multimillion dollar budget at their disposal. While these advances are of tremendous importance, they are virtually unusable for the average person. Even for the technically-inclined individual, installing all of the proper dependencies, toolkits, libraries, and so on can be a nightmare. And if you want to install a second model, chances are high that you will break the first thing that you finally got working after days of aggravation.

What good is the best technology in the world if it is inaccessible? This must have been the question kicking around in the minds of engineers at Mozilla Ocho that spearheaded a new open-source initiative called llamafile. This project makes it possible to package up the LLM model, dependencies, frameworks, UI, and everything else that is needed for operation into a single executable file. And as if that were not easy enough, the same executable can run on six different operating systems, ranging from Windows and Mac OS to Linux and Unix. What, you still want more? Fine then, a single llamafile can even run on multiple architectures, like x86 and Arm.

The project achieved this goal by utilizing two existing projects — llama.cpp and Cosmopolitan. Georgi Gerganov’s llama.cpp enables LLMs to run on consumer-grade hardware with acceptable performance, and will even work well with just a CPU if you do not have a fancy GPU. Any decent laptop or desktop from the past several years should give good performance, and even single board computers like the Raspberry Pi have enough power to get in on the fun. The other piece of the puzzle is Justine Tunney’s Cosmopolitan, which allows llamafile to run on virtually any modern operating system or hardware architecture.

In the project’s GitHub repository, there are several downloadable llamafiles ranging in size from 4 GB to 30 GB each. There are both command-line versions and server versions of each. If you choose the server version, all you need to do is download and execute it. That will trigger a web UI to pop up where you can interact with the chatbot just like you would with an online service. But of course, the llamafile is completely offline, totally private, and is never unavailable. My own tests have found llamafile to be just as easy as advertised, and the performance has been excellent. Unlike traditional methods, this will not take hours or days to get working. Rather, you can be up and running within seconds of finishing the download. As such, it is well worth giving llamafile a try.