Alpaca: The Large Language Model That Won't Fleece You

Have you heard any interesting news in the world of machine learning lately? If not, you have not been reading Hackster News! We are a bit sad if that is the case, but let's not dwell on it. No, no, don’t worry about us, we will be alright. Let us quickly bring you up to speed while we regain our composure.

We recently reported on Meta AI’s Large Language Model (LLM) that favors training depth over parameter count to run on far more modest hardware, then shortly after, we reported on the leak of this model’s trained weights. When Meta AI’s role as sole gatekeeper disappeared, we saw hackers running LLMs on everything from smartphones to Raspberry Pi 4s. After this, a surprise announcement revealed the release of ChatGPT’s next major upgrade — the GPT-4 model. We even saw a practical method to perform visual question answering by leaning on existing, pretrained models. Since no training is needed, this puts the power in the hands of the people that do not have a multibillion dollar R&D budget.

This trend of bringing powerful LLMs to a much larger audience does not appear to be slowing down any time soon. Stanford University’s Center for Research on Foundation Models has recently reported on an instruction-following LLM called Alpaca. Not only does this model run on modest hardware, but it can even be retrained on a modest budget to fine-tune it for new use cases. Using their methods, the team showed it was possible to retrain their LLM for less than $600.

Instruction-following models like ChatGPT, Claude, and Bing Chat have taken the world by storm, but they are closed-source, and require massive amounts of computational resources to experiment with. The Stanford researchers seized upon the opportunities presented by Meta AI’s LLaMA model to run on smaller computing platforms and took it one step further in devising a means to inexpensively retrain such models. This puts the technology into the hands of academic researchers and tinkerers to help address some of the deficiencies that presently exist in these models.

With a strong pretrained language model in hand, thanks to LLaMA’s availability, they only needed high-quality instruction-following data to build a system on par with the instruction-following models of the big players. Recent research suggested that this type of data could automatically be generated by prompting an existing strong LLM to produce it. The team did exactly that, and with a seed of 175 human-written instruction-output pairs, they created a dataset of 52,000 examples generated by OpenAI’s text-davinci-003 model. The cost for this number of API queries came in at just under $500.

Hugging Face’s training framework was then used to retrain LLaMA with this additional dataset. Using eight NVIDIA A100 Tensor Core GPUs, the model retraining took about three hours to complete. Relying on cloud computing providers, the cost for this compute time would generally be under $100. The team also noted that there are still areas where efficiency could be improved, which would reduce the cost further.

Five of the authors took part in a blind evaluation of Alpaca vs. text-davinci-003 using a diverse list of user-oriented instructions. The comparison revealed that the models performed very similarly, with Alpaca ultimately being given a slight edge in terms of performance.

The success of this approach has been somewhat surprising, given the small size of the dataset that was used for retraining. There may still be limitations that have not yet been discovered, however. Alpaca is still being subjected to additional testing at this time.