Article complete

UC Berkeley Releases Open LLaMA, an Open-Source Alternative to Meta's LLaMA

The team is also working on evaluations and a 3B model is also being actively trained which will be released soon.

A cartoon llama wearing sunglasses against a bright yellow background with the text "LLAMA" in large letters.

Key takeaways

  • UC Berkeley has launched the 7B OpenLLaMA model as an open-source alternative to Meta's LLaMA.
  • The model is trained on the RedPajama dataset, which includes 200 billion tokens, and is available in PyTorch and Jax.
  • OpenLLaMA demonstrates comparable performance to LLaMA and GPT-J, with potential for further improvements as training progresses.
  • The team is also developing a 3B model and conducting ongoing evaluations to enhance performance.
  • OpenLLaMA allows for the re-training of non-commercial models under a permissive licence, expanding accessibility in the research community.

Takeaways drafted with AI from the article text.

Researchers at the University of California, Berkeley have released a preview of the 7B OpenLLaMA model, an open-source alternative to Meta's LLaMA language model. The model has been trained on the RedPajama dataset with 200 billion tokens, and its weights are available in PyTorch and Jax. With the latest release all non-commercial models stemming from LLaMA can now be re-trained with a permissive licence. 

Check out GitHub repo and model weights here. 

The RedPajama dataset used for training contains a whopping 1.2 trillion tokens, equivalent to that used by LLaMA. To train OpenLLAMA, the researchers used a cloud TPUv4 pod with data parallelism and FSDP or Zero3 for balancing throughput and memory usage. Their training run achieved a throughput of over 1900 tokens/second/TPUv4 chip.

Evaluation of OpenLLAMA using the lm-evaluation-harness from EleutherAI showed that it performs comparably to LLaMA and GPT-J in most tasks and even outperforms them in some. The team expects that OpenLLAMA's performance will improve even more once it completes its training on 1 trillion tokens. The authors are also working on evaluations and sharing that OpenLLaMA is on par with the original model, or even better in most cases.

The team is also working on evaluations and a 3B model is also being actively trained which will be released soon. 

Meta’s LLaMA Saga

As Meta’s LLaMA is bound by industrial licences it was not possible to directly distribute LLaMa-based models but so is not the case anymore. Several attempts have been constantly made to open source the models. Open LlaMA is not the first one in the domain. Less than two weeks ago, Hugging Face, the open source AI platform,  released an open-source alternative to, ChatGPT named HuggingChat. The chatbot runs on OpenAssistant’s latest LLaMA based model which provides XOR weights for the OA models. 

Furthermore, Databricks also figured out a way around this with Dolly 2.0. The differentiating factor between other ‘open source’ models and Dolly 2.0 is that it is available for commercial purposes without the need to pay for API access or share data with third parties unlike the rest.

Here are: 14 Open Source LLMs You Need to Know

Published

Spotted an error? Tell our editors. Read our editorial standards and corrections.

More in this story

OpenAI · 5 stories
  1. Microsoft’s Bing-ChatGPT Combo Can’t Stand Up to Google’s LaMDA
  2. Stop Questioning OpenAI’s Open-Source Policy
  3. You are here: UC Berkeley Releases Open LLaMA, an Open-Source Alternative to Meta's LLaMA
  4. OpenAI's Desperate Attempt to Create Its Own Open-Source Ecosystem
  5. You Can Now Write CUDA Kernels in Rust
About the author

Contributor

All stories
Belamy · Daily

The day’s AI news, before your first meeting.