UC Berkeley Releases Open LLaMA, an Open-Source Alternative to Meta's LLaMA

Researchers at the University of California, Berkeley have released a preview of the 7B OpenLLaMA model, an open-source alternative to Meta's LLaMA language model. The model has been trained on the RedPajama dataset with 200 billion tokens, and its weights are available in PyTorch and Jax. With the latest release all non-commercial models stemming from LLaMA can now be re-trained with a permissive licence. 

Check out GitHub repo and model weights here. 

The RedPajama dataset used for training contains a whopping 1.2 trillion tokens, equivalent to that used by LLaMA. To train OpenLLAMA, the researchers used a cloud TPUv4 pod with data parallelism and FSDP or Zero3 for balancing throughput and memory usage. Their training run achieved a throughput of over 1900 tokens/second/TPUv4 chip.

Evaluation of OpenLLAMA using the lm-evaluation-harness from EleutherAI showed that it performs comparably to LLaMA and GPT-J in most tasks and even outperforms them in some. The team expects that OpenLLAMA's performance will improve even more once it completes its training on 1 trillion tokens. The authors are also working on evaluations and sharing that OpenLLaMA is on par with the original model, or even better in most cases.

The team is also working on evaluations and a 3B model is also being actively trained which will be released soon. 

Meta’s LLaMA Saga

As Meta’s LLaMA is bound by industrial licences it was not possible to directly distribute LLaMa-based models but so is not the case anymore. Several attempts have been constantly made to open source the models. Open LlaMA is not the first one in the domain. Less than two weeks ago, Hugging Face, the open source AI platform,  released an open-source alternative to, ChatGPT named HuggingChat. The chatbot runs on OpenAssistant’s latest LLaMA based model which provides XOR weights for the OA models. 

Furthermore, Databricks also figured out a way around this with Dolly 2.0. The differentiating factor between other ‘open source’ models and Dolly 2.0 is that it is available for commercial purposes without the need to pay for API access or share data with third parties unlike the rest.

Here are: 14 Open Source LLMs You Need to Know