Ai2’s New Language Models Compete Directly with Qwen and Llama

The Allen Institute for AI (Ai2), founded by late Microsoft founder Paul Allen, has released a new family of language models, OLMo 2, to improve natural language understanding and generation. These models were recently updated on Hugging Face.

The models, available in both 7 billion (OLMo-2-1124-7B) and 13 billion (OLMo-2-1124-13B) parameter versions, are trained on a wide range of datasets to help improve the performance of AI systems in various applications, from scientific research to customer service.

Ai2 announced the development on X saying, “Meet OLMo 2, the best fully open language model to date, including a family of 7B and 13B models trained up to 5T tokens.”

Meet OLMo 2, the best fully open language model to date, including a family of 7B and 13B models trained up to 5T tokens. OLMo 2 outperforms other fully open models and competes with open-weight models like Llama 3.1 8B — As always, we released our data, code, recipes and more 🎁 pic.twitter.com/YQ7z8W9lE6

— Ai2 (@allen_ai) November 26, 2024

These new OLMo 2 models are trained on OLMo-mix-1124 and Dolmino-mix-1124 datasets, which gives them an advantage over the original OLMo 7B model. 

Ai2 has also released instruction-tuned versions of the OLMo 2 models. These ‘instruct’ models utilise the Tülu 3 dataset to perform better on tasks that require understanding and following specific instructions. 

OLMo-2-1124-7B-Instruct and OLMo-2-1124-13B-Instruct are optimised to handle more structured and goal-oriented tasks. These models have shown significant improvements in benchmarks such as MATH, GSM8K, and IFEval, displaying the ability to tackle complex reasoning tasks and respond more effectively to inputs.

One of the key features of the OLMo 2 release is its emphasis on open access. Ai2 has made the models and training data publicly available, including all code and intermediate checkpoints, under the Apache 2.0 license. 

This openness is in line with Ai2’s commitment to fostering transparency and reproducibility in AI research. By providing access to the full model architecture, training processes, and evaluation results, Ai2 claims to enable other researchers to build upon their work and contribute to further advancements in the field of language modelling.

OLMo 2 Outperforms Qwen and Llama

The OLMo 2 models were trained using up to 5 trillion tokens, a vast amount of text data which has enabled them to achieve high performance in multiple natural language processing tasks. 

Applying our state-of-the-art Tülu 3 post-training recipe, we also built OLMo 2 Instruct, which are competitive with even the best open-weight models—OLMo 2 13B Instruct outperforms Qwen 2.5 14B instruct, Tülu 3 8B, and Llama 3.1 8B Instruct models. pic.twitter.com/VIuWW56O2L

— Ai2 (@allen_ai) November 26, 2024

These models compete directly with leading frontier models such as Qwen 2.5, Llama 3.1, Mistral NeMo and Gemma 2. 

In terms of performance, OLMo 2 has achieved notable results across various benchmarks. For instance, the OLMo 2 7B Instruct model has demonstrated strong performance on tasks like GSM8K and MATH, indicating its proficiency in mathematical reasoning.  

Similarly, the OLMo 2 13B Instruct model has shown competitive results on benchmarks like GSM8K and IFEval, demonstrating its ability to handle diverse tasks. 

As reported by AIM in the past, Ai2’s Macaw also competed directly with OpenAI’s GPT-3.

Most of the performance metrics and benchmarks now suggest that OLMo 2 is at par with these frontier models. Its open-weight availability and comprehensive training approach position it as a strong contender in the landscape of advanced language models.