Hugging Face Launches Open LLM Leaderboard, Chinese Models Dominate

Hugging Face, the AI community platform, has unveiled a brand new open large language model (LLM) leaderboard, with Chinese models taking the top spots, according to an announcement today from co-founder and CEO Clem Delangue.

The leaderboard ranks open-source LLMs based on extensive new evaluations, including the MMLU-pro benchmark, which tests models on high school and college-level problems.

Hugging Face utilised 300 NVIDIA H100 GPUs to re-evaluate all major open LLMs for the updated rankings.

Alibaba's Qwen-72B model emerged as the top performer overall, outpacing other open-source models, highlighting the rapid progress of Chinese AI companies in the LLM space.

The dominance of Chinese models on the leaderboard underscores the increasingly competitive and global nature of the open-source AI ecosystem.

Delangue noted that previous LLM benchmarks have become "too easy" for the latest models, comparing it to "grading high school students on middle school problems." This suggests the need for more rigorous and challenging evaluations as open-source language models grow more sophisticated.

However, Delangue also cautioned that some AI developers may be overly focused on optimising for specific benchmarks at the expense of well-rounded model performance. The leaderboard results also indicated that simply increasing model size does not always translate to superior performance.

The launch of Hugging Face's new open LLM leaderboard marks an important step in the field's maturation in terms of transparent and comprehensive evaluation. With Chinese models leading the pack, the leaderboard will likely spur further innovation and investment in open-source AI technologies worldwide.

Hugging Face, founded in 2016, has become a central hub for open-source machine learning, hosting over 250,000 models and datasets used by a community of 200,000 developers.