An Indic LLM leaderboard is finally here. CognitiveLab has released its Indic LLM Leaderboard for the growing number of Indic Language Models entering the scene without a uniform evaluation framework. 

The Indic LLM Leaderboard offers support for 7 Indic languages including Hindi, Kannada, Tamil, Telugu, Malayalam, Marathi, and Gujarati, providing a comprehensive assessment platform. Hosted on Hugging Face, it supports 4 Indic benchmarks initially, with plans for additional benchmarks in the future. 

Click here to check it out.

Along with this, Adithya S Kolavi, the founder of CognitiveLab has also unveiled the indic_eval evaluation framework which supports Arc Easy, Challenge, Hellaswag, MMLU, BoolQ, and Translation.

The leaderboard also seamlessly integrates with indic_eval, simplifying the process of uploading evaluation scores.

This entire system is deployed within India, ensuring robust security measures.

As an alpha release, the Leaderboard promises ongoing enhancements and tested features in subsequent updates. As of this release, base models ‘meta-llama/Llama-2-7b-hf’ and ‘google/gemma-7b’ have been added into the leaderboard to use as reference.

With a commitment to bolstering its capabilities, CognitiveLab aims to establish the Indic LLM Leaderboard as a pivotal tool for evaluating and advancing Indic Language Models.

The leaderboard operates by executing indic_eval on the chosen model, then transmitting the outcomes to a server for storage in a database. The Frontend Leaderboard subsequently accesses this server to retrieve the most recent models from the database, alongside their respective benchmarks and metadata. 

In contrast to the Open LLM leaderboard, this project draws inspiration from it but introduces standardized evaluation with common benchmarks due to computational resource limitations. Users can conduct evaluations on their GPUs, while the leaderboard acts as a centralized platform for model comparisons. 

To ensure reliability and consistency in the output, the company employed indictrans2 from AI4Bharat and other translation APIs to translate the benchmarking dataset into seven Indian languages.

In March, CognitiveLab introduced Ambari, an open-source Bilingual Kannada-English LLMs series. The initiative addresses the challenges posed by the dynamic landscape of LLMs, with a primary focus on bridging the linguistic gap between Kannada and English.