When it comes to building Indic language models, Tokenization has been one of the most discussed topics as it is very different for each model. Looking at this, Cognitive Lab has introduced Tokenizer Arena on Hugging Face.
The arena lets users compare different tokenizers simultaneously and is built on top of the transformer js library.
The arena hosts several models such as Gemma, Mistral, all versions of Llama, GPT-3, GPT-4, Grok-1, Claude, Phi-3, and Command R.
This is ideal for developers who are trying to build Indic LLM models on top of existing open source models for tokenizing on devanagari text, which is very different from English language.
Recently, Google Researchers from India created the IndicGenBench dataset for multilingual benchmarking of different LLMs on 29 Indic languages, 13 scripts, and 4 language families.
Adithya S Kolavi, the founder of Cognitive Lab recently created the Indic LLM Leaderboard for measuring different Indic LLMs rising in the country. The team also released Ambari, the first bilingual Kannada model built on top of Llama 2.
The Indic LLM Leaderboard offers support for 7 Indic languages, including Hindi, Kannada, Tamil, Telugu, Malayalam, Marathi, and Gujarati, providing a comprehensive assessment platform. Hosted on Hugging Face, it initially supports 4 Indic benchmarks, with plans for additional benchmarks in the future.
