Article complete

Cognitive Lab Introduces Tokenizer Arena for Devanagari Text

The arena hosts several models such as Gemma, Mistral, all versions of Llama, GPT-3, GPT-4, Grok-1, Claude, Phi-3, and Command R.

Hands holding a ruler and pencil, with a chalkboard showing Devanagari text and labels for Gemma and LLaMA 3.

Key takeaways

  • Cognitive Lab launches Tokenizer Arena for comparing tokenizers tailored for Devanagari text.
  • Arena features multiple models including GPT-3, GPT-4, and Llama variants for enhanced tokenization.
  • Supports developers building Indic language models, addressing unique tokenization challenges of non-English scripts.
  • Indic LLM Leaderboard tracks performance across seven Indic languages, fostering advancements in AI for regional languages.
  • Google's IndicGenBench dataset aids multilingual benchmarking of LLMs across 29 Indic languages and scripts.

Takeaways drafted with AI from the article text.

When it comes to building Indic language models, Tokenization has been one of the most discussed topics as it is very different for each model. Looking at this, Cognitive Lab has introduced Tokenizer Arena on Hugging Face.

The arena lets users compare different tokenizers simultaneously and is built on top of the transformer js library. 

https://twitter.com/adithya_s_k/status/1788670846136930506

The arena hosts several models such as Gemma, Mistral, all versions of Llama, GPT-3, GPT-4, Grok-1, Claude, Phi-3, and Command R. 

This is ideal for developers who are trying to build Indic LLM models on top of existing open source models for tokenizing on devanagari text, which is very different from English language.

Recently, Google Researchers from India created the IndicGenBench dataset for multilingual benchmarking of different LLMs on 29 Indic languages, 13 scripts, and 4 language families. 

Adithya S Kolavi, the founder of Cognitive Lab recently created the Indic LLM Leaderboard for measuring different Indic LLMs rising in the country. The team also released Ambari, the first bilingual Kannada model built on top of Llama 2.

The Indic LLM Leaderboard offers support for 7 Indic languages, including Hindi, Kannada, Tamil, Telugu, Malayalam, Marathi, and Gujarati, providing a comprehensive assessment platform. Hosted on Hugging Face, it initially supports 4 Indic benchmarks, with plans for additional benchmarks in the future.

Published

Spotted an error? Tell our editors. Read our editorial standards and corrections.

More in this story

cognitive computing human capital · 5 stories
  1. Mark vs Musk: Tech titans lock horns over fear of dystopian views of AI
  2. Companies Must Adopt AI And VR; Or They Might Lag Behind The Competition, Says Sameet Gupte, CEO, Servion Global
  3. Flashback 2017: All The News Stories About Analytics And AI That Made Headlines In India
  4. State of Enterprise AI In India 2019 | Analytics India Magazine & BRIDGEi2i
  5. You are here: Cognitive Lab Introduces Tokenizer Arena for Devanagari Text
About the author

Journalist

Mohit writes about AI in simple, explainable, and often funny words. He's especially passionate about chatting with those building AI for Bharat, with the occasional detour into AGI.

All stories
Belamy · Daily

The day’s AI news, before your first meeting.