The Battle for Indian AI Will Be Won on Indic Benchmarks

Ahead of the India AI Impact Summit, the need for AI sovereignty is becoming more apparent. 

AI giants have been releasing models for Indic languages. Google expanded Gemini to support seven Indian languages, OpenAI introduced IndQA to benchmark AI understanding of Indian cultures and languages, and ElevenLabs is working on voice AI for Indian languages. However, it’s difficult to assess them on Western benchmarks.

Several models like GPT, Gemini, and Mistral may look strong on Western tests yet falter the moment they encounter Tamil tables, Bengali newspapers, or Marathi invoices. Much of India’s public record exists as low-quality scans across 22 scheduled languages. Moreover, assessing them on existing benchmarks against Indian competitors like Sarvam and Gnani may be unfair.

Sarvam AI’s solution is a dedicated Indic OCR benchmark. At 3 billion parameters, the vision language model is designed to handle image captioning, scene text recognition, chart interpretation, and complex table parsing. 

Sarvam’s Assessment

On English-heavy global benchmarks such as olmOCR Bench and OmniDocBench, Sarvam’s model keeps pace with or exceeds several larger systems. However, on the Sarvam Indic OCR Bench, accuracy drops sharply for many global models as scripts move away from English and Hindi. In languages such as Telugu, Odia, Kashmiri, and Maithili, the gap widens. 

The benchmark contains 20,267 samples across 22 languages, some dating back to 1800, with varied scan quality. Performance is reported as word accuracy.

Sarvam’s model remains more stable across the board, and its rapport has been rising. Recently, Debarghya ‘Deedy’ Das, Partner at Menlo Ventures, walked back his criticism of the company for focusing on Indic language tasks. He also accepted that there is a need to solve the Indic language problems, something that the West isn’t as capable of.

Trust Deficit

Still, benchmarks built by AI companies raise an obvious question. Can they be trusted? Last year, OpenAI faced scrutiny for its o3 model that would sometimes manipulate its own process to stay on top of the leaderboard in a “reward hacking” behaviour.

Adithya S Kolavi, Founder of open-source AI research lab CognitiveLab, has been building independent evaluation tools and leaderboards for Indic models. He tested Sarvam’s new multimodal AI model, Sarvam Vision. “I tested the model as well, and it's doing pretty good, very, very hard to get that level or performance with a 3 billion model,” he tells AIM.

Kolavi believes there’s a pressing need for shared evaluation. “I think multilingual benchmarks are important now. Not only Indic but Indic + other languages as well. As now Indian models will come out, there will be a proper need for evaluations, either the companies developing the models release them, or there should be an open standard.”

Kolavi had earlier created the Indic LLM Leaderboard as a comprehensive assessment platform for seven Indic languages—Hindi, Kannada, Tamil, Telugu, Malayalam, Marathi, and Gujarati. Hosted on open-source AI community Hugging Face, it currently supports four Indic languages.

The CognitiveLab team is now preparing a toolkit and multilingual benchmark to compare many systems. Synthetic data is ready, real world samples are being hand-annotated. Kolavi plans to test Sarvam’s claims independently. 

The trust issue is central. Benchmarks determine funding, adoption, and credibility, blurring the line between marketing and measurement.

Read: The Dire Need for an Indic LLM Leaderboard

A Glaring Need

Vivekananda Pani, Co-founder and CTO of Indic language-focused Reverie Language Technologies, argues that the ecosystem needs neutral standards. “India-specific benchmark data are essential. With the focus on India and the swarm of models being released, it is hard for anyone to compare performances and benchmark. Benchmark data created by model builders is not an ideal expectation,” he notes.

He adds that even without deliberate manipulation, bias creeps in. “One may not deliberately manipulate benchmarks, but without any standard, neutral and open benchmark process and data available, every organisation, team or even individuals developing models can have inadvertent or deliberate biases.”

Reverie has shared inputs with the Ministry of Electronics and Information Technology and the Central Institute of Indian Languages to help shape broader public datasets. The idea is to treat evaluation as shared infrastructure rather than competitive IP.

Bharath Shankar, Co-founder and Chief Product and Engineering Officer at Gnani.ai, sees the demand for Indic benchmarks rising as AI moves from demos to deployment. “India’s linguistic diversity, code-mixing, dialectal variations, and culturally grounded usage patterns are fundamentally different from Western language contexts. Without benchmarks that reflect real Indian data across speech, text, and multimodal use cases, model evaluations risk being incomplete or misleading.”

For him, the issue is not just domestic models. Global AI systems must also be tested on Indian ground truth. 

“Models often perform well on benchmarks closely aligned with their training data, which can mask gaps in Indic language understanding, accent handling, and contextual nuance. Independent Indic benchmarks create a level playing field, expose performance drop-offs, and push the ecosystem toward building truly inclusive, high-quality AI systems that work at scale for India, not just in English-first or Western settings,” he explains.

Shankar also addresses the credibility question. “Indic benchmarks become reliable not because of who builds them, but because of how they are designed, governed, and validated. Transparency in dataset creation, clear documentation of sources, annotation guidelines, and evaluation metrics are critical.”

He argues that trust comes from process. “When benchmarks are built using diverse, representative data across regions, accents, domains, and socio-economic contexts and are peer-reviewed or audited by independent experts, they reduce the risk of bias or gaming. Open methodologies and reproducible evaluation pipelines make it harder for any single player to manipulate outcomes.”

The solution, he says, is shared ownership. “Credibility increases when benchmarks are maintained by neutral bodies or consortia involving academia, startups, enterprises, and policymakers rather than a single company. Ultimately, trust in Indic benchmarks will come from community adoption and consistent correlation between benchmark scores and real-world performance in Indian deployments.”

Only the Beginning

This push is multipolar. 

Last year, open-source research initiative AI4Bharat, backed by IIT Madras, launched the Indic LLM Arena, a crowd-sourced platform where users compare two anonymous models in Indian languages and vote on which performs better. Instead of static scores, it relies on thousands of human judgements across language, context, and safety. 

There’s work being done on speech, too. Supriya Paul, co-founder and CEO of Josh Talks, says the bigger problem with current speech AI is not accuracy on paper but representation. “Much of today’s speech AI is trained and evaluated using global, largely first-world datasets. As a result, performance assumptions are often shaped by urban, English-forward, and relatively homogeneous speech patterns,” she explains.

“What remains underexamined is how these systems perform when exposed to the full range of Indian speech, including rural and urban variation, strong dialectal differences, code switching, and everyday conversational use,” Paul adds.

Josh Talks, in collaboration with AI4Bharat, is preparing a nationally representative public benchmark for Indian-language automatic speech recognition. Instead of scripted or studio-quality audio, the dataset captures messy, natural speech across regions, accents, and demographic groups.

For Paul, the stakes go beyond research. Benchmarks shape access. If systems fail to understand large sections of the population, digital services quietly become exclusionary.

Indic benchmarks would also create pressure on global players. If Western models want to serve India, they will have to compete on Indian tests, not just English leaderboards.

There is still work left. Independent governance, shared datasets, and transparent protocols remain unresolved. Sarvam’s benchmark is only the beginning.