AI4Bharat Wants to Test Every AI Claim India Makes

AI4Bharat and Josh Talks AI recently launched Voice of India, an independent multimodal evaluation platform designed to test AI models across Indian languages, accents, dialects, and real-world deployment scenarios. 

Instead of relying on English language benchmarks or vendor-reported numbers, governments and enterprises can compare models based on how they perform in Indian banking, healthcare, agriculture, and public services.

That sounds like another benchmarking platform. However, Mitesh Khapra, Associate Professor at IIT Madras and Head of the research lab AI4Bharat, believes it is much bigger than that.

“Evaluation determines what gets built,” he tells AIM. “With AI systems becoming more deeply integrated into society, rigorous evaluation becomes as important as model development itself.”

The statement quietly challenges the direction India’s AI conversation has taken over the past year.

The AI debate in India has centred around GPUs, sovereign models, compute capacity, and who can build the country’s equivalent of Mythos. Evaluation has barely featured in the discussion. Yet the success of every government procurement, enterprise deployment, and AI assistant ultimately depends on one question: do people actually find the model useful?

Khapra argues that India has reached the point where building models is no longer the biggest bottleneck. “We've got the data for training the models,” he says. “Now we have to focus on the output problem, which is evaluating AI models.”

That distinction is important.

The Need for Benchmarks

AI4Bharat, founded in 2019 by Khapra along with Pratyush Kumar from Sarvam, has spent years building datasets and multilingual models. Last month, its researchers published FOCUS, a benchmark that exposed another uncomfortable truth. Vision-language models acting as AI evaluators for other models often fail to identify degraded outputs.

Another evaluation spanning automatic speech recognition systems across 15 languages and more than 35,000 speakers revealed sharp accuracy disparities across dialects, demographics, and everyday conversational settings.

OpenAI transcription models recorded word error rates of over 55% on Indian speech. In languages such as Maithili and Tamil, the models fail to transcribe nearly two of every three words correctly.

Now, the organisation is extending that thinking beyond research papers and into deployment.

Instead of asking whether one model scores better than another on a generic leaderboard, Voice of India asks whether a voice agent understands Bhojpuri inside a noisy bank branch, whether OCR works on multilingual government documents, or whether a legal assistant reasons correctly over Indian statutes.

The shift is subtle but significant.

Western benchmarks largely measure intelligence. Voice of India is attempting to measure usefulness. 

That is exactly why Josh Talks AI entered the picture.

“The first phase of AI was about intelligence,” said co-founder Supriya Paul. "The second phase was about deployment. Now, we're talking about responsibility, inclusivity, safety, and eventually, trust."

Paul believes the industry has built sophisticated models without building an independent way to verify vendor claims. "There is no infrastructure that supports the claims vendors make about how good they are."

For enterprises, that gap is becoming expensive.

An AI system claiming 95% accuracy inside a lab can fail spectacularly once deployed. Paul shared one example where a voice model repeatedly interpreted “pandrah (15) lakh” as “pachaas (50) lakh” during a financial conversation. In another healthcare example, the Hindi word ‘mahamaari’ (epidemic) was transcribed as Mahavir.

On benchmark leaderboards, those systems still look impressive.

However, for someone trying to collect a loan payment or seek medical advice, they become unusable. Voice of India is trying to expose those failures before deployment.

Broader Ambition

The platform does not stop at testing individual models. It evaluates complete enterprise deployments and proposes continuous post-deployment audits rather than treating benchmarking as a one-time exercise.

Khapra says India has an opportunity to build something that the US and China currently do not possess. "I don't think evaluations at this scale have been done anywhere in the world for this diverse a demographic, these many languages or these many use cases."

His vision is for India to become "the blueprint" for AI evaluation globally.

Interestingly, neither AI4Bharat nor Josh Talks AI believe Indian models should automatically win these evaluations. "The benchmark should show you the mirror," Khapra adds.

"If there is an exam, everyone prepares." That philosophy also shapes how the platform intends to remain independent.

Josh Talks AI already licenses speech datasets to AI labs, raising an obvious question about conflicts of interest. Co-founder Shobhit Banga insists the benchmark will remain free for every lab to participate in and that datasets are licensed under identical terms regardless of who buys them.

"The public evaluations are not paid for. You cannot pay to be evaluated." More importantly, Banga argues that labs themselves want this feedback.

"The next frontier in AI isn't building larger models. It's building better ways to measure whether they actually work," he reiterates.

According to him, most AI companies simply do not know where their models fail across India's thousands of dialects and regional variations. "The labs are eagerly waiting to improve on those exams." 

Banga reveals that one global AI laboratory told them its benchmark report was detailed enough to be published as a research paper.

It is a reminder that evaluation is becoming a research discipline of its own rather than a simple leaderboard.

The country has spent years building digital public infrastructure through Aadhaar, UPI, and ONDC. Khapra sees AI evaluation following a similar trajectory, becoming shared infrastructure that everyone builds on rather than another proprietary model competing for market share.

If that happens, India's contribution to global AI may not be another frontier model. It could be the yardstick every frontier model has to pass. And that is perhaps the biggest shift in thinking.