AI4Bharat, Josh Talks AI Design Real-World Multimodal AI Evaluation Platform

Research lab AI4Bharat at IIT Madras and media edtech platform Josh Talks AI have launched Voice of India, an independent multimodal AI evaluation platform that will benchmark AI systems against real-world Indian languages, accents, dialects, and deployment environments.

The platform targets governments, enterprises, researchers, and AI developers to independently assess how AI models perform in India rather than relying only on vendor-reported benchmarks or evaluation standards developed for Western markets.

The platform features a suite of benchmarks, including a finance-focused benchmark for evaluating AI systems across banking workflows. Over time, it plans to expand evaluations across voice AI, large language models (LLMs), OCR and document processing, safety and cultural assessment, and Indian legal reasoning.

For enterprises and government agencies deploying AI, the platform introduces a three-stage evaluation process comprising constituent model evaluation, end-to-end deployment testing under real operating conditions, and periodic post-deployment audits. The framework aims to measure how AI systems perform in production environments rather than only in controlled benchmark settings.

“Evaluation determines what gets built. With AI systems becoming more deeply integrated into society, rigorous evaluation becomes as important as model development itself,” said Prof Mitesh Khapra of AI4Bharat at IIT Madras in a statement.

He added that Voice of India aims to provide “scientifically robust evaluation frameworks that reflect India's linguistic diversity,” and said India has an opportunity “not just to build AI for the world, but to help define how the world measures it.”

The platform will benchmark technologies including automatic speech recognition (ASR), text-to-speech (TTS), end-to-end voice agents, and speech translation across sectors such as finance, agriculture, healthcare, and government services. It will also evaluate LLMs on text reasoning, instruction following in Indian languages, retrieval-grounded search, long-document processing, and translation involving code-mixed languages.

Additional evaluation areas include OCR accuracy for Indic scripts and multilingual documents, safety and bias assessment across Indian languages, and legal reasoning involving statutes, contracts, and case law.

Supriya Paul, Co-founder of Josh Talks AI, said trust in AI will depend on how well systems understand the people they serve rather than only on their intelligence.

“India has a once-in-a-generation opportunity to contribute something unique to global AI, not necessarily the biggest models, but the evaluation infrastructure that makes AI trustworthy for the world,” she said.

Shobhit Banga, Co-founder of Josh Talks AI, said the focus of the next phase of AI development should shift towards measuring real-world performance.

“The next frontier in AI isn't building larger models. It’s building better ways to measure whether they actually work,” Banga said. "The Voice of India AI Evaluation Platform aims to bridge that gap by creating independent, transparent evaluation infrastructure that helps AI builders understand where models succeed, where they fail, and why.”