AI4Bharat, Josh Talks Benchmark Flags 55% Error Rate for OpenAI in Indian Speech

A new national benchmark for speech recognition in India has revealed significant performance gaps in leading global AI systems when tested on real-world Indian speech. 

The benchmark, titled ‘Voice of India’, has been launched by Josh Talks in collaboration with AI4Bharat. It evaluates automatic speech recognition systems across 15 languages and more than 35,000 speakers, revealing sharp accuracy disparities across dialects, demographics, and everyday conversational settings.

OpenAI transcription models record word error rates of over 55% on Indian speech. In languages such as Maithili and Tamil, the models fail to transcribe nearly two of every three words correctly. 

In Urdu, OpenAI models post a 35.4% word error rate, compared to 6.95% for Sarvam Audio.

In dialects like Bhojpuri and Maithili, some global models fail to transcribe two of every three words correctly. Bhojpuri alone has more than 50 million speakers, yet remains among the most challenging dialects for AI systems. 

Meta’s performance in Tamil and Malayalam shows error rates often two to three times higher than Indian systems. Microsoft’s speech-to-text system doesn’t support six of 15 languages tested, including Punjabi, Odia, and Kannada. 

Overall, Microsoft STT is not supported for nearly half the benchmark languages.

The benchmark also highlights what it terms the ‘OpenAI Gap’. While Google Gemini remains competitive with Indian systems, OpenAI’s GPT-4o models trail Sarvam by over 50% points in overall average accuracy. 

Sarvam Audio ranks first or second across nearly every language and dialect tested, including Hindi, Bengali, Odia, and Assamese, claiming 93% or higher accuracy in key regional dialects where global models falter. Google Gemini emerges as the leading global contender, maintaining high performance parity with local systems.

The benchmark also finds limited efficiency gains in some global models. Meta’s 7B parameter model is only around 4% more accurate than its smaller 1B parameter model on average across Indian languages.

Voice of India evaluates conversational speech collected from around 2,000 speakers per language. The dataset spans age groups, genders, regions, socio-economic backgrounds, device types, and acoustic environments. 

It explicitly includes code-switched speech such as Hindi-English, Tamil-English, and Urdu-Hindi, along with background noise and informal speaking styles common in everyday use.

Professor Mitesh Khapra of AI4Bharat at IIT Madras said, “This is one of the most rigorous large-scale evaluations of speech recognition for Indian languages, containing district-level cohorts with balanced representation across gender and age to truly reflect India’s diversity,” he said, adding that the intensive effort sets a new benchmark for fair and representative ASR evaluation in India.

Meanwhile, Shobhit Banga, Co-Founder of Josh Talks, said, “The data shows that when we build AI that understands the soul of Indian speech, our dialects, our accents, and our rural context, we can unlock a level of digital inclusion that was previously unimaginable.”

Notably, in a previous conversation AIM, Supriya Paul, co-founder and CEO of Josh Talks, has explained that the bigger problem with current speech AI is not accuracy on paper but representation. “Much of today’s speech AI is trained and evaluated using global, largely first-world datasets. As a result, performance assumptions are often shaped by urban, English-forward, and relatively homogeneous speech patterns.”

“What remains underexamined is how these systems perform when exposed to the full range of Indian speech, including rural and urban variation, strong dialectal differences, code switching, and everyday conversational use,” Paul added.

Read: The Battle for Indian AI Will Be Won on Indic Benchmarks