Humyn Labs Highlights Multilingual AI Speech Recognition Gaps With New Benchmark Report

Humyn Labs, a physical and voice AI data infrastructure company, has launched BRIDGE (Benchmark of Regional & International Data for Global Evaluation), an independent benchmarking report that evaluates the performance of AI speech recognition systems across Indian and Global South languages. 

The company claims it is the largest benchmark of its kind, covering 15 commercial AI models across more than 22 non-English languages and dialects spoken by over 5.5 billion people globally.

The report evaluated popular speech recognition tools from companies including OpenAI, Google, Amazon, Microsoft, Deepgram, ElevenLabs, and Indian AI providers, Sarvam AI and Gnani.ai.

According to the findings in the report, many widely used AI speech tools struggle to accurately recognise Indian language conversations, with some systems reportedly mishearing one in every three words. The report noted that traditional industry metrics such as Word Error Rate (WER) fail to fully capture how AI systems perform on multilingual and code-switched speech common in countries like India.

Manish Agarwal, Co-founder of Humyn Labs, said, “The models are grading their own work. ASR providers published their own accuracy scores using benchmarks built on English-first, internet-trained datasets with little independent validation.”

He added that enterprises have been making “million-dollar deployment decisions” based on benchmarks that often do not reflect how users in the Global South actually speak.

Unlike conventional benchmarks, BRIDGE uses a seven-metric evaluation system that includes Semantic Similarity, Code-Switch F1, Loan Word WER, Phoneme-Informed Error Rate, and Word Information Lost alongside standard WER and Character Error Rate.

The report found that handling code-switched conversations where speakers mix English with Indian languages mid-sentence remains a major challenge for AI systems. Deepgram Nova-3 recorded the highest Code-Switch F1 score of 0.906 while several other global models scored significantly lower.

“The models aren’t the only problem, but the metrics are. You cannot evaluate non-English speech with a scoring system designed for English phonology and call it rigorous,” said Ishank Gupta, Co-founder of Humyn Labs.

The benchmark was built using human-verified, field-collected conversational audio from 22 Indian states rather than scripted or internet-scraped speech data.