Beyond Text-to-Speech: Inside the Tech Making AI Conversations Sound More Human

B2C companies, especially banks and other financial institutions, love to use Interactive Voice Response (IVR) technologies to process a high volume of customer calls. These drastically reduce operational costs, enabling them to offer 24/7 service without additional staffing.

But not everyone’s a fan. A New York University study found that 83% of consumers said IVRs provide them with no benefit at all. That’s because voice technologies often sound robotic and fail to grasp user intent.

Voice AI systems originally relied on a straightforward pipeline: speech recognition converted spoken words into text, a basic natural language processor interpreted the text, and text‑to‑speech converted the response back into audio. This model works well for simple queries such as account balances, ticket bookings, or password resets.

However, users have been noticing its limitations more than ever. Responses often sound robotic, lack contextual understanding, and fail to handle complex or ambiguous queries. Customers recognise that they have been speaking to a machine, and many ask for human agents to resolve issues that require empathy or nuanced reasoning.

“Voice is the natural communication method for humans. For decades, powering personalised, human-like voice experiences was impossible, with most modern technologies taking the easy way out: chat and text," Yazan El-Baba, Partner at Emergence Capital, tells AIM.

Even OpenAI struggles to process natural language with the speech-to-speech GPT-4o Realtime API launched in 2024, with users online flagging audio overlap, inability to maintain voice continuity, and latency, resulting in a robotic sound. 

"OpenAI's speech-to-speech preview is impressive. But it still has the same flaw every cascaded system has had for a decade. It waits for you to finish your sentence. And in that wait, calls drop, cues get missed, and the conversation stops feeling like one. Humans don't act like that. Humans are listening, reasoning, predicting, and responding all at once," Sudarshan Kamath, Founder and CEO of Bengaluru-based AI research lab smallest.ai, notes.

Firms are closing in on this gap by building LLM models that interact just like humans. They are designing AI agents to handle multi‑turn conversations, understand intent, and adapt tone, providing customers with personalised answers over scripted responses. 

According to a recent study published in Nature, modular pipelines—often in the form of RAG (Retrieval-Augmented Generation) or multi-agent systems—outperform monolithic LLMs, offering higher accuracy and flexibility, especially in multilingual contexts such as India, where regional language support is critical. This is especially relevant as the Indian conversational AI market is set to grow at a compound annual growth rate of 26% over the next five years, reaching $1.85 billion by 2030, according to Grand View Research.

Hydra AI vs Hunar.ai: A Case Study

Last month, smallest.ai beta-launched a speech-to-speech AI model called Hydra. It uses a unified speech‑to‑speech pipeline, enabling sub‑300ms latency, emotional fidelity, and true full‑duplex conversations. 

According to smallest.ai, Hydra preserves acoustic signals like urgency and warmth, making exchanges more human‑like. This differs from OpenAI’s cascaded model, which stitches recognition, language, and synthesis layers. For enterprise use cases such as debt collection, recruitment, or healthcare intake, Hydra’s low latency and conversational naturalness improve completion rates, the startup claims.

Meanwhile, Hunar.ai, a voice AI company, includes a hybrid approach to input and output. 

On the input side, it begins with audio rather than text, applying extensive pre‑processing before moving into audio‑aware inference. On the output side, the inference engine produces text, which is then converted into speech through a text‑to‑speech layer. This layer can switch providers during a call depending on the language being spoken, ensuring fluid multilingual conversations.

The process starts with raw audio from the user, which is handled by the Dynamic Config Generator (DCG). This patented component detects the language being spoken—whether Hindi, English, Tamil, Telugu, or Malayalam—and identifies filler words specific to each dialect. It interprets pauses in context, distinguishes backchannel acknowledgements such as "hmm" or "haan ji", and filters out irrelevant segments, such as background noise or repetitive acknowledgements. 

"By reducing the audio to only the semantically relevant parts, DCG preserves cues that matter for turn‑taking, latency control, and user intent. It produces a structured audio representation that the inference model can process reliably across languages and code‑mixed speech," says Krishna Khandelwal, Founder of Hunar.ai.

The cleaned audio is then fed into a voice‑aware inference engine. Unlike conventional text‑only models, this engine retains intonation, timing, and spoken behaviour while removing fillers. Because DCG has already normalised pause patterns and acknowledgements, the model avoids unnecessary interruptions and responds only when required. It is optimised not just for content but also for context, recognising whether a statement is an answer, a complaint, a hesitation, or simply an acknowledgement.

"On the output side, the system uses multiple text‑to‑speech engines, each strong in particular languages. Some excel in Indian English and Hindi, while others are better at Tamil, Telugu, or Malayalam. Because DCG continuously tracks the language being spoken, the system can switch providers mid‑call without latency," says Khandelwal.

This supports natural code‑switching, such as English mixed with Tamil or Hindi mixed with English, within the same conversation. The agent, therefore, avoids sounding locked into a single language, which is crucial for use cases like HR screening.

ARM and Its Cost

Audio regenerative models (ARMs) feature a significant improvement over the traditional text-to-speech models by focusing on rebuilding, enhancing, and editing existing audio rather than just generating new audio from scratch. They are adept at managing interruptions and context recomposition. 

In long conversations, users often cut in, change their mind, or add clarifications mid‑sentence. ARM freezes the agent's current response trajectory, captures the new input, and merges it with the pre‑interruption context. It then regenerates a coherent response plan, much like a human adjusting mid‑thought when interrupted. This plan is passed to the text‑to‑speech layer, making the system feel natural and adaptive.

This design choice avoids the poor quality of direct audio generation from language models, reduces costs since text tokens are cheaper than audio tokens, and allows flexibility to swap in better text‑to‑speech providers. 

Raoul Nanavati, CEO of B2B voice AI startup Navana.ai, notes that the infrastructure required to overcome these barriers is also fundamentally different. “Automatic speech recognition, text-to-speech, dialogue management, core system integrations, authentication, compliance guardrails, and human handoff logic need to all work together under two seconds of latency,” he says.

Many Big Tech companies have already deployed ARMs. Meta’s MusicGen and AudioGen use transformer models for audio synthesis from text prompts. NVIDIA uses BigVGAN and Audio2Face to generate facial animation from voice input in gaming. Meanwhile, ElevenLabs creates entirely new synthetic voices from scratch, allowing users to define age, gender, and accent.

However, that comes at a great computational cost. 

According to research from Sorbonne University, training ARMs, especially those using raw waveform modelling, can cost between $50,000 and $500,000. Another research pegs the service charge at $0.008 per minute of speech. 

For India, the math is different. A startup can deploy a fully functional voice agent handling customer queries for approximately ₹5 to ₹15 per minute of conversation. For comparison, a single part-time call centre representative costs between ₹10,000 and ₹15,000 per month and handles a fraction of the concurrent call volume, as per data from no-code voice AI platform Vomyra.

But the true cost of voice AI is not just in the API bill. It includes the hidden toll of poor performance. Over 61% of consumers report dissatisfaction with traditional automated phone systems, and more than half abandon calls due to poor experiences, potentially costing companies up to $262 per customer annually, according to VoiceAIWrapper.

For businesses operating in collections, insurance, or BFSI, where every call is a revenue opportunity, that leakage is significant. 

Architectures like the hybrid audio-in, text-out model that Hunar.ai uses are, in part, a cost play: text token generation is substantially cheaper than end-to-end audio synthesis, and the ability to swap text-to-speech providers mid-call allows businesses to optimise for both quality and spend simultaneously.

The next step is full agentic workflows: multi-step processes that a voice agent runs autonomously across systems, without a human in the loop.