Now the voice dubbing industry is being disrupted by AI

The Israeli startup DeepDub claims that it can dub films, television shows, and video games while retaining the pitch and tone of the original actors in any given process. This allows them to recreate the voices of famous actors in other languages, as well as provide an efficient alternative to hiring voice actors to dub individual films or television shows. 

Another company, the UK-based Synthesia, is using AI-based video-editing software to improve the quality of dubbing by tweaking the actor’s mouth movements to fit the local language they are speaking. 

Meanwhile, WellSaid Labs—a Seattle-based startup—is offering AI voices with different voice personalities (such as “energetic and insightful” or “poised and expressive”) to its clients.

Subscribe to our Newsletter

Join our editors every weekday evening as they steer you through the most significant news of the day, introduce you to fresh perspectives, and provide unexpected moments of joy
Your newsletter subscriptions are subject to AIM Privacy Policy and Terms and Conditions.

What’s the technology behind these products?

While synthetic voices have been around for a while, recent breakthroughs in deep learning have made replicating the subtleties of human speech easier than ever. 

Previously, synthetic voices were just words glued together to produce a clunky, robotic effect. Nowadays, voice developers no longer need to provide the exact pacing, pronunciation, or intonation of the generated speech themselves. Instead, they simply fill hours of audio into an algorithm that deciphers those patterns for itself. These voices can change in style and emotion, take pauses and breathe in the correct places, and learn the tone and patterns of the speech of an actor. 

The voice engine constructed by WellSaid consists of two primary deep-learning models: the first predicts what a speaker will sound like from a passage of text, and the second fills in the details—including breaths and the way the voice would resonate in a particular environment. 

Traditionally, dubbing is done without making any alterations to the source video—which means that local script writers have to struggle to make the dialogue match the action taking place on the screen, and voice actors have to deliver their dialogue perfectly to make sure the dub doesn’t feel off. The time to write these scripts and to dub the movie is significantly shorter when artificial intelligence is used to simply change the lip movement of the actor post production (which also makes the scene more realistic to a viewer). While Synthesia’s deep learning algorithm has perfected making it look like an actor’s mouth is forming foreign words, it still isn’t capable of manufacturing the facial idiosyncrasies of individual actors—which would likely change depending on the language the actor is speaking in. 

Why does this innovation matter? 

AI voices are increasing in popularity amongst brands who want to maintain a consistent sound in millions of interactions with customers. Its popularity has also increased due to the omnipresence of smart speakers, automated service, and digital assistants today—that require brands to produce more than a hundred hours of audio every month. 

It’s also getting increasingly vital for film and television producers to take their products to international markets. China’s influence on the global box office has grown exponentially in recent years; in 2020, a vast majority of Netflix’s subscriber gains were from outside the United States; and one-third of Disney+ subscribers are in India. 

Traditional dubbing is an expensive and time-consuming process, and AI voices provide a cheap and scalable alternative to them. With DeepDub, an eight-episode television series—that would ordinarily take 14-16 weeks to dub—can be dubbed in just six weeks. This quicker turnaround for dubbed content could also help reduce piracy—which often occurs because of the time window between content released in the United States and other regions. The bigger the time window is, the greater the threat of piracy. 

Srishti Mukherjee
Drowned in reading sci-fi, fantasy, and classics in equal measure; Srishti carries her bond with literature head-on into the world of science and tech, learning and writing about the fascinating possibilities in the fields of artificial intelligence and machine learning. Making hyperrealistic paintings of her dog Pickle and going through succession memes are her ideas of fun.

Download our Mobile App

MachineHack | AI Hackathons, Coding & Learning

Host Hackathons & Recruit Great Data Talent!

AIMResearch Pioneering advanced AI market research

With a decade of experience under our belt, we are transforming how businesses use AI & data-driven insights to succeed.

The Gold Standard for Recognizing Excellence in Data Science and Tech Workplaces

With Best Firm Certification, you can effortlessly delve into the minds of your employees, unveil invaluable perspectives, and gain distinguished acclaim for fostering an exceptional company culture.

AIM Leaders Council

World’s Biggest Community Exclusively For Senior Executives In Data Science And Analytics.

3 Ways to Join our Community

Telegram group

Discover special offers, top stories, upcoming events, and more.

Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

Subscribe to our Daily newsletter

Get our daily awesome stories & videos in your inbox