Indian-origin researcher Prady Modukuru has made a remarkable tool that effortlessly translates any video into the language of your choice. All you need is a YouTube link!
Modukuru posted a video on Twitter translating a conversation between Meta CEO Zuckerberg and Lex Fridman into Hindi. Here is a glimpse of the sorcery that's unfolding.
A serial entrepreneur, Modukuru is currently working on a stealth startup. Some of his previous ventures include marketing startup SŪNYA and mental wellness startup Oneself (inactive). He has previously worked with companies like Microsoft, Cisco, and Qknow among others.
How it works?
Modukuru leveraged the power of GPT-4 and prompt engineering techniques for seamless translation. To ensure accurate voice representation in different languages while preserving the original speaker's voice, he collaborated with ElevenLabs for voice training and text-to-speech capabilities.
In order to achieve precise lip syncing, Modukuru utilised wav2lip-2, which harnesses AI for generating lifelike lip movements. This breakthrough in the AI industry opens up new possibilities, allowing users to easily comprehend informative content in their preferred language.
In the near future, this technology can revolutionise dubbing, and can be used by major OTT platforms to translate content including series and movies to any other language.
- Amazon Warns Engineers Against AI Productivity Traps in Latest Memo
- Gnani AI Launches Prisma v3.0 Speech Recognition Model for 10 Languages
- Venttup Raises $3 Million To Expand Precision Manufacturing
- Articul8 Launches Bivia To Help Enterprise AI Agents Make Decisions
- NVIDIA Launches Open-Weight Transcription Model for 40 Languages
- Sarvam AI Rolls Out Indus Cowork to 2,500 Maharashtra Govt Officials