Article complete

Meta AI's MuAViC Sets New Benchmark for Highly Accurate Speech Translation

MuAViC can deliver superior speech translation in challenging amid noisy environments.

A stylized portrait of a man in a suit against a purple background, with graphic elements resembling a map and question marks

Key takeaways

  • Meta AI has introduced MuAViC, a benchmark that combines audio-visual learning for enhanced accuracy in speech translation, especially in noisy environments.
  • The MuAViC model is built on previous AI frameworks such as AV-HuBERT and RAVen, which incorporate visual data to improve speech recognition.
  • The training of MuAViC addresses the challenges of limited audio-visual training data, showcasing its capability to perform well even with distorted audio inputs.
  • MuAViC is the largest multilingual benchmark for audio-video speech recognition, featuring approximately 1,200 hours of transcribed data across nine languages.

Takeaways drafted with AI from the article text.

Based on their previous AI models such as AV-HuBERT and RAVen models that use visual information to improve English speech recognition, through MuAViC, Meta AI has trained its AV-HuBERT model to deliver superior speech translation in challenging amid noisy environments.

The model can effortlessly handle noise, with the visual modality being relied upon more heavily if the audio modality is distorted. The models were tested in noisy and noise-free environments against a top-performing model for speech recognition and X-En speech translation tasks.

Read the full paper here. 

Training Process

The shortage of adequate training data previously hindered the exploration of audio-visual understanding for speech translation. Compared to audio data alone, gathering and processing audio-video data requires more resources.

The MuAViC is the most extensive multilingual benchmark for audio-video speech recognition because it comprises about 1,200 hours of transcribed data across nine languages.

For speeches in English, the team repurposed the audio-visual data from the LRS3 dataset and aligned it with a machine translation corpus using a text-matching algorithm. They matched the examples with the corresponding target sentences in the machine translation corpus to generate translation labels and employed exact text matching for examples in the development and test sets. For training set examples without matches, they used a machine translation model to obtain pseudo-translation labels.

On the other hand, speeches in non-English languages, Meta used the audio-only data, transcriptions, and text translations from the speech translation dataset. The team obtained video tracks of the original recordings and aligned processed video data with the audio data to create audio-visual data. Although all audio data is transcribed, we only translated a subset of it. To create pseudo-translation labels, they also utilized the same machine translation model as earlier.

The team utilized Meta's AV-HuBERT design to produce speech recognition and speech translation models that process both audio and video data end-to-end. When given paired audio and video inputs, the model combines their representations into a single space that can be used for either task. Even if one modality is absent, the AV-HuBERT can still process the available data, albeit with reduced efficiency. 

Last week, Meta released LLaMA, a set of foundation language models that range from 7B to 65B parameters, but it got leaked, along with its weights and is now available to download through torrents. Christopher King, a GitHub user, submitted a pull request to the LLaMA GitHub page which included a torrent link to the open model. LLaMA-13B surpasses OpenAI’s GPT-3 (175B) while being over ten times smaller, and LLaMA-65B is comparable to DeepMind’s Chinchilla-70B and Google’s PaLM-540B.

Published

Spotted an error? Tell our editors. Read our editorial standards and corrections.

More in this story

Meta AI · 5 stories
  1. What's Meta's Project CAIRaoke?
  2. Meta AI Chief Yann LeCun’s Self-Supervised Idea Finally Sees Open Source Light 
  3. Meta AI Releases A Multimodal Model "CM3leon"  — But Won’t Release It
  4. You are here: Meta AI's MuAViC Sets New Benchmark for Highly Accurate Speech Translation
  5. Llama 3 Might Not be Open Source
About the author

Contributor

All stories
Belamy · Daily

The day’s AI news, before your first meeting.