Article complete

Google Used Fréchet Distance To Assess AI-Generated Audio & Video Quality

A close-up view of numerous white Buddha statues arranged closely together, creating a serene and repetitive pattern.

Key takeaways

  • Artificial Neural Networks have advanced significantly, enabling the generation of deep fake media and raising various technological possibilities.
  • Generative Adversarial Networks are commonly used for image generation, with Fréchet Inception Distance being a key metric for their validation.
  • Fréchet Inception Distance measures the similarity between generated and target image distributions, aiding in the evaluation of generative models.
  • Google researchers have introduced Fréchet Video Distance and Fréchet Audio Distance metrics to assess the quality of synthesised audio and video.
  • These new metrics are essential for evaluating progress in audio and video understanding, addressing a gap in current evaluation methods.

Takeaways drafted with AI from the article text.

From generating deep fake faces to fake videos, Artificial Neural Network has touched new heights as well as opened up various directions in the domain of emerging technologies. While modelling a deep ANN, it is important to use the right metrics with the right dataset to build up a robust model.  Generative adversarial networks (GANs) are one of the most popular methods for generating images and Frechet Inception Distance (FID) is the most popular metric used to validate it. This metric serves as a remedy for the pitfalls of GANs and is specifically designed for images. The Fréchet Inception Distance (FID) works by taking a large number of images from both the target distribution and the generative model and uses the Inception object-recognition network to embed each image into a lower-dimensional space which captures the important features. Computing the Fréchet distance between these samples helps in providing a quantitative measure of how similar the two distributions actually are. According to the researchers at the tech giant, the access to robust metrics for evaluation of generative models is crucial for measuring (and making) progress in the fields of audio and video understanding, but currently, no such metrics exist until now. Recently, the researchers at Google proposed two new metrics for audio and video which are built on the principles of FID. They are Fréchet Video Distance (FVD) and Fréchet Audio Distance (FAD) which are useful in measuring the quality of synthesised video and audio respectively.  Figure: The key component for both metrics is a pre-trained model that converts the video or audio clip into an N-dimensional embedding. (Source)

Fréchet Video Distance (FVD)

Fréchet Video Distance (FVD) is a new metric for generative models of video which correlates well with the qualitative human judgment of generated videos. FVD can be used in situations such as unconditional video generation via Generative Adversarial Networks. There are several features which makes this metric better than the existing ones. They are mentioned below
  • FVD is sensitive to both temporal, and frame-level perturbations.
  • It coincides well with the qualitative human judgment of generated videos.
  • This metric is accurate in evaluating videos that were modified to include static noise, and temporal noise.
This metric avoids the drawbacks of frame-level metrics which is common among the existing video evaluating metrics. Image: Examples of videos of a robot arm, judged by the new FVD metric. (Source)

Fréchet Audio Distance (FAD)

Fréchet Audio Distance (FAD) is a reference-free evaluation metric for music enhancement algorithms which is designed to measure how a given audio clip compares to clean, studio-recorded music. It compares statistics computed on a set of reconstructed music clips to background statistics computed on a large set of studio-recorded music.  This metric is different from the other existing metrics as the existing metrics for generating audio quality either require a time-aligned ground truth signal, for instance, source-to-distortion ratio (SDR) or only target a specific domain like speech quality. But FAD, on the other hand, is a reference-free metric and can be used on any type of audio.  For the evaluation of these two metrics, the researchers at Google performed a large-scale human study in order to determine how well these two new metrics align with the qualitative human judgment of generated audio and video.

Outlook

For a few years now, Google has been doing several breakthroughs among intelligent platforms. The tech giant has also released the source codes for both Fréchet Video Distance and Fréchet Audio Distance on GitHub. According to the researchers at the tech giant, the two metrics, FAD and FVD will assist in keeping this progress measurable as well as improve the models for audio and video generation.

Published

Spotted an error? Tell our editors. Read our editorial standards and corrections.

More in this story

GAN · 5 stories
  1. Making A Case For Art With GANs
  2. GANs Can Swap Faces Now With FSGAN: A New Deep Learning Approach To Face Swapping
  3. You are here: Google Used Fréchet Distance To Assess AI-Generated Audio & Video Quality
  4. What Is Google’s Recently Launched BigBird
  5. Google Creates Videos and 3D Models from Single Images
About the author

Contributor

A Technical Journalist who loves writing about Machine Learning and Artificial Intelligence. A lover of music, writing and learning something out of the box.

All stories
Belamy · Daily

The day’s AI news, before your first meeting.