Advertisement

Active Hackathon

Meta open sources three new AI-driven acoustic models for better sound in the metaverse

While there has been considerable research done on creating better visuals, Meta AI intends to also create sound that is just as immersive for users.

Meta AI has partnered with the University of Texas to open source three new models based on audio-visual perception that can help improve AR/VR experiences. The release is another step toward the direction Meta has taken to shift to a virtual universe. 

The first model, the Visual Acoustic Matching model or AViTAR, can help transform the acoustics in audio clips and make them sound like the target space in a specific image. For instance, an audio clip that sounded like it was recorded in an empty space could be matched with the image of a crowded restaurant and result in audio that sounded like it was in the restaurant. 

THE BELAMY

Sign up for your weekly dose of what's up in emerging technology.

The second model, called Visually-Informed Dereverberation or VIDA, as the name suggests, performs the opposite function. VIDA uses observed sounds and visual cues to remove reverberations of a certain audio modality only. This model enhances the quality of speech, which also helps automatic speech recognition. 

The third model, VisualVoice, helps separate speech from background noise using audio visual cues. 

While there has been considerable research done on creating better visuals, Meta AI intends to also create sound that is just as immersive for users. “Getting spatial audio right is key to delivering a realistic sense of presence in the metaverse,” said Mark Zuckerberg, founder and chief executive of the company. “If you’re at a concert or just talking with friends around a virtual table, a realistic sense of where sound is coming from makes you feel like you’re actually there.”

More Great AIM Stories

Poulomi Chatterjee
Poulomi is a Technology Journalist with Analytics India Magazine. Her fascination with tech and eagerness to dive into new areas led her to the dynamic world of AI and data analytics.

Our Upcoming Events

Conference, Virtual
Genpact Analytics Career Day
3rd Sep

Conference, in-person (Bangalore)
Cypher 2022
21-23rd Sep

Conference, in-person (Bangalore)
Machine Learning Developers Summit (MLDS) 2023
19-20th Jan, 2023

Conference, in-person (Bangalore)
Data Engineering Summit (DES) 2023
21st Apr, 2023

Conference, in-person (Bangalore)
MachineCon 2023
23rd Jun, 2023

3 Ways to Join our Community

Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

Telegram Channel

Discover special offers, top stories, upcoming events, and more.

Subscribe to our newsletter

Get the latest updates from AIM
MOST POPULAR
[class^="wpforms-"]
[class^="wpforms-"]