Meta open sources three new AI-driven acoustic models for better sound in the metaverse

While there has been considerable research done on creating better visuals, Meta AI intends to also create sound that is just as immersive for users.

Meta AI has partnered with the University of Texas to open source three new models based on audio-visual perception that can help improve AR/VR experiences. The release is another step toward the direction Meta has taken to shift to a virtual universe. 

The first model, the Visual Acoustic Matching model or AViTAR, can help transform the acoustics in audio clips and make them sound like the target space in a specific image. For instance, an audio clip that sounded like it was recorded in an empty space could be matched with the image of a crowded restaurant and result in audio that sounded like it was in the restaurant. 

The second model, called Visually-Informed Dereverberation or VIDA, as the name suggests, performs the opposite function. VIDA uses observed sounds and visual cues to remove reverberations of a certain audio modality only. This model enhances the quality of speech, which also helps automatic speech recognition. 

THE BELAMY

Sign up for your weekly dose of what's up in emerging technology.

The third model, VisualVoice, helps separate speech from background noise using audio visual cues. 

While there has been considerable research done on creating better visuals, Meta AI intends to also create sound that is just as immersive for users. “Getting spatial audio right is key to delivering a realistic sense of presence in the metaverse,” said Mark Zuckerberg, founder and chief executive of the company. “If you’re at a concert or just talking with friends around a virtual table, a realistic sense of where sound is coming from makes you feel like you’re actually there.”

More Great AIM Stories

Poulomi Chatterjee
Poulomi is a Technology Journalist with Analytics India Magazine. Her fascination with tech and eagerness to dive into new areas led her to the dynamic world of AI and data analytics.

Our Upcoming Events

Conference, in-person (Bangalore)
Machine Learning Developers Summit (MLDS) 2023
19-20th Jan, 2023

Conference, in-person (Bangalore)
Rising 2023 | Women in Tech Conference
16-17th Mar, 2023

Conference, in-person (Bangalore)
Data Engineering Summit (DES) 2023
27-28th Apr, 2023

Conference, in-person (Bangalore)
MachineCon 2023
23rd Jun, 2023

Conference, in-person (Bangalore)
Cypher 2023
20-22nd Sep, 2023

3 Ways to Join our Community

Whatsapp group

Discover special offers, top stories, upcoming events, and more.

Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

Subscribe to our newsletter

Get the latest updates from AIM