“How do astronomers look for exoplanets?” – after pondering over this question for what seems like a split second, the avatar of NVIDIA CEO Jensen Huang responds, “Most exoplanets are found via the transit method. That was the case for the TRAPPIST-1 planets.”

This avatar, also nicknamed Toy Jensen, is one of the latest offerings from NVIDIA. It was demonstrated at the GTC 2021 Fall event recently. These conversational avatars called Toy-Me are trained for speech understanding and natural language processing on large models. These avatars can help synthesise users voices and character animation. Toy Me is just one of the many developments of NVIDIA in the metaverse space.

Launched officially in April this year, NVIDIA has been speeding up its metaverse efforts or what it calls — the NVIDIA Omniverse. It was hardly surprising that a major chunk of Huang’s keynote speech revolved around developments in this field, subtly indicating that the company is ready for the transition in the tech world, which is levitating towards the new wonder word — metaverse. Good news for Omniverse! It has been ranked by TIME magazine as one of the top 100 innovations of 2021.

NVIDIA Omniverse is a scalable, multi-GPU real-time reference development platform that helps users build 3D simulations and collaborate for designing. This Omniverse is based on Pixar’s Universal Scene Description and NVIDIA RTX technology. The Omniverse is in the open beta stage and is being used by 70,000 users. Talking about the technology that was utilised in building this Omniverse, Huang said, “USD is the HTML of the 3D world.”

NVIDIA Omniverse Avatar

The Omniverse Avatar is an amalgamation of the company’s technologies in speech AI, NLU, recommendation engine, computer vision, and simulation technologies. These 3D avatars are interactive characters that can see and interact with a wide variety of subjects, capable of understanding naturally spoken intent. Its range of applications is very diverse and can be used in several customer service interactions in the field of hospitality, banking, and appointment making. 

One of the most important announcements about Omniverse Avatars, Huang, spoke about Project Tokkio. Toy Me is part of this project. This toy version can hold a conversation with real people on subjects of climate change, astronomy, human biology, and much more. For speech and voice, Project Tokkio uses a software called the Riva Speech AI from NVIDIA. This software helps create a virtual assistant with a unique voice. Apart from its applications in customer support roles, Riva can also prove to be a boon for those with speech and language deficiencies.

NVIDIA Riva will now also be integrated into Project Maxine to enable better communication and understanding. It will help in real-time translation and text-to-speech with photo animation and eye contact in real-time. Project Maxine was first announced at the 2020 GTC event. It is a platform that provides a cloud-based suite of GPU-accelerated AI video conferencing software.

The Omniverse Avatar also DRIVE Concierge to serve as a digital assistant to make everyday driving stress-free and simplistic. Passengers can have natural conversations with the vehicle and use voice to control functions that previously required physical control or touch screens.

Omniverse Replicator

Omniverse Replicator is the latest offering from NVIDIA. It is a synthetic data generator that produces physically simulated synthetic data to train deep neural networks. Under its initial offering, NVIDIA introduced two applications of Omniverse Replicator — NVIDIA DRIVE Sim, a virtual world that hosts the digital twin of autonomous vehicles and NVIDIA Isaac Sim, a virtual world for the digital twin for robot manipulation. In both the applications, the Replicators help developers bootstrap AI models, fill data gaps, and label the ground truth like velocity, depth, adverse weather conditions, etc. It is particularly useful for dangerous situations which are not regularly and safely experienced in real life.

Other new features

Other major features announced with Omniverse are:

• NVIDIA CloudXR: It is an enterprise-class streaming framework integrated into Omniverse Kit (a toolkit for building native Omniverse applications and microservices). CloudXR allows users to interactively stream the Omniverse experience onto their mobile AR and VR devices.

• Omniverse Farm: It is a system layer to orchestrate multiple computing resources to execute batch and interactive tasks, performing tasks like rendering, streaming, file conversion, simulation, etc. It can work across typical workstations, bare metal servers, advanced cloud scheduling platforms like Kubernetes or VMs. Developers can build tools to leverage Farm in batch mode or interactive mode. 

• Omniverse VR is the world’s first full-image, real-time ray-traced VR. It helps developers in building their own VR-capable tools on the platform for end-users to utilise them directly.

• Omniverse XR Remote offers augmented reality capabilities and virtual cameras. Designers can leverage it to view their assets fully ray-traced through both iOS and Android devices.

• Omniverse Showroom: As an app in Omniverse Open Beta, Omniverse Showroom lets even non-technical users play with tech demos that demonstrate the platform’s real-time physics and rendering technologies.