Virtual meetings have become the order of the day in the wake of the pandemic. The stocks of video conferencing apps like Google Meet, Zoom and Webex soared in 2020. The trend is continuing into 2021. Taking the cue, Microsoft has released Mesh, a platform that helps build multi-user, cross-platform mixed reality applications based on AI-powered tools.
Mesh contains tools to enable immersive experiences, holographic rendering, spatial maps, and multi-user sync. The team said the platform can be used to conduct virtual meetings, conduct design sessions and classes, and host virtual social gatherings with participants worldwide.
What Is Mesh?
Mixed reality is touted as the fourth wave in computing after Mainframes, PCs, and smartphones. Up to 50 percent of the Fortune 500 companies currently use mixed reality solutions such as HoloLens. However, the options in Mixed Reality applications and experiences remain minimal. Lack of time and resources for people representation, hologram stability, building high-fidelity models into mixed reality, and synchronisation of people’s actions and gestures are the major pain points in developing such platforms.
With Mesh, the Microsoft team hopes to help developers design immersive MR apps without worrying about complex technical issues. It provides a comprehensive developer platform and tooling based on Microsoft Azure.
Developers don’t need to worry about core infrastructure around live-state management, audio/video transmission, and billing. To enable secure and trusted sessions, Mesh uses identity services such as Azure Active Directory and Microsoft Accounts. With Microsoft Graph API, users can bring connections, content, and preferences from the commercial and consumer space. Mesh also has AI-capabilities for enabling massive multi-user online (MMO) scenarios for mixed reality.
Credit: Microsoft
Salient Features
Mesh’s significant features include:
- The platform allows ‘meet users where they are’. For a 3D volumetric experience, Mesh supports fully immersive head-mounted displays (HMDs) such as Microsoft HoloLens, HP Reverb G2, and Oculus Quest 2. It provides an accessible 3D presence with representative avatars created with inside-out sensors of the devices. Mesh contains an avatar rig and a customisation studio for creating out-of-the-box avatars. Due to its AI-powered motion models that accurately capture motions, the platform can function with existing avatar rigs. Along with avatars, Mesh enables 360° 3D representation creation and transmission called holoportation, using outside-in sensors. These sensors can be custom camera setups.
- Regardless of the device’s (mobile, PC, tablet) compute and thermal budget, Mesh allows both local stand-alone and cloud-connected remote holographic rendering for each scene and model. This helps in optimising for latency and fidelity both, subject to the device being used. Additionally, 3D file formats can be natively rendered on Mesh-enabled apps.
- Understanding each participant’s physical world and the space is a prerequisite when building apps that persist holographic content in the real world. It is important to place holograms in a way that can persist across time, space, and devices. Mesh fulfils this need by delivering ‘world-locked holograms’ tied to specific points of interest. Mesh also generates holographic content aligned precisely to the layout and geometry of objects. This helps in building apps with visual information, service records, and other data.
- With the multi-user sync feature in Mesh, users can create a hologram’s common perspective in a collaborative session. As per the company, all the motions, expressions or holographic transformation happens within 100 milliseconds of latency, irrespective of whether the participant is in the same physical location or not. With this, Mesh creates a sense of being in the same physical space in a multi-user situation.
Wrapping Up
At the Microsoft Ignite event, the tech giant announced that it had built two apps on the Mesh platform–HoloLens and AltspaceVR.
“This has been the dream for mixed reality, the idea from the very beginning. You can feel like you’re in the same place with someone sharing content or you can teleport from different mixed reality devices and be present with people even when you’re not physically together,” said Microsoft Technical Fellow Alex Kipman about Mesh.
