Microsoft’s Holoportation Demo Showcases Its Prowess In Mixed Reality, AI & Neural Text-To-Speech

Be it a stylus which can turn onto an earpiece or an automated travel diary system, Microsoft has never ceased to amaze its audiences and customers. At the recently concluded Microsoft Inspire 2019, the tech giant showed a surreal technology known as Holoportation.

Researchers at Microsoft have been working on holoportation and mixed realities for a while now. Holoportation is a type of 3D capture technology that allows high-quality 3D models of persons to be reconstructed, compressed and transmitted anywhere, real-time.

Now, the researchers from Microsoft have been trying to make the approach of holoportation cellular by reducing the bandwidth, changing lighting and background, etc. and have successfully accomplished to a great extent. 

Julia White, Azure Corporate Vice President demonstrated how holoportation works through Hololens 2 headset and the technologies used behind this innovation. She wore a Hololens 2 headset which generated a complete hologram of herself speaking in a foreign language.

Technologies Behind

The technology is developed using four approaches:

1| Mixed Reality: This technology is used to create the hologram of the speaker and render it in real-time at the event. Holoportation is a 3D capture technology which allows high-quality 3D models of people to be reconstructed, compressed and transmitted anywhere in the world in real-time. This technology is combined with mixed reality display such as, in this case, Hololens 2 headset in order to see, hear and communicate with people who are not present at that particular place.

2| Azure Speech-To-Text Capability:  Azure Speech-To-Text enables real-time transcription of audio streams into text. This capability is used in the English transcription to extract the speech which can be consumed by the device and later take action on as command input.

3| Azure Translate: The Azure Speech Translator is a cloud-based automatic translation service which enables to create an end-to-end, real-time speech translation. This approach helps the device to translate the English speech into the Japanese language.

4| Neural Text-To-Speech Technology: Microsoft’s neural text-to-speech uses deep neural networks to overcome the limits of traditional text-to-speech systems in matching the patterns of stress and intonation in spoken language. In this case, the neural text-to-speech technology was applied to sound exactly the same way as White was. The only difference is that this time the 3D replica was speaking in the Japanese language. 

The Demo

The demo was a jaw-dropping highlight from the event. The main idea behind this innovation is to fundamentally change the way of conversations with the persons who are miles away. It is basically a method of virtual teleportation from one space to another regardless of language and distance.

Azure Corporate VP, White said that the “game-changing” experience used mixed reality and Azure AI services to create the hologram of the speaker. The life-size hologram of White was created at Microsoft’s Mixed Reality Capture Studios. At the event, White put on the headset which created a smaller version of herself which she called “mini-me” and with the help of Hololens 2, she shows it off as if she is actually holding the smaller version of herself.

The speaker then commands “render keynote” and with the help of some special effects, the “mini-me” version is transformed into a real and full-sized hologram of the speaker. The hologram of White is rendered in real-time and spoke in the Japanese language with the speaker’s unique voice signature. The interesting part is that White personally does not know how to speak Japanese. 

Watch the video here:


More Great AIM Stories

Ambika Choudhury
A Technical Journalist who loves writing about Machine Learning and Artificial Intelligence. A lover of music, writing and learning something out of the box.

More Stories


8th April | In-person Conference | Hotel Radisson Blue, Bangalore

Organized by Analytics India Magazine

View Event >>

30th Apr | Virtual conference

Organized by Analytics India Magazine

View Event >>


3 Ways to Join our Community

Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

Telegram Channel

Discover special offers, top stories, upcoming events, and more.

Subscribe to our newsletter

Get the latest updates from AIM