“Modern smartphones come with built-in LiDAR and multi-cameras. This can be used for computer generated holograms ”

Holography is making a comeback, thanks to AI. Invented in the 1940s by Dennis Gabor, holograms have joined forces with deep learning to enrich the virtual world experience. A team of researchers at MIT has figured out how to use artificial intelligence to generate holograms that can run on a laptop.

Typically, holograms need a good setup. The usual 2D projections that get duped for holograms are cheap rip-offs. Originally, holograms are designed to display imagery with great detail. Today, the world has seen 8K footage. For the holograms to be high quality, a lot of data points must be transferred at high speeds. While 5G promises to take care of high transfer rates soon, deep learning assures the whole process can be efficient enough.

The MIT researchers said the VR’s biggest challenge is its pressure on the user’s eye. According to them, the illusions of 3D viewing of VR can result in nausea and eye strain. The solution for better 3D visualization, said the researchers, could lie in a 60-year-old technology remade for the digital world: holograms.

How Deep Learning Helps Holography

Holographic display prototype used for the experiments (Image credits: Paper by Shi et al.,)

“The researchers took a different approach; they asked the computer to teach itself the physics of simulation.”

Despite their realism, holograms are a challenge to make and share. Older holographic techniques produced images that were static, difficult to reproduce and transmit. The introduction of computer-generated holography solved this to an extent. But, the challenge here was the computational heavy lifting. According to the researchers, each point in the scene has a different depth and the same operation cannot be applied for all. To run physics simulations over all these data points would require a clustered supercomputer. Plus, the process is time-consuming. So, the MIT researchers made the machine learn physics independently while not sacrificing the photorealistic nature of the output.

For real-time hologram generation, the researchers used deep learning, which can increase the rate at which holography is generated. The team used a variant of the popular convolutional neural network (CNNs). For training the model, the researchers built 4,000 pairs of computer-generated images. Each pair from the custom database is used to match a picture that includes colour and depth information for each pixel with its corresponding hologram.

By learning from each image pair, the tensor network in the CNNs tweaked the parameters and, in due course, became efficient at creating holograms. According to MIT News, this fully optimized tensor network operated at orders of magnitude faster than physics-based calculations. The compact tensor network requires less than 1 MB of memory. “It’s negligible, considering the tens and hundreds of gigabytes available on the latest cell phone,” explained the researchers.

To handle occlusion, the researchers used physics-based calculations. To create the holograms in the new database, they used scenes with complex and variable shapes and colours and the depth of pixels, which are distributed evenly from the background to the foreground. This approach, stated researchers, resulted in photorealistic training data and hence a workable algorithm. The whole method is called “tensor holography”.

According to the researchers, tensor holography would need only a few seconds to craft holograms from images with depth information provided by typical computer-generated images and can be calculated from a multi-camera setup or LiDAR sensor. Since most smartphones come with LiDAR sensors, this work can help further research to bring real-time 3D holography into our hands.

Future Direction

Image credits: Microsoft Hololens

According to the researchers, holography has many uses:

• Real-time 3D holography has great significance for virtual reality.
• 3D holography could also boost the development of volumetric 3D printing.
• It can change the way we use microscopy, the visualization of medical data, and the design of surfaces with unique optical properties.

Microsoft Hololens and Facebook’s Reality Labs are two of the big names at the forefront of augmented reality research. They have demonstrated that they can change how we do product maintenance, education, entertainment and even high precision surgery. For instance, the Balgrist University Hospital surgeons have used Microsoft’s augmented reality in spinal surgery. The surgeons used the generated CT imaging and the 3D representations of the affected anatomy directly projected into the surgical field overlaying the real anatomy during the operation. Throughout the surgery, the surgeons were prompted for various actions such as the exact insertion point and trajectory of a screw.

The big tech has always paraded their virtual prowess. Be it Google’s failed smart glasses or Microsoft’s recent hit Hololens, top companies are fully immersed in AR/VR for a while now. And, the pandemic turned out to be a shot in the arm for the industry. 

Companies like Facebook and Microsoft are pulling out all stops to offer sci-fi style services. Facebook chief Mark Zuckerberg even went a step further and linked virtual experiences with combating increasing carbon footprint. Zuckerberg believes AR/VR (think: Oculus) can discourage commuters and reduce global emissions!

Find the original paper here.