MIT researchers use AI to create 3D holograms in real-time

Researchers at the Massachusetts Institute of Technology developed a method to produce holograms instantaneously. The method relies on deep learning and can be run on a smartphone. The team used deep learning to accelerate computer-generated holography, allowing for real-time hologram generation. 

The researchers designed a convolutional neural network and built a custom database of 4,000 pairs of computer-generated images. Each pair-matched a picture — including colour and depth information for each pixel — with its corresponding hologram. To create the holograms in the new database, the researchers used scenes with complex and variable shapes and colours, with the depth of pixels distributed evenly from the background to the foreground, and with a new set of physics-based calculations to handle occlusion. That approach resulted in photorealistic training data.

By learning from each image pair, the tensor network tweaked the parameters of its own calculations, successively enhancing its ability to create holograms. The fully optimised network operated orders of magnitude faster than physics-based calculations.

“People previously thought that with existing consumer-grade hardware, it was impossible to do real-time 3D holography computations,” said Liang Shi, the study’s lead author and a PhD student in MIT’s Department of Electrical Engineering and Computer Science (EECS). 

“It’s often been said that commercially available holographic displays will be around in 10 years, yet this statement has been around for decades,” Shi added.

Shi believes the help of ‘tensor holography’ will accelerate the 3D holography development timelines by 10 years.