Meta Releases HyperReel for High-Resolution 3D Experience

Recently, MetaAI released HyperReel, a new model which uses the Six–Degrees-of-Freedom (6-DoF) technique to enable photorealistic view synthesis for static scenes. This technique allows new, unobserved views of an environment—static or dynamic—to be processed from a set of posed images or videos. 

The 6-DoF videos provide immersive experiences with various AR/VR applications. For instance, when a VR player changes their head position (3 DoF) and orientation (3 DoF), the view also synthesises accordingly. 

However, the volume rendering procedures that drive these representations require careful trade-offs between quality, rendering speed, and memory efficiency. Existing methods have struggled to simultaneously achieve real-time performance, a small memory footprint, and high-quality rendering for difficult real-world scenarios. 

HyperReel is based on Neural Radiance Fields (NeRF), using a ray-conditioned sample prediction network that predicts sparse point samples for volume rendering. The researchers claimed their system renders up to 18 frames-per-second at megapixel resolution without using any custom CUDA code. 

The dynamic rendering of scenes was produced with datasets collected from Google Immersive, Technicolor, and Neural 3D Video, and the static reconstruction was made through datasets from Shiny and Stanford. The following is a chart on where the HyperReel fares in comparison to other state of the art models when it comes to ensuring speed-quality trade-off for both static and dynamic modelling.   

NVIDIA’s Jim Fan described HyperReel as the “new killer app” for VR. He said that one can capture videos with multiple cameras at different angles, then run HyperReel, and later step into the dynamic scene and freely walk around. This is essentially a high-resolution 4D experience replay. 

Fan further added, “The biggest strength of HyperReel over prior works is the memory and computational efficiency, both crucial to portable VR headsets. It [the application] runs 18 frames-per-second at megapixel resolution on an NVIDIA RTX 3090, using only vanilla PyTorch.” 

The code released for HyperReel is open-source and can be accessed here.