NVIDIA launches ‘Neuralangelo’ to Transform 2D Videos Into Detailed 3D Structures Using AI

The 3D reconstruction capabilities offered by Neuralangelo will greatly benefit creators by enabling them to recreate the real world in the digital realm.
Listen to this story

NVIDIA Research has developed Neuralangelo, an AI model for 3D reconstruction that utilises neural networks to transform 2D video clips into detailed 3D structures. This innovative model generates lifelike virtual replicas of real-world objects, including buildings, sculptures, and more. Similar to how Michelangelo sculpted intricate visions from blocks of marble, Neuralangelo creates 3D structures with remarkable details and textures. Creative professionals can import these 3D objects into design applications, further editing them for use in various fields such as art, video game development, robotics, and industrial digital twins.

What sets Neuralangelo apart is its ability to accurately translate complex textures from 2D videos to 3D assets, surpassing previous methods. This high fidelity makes it easier for developers and creative professionals to rapidly create usable virtual objects for their projects, utilising footage captured by smartphones. The 3D reconstruction capabilities offered by Neuralangelo will greatly benefit creators by enabling them to recreate the real world in the digital realm. Ming-Yu Liu, senior director of research and co-author of the paper, emphasised that this tool will eventually allow developers to import detailed objects, from small statues to massive buildings, into virtual environments for video games or industrial digital twins.

During a demonstration, NVIDIA researchers showcased Neuralangelo’s ability to recreate a variety of objects, ranging from iconic sculptures like Michelangelo’s David to everyday objects like a flatbed truck. The model also demonstrated its capability to reconstruct building interiors and exteriors, as evidenced by a detailed 3D model of the park at NVIDIA’s Bay Area campus.

Subscribe to our Newsletter

Join our editors every weekday evening as they steer you through the most significant news of the day, introduce you to fresh perspectives, and provide unexpected moments of joy
Your newsletter subscriptions are subject to AIM Privacy Policy and Terms and Conditions.

Neuralangelo employs instant neural graphics primitives, the technology behind NVIDIA Instant NeRF, to capture fine details that previous AI models struggled with, including repetitive texture patterns, homogenous colors, and strong color variations. The model utilises a 2D video of an object or scene filmed from multiple angles, selecting several frames that provide different viewpoints. This approach mimics an artist considering a subject from various sides to grasp its depth, size, and shape. Neuralangelo’s AI then creates a rough 3D representation of the scene, similar to a sculptor chiseling the shape of the subject. The model optimises the render to enhance details, much like a sculptor carefully hewing stone to mimic fabric texture or human figures. The final output is a high-quality 3D object or large-scale scene suitable for virtual reality applications, digital twins, or robotics development.

Neuralangelo is among the nearly 30 projects by NVIDIA Research that will be presented at the Conference on Computer Vision and Pattern Recognition (CVPR), taking place from June 18-22 in Vancouver. These projects cover a wide range of topics, including pose estimation, 3D reconstruction, and video generation. One notable project, DiffCollage, utilises a diffusion method to create large-scale content such as landscape images, 360-degree panoramas, and looped-motion visuals. By treating smaller images as sections of a larger visual, like pieces of a collage, DiffCollage enables diffusion models to generate cohesive-looking content without being trained on images of the same scale. The technique can also transform text prompts into video sequences, as demonstrated by a pretrained diffusion model capturing human motion.

Shyam Nandan Upadhyay
Shyam is a tech journalist with expertise in policy and politics, and exhibits a fervent interest in scrutinising the convergence of AI and analytics in society. In his leisure time, he indulges in anime binges and mountain hikes.

Download our Mobile App

MachineHack | AI Hackathons, Coding & Learning

Host Hackathons & Recruit Great Data Talent!

AIMResearch Pioneering advanced AI market research

With a decade of experience under our belt, we are transforming how businesses use AI & data-driven insights to succeed.

The Gold Standard for Recognizing Excellence in Data Science and Tech Workplaces

With Best Firm Certification, you can effortlessly delve into the minds of your employees, unveil invaluable perspectives, and gain distinguished acclaim for fostering an exceptional company culture.

AIM Leaders Council

World’s Biggest Community Exclusively For Senior Executives In Data Science And Analytics.

3 Ways to Join our Community

Telegram group

Discover special offers, top stories, upcoming events, and more.

Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

Subscribe to our Daily newsletter

Get our daily awesome stories & videos in your inbox