OpenAI Releases Point-E, A 3D DALL.E

Point-E will do for 3D image generation what DALL.E has done for 2D images.
Listen to this story

DALL-E 2 was one of the hottest transformer-based models in 2022, but OpenAI just released a brother to this highly capable diffusion model. In a paper submitted on 16th December, the OpenAI team described Point-E, a method for generating 3D point clouds from complex text prompts. 

With this, AI enthusiasts can move beyond text-to-2D-image and generatively synthesize 3D models with text. The project has also been open-sourced on Github, as well as the model’s weights for various numbers of parameters. 

The model is just one of the parts that make the solution work. The crux of the paper lies in the method proposed for creating 3D objects through a diffusion method that works on point clouds. The algorithm was created with a focus on virtual reality, gaming, and industrial design, as it can generate 3D objects up to 600x faster than current methods. 

AIM Daily XO

Join our editors every weekday evening as they steer you through the most significant news of the day, introduce you to fresh perspectives, and provide unexpected moments of joy
Your newsletter subscriptions are subject to AIM Privacy Policy and Terms and Conditions.

There are two ways that text-to-3D models currently work. The first is to train generative models on data which has 3D object to text pairing. This results in incapability to understand more complex prompts as well as issues with 3D datasets. The second approach is to leverage text-image models to optimize the creation of 3D representations of the prompt. 

Point- E combines traditional methods of training algorithms for text-to-3D synthesis. Using two separate models paired together, Point-E can cut down on the amount to create a 3D object. The first set of algorithms is a text-to-image model, likely DALL-E 2, which can create an image of the prompt given by the user. This image is then used as a base for the second model, which converts the image into a 3D object. 

Download our Mobile App

The OpenAI team created a dataset of several million 3D models, which they then exported through Blender. These renders were then processed to extract the image data as a point cloud, which is a way of denoting the density of composition of the 3D object. After further processing, such as removing flat objects and clustering by CLIP features, the dataset was ready to be fed into the View Synthesis GLIDE model. 

The researchers then created a new method for point cloud diffusion by representing the point cloud as a tensor of a shape. These tensors are then whittled down from a random shape to the shape of the required 3D object through progressive denoising. The output from this diffusion model is then run through a point cloud upsampler that improves the quality of the final output. For compatibility with common 3D applications, the point clouds are then converted into meshes using Blender.

These meshes can then be used in games, metaverse applications, or other 3D intensive tasks like post processing for movies. While DALL-E has already revolutionized the text-to-image generation process, Point-E aims to do the same for the 3D space. Creating on-demand 3D objects and shapes fast is an important step towards generating 3D landscapes using artificial intelligence.

Sign up for The Deep Learning Podcast

by Vijayalakshmi Anandan

The Deep Learning Curve is a technology-based podcast hosted by Vijayalakshmi Anandan - Video Presenter and Podcaster at Analytics India Magazine. This podcast is the narrator's journey of curiosity and discovery in the world of technology.

Anirudh VK
I am an AI enthusiast and love keeping up with the latest events in the space. I love video games and pizza.

Our Upcoming Events

24th Mar, 2023 | Webinar
Women-in-Tech: Are you ready for the Techade

27-28th Apr, 2023 I Bangalore
Data Engineering Summit (DES) 2023

23 Jun, 2023 | Bangalore
MachineCon India 2023 [AI100 Awards]

21 Jul, 2023 | New York
MachineCon USA 2023 [AI100 Awards]

3 Ways to Join our Community

Telegram group

Discover special offers, top stories, upcoming events, and more.

Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

Subscribe to our Daily newsletter

Get our daily awesome stories & videos in your inbox

Council Post: From Promise to Peril: The Pros and Cons of Generative AI

Most people associate ‘Generative AI’ with some type of end-of-the-world scenario. In actuality, generative AI exists to facilitate your work rather than to replace it. Its applications are showing up more frequently in daily life. There is probably a method to incorporate generative AI into your work, regardless of whether you operate as a marketer, programmer, designer, or business owner.