TensorFlow brings out Depth API to convert individual images to 3D photos

Tensorflow has launched a live demo for enthusiasts to try and convert their photographs into 3D versions.

TensorFlow has introduced Depth API and ARPortraitDepth which estimate a depth map for a single portrait image. It has also presented a computational photography application, 3D photo, which uses the predicted depth and enables a 3D parallax effect on the given portrait image. Tensorflow has launched a live demo for enthusiasts to try and convert their photographs into 3D versions.


TensorFlow explains that ARPortraitDepth takes a single colour portrait image as the input and produces a depth map. The encoder gradually downscales the image or feature map resolution by half, and the decoder increases the feature resolution to the same as the input. 

Image: TensorFlow

It adds, “Deep learning features from the encoder are concatenated to the corresponding layers with the same spatial resolution in the decoders to bring high-resolution signals for depth estimation. During training, we force the decoder to produce depth predictions with increasing resolutions at each layer and add a loss for each of them with the ground truth. This empirically helps the decoder to predict accurate depth by gradually adding details.”

To improve the robustness against background variation, it runs an off-the-shelf body segmentation model with MediaPipe and TensorFlow.js before sending the image into the neural network of depth estimation.

TensorFlow added that for the 3D photo application, it created a high-performance rendering pipeline. Firstly, it generates a segmented mask using the TensorFlow.js existing body segmentation API. After that, it passes the masked portrait into the Portrait Depth API and obtains a depth map on the GPU. Then it generates a depth mesh in three.js, with vertices arranged in a regular grid and displaced by re-projecting corresponding depth values.

Image: TensorFlow

Then it applies texture projection to the depth mesh and rotates the camera around the z-axis in a circle. 

Download our Mobile App

Sreejani Bhattacharyya
I am a technology journalist at AIM. What gets me excited is deep-diving into new-age technologies and analysing how they impact us for the greater good. Reach me at sreejani.bhattacharyya@analyticsindiamag.com

Subscribe to our newsletter

Join our editors every weekday evening as they steer you through the most significant news of the day.
Your newsletter subscriptions are subject to AIM Privacy Policy and Terms and Conditions.

Our Recent Stories

Our Upcoming Events

3 Ways to Join our Community

Telegram group

Discover special offers, top stories, upcoming events, and more.

Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

Subscribe to our Daily newsletter

Get our daily awesome stories & videos in your inbox

6 IDEs Built for Rust

Rust IDEs aid efficient code development by offering features like code completion, syntax highlighting, linting, debugging tools, and code refactoring