Active Hackathon

TensorFlow brings out Depth API to convert individual images to 3D photos

Tensorflow has launched a live demo for enthusiasts to try and convert their photographs into 3D versions.

TensorFlow has introduced Depth API and ARPortraitDepth which estimate a depth map for a single portrait image. It has also presented a computational photography application, 3D photo, which uses the predicted depth and enables a 3D parallax effect on the given portrait image. Tensorflow has launched a live demo for enthusiasts to try and convert their photographs into 3D versions.

Mechanism

TensorFlow explains that ARPortraitDepth takes a single colour portrait image as the input and produces a depth map. The encoder gradually downscales the image or feature map resolution by half, and the decoder increases the feature resolution to the same as the input. 

THE BELAMY

Sign up for your weekly dose of what's up in emerging technology.

Image: TensorFlow

It adds, “Deep learning features from the encoder are concatenated to the corresponding layers with the same spatial resolution in the decoders to bring high-resolution signals for depth estimation. During training, we force the decoder to produce depth predictions with increasing resolutions at each layer and add a loss for each of them with the ground truth. This empirically helps the decoder to predict accurate depth by gradually adding details.”

To improve the robustness against background variation, it runs an off-the-shelf body segmentation model with MediaPipe and TensorFlow.js before sending the image into the neural network of depth estimation.

TensorFlow added that for the 3D photo application, it created a high-performance rendering pipeline. Firstly, it generates a segmented mask using the TensorFlow.js existing body segmentation API. After that, it passes the masked portrait into the Portrait Depth API and obtains a depth map on the GPU. Then it generates a depth mesh in three.js, with vertices arranged in a regular grid and displaced by re-projecting corresponding depth values.

Image: TensorFlow

Then it applies texture projection to the depth mesh and rotates the camera around the z-axis in a circle. 

More Great AIM Stories

Sreejani Bhattacharyya
I am a technology journalist at AIM. What gets me excited is deep-diving into new-age technologies and analysing how they impact us for the greater good. Reach me at sreejani.bhattacharyya@analyticsindiamag.com

Our Upcoming Events

Conference, in-person (Bangalore)
Machine Learning Developers Summit (MLDS) 2023
19-20th Jan, 2023

Conference, in-person (Bangalore)
Data Engineering Summit (DES) 2023
27-28th Apr, 2023

Conference, in-person (Bangalore)
MachineCon 2023
23rd Jun, 2023

3 Ways to Join our Community

Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

Telegram Channel

Discover special offers, top stories, upcoming events, and more.

Subscribe to our newsletter

Get the latest updates from AIM