NVIDIA has released a new Kaolin library that will allow researchers to simplify and accelerate workflows. The new library includes a new representation, structured point clouds (SPC), a sparse octree-based acceleration data structure with highly efficient convolution, and ray-tracing capabilities.
SPCs are very popular in 3D deep learning research today. It helps in scaling up and accelerating implicit neural representations. It is also behind the latest version of NeuralLOD training and delivers up to 30x reduction in memory and speeds up training time by a factor of 3.
NVIDIA Kaolin New Release
NVIDIA introduced the Kaolin library in 2019, originally intended for the NVIDIA Toronto AI lab as an internship project. Before that, researchers lacked the utility to develop 3D models for use with deep learning tools; they were forced to write lines of repetitive code and copy the algorithmic components for several projects. The researchers then started developing a PyTorch library that could bring common functionality for 3D deep learning; this led to the development of Kaolin, a software library that supports all types of 3D applications. Since its release, this library has grown into a codebase with optimised utilities and algorithms for 3D deep learning.
Some of Kaolin’s applications include simplification of complex 3D dataset processing to be used for training. Kaolin also provides building blocks for conversions between 3D representations, useful 3D loss functions for training, and differentiable rendering.
The new Kaolin library release also includes a lightweight Tensorboard-style web dashboard called Dash3D visualiser that allows visualisation of local and remote checkpoints without the use of specialised hardware or applications. This tool is combined with the latest builds like a command-line utility. Users can leverage this tool to inspect checkpoints of 3D predictions produced by deep learning models during training and on remote hardware configurations.
Credit: NVIDIA
The new library release also improves support for 3D datasets and new datasets like SHREC and ModelNet, speedups for USD 3D file format that enables a 5x improvement in the load time efficiency during training. It also includes new tutorials for differentiable rendering and 3D checkpoints.
Omniverse Kaolin App
Earlier this year, NVIDIA made its Omniverse Kaolin App available for 3D Deep Learning Researchers. The Omniverse platform offers researchers the ability to collaborate virtually and work across different software applications. The Omniverse Kaolin app is an interactive application, which along with the NVIDIA Kaolin library, helps 3D deep learning researchers in accelerating their process. This app leverages the Omniverse platform, USD format, and RTX rendering to provide interactive tools. These tools allow the visualisation of 3D outputs of any deep learning model as it trains and inspects 3D datasets to find inconsistencies and gain intuitive insights from collections of 3D data.
Need For Kaolin
Deep learning is one of the most rapidly growing areas of research with applications in areas such as self-driving vehicles, autonomous robots, 3D graphics and games, augmented reality, and virtual reality.
Unlike 2D, 3D data has more parameters and features and is more complex. Collecting and transforming 3D data from one representation to another is a tedious task and is more time consuming and error-prone than 2D computer vision. Recently, many better performing models, datasets, metrics, visualisation tools, and graphic tools have been introduced in recent years. Integrating the approaches is still a non-trivial job for researchers and practitioners.
To this end, Kaolin proves to be an efficient tool for the manipulation of 3D content. It can wrap into PyTorch tensors 3D datasets implemented as polygon meshes, point clouds, voxel grids, etc. The interface provides a repository of baseline and state-of-the-art models for classification, 3D reconstruction, segmentation, super-resolution, etc.
Your reaction
Discussion
What's your take on this story?
No comments yet — be the first to share your view.