NVIDIA Introduces TensorRT-LLM To Accelerate LLM Inference on H100 GPUs

NVIDIA recently announced it  is set to release TensorRT-LLM in coming weeks, an open source  software that promises to accelerate and optimize LLM inference.

TensorRT-LLM encompasses a host of optimizations, pre- and post-processing steps, and multi-GPU/multi-node communication primitives, all designed to unlock unprecedented performance levels on NVIDIA GPUs. 

Notably, this software empowers developers to experiment with new LLMs, offering peak performance and customization capabilities without necessitating expertise in C++ or NVIDIA CUDA.

Naveen Rao, Vice President of Engineering at Databricks, lauded TensorRT-LLM, describing it as "easy to use, feature-packed with streaming of tokens, in-flight batching, paged-attention, quantization, and more." He emphasized that it delivers state-of-the-art performance for LLMs on NVIDIA GPUs, ultimately benefiting customers with cost savings.

Performance benchmarks demonstrate the significant improvements brought by TensorRT-LLM on the latest NVIDIA Hopper architecture. For instance, the H100 alone is 4x faster than A100. Adding TensorRT-LLM and its benefits, including in-flight batching, result in an 8X total increase to deliver the highest throughput. 

Furthermore, TensorRT-LLM demonstrated its ability to accelerate inference performance for Meta's 70-billion-parameter Llama 2 model by a staggering 4.6x when compared to A100 GPUs.

Today's LLMs are incredibly versatile, serving a multitude of tasks with varying output sizes. TensorRT-LLM addresses this challenge with in-flight batching, an optimized scheduling technique that allows for the concurrent execution of requests.

With the rapid innovation in the LLM ecosystem and the emergence of larger, more advanced models, the need for multi-GPU coordination and optimization has become paramount. TensorRT-LLM leverages tensor parallelism, a model parallelism technique, to efficiently scale LLM inference across multiple GPUs and servers. This automation eliminates the need for developers to manually split models and manage execution across GPUs.

TensorRT-LLM also equips developers with a wealth of open-source NVIDIA AI kernels, including FlashAttention and masked multi-head attention, to optimize models as they evolve.

​​To access TensorRT-LLM, developers can apply for early access through the NVIDIA Developer Program