Article complete

NVIDIA Hopper Leads in Generative AI Inference at MLPerf Results

NVIDIA has commenced the shipment of H200 GPUs today, which will soon be accessible through nearly 20 prominent system builders and cloud service providers.

A sleek, futuristic GPU design showcased against a dark background, highlighting NVIDIA's technology.

Key takeaways

  • NVIDIA's H200 GPUs achieve 3x performance increase in generative AI inference over six months.
  • Leverage TensorRT-LLM for optimized LLM performance and simplified deployment of inference tasks.
  • H200 GPUs generate up to 31,000 tokens per second on the Llama 2 70B benchmark.
  • Innovative thermal management solutions contribute to performance gains of up to 14%.
  • H200 GPUs are now shipping through nearly 20 system builders and cloud service providers.

Takeaways drafted with AI from the article text.

Companies at the forefront of innovation are harnessing TensorRT-LLM to optimise their models, facilitated further by NVIDIA NIM - a suite of inference microservices encompassing powerful engines such as TensorRT-LLM. This integrated approach simplifies the deployment of NVIDIA's inference platform, offering businesses unparalleled efficiency and flexibility.

The recent MLPerf benchmarks unveiled a significant leap in generative AI capabilities, with TensorRT-LLM running on NVIDIA's latest H200 Tensor Core GPUs. These memory-enhanced GPUs, making their debut in the MLPerf arena, achieved remarkable throughput, generating up to 31,000 tokens per second on the Llama 2 70B benchmark.

The success of the H200 GPUs also underscores the innovative strides made in thermal management, with custom solutions contributing to performance gains of up to 14%. These advancements, exemplified by systems builders' creative implementations in NVIDIA MGX designs, further elevate the performance capabilities of Hopper GPUs.

NVIDIA has commenced the shipment of H200 GPUs today, which will soon be accessible through nearly 20 prominent system builders and cloud service providers.

With an impressive memory bandwidth of almost 5 TB/second, the GH200 Superchips have demonstrated exceptional performance, particularly in memory-intensive MLPerf assessments such as recommender systems.

Employing a technique known as structured sparsity—a method aimed at reducing computations, initially introduced with NVIDIA A100 Tensor Core GPUs—NVIDIA engineers achieved speed enhancements of up to 33% on inference with Llama 2.

In response to the escalating sizes of LLM models, NVIDIA's founder and CEO Jensen Huang revealed during last week's GTC that the upcoming NVIDIA Blackwell architecture GPUs will deliver heightened performance levels necessary for multi trillion-parameter AI models.

Published Updated

Spotted an error? Tell our editors. Read our editorial standards and corrections.

More in this story

NVIDIA · 5 stories
  1. The Uber Rich Get $150 Billion Richer, Thanks to The AI Boom
  2. You are here: NVIDIA Hopper Leads in Generative AI Inference at MLPerf Results
  3. NVIDIA to Invest $26 Bn in Open-Weight AI Models Over 5 Years
  4. Microsoft & Mistral Announce Multibillion-Dollar AI Infrastructure Partnership in Europe
  5. Razorpay Announces India’s First AI Payments Foundation Model
About the author

Journalist

Mohit writes about AI in simple, explainable, and often funny words. He's especially passionate about chatting with those building AI for Bharat, with the occasional detour into AGI.

All stories
Belamy · Daily

The day’s AI news, before your first meeting.