NVIDIA’s Groq 3 LPX Brings 3,400 Tokens Per Second to AI Inference

NVIDIA has put its Groq 3 LPX inference accelerator into full production, targeting AI systems that need to generate large volumes of tokens with low latency.

The company said Groq 3 LPX recorded 3,400 output tokens per second while running Gemma 4 31B with a 100,000-token context in Artificial Analysis benchmarking. NVIDIA said this is the fastest result recorded for the model in the benchmark.

Groq 3 LPX extends NVIDIA’s Vera Rubin NVL72 platform by focusing on token generation, a key bottleneck for agentic AI systems that repeatedly reason, call tools, inspect files, write code and verify results.

NVIDIA said Groq 3 LPX can deliver four times faster responsiveness for agents and latency-sensitive workloads than the nearest alternative platform. The company also said the system can reduce some coding tasks from hours to minutes.

“Generation is the phase of inference that determines how responsive an AI system actually is, and that’s exactly what NVIDIA Groq 3 LPX is built to accelerate,” said Danila Shtan, Chief Technology Officer of Nebius.

Nebius will be the first AI cloud provider to deploy Groq 3 LPX through its Nebius Token Factory inference platform. The company plans to make the accelerator available to developers through its existing API.

“As the first AI cloud bringing it to production via Nebius Token Factory, we’re making sure every step of an agent’s loop feels instant,” Shtan said.

Groq will also deploy NVIDIA Groq 3 LPX alongside NVIDIA Vera Rubin NVL72, expanding its AI inference cloud for developers and enterprises.

The company is working with Dell Technologies to deploy the systems in its purpose-built inference cloud. Groq said the combination will increase token generation speeds for context-heavy and agentic AI workloads.

NVIDIA CEO Jensen Huang said the new accelerator extends the company’s focus on inference as AI workloads shift towards agentic systems.

“Inference is the growth engine of AI,” Huang said. “Vera Rubin extends that vision with workload-optimised AI factory configurations designed for the era of agentic AI.”

Groq 3 LPX is built to handle the token-generation stage of inference, while Vera Rubin NVL72 is designed to handle broader training and inference workloads. NVIDIA said the two platforms can work together within AI factories.

The systems also use NVIDIA BlueField-4 DPUs and Spectrum-6 SPX Ethernet, alongside Vera CPU racks and Vera BlueField-4 storage systems.

Following Nebius, Groq plans to be an early adopter of Groq 3 LPX and Vera Rubin NVL72.

The focus on token generation comes as AI agents perform more inference steps for a single task. Faster generation can reduce the time between an agent's reasoning, tool calls and subsequent actions, making inference speed increasingly important for coding and other interactive applications.