Groq to Now Help NVIDIA Build Inference Tech

Groq, the US-based company that designs specialised hardware for AI inference, has announced a non-exclusive licensing agreement with NVIDIA.

As part of the deal, Groq founder Jonathan Ross, along with Groq president Sunny Madra and several other employees, will join NVIDIA to “help advance and scale the licensed technology” for NVIDIA. 

The announcement comes after CNBC earlier reported that NVIDIA was in talks to acquire Groq for $20 billion. But Groq said it will continue to operate as an independent company, with Simon Edwards stepping into the CEO role.

Ross spent more than four years at Google before founding Groq, where he was a key hardware engineer behind the early development of Tensor Processing Units (TPUs). 

TPUs are now central to training and running Google’s Gemini family of models and are widely viewed as one of the few credible alternatives to NVIDIA’s AI hardware stack.

“Jonathan was not only the father of TPU when he was at Google, but he is a technical genius of biblical proportions,” said Chamath Palihapitiya, an early investor in Groq, congratulating Ross on the move, in a post on X. 

After leaving Google, Ross founded Groq to build the Language Processing Unit (LPU), a chip architecture designed for deterministic, low-latency inference. 

Since its launch, Groq has positioned its systems as delivering higher inference speeds for specific models than NVIDIA’s GPUs on certain AI models. 

The deal has triggered speculation across the industry about NVIDIA’s motives. “What this also says to me is that Nvidia sensed a threat to scaling their own inference business,” wrote Naveen Rao, CEO of Unconventional AI and former VP of AI at Databricks, in a post on X.

Max Weinbach, an analyst at Creative Strategies, suggested in a post on X that the agreement could help NVIDIA rethink its inference roadmap.

“This gets Nvidia the IP they need to bypass CoWoS and HBM for a fast inference-focused chip, and use NVLink for better chip-to-chip interconnect of the LPU,” he wrote.

This indicates NVIDIA may be looking to absorb ideas from Groq’s LPU architecture to design inference-optimised chips that rely less on costly advanced packaging and memory stacks, while still leveraging its NVLink ecosystem. 

This would strengthen its position in low-latency, high-throughput AI inference without requiring a complete acquisition of Groq.

In September, Groq raised a $750 million funding round at a valuation of $6.9 billion, underscoring investor confidence in its approach to inference-focused hardware.