How Google Used High School Math to Deliver 8x Performance Boost on NVIDIA H100s

Google’s latest research has reduced one of the largest cost centres in large language models without changing model outputs. 

TurboQuant, a compression method introduced by Google Research, reduces key-value cache memory usage by up to 6x while maintaining accuracy across long-context benchmarks.

Quantisation has long been used to reduce model size and memory usage, typically by lowering numerical precision at the cost of some accuracy. 

However, Google argues that existing methods struggle at the KV cache level, where aggressive compression often introduces errors or adds extra memory overhead. 

TurboQuant addresses this by restructuring how values are represented, allowing the cache to be compressed to much lower bit-widths without affecting outputs.

At 4-bit precision, TurboQuant achieves up to an 8x performance increase over 32-bit unquantised keys on H100 GPUs.

The research has raised eyebrows across the ecosystem. 

“I’m surprised Google is publishing this—it seems like a good intellectual property,” Ryan Dahl, Co-Founder of Node.js, wrote on X.

On benchmarks such as LongBench, Needle-in-a-Haystack, ZeroSCROLLS, RULER and L-Eval, models like Gemma and Mistral retain their baseline scores even as cache sizes shrink.

Google reports that TurboQuant “achieves perfect downstream results across all benchmarks while reducing the key value memory size by a factor of at least 6x.” 

According to the company, it also “quantise[s] the key-value cache to just 3 bits without requiring training or fine-tuning and causing any compromise in model accuracy”.

The same system shows higher recall in vector search without relying on dataset-specific optimisation, which typically increases memory use and setup complexity. 

“TurboQuant consistently achieves superior recall ratios compared to baseline methods, despite those baselines utilising inefficient large codebooks and dataset-specific tuning,” Google said.

In an interaction with AIM, Karan Shingde, a machine learning consultant, offered a simplistic explaination of how this works. 

“When you run something like ChatGPT… a lot of GPU memory is used just to store past tokens in something called KV cache,” he says. 

“TurboQuant compresses that heavily, so you can either serve more users on the same GPU, or allow much longer conversations of the context.” 

He adds that this is helpful in running models on smaller machines or edge devices. 

However, he also speaks about the trade-offs. “You save memory, but you add some extra computation because the system has to compress and decompress things.” 

In practice, however, this should not be an issue, as GPUs struggle more with memory than with compute. 

“There is also a small risk of approximation errors in extreme cases, like very long sequences. But overall, the research shows it stays very stable,” he explains. 

Some developers describe the approach as a clever mathematical reframing. Benedict Brady, Founder at Meridian, joked on X: “All you had to do was pay attention to the polar coordinates lecture in [trigonometry], and you could have discovered a 6x reduction in KV cache memory.”

“High school math vindicated.”

This refers to how Google changes vector representation before compression. In its research blog, the company describes moving from standard Cartesian coordinates to polar coordinates, which, in simplified terms, makes values easier to compress and reduces the extra memory typically required for quantisation.

“Most innovations stem back to deep understanding of high school fundamentals and applying it at the right place,” said Sudarshan Kamath, Founder and CEO of Smallest.ai, in a post on X.

How Different is it Compared to MLA? 

However, there are also questions about novelty. 

The Johnson-Lindenstrauss Transform, the mathematical technique at the heart of TurboQuant’s error-correction layer, has been around for decades. It shrinks complex data without scrambling the relationships between points. 

Google’s contribution lies in applying it aggressively, stripping each value down to a single bit to clean up errors introduced by the main compression step. 

However, Will Brown, research lead at Prime Intellect, notes that this trick is not entirely new, arguing that it is “basically why MLA works”— a reference to Multi-head Latent Attention, an architecture used in models like DeepSeek that achieves similar memory savings through a different route. 

His point is that, for the growing number of models already built on MLA, such as the DeepSeek family, Moonshot AI’s Kimi K2, and others, TurboQuant may not move the needle much.

As a result, the reported speedups on H100 GPUs likely reflect reduced bandwidth pressure rather than changes to the underlying computation.

Shingde offers further clarification. “They (Google and DeepSeek) are solving a similar problem but in different ways,” he says. According to him, MLA changes how the model itself works by reducing the size of attention internally by using a compressed representation. 

“But you need to train the model that way from the start,” Shingde says. “TurboQuant does not touch the model. It works after the model runs and compresses the memory it produces.” 

He added that while MLA is inside the model, TurboQuant is an external system. 

“MLA reduces how much data is created. TurboQuant reduces the amount of memory data takes. If you combine both, you get much bigger gains… So they are not competitors. They are complementary.”

Shift in System Design and Real-World Impact

In an interaction with AIM, Sachin Dharashivkar, an AI researcher and Founder of Athena Agent, says the significance of TurboQuant lies in the fact that the bottleneck in modern systems has already shifted.

“The model weights don't remain a bigger issue… The activations become a bigger thing,” he adds, pointing to KV cache and runtime state as the dominant memory constraint as context lengths grow.

As sequence lengths increase, memory usage scales with every additional token, forcing trade-offs between context limits and repeated recomputation. “After a certain length… sequence length becomes a huge factor which requires larger memory,” he points out. Compression at the KV cache level reduces the frequency with which models need to reset or recompute long contexts.

“I can allocate a larger amount of memory to model weights and smaller to the activation function… the equations change,” he says, describing how reducing KV cache footprint frees up capacity for larger models on the same hardware. For larger labs, this enables longer context windows. For smaller teams, it allows running or training larger models within the same memory limits.

Thus, the impact is immediate in agentic workflows. “We need to do frequent compaction… If this works out, our compaction windows might get longer… our coding agents will get a lot better,” he adds. Fewer compaction cycles reduce context loss in multi-step tasks such as code generation.

He also points to search systems as a primary use case, where large-scale vector retrieval dominates memory usage and cost. Reducing KV cache footprint lowers both infrastructure cost and the amount of data moved across systems.

At the same time, he emphasises that the results need validation beyond inference benchmarks. “On the training side, people need to just check once again… I’m cautious, but more optimistic than cautious,” he says.

A Timely Response to the Memory Constraints?

The research arrives as memory, not compute, and becomes the primary constraint in scaling AI systems. Citing shortages of memory chips as a constraint on expanding capacity, OpenAI COO Brad Lightcap recently told Bloomberg that memory, not compute, has become the primary bottleneck in scaling AI infrastructure.

Similarly, SK Group Chairman Chey Tae-won has warned that shortages in high-bandwidth memory could persist until 2030, with supply lagging demand. The shift is already visible in cost structures, where memory bandwidth and capacity, rather than raw compute, increasingly determine how far models can scale and how efficiently they run.

“Another big direction of this research is hardware,” Shingde points out. “Future GPUs or AI chips might directly support these kinds of compressed representations, which will make this even faster.” 

That, according to him, is one of the techniques that users do not see directly, but can completely change the economics of running AI systems. “Long term, models themselves might be designed in a way that makes them easier to compress from the start.” 

Shingde explains that, soon, we can expect the research outcomes to be added to inference engines like vLLM, followed by compressions that can go beyond KV cache, including weights and activations. 

Furthermore, there is potential for dynamic compression in which systems will decide which tokens require more precision or more compression. “This makes models more efficient without hurting quality.” 

Besides Google, the broader industry focus is converging on the same constraint. 

Recent work from NVIDIA highlights how KV cache size is now a primary bottleneck in inference, with caches for modern models running into hundreds of megabytes per thousand tokens and scaling directly with sequence length.

The company’s ‘kvtc’ research shows compression ratios of up to 20× while maintaining accuracy, with even higher reductions possible in specific cases, reinforcing the substantial and exploitable redundancy in KV representations.

NVIDIA also notes that in production systems, caches are treated as reusable across user sessions, but compete for limited GPU memory, forcing trade-offs between recomputation, offloading, and latency.