Why Google TPU 8 May Be Death Knell for General-Purpose AI Chips

Rather than following a one-size-fits-all approach, Google has done something that would have seemed unlikely just a few years ago: it has split its most prized piece of silicon in two. With TPU 8, Google has produced two architecturally distinct chips within a single generation—one built for training, the other for inference. It is a quiet but seismic shift in how the world's most powerful AI systems are built.

The two chips carry aquatic codenames. "Sunfish", officially the TPU 8t, handles AI training and recommendation engines, co-developed with Broadcom. "Zebrafish", the TPU 8i, is dedicated to inference and reasoning, built in partnership with MediaTek. They represent Google's clearest acknowledgement yet that fundamentally different problems deserve fundamentally different hardware.

A Decade in the Making

Google first deployed its TPUs in 2015, born out of a pressing necessity: machine learning algorithms were threatening to double the company's entire data centre footprint if left running on conventional CPUs or GPUs. 

For years, different TPU variants were largely bin-sorted versions of the same basic design, lower-performing dies reassigned for inference, higher-performing ones reserved for training. That pragmatic approach served Google well enough. But the rise of generative AI has changed everything.

Amin Vahdat, Chief Technologist and Senior Vice President of AI and Infrastructure at Google, made the case bluntly at the recent Google Cloud Next event. Google optimised the Sunfish TPU 8t for training by moving block scale multiplication directly inside the MX units. "This native quantisation eliminates CPU overhead, delivering nearly three times the compute performance per pod over previous generations," he said. 

That is not merely incremental progress—it is a redesign of the computational pipeline itself.

Under the Hood

The Sunfish TPU 8t is a multi-chiplet design featuring two compute dies, one I/O die, and six stacks of HBM3e memory, delivering 216 GB of capacity and 6,528 GB/sec of bandwidth. It supports BF16, INT8, FP8, and a new FP4 format that doubles performance over FP8 for portions of the training stack that can tolerate a precision trade-off, with peak FP4 throughput of 12.6 petaflops per chip. 

Google claims the Sunfish TPU 8t delivers 2.7 times better value on AI training compared to its predecessor, the "Ironwood" TPU v7e.

The Zebrafish TPU 8i is where things become particularly interesting. Designed to excel at the decode phase of AI inference, generating tokens in response to a query, it features a massive 384 MB of on-chip SRAM cache, three times more per TensorCore than either its Sunfish sibling or the previous Ironwood generation. 

“In place of SparseCores, which the Sunfish retains for training workloads, the Zebrafish introduces a new Collectives Acceleration Engine, or CAE, which Google says reduces on-chip latency of collectives by 5x by aggregating results across cores with near-zero latency during auto-regressive decoding and chain-of-thought processing,” Vahdat noted.

This focus on inference is no accident. As Harshkumar Vekariya, Senior AI Software Engineer at Apexon Labs, observes in a conversation with AIM, "Training grabs the headlines, but inference brings AI to life." 

For a world moving rapidly towards millions of concurrent AI agents, that latency reduction has become a necessity.

A Shifting Competitive Landscape

Google is not alone in recognising the divergence between training and inference hardware. 

NVIDIA—the dominant force in AI chips—has already pursued a similar bifurcation, with its Blackwell B200 GPU designed to handle both training and inference, while the B300 was more explicitly tuned for inference workloads. NVIDIA's $20 billion acquisition of Groq last December underscored just how seriously the company is taking the inference challenge. 

Meta, meanwhile, is developing its own in-house MTIA chips, while Amazon's Trainium and Inferentia follow a comparable logic of separate silicon for separate tasks.

The pattern is unmistakable. As Vekariya puts it, "AI infrastructure is becoming increasingly specialised, heterogeneous, and operationally complex." The era of the one-chip-fits-all AI accelerator is drawing to a close.

What distinguishes Google's approach is the depth of co-design between hardware and networking. Alongside the TPU 8 chips, the company has introduced a new data centre fabric codenamed "Virgo" to connect superpods within a data centre, with separate topologies for training and inference. The Sunfish training system scales up to 9,600 TPUs in a single memory domain pod, connected via a 3D torus topology and offering twice the inter-chip interconnect bandwidth of the Ironwood generation. 

"The interconnect is now as important as the accelerator," Vekariya comments.

A New Era of AI Infrastructure

The implications extend well beyond raw performance benchmarks. Google's decision to fork its TPU line signals a broader truth about where generative AI is heading: scale alone will no longer be sufficient. The next frontier is efficiency—doing more with less energy, less latency, and less cost. 

Both the Sunfish and Zebrafish deliver 2x better performance per watt than the Ironwood TPU v7e. That’s crucial as energy consumption is increasingly scrutinised by regulators, investors, and engineers alike.

“On the surface, this looks like maybe another custom silicon chip launch. But digging into this, a few things to consider: AI workloads and infrastructure are fragmenting. Inference is becoming the enterprise battleground. The interconnect is now as important as the accelerator,” he notes. 

The rise of agentic AI adds another layer of complexity to this picture. Multi-step reasoning, persistent memory, and orchestration pressure infrastructure in ways that traditional LLM prompting never did, and the Zebrafish's low-latency architecture is clearly designed with that future in mind.

“Agentic AI changes infrastructure assumptions. And enterprise IT deploying AI should separate 'impressive hyperscaler architecture' from 'practical enterprise consumption'—those are often not the same thing,” says Vekariya. 

For now, pricing for TPU 8 instances is yet to be announced, and further architectural details are expected to emerge at the Hot Chips conference in August 2026. But what is already plain is that Google, a company that has been building its own AI chips longer than almost anyone, has just raised the stakes for everyone else.