SambaNova’s 24TB Memory Powers 12T Parameters, Surpasses Cerebras and Groq in AI Scalability

The battle for token speed is intensifying as SambaNova, Cerebras, and Groq push the limits of inference performance. SambaNova recently made headlines by setting a new record for inference on Meta’s Llama 3.1 405B. The platform achieved 132 output tokens per second while running the model in native 16-bit precision. 

In an exclusive interview with AIM, SambaNova’s chief architect Sumti Jairath and Raghu Prabhakar, Architect and founding engineer, revealed how SambaNova is different from its competitors. Notably, among the three—Groq, Cerebras, and SambaNova—SambaNova is the only platform offering Llama 3.1 405B.

The API inference on SambaNova cloud is powered by its SN40L custom AI chip, which features their Reconfigurable Dataflow Unit (RDU) architecture. Manufactured on TSMC’s 5 nm process, the SN40L RDU integrates DRAM, HBM, and SRAM on each chip.

Jairath explained that one of the key differences SambaNova has over Cerebras and Groq is its three levels of memory hierarchy. “If you look at Groq and Cerebras, they only have SRAM. There is no HBM, and there is no high-capacity memory.”

He further explained that running the Llama 70B model on Groq requires nine racks of hardware, with each rack containing eight usable nodes and eight LPUs per node. The challenge, he explained, is having enough SRAM to accommodate 140 gigabytes of memory, which is necessary for the 70 billion parameters.

He added that while Groq and Cerebras handle the 70B model, scaling up would be difficult if the parametres increase. “The reason Cerebras and Groq don't have Llama 3.1 405B yet is that the required capacity would need to grow by another factor of five to ten. The amount of hardware needed becomes impractical,” he said.

Speaking about SambaNova, he explained that they use just one rack, or 16 SN40L RDUs for both Llama3.1 70B and 405B models because of their dataflow architecture and three levels of memory hierarchy. “We have 24 terabytes of DDR, one terabyte of HBM, and around eight gigabytes of SRAM. This combination allows us to manage up to 12 trillion parameters in a single rack,” he said.

Referring to Cerebras, he explained that they would need four wafers, deployed across four hardware racks, for Llama3.1 70B. “The reason they haven’t made Llama3.1 405B available is that it would require 12 racks of hardware—12 wafers in total—making it costly for them,” he said.

Would Love to Work With OpenAI

SambaNova Systems recently launched a new demo on Hugging Face, offering a high-speed, open-source alternative to OpenAI’s o1 model. The demo uses Meta’s Llama 3.1 Instruct model and competes directly with OpenAI’s latest release.

Jairath said that the company is willing to collaborate with any open-source model that matches the capabilities of OpenAI’s o1. Interestingly, he added that SambaNova is also willing to partner with OpenAI to run their models.

“If OpenAI is interested in using our hardware and running their models behind their firewall, I see no reason why it wouldn’t deliver better performance,” he said. Jairath also pointed out that OpenAI’s models would likely benefit from SambaNova’s hardware, saying, “They are currently running on GPUs, and their model architectures are well-aligned with what SambaNova offers.”

Better than Traditional GPUs

Prabhakar explained that running a complex operation like a Transformer decoder on a GPU often involves multiple kernel launches, each with its own memory access patterns and synchronisation costs. In contrast, the SN40L RDU’s dataflow architecture allows for the fusion of these operations into a single kernel call, drastically reducing overhead and increasing throughput.

“On GPUs, each executing a decoder might require multiple trips to high-bandwidth memory (HBM), resulting in significant bandwidth and latency penalties. By fusing all operators into a single unit of execution, we eliminate these inefficiencies and achieve up to 3x to 4x better performance over NVIDIA DGX H100. Also, by delivering this performance with just 16 chips, or one rack, the SN40L has a 10X better Performance/Area over Cerebras, and 40X better Performance/Area over Groq, translating to much better Perf/TCO,” said Prabhkar. 

Speaking about the SN40L chip, he explained, “The chip features over half a gigabyte of SRAM on a single socket, approximately 64 gigabytes of HBM, and 1.5 terabytes of off-package DDR high-capacity memory. The system delivers 638 teraflops, which is roughly comparable to, though slightly below, the compute power of an air-cooled NVIDIA Hopper H100 for BF16 teraflops.

The Business Model of SambaNova

Jairath said that SambaNova is a full-stack company. “We provide everything from the chip to the APIs, basically everything in between.” He explained that back in 2020, they started selling chip systems.

“Our box was very much similar to what DGX boxes are from NVIDIA. Anybody can take it and then code and we still sell that. That product is called the SambaNova Datascale,” he said. “In addition, SambaNova Cloud (cloud.sambanova.ai) offers API access to Llama3.1 models with free and paid tiers.” 

Jairath explained that their current business model caters to enterprises that want a ChatGPT-like service but behind their own firewall. “If someone wants the entire stack—chip system, all the APIs, and the model—but behind their firewall, where they can train and run the model without sending data out, that’s the business we are in,” he said.

He said their goal is to offer enterprise AI solutions that allow customers to fine-tune and deploy models in the cloud. The company also provides an option for deploying these models securely behind the customer's firewall.

Moreover, he explained that in the near future with agentic AI soulutions coming in, companies won’t just deploy one model, instead there will be multiple models working together, which different departments within the company will be fine-tuning. 

“Enterprises will be running hundreds of these models. When they need to operate these many models, they can’t deploy a separate set of GPUs for each one, as the costs would grow exponentially,” concluded Jairath.