Article complete

SambaNova's Llama 3.1 405B Model Hits 114 Tokens Per Second, Setting Speed Record

The company's technology is built around the SN40L chip, which features a reconfigurable dataflow architecture.

Abstract graphic featuring a classical bust, digital elements, and the SambaNova logo against a blue background.

Key takeaways

  • SambaNova's Llama 3.1 405B model sets a world record at 114 tokens per second.
  • Leverage SambaNova's SN40L chip for enhanced AI speed and efficiency in applications.
  • Utilize the verified benchmark for real-time applications like customer support and AI agents.
  • Explore enterprise use cases, including intelligent document processing and explainable AI.
  • Access demos and APIs on SambaNova's website for building advanced AI solutions.

Takeaways drafted with AI from the article text.

“I've been playing with SambaNova Systems's API serving fast Llama 3.1 405B tokens. Really cool to see the leading model running at speed. Congrats to Samba Nova for hitting a 114 tokens/sec speed record,” said DeepLearning.ai founder Andrew Ng.  

The benchmark was set using a single 16-socket node, operating with full 16-bit precision on SambaNova’s custom RDU chips. This advancement addresses the challenge of balancing quality and speed in large models like Llama 3.1 405B, enabling the deployment of the model in more speed-sensitive applications, such as customer support and AI agents.

George Cameron, Co-Founder of Artificial Analysis, confirmed the record, saying that SambaNova's platform reduces the trade-off between model size and operational speed, making it viable for real-time applications.

SambaNova’s fourth-generation RDU chip, the SN40L, plays a critical role in this achievement, facilitating real-time processing that opens up new enterprise use cases. These include intelligent document processing, real-time AI copilots, and explainable AI, all of which benefit from the platform's speed.

The company is offering a demo of the Llama 3.1 405B model on its website and is inviting developers to access its APIs for building enterprise-level AI applications.

SambaNova Systems is a technology company specializing in artificial intelligence (AI) hardware and software solutions. Founded in 2017 in Palo Alto, California, by Kunle Olukotun, Rodrigo Liang, and Christopher Ré, the company provides purpose-built solutions for deep learning and AI applications.

The company's technology is built around the SN40L chip, which features a reconfigurable dataflow architecture. This design optimizes data movement and reduces latency, making it highly efficient for AI tasks compared to traditional GPU-based systems.

Published

Corrections: AIM corrects errors of fact and notes the change on the story. Spotted one? Tell the editors. Read our editorial standards.

About the author

Tech Journalist

Siddharth is a media graduate who loves to explore tech through journalism and putting forward ideas worth pondering about in the era of artificial intelligence.

All stories
Belamy · Daily

The day’s AI news, before your first meeting.