Benchmarks Don’t Tell the Full Story of Sarvam AI

For years, India talked about building a sovereign AI model. That day arrived at the India AI Impact Summit last month when Bengaluru-based AI startup Sarvam AI unveiled two open-weight large language models—Sarvam 30B and Sarvam 105B—trained entirely in India using compute provided under the IndiaAI Mission. 

Both models were rolled out earlier this month under the open-source Apache 2.0 license, and weights are available on the government-backed AI repository AIKosh as well as Hugging Face. The models can also be accessed through the company’s API.

The company said the models were developed from scratch using internally curated datasets and deployed across multiple production systems. 

Sarvam 30B was trained on 16 trillion tokens, while Sarvam 105B used 12 trillion tokens. The training mixture includes code, web data, specialised knowledge corpora, mathematics datasets, and multilingual content. A substantial portion of the multilingual corpus focuses on the 10 most spoken Indian languages. 

The two models already power products built by Sarvam. Sarvam 30B runs Samvaad, the company’s conversational agent platform, while Sarvam 105B powers Indus, an AI assistant for reasoning and agent-based workflows.

Sanchit Vir Gogia, CEO of Greyhound Research, notes that Sarvam’s models are entering a global environment where open model innovation has accelerated rapidly. From a technical standpoint, he says Sarvam appears to be working within the same architectural direction that defines many modern foundation models. 

“The models rely on a Mixture-of-Experts routing, which allows a large parameter count without activating every parameter for each token processed. Instead, only a subset of experts (subnetworks) participates in generating a response. This reduces the computational cost associated with inference while maintaining model capacity,” he says. 

Full-Stack Approach

Sarvam said the models were built through a full-stack development process covering dataset creation, architecture design, training infrastructure, and inference optimisation. The company also developed internal systems covering tokenisation, execution kernels, scheduling, and inference pipelines to enable deployment across different hardware configurations ranging from data-centre GPUs to personal laptops.

It said the 105B model is good at reasoning and agentic tasks, while the smaller 30B model is built for real-time conversational deployment

The 105B model beats DeepSeek R1 on Browse Comp, a benchmark that tests how well AI models can search the web and use the information to complete tasks. Sarvam 105B scored 49.5, compared with 3.2 for DeepSeek R1.

However, the model trails some competitors on other benchmarks. On SWE Bench Verified, which evaluates software engineering and coding capabilities, Sarvam 105B scored 45.0, lower than DeepSeek R1 at 57.6 and Gemini 2.5 Flash at 48.9. 

It scored 78.7 on GPQA Diamond, which evaluates advanced reasoning in STEM fields, behind DeepSeek R1 at 81.0 and Gemini 2.5 Flash at 82.8.

On the other hand, the 30B model, which activates up to 2.4 billion active parameters per token, performs competitively with larger models across several benchmarks. It scored 97.0 on MATH-500, 92.1 on HumanEval (evaluating functional correctness of code generation), and 92.7 on MBPP (on translating natural language descriptions into working code). 

On LiveCodeBench v6 (solving high-quality competitive programming problems), it scored 70.0 and achieved 85.1 on the MMLU knowledge benchmark. 

The DeepSeek Connection

In a post on X, Pratyush Kumar, Co-founder of Sarvam AI, said the company’s 105B model is already posting reasoning benchmark scores comparable to, and in some cases better than, those achieved by DeepSeek R1 when it was released.

“We got there with a small team and with a smaller model size. This only points to the rapidly improving productivity gains in research as AI gets adopted by LLM teams across the world,” he added. Sarvam’s 105B model was built by a team of 15 engineers, out of a total team of 40 technologists, according to ET.

The comparison with DeepSeek also sparked debate in the AI community, with some observers suggesting that Sarvam’s architecture closely mirrors techniques used by DeepSeek, alleging it is a clone. 

In a blog post, Sarvam clarified that the 105B  extends its architecture with Multi-head Latent Attention (MLA), a compressed attention technique popularised by DeepSeek’s V2 models that reduces KV-cache memory usage during long-context inference, managing inactive cache data to CPU memory.

On the other hand, Sarvam 30B uses Grouped Query Attention (GQA), which reduces KV-cache memory usage while maintaining performance. GQA, first introduced by researchers at Google in 2023, reduces memory usage during inference by allowing multiple query heads to share key and value representations.

“Models like DeepSeek have shown what is possible when open models are pushed aggressively on performance and efficiency. It will take time for Indian labs to reach that level, but what matters right now is momentum,” says Ashutosh Prakash Singh, CEO and Co-founder of fintech-focused AI startup RevRag.ai. 

“The more teams that experiment with training and releasing models, the faster the ecosystem matures,” he adds. 

The Real Moat 

Sarvam’s models should not be judged by how they perform on coding benchmarks but by how good they are at Indic languages.

The company built a tokeniser designed for all 22 scheduled Indian languages across 12 scripts. The tokeniser reduces the number of tokens required to represent words, improving inference speed and reducing costs. It is particularly efficient for low-resource languages such as Odia, Santali, and Manipuri.

Singh says that for startups, Sarvam models are definitely helpful. “Even if many companies continue to rely on global models for production today, having strong open models built in India gives developers more flexibility, especially for localisation, cost control, and building applications tailored to Indian languages and markets,” he adds.

Sarvam also rebuilt its inference system to run models more efficiently. It used fused attention kernels, better scheduling, and disaggregated serving pipelines to improve performance. On H100 GPUs, Sarvam 30B delivered 3–6x higher throughput per GPU compared with a Qwen3 baseline at similar operating conditions.

The model can also run on mid-tier GPUs such as NVIDIA L40S and even locally on Apple Silicon devices, while maintaining good token throughput.

“One thing that is not captured by the benchmarks is Sarvam’s good performance on Indian languages. According to a judge model, the Sarvam team found that their model is preferred 90% of the time compared to others when it comes to Indian texts,” said Sebastian Raschka, Founder and LLM Research Engineer at RAIR Lab and author of  Build a Large Language Model (From Scratch), in a post on X. 

Similarly, Gogia says Sarvam may not find its strongest footing in raw benchmarking against global models. Instead, its advantage could lie in solving problems that global systems are not optimised for, particularly in India’s diverse linguistic environment.

“Everyday communication often involves multiple languages within the same sentence. Models trained with a strong focus on Indian languages and dialect patterns can perform significantly better in local enterprise applications,” he adds.

For instance, Sarvam 30B powers Samvaad, a conversational AI platform that supports multilingual interactions such as banking assistance and agricultural surveys. The system can handle telephone conversations in languages including Hindi and Tamil.

The company also tested the models on several real-world tasks. The 105B model was evaluated using the JEE Main 2026 exam. Under the Pass@1 evaluation metric, which assesses correctness at first attempt, the model scored 25 out of 25 in mathematics, 23 out of 25 in chemistry, and 22 out of 25 in physics. Under Pass@2 evaluation the model achieved perfect scores across all subjects. 

Will Enterprises Adopt?

Gogia says Sarvam’s open models could make experimentation easier for startups, but they do not eliminate the deeper structural challenges that shape AI innovation ecosystems.

“A common misunderstanding is that once a model’s weights are public, building AI products becomes straightforward,” he says. “In practice, the model itself is only one part of the equation.”

Startups still need compute infrastructure to run the models, tools to optimise inference, datasets for fine-tuning, and engineers who understand large-scale machine learning systems.

At the same time, he says Sarvam’s release improves certain parts of the development environment. “The permissive Apache licensing model reduces legal barriers for commercial use. Developers can build applications on top of the models without navigating restrictive licensing terms.”

In an interview with CNBC-TV18, Kumar said Sarvam’s enterprise strategy will be initially focused on the BFSI sector. The company is also expanding into education and exploring opportunities with the central and state governments.