Sarvam Is Hell-Bent on Pre-Training a Trillion-Parameter Model

Before announcing nearly a dozen updates to its platform, or even teasing its trillion-parameter AI model, to a packed crowd in Bengaluru last week, Sarvam co-founder Pratyush Kumar wanted to get something out of the way. He wanted to address a question swirling in the minds of proponents of building on top of open-weight models in India instead of focusing on foundational models.

‘Why does Sarvam need to build a foundation model from scratch, an effort that is capital-, compute-, and talent-intensive?’

For tech enthusiasts, India’s AI ambitions have always been about maximising the capabilities of building on top of open-weight models. For many others, the bigger concern is whether the same resources could arguably find a better use solving a more pressing problem.

Kumar offered an explanation, both from a technical standpoint and a technological sovereignty perspective.

‘Don’t Bet on Fair Access to Open Source’

“There are still voices in the country that say we should not be training our models,” Kumar began, addressing the idea that Sarvam can just build on top of Moonshot AI’s Kimi K3 model, China’s latest frontier open-weight model.

“I want to say that this is absolutely the wrong thing to do,” he affirmed.

Sarvam’s conviction stems from what it has learnt over the past three years.

He traced the company’s progression from contributing to Meta’s Llama 3, to pre-training a 2-billion-parameter model on India’s first 1,000 Hopper GPUs, developing a reasoning model based on Mistral Small, and eventually training a 100-billion-plus parameter model from scratch on 4,000 Hopper GPUs in India.

It recently bagged major investments from HCLTech, NVIDIA, and others for its $300 million Series B round to build a sovereign AI stack in India.

“The key moat in the space of AI is not that you have an application, not that you have a user base, but really that you have accumulated judgement, maybe you call it taste, that you build over time,” Kumar said.

He argued that training models builds expertise in areas ranging from data quality and reinforcement learning to large-scale GPU orchestration and turning research models into deployable products.

He also framed the effort as striving towards long-term technological sovereignty. While acknowledging that open models are currently driving rapid innovation, Kumar cautioned against assuming that India would always have access to those models.

“You cannot bet the country’s future on fair access to open models,” he said, adding that India needed the ability to build frontier models for its own strategic and commercial needs.

His comments come at a time when AI models from the US briefly restricted OpenAI and Anthropic’s frontier models because of their advanced capabilities. Now, China’s AI Tigers, adept at furnishing open-weight AI, are considering keeping their best models closed-source in the near future.

Kumar further argued that training models gives Sarvam a unique vantage point across the AI stack—from infrastructure and inference to data, model customisation and applications—allowing it to innovate beyond the model layer itself.

However, Kumar did not dismiss the idea of building on top of open-weight models.

“We should absolutely be using the best models out there, both closed-source, frontier, open-source, but at the same time making our bets to train models,” he noted.

That balance is reflected in Sarvam’s latest updates and products. It released Sarvam Coding Agent, part of its full-stack Indus AI platform and built on top of Z.ai’s GLM-5.2 model—one of the best-performing open-weight models in the ecosystem.

As per the benchmark performance claimed by the company, the setup, where Sarvam has built a harness (software infrastructure that wraps around an LLM) around GLM-5.2, offers competitive performance against Claude Sonnet 5, Opus 4.7, and OpenAI’s GPT-5.5-based Codex workflow at a fraction of the cost.

Building for an Unpredictable Future

However, Kumar’s thesis that the company will have a long-term advantage from the sheer experience of building trillion-parameter models does need examining.

In an interaction with AIM, Sajo Mathews, the machine learning lead at Deccan AI, a startup that provides post-training expertise to US-based AI labs, says there are tangential benefits to building a foundational model. “It creates an ecosystem that will allow you to pursue new trajectories and areas of thought.”

He argues that without investing in foundational AI capabilities, countries risk losing the ability to identify and pursue the next wave of technological breakthroughs. Building from first principles, he believes, creates the expertise needed to recognise what is worth investing in next.

He points to Samsung as an example. The company is now benefiting from the AI-driven surge in demand for memory chips, but those gains are the result of decades of patient investment in the memory business, not a short-term bet.

“Trying to short-circuit does not create the ecosystem and the capabilities that you actually need underneath to be able to create something which is valuable,” says Mathews.

That said, fine-tuning existing models or building on open-weight small models is more suited for identifiable enterprise use cases that require specialised intelligence rather than broad reasoning.

For example, small language models, he says, are more parameter-efficient for narrowly defined tasks, but they cannot replace general-purpose foundation models designed to tackle unfamiliar problems across domains.

“Both are valid strategies. Both have their place, and we should be investing in both,” he affirms.

Post-Training Gives Immediate Value

However, many industry stakeholders share a different perspective.

“I think pre-training is important, but I don’t think it’s the first thing Indian AI companies should invest heavily in,” says Adithya S Kolavi, an AI researcher and Founder of open-source research lab CognitiveAI.

Specifically, when it comes to creating value, Kolavi argues that Indian AI startups should focus less on building foundation models from scratch and more on post-training open-weight models.

Thankfully, Indian AI startups are increasingly subscribing to this approach. Avataar AI recently launched Varya, a video generation model distilled (transferring learning) from Alibaba’s open-weight Wan 2.2 and adapted for Indian use cases.

On the technical advantages of pre-training, Kolavi says it allows the model maker to “have complete control over the tokenizer, the data mixture, the architecture, and every stage of training”.

Elements of the model such as the tokenizer, which determines how text is broken down into units that the model can understand, help improve how well the model handles Indian languages, mixed-language inputs, and local scripts.

Controlling the datasets and the underlying architecture, meanwhile, allows model builders to optimise the system for Indian use cases from the ground up.

Despite this, Kolavi argues that far more value can be created by building on top of open-weight models. Techniques such as supervised fine-tuning, reinforcement learning, preference optimisation, and domain-specific adaptation allow companies to improve an existing model by training it with proprietary data and aligning it to specific business requirements rather than starting from scratch.

He also says companies should invest capital in the infrastructure required to carry out post-training tasks.

“Of course, there are strategic advantages to having a strong Indian foundation model, especially from a sovereignty and long-term research perspective. But if I had to prioritise where investment should go today, I would put post-training ahead of pre-training.”