India now has a company building solutions to achieve AGI and beyond. Soket AI Labs, the latest entrant, has taken everyone by surprise. This AI research lab plans to do this by starting with smaller language models and eventually building advanced AI systems capable of achieving human-level intelligence. 

“We are a research-first company, technically building products for enterprises. But our primary thesis is that we want to converge towards AGI,” said Abhishek Upperwal, the founder and CEO of Soket AI Labs, adding that artificial general intelligence (AGI) might be born out of India, and NOT just ‘A Guy in India’.

Founded in 2019 by Upperwal, Soket AI Labs’ focus was on building a decentralised data exchange for smart cities. However, things changed significantly after OpenAI CEO Sam Altman’s visit to India, which motivated him and his team to build the best AI models in the country. 

The company is part of NVIDIA’s Inception Programme and AWS Activate for training compute access. Upperwal said that the company plans to obtain access to compute from other cloud providers, without revealing their names. It also made it to Nasscom’s GenAI Founder Programme, alongside getting exclusive access to CDAC’s GPU cluster in Pune. 

A data scientist himself, Upperwal did his master’s at IISc Bangalore with a specialisation in high-performance computing (HPC) and distributed systems and has worked with the ministry of housing and urban affairs on Smart Cities project, where he used to work with a lot of diverse data.

This expertise led him to experiment with transfer learning for building Pragna-1B, India’s first open-source multilingual model designed to cater to India’s linguistic diversity. Available in Hindi, Gujarati, Bangla, and English, the model comes with 1.25 billion parameters and a context length of 2048 tokens.

It was easier said than done 

“The only thing lacking for Indic language models is the availability of data,” said Upperwal. 

Upperwal said that it took the company six months to train the model, which involved many experiments with different models and a total of 150 billion tokens.

“We found that TinyLlama was an amazing model offered under the Apache 2.0 licence,” said Upperwal, but only for English, while also highlighting that the Llama 2 is only an open model and is not the best for Indic languages. The same goes for Llama 3, which struggles to efficiently tokenise Indic languages. 

Upperwal also noted that transfer learning using TinyLlama (which uses Llama 2’s architecture) did not work as efficiently as he expected for Indian languages. That is when the team decided to build from scratch and pre-train the model. 

It took close to 8000 GPU hours on NVIDIA A100s to train the model on 150 billion tokens, which Upperwal said are all fresh and in Indic language.

“When we take a large corpus of mixed-language data, the dominant language is best compressed, but the languages underrepresented are not compressed well,” he explained. That is why Soket AI Labs trained each tokeniser individually for each language separately, and then merged them to get the maximum efficiency. 

To ensure that the quality and quantity of data are enough, Soket AI Labs embarked on the journey to create Bhasha-Wiki, a translation of 6.3 million Wikipedia articles into six languages for training Indic models. 

“One thing is absolutely critical for us: We want to keep the form factor small for all generative AI models,” said Upperwal.

Towards AGI and beyond

Bengaluru and Gurugram-based Soket AI Labs is not alone in the generative AI race. Other companies building full-stack AI solutions for India include Hanooman, Krutrim, and Sarvam AI. Surprisingly, most of them are building solutions focussed on enterprise AI. 

Soket AI Labs, on the other hand, claimed it is not just building full-stack AI solutions for enterprises and Indian customers, but also aiming to achieve AGI. The company recently built the first-of-its-kind GenAI Studio, which offers a unified stack for fine-tuning and building AI models completely in-house. 

Questioning the narrative that India does not need to build its foundational models, Upperwal said even though India is touted as a service provider, AI is about building a sovereign technology. “I think it is really important and because I have worked with the government, I know how necessary it is,” he explained. 

“I think a majority of people working on AGI in the western world are of Indian origin,” quipped Upperwal, cautiously adding that he is not discrediting anyone’s effort. He’s a big fan of Ilya Sutskever and Mustafa Suleyman, after all.