OpenAI CEO Sam Altman’s India visit has created quite a flutter. When managing director of PeakXV Partners and former VP Google India, Rajan Anandan asked Sam about how India can build substantial foundational models in India with a budget of 10 million versus 100 million, the answer was simple- “totally hopeless”. While this caused a massive stir with people and their bruised egos jumping in to “accept it as a challenge” to prove him wrong, there is truth in what Altman said. The OpenAI founder even reiterated that a $10 million budget will not work. However, not all hopeless, India has been making progress on the LLM front. 

Focus Where It Matters

India’s focus is tilted towards capitalising the talent pool to build solutions that are specific to the country. As opposed to the West, where English is the primary language, and with prompt engineering boosting the language, the market is heavily skewed to cater to an audience that relies on English alone. However, for a country like India, which is home to over 390 languages, and only 10% of the workforce speak English, the development of models needs to be looked at through a different lens. Research institutes across the country are understanding this gap and building on it. They are working and making significant progress in building models that are catering to the country. 

Nascent and Promising Progress

AI research labs across the country are working on technologies to cater to the LLM market specific to our country. IIT Madras is one place where they have been working on models with Indic languages.  

AI4Bharat, an initiative of IIT Madras, is focussed on building open-source language AI for Indian languages including datasets, models and applications. Backed by Nandan Nilekani, AI4Bharat is working towards making these foundational models across tasks and 22 Indian languages. 

One of the many models trained by AI4Bharat, IndicBart, a multilingual, sequence-to-sequence pre-trained model supports 12 languages and is trained on mBART architecture. Considered a compact model, IndicBART is built on 244M parameters and tasks vary from question answering, machine translation and natural language generation. 

IndicBERT, another model, has much fewer parameters than other public models such as mBERT but is able to give a better performance. Its tasks range from generating text, translating languages and writing different kinds of creative content. 

Leveraging Advancements

With significant progress happening in the open-sourced LLM market, the developer community is able to get access to some of the highly trained models such as Meta’s OPT which is trained on parameters ranging from 125 million to 175 billion parameters (same as GPT-3). In addition, there has also been developments in creating lower cost models as low as $300 which reportedly achieves 90% of ChatGPT’s quality. 

Comprehending The Reality of Situation 

Challenges to building a model to the scale of GPT-4 will remain. Financial capabilities is one of the biggest roadblock. Training models is a costly affair with large computational resource costs. Training GPT-3 is said to have cost millions of dollars requiring extensively large computational resources. It is seen that these levels of resources are available with only a few research labs and big techs in the world which thereby curtails the production of new ones. 

In an earlier interview with AIM, Amrutur Bharadwaj, research head and director of ARTPARK, mentioned that government involvement via financial support in the research ecosystem will help boost development. “The current system is not geared to produce and support brilliant outliers.” 

Another challenge that lies ahead is the education system in India where the brightest students leave the country to pursue higher studies. This leads to a talent drain, and you would often find Indian researchers working with global firms. 

The path ahead for India to develop foundational models should not be considered as building mere competitive products to some of the largest foundational models in the world. Instead, identifying the actual use-cases required for our country and working towards catering those needs will be a true benchmark to know where India lies in the LLM race. 

So, dear Sam, like you said, India will ‘try anyway’, which we already have been doing and even progressing at a steady pace. Your statement might serve as an impetus, but we are already on our way.