Nolano.ai Launches Open-Source English-Hindi Bilingual Model, Hi-NOLIN

Young Indian startup Nolano.ai has introduced Hi-NOLIN, an open-source English-Hindi bilingual model, making it the first Indian model that has beaten all benchmarks for models of its size.

Hi-NOLIN's journey began with the goal of creating the first open-source English-Hindi bilingual model. Researchers expanded the 7B Pythia architecture to a 9B model, enhancing efficiency on their hardware while training on the 300B token Pile text corpus, encompassing both English and code data. Hi-NOLIN stands out for its ability to transition seamlessly between languages, mastering both Hindi and English, while processing code.

As researchers continue training Hi-NOLIN, leveraging the Summit supercomputer with its unique 6 GPUs per node configuration, preliminary results demonstrate remarkable potential. Despite being far from convergence, the 9B model shows a steady reduction in training loss and promises substantial improvements. 

Employing advanced techniques from GPT-NeoX, Megatron-LM, and DeepSpeed, Hi-NOLIN utilizes 3D parallelism and ZeRO redundancy optimizer, maximizing its training resources and computational prowess.

Hi-NOLIN shines through in various standard LLM benchmarks, including HellaSwag, TruthfulQA, Arc, and Human Eval. Remarkably, even in its preliminary stage with 600B tokens, Hi-NOLIN outperforms Pythia 12B and multilingual Bloom models across most evaluation benchmarks, narrowing the gap with LLaMa 2 models.

In a landscape dominated by English language models, Hi-NOLIN is a significant stride towards linguistic inclusivity, addressing the gap in state-of-the-art language models for non-English languages. 

The team behind Hi-NOLIN includes the founding trio of Ayush Kaushal, Tejas Vaidhya and Irina Rish. While Vaidhya is a graduate research student in computer science at MILA and the University of Montreal, focusing on NLP and Transformer-based Language models, Kaushal, has already finished his master's at the University of Texas, specialising in ML and DL, with expertise in data modalities. Irina Rish, a full Professor at the Université de Montréal, is a core faculty member of MILA, holds research chairs, and leads the US Department of Energy's INCITE project on Scalable Foundation Models while serving as the co-founder and chief scientific officer of Nolano.ai.

Meanwhile, Indian IT firm Tech Mahindra intends to launch Project Indus, its LLM designed for Hindi and its 37 dialects, by the end of December or early January.

[Update: 2nd November 2023 20:17 |This article previously mentioned EleutherAI as the creator of Hi-NOLIN. The headline and story have now been updated to reflect the changes.]