Tamil Llama Creator Unveils Malayalam and Telugu Llamas

In a recent update, Abhinand Balachandran, the creator of the Tamil LLaMA, has included support for Telugu and Malayalam, enhancing its performance over Meta’s LLaMA 2 across various benchmarks. This development builds on the success of the original Tamil LLaMA project, released on GitHub in November 2023.

To address limitations and widen the scope, Abhinand introduced Tamil LLaMA v0.2. This bilingual model excels in English and Tamil, marking a significant step forward.

The team also had JarvisLabs.ai's support with GPUs, facilitating the development of Telugu and Malayalam LLaMA models. These models are now accessible on HuggingFace Hub.

The model adaptation process involved pretraining, fine-tuning, and alignment. The pretraining phase focused on expanding vocabulary and enhancing language generation capabilities. Fine-tuning involved training on a substantial set of instruction-response pairs, while alignment ensured human-preferred responses using techniques like RLHF and DPO.

Despite the alignment stage, the models remain largely uncensored.

Inspired by Sarvam AI's OpenHathi, Abhinand's approach took a distinct technical route. Improvements were made to the Tamil LLaMA tokeniser, and a comparative analysis against Indic Language LLMs on English benchmark scores was conducted. Based on the Open LLM Leaderboard, the evaluation positioned the new models favourably against LLaMA 2 and OpenHathi.

The fine-tuning stage aimed to match or surpass the original LLaMA 2 model's English performance while enhancing language abilities in Tamil, Telugu, and Malayalam. This involved fine-tuning on a vast corpus of instructions, creating a synthetic dataset for regional knowledge, and performing DPO for further enhancement.

The result is the Tamil LLaMA v0.2 model marginally outperforming LLaMA 2 Chat on various benchmarks, showcasing advancements in linguistic capabilities.

The Tamil LLaMA project, initiated in September 2023, aimed to adapt the features of LLaMA 2 for the Tamil language. The project's success led to the release of 7B and 13B parameter model variants. Open-sourcing the project facilitated collaboration and adaptations for other Indian languages like Hindi, Odia, and Kannada.

In an exclusive interview with AIM, Balachandran revealed the genesis of the Tamil LLaMA project, citing inspiration drawn from the Chinese LLaMA Alpaca model. 

He stated, "Chinese is a bit of a complex language, but if they can make it work for Chinese, then surely we will also be able to make it work for Indian languages, right? So that was the motivation," emphasised Balachandran.