Article complete

Kannada Llama Finally Arrives

"A Continually LoRA PreTrained & FineTuned 7B Indic model"

A stylized llama wearing sunglasses sits cross-legged in a vibrant outfit, surrounded by city skyscrapers and cultural imager

Key takeaways

  • Tensoic releases Kannada Llama, a 7B Llama-2 model fine-tuned for Kannada language processing.
  • Enhance Llama-2's capabilities by training on 600 million Kannada tokens from CulturaX dataset.
  • Develop a new tokenizer, increasing vocabulary from 32K to 48K to better handle Kannada text.
  • Release models, code, and datasets under permissive licenses to promote accessibility.
  • Utilize advanced fine-tuning on state-of-the-art instruction datasets for improved performance.

Takeaways drafted with AI from the article text.

Mumbai-based software development company Tensoic has released Kannada Llama aka Kan-LLaMA [ಕನ್-LLama] — A 7B Llama-2 model, LoRA PreTrained and FineTuned on "Kannada" token.

The company said it expanded Llama-2's existing linguistic capabilities for Low Resource Indic languages, specifically Kannada, by fine-tuning on 600 Million Kannada tokens and subsequently fine-tuning on SOTA Instruction Datasets. The company added that it will release the models, code, datasets and the paper(eventually) under permissive licenses.

“We Continually Pre-Train Llama-2 on ~600 Million Kannada Tokens from the popular CulturaX Dataset. The dataset consists of multiple de-duplicated Multilingual dumps from popular scrapes such as mC4 and OSCAR. We randomly select documents from the same, resulting in a text corpus of ~11GB for the pre-training step.” the company wrote in its blog post. 

The model expansion involves the development of a tokeniser, a crucial tool for splitting text into smaller units or tokens. The vocabulary is increased from Llama-2's existing 32K tokens to a total of 48K tokens, with a specific emphasis on efficiently processing Kannada text.

To achieve this, a sentence piece tokeniser with a vocab size of 20K is trained on a Kannada text corpus, also used for pretraining. This new tokenizer is then merged with the existing Llama-2 tokenizer, resulting in improved processing capabilities for Kannada text.

The subsequent fine-tuning phase is carried out on chat-optimised and translated datasets to enhance the model's conversational capabilities. The curated datasets are released under various licenses, such as cc-by-4.0 and Apache 2.0, to encourage community contributions.

Axolotl is employed for the fine-tuning phase, providing an easy yet powerful environment through YAML configs to fine-tune Large Language Models (LLMs). The resulting Kannada Llama model showcases its generation capabilities through quantized versions.

Here is the dataset of Kan-LLaMA [ಕನ್-LLama]

Published

Corrections: AIM corrects errors of fact and notes the change on the story. Spotted one? Tell the editors. Read our editorial standards.

About the author

Tech Journalist

Siddharth is a media graduate who loves to explore tech through journalism and putting forward ideas worth pondering about in the era of artificial intelligence.

All stories
Belamy · Daily

The day’s AI news, before your first meeting.