Article complete

Top 8 Vernacular Language Models Based on Llama 2

Using Llama 2, developers have created various ‘Local Llamas' based on the languages.

Two llamas against a blue background, with letters and symbols emerging from their mouths, symbolizing language models.

Key takeaways

  • Leverage Llama 2's open-source model for diverse vernacular applications worldwide.
  • Create tailored 'Local Llamas' to effectively represent regional languages and improve performance.
  • Utilize Llama 2's lower computational requirements for local training, even in infrastructure-limited areas.
  • Explore OpenHathi's introduction of the first Hindi LLM, showcasing Llama 2's adaptability.
  • Capitalize on Llama 2's ease of fine-tuning for specific tasks using smaller local datasets.

Takeaways drafted with AI from the article text.

It has gained popularity in India and other countries worldwide, where users have created ‘Local Llamas' based on the local language of the region. In models like GPT-3.5, there is a challenge with 'tokenisation,' where Indic language text is not efficiently represented.

Compared to massive LLMs like GPT-4 or 3.5, Llama 2 requires less computational power and training data. This makes it more feasible to train and run on local hardware, even in areas with limited infrastructure. 

Moreover, Llama 2 can be easily fine-tuned for specific tasks and domains using smaller datasets of local language text. This allows developers to tailor the model to their specific language and needs, improving its performance and relevance.

OpenHathi 

Recently in India, Sarvam AI introduced OpenHathi-Hi-v0.1, marking the debut of the first Hindi LLM in the OpenHathi series. Built on a cost-effective platform, this model, extending from Llama2-7B, exhibits performance akin to GPT-3.5 for Indic languages.

OpenHathi, featuring a 48K-token extension of Llama2-7B’s tokenizer, undergoes a two-phase training process. The initial phase focuses on embedding alignment, aligning randomly initialized Hindi embeddings, followed by bilingual language modeling, teaching the model cross-lingual attention across tokens.

The model demonstrates robust performance across various Hindi tasks, comparable to, if not surpassing, GPT-3.5, while maintaining English proficiency. Sarvam AI’s evaluation includes non-academic, real-world tasks alongside standard Natural Language Generation (NLG) tasks. Evaluations against GPT-3.5 generation with GPT-4 as the judge revealed superior performance in Hindi, both in native and Romanised scripts.

Tamil Llama 

Kaggle ML Engineer and Kaggle Master Abhinand Balachandra, recently introduced Tamil Llama, an Indic LLM engineered specifically to elevate the Tamil language domain. This AI model is built on top of Meta’s Llama 2.

It was trained with an additional 16,000 Tamil tokens, aiming to achieve superior text generation and comprehension in the Tamil language. This model serves as an extension of the LLaMA model and has been enhanced by incorporating extra Tamil tokens, utilizing the LoRA methodology for efficient training.

It presents four distinct variations: Tamil LLaMA 7B, 13B, 7B Instruct, and 14B Instruct. Throughout the training phase, the model's vocabulary has been expanded to encompass 16,000 Tamil tokens, supplementing the original 32K tokens. 

Telugu Llama 

Interestingly, Telugu language consumes less tokens as compared to English as per Golla. This breakthrough not only promises faster and cost-effective text generation in Telugu but also sets the stage for LLama 2 to excel in Indic languages, revolutionising the landscape of natural language processing. 

Odia Generative AI 

SeaLLMs - Large Language Models for Southeast Asia

Alibaba Group Holding's research division, Damo Academy, has introduced LLMs  specifically designed for Southeast Asian languages. SeaLLMs are built upon the Llama-2 model and further advanced through continued pre-training with an extended vocabulary, specialized instruction and alignment tuning to better capture the intricacies of regional languages.  

The Southeast Asia LLM (SeaLLM) was pre-trained on Vietnamese, Indonesian, Thai, Malay, Khmer, Lao, Tagalog, and Burmese data sets, and has outperformed other open-source models in linguistic and safety tasks. SeaLLMs exhibit remarkable proficiency in language understanding and generation tasks, presenting a formidable challenge to dominant models like ChatGPT-3.5, especially in Southeast Asian (SEA) languages.

VinaLLaMA: LLaMA-based Vietnamese Foundation Model

VinaLLaMA, a foundational LLM designed specifically for the Vietnamese language. VinaL- LaMA, built on top of LLaMA-2, represents a vital stride towards linguistic inclusivity in AI, adeptly addressing the syntactic and semantic intricacies of Vietnamese.

LLaMAntino: LLaMA 2 Models for Effective Text Generation in Italian Language

This model aims to provide Italian NLP researchers with a tool to tackle tasks such as information extraction and closed qa. 

Dutch Llama v2 13B

It is a Dutch model based on Llama 2. This model is a fine-tuned version of BramVanroy/llama2-13b-ft-mc4_nl_cleaned_tiny on the BramVanroy/dutch_chat_datasets dataset on a context of 4096 tokens. 

Depending on the input, the model can provide satisfactory outcomes, taking into account its 13B size and limited pretraining in Dutch. However, it's essential to note that the model lacks human feedback during training and lacks safeguards, potentially leading to unexpected or offensive content based on the query.

Published Updated

Spotted an error? Tell our editors. Read our editorial standards and corrections.

More in this story

Top Trend · 5 stories
  1. L&T Finance is Certified as a Best Firm for Data Scientists
  2. Top 8 Generative AI Startups in India
  3. You are here: Top 8 Vernacular Language Models Based on Llama 2
  4. Top 7 Apps on Hugging Face Spaces
  5. India to Host Global AI Summit 2026 with Anthropic
About the author

Tech Journalist

Siddharth is a media graduate who loves to explore tech through journalism and putting forward ideas worth pondering about in the era of artificial intelligence.

All stories
Belamy · Daily

The day’s AI news, before your first meeting.