Article complete

Telugu LLM Labs is Here with Native and Romanised Dataset

The dataset includes a romanised pre-training dataset and a supervised fine-tuning dataset in native and romanised scripts in Telugu.

A stylized graphic featuring a person with a blurred face, surrounded by letters and symbols representing Telugu language.

Key takeaways

  • Launch Telugu LLM Labs aims to enhance Telugu NLP for 100 million speakers globally.
  • Introduce comprehensive datasets in both native and romanized Telugu for AI applications.
  • Develop new pre-training and fine-tuning datasets addressing scarcity in Indic languages.
  • Collaborate with industry experts to share models and embeddings focused on Telugu.
  • Highlight growing importance of romanized Telugu in online communication for dataset creation.

Takeaways drafted with AI from the article text.

Aimed at providing an enriched AI experience for the vast community of 100 million Telugu speakers globally, the initiative marks a pivotal moment for linguistic technology.

The primary objectives of Telugu LLM Labs involve contributing to open datasets with a specific focus on Telugu, encompassing both native script and romanised versions. For this, the initiative aims to share their experiments, including models and embeddings, concentrating on LLMs and tailored for Telugu.

The Romanized Telugu Pretraining Dataset, recognising the prevalent use of romanized Telugu in online conversations, such as WhatsApp or YouTube comments, Telugu LLM Labs unveils the "uonlp_culturaX_telugu_romanized_100k" dataset. 

Comprising the romanized version of the initial 108,000 rows from the culturaX_telugu dataset, this resource addresses the scarcity of datasets for additional pre-training on models or fine tuning, particularly in romanized Telugu.

The Supervised Finetuning Dataset in Telugu (Native + Romanized) addresses the scarcity of instruction datasets in Indic languages, Telugu LLM Labs introduces two datasets –

  • yahma_alpaca_cleaned_telugu_filtered_and_romanized
  • teknium_GPTeacher_general_instruct_telugu_filtered_and_romanized.

These datasets, available on HuggingFace Hub, provide translated and transliterated rows to Telugu, while also undergoing filtering using NLP classification systems to remove content deemed "English Language Specific" or "Coding related." 

Telugu LLM Labs is actively engaged in the finetuning and training of open-source models, including Llama 2, Mistral, and TinyLlama, using the newly released Telugu translation and transliteration datasets.

Published Updated

Spotted an error? Tell our editors. Read our editorial standards and corrections.

More in this story

LLMs · 5 stories
  1. Andrew Ng Partners with Google Cloud for New Course on LLMOps
  2. AI4Bharat Releases Airavata: An Instruction-tuned Hindi LLM
  3. You are here: Telugu LLM Labs is Here with Native and Romanised Dataset
  4. Researcher from Upwork Releases Nandi, Gemma-based Telugu Model
  5. Meta Releases MEGALODON, Efficient LLM Pre-Training and Inference on Infinite Context Length
About the author

Journalist

Mohit writes about AI in simple, explainable, and often funny words. He's especially passionate about chatting with those building AI for Bharat, with the occasional detour into AGI.

All stories
Belamy · Daily

The day’s AI news, before your first meeting.