Telugu LLM Labs is Here with Native and Romanised Dataset

Ravi Theja Desetty from LlamaIndex and Ramsri Goutham Golla, through their collaborative venture have introduced Telugu LLM Labs, initiating a significant stride towards enhancing the Telugu Natural Language Processing (NLP) space. 

Click here to check out the Hugging Face page.

Aimed at providing an enriched AI experience for the vast community of 100 million Telugu speakers globally, the initiative marks a pivotal moment for linguistic technology.

The primary objectives of Telugu LLM Labs involve contributing to open datasets with a specific focus on Telugu, encompassing both native script and romanised versions. For this, the initiative aims to share their experiments, including models and embeddings, concentrating on LLMs and tailored for Telugu.

The Romanized Telugu Pretraining Dataset, recognising the prevalent use of romanized Telugu in online conversations, such as WhatsApp or YouTube comments, Telugu LLM Labs unveils the "uonlp_culturaX_telugu_romanized_100k" dataset. 

Comprising the romanized version of the initial 108,000 rows from the culturaX_telugu dataset, this resource addresses the scarcity of datasets for additional pre-training on models or fine tuning, particularly in romanized Telugu.

The Supervised Finetuning Dataset in Telugu (Native + Romanized) addresses the scarcity of instruction datasets in Indic languages, Telugu LLM Labs introduces two datasets –

• yahma_alpaca_cleaned_telugu_filtered_and_romanized
• teknium_GPTeacher_general_instruct_telugu_filtered_and_romanized.

These datasets, available on HuggingFace Hub, provide translated and transliterated rows to Telugu, while also undergoing filtering using NLP classification systems to remove content deemed "English Language Specific" or "Coding related." 

Telugu LLM Labs is actively engaged in the finetuning and training of open-source models, including Llama 2, Mistral, and TinyLlama, using the newly released Telugu translation and transliteration datasets.