Article complete

Meta's ‘No Language Left Behind’ 450GB training dataset reproduced & released online

An open book on a table with floating text representing various languages around it, highlighting Meta's translation technolo

Key takeaways

  • Meta and AllenNLP have released a significant 450GB training dataset for AI research.
  • The dataset enables researchers to replicate the training of Meta's NLLB-200 models for better analysis.
  • It includes bitext for a variety of language pairs, enhancing accessibility for multilingual AI development.
  • The structured data is available on platforms like Hugging Face, promoting wider use and collaboration.
  • The project was led by a team of researchers at Meta, focusing on open sourcing AI resources.

Takeaways drafted with AI from the article text.

This newly released data was the missing piece in being able to fully recreate the dataset used to train NLLB-200. AI researchers can now access the full dataset.

The announcement was made in a Twitter post on Wednesday.

The researchers behind this project are Jesse Dodge, Akshita Bhagia and Kenneth Heafield, along with other researchers at Meta for open sourcing info for this reproduction. 

The new dataset contains bitext for 148 English-centric and 1,465 non-English-centric language pairs using the stopes mining library and the LASER3 encoders. The complete dataset is estimated to be 450 GB of text. 

The dataset structure is made up of gzipped tab delimited text files for each direction, with each text file containing lines with parallel sentences. The data was filtered based on language identification, emoji-based filtering, and implemented language models for some high-resource languages. 

Moreover, the dataset can also be accessed on the data science platform Hugging Face. 

Based in Seattle, AI2 is a non-profit research institute founded in 2014 that conducts high-impact AI research and engineering services, taking a results-oriented approach to complex challenges.

Published

Spotted an error? Tell our editors. Read our editorial standards and corrections.

More in this story

Meta AI · 5 stories
  1. What's Meta's Project CAIRaoke?
  2. Meta open-sources an AI model that can translate across 200 languages
  3. You are here: Meta's ‘No Language Left Behind’ 450GB training dataset reproduced & released online
  4. Meta AI Releases A Multimodal Model "CM3leon"  — But Won’t Release It
  5. Meta Reports 33% Revenue Growth, Testing Early Versions of Business AI
About the author

Contributor

All stories
Belamy · Daily

The day’s AI news, before your first meeting.