Microsoft Research announced the launch of NTREX, the second largest human-translated parallel test set, featuring 128 languages, each having 2000 sentences translated with a document context without post-editing.

NTREX, a data set containing “News Text References of English into X Languages”, expands multilingual testing for translating 123 documents (1,997 sentences, 42k words) from English into 128 target languages. The test data is based on WMT19 and compatible with SacreBLEU. 

Read the full paper here. 

It can be used to evaluate English-sourced translation models but not in the reverse direction. The test set release also introduces another benchmark for evaluating massively multilingual machine translation research.

To produce this data set, the team sent the original English WMT19 test set to professional human translators. This work started after the release of the WMT19 test data and has continued in parallel with the work on new translation models since then. Translators could access the full document context. 

The team compared the NTREX-128 data set with COMET-src, a neural framework for MT evaluation, for the authentic translation direction against the scores obtained in the reverse direction. They also investigated how COMET-src behaves for languages it has yet to be trained.

Microsoft Research revealed the following results: 

• Using COMET-src for test quality estimation is feasible but constrained due to the non-comparability of score ranges across language pairings. 
• A significant subset of languages sees COMET-src scores on translationese input performed than the corresponding authentic source data. 
• Although COMET-src relative comparisons are valid across all language pairings, there is a subset of languages for which the scores seem faulty.  

The data set consists of the following set of 128 languages: Afrikaans, Albanian, Amharic, Arabic, Azerbaijani, Bangla, Bashkir, Bosnian, Bulgarian, Burmese, Cantonese, Catalan, Central Kurdish, Chinese, Chuvash, Croatian, Czech, Danish, Dari, Divehi, Dutch, English, Estonian, Faroese, Fijian, Filipino, Finnish, French, Galician, Georgian, German, Greek, Gujarati, Haitian Creole, Hebrew, Hindi, Hmong, Hungarian, Icelandic, Indonesian, Inuinnaqtun, Inuktitut, Irish, isiZulu, Italian, Japanese, Kannada, Kazakh, Khmer, Kiswahili, Korean, Kurdish, Kyrgyz, Lao, Latvian, Lithuanian, Macedonian, Malagasy, Malay, Malayalam, Maltese, Maori, Marathi, Maya, Yucatán, Mongolian, Nepali, Norwegian, Odia, Otomi, Querétaro, Pashto, Persian, Polish, Portuguese, Punjabi, Romanian, Russian, Samoan, Serbian, Slovak, Slovenian, Somali, Spanish, Swedish, Tahitian, Tajik, Tajiki, Tamil, Tatar, Telugu, Thai, Tibetan, Tigrinya, Tongan, Turkish, Turkmen, Ukrainian, Upper Sorbian, Urdu, Uyghur, Uzbek, Vietnamese, Welsh. 

The total count of language names is less than 128, as there are some languages for which multiple scripts or variants are supported.

The number of supported languages for three multilingual test data sets, TICO-19, FLORES-101, and FLORES-200, is 37,101,200, respectively. 

The “Translation Initiative for Covid-19” released the TICO-19 dataset. It was a collaborative endeavour between several academic and industrial partners. The benchmark consists of 30 documents translated into 37 target languages from English (3,071 sentences, 69.7k words).

Meta also unveiled their open-source AI model—’ No Language Left Behind‘ (NLLB-200), capable of providing high-quality translations across 200 different languages, validated through extensive evaluations. Meta developed data set FLORES-101 with 3,001 sentences in 842 documents translated from English into 101 target languages. FLORES-200 expands FLORES-101 to 200 target languages and can assess NLLB-200’s performance. The same English source data that FLORES-101 is used to create FLORES-200.