More than English: NLP Datasets have a Language Overfitting Problem

Up until a few years back, NLP research was predominantly focused on English. With time however there is an understanding that the benefits of including non-English languages in NLP datasets outweigh the challenges in developing them. The argument often levelled against having multilingual datasets in NLP was that English was the language of the Internet. After English, the second most spoken language is Mandarin Chinese—still there's a difference of 300 million users between the two. 

Need for multilingual datasets in NLP

Things have changed vastly now. Enterprise customers and brands have started to recognise the economic gains that come with catering to markets that may not speak the same language across. The number of these brands are only growing with time. These big tech companies usually serve consumers that cover more than 20 languages, focusing heavily on the Nordic regions, Latin America and Asia-Pacific. And this is merely a global breakdown—countries like India speak 22 languages which can then be divided into subgroups of different dialects.

An obvious advantage to this from a social perspective is that to make technology accessible to minority groups that are usually ignored and lack representation online. A user’s access to technology is normally restrained by their knowledge of English. At times, even when the user is English-speaking, biased algorithms discriminate against users with different accents. The reason behind this language divide is that earlier companies were unwilling to spend on support for low-resource languages—even for spell checking and keyboard support. 

How did NLP datasets become overfitted in English? 

Aside from the virtuosity, there are other advantages to including more non-English datasets from the ML perspective, the foremost of which is to avoid overfitting. Model architectures are encoded with the assumptions based on the data it is applied on. Despite the fact that these models claim to be generalised, a bunch of inductive biases they have are very specific to English. This one-sided approach will become more and more apparent as NLP grows to engage with processes meant for a global user base. 

English and other languages similar to English are not representative of the other so-called foreign languages with unique structures and grammar that are pertinent only to them. The repeated support for English, has led to a normative bias which has made it the default language for technologies. Sufficient past research concludes that a model is not exactly language agnostic just because it doesn’t have explicitly encoded information. 

The 2011 research paper titled, ‘On Achieving and Evaluating Language-Independence in NLP’ by Emily M. Bender gives the example of n-gram language models which performed slightly worse at studying languages which were ‘more complex’ or had a comparatively more free word order than English. 

Another research by Reut Tsarfaty published in 2020, titled, ‘From SPMRL to NMRL: What Did We Learn (and Unlearn) in a Decade of Parsing Morphologically-Rich Languages (MRLs)?’ discussed how neural networks tend to overlook the complexities of rich languages. 

According to a 2017 study by Claria Vania and Adam Lopez, called ‘From Characters to Words to in Between: Do We Capture Morphology?’ models are poor at the process of subword tokenisation for languages with reduplication. Another study, ‘What Kind of Language Is Hard to Language-Model?’ authored by Sabrina J. Mielke, that came out in 2019, noted that languages which have larger vocabularies are inherently harder to process for language models. 

How to improve multilingual speech research?

Owing to the varied-level of complications these challenges in model structuring have posed, researchers are being forced to change their approach to embrace multilingual qualities. This was a two-pronged approach: changing the pre-training process for NLP models building bigger datasets for multiple languages. 

The paper, ‘Common Sense Beyond English: Evaluating and Improving Multilingual Language Models for Commonsense Reasoning,’ published in 2021 determined this weakness and introduced a process called ‘multilingual contrastive pre-training’ or MCP. MCP performed the task of selecting the apt assertion in a set of ‘N’ contrastive assertions among ‘N’ different languages. 

While there has been marked improvement in pre-training language models, there is a scarcity of huge amounts of unlabelled data for many non-English languages. Companies have now taken note of this chasm. 

https://youtu.be/uCxSPPiwrNE

Release of multilingual datasets

Early this year in January, Meta AI released Multilingual LibriSpeech or MLS, an open source data set with more than 50,000 hours of audio for eight languages including English, German, Dutch, French, Spanish, Italian, Portuguese and Polish. MLS is designed to help the speech research community’s work in languages beyond just English so people around the world can benefit from improvements in a wide range of AI-powered services.

The research also contained the datasets and the pre-trained language models along with the baseline results for researchers to compare between different Automatic Speech Recognition (ASR) systems. These were just the first baby steps taken by Meta. 

As a part of their ‘No Model Left Behind’ project, in August, Meta AI open sourced NLLB-200, a huge AI model that translated between more than 200 languages. The model contains 54.5 billion parameters that were trained on a dataset with more than 18 billion sentence pairs. 

Other giant tech organisations that can afford the investment needed to build these datasets have also chimed in since then. In April this year, Amazon has open sourced a new multilingual dataset called MASSIVE that comprises a million labelled voice notes that cover 51 languages. The research also gave examples of how to use the dataset for modelling while allowing researchers to re-create the baseline results produced in the paper.