One of the crucial goals of machine learning (ML) research is to build models that understand and generate natural language efficiently. This has a direct impact on building smart systems for everyday applications, where researchers keep a target of improving the quality of language models. 

Researchers at Google AI in ‘Unifying Language Learning Paradigms’, have presented a language pre-training paradigm called Unified Language Learner (UL2) that focuses on improving the performance of language models across datasets and setups around the world. 

Some of the most common paradigms to build and train language models either use autoregressive decoder-only architectures such as PaLM or GPT-3, where the model is trained to predict the next word for a given phrase. Whereas, other models such as T5, ST-MoE span corruption-based encoder-decoder architectures. However, there remains an opportunity to create an effective unified framework for pre-training models.

According to the company’s blog, the UL2 forms different objective functions for training language models as denoising tasks, where the model has to recover missing sub-sequences of a given input. 

Furthermore, a novel mixture-of-denoisers is used during pre-training – which samples from a varied set of objectives – each with different configurations. The team then demonstrates the models trained using the framework in a variety of language domains that includes models fine-tuned for down-stream tasks and prompt-based few-shot learning. 
Google AI says, “UL2 demonstrates superior performance on a plethora of fine-tuning and few-shot tasks. Additionally, we show that UL2 excels in generation, language understanding, retrieval, long-text understanding and question answering tasks. We publicly release checkpoints of our best performing UL2 model with 20 billion parameters, which we hope will inspire faster progress in developing better language models in the machine learning community as a whole.”