Advertisement

Google Releases New Language Model That Kicks GPT-3’s Butt

Google AI’s GLaM model achieves competitive results on zero-shot and one-shot learning

Google has introduced the Generalist Language Model (GLaM). It is a trillion weight model that uses sparsity. It not only makes it more efficient in terms of training and serving, but it also achieves a competitive advantage on multiple few-shot learning tasks. In terms of performance, GLaM demonstrates improved learning efficiency across 29 public NLP benchmarks in seven categories like language completion, open domain question answering, and inference tasks.

Over the past few years, leading AI institutes and tech companies have been releasing several language models – each bigger and more advanced than the previous. GPT-3’s launch was no less than a watershed moment in this space – never had the world seen such a large model with 175B parameters. GPT-3 and other similar models can perform tasks like few-shot learning across a wide array of tasks, including reading comprehension and question answering with very few or no training examples with much ease.

That said, this innovation and superior performance come at a cost. They are computationally intensive and have adverse effects on the environment. Researchers are now working to develop models that can be trained and used more efficiently.

THE BELAMY

Sign up for your weekly dose of what's up in emerging technology.

To build GLaM, Google’s team built a high-quality 1.6 trillion token dataset that contains language usage representative of a wide range of use cases For each token, the gating network selects the two most appropriate experts to process the data. The full version of GLaM has 1.2 trillion total parameters across 64 experts per MoE layer with 32 MoE layers in total, but only activates a subnetwork of 97 billion (8% of 1.2 trillion) parameters per token prediction during inference. Each input token is dynamically routed to two selected expert networks out of 64 for prediction. 

Credit: Google AI

It is a mixture of experts (MoE) model, which means that it has different submodels that are specialised for different inputs. Experts in each layer are controlled by a gating network. They activate experts based on the input data. The gating network selects two most appropriate experts to process the data for each token. The full version of GLaM has 1.2 trillion total parameters across 64 experts per MoE layer with 32 MoE layers in total, but only activates a subnetwork of 97 billion (8% of 1.2 trillion) parameters per token prediction during inference. Compared with Megatron-Turing model, GLaM is on-par on the seven respective tasks if using a 5 percent margin, while using 5x less computation during inference.


Download our Mobile App



More Great AIM Stories

Meeta Ramnani
Meeta’s interest lies in finding out real practical applications of technology. At AIM, she writes stories that question the new inventions and the need to develop them. She believes that technology has and will continue to change the world very fast and that it is no more ‘cool’ to be ‘old-school’. If people don’t update themselves with the technology, they will surely be left behind.

AIM Upcoming Events

Early Bird Passes expire on 3rd Feb

Conference, in-person (Bangalore)
Rising 2023 | Women in Tech Conference
16-17th Mar, 2023

Conference, in-person (Bangalore)
Data Engineering Summit (DES) 2023
27-28th Apr, 2023

3 Ways to Join our Community

Telegram group

Discover special offers, top stories, upcoming events, and more.

Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

Subscribe to our Daily newsletter

Get our daily awesome stories & videos in your inbox
AIM TOP STORIES