Active Hackathon

Google introduces Pathways Language Model with 540 Bn parameters

PaLM achieves a training efficiency of 57.8% hardware FLOPs utilisation, the highest yet achieved for LLMs at this scale

Google AI had introduced the Pathways Language Model (PaLM), a 540-billion parameter, dense decoder-only Transformer model trained with the Pathways system used to train a single model across multiple TPU v4 Pods. The researchers evaluated PaLM on hundreds of language understanding and generation tasks and achieved state-of-the-art few-shot performance across most tasks, by significant margins in many cases.

PaLM achieves a training efficiency of 57.8% hardware FLOPs utilisation, the highest yet achieved for LLMs at this scale, thanks to a combination of the parallelism strategy and a reformulation of the Transformer block that allows for attention and feedforward layers to be computed in parallel, enabling speedups from TPU compiler optimisations.

THE BELAMY

Sign up for your weekly dose of what's up in emerging technology.

PaLM was trained using a combination of English and multilingual datasets that include high-quality web documents, books, Wikipedia, conversations, and GitHub code. The researchers also created a “lossless” vocabulary that preserves all whitespace (especially important for code), splits out-of-vocabulary Unicode characters into bytes, and splits numbers into individual tokens, one for each digit.

PaLM showed breakthrough capabilities on numerous difficult tasks. When tested against other language models, PaLM 540B surpassed few-shot performance on language understanding and generation when evaluated on 29 widely-used English natural language processing (NLP) tasks. In addition, PaLM demonstrated impressive natural language understanding and generation capabilities on several BIG-bench tasks.
PaLM exhibited breakthrough capabilities on reasoning tasks that require multi-step arithmetic or common-sense reasoning. Prior LLMs, like Gopher, saw less benefit from model scale in improving performance.

More Great AIM Stories

Kartik Wali
A writer by passion, Kartik strives to get a deep understanding of AI, Data analytics and its implementation on all walks of life. As a Senior Technology Journalist, Kartik looks forward to writing about the latest technological trends that transform the way of life!

Our Upcoming Events

Conference, Virtual
Genpact Analytics Career Day
3rd Sep

Conference, in-person (Bangalore)
Cypher 2022
21-23rd Sep

Conference, in-person (Bangalore)
Machine Learning Developers Summit (MLDS) 2023
19-20th Jan

Conference, in-person (Bangalore)
Data Engineering Summit (DES) 2023
21st Apr, 2023

3 Ways to Join our Community

Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

Telegram Channel

Discover special offers, top stories, upcoming events, and more.

Subscribe to our newsletter

Get the latest updates from AIM
MOST POPULAR

The curious case of Google Cloud revenue

Porat had earlier said that Google Cloud was putting in money to make more money, but even with the bucket-loads of money that it was making, profitability was still elusive.

Global Parliaments can do much more with Artificial Intelligence

The world is using AI to enhance the performance of its policymakers. India, too, has launched its own machine learning system NeVA, which at the moment is not fully implemented across the nation. How can we learn and adopt from the advancement in the Parliaments around the world? 

Why IISc wins?

IISc was selected as the world’s top research university, trumping some of the top Ivy League colleges in the QS World University Rankings 2022

[class^="wpforms-"]
[class^="wpforms-"]