Google introduces Pathways Language Model with 540 Bn parameters

PaLM achieves a training efficiency of 57.8% hardware FLOPs utilisation, the highest yet achieved for LLMs at this scale

Google AI had introduced the Pathways Language Model (PaLM), a 540-billion parameter, dense decoder-only Transformer model trained with the Pathways system used to train a single model across multiple TPU v4 Pods. The researchers evaluated PaLM on hundreds of language understanding and generation tasks and achieved state-of-the-art few-shot performance across most tasks, by significant margins in many cases.

PaLM achieves a training efficiency of 57.8% hardware FLOPs utilisation, the highest yet achieved for LLMs at this scale, thanks to a combination of the parallelism strategy and a reformulation of the Transformer block that allows for attention and feedforward layers to be computed in parallel, enabling speedups from TPU compiler optimisations.

PaLM was trained using a combination of English and multilingual datasets that include high-quality web documents, books, Wikipedia, conversations, and GitHub code. The researchers also created a “lossless” vocabulary that preserves all whitespace (especially important for code), splits out-of-vocabulary Unicode characters into bytes, and splits numbers into individual tokens, one for each digit.


Sign up for your weekly dose of what's up in emerging technology.

PaLM showed breakthrough capabilities on numerous difficult tasks. When tested against other language models, PaLM 540B surpassed few-shot performance on language understanding and generation when evaluated on 29 widely-used English natural language processing (NLP) tasks. In addition, PaLM demonstrated impressive natural language understanding and generation capabilities on several BIG-bench tasks.
PaLM exhibited breakthrough capabilities on reasoning tasks that require multi-step arithmetic or common-sense reasoning. Prior LLMs, like Gopher, saw less benefit from model scale in improving performance.

More Great AIM Stories

Kartik Wali
A writer by passion, Kartik strives to get a deep understanding of AI, Data analytics and its implementation on all walks of life. As a Senior Technology Journalist, Kartik looks forward to writing about the latest technological trends that transform the way of life!

Our Upcoming Events

Conference, in-person (Bangalore)
Machine Learning Developers Summit (MLDS) 2023
19-20th Jan, 2023

Conference, in-person (Bangalore)
Rising 2023 | Women in Tech Conference
16-17th Mar, 2023

Conference, in-person (Bangalore)
Data Engineering Summit (DES) 2023
27-28th Apr, 2023

Conference, in-person (Bangalore)
MachineCon 2023
23rd Jun, 2023

Conference, in-person (Bangalore)
Cypher 2023
20-22nd Sep, 2023

3 Ways to Join our Community

Whatsapp group

Discover special offers, top stories, upcoming events, and more.

Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

Subscribe to our newsletter

Get the latest updates from AIM