- Published on April 19, 2022
- In AI Features
The rise of decoder-only Transformer models
Apart from the various interesting features of this model, one feature that catches the attention is its decoder-only architecture. In fact, not just PaLM, some of the most popular and widely used language models are decoder-only.
- By Shraddha Goled
Recently, Google's team introduced PaLM, a 540 billion parameter dense decoder-only Transformer model that is trained with Google's own Pathway systems. The researchers were able to demonstrate that the model could achieve state-of-the-art few-shot performance across most tasks, in some cases, by a significant margin.
Apart from the various interesting features of this model, one feature that catches the attention is its decoder-only architecture. In fact, not just PaLM, some of the most popular and widely used language models are decoder-only.
https://twitter.com/LiamFedus/status/1514244515078365186?s=20&t=vrLvnsYioG-I86JuJamyEA
Decoder-only models
In the last few years, large neural networks have achieved impressive results across a wide range of tasks. Models lik
📣 Want to advertise in AIM? Book here
Shraddha Goled
I am a technology journalist with AIM. I write stories focused on the AI landscape in India and around the world with a special interest in analysing its long term impact on individuals and societies. Reach out to me at shraddha.goled@analyticsindiamag.com.
Related Posts
Fighting Deepfakes May Not Be a Technology Problem
Defenders must be active at all times, while attackers need only one opportunity.
India’s Data Centre Expansion Is Decentralising
Without compute buildup beyond metros, the next wave of digital adoption will be constrained
How Mumbai Keeps Winning India’s Data-Centre Race
Land prices are among the highest in the country, but total build economics remain competitive by global standards.
From Shortages to Scale, io.net’s Approach to Rewriting AI Compute Access
A decentralised GPU marketplace may scale AI compute faster than traditional clouds, as GPU demand towers over supply
Why Deloitte Built a Tax AI That Knows When to Say ‘I Don’t Know’
The company has launched an agentic AI platform for tax research that’s targeting something radical in a conservative profession.
Can India’s AI Copyright Plan Survive Legal and Technical Scrutiny?
India’s ambitious proposal for a single mandatory AI training licence faces feasibility, legal and innovation concerns.
2026 Could be India’s Year in AI, But Only the Resilient Will Survive
“This isn’t a freeze. It’s a filter… By 2026, the real signal will be resilience, not rhetoric.”
Enterprises Won’t Choose Sovereign Models For Patriotism’s Sake
The first set of sovereign models is aimed at Indic languages and national datasets. Their value lies in cultural nuance.
Download the easiest way to
stay informed
stay informed
