Article complete
AI NewsRetrograde Reinvention

Google Launches T5Gemma to Reclaim Encoder-Decoder Architecture Benefits

Google’s new AI models take a new approach with an older architecture.

Google Launches T5Gemma to Reclaim Encoder-Decoder Architecture Benefits

Key takeaways

  • Google launches T5Gemma, enhancing encoder-decoder architecture for improved language model performance.
  • Adaptation technique allows pretrained decoder-only models to leverage strengths of encoder-decoder frameworks.
  • T5Gemma models outperform decoder-only counterparts in accuracy and latency on benchmark tasks.
  • New model variations optimize performance for specific tasks by balancing encoder and decoder sizes.
  • T5Gemma's development reflects a shift back to traditional architectures to enhance AI model efficiency.

Takeaways drafted with AI from the article text.

Unlike the current trend that favours decoder-only LLMs, T5Gemma revisits the classic encoder-decoder architecture used in models like T5. The company introduced an adaptation technique that converts pretrained decoder-only models into encoder-decoder ones. 

“We study a novel problem: adapting pretrained decoder-only LLMs to encoder-decoder, with the goal of leveraging the strengths of both approaches to achieve a more favourable quality efficiency trade-off,” the research paper mentioned.

The researchers further highlight that this adaptation not only enables inheriting the capability of decoder-only LLMs but also reduces the computational demand compared to pretraining from scratch.

“Can we build top-tier encoder-decoder models based on pretrained decoder-only models? We answer this question by exploring model adaptation,” the company explained in the blog post.

T5Gemma includes both newly trained T5-sized models, ranging from small to XL, and adapted Gemma 2 models with 2B and 9B parameters. It also offers unbalanced combinations such as a 9B encoder with a 2B decoder, aimed at optimising performance for tasks where input understanding is more important than output complexity.

According to benchmark results shared by Google, T5Gemma dominates the quality-inference efficiency Pareto frontier. On SuperGLUE and GSM8K, the models outperform comparable decoder-only models in both accuracy and latency. For example, T5Gemma 9B-9B delivered higher GSM8K accuracy than Gemma 2 9B while maintaining similar latency.

The gains extend beyond pretraining. After instruction tuning, T5Gemma models showed dramatic improvements. The 2B-2B model's MMLU score jumped 12 points, while GSM8K accuracy rose from 58.0% to 70.7%, highlighting the architecture’s responsiveness to fine-tuning.

Google has released a wide range of T5Gemma checkpoints, including pretrained and instruction-tuned variants, with multiple training objectives such as PrefixLM and UL2. 

The models are now available on Hugging Face, Kaggle, and Vertex AI for further experimentation and deployment.

Published

Corrections: AIM corrects errors of fact and notes the change on the story. Spotted one? Tell the editors. Read our editorial standards.

More in this story

Google · 5 stories
  1. Google's Artificial Intelligence Successfully Spawns An AI 'Child' That Easily Outperforms Man-Made AI
  2. John Giannandrea Steps Down As Google AI And Search Head
  3. Google Cloud Launches Digital Twin For Supply-Chain Management
  4. You are here: Google Launches T5Gemma to Reclaim Encoder-Decoder Architecture Benefits
  5. Gemini Introduces Personal Intelligence to Connect Gmail, Photos and Other Google Apps
About the author

Editor

I am a tech aficionado and a computer science graduate with a keen interest in AI, Coding, Open Source, Global SaaS, and Cloud. Have a tip? Reach out to

All stories
Belamy · Daily

The day’s AI news, before your first meeting.