Finally, A BetterTransformer

To use BetterTransformer, install PyTorch 1.12 and start using high-quality, high-performance Transformer models with the PyTorch API today.
Listen to this story

The newly released PyTorch 1.12 has introduced BetterTransformer which implements a backwards-compatible fastpath of torch.nn.TransformerEncoder for Transformer Encoder Inference. It helps in 2x in speedup and throughput for many common execution scenarios.

Image: Transformer Encoder architecture

BetterTransformer launches with accelerated native implementations of MultiHeadAttention and TransformerEncoderLayer for CPUs and GPUs. These fast paths are integrated into the standard PyTorch Transformer APIs and help in accelerating TransformerEncoder, TransformerEncoderLayer and MultiHeadAttention nn.modules. 

THE BELAMY

Sign up for your weekly dose of what's up in emerging technology.

Source: pytorch.org

These new modules implement two types of optimizations:


Download our Mobile App



  1. Fused kernels combine multiple individual operators normally used to implement Transformers to provide a more efficient implementation.
  2. Take advantage of sparsity in the inputs to avoid performing unnecessary operations on padding tokens. Padding tokens frequently account for a large fraction of input batches in many Transformer models used for Natural Language Processing.

Backwards Compatibility

Advantageously, BetterTransformer does not need any model change. To benefit from fast path execution, inputs and operating conditions must satisfy some access conditions. While the internal implementation of Transformer APIs has changed, PyTorch 1.12 maintains strict compatibility with Transformer modules shipped in previous versions, enabling PyTorch users to use models created and trained with previous PyTorch releases while benefiting from BetterTransformer improvements.
For more details, click here

More Great AIM Stories

Kartik Wali
A writer by passion, Kartik strives to get a deep understanding of AI, Data analytics and its implementation on all walks of life. As a Senior Technology Journalist, Kartik looks forward to writing about the latest technological trends that transform the way of life!

AIM Upcoming Events

Early Bird Passes expire on 3rd Feb

Conference, in-person (Bangalore)
Rising 2023 | Women in Tech Conference
16-17th Mar, 2023

Conference, in-person (Bangalore)
Data Engineering Summit (DES) 2023
27-28th Apr, 2023

3 Ways to Join our Community

Telegram group

Discover special offers, top stories, upcoming events, and more.

Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

Subscribe to our Daily newsletter

Get our daily awesome stories & videos in your inbox
AIM TOP STORIES

Is AI sexist?

Genderify, launched in 2020, determines the gender of a user by analysing their name, username and email address using AI.