What is Google’s New Benchmark For Efficient Transformers?

Advertisement

Recently, researchers from Google and DeepMind introduced a new benchmark for evaluating the performance and quality of Transformer models, known as Long-Range Arena (LRA). The benchmark specifically focused on evaluating the quality of Transformers under long-context scenarios.

For a few years now, Transformer models have played an important role in NLP development and have gained much traction among researchers and organisations. From images to languages, Transformer models have been used for various reasons and are widely used for tackling various natural language processing (NLP) problems.

THE BELAMY

Sign up for your weekly dose of what's up in emerging technology.

However, a prevalent weakness of the Transformer models is their quadratic memory complexity which prevails within the self-attention mechanism. This restricts the potential application of these models to domains that require longer sequence lengths. According to the researchers, Transformers do not scale very well to long sequence lengths largely because of quadratic self-attention complexity. 

Also, researchers around the globe have been trying to mitigate such issues by proposing various efficient Transformers. Moreover, irregular benchmarking on a broad spectrum of tasks as well as datasets makes it challenging to assess relative model quality amongst various models.

Behind Long-Range Arena

Long-Range Arena (LRA) is a systematic and unified benchmark that is specifically focused on evaluating model quality under long-context scenarios. The benchmark is a suite of tasks that consists of sequences ranging from 1K to 16K tokens. 

LRA encompasses an extensive range of data types as well as modalities, such as text, synthetic images, natural images and mathematical expressions that require similarity, structural as well as visual-spatial reasoning. 

LRA includes the task, evaluators and models and is implemented in Python 3. It also implements different variants of Transformer models in JAX, using Flax. For creating the Long-Range Arena benchmark, the researchers created several prerequisites, such as all efficient Transformers models should be applicable to the tasks, the tasks should be difficult enough for current models, the input sequence lengths should be longer in size, among others.  

The researchers evaluated the LRA benchmark on ten of the most recently proposed efficient Transformer models. Besides the standard vanilla Transformer and a simple local attention baseline, they compared the following Transformer models:

  • Sparse Transformers 
  • Longformers
  • Linformers
  • Reformers
  • Sinkhorn Transformers
  • Synthesisers
  • BigBird
  • Linear Transformers and 
  • Performers

Benchmarking the Models

The purpose of LRA is to benchmark sequence models under the long-context scenario. The researchers designed this benchmark suite by compiling both synthetic probing tasks and real-world tasks. 

There are several important tasks in the LRA benchmark that are specifically designed for assessing various aspects of efficient Transformers. They are mentioned below:

  • Long ListOps: This benchmark task is a longer variation of the standard ListOps task that is created to examine the parsing ability of neural network models. 
  • Byte-Level Text Classification: This benchmark task uses real-world data and represents a common use-case of efficient Transformers that are usually needed to process long documents.
  • Byte-Level Document Retrieval: This task is about modelling a similarity score between two documents.
  • Image Classification on Sequences of Pixels: This benchmark task is an image classification task, where the inputs are the sequences of pixels.
  • Pathfinder (Long- Range Spatial Dependency): It is a synthetic visual task that is used for learning long-range spatial dependencies. 
  • Pathfinder-X (Long-Range Spatial Dependencies with Extreme Lengths): This benchmark task checks if the same algorithmic challenges carry a varying extent of difficulty when sequence lengths are longer in size.

Wrapping Up

The researchers stated that this benchmark is an effort toward a systematic evaluation of efficient transformer models. The project aimed at establishing benchmark tasks or datasets using which can be used to evaluate transformer-based models by assessing their generalisation power, computational efficiency, memory foot-print and other such.

They added, “While the focus of this benchmark is the ability of these architectures to reason in long-context scenarios, we are also fundamentally interested in understanding the capabilities and properties of various Transformer architectures when exposed to different types of data and conditions.” And that’s why the benchmark has been purposefully designed to be capability probing, i.e., we select datasets and tasks with certain innate structure.”

The research showed that the LRA model better understands the quality and performance of the above mentioned popular Transformer models. They also mentioned that they are currently working on the second update of this benchmark and will soon release more models and baselines for this benchmark suite.

More Great AIM Stories

Ambika Choudhury
A Technical Journalist who loves writing about Machine Learning and Artificial Intelligence. A lover of music, writing and learning something out of the box.

Our Upcoming Events

Conference, in-person (Bangalore)
MachineCon 2022
24th Jun

Conference, Virtual
Deep Learning DevCon 2022
30th Jul

Conference, in-person (Bangalore)
Cypher 2022
21-23rd Sep

3 Ways to Join our Community

Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

Telegram Channel

Discover special offers, top stories, upcoming events, and more.

Subscribe to our newsletter

Get the latest updates from AIM
MORE FROM AIM
Sreejani Bhattacharyya
Why is edtech falling first?

With the lockdown being imposed due to the COVID-19 pandemic and schools being shut down, the edtech startups witnessed some of their best times during 2020 and 2021.