What is Google’s New Benchmark For Efficient Transformers?

Recently, researchers from Google and DeepMind introduced a new benchmark for evaluating the performance and quality of Transformer models, known as Long-Range Arena (LRA). The benchmark specifically focused on evaluating the quality of Transformers under long-context scenarios.

For a few years now, Transformer models have played an important role in NLP development and have gained much traction among researchers and organisations. From images to languages, Transformer models have been used for various reasons and are widely used for tackling various natural language processing (NLP) problems.

However, a prevalent weakness of the Transformer models is their quadratic memory complexity which prevails within the self-attention mechanism. This restricts the potential application of these models to domains that require longer sequence lengths. According to the researchers, Transformers do not scale very well to long sequence lengths largely because of quadratic self-attention complexity. 

Also, researchers around the globe have been trying to mitigate such issues by proposing various efficient Transformers. Moreover, irregular benchmarking on a broad spectrum of tasks as well as datasets makes it challenging to assess relative model quality amongst various models.

Behind Long-Range Arena

Long-Range Arena (LRA) is a systematic and unified benchmark that is specifically focused on evaluating model quality under long-context scenarios. The benchmark is a suite of tasks that consists of sequences ranging from 1K to 16K tokens. 

LRA encompasses an extensive range of data types as well as modalities, such as text, synthetic images, natural images and mathematical expressions that require similarity, structural as well as visual-spatial reasoning. 

LRA includes the task, evaluators and models and is implemented in Python 3. It also implements different variants of Transformer models in JAX, using Flax. For creating the Long-Range Arena benchmark, the researchers created several prerequisites, such as all efficient Transformers models should be applicable to the tasks, the tasks should be difficult enough for current models, the input sequence lengths should be longer in size, among others.  

The researchers evaluated the LRA benchmark on ten of the most recently proposed efficient Transformer models. Besides the standard vanilla Transformer and a simple local attention baseline, they compared the following Transformer models:

• Sparse Transformers 
• Longformers
• Linformers
• Reformers
• Sinkhorn Transformers
• Synthesisers
• BigBird
• Linear Transformers and 
• Performers

Benchmarking the Models

The purpose of LRA is to benchmark sequence models under the long-context scenario. The researchers designed this benchmark suite by compiling both synthetic probing tasks and real-world tasks. 

There are several important tasks in the LRA benchmark that are specifically designed for assessing various aspects of efficient Transformers. They are mentioned below:

• Long ListOps: This benchmark task is a longer variation of the standard ListOps task that is created to examine the parsing ability of neural network models. 
• Byte-Level Text Classification: This benchmark task uses real-world data and represents a common use-case of efficient Transformers that are usually needed to process long documents.
• Byte-Level Document Retrieval: This task is about modelling a similarity score between two documents.
• Image Classification on Sequences of Pixels: This benchmark task is an image classification task, where the inputs are the sequences of pixels.
• Pathfinder (Long- Range Spatial Dependency): It is a synthetic visual task that is used for learning long-range spatial dependencies. 
• Pathfinder-X (Long-Range Spatial Dependencies with Extreme Lengths): This benchmark task checks if the same algorithmic challenges carry a varying extent of difficulty when sequence lengths are longer in size.

Wrapping Up

The researchers stated that this benchmark is an effort toward a systematic evaluation of efficient transformer models. The project aimed at establishing benchmark tasks or datasets using which can be used to evaluate transformer-based models by assessing their generalisation power, computational efficiency, memory foot-print and other such.

They added, “While the focus of this benchmark is the ability of these architectures to reason in long-context scenarios, we are also fundamentally interested in understanding the capabilities and properties of various Transformer architectures when exposed to different types of data and conditions.” And that’s why the benchmark has been purposefully designed to be capability probing, i.e., we select datasets and tasks with certain innate structure.”

The research showed that the LRA model better understands the quality and performance of the above mentioned popular Transformer models. They also mentioned that they are currently working on the second update of this benchmark and will soon release more models and baselines for this benchmark suite.