Transformers Can Now Work Pixel by Pixel, Says Meta AI’s New Study

A latest research by Meta AI and the University of Amsterdam have shown that transformers, a popular neural network architecture, can operate directly on individual pixels of an image without relying on the locality inductive bias present in most modern computer vision models.

The study, exploring “Transformers on Individual Pixels," challenges the long-held belief that locality – the notion that neighboring pixels are more related than distant ones – is a fundamental requirement for vision tasks.

Traditionally, computer vision architectures like Convolutional Neural Networks (ConvNets) and Vision Transformers (ViTs) have incorporated locality bias through techniques such as convolutional kernels, pooling operations, and patchification, assuming neighboring pixels are more related. 

However, researchers introduced Pixel Transformers (PiTs), which treat each pixel as an individual token, removing any assumptions about the 2D grid structure of images. Surprisingly, PiTs achieved highly performant results across various tasks.

Following the architecture of Diffusion Transformers (DiTs), PiTs operating on latent token spaces from VQGAN achieved better quality metrics like Fréchet Inception Distance (FID) and Inception Score (IS) than their locality-biased counterparts.

Perceiver IO Transformers (PiTs) are computationally expensive due to longer sequences, but they challenge the need for locality bias in vision models. Advances in handling large sequence lengths may make PiTs more practical. 

The study highlights reducing inductive biases in neural architectures, potentially leading to more versatile and capable systems for diverse vision tasks and data modalities.

Image generation using transformers

There are different models for image generation, such as Midjourney, Stable Diffusion, and Invoke, whose images can be reimagined with these technologies. Recently Midjourney has released the new feature "Character Reference" claiming to generate consistent characters across multiple reference images.

Stability AI announced Stable Diffusion 3, the most capable text-to-image model, featuring significantly enhanced performance in multi-subject prompts, image quality, and spelling abilities.