Perceiver IO: a scalable, fully-attentional model that works on any modality

TLDR

We've added Perceiver IO to Transformers, the first Transformer-based neural network that works on all kinds of modalities (text, images, audio, video, point clouds,...) and combinations thereof. Take a look at the following Spaces to view some examples:

• predicting optical flow between images
• classifying images.

We also provide several notebooks.

Below, you can find a technical explanation of the model.

Introduction

The Transformer, originally introduced by Vaswani et al. in 2017, caused a revolution in the AI community, initially improving state-of-the-art (SOTA) results in machine translation. In 2018, BERT was released, a Transformer encoder-only model that crushed the benchmarks of natural language processing (NLP), most famously the GLUE benchmark.

Not long after that, AI researchers started to apply the idea of BERT to other domains. To name a few examples:

• Wav2Vec2 by Facebook AI illustrated that the architecture could be extended to audio
• the Vision Transformer (ViT) by Google AI showed that the architecture works really well for vision
• most recently the Video Vision transformer (ViViT), also by Google AI, applied the architecture to video.

In all of these domains, state-of-the-art results were improved dramatically, thanks to the combination of this powerful architecture with large-scale pre-training.

The Perceiver

In the following section, we look in a bit more detail at how Perceiver IO actually works by going over its implementation in HuggingFace Transformers, a popular library that initially implemented Transformer-based models for NLP, but is now starting to implement them for other domains as well. In the sections below, we explain in detail - in terms of shapes of tensors - how the Perceiver actually pre and post processes modalities of any kind.

All Perceiver variants in HuggingFace Transformers are based on the PerceiverModel class. To initialize a PerceiverModel, one can provide 3 additional instances to the model:

The Perceiver architecture.