# SparseFormer: Sparse Visual Recognition via Limited Latent Tokens

Ziteng Gao1 Zhan Tong2 Limin Wang3 Mike Zheng Shou1 ✉
1Show Lab, National University of Singapore 2Tencent AI Lab 3Nanjing
University

###### Abstract

Human visual recognition is a _sparse_ process, where only a few salient
visual cues are attended to rather than traversing every detail uniformly.
However, most current vision networks follow a _dense_ paradigm, processing
every single visual unit (_e.g_., pixel or patch) in a uniform manner. In this
paper, we challenge this dense paradigm and present a new method, coined
_SparseFormer_ , to imitate human’s _sparse_ visual recognition in an end-to-
end manner. SparseFormer learns to represent images using a highly limited
number of tokens (down to $49$) in the latent space with sparse feature
sampling procedure instead of processing dense units in the original pixel
space. Therefore, SparseFormer circumvents most of dense operations on the
image space and has much lower computational costs. Experiments on the
ImageNet classification benchmark dataset show that SparseFormer achieves
performance on par with canonical or well-established models while offering
better accuracy-throughput tradeoff. Moreover, the design of our network can
be easily extended to the video classification with promising performance at
lower computational costs. We hope that our work can provide an alternative
way for visual modeling and inspire further research on sparse neural
architectures. The code will be publicly available at
https://github.com/showlab/sparseformer.

†† ✉: Corresponding Author.

##  1 Introduction

Designing neural architectures for visual recognition has long been an
appealing yet challenging topic in the computer vision community.
Convolutional neural networks (CNNs) [28, 18, 20, 42, 33] use convolutional
filters on every unit of images or feature maps to build features. Recently
proposed vision transformers [11, 54, 32, 60] use attention operation on each
patch or unit to dynamically interact with other units to mimic human
attention mechanism. Both convolution-based and Transformer-based
architectures need to traverse every unit, like pixel or patch on the grid
map, to densely perform operations. This dense per-unit traversal originates
from sliding windows [39], reflecting the assumption that foreground objects
may appear uniformly with respect to spatial locations in an image.

Figure 1: Dense versus our proposed sparse paradigm for visual recognition.
The dense paradigm requires traversing $H\times W$ units to perform
convolution or attention, while our proposed sparse paradigm performs
transformers over only $N$ latent tokens where $N\ll H\times W$.

However, as humans, we do not need to examine every detail in a scene to
recognize it. Instead, we are fast enough to first roughly find discriminative
regions of interest with several glimpses and then recognize textures, edges,
and high-level semantics within these regions [10, 37, 23, 46]. This contrasts
greatly with existing visual networks, where the convention is to exhaustively
traverse every visual unit. The dense paradigm incurs soaring computational
costs with larger input resolutions and does not directly provide details
about what a vision model is looking at in an image.

In this paper, we propose a new vision architecture, coined SparseFormer, to
explore _sparse visual recognition_ by explicitly imitating the perception
mechanism of human eyes. Specifically, SparseFormer learns to represent an
image by latent transformers along with a highly limited number of tokens
(_e.g_., down to $49$) in the latent space _from the very beginning_. Each
latent token is associated with a region of interest (RoI) descriptor, and the
token RoI can be refined across stages. Given an image, SparseFormer first
extracts image features by a lightweight early convolution module. A latent
focusing transformer adjusts token RoIs to focus on foregrounds and sparsely
extracts image features according to these token RoIs to build latent token
embeddings in an iterative manner. SparseFormer then feeds tokens with these
region features into a larger and deeper network, _i.e_. a standard
transformer encoder in the latent space, to enable precise recognition. The
transformer operations are only applied over the limited tokens in the latent
space. Since the number of latent tokens is _highly limited_ (_e.g_., $49$)
and the feature sampling procedure is _sparse_ (_i.e_., based on direct
bilinear interpolation), it is reasonable to call our architecture a _sparse_
approach for visual recognition. The overall computational cost of
SparseFormer is _almost irrelevant_ to the input resolution except for the
early convolution part, which is lightweight in our design. Moreover,
SparseFormer can be supervised _solely with classification signals_ in an
_end-to-end_ manner without requiring separate prior training with localizing
signals.

As an initial step to sparse visual recognition, the aim of SparseFormer is to
explore an alternative paradigm for vision modeling, rather than state-of-the-
art results with bells and whistles. Nonetheless, SparseFormer still delivers
very promising results on the challenging ImageNet classification benchmark,
on par with dense counterparts but at lower computational costs. Since most
operators of SparseFormer are performed on tokens in the latent space rather
than the dense image space, and the number of tokens is limited, the memory
footprints are lower and throughputs are higher compared to dense
architectures, yielding a better accuracy-throughput trade-off, especially in
the low-compute region. For instance, with ImageNet 1K training, the tiny
variant of SparseFormer with $2.0$G FLOPs gives $81.0$ top-1 accuracy at a
throughput of $1270$ images/s, while Swin-T with $4.5$G FLOPs achieves $81.3$
at a lower throughput of $726$ images/s. Visualizations of SparseFormer
validate its ability to differentiate foregrounds from backgrounds in an end-
to-end manner using only classification signals. Furthermore, we also
investigate several scaling-up strategies of SparseFormer for better
performance.

The simple design of SparseFormer can also be easily extended to video
classification, which is more data-intensive and computationally expensive for
dense vision models but is suitable for the SparseFormer architecture.
Experimental results on the challenging video classification Kinetics-400 [7]
benchmark show that our extension of SparseFormer in video classification
yields promising performance with lower computation than dense architectures.
This highlights the efficiency and effectiveness of the proposed sparse vision
architecture with denser input data.

##  2 Related Work

Convolutional neural networks. AlexNet [28] pioneered convolutional neural
networks for general visual recognition by using stacked local convolutions on
dense pixels to build semantic features. Since then, CNNs have dominated
visual understanding with improvement on several aspects, _e.g_., pathway
connections [18, 20], convolutional filter configurations [42, 53, 33], multi
branches [52, 51], normalizations [22, 63]. To cover discriminative details,
all convolutional networks require dense convolution applied over pixels.
Although pooling layers are used to downsize feature maps spatially,
exhaustive convolution over dense units still slows down networks,
particularly for large inputs. The case becomes more severe when dealing with
data-intensive video inputs, as the dense paradigm for videos introduces a
significantly heavier computational burden [55, 7, 7]. Even with adopted
factorized convolution [56, 41, 67] or sampling strategies [59, 14], the
computational burden is still significant.

(Efficient) vision transformers. Recently proposed Vision Transformer (ViT)
[11] makes use of a fully Transformer-based architecture originating in NLP
[58] to enable visual recognition with a larger model capacity. ViT first
splits an input image into patches with a moderate number (_e.g_., 196) and
feeds these patches as tokens into a standard Transformer for recognition.
Attention operators inside ViT learn dynamic and long-term relationships
between patches with a minimum of inductive bias. This makes ViT a model with
greater capacity and friendly for a dataset of massive samples like JFT [49].
Since ViT, attention-based methods have deeply reformed vision understanding,
and many works attempted to find better attention operators for computer
vision [32, 69, 60, 12, 17, 64, 57]. Among these, Swin Transformer [32]
introduces shifting windows for vision transformers and truncates the
attention range inside these windows for better efficiency. Reducing the
number of tokens in the attention operator in the Pyramid or Multi-Scale
Vision Transformers (PVT or MViT) [60, 12] also maintains a better trade-off
between performance and computation.

However, despite their ability to stress semantic visual areas dynamically,
attention in vision transformers still requires dense per-unit modeling and
also introduces computation up to the quadratic level with respect to input
resolution. Zhang et al. [70] use the sparse attention variant first proposed
in NLP [3, 27] to speed up vision transformers for larger inputs. Another
attempts for faster vision transformer inference is to keep only
discriminative tokens and reduce the number of tokens at the inference stage
via various strategies and patch score criteria [30, 13, 5, 44, 68].

In contrast to sparsifying attention operators or reducing the number of
visual tokens in pre-trained models at the inference stage, SparseFormer
learns a highly limited number of latent tokens to represent an image and
performs transformers solely on these tokens in the latent space efficiently.
Since the number of latent tokens is highly constrained and irrelevant to the
input resolution, the computational cost of SparseFormer is low, controllable,
and practical.

Perceiver architectures and detection transformers. Perceiver architectures
[25, 24] aim to unify different modal inputs using latent transformers. For
image inputs, one or more cross attentions are deployed to transform image
grid-form features into latent tokens, and then several Transformer stages are
applied to these latent tokens. Our proposed SparseFormer is greatly inspired
by this paradigm, where sparse feature sampling can be regarded as “cross
attention” between the image space and the latent space. However, SparseFormer
differs from Perceiver architectures in several aspects: First, in
SparseFormer, the “cross attention” or sparse feature sampling is performed
sparsely and based on direct bilinear interpolation. SparseFormer does not
need to traverse every pixel in an image to build a latent token, while
Perceiver architectures require an exhausting traversal. Second, the number of
latent tokens in Perceiver is large (_i.e_., 512). In contrast, the number of
tokens in SparseFormer is much smaller, down to $49$, only $0.1\times$ times
compared to Perceiver. As a result, the proposed SparseFormer enjoys better
efficiency than Perceiver.

Recently proposed detection transformers (DETR models) [6, 72, 50] also use
cross attention to directly represent object detections by latent tokens from
encoded image features. SparseFormer shares similar ideas with DETR to
represent an image by latent tokens but in a more efficient manner.
SparseFormer does not require a heavy CNN backbone or a Transformer encoder to
encode image features first. The “cross attention” layer in SparseFormer is
based on the image bilinear interpolation and is also efficient, inspired by
the adaptive feature sampling in AdaMixer [15] as a detection transformer.
Furthermore, SparseFormer does not require localization signals in object
detection, and it can be optimized to spatially focus tokens in an end-to-end
manner solely with classification signals.

Glimpse models. In the nascent days of neural vision understanding research,
glimpse models [38, 43] are proposed to imitate human visual perception by
capturing several glimpses over an image for recognition. These glimpse models
only involve computation over limited parts of an image, and their computation
is fixed. While glimpse models are efficient, they are commonly non-
differentiable regarding where to look and necessitate workarounds like the
expectation-maximization algorithm [8] or reinforcement learning to be
optimized. Furthermore, the performance of glimpse models has only been
experimented with promising results on rather small datasets, like MNIST [29],
and the potential to extend to larger datasets has not yet been proven.

Our presented SparseFormer architecture can be seen as an extension of glimpse
models, where multiple glimpses (latent tokens) are deployed in each stage.
Since bilinear interpolation is adopted, the entire pipeline can be optimized
in an end-to-end manner. Furthermore, the proposed method has been shown to be
empirically effective on large benchmarks.

##  3 SparseFormer

In this section, we describe the SparseFormer architecture in detail: we first
introduce SparseFormer as a method for the image classification task and then
we discuss how to extend it to video classification with minimal additional
efforts. As SparseFormer performs most of the operations over tokens in the
latent space, we begin with the definition of latent tokens. Then we discuss
how to build latent tokens and how to perform recognition based on them.

Figure 2: The overall architecture of the proposed SparseFormer. The MHSAs and
FFNs are multi-head self-attention over latent tokens and feed-forward
networks applied on these tokens, respectively. Note that different from the
illustration aimed for details, the cortex Transformer is actually much deeper
and wider than the focusing Transformer. We omit layer normalization here for
more clarity. All of these operations are performed in the latent token space,
except for image feature sampling in the original image space.

Latent tokens. Different from dense models, which involve per-pixel or per-
patch modeling in the original image space, SparseFormer recognizes an image
in a _sparse_ way by learning a limited number of tokens in the latent space
and applying transformers to them. Similar to tokens or queries in the
Transformer decoder [58], a token in SparseFormer is an embedding
$\mathbf{t}\in\mathbb{R}^{d}$ in the latent space. To explicitly model the
spatial focusing area, we associate each latent token $\mathbf{t}$ in
SparseFormer with an RoI descriptor $\mathbf{b}=(x,y,w,h)$, where $x$, $y$,
$w$, and $h$ are the center coordinates, width, and height, respectively. We
choose to normalize the components of $\mathbf{b}$ by the image size,
resulting in a range of $[0,1]$. Thus, a latent token consists of an embedding
$\mathbf{t}$ and an RoI descriptor $\mathbf{b}$ as its geometric property. The
entire set of latent tokens in SparseFormer can be described as follows:

$\displaystyle\mathbf{T}=\\{(\mathbf{t}_{1},\mathbf{b}_{1}),(\mathbf{t}_{2},\mathbf{b}_{2}),\cdots,(\mathbf{t}_{N},\mathbf{b}_{N})\\},$
(1)

where $N$ is the number of latent tokens, and both the embedding $\mathbf{t}$
and the RoI $\mathbf{b}$ can be refined throughout the network. The initial
$\mathbf{t}$ and $\mathbf{b}$ are learnable parameters of SparseFormer, and
their initialization will be described in the experiment section.

###  3.1 Building Latent Tokens

SparseFormer includes two successive parts for building tokens in the latent
space, the focusing Transformer and the cortex Transformer. The focusing
Transformer addresses the challenge of sparsely extracting image features,
decoding them into latent tokens, and adjusting token RoIs. The subsequent
cortex Transformer accepts these latent token embeddings as inputs and models
them with a standard Transformer encoder.

Sparse feature sampling. In sampling stages, a latent token in SparseFormer
generates $P$ sampling points in the image space according to its geometric
property, _i.e_., the token RoI. These sampling points are explicitly
described as image coordinates (_i.e_., $x$ and $y$) in an image and can be
directly used for feature sampling via bilinear interpolation. Since bilinear
interpolation takes $\mathcal{O}(1)$ time for every sampling point, we term
this procedure as _sparse feature sampling_. In contrast, cross attention
requires $\mathcal{O}(n)$ time to traverse the input, where $n$ is the input
size. To produce such sampling points, SparseFormer first produces relative
offsets for each latent token RoI and then derives absolute sampling locations
based on these RoIs. SparseFormer uses a learnable linear layer to generate a
set of sampling offsets for a token conditionally on its embedding
$\mathbf{t}$:

$\displaystyle\left\\{(\bigtriangleup x_{i},\bigtriangleup
y_{i})\right\\}_{P}=\mathrm{Linear}(\mathbf{t}),$ (2)

where the $i$-th sampling offset $(\bigtriangleup x_{i},\bigtriangleup y_{i})$
indicates the relative position of a sampling point $i$ with respect to a
token RoI. The linear layer contains layer normalization [2] over
$\mathbf{t}$, which is omitted here and below for more clarity. These offsets
are then translated to absolute sampling locations $(\tilde{x},\tilde{y})$ in
an image with the RoI $\mathbf{b}=(x,y,w,h)$:

$\displaystyle\begin{cases}\tilde{x}_{i}=x+0.5\cdot\bigtriangleup x_{i}\cdot
w,\\\ \tilde{y}_{i}=y+0.5\cdot\bigtriangleup y_{i}\cdot h,\end{cases}$ (3)

for every $i$. To stabilize training, we perform standard normalization over
$\left\\{(\bigtriangleup x_{i},\bigtriangleup y_{i})\right\\}_{P}$ by three
standard deviations to keep most of the sampling points inside the RoI.

SparseFormer can directly and efficiently extract image features through
bilinear interpolation based on these explicit sampling points, without the
need for dense traversal over the grid. Given an input image feature
$\mathbf{I}\in\mathbb{R}^{C\times H\times W}$ with $C$ channels, the shape of
the sampled feature matrix $\mathbf{x}$ for a token is $\mathbb{R}^{P\times
C}$. The computational complexity of this sparse feature sampling procedure is
$\mathcal{O}(N\cdot P\cdot C)$, with the number of latent tokens $N$ given and
independent of the input image size $H\times W$.

Adaptive feature decoding. Once image features $\mathbf{x}$ have been sampled
for a token, the key question for our sparse architecture is how to
effectively decode them and build a token in the latent space. While a linear
layer of $\mathbb{R}^{P\times C}\rightarrow\mathbb{R}^{d}$ can be a simple
method to embed features into the latent token, we have found it to be rather
ineffective. Inspired by [15], we use the adaptive mixing layer to decode
sampled features to leverage spatial and channel semantics in an _adaptive_
way. Specifically, we use a lightweight network
$\mathcal{F}:\mathbb{R}^{d}\rightarrow\mathbb{R}^{C\times C+P\times P}$
conditionally on the token embedding $\mathbf{t}$ to produce _adaptive_
channel decoding weight $M_{c}\in\mathbb{R}^{C\times C}$ and spatial decoding
weight $M_{s}\in\mathbb{R}^{P\times P}$. We then decode sampled features
$\mathbf{x}$ in the channel axis and spatial axis with adaptive weights in the
following order:

$\displaystyle\left[M_{c}|M_{s}\right]$
$\displaystyle=\mathcal{F}(\mathbf{t}),M_{c}\in\mathbb{R}^{C\times
C},M_{s}\in\mathbb{R}^{P\times P},$ (4) $\displaystyle\mathbf{x}^{(1)}$
$\displaystyle=\mathrm{GELU}(\mathbf{x}^{(0)}M_{c})\in\mathbb{R}^{P\times C},$
(5) $\displaystyle\mathbf{x}^{(2)}$
$\displaystyle=\mathrm{GELU}(M_{s}\mathbf{x}^{(1)})\in\mathbb{R}^{P\times C}.$
(6)

Here, $\mathbf{x}^{(0)}$ represents the sampled feature $\mathbf{x}$, while
$\mathbf{x}^{(2)}$ is the output of the adaptive feature decoding process.
GELU refers to the Gaussian Error Linear Unit activation function [19], and
learnable biases before GELU are omitted for clarity. We choose two successive
linear layers without activation functions as our $\mathcal{F}$, where the
hidden dimension is $d/4$, for efficiency. The final output,
$\mathbf{x}^{(2)}$, is passed through a learnable linear layer to dimension
$d$ and then added back to the token embedding to update it. Adaptive feature
decoding can be viewed as a token-wise spatial-channel factorization of
dynamic convolution [26], adding convolutional inductive bias for
SparseFormer.

The adaptive feature decoding method enables SparseFormer to reason
conditionally on the embedding $\mathbf{t}$ about _what a token expects to
see_. Since SparseFormer performs feature sampling and decoding several times,
the token embedding $\mathbf{t}$ contains information about _what a token has
seen before_. Thus, a token can infer _where to look_ and focus on
discriminating foregrounds with the RoI adjusting mechanism, which is
described below.

Adjusting RoI. A first, quick glance at an image with human eyes is usually
insufficient to fully understand its contents. Our eyes need several
adjustments to bring the foreground into focus before we can recognize
objects. This is also the case for SparseFormer. The RoI of a latent token in
SparseFormer can be iteratively adjusted together with the update of its
corresponding token embedding. In a stage, a token RoI $\mathbf{b}=(x,y,w,h)$
is adjusted to $(x^{\prime},y^{\prime},w^{\prime},h^{\prime})$ in the
following way:

$\displaystyle x^{\prime}=x+t_{x}\cdot w,\hskip
13.00005pty^{\prime}=y+t_{y}\cdot h,$ (7) $\displaystyle
w^{\prime}=w\cdot\exp(t_{w}),\hskip 5.0pth^{\prime}=h\cdot\exp(t_{h}),$ (8)

where $(t_{x},t_{y},t_{w},t_{h})$ are normalized adjustment deltas for an RoI,
a parameterization adopted from object detectors [45]. The adjustment deltas
are produced by a linear layer, which takes the token embedding as input in a
token-wise manner:

$\displaystyle\left\\{t_{x},t_{y},t_{w},t_{h}\right\\}=\mathrm{Linear}(\mathbf{t}).$
(9)

With the RoI adjusting and sufficient training, SparseFormer can focus on
foregrounds after several stages. It is worth noting that unlike object
detectors, we do not supervise SparseFormer with any localizing signals. The
RoI adjustment optimization is achieved end-to-end by back-propagating
gradients from sampling points in bilinear interpolation. Though the bilinear
interpolation might incur noisy gradients due to local and limited sampling
points, the optimization direction for RoI adjusting is still non-trivial by
ensembling gradients of these points according to experiments.

Focusing Transformer. In practice, we perform token RoI adjustment first,
generate sparse sampling points, and then use bilinear interpolation to obtain
sampled features. We then apply adaptive decoding to these sampled features
and add the decoded output back to the token embedding vector. The RoI
adjusting, sparse feature sampling, and decoding in all can be regarded as a
“cross attention” alternative in an _adaptive_ and _sparse_ way. Together with
self-attention over latent tokens and feed-forward network (FFN), we can stack
these sampling and decoding as a repeating Transformer stage in the latent
space to perform iterative refinement for latent token embedding
$\\{\mathbf{t}_{i}\\}_{N}$ and RoI $\\{\mathbf{b}_{i}\\}_{N}$. We name this
repeating Transformer as _the focusing Transformer_. The focusing Transformer
in our design is lightweight, and parameters are shared between repeating
stages for efficiently focusing on foregrounds. These latent tokens are then
fed into a standard large Transformer encoder, termed as _the cortex
Transformer_.

Cortex Transformer. The cortex Transformer follows a standard Transformer
encoder architecture except for the first stage with the “cross attention” in
the focusing Transformer. As the name implies, the Cortex Transformer behaves
like the cerebral cortex in the brain, processing visual signals from the
focusing Transformer in a manner more similar to the way the cortex processes
signals from the eyes. Parameters of different cortex Transformer stages are
independent.

###  3.2 Overall SparseFormer Architecture

The overall architecture in SparseFormer is depicted in Figure 2. The input
image features are the same and shared across all sampling stages. The final
classification is done by averaging embeddings $\\{\mathbf{t}_{i}\\}_{N}$ over
latent tokens and applying a linear classifier following.

Early convolution. As discussed, gradients regarding sampling points through
bilinear interpolation might be very noisy. This case is even worse for raw
RGB inputs since the nearest four RGB values on the grid are usually too noisy
to estimate local gradients. Thus, it is necessary to introduce early
convolution for more interpolable feature maps to achieve better training
stability and performance, like ones for vision transformers [66]. In the
experiment, the early convolution is designed to be as lightweight as
possible.

Sparsity of the architecture. It is worth noting that the number of latent
tokens in SparseFormer is limited and independent of the input resolution. The
sparse feature sampling procedure extracts features from image feature maps in
a non-traversing manner. Therefore, the computational complexity and memory
footprints of transformers in the latent space are _independent_ of the input
size. So it is reasonable to call our method a sparse visual architecture.
Moreover, the SparseFormer latent space capacity also demonstrates sparsity.
The largest capacity of the latent space, $N\cdot d_{c}$, is $81\cdot 768$
according to Table 1, still smaller than the input image size $3\cdot
224^{2}$. This distinguishes SparseFormer from Perceivers [25, 24], whose
latent capacity is typically $512\cdot 1024$, exceeding the input image size.

It should be also noted that our presented method is quite different from
post-training token sparsifying techniques [30, 13, 5, 44, 68]: SparseFormer,
as a new visual architecture, learns to represent an image with sparse tokens
from scratch. In contrast, token sparsifying techniques are usually applied to
pre-trained vision transformers. In fact, token sparsification can also be
further applied to latent tokens in SparseFormer, but this is beyond the
scope.

Extension to video classification. Video classification is similar to image
classification but requires more intensive computation due to multiple frames.
Fortunately, SparseFormer architecture can also be extended to video
classification with minor additional efforts. Given the video feature
$\mathbf{V}\in\mathbb{R}^{C\times T\times H\times W}$, the only problem is to
deal with an extra temporal axis compared to $\mathbf{I}\in\mathbb{R}^{C\times
H\times W}$. We associate the token RoI with an extension $(t,l)$, where $t$
is the center temporal coordinate and $l$ is the temporal length of the tube,
to make the RoI a tube. In the sparse feature sampling procedure, an extra
linear layer produces temporal offsets, and we transform them to 3D sampling
points $\\{(\tilde{x}_{i},\tilde{y}_{i},\tilde{z}_{i})\\}_{P}$. Bilinear
interpolation is replaced by trilinear interpolation for 4D input data. Alike,
the RoI adjusting is extended to the temporal dimension. Other operators like
early convolution, adaptive feature decoding, self attention, and FFN remain
untouched. For larger capacity for videos, we inflate tokens in the temporal
axis by $n_{t}$ times and initialize their $(t,l)$ to cover all frames, where
$n_{t}$ is much smaller than the frame count $T$ (_e.g_., $8$ versus $32$).

##  4 Experiments

We conduct experiments on the canonical ImageNet classification [9] and
Kinetics [7] video classification benchmarks to investigate our proposed
SparseFormer. We also report our preliminary trials of SparseFormer on
downstream tasks, semantic segmentation and object detection, in the
supplementary.

variant |  #tokens $N$ |  foc. dim $d_{f}$ |  cort. dim $d_{c}$ |  cort. stage $L_{c}$ | FLOPs | #params
---|---|---|---|---|---|---
tiny (T) | 49 | 256 | 512 | 8 | 2.0G | 32M
small (S) | 64 | 320 | 640 | 8 | 3.8G | 48M
base (B) | 81 | 384 | 768 | 10 | 7.8G | 81M
Table 1: Configurations of SparseFormer variants. FLOPs is calculated with the
input image size $224^{2}$.

Model configurations. We use ResNet-like early convolutional layers (a
$7\times 7$ stride-$2$ convolution, a ReLU, and a $3\times 3$ stride-$2$ max
pooling) to extract initial $96$-d image features. We design several variants
of SparseFormer with the computational cost from 2G to $\sim$8G FLOPs, as
shown in Table 1. We mainly scale up the number of latent tokens $N$, the
dimension of the focusing and cortex Transformer $d_{f}$ and $d_{c}$, and
layers of the cortex Transformer $L_{c}$. For all variants, we keep the
repeats of the focusing Transformer, $L_{f}$, to $4$. We bridge the focusing
Transformer and the cortex Transformer with a linear layer to increase the
token dimension. Although the token number is scaled up, it is still smaller
than that of conventional vision transformers. The center of the latent token
RoIs and their sampling points are initialized to a grid. The width and height
of these RoIs are initialized to half the image. For the sake of unity in
building blocks, the first cortex Transformer stage also performs RoI
adjusting, feature sampling, and decoding. We do not inject any positional
information into latent tokens. For more details, please refer to the
supplementary.

method | top-1 | FLOPs | #params | throughput (img/s)
---|---|---|---|---
ResNet-50 [62] | 80.4 | 4.1G | 26M | 1179
ResNet-101 [18] | 81.5 | 7.9G | 45M | 691
RegNetY-8G [42] | 81.7 | 8.0G | 39M | 592
EfficientNet-B3∗ [53] | 81.6 | 1.8G | 12M | 745
DeiT-S [54] | 79.8 | 4.6G | 22M | 983
DeiT-B [54] | 81.8 | 17.5G | 86M | 306
Swin-T [32] | 81.3 | 4.5G | 29M | 726
Swin-S [32] | 83.0 | 8.7G | 50M | 437
Perceiver [25] | 78.0 | 707G | 45M | 17
Perceiver IO [24] | 82.1 | 369G | 49M | 30
SparseFormer-T | 81.0 | 2.0G | 32M | 1270
SparseFormer-S | 82.0 | 3.8G | 48M | 898
SparseFormer-B | 82.6 | 7.8G | 81M | 520
Table 2:  Comparison of different networks on ImageNet-1K classification. The
input resolution is $224^{2}$ except for Efficient-B3 with the resolution
$300^{2}$. The throughput is measured on a single V100 GPU, following [32]

Training recipes. For ImageNet-1K classification [9], we train the proposed
Transformer according to the recipe in [32], which includes the training
budget of 300 epochs, the AdamW optimizer [36] with an initial learning rate
0.001, the weight decay 0.05 and sorts of augmentation and regularization
strategies. The input resolution is fixed to $224^{2}$. We add EMA [40] to
stabilize the training. The stochastic depth (_i.e_., drop path) [21] rate is
set to 0.2, 0.3, and 0.4 for SparseFormer-T, -S, and -B.

To pre-train on ImageNet-21K [9], we use a subset suggested by [47] due to the
availability of the winter 2021 release. We follow the training recipe in [32]
and use a similar recipe to 1K classification but with 60 epochs, an initial
learning rate $2\times 10^{-3}$, weight decay 0.05, and drop path 0.1. After
pre-training, we fine-tune models with a recipe of 30 epochs, an initial
learning rate $2\times 10^{-4}$ with cosine decay and weight decay $10^{-8}$.

We adopt the ImageNet pre-trained models to initialize parameters for training
on Kinetics-400. Since our architecture is endurable to large input sizes, the
number of input frames is set to $T=32$. Formally, 32 frames are sampled from
the 128 consecutive frames with a stride of 4. We mildly inflate initial
latent tokens by $n_{t}=8$ times in the temporal axis to cover all input
frames. We use AdamW [36] for optimization with 32 GPUs, following the
training protocol in [12]. We train the model for 50 epochs with 5 linear
warm-up epochs. The mini-batch size is 8 video samples per GPU. The learning
rate is set to $5\times 10^{-4}$, and we adopt a cosine learning rate schedule
[35]. For evaluation, we apply a 12-view testing scheme (three 3 spatial crops
and 4 temporal clips) as previous work [34].

$N$ |  9 |  16 |  25 |  36 |  49 |  64 |  81
---|---|---|---|---|---|---|---
top-1 |  74.5 |  77.4 |  79.3 |  80.1 |  81.0 |  81.4 |  81.9
GFLOPs |  0.5 |  0.8 |  1.1 |  1.6 |  2.0 |  2.7 |  3.3
(a)

$L_{s}$ |  top-1 |  GFLOPs
---|---|---
nil |  77.8 |  1.6
1 |  79.7 |  1.7
4 |  81.0 |  2.0
8 |  81.0 |  2.5
(b)

$P$ |  top-1 |  GFLOPs
---|---|---
16 |  80.3 |  1.9
36 |  81.0 |  2.0
64 |  81.3 |  2.3
|  |
(c)

img feat. |  top-1 |  GFLOPs
---|---|---
RGB |  fail |  1.5
ViT/8-embed |  78.4 |  1.9
early conv |  81.0 |  2.0
|  |

(d)

decode |  top-1 |  GFLOPs
---|---|---
linear |  78.5 |  1.9
static, mix |  80.1 |  1.9
adaptive, mix |  81.0 |  2.0
|  |
(e)

Table 3: Ablations study on SparseFormer. The default choice for SparseFormer
is colored gray.

###  4.1 Main Results

ImageNet-1K classification. We first benchmark SparseFormer on the ImageNet-1K
classification and compare them to other well-established methods in Table 2.
SparseFormer-T achieves 81.0 top-1 accuracy on par with the well-curated dense
transformer Swin-T [32], with less than half FLOPs of it (2.0G versus 4.5G)
and 74% higher throughput (1270 versus 726). The small and base variants of
SparseFormer, SparseFormer-S, and -B also maintain a good balance between the
performance and actual throughput over highly-optimized CNNs or transformers.
We can see that Perceiver architectures [25, 24], which also adopt the latent
transformer as ours, incur extremely large FLOPs and have impractical
inference speed due to a large number of tokens (_i.e_., $512$) and dense
cross-attention traversal.

variant | pre-train | resolution | top-1 | FLOPs | throughput (img/s)
---|---|---|---|---|---
B | 1K | $224^{2}$ | 82.6 | 7.8G | 520
B | 21K | $224^{2}$ | 83.6 | 7.8G | 520
B | 21K | $384^{2}$ | 84.1 | 8.2G | 444
B | 21K | $512^{2}$ | 84.0 | 8.6G | 419
B, $N=144\uparrow$ | 21K | $384^{2}$ | 84.6 | 14.2G | 292
B, $N=196\uparrow$ | 21K | $384^{2}$ | 84.8 | 19.4G | 221
Table 4:  Scaling up of SparseFormer-B. Except for the 1K entry, all follow
first the same pre-training on ImageNet-21K ($224^{2}$ input, $81$ tokens) and
then individual fine-tuning on ImageNet-1K.

Scaling up SparseFormer. We scale up the base SparseFormer variant in Table 4.
We first adopt ImageNet-21K pre-training, and it brings 1.0 top-1 accuracy
improvement. Then we investigate SparseFormer with large input resolution
fine-tuning. Large resolution inputs benefit SparseFormer (0.5$\uparrow$ for
$384^{2}$) only extra 5% FLOPs. Moreover, we try a more aggressive way to
scale up the model by reinitializing tokens (_i.e_., embeddings and RoIs) with
a more number _in fine-tuning_ , and find it with better results. We leave
further scaling up to future work.

Kinetics-400 classification.

method | top-1 | pre-train | #frames | GFLOPs | #params
---|---|---|---|---|---
NL I3D [61] | 77.3 | ImageNet-1K | 128 |  359$\times$10$\times$3 | 62M
SlowFast [14] | 77.9 | - | 8+32 |  106$\times$10$\times$3 | 54M
TimeSFormer [4] | 75.8 | ImageNet-1K | 8 |  196$\times$1$\times$3 | 121M
Video Swin-T [34] | 78.8 | ImageNet-1K | 32 |  88$\times$4$\times$3 | 28M
ViViT-B FE [1] | 78.8 | ImageNet-21K | 32 |  284$\times$4$\times$3 | 115M
MViT-S [12] | 76.0 | - | 16 |  33$\times$5$\times$1 | 26M
MViT-B [12] | 78.4 | - | 16 |  71$\times$5$\times$1 | 37M
VideoSF-T | 77.9 | ImageNet-1K | 32 |  22$\times$4$\times$3 | 31M
VideoSF-S | 79.1 | ImageNet-1K | 32 |  38$\times$4$\times$3 | 48M
VideoSF-B | 79.8 | ImageNet-21K | 32 |  74$\times$4$\times$3 | 81M
Table 5: Comparison with well-established video classification methods on
Kinetics-400. The GFLOPs is in the format of a single view $\times$ the number
of views. “N/A” indicates the numbers are not available.

We also investigate the extension of SparseFormer to the video classification
task. Results on the Kinetics-400 dataset are reported in Table 5. Our
VideoSparseFormer-T achieves the performance of well-established video CNNs
(I3D or SlowFast) with a much lower computational burden. Surprisingly, our
VideoSparseFormer-S pre-trained on ImageNet-1K even surpasses the Transformer-
based architectures pre-trained on ImageNet-21K, like TimeSFormer [4] and
ViViT [1]. Furthermore, our VideoSparseFormer-S pre-trained on ImageNet-21K
can improve the performance to 79.8 with only 74 GFLOPs.

###  4.2 Ablation Study

In this section, we ablate key designs in the proposed SparseFormer. Limited
by computational resources, we resort to SparseFormer-T on ImageNet-1K
classification.

The number of latent tokens. The number of latent tokens $N$ is a key hyper-
parameter of SparseFormer as it controls the latent space capacity. Table
LABEL:tab:tokennumber shows the performance with different numbers of latent
tokens. The performance of SparseFormer is highly determined by the number of
tokens $N$, and so is the computational cost. We can see when increasing $N$
to 81, SparseFormer-T reaches the performance of SparseFormer-S. As there are
no dense units in the latent space, the number of tokens serves a crucial role
in the information flow in the network. We are still in favor of fewer tokens
in the design of SparseFormer for efficiency.

Focusing stages. Given a limited number of latent tokens, SparseFormer attempt
to focus them on foregrounds by adjusting their RoI and sample corresponding
features to make a visual recognition. Table LABEL:tab:focusingstage
investigates repeats of the parameter-shared focusing Transformer. The ‘nil’
stands for fixing token RoIs to specific areas and making them unlearnable.
Sampling points of tokens in the ‘nil’ row are also set to fixed spatial
positions. The adjusting RoI mechanism and its repeats are vital to the
SparseFormer performance.

Sparsity of sampling points. The other important factor for the sparsity of
the proposed method is the number of sampling points in the feature sampling.
Table LABEL:tab:samplingpoints shows the ablation on this. Compared to
increasing the number of latent tokens (_e.g_., $49\to 64$, 30% up, 81.4
accuracy), more sampling points are not economical for better performance, and
81.3 accuracy needs 77% more ($36\to 64$) sampling points. To maintain a
tradeoff between sparsity and performance, we choose 36 sampling points as our
default.

Image features and how to decode sampled features. Table
LABEL:tab:imagefeatures investigates input image features to be sampled into
the latent space in SparseFormer. As discussed before, input image features
can be in the raw RGB format, but we find it hard to train111Attempts of a
preliminary design of SparseFormer with the raw RGB input are with about 60
top-1 accuracy.. We also ablate it with ViT-like embedding layer [11], namely,
patchifying but keeping the spatial structure, and find it worse than the
ResNet-like early convolutional image feature. Table LABEL:tab:howtodecode
ablates how to decode sampled features for a token. The static mixing uses the
static weights, which are not conditional on token embedding, to perform
mixing on sampled features. We can find that the adaptive mixing decoding
design is better.

Inflation of latent tokens on video classification. We also investigate the
inflation rate of tokens on videos. Intuitively, video data with multiple
frames need more latent tokens than a static image to model. Results in Table
6 show this. Note that the input video has 32 frames, but the token inflation
rate 8 is already sufficient for the favorable performance of
VideoSparseFormer. As a contrast, dense CNNs or Transformers usually require
at least exactly $\\#\textrm{frames}$ times the computational cost if no
temporal reduction is adopted. This also validates the sparsity of the
proposed SparseFormer method on videos.

inflation | top-1 | GFLOPs
---|---|---
1 | 69.5 |  7$\times$4$\times$3
4 | 74.7 |  13$\times$4$\times$3
8 | 77.9 | 22$\times$4$\times$3
16 | 78.2 |  32$\times$4$\times$3
Table 6: Different inflation rates of VideoSparseFormer-T on Kinetics-400.

###  4.3 Visualizations

Figure 3: Visualizations of sampling points and their sampling density maps
across sampling stages in SparseFormer-S. Stage 1-4 refer to the feature
sampling in the focusing Transformer, and Stage 5 refers to the cortex
Transformer. Better view with zoom-in.

As discussed in Section 3, we argue that SparseFormer, with the RoI adjusting
and sparse feature sampling, can focus on foregrounds by reasoning about where
to look. To show this, we perform visualizations of token sampling points
across different sampling stages in Figure 3. We apply kernel density
estimation (KDE) [48] spatially about sampling points with top-hat kernels to
obtain the sampling density map. We can find that SparseFormer initially looks
at the image in a relatively uniform way and gradually focuses on
discriminative details of foregrounds. It is worth noting that SparseFormer is
supervised with only classification signals, and it can roughly learn _where
discriminative foregrounds are_ by _weak supervision_.

##  5 Conclusion

In this paper, we have presented a neural architecture, SparseFormer, to
perform visual recognition with a limited number of tokens along with the
Transformer in the latent space. To imitate human eye behavior, we design
SparseFormer to focus these sparse latent tokens on discriminative foregrounds
and make a recognition sparsely. As a very initial step to the sparse visual
architecture, SparseFormer consistently yields promising results on
challenging image classification and video classification benchmarks with a
good performance-throughput tradeoff. We hope our work can provide an
alternative way and inspire further research about sparse visual
understanding.

Figure 4: More visualizations of token RoIs, their sampling points, and
density across sampling stages in SparseFormer-S ($64$ tokens). RoIs and
sampling points of different tokens are colored with different colors. Better
view with zoom-in.

## Appendix

### A.1. SparseFormer itself as Object Detector

Since the SparseFormer architecture produces embedding and RoI together for a
token given an image, it is natural to ask whether SparseFormer _per se_ can
perform object detection task? The answer is _yes_. In other words, we can
train a SparseFormer model to detect objects _without making any architectural
changes_ by simply adding a final classifier and a RoI refining layer upon it.

Specifically, we follow the training strategy of DETR [6] to train a
SparseFormer-S for object detection. We adopt the ImageNet-1K pre-trained
SparseFormer-S in the main paper. We first inflate the number of latent tokens
to $400$ by re-initializing token embeddings to the normalization distribution
and the center of token RoIs to the uniform distribution on $[0,1]$. The RoI
height and width is $0.5\times 0.5$ still. We use a fixed set of $100$ latent
tokens to detect objects. The other tokens, which are not used for detection,
aim to enrich the semantics in the latent space. We do not change the matching
criterion together with the loss function of DETR and we train for $300$
epochs. The final classifier is simply one-layer FC layer and the RoI refining
layer is 2-layer FC layer, also following DETR. The final refining of RoIs is
performed in the same way as RoI adjustment in the main paper. The result is
shown in Table 7.

detector | GFLOPs | AP |  AP50 |  AP75 |  APs |  APm |  APl
---|---|---|---|---|---|---|---
DETR | 86 | 42.0 | 62.4 | 44.2 | 20.5 | 45.8 | 61.1
DETR-DC5 | 187 | 43.3 | 63.1 | 45.9 | 22.5 | 47.3 | 61.1
SparseFormer-S | 27 | 26.4 | 43.8 | 26.6 | 8.3 | 26.0 | 45.0
Table 7: Detection performance of SparseFormer-S on MS COCO [31] val set.

Although the performance of SparseFormer is currently inferior to DETR, it is
important to note that this is very preliminary result and we do not add any
additional attention encoders or decoders to SparseFormer for object
detection. Actually, SparseFormer can be considered a decoder-only
architecture for object detection if we treat the early convolution part as
the embedding layer.

### A.2. SparseFormer Performing Per-Pixel Task

SparseFormer learns to represent an image by limited tokens in the latent
space and outputs token embeddings with their corresponding RoIs. It is
appealing to investigate whether SparseFormer can perform per-pixel tasks,
like semantic segmentation. Yet, SparseFormer itself cannot perform dense
tasks since it outputs discrete token set. However, we can _restore_ a dense
structured feature map from these discrete tokens by the vanilla cross-
attention operator and build a final classifier upon the dense feature map.

Specifically, to perform semantic segmentation, we use a location-aware cross-
attention operator to map latent token embeddings back to the structured
feature map, whose height and width is one fourth the input image (namely,
stride $4$, $H/4$ and $W/4$). The location-aware cross-attention is the
vanilla cross-attention with geometric prior as biases in attention matrix:

$\displaystyle{\rm Attn}(Q_{ds},K_{lt},V_{lt})={\rm
Softmax}(Q_{ds}K_{lt}^{T}/\sqrt{d}+B)V_{lt},$

where $Q_{ds}\in\mathbb{R}^{N_{ds}\times d}$ is the query matrix for the dense
map ($N_{ds}=H/4*W/4$), $K_{lt},V_{lt}\in\mathbb{R}^{N\times d}$ is the key
and value matrix for the latent tokens,

$\displaystyle
B_{i,j}=-(\frac{x_{ds,i}-x_{lt,j}}{w_{lt,j}})^{2}-(\frac{y_{ds,i}-y_{lt,j}}{h_{lt,j}})^{2}$

, where $(x_{lt},y_{lt},w_{lt},h_{lt})$ is the RoI descriptor for a latent RoI
and $(x_{ds},y_{ds})$ are $x$ and $y$ coordinates for a unit on the dense
feature map. In our current design, the input dense map to cross attend latent
tokens is the early convolved feature map, which has the same height and width
$H/4$ and $W/4$. We put two $3\times 3$ convolution layers and a following
classifier on the restored feature map following common pratice. The results
are shown in Table 8.

segmentor | GFLOPs | mIoU | mAcc
---|---|---|---
Swin-T [32] \+ UperNet [65] | 236 | 44.4 | 56.0
SF-T w/ 49 tokens | 33 | 36.1 | 46.0
SF-T w/ 256 tokens | 39 | 42.9 | 53.7
SF-T w/ 400 tokens | 43 | 43.5 | 54.7
Table 8: Semantic segmentation performance of SparseFormer-T on Ade20K [71]
validation set. The GFLOPs are computed with $512\times 512$ input resolution.

We also inflate the number of latent tokens in the semantic segmentation as we
do in object detector for better performance. The performance of
SparseFormer-T with 400 latent tokens is near the well-established Swin [32]
and UperNet [65] but with merely $1/8$ Swin-T’s GFLOPs. This validates that
our proposed SparseFormer can perform per-pixel task and is suitable to handle
high resolution input data with limited latent tokens.

### A.3. Model Initializations in Details

We initialize all weights of linear layers in SparseFormer unless otherwise
specified below to follow a truncated normalization distribution with a mean
of $0$, a standard deviation of $0.02$, and a truncated threshold of $2$.
Biases of these linear layers are initialized to zeros if exisiting.

Sampling points for every token are initialized in a grid-like shape ($6\times
6$ for $36$ sampling points by default) by zeroing weights of the linear layer
to generate offsets and setting its bias using meshgrid. Alike, we initialize
the center of initial token RoIs (as parameters of the model) to the grid-like
(_e.g_., $7\times 7$ for $49$ SF-Tiny variant) shape in the same way. The
token’s height and width are set to half of the image’s height and width,
which is expressed as $0.5\times 0.5$. We also try other initializations for
tokens’ height and width in Table 9. For training stability, we also
initialize adaptive decoding in SparseFormer following [15] with an initial
Xavier decoding weight [16]. This initialization makes the adaptive decoding
behaves like unconditional convolution (weights not dependent on token
embeddings) at the beginning of the training procedure.

width and height initialization | top-1
---|---
half, $0.5\times 0.5$ | 81.0
cell, $1/\sqrt{N}\times 1/\sqrt{N}$ | 81.0
whole, $1.0\times 1.0$ | 80.2
Table 9: Alternative ways to initialize the token height and width for
SparseFormer-T. $N$ is the number of latent tokens for the ‘cell’
initialization. The ‘cell’ initialization tiles RoIs without overlapping over
the image. The ‘whole’ initialization is with all token RoIs centered at the
image center.

For alternative ways for token height and width initializations, we can find
that there is no significant difference between the ‘half’ and ‘cell’
initializations. We prefer the ‘half’ initialization as tokens can see more
initially. However, setting all token RoIs to the whole image, the ‘whole’
initialization, is lagging before other initializations. We suspect that the
model is unable to differentiate between different tokens and is causing
training instability due to identical RoIs and sampling points for all tokens.

### A.4. Visualizations

More visualizations on RoIs and sampling points. In order to confirm the
general ability of SparseFormer to focus on foregrounds, we present additional
visualizations in Figure 4 and 5 with ImageNet-1K [9] validation set inputs.
Note that these samples are not cherry-picked. we observe that SparseFormer
progressively directs its attention towards the foreground, beginning from the
roughly high contrasting areas and eventually towards more discriminative
areas. The focal areas of SparseFormer adapt to variations in the image and
mainly concentrate on discriminative foregrounds when the input changes. This
also validate the semantic adaptability of SparseFormer to different images.

Visualizations on specific latent tokens. We also provide visualizations of
specific latent tokens across stages to take a closer look at how the token
RoI behaves at the token level. We choose 5 tokens per image that respond with
the highest values to the ground truth category. To achieve this, we remove
token embedding average pooling and place the classifier layer on individual
tokens. The visualizations are shown in Figure 6. We can observe that the
token RoIs progressively move towards the foreground and adjust their aspect
ratios at a mild pace by stage.

Visualizations on disturbed input images. We also show visualizations on
disturbed input images in Figure 7, where images are either random erased or
heavily padding with zero values or reflection. We can see that although
SparseFormer initially views the image in a almost uniform way, it learns to
avoid sampling in uninformative areas in subsequent stages. This illustrates
the robustness and adaptability of SparseFormer when dealing with perturbed
input images.

## References

  * [1] Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid.  Vivit: A video vision transformer.  In ICCV, 2021.
  * [2] Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton.  Layer normalization.  ArXiv, 2016.
  * [3] Iz Beltagy, Matthew E Peters, and Arman Cohan.  Longformer: The long-document transformer.  arXiv preprint arXiv:2004.05150, 2020.
  * [4] Gedas Bertasius, Heng Wang, and Lorenzo Torresani.  Is space-time attention all you need for video understanding?  In ICML, 2021.
  * [5] Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman.  Token merging: Your vit but faster.  ArXiv, 2022.
  * [6] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In European Conference on Computer Vision, 2020.
  * [7] João Carreira and Andrew Zisserman.  Quo vadis, action recognition? A new model and the kinetics dataset.  In CVPR, pages 4724–4733. IEEE Computer Society, 2017.
  * [8] Arthur P. Dempster, Nan M. Laird, and Donald B. Rubin.  In Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper, 1977.
  * [9] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li.  Imagenet: A large-scale hierarchical image database.  In CVPR, 2009.
  * [10] Robert Desimone and John Duncan.  Neural mechanisms of selective visual attention.  Annual Review of Neuroscience, 18(1):193–222, 1995.  PMID: 7605061.
  * [11] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In ICLR. OpenReview.net, 2021.
  * [12] Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer.  Multiscale vision transformers.  In ICCV, 2021.
  * [13] Mohsen Fayyaz, Soroush Abbasi Koohpayegani, Farnoush Rezaei Jafari, Sunando Sengupta, Hamid Reza Vaezi Joze, Eric Sommerlade, Hamed Pirsiavash, and Jürgen Gall.  Adaptive token sampling for efficient vision transformers.  In ECCV (11), volume 13671 of Lecture Notes in Computer Science, pages 396–414. Springer, 2022.
  * [14] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He.  Slowfast networks for video recognition.  In ICCV, 2019.
  * [15] Ziteng Gao, Limin Wang, Bing Han, and Sheng Guo.  Adamixer: A fast-converging query-based object detector.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5364–5373, 2022.
  * [16] Xavier Glorot and Yoshua Bengio.  Understanding the difficulty of training deep feedforward neural networks.  In AISTATS, volume 9 of JMLR Proceedings, pages 249–256. JMLR.org, 2010.
  * [17] Qiqi Gu, Qianyu Zhou, Minghao Xu, Zhengyang Feng, Guangliang Cheng, Xuequan Lu, Jianping Shi, and Lizhuang Ma.  Pit: Position-invariant transform for cross-fov domain adaptation.  In IEEE/CVF International Conference on Computer Vision, 2021.
  * [18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, 2016.
  * [19] Dan Hendrycks and Kevin Gimpel.  Gaussian error linear units (gelus).  arXiv preprint arXiv:1606.08415, 2016.
  * [20] Gao Huang, Zhuang Liu, and Kilian Q. Weinberger.  Densely connected convolutional networks.  In CVPR, 2017.
  * [21] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger.  Deep networks with stochastic depth.  In ECCV (4), volume 9908 of Lecture Notes in Computer Science, pages 646–661. Springer, 2016.
  * [22] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In ICML, 2015.
  * [23] Laurent Itti and Christof Koch.  Computational modelling of visual attention.  Nature Reviews Neuroscience, 2(3):194–203, 2001.
  * [24] Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, Olivier J. Hénaff, Matthew M. Botvinick, Andrew Zisserman, Oriol Vinyals, and João Carreira.  Perceiver IO: A general architecture for structured inputs & outputs.  In ICLR. OpenReview.net, 2022.
  * [25] Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and João Carreira.  Perceiver: General perception with iterative attention.  In ICML, volume 139 of Proceedings of Machine Learning Research, pages 4651–4664. PMLR, 2021.
  * [26] Xu Jia, Bert De Brabandere, Tinne Tuytelaars, and Luc Van Gool.  Dynamic filter networks.  In NIPS, 2016.
  * [27] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya.  Reformer: The efficient transformer.  In International Conference on Learning Representations, 2019.
  * [28] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton.  Imagenet classification with deep convolutional neural networks.  In Peter L. Bartlett, Fernando C. N. Pereira, Christopher J. C. Burges, Léon Bottou, and Kilian Q. Weinberger, editors, NIPS, pages 1106–1114, 2012.
  * [29] Yann LeCun and Corinna Cortes.  In The mnist database of handwritten digits, 2005.
  * [30] Youwei Liang, Chongjian Ge, Zhan Tong, Yibing Song, Jue Wang, and Pengtao Xie.  Not all patches are what you need: Expediting vision transformers via token reorganizations.  In International Conference on Learning Representations, 2022.
  * [31] Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick.  Microsoft COCO: common objects in context.  In ECCV, 2014.
  * [32] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  CVPR, 2021.
  * [33] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie.  A convnet for the 2020s.  In CVPR, pages 11966–11976. IEEE, 2022.
  * [34] Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu.  Video swin transformer.  arXiv preprint arXiv:2106.13230, 2021.
  * [35] Ilya Loshchilov and Frank Hutter.  Sgdr: Stochastic gradient descent with warm restarts.  arXiv preprint arXiv:1608.03983, 2016.
  * [36] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  arXiv preprint arXiv:1711.05101, 2017.
  * [37] David Marr.  Vision: A Computational Investigation into the Human Representation and Processing of Visual Information.  The MIT Press, 07 2010.
  * [38] Volodymyr Mnih, Nicolas Heess, Alex Graves, and Koray Kavukcuoglu.  Recurrent models of visual attention.  In NIPS, pages 2204–2212, 2014.
  * [39] Timo Ojala, Matti Pietikäinen, and David Harwood.  A comparative study of texture measures with classification based on featured distributions.  Pattern Recognit., 29(1):51–59, 1996.
  * [40] B. T. Polyak and A. B. Juditsky.  Acceleration of stochastic approximation by averaging.  SIAM Journal on Control and Optimization, 30(4):838–855, 1992.
  * [41] Zhaofan Qiu, Ting Yao, and Tao Mei.  Learning spatio-temporal representation with pseudo-3d residual networks.  In ICCV, pages 5534–5542. IEEE Computer Society, 2017.
  * [42] Ilija Radosavovic, Raj Prateek Kosaraju, Ross B. Girshick, Kaiming He, and Piotr Dollár.  Designing network design spaces.  In CVPR, 2020.
  * [43] Marc’Aurelio Ranzato.  On learning where to look.  ArXiv, 2014.
  * [44] Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh.  Dynamicvit: Efficient vision transformers with dynamic token sparsification.  In Advances in Neural Information Processing Systems, 2021.
  * [45] Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun.  Faster R-CNN: towards real-time object detection with region proposal networks.  In NIPS, 2015.
  * [46] Ronald A. Rensink.  The dynamic representation of scenes.  Visual Cognition, 7:17 – 42, 2000.
  * [47] Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor.  Imagenet-21k pretraining for the masses.  In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2021.
  * [48] Murray Rosenblatt.  Remarks on some nonparametric estimates of a density function.  Annals of Mathematical Statistics, 27:832–837, 1956.
  * [49] Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta.  Revisiting unreasonable effectiveness of data in deep learning era.  In ICCV, pages 843–852. IEEE Computer Society, 2017.
  * [50] Peize Sun, Rufeng Zhang, Yi Jiang, Tao Kong, Chenfeng Xu, Wei Zhan, Masayoshi Tomizuka, Lei Li, Zehuan Yuan, Changhu Wang, et al.  Sparse r-cnn: End-to-end object detection with learnable proposals.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
  * [51] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich.  Going deeper with convolutions.  In CVPR, 2015.
  * [52] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna.  Rethinking the inception architecture for computer vision.  In CVPR, 2016.
  * [53] Mingxing Tan and Quoc V. Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  In ICML, volume 97 of Proceedings of Machine Learning Research, pages 6105–6114. PMLR, 2019.
  * [54] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  In ICML, volume 139 of Proceedings of Machine Learning Research, pages 10347–10357. PMLR, 2021.
  * [55] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri.  Learning spatiotemporal features with 3d convolutional networks.  In IEEE/CVF International Conference on Computer Vision, 2015.
  * [56] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri.  A closer look at spatiotemporal convolutions for action recognition.  In CVPR, pages 6450–6459. Computer Vision Foundation / IEEE Computer Society, 2018.
  * [57] Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan C. Bovik, and Yinxiao Li.  Maxvit: Multi-axis vision transformer.  In ECCV (24), volume 13684 of Lecture Notes in Computer Science, pages 459–479. Springer, 2022.
  * [58] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In NIPS, pages 5998–6008, 2017.
  * [59] Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool.  Temporal segment networks for action recognition in videos.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019\.
  * [60] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao.  Pyramid vision transformer: A versatile backbone for dense prediction without convolutions.  ICCV, 2021.
  * [61] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He.  Non-local neural networks.  In CVPR, 2018.
  * [62] Ross Wightman, Hugo Touvron, and Hervé Jégou.  Resnet strikes back: An improved training procedure in timm.  arXiv preprint arXiv:2110.00476, 2021.
  * [63] Yuxin Wu and Kaiming He.  Group normalization.  Int. J. Comput. Vis., 128(3):742–755, 2020.
  * [64] Zhuofan Xia, Xuran Pan, Shiji Song, Li Erran Li, and Gao Huang.  Vision transformer with deformable attention.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
  * [65] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun.  Unified perceptual parsing for scene understanding.  In ECCV (5), volume 11209 of Lecture Notes in Computer Science, pages 432–448. Springer, 2018.
  * [66] Tete Xiao, Mannat Singh, Eric Mintun, Trevor Darrell, Piotr Dollár, and Ross B. Girshick.  Early convolutions help transformers see better.  In NeurIPS, pages 30392–30400, 2021.
  * [67] Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy.  Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification.  In ECCV (15), volume 11219 of Lecture Notes in Computer Science, pages 318–335. Springer, 2018.
  * [68] Hongxu Yin, Arash Vahdat, Jose M. Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov.  A-vit: Adaptive tokens for efficient vision transformer.  In CVPR, pages 10799–10808. IEEE, 2022.
  * [69] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Francis E. H. Tay, Jiashi Feng, and Shuicheng Yan.  Tokens-to-token vit: Training vision transformers from scratch on imagenet.  ICCV, 2021.
  * [70] Pengchuan Zhang, Xiyang Dai, Jianwei Yang, Bin Xiao, Lu Yuan, Lei Zhang, and Jianfeng Gao.  Multi-scale vision longformer: A new vision transformer for high-resolution image encoding.  In IEEE/CVF International Conference on Computer Vision, 2021.
  * [71] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba.  Semantic understanding of scenes through the ade20k dataset.  International Journal of Computer Vision, 127(3):302–321, 2019\.
  * [72] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai.  Deformable detr: Deformable transformers for end-to-end object detection.  In International Conference on Learning Representations, 2020.

Figure 5:  More visualizations (cont’d).  Figure 6:  Visualizations on top-5
tokens responding to the ground-truth category across stages.  Figure 7:
Visualizations on sampling density maps when disturbing input images.