ResNeXt101-64x4d [244] $\star$ | 83.5 | 15.6 | 79.6
RegNetY-16G [86] $\star$ | 84.0 | 16.0 | 82.9
EfficientNet-B6 [87] $\star$ | 43.0 | 19.0 | 84.0
NesT-B [111] | 68.0 | 17.9 | 83.8
ViT-B/16 [11] | 86.6 | 17.6 | 79.8
DeiT-B/16 [12] | 86.6 | 17.6 | 81.8
Swin-B [36] | 88.0 | 15.4 | 83.3
Twins-SVT-L [37] | 99.2 | 14.8 | 83.7
PVTv2-B5 [97] | 82.0 | 11.8 | 83.8
Lv-ViT-M [89] | 56.0 | 16.0 | 84.1

TABLE III: A Comparative analysis between different vision transformer and CNN
models in terms of their parameter complexity and top-1 (%) accuracy on
ImageNet validation set. For a direct comparison, we consider models that are
trained on ImageNet from scratch on input of size 224x224. $\star$ denotes
pure CNN-based methods.

In the language domain, recent works focus on reducing the high complexity of
Transformer models (basically arising from the self-attention mechanism [1]
where a token’s representation is updated by considering all tokens from the
previous layer). For example, [245, 103] explore selective or sparse attention
to previous layer tokens while updating each next layer token. Linformer [38]
reduces complexity of standard self-attention operation from
$\mathcal{O}(n^{2})$ to $\mathcal{O}(n)$ (both in time and memory
requirements). The main idea is to show that a low-rank matrix is sufficient
to model the self-attention mechanism. The Reformer model [246] employed
locally-sensitive hashing (LSH) to minimize the complexity of self-attention
from $\mathcal{O}(n^{2})$ to $\mathcal{O}(nlog(n))$. In similar pursuit, the
recent Lambda Networks propose to model local context as a linear function
which helps reduce complexity of self-attention [247]. These linear function
lambdas are applied to the input query to model contextual relationships
between pixels.

Vyas et al.[248] developed an efficient _cluster attention_ to deal with large
input sequences that approximates the original self-attention. The cluster
attention groups queries into clusters and then computes attention between
cluster centers (instead of attention between all the queries that leads to
quadratic complexity). The main idea is that the queries close in the
Euclidean space should have similar attention distributions. With a fixed
number of clusters, this intuition helps reduce the quadratic complexity to
linear complexity of $\mathcal{O}(nc)$ with respect to the input sequence
length $n$ (where $c$ is the number of clusters). We refer interested readers
to a survey on efficient Transformers in NLP [34].

Similar to the NLP domain, computer vision models also suffer from the high
computational cost of Transformer models. For example, image generators that
are based on sequence-based Transformers (_e.g._ , iGPT) have a high compute
cost limiting their applicability to high-resolution inputs. The time and
memory cost of core self-attention operation in Transformers increases
quadratically with the number of patches, i.e. $\mathcal{O}(n^{2})$, for $n$
image patches (in some applications, e.g., low-level vision, $n=H\times W$
where $H,W$ denote the height and width of the image). This is a major
drawback of existing Transformers that hinders their application to most tasks
involving high-resolution (HR) images, such as object detection and
segmentation (in high-level vision), and super-resolution, deblurring,
denoising, etc. (in low-level vision). Numerous methods have been proposed
that make special design choices to perform self-attention more ‘efficiently’,
for instance employing pooling/downsampling in self-attention [97, 219, 249],
local window-based attention [36, 250], axial-attention [179, 251], low-rank
projection attention [38, 252, 253], kernelizable attention [254, 255], and
similarity-clustering based methods [246, 256]. However, almost all of these
approaches either come with a trade-off between complexity and accuracy,
require special hardware specifications or are still not applicable to very
large images. Therefore, there is a pressing need to develop an efficient
self-attention mechanism that can be applied to HR images on resource-limited
systems without compromising accuracy. It will be interesting to explore how
existing models can be extended to high-dimensional cases _e.g._ , using a
_multi-scale transformer_ design with a somewhat local context modeling. By
inducing inductive biases based on our understanding of the visual learning
tasks (e.g., spatial relationships in the local neighbourhood), the high
computational cost can be reduced. Similarly, using sparse attention maps
modeled with low-rank factorization in the matrices can also help towards
reducing the computational cost [211].

###  4.2 Large Data Requirements

Since Transformer architectures do not inherently encode inductive biases
(prior knowledge) to deal with visual data, they typically require large
amount of training to figure out the underlying modality-specific rules. For
example, a CNN has inbuilt translation invariance, weight sharing, and partial
scale invariance due to pooling operations or multi-scale processing blocks.
However, a Transformer network needs to figure out these image-specific
concepts on its own from the training examples. Similarly, relationships
between video frames need to be discovered automatically by the self-attention
mechanism by looking at a large database of video sequences. This results in
longer training times, a significant increase in computational requirements,
and large datasets for processing. For example, the ViT [11] model requires
hundreds of millions of image examples to obtain reasonable performance on the
ImageNet benchmark dataset. The question of learning a Transformer in a data-
efficient manner is an open research problem and recent works report
encouraging steps towards its resolution. For example, DeiT [12] uses a
distillation approach to achieve data efficiency while T2T (Tokens-to-Token)
ViT [35] models local structure by combining spatially close tokens together,
thus leading to competitive performance when trained only on ImageNet from
scratch (without pre-training). By incorporating CNNs like feature hierarchies
in ViTs to effectively capture local image cues, ViTs (e.g., CCT [106], NesT
[111]) can be trained from scratch even on small-scale datasets (e.g.,
CIFAR-10). Another approach to data efficient training of ViTs is proposed in
et al.[257]. The authors show that by smoothing the local loss surface using
sharpness-aware minimizer (SAM) [258], ViTs can be trained with simple data
augmentation scheme (random crop, and horizontal flip) [259], instead of
employing compute intensive strong data augmentation strategies, and can
outperform their counterpart ResNet models.

###  4.3 Vision Tailored Transformer Designs

We note that most of the existing works focused on vision tasks tend to
directly apply NLP Transformer models on computer vision problems. These
include architectures designed for image recognition [11], video understanding
[17] and especially multi-modal processing [181]. Although the initial results
from these simple applications are quite encouraging and motivate us to look
further into the strengths of self-attention and self-supervised learning,
current architectures may still remain better tailored for language problems
(with a sequence structure) and need further intuitions to make them more
efficient for visual inputs. For example, vector attention from [82] is a nice
work in this direction which attempts to specifically tailor self-attention
operation for visual inputs via learning channel-wise attentions. Similarly,
[260] uses a Jigsaw puzzle based self-supervision loss as a parallel branch in
the Transformers to improve person re-identification. A recent work [35]
rearranges the spatially close tokens to better model relationships in
spatially proximal locations. Token distillation [12] from pre-trained CNN
models has also been used as a remedy to inject domain biases in the
representations. One may argue that the architectures like Transformer models
should remain generic to be directly applicable across domains, we notice that
the high computational and time cost for pre-training such models demands
novel design strategies to make their training more affordable on vision
problems.

###  4.4 Neural Architecture Search for ViTs

While Nerual Architecuter Search (NAS) has been well explored for CNNs to find
an optimized architecture, it is relatively less explored in Transformers
(even for language transformers [261, 262]). Chen et al.[263] propose a one-
shot NAS for vision transformers, called AutoFormer. BossNAS [264] searches
for a hybrid architecture (CNN and Transformer). Another recent effort studies
the trade-off between global and local information in Transformers in the
context of vision applications [265]. It will be insightful to further explore
the domain-specific design choices (e.g., the contrasting requirements between
language and vision domains) using NAS to design more efficient and light-
weight models similar to CNNs [87].

###  4.5 Interpretability of Transformers

Through an extensive set of carefully designed experiments, Naseer et al.[266]
investigate multiple intriguing properties of ViTs in terms of their
generalization and robustness. They show that, compared with CNNs, ViTs
demonstrate strong robustness against texture changes and severe occlusions,
_e.g._ ViTs retain upto 60% top-1 accuracy on ImageNet once 80% of the image
content is randomly occluded.  Given the strong performance of Transformer
architectures, it is interesting and critical to interpret their decisions,
_e.g._ , by visualizing relevant regions in an image for a given
classification decision. The main challenge is that the attention originating
in each layer, gets inter-mixed in the subsequent layers in a complex manner,
making it difficult to visualize the relative contribution of input tokens
towards final predictions. This is an open problem, however, some recent works
[267, 268, 269] target enhanced interpretability of Transformers and report
encouraging results. Attention roll-out and attention flow methods were
proposed in [268] to estimate the accurate attentions. However, this method
functions in an ad-hoc manner and makes simplistic assumptions _e.g._ , input
tokens are linearly combined using attention weights across the layers. Chefer
et al. [269] note that the attention scores obtained directly via the self-
attention process (encoding relationships between tokens) or reassignments in
[268] do not provide an optimal solution. As an alternative, they propose to
assign and propagate _relevancy scores_ in the Transformer network such that
the sum of relevancy is constant throughout the network. Their design can
handle both the positive and negative attributions experienced in the self-
attention layer. The proposed framework has an added advantage of being able
to provide class-specific visualizations. Despite these seminal works,
visualizing and interpreting Transformers is an unsolved problem and methods
are needed to obtain spatially precise activation-specific visualizations.
Further progress in this direction can help in better understanding the
Transformer models, diagnosing any erroneous behaviors and biases in the
decision process. It can also help us design novel architectures that can help
us avoid any biases.

###  4.6 Hardware Efficient Designs

Large-scale Transformer networks can have intensive power and computation
requirements, hindering their deployment on edge devices and resource-
constrained environments such as internet-of-things (IoT) platforms. Some
recent efforts have been reported to compress and accelerate NLP models on
embedded systems such as FPGAs [270]. Li et al.[270] used an enhanced block-
circulant matrix-based representation to compress NLP models and proposed a
new Field Programmable Gate Array (FPGA) architecture design to efficiently
manage resources for high throughput and low latency. They could achieve 27x,
3x and 81x improvements in performance (throughput measured in FPS), reduced
power consumption, and energy efficiency relative a CPU for RoBERTa model [7].
Towards this goal, [262] proposed to design Hardware-Aware Transformers (HAT)
using neural architecture search strategies [271, 272, 273]. Specifically, a
SuperTransformer model is first trained for performance approximation which
can estimate a model’s performance without fully training it. This model
comprises the largest possible model in the search space while sharing weights
between common parts. Eventually, an evolutionary search is performed
considering the hardware latency constraints to find a suitable SubTransformer
model for a target hardware platform (_e.g._ , IoT device, GPU, CPU). However,
such hardware efficient designs are currently lacking for the vision
Transformers to enable their seamless deployment in resource-constrained
devices. Further, the search cost of the evolutionary algorithms remains
significant with the associated impact of CO2 emissions on the environment.

###  4.7 Towards Integrating All Modalities

Since Transformers provide a unified design to process different modalities,
recent efforts also focus on proposing more generic general purpose reasoning
systems based on Transformers. Inspired by the biological systems that can
process information from a diverse range of modalities, Perceiver model [274]
aims to learn a unified model that can process any given input modality
without making domain-specific architectural assumptions. In order to scale to
high-dimensional inputs, Perceiver uses an asymmetric cross attention method
to distill input information into low-dimensional latent bottleneck features.
Once the features are distilled in a compact and fixed-dimensional form,
regular Transformer blocks are applied in the latent space. The original
Perceiver model shows performance competitive to ResNets and ViTs on image
classification and can process 3D data, audio, images, video or their
combinations. However, this model can only generate fixed outputs e.g., class
probabilities. A recent improvement called Perceiver IO [275] aims to learn
models with both flexible inputs as well as arbitrary sized outputs. This
allows application to problems which demand structured outputs such as natural
language tasks and visual comprehension. While these models avoid modality
dependent architectural choices, the learning itself still involves modality
dependent choices e.g., specific augmentations or positional encodings. An
interesting and open future direction is to achieve total modality-agnosticism
in the learning pipeline.

##  5 Conclusion

Attention has played a key role in delivering efficient and accurate computer
vision systems, while simultaneously providing insights into the function of
deep neural networks. This survey reviews the self-attention approaches and
specifically focuses on the Transformer and bi-directional encoding
architectures that are built on the principle of self-attention. We first
cover fundamental concepts pertaining to self-attention architectures and
later provide an in-depth analysis of competing approaches for a broad range
of computer vision applications. Specifically, we include state of the art
self-attention models for image recognition, object detection, semantic and
instance segmentation, video analysis and classification, visual question
answering, visual commonsense reasoning, image captioning, vision-language
navigation, clustering, few-shot learning, and 3D data analysis. We
systematically highlight the key strengths and limitations of the existing
methods and particularly elaborate on the important future research
directions. With its specific focus on computer vision tasks, this survey
provides a unique view of the recent progress in self-attention and
Transformer-based methods. We hope this effort will drive further interest in
the vision community to leverage the potential of Transformer models and
improve on their current limitations _e.g._ , reducing their carbon footprint.

## Acknowledgments

The authors would like to thank Tim Prangemeier (TU Darmstadt), Luowei Zhou
(Microsoft Research), Jason Corso (University of Michigan), Pichao Wang
(Alibaba Group), Yuqing Wang (Meituan), Alex Meinke (Uni-Tuebingen), Irwan
Bello (Google Brain) and Manoj Kumar (Google Brain) for their helpful feedback
on the survey. We would also like to thank Mohamed Afham for his help with a
figure.

## References

  * [1] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in NeurIPS, 2017.
  * [2] M. Ott, S. Edunov, D. Grangier, and M. Auli, “Scaling neural machine translation,” in WMT, 2018.
  * [3] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018.
  * [4] A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever, “Improving language understanding by generative pre-training,” tech. rep., OpenAI, 2018.
  * [5] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language models are unsupervised multitask learners,” tech. rep., OpenAI, 2019.
  * [6] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., “Language models are few-shot learners,” arXiv preprint arXiv:2005.14165, 2020.
  * [7] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “RoBERTa: A robustly optimized bert pretraining approach,” arXiv preprint arXiv:1907.11692, 2019.
  * [8] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” arXiv preprint arXiv:1910.10683, 2019\.
  * [9] D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen, “Gshard: Scaling giant models with conditional computation and automatic sharding,” arXiv preprint arXiv:2006.16668, 2020.
  * [10] W. Fedus, B. Zoph, and N. Shazeer, “Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,” arXiv preprint arXiv:2101.03961.
  * [11] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929, 2020.
  * [12] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou, “Training data-efficient image transformers & distillation through attention,” arXiv preprint arXiv:2012.12877, 2020.
  * [13] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with transformers,” arXiv preprint arXiv:2005.12872, 2020.
  * [14] X. Zhu, W. Su, L. Lu, B. Li, X. Wang, and J. Dai, “Deformable DETR: Deformable transformers for end-to-end object detection,” arXiv preprint arXiv:2010.04159, 2020.
  * [15] L. Ye, M. Rochan, Z. Liu, and Y. Wang, “Cross-modal self-attention network for referring image segmentation,” in CVPR, 2019.
  * [16] F. Yang, H. Yang, J. Fu, H. Lu, and B. Guo, “Learning texture transformer network for image super-resolution,” in CVPR, 2020.
  * [17] C. Sun, A. Myers, C. Vondrick, K. Murphy, and C. Schmid, “VideoBERT: A joint model for video and language representation learning,” in ICCV, 2019.
  * [18] R. Girdhar, J. Carreira, C. Doersch, and A. Zisserman, “Video action transformer network,” in CVPR, 2019.
  * [19] H. Chen, Y. Wang, T. Guo, C. Xu, Y. Deng, Z. Liu, S. Ma, C. Xu, C. Xu, and W. Gao, “Pre-trained image processing transformer,” arXiv preprint arXiv:2012.00364, 2020.
  * [20] A. Ramesh, M. Pavlov, G. Goh, and S. Gray, “DALL·E: Creating images from text,” tech. rep., OpenAI, 2021.
  * [21] H. Tan and M. Bansal, “LXMERT: Learning cross-modality encoder representations from transformers,” in EMNLP-IJCNLP, 2019.
  * [22] W. Su, X. Zhu, Y. Cao, B. Li, L. Lu, F. Wei, and J. Dai, “VL-BERT: Pre-training of generic visual-linguistic representations,” arXiv preprint arXiv:1908.08530, 2019.
  * [23] X. Wang, C. Yeshwanth, and M. Nießner, “SceneFormer: Indoor scene generation with transformers,” arXiv preprint arXiv:2012.09793, 2020.
  * [24] M. Kumar, D. Weissenborn, and N. Kalchbrenner, “Colorization transformer,” in ICLR, 2021.
  * [25] C. Doersch, A. Gupta, and A. Zisserman, “CrossTransformers: spatially-aware few-shot transfer,” NeurIPS, 2020.
  * [26] H.-J. Ye, H. Hu, D.-C. Zhan, and F. Sha, “Few-shot learning via embedding adaptation with set-to-set functions,” in CVPR, 2020.
  * [27] S. Chaudhari, G. Polatkan, R. Ramanath, and V. Mithal, “An attentive survey of attention models,” arXiv preprint arXiv:1904.02874, 2019.
  * [28] A. de Santana Correia and E. L. Colombini, “Attention, please! asurvey of neural attention models in deep learning,” arXiv preprint arXiv:2103.16775, 2021.
  * [29] O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, “Show and tell: A neural image caption generator,” in CVPR, 2015.
  * [30] Y. Bengio, I. Goodfellow, and A. Courville, Deep learning.  MIT press, 2017.
  * [31] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, 2015.
  * [32] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural computation, 1997.
  * [33] D. Hu, “An introductory survey on attention mechanisms in nlp problems,” in IntelliSys, 2019.
  * [34] Y. Tay, M. Dehghani, D. Bahri, and D. Metzler, “Efficient transformers: A survey,” arXiv preprint arXiv:2009.06732, 2020.
  * [35] L. Yuan, Y. Chen, T. Wang, W. Yu, Y. Shi, F. E. Tay, J. Feng, and S. Yan, “Tokens-to-token vit: Training vision transformers from scratch on imagenet,” arXiv preprint arXiv:2101.11986, 2021.
  * [36] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” arXiv preprint arXiv:2103.14030, 2021.
  * [37] X. Chu, Z. Tian, Y. Wang, B. Zhang, H. Ren, X. Wei, H. Xia, and C. Shen, “Twins: Revisiting the design of spatial attention in vision transformers,” arXiv preprint arXiv:2104.13840, 2021.
  * [38] S. Wang, B. Li, M. Khabsa, H. Fang, and H. Ma, “Linformer: Self-attention with linear complexity,” arXiv preprint arXiv:2006.04768, 2020.
  * [39] H. Zhang, I. Goodfellow, D. Metaxas, and A. Odena, “Self-attention generative adversarial networks,” in International conference on machine learning, pp. 7354–7363, PMLR, 2019.
  * [40] J. Pérez, J. Marinković, and P. Barceló, “On the turing completeness of modern neural network architectures,” in International Conference on Learning Representations, 2018.
  * [41] J.-B. Cordonnier, A. Loukas, and M. Jaggi, “On the relationship between self-attention and convolutional layers,” in International Conference on Learning Representations, 2019.
  * [42] J. Dai, H. Qi, Y. Xiong, Y. Li, G. Zhang, H. Hu, and Y. Wei, “Deformable convolutional networks,” in Proceedings of the IEEE international conference on computer vision, pp. 764–773, 2017.
  * [43] Y.-C. Chen, L. Li, L. Yu, A. El Kholy, F. Ahmed, Z. Gan, Y. Cheng, and J. Liu, “UNITER: Universal image-text representation learning,” in ECCV, 2020\.
  * [44] X. Li, X. Yin, C. Li, P. Zhang, X. Hu, L. Zhang, L. Wang, H. Hu, L. Dong, F. Wei, et al., “Oscar: Object-semantics aligned pre-training for vision-language tasks,” in ECCV, 2020.
  * [45] K. Lin, L. Wang, and Z. Liu, “End-to-end human pose and mesh reconstruction with transformers,” arXiv preprint arXiv:2012.09760, 2020.
  * [46] S. Gidaris, P. Singh, and N. Komodakis, “Unsupervised representation learning by predicting image rotations,” in ICLR, 2018.
  * [47] “Revisiting the unreasonable effectiveness of data.” https://ai.googleblog.com/2017/07/revisiting-unreasonable-effectiveness.html.  Accessed: 2020-12-31.
  * [48] L. Jing and Y. Tian, “Self-supervised visual feature learning with deep neural networks: A survey,” TPAMI, 2020.
  * [49] X. Liu, F. Zhang, Z. Hou, Z. Wang, L. Mian, J. Zhang, and J. Tang, “Self-supervised learning: Generative or contrastive,” arXiv preprint arXiv:2006.08218, 2020.
  * [50] “Aaai 2020 keynotes turing award winners event.” https://www.youtube.com/watch?v=UX8OubxsY8w.  Accessed: 2020-12-31.
  * [51] R. Zhang, P. Isola, and A. A. Efros, “Colorful image colorization,” in ECCV, 2016.
  * [52] C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang, et al., “Photo-realistic single image super-resolution using a generative adversarial network,” in CVPR, 2017.
  * [53] D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, and A. Efros, “Context encoders: Feature learning by inpainting,” in CVPR, 2016.
  * [54] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in NeurIPS, 2014.
  * [55] D. Lin, K. Fu, Y. Wang, G. Xu, and X. Sun, “MARTA GANs: Unsupervised representation learning for remote sensing image classification,” GRSL, 2017.
  * [56] U. Ahsan, R. Madhok, and I. Essa, “Video jigsaw: Unsupervised learning of spatiotemporal context for video action recognition,” in WACV, 2019.
  * [57] M. Noroozi and P. Favaro, “Unsupervised learning of visual representations by solving jigsaw puzzles,” in ECCV, 2016.
  * [58] D. Kim, D. Cho, D. Yoo, and I. S. Kweon, “Learning image representations by completing damaged jigsaw puzzles,” WACV, 2018.
  * [59] L. Jing, X. Yang, J. Liu, and Y. Tian, “Self-supervised spatiotemporal feature learning via video rotation prediction,” arXiv preprint arXiv:1811.11387, 2018.
  * [60] H.-Y. Lee, J.-B. Huang, M. Singh, and M.-H. Yang, “Unsupervised representation learning by sorting sequences,” in ICCV, 2017.
  * [61] I. Misra, C. L. Zitnick, and M. Hebert, “Shuffle and learn: unsupervised learning using temporal order verification,” in ECCV, 2016.
  * [62] D. Wei, J. J. Lim, A. Zisserman, and W. T. Freeman, “Learning and using the arrow of time,” in CVPR, 2018.
  * [63] L. H. Li, M. Yatskar, D. Yin, C.-J. Hsieh, and K.-W. Chang, “VisualBERT: A simple and performant baseline for vision and language,” in Arxiv preprint arXiv:1908.03557, 2019.
  * [64] B. Korbar, D. Tran, and L. T., “Cooperative learning of audio and video models from self-supervised synchronization,” in NeurIPS, 2018.
  * [65] R. Arandjelovic and A. Zisserman, “Look, listen and learn,” in ICCV, 2017\.
  * [66] N. Sayed, B. Brattoli, and B. Ommer, “Cross and learn: Cross-modal self-supervision,” in GCPR, 2018.
  * [67] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in CVPR, 2016.
  * [68] J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,” arXiv preprint arXiv:1607.06450, 2016.
  * [69] A. Buades, B. Coll, and J.-M. Morel, “A non-local algorithm for image denoising,” in CVPR, 2005.
  * [70] X. Wang, R. Girshick, A. Gupta, and K. He, “Non-local neural networks,” in CVPR, 2018.
  * [71] W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijayanarasimhan, F. Viola, T. Green, T. Back, P. Natsev, et al., “The kinetics human action video dataset,” arXiv preprint arXiv:1705.06950, 2017.
  * [72] Z. Huang, X. Wang, L. Huang, C. Huang, Y. Wei, and W. Liu, “CCNet: Criss-cross attention for semantic segmentation,” in ICCV, 2019.
  * [73] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, “The cityscapes dataset for semantic urban scene understanding,” in CVPR, 2016.
  * [74] B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso, and A. Torralba, “Scene parsing through ade20k dataset,” in CVPR, 2017.
  * [75] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft COCO: Common objects in context,” in ECCV, 2014.
  * [76] X. Liang, K. Gong, X. Shen, and L. Lin, “Look into person: Joint body parsing & pose estimation network and a new benchmark,” TPAMI, 2018.
  * [77] G. J. Brostow, J. Fauqueur, and R. Cipolla, “Semantic object classes in video: A high-definition ground truth database,” Pattern Recognition Letters, 2009\.
  * [78] H. Hu, Z. Zhang, Z. Xie, and S. Lin, “Local relation networks for image recognition,” in ICCV, 2019.
  * [79] I. Bello, B. Zoph, A. Vaswani, J. Shlens, and Q. V. Le, “Attention augmented convolutional networks,” in ICCV, 2019.
  * [80] P. Shaw, J. Uszkoreit, and A. Vaswani, “Self-attention with relative position representations,” in NAACL, 2018.
  * [81] N. Parmar, P. Ramachandran, A. Vaswani, I. Bello, A. Levskaya, and J. Shlens, “Stand-alone self-attention in vision models,” in NeurIPS, 2019.
  * [82] H. Zhao, J. Jia, and V. Koltun, “Exploring self-attention for image recognition,” in CVPR, 2020.
  * [83] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus, “Intriguing properties of neural networks,” arXiv preprint arXiv:1312.6199, 2013.
  * [84] M. M. Naseer, S. H. Khan, M. H. Khan, F. S. Khan, and F. Porikli, “Cross-domain transferability of adversarial perturbations,” in NeurIPS, 2019.
  * [85] M. Naseer, K. Ranasinghe, S. Khan, F. S. Khan, and F. Porikli, “On improving adversarial transferability of vision transformers,” arXiv preprint arXiv:2106.04169, 2021.
  * [86] I. Radosavovic, R. P. Kosaraju, R. Girshick, K. He, and P. Dollár, “Designing network design spaces,” in CVPR, 2020.
  * [87] M. Tan and Q. V. Le, “EfficientNet: Rethinking model scaling for convolutional neural networks,” in ICML, 2019.
  * [88] K. Han, A. Xiao, E. Wu, J. Guo, C. Xu, and Y. Wang, “Transformer in transformer,” arXiv preprint arXiv:2103.00112, 2021.
  * [89] Z. Jiang, Q. Hou, L. Yuan, D. Zhou, Y. Shi, X. Jin, A. Wang, and J. Feng, “All tokens matter: Token labeling for training better vision transformers,” arXiv preprint arXiv:2104.10858, 2021.
  * [90] S. Yun, D. Han, S. J. Oh, S. Chun, J. Choe, and Y. Yoo, “Cutmix: Regularization strategy to train strong classifiers with localizable features,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6023–6032, 2019.
  * [91] A. El-Nouby, H. Touvron, M. Caron, P. Bojanowski, M. Douze, A. Joulin, I. Laptev, N. Neverova, G. Synnaeve, J. Verbeek, and H. Jegou, “Xcit: Cross-covariance image transformers,” 2021.
  * [92] D. Zhou, B. Kang, X. Jin, L. Yang, X. Lian, Z. Jiang, Q. Hou, and J. Feng, “Deepvit: Towards deeper vision transformer,” 2021.
  * [93] W. Wang, E. Xie, X. Li, D.-P. Fan, K. Song, D. Liang, T. Lu, P. Luo, and L. Shao, “Pyramid vision transformer: A versatile backbone for dense prediction without convolutions,” arXiv preprint arXiv:2102.12122, 2021\.
  * [94] J. Yang, C. Li, P. Zhang, X. Dai, B. Xiao, L. Yuan, and J. Gao, “Focal self-attention for local-global interactions in vision transformers,” 2021.
  * [95] Z. Huang, Y. Ben, G. Luo, P. Cheng, G. Yu, and B. Fu, “Shuffle transformer: Rethinking spatial shuffle for vision transformer,” 2021.
  * [96] H. Wu, B. Xiao, N. Codella, M. Liu, X. Dai, L. Yuan, and L. Zhang, “Cvt: Introducing convolutions to vision transformers,” arXiv preprint arXiv:2103.15808, 2021.
  * [97] W. Wang, E. Xie, X. Li, D.-P. Fan, K. Song, D. Liang, T. Lu, P. Luo, and L. Shao, “Pvtv2: Improved baselines with pyramid vision transformer,” 2021.
  * [98] W. Xu, Y. Xu, T. Chang, and Z. Tu, “Co-scale conv-attentional image transformers,” 2021.
  * [99] W. Wang, L. Yao, L. Chen, D. Cai, X. He, and W. Liu, “Crossformer: A versatile vision transformer based on cross-scale attention,” arXiv preprint arXiv:2108.00154, 2021.
  * [100] C.-F. Chen, R. Panda, and Q. Fan, “Regionvit: Regional-to-local attention for vision transformers,” 2021.
  * [101] E. Xie, W. Wang, Z. Yu, A. Anandkumar, J. M. Alvarez, and P. Luo, “Segformer: Simple and efficient design for semantic segmentation with transformers,” 2021\.
  * [102] P. Zhang, X. Dai, J. Yang, B. Xiao, L. Yuan, L. Zhang, and J. Gao, “Multi-scale vision longformer: A new vision transformer for high-resolution image encoding,” ICCV 2021, 2021.
  * [103] I. Beltagy, M. E. Peters, and A. Cohan, “Longformer: The long-document transformer,” arXiv preprint arXiv:2004.05150, 2020.
  * [104] C.-F. Chen, Q. Fan, and R. Panda, “Crossvit: Cross-attention multi-scale vision transformer for image classification,” arXiv preprint arXiv:2103.14899, 2021.
  * [105] K. Yuan, S. Guo, Z. Liu, A. Zhou, F. Yu, and W. Wu, “Incorporating convolution designs into visual transformers,” arXiv preprint arXiv:2103.11816, 2021\.
  * [106] A. Hassani, S. Walton, N. Shah, A. Abuduweili, J. Li, and H. Shi, “Escaping the big data paradigm with compact transformers,” 2021.
  * [107] Y. Li, K. Zhang, J. Cao, R. Timofte, and L. V. Gool, “Localvit: Bringing locality to vision transformers,” 2021.
  * [108] B. Graham, A. El-Nouby, H. Touvron, P. Stock, A. Joulin, H. Jégou, and M. Douze, “Levit: a vision transformer in convnet’s clothing for faster inference,” 2021.
  * [109] Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel, “Backpropagation applied to handwritten zip code recognition,” Neural computation, vol. 1, no. 4, pp. 541–551, 1989.
  * [110] Q. Zhang and Y. Yang, “Rest: An efficient transformer for visual recognition,” arXiv preprint arXiv:2105.13677, 2021.
  * [111] Z. Zhang, H. Zhang, L. Zhao, T. Chen, and T. Pfister, “Aggregating nested transformers,” in arXiv preprint arXiv:2105.12723, 2021.
  * [112] Z. Dai, H. Liu, Q. V. Le, and M. Tan, “Coatnet: Marrying convolution and attention for all data sizes,” 2021.
  * [113] X. Chu, Z. Tian, B. Zhang, X. Wang, X. Wei, H. Xia, and C. Shen, “Conditional positional encodings for vision transformers,” 2021.
  * [114] Y. Liu, G. Sun, Y. Qiu, L. Zhang, A. Chhatkuli, and L. Van Gool, “Transformer in convolutional neural networks,” arXiv preprint arXiv:2106.03180, 2021\.
  * [115] X. Chen, S. Xie, and K. He, “An empirical study of training self-supervised visual transformers,” arXiv e-prints, pp. arXiv–2104, 2021.
  * [116] X. Chen, H. Fan, R. Girshick, and K. He, “Improved baselines with momentum contrastive learning,” arXiv preprint arXiv:2003.04297, 2020.
  * [117] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729–9738, 2020.
  * [118] Z. Xie, Y. Lin, Z. Yao, Z. Zhang, Q. Dai, Y. Cao, and H. Hu, “Self-supervised learning with swin transformers,” arXiv preprint arXiv:2105.04553, 2021\.
  * [119] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. H. Richemond, E. Buchatskaya, C. Doersch, B. A. Pires, Z. D. Guo, M. G. Azar, et al., “Bootstrap your own latent: A new approach to self-supervised learning,” arXiv preprint arXiv:2006.07733, 2020.
  * [120] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin, “Emerging properties in self-supervised vision transformers,” arXiv preprint arXiv:2104.14294, 2021.
  * [121] C. Li, J. Yang, P. Zhang, M. Gao, B. Xiao, X. Dai, L. Yuan, and J. Gao, “Efficient self-supervised vision transformers for representation learning,” arXiv preprint arXiv:2106.09785, 2021.
  * [122] Y. Wang, X. Zhang, T. Yang, and J. Sun, “Anchor detr: Query design for transformer-based detector,” 2021.
  * [123] T. Chen, S. Saxena, L. Li, D. J. Fleet, and G. Hinton, “Pix2seq: A language modeling framework for object detection,” 2021.
  * [124] Y. Fang, B. Liao, X. Wang, J. Fang, J. Qi, R. Wu, J. Niu, and W. Liu, “You only look at one sequence: Rethinking transformer in vision through object detection,” 2021.
  * [125] S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards real-time object detection with region proposal networks,” TPAMI, 2016.
  * [126] R. Girshick, “Fast R-CNN,” in ICCV, 2015.
  * [127] K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask R-CNN,” in ICCV, 2017.
  * [128] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” in CVPR, 2016.
  * [129] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg, “SSD: Single shot multibox detector,” in ECCV, 2016.
  * [130] T. Prangemeier, C. Reich, and H. Koeppl, “Attention-based transformers for instance segmentation of cells in microstructures,” in 2020 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp. 700–707, IEEE, 2020.
  * [131] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie, “Feature pyramid networks for object detection,” in CVPR, 2017.
  * [132] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” 2020.
  * [133] H. Wang, Y. Zhu, B. Green, H. Adam, A. Yuille, and L.-C. Chen, “Axial-DeepLab: Stand-alone axial-attention for panoptic segmentation,” arXiv preprint arXiv:2003.07853, 2020.
  * [134] S. Zheng, J. Lu, H. Zhao, X. Zhu, Z. Luo, Y. Wang, Y. Fu, J. Feng, T. Xiang, P. H. S. Torr, and L. Zhang, “Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers,” 2021.
  * [135] R. Strudel, R. Garcia, I. Laptev, and C. Schmid, “Segmenter: Transformer for semantic segmentation,” 2021.
  * [136] A. Kirillov, K. He, R. Girshick, C. Rother, and P. Dollár, “Panoptic segmentation,” in CVPR, 2019.
  * [137] G. Neuhold, T. Ollmann, S. Rota Bulo, and P. Kontschieder, “The mapillary vistas dataset for semantic understanding of street scenes,” in ICCV, 2017\.
  * [138] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “ImageNet: A large-scale hierarchical image database,” in CVPR, 2009.
  * [139] L. Yu, P. Poirson, S. Yang, A. C. Berg, and T. L. Berg, “Modeling context in referring expressions,” in ECCV, 2016.
  * [140] J. Mao, J. Huang, A. Toshev, O. Camburu, A. L. Yuille, and K. Murphy, “Generation and comprehension of unambiguous object descriptions,” in CVPR, 2016.
  * [141] S. Kazemzadeh, V. Ordonez, M. Matten, and T. Berg, “Referitgame: Referring to objects in photographs of natural scenes,” in EMNLP, 2014.
  * [142] N. Parmar, A. Vaswani, J. Uszkoreit, Ł. Kaiser, N. Shazeer, A. Ku, and D. Tran, “Image transformer,” in ICML, 2018.
  * [143] M. Chen, A. Radford, R. Child, J. Wu, H. Jun, D. Luan, and I. Sutskever, “Generative pretraining from pixels,” in ICML, 2020.
  * [144] P. Esser, R. Rombach, and B. Ommer, “Taming transformers for high-resolution image synthesis,” arXiv:2012.09841, 2020.
  * [145] Y. Jiang, S. Chang, and Z. Wang, “Transgan: Two transformers can make one strong gan,” 2021.
  * [146] A. K. Bhunia, S. Khan, H. Cholakkal, R. M. Anwer, F. S. Khan, and M. Shah, “Handwriting transformers,” arXiv preprint arXiv:2104.03964, 2021.
  * [147] A. Van den Oord, N. Kalchbrenner, L. Espeholt, O. Vinyals, A. Graves, et al., “Conditional image generation with pixelcnn decoders,” in NeurIPS, 2016.
  * [148] A. Krizhevsky, “Learning multiple layers of features from tiny images,” tech. rep., 2009.
  * [149] A. Coates, A. Ng, and H. Lee, “An analysis of single-layer networks in unsupervised feature learning,” in AISTATS, 2011.
  * [150] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” arXiv preprint arXiv:2002.05709, 2020.
  * [151] P. Bachman, R. Hjelm, and W. Buchwalter, “Learning representations by maximizing mutual information across views,” in NeurIPS, 2019.
  * [152] O. J. Hénaff, A. Srinivas, J. De Fauw, A. Razavi, C. Doersch, S. Eslami, and A. v. d. Oord, “Data-efficient image recognition with contrastive predictive coding,” arXiv preprint arXiv:1905.09272, 2019.
  * [153] Y. Tian, D. Krishnan, and P. Isola, “Contrastive multiview coding,” arXiv preprint arXiv:1906.05849, 2019.
  * [154] S. Khan, H. Rahmani, S. A. A. Shah, and M. Bennamoun, “A guide to convolutional neural networks for computer vision,” Synthesis Lectures on Computer Vision, 2018.
  * [155] A. Radford, L. Metz, and S. Chintala, “Unsupervised representation learning with deep convolutional generative adversarial networks,” arXiv preprint arXiv:1511.06434, 2015.
  * [156] C. Gao, Y. Chen, S. Liu, Z. Tan, and S. Yan, “Adversarialnas: Adversarial neural architecture search for gans,” in CVPR, pp. 5680–5689, 2020.
  * [157] T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila, “Analyzing and improving the image quality of stylegan,” in CVPR, pp. 8110–8119, 2020.
  * [158] S. Reed, Z. Akata, X. Yan, L. Logeswaran, B. Schiele, and H. Lee, “Generative adversarial text to image synthesis,” in ICML, 2016.
  * [159] H. Zhang, T. Xu, H. Li, S. Zhang, X. Wang, X. Huang, and D. N. Metaxas, “StackGAN: Text to photo-realistic image synthesis with stacked generative adversarial networks,” in ICCV, 2017.
  * [160] H. Zhang, T. Xu, H. Li, S. Zhang, X. Wang, X. Huang, and D. N. Metaxas, “StackGAN++: Realistic image synthesis with stacked generative adversarial networks,” TPAMI, 2018.
  * [161] T. Xu, P. Zhang, Q. Huang, H. Zhang, Z. Gan, X. Huang, and X. He, “AttnGAN: Fine-grained text to image generation with attentional generative adversarial networks,” in CVPR, 2018.
  * [162] D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” arXiv preprint arXiv:1312.6114, 2013.
  * [163] A. Razavi, A. van den Oord, and O. Vinyals, “Generating diverse high-fidelity images with vq-vae-2,” in NeurISP, 2019.
  * [164] J. Liang, J. Cao, G. Sun, K. Zhang, L. Van Gool, and R. Timofte, “Swinir: Image restoration using swin transformer,” in ICCVW, 2021.
  * [165] Z. Wang, X. Cun, J. Bao, and J. Liu, “Uformer: A general u-shaped transformer for image restoration,” arXiv preprint arXiv:2106.03106, 2021.
  * [166] Z. Lu, H. Liu, J. Li, and L. Zhang, “Efficient transformer for single image super-resolution,” arXiv preprint arXiv:2108.11084, 2021.
  * [167] Y. Zhang, K. Li, K. Li, L. Wang, B. Zhong, and Y. Fu, “Image super-resolution using very deep residual channel attention networks,” in ECCV, 2018.
  * [168] T. Dai, J. Cai, Y. Zhang, S. Xia, and L. Zhang, “Second-order attention network for single image super-resolution,” in CVPR, 2019.
  * [169] B. Niu, W. Wen, W. Ren, X. Zhang, L. Yang, S. Wang, K. Zhang, X. Cao, and H. Shen, “Single image super-resolution via a holistic attention network,” in ECCV, 2020.
  * [170] B. Lim, S. Son, H. Kim, S. Nah, and K. Mu Lee, “Enhanced deep residual networks for single image super-resolution,” in CVPRW, 2017.
  * [171] Y. Tai, J. Yang, and X. Liu, “Image super-resolution via deep recursive residual network,” in CVPR, 2017.
  * [172] W. Han, S. Chang, D. Liu, M. Yu, M. Witbrock, and T. Huang, “Image super-resolution via dual-state recurrent networks,” in CVPR, 2018.
  * [173] Y. Zhang, Y. Tian, Y. Kong, B. Zhong, and Y. Fu, “Residual dense network for image restoration,” TPAMI, 2020.
  * [174] X. Wang, K. Yu, S. Wu, J. Gu, Y. Liu, C. Dong, Y. Qiao, and C. Change Loy, “ESRGAN: enhanced super-resolution generative adversarial networks,” in ECCVW, 2018.
  * [175] S.-J. Park, H. Son, S. Cho, K.-S. Hong, and S. Lee, “SRFEAT: Single image super-resolution with feature discrimination,” in ECCV, 2018.
  * [176] M. S. Sajjadi, B. Scholkopf, and M. Hirsch, “EnhanceNet: Single image super-resolution through automated texture synthesis,” in ICCV, 2017.
  * [177] C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang, et al., “Photo-realistic single image super-resolution using a generative adversarial network,” in CVPR, 2017.
  * [178] J. Johnson, A. Alahi, and L. Fei-Fei, “Perceptual losses for real-time style transfer and super-resolution,” in ECCV, 2016.
  * [179] J. Ho, N. Kalchbrenner, D. Weissenborn, and T. Salimans, “Axial attention in multidimensional transformers,” arXiv preprint arXiv:1912.12180, 2019.
  * [180] G. Li, N. Duan, Y. Fang, M. Gong, D. Jiang, and M. Zhou, “Unicoder-VL: A universal encoder for vision and language by cross-modal pre-training.,” in AAAI, 2020.
  * [181] J. Lu, D. Batra, D. Parikh, and S. Lee, “Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks,” in NeurIPS, 2019.
  * [182] S. Lee, Y. Yu, G. Kim, T. Breuel, J. Kautz, and Y. Song, “Parameter efficient multimodal transformers for video representation learning,” arXiv preprint arXiv:2012.04124, 2020.
  * [183] S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. Lawrence Zitnick, and D. Parikh, “VQA: Visual question answering,” in ICCV, 2015.
  * [184] R. Zellers, Y. Bisk, A. Farhadi, and Y. Choi, “From recognition to cognition: Visual commonsense reasoning,” in CVPR, 2019.
  * [185] K.-H. Lee, X. Chen, G. Hua, H. Hu, and X. He, “Stacked cross attention for image-text matching,” in ECCV, 2018.
  * [186] A. Suhr, S. Zhou, A. Zhang, I. Zhang, H. Bai, and Y. Artzi, “A corpus for reasoning about natural language grounded in photographs,” arXiv preprint arXiv:1811.00491, 2018.
  * [187] J. Carreira, E. Noland, C. Hillier, and A. Zisserman, “A short note on the kinetics-700 human action dataset,” arXiv:1907.06987, 2019.
  * [188] K. Soomro, A. R. Zamir, and M. Shah, “UCF101: A dataset of 101 human actions classes from videos in the wild,” arXiv preprint arXiv:1212.0402, 2012\.
  * [189] J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, “Audio set: An ontology and human-labeled dataset for audio events,” in ICASSP, 2017.
  * [190] G. A. Sigurdsson, G. Varol, X. Wang, A. Farhadi, I. Laptev, and A. Gupta, “Hollywood in homes: Crowdsourcing data collection for activity understanding,” in ECCV, 2016.
  * [191] H. Tan and M. Bansal, “Vokenization: Improving language understanding with contextualized, visual-grounded supervision,” in EMNLP, 2020.
  * [192] W. Hao, C. Li, X. Li, L. Carin, and J. Gao, “Towards learning a generic agent for vision-and-language navigation via pre-training,” in CVPR, 2020.
  * [193] A. Majumdar, A. Shrivastava, S. Lee, P. Anderson, D. Parikh, and D. Batra, “Improving vision-and-language navigation with image-text pairs from the web,” arXiv preprint arXiv:2004.14973, 2020.
  * [194] K. Chen, J. K. Chen, J. Chuang, M. Vázquez, and S. Savarese, “Topological planning with transformers for vision-and-language navigation,” arXiv preprint arXiv:2012.05292, 2020.
  * [195] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al., “Learning transferable visual models from natural language supervision,” Image, vol. 2, p. T2, 2021.
  * [196] P. Sharma, N. Ding, S. Goodman, and R. Soricut, “Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning,” in ACL, 2018.
  * [197] L. Zhou, H. Palangi, L. Zhang, H. Hu, J. Corso, and J. Gao, “Unified vision-language pre-training for image captioning and vqa,” in AAAI, vol. 34, pp. 13041–13049, 2020.
  * [198] C. Sun, F. Baradel, K. Murphy, and C. Schmid, “Learning video representations using contrastive bidirectional transformer,” arXiv preprint arXiv:1906.05743, 2019.
  * [199] C. Alberti, J. Ling, M. Collins, and D. Reitter, “Fusion of detected objects in text for visual question answering,” in EMNLP, 2019.
  * [200] R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L.-J. Li, D. A. Shamma, et al., “Visual genome: Connecting language and vision using crowdsourced dense image annotations,” IJCV, 2017.
  * [201] V. Ordonez, G. Kulkarni, and T. L. Berg, “Im2text: Describing images using 1 million captioned photographs,” in NeurIPS, 2011.
  * [202] P. Zhang, X. Li, X. Hu, J. Yang, L. Zhang, L. Wang, Y. Choi, and J. Gao, “Vinvl: Revisiting visual representations in vision-language models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5579–5588, 2021.
  * [203] A. Kamath, M. Singh, Y. LeCun, I. Misra, G. Synnaeve, and N. Carion, “Mdetr–modulated detection for end-to-end multi-modal understanding,” arXiv preprint arXiv:2104.12763, 2021.
  * [204] J. Deng, Z. Yang, T. Chen, W. Zhou, and H. Li, “Transvg: End-to-end visual grounding with transformers,” 2021.
  * [205] M. Li and L. Sigal, “Referring transformer: A one-step approach to multi-task visual grounding,” arXiv preprint arXiv:2106.03089, 2021.
  * [206] Y. Du, Z. Fu, Q. Liu, and Y. Wang, “Visual grounding with transformers,” arXiv preprint arXiv:2105.04281, 2021.
  * [207] S. Ging, M. Zolfaghari, H. Pirsiavash, and T. Brox, “COOT: Cooperative hierarchical transformer for video-text representation learning,” arXiv preprint arXiv:2011.00597, 2020.
  * [208] H. Seong, J. Hyun, and E. Kim, “Video multitask transformer network,” in ICCV Workshops, pp. 0–0, 2019.
  * [209] Y. Wang, Z. Xu, X. Wang, C. Shen, B. Cheng, H. Shen, and H. Xia, “End-to-end video instance segmentation with transformers,” arXiv preprint arXiv:2011.14503, 2020.
  * [210] L. Zhou, Y. Zhou, J. Corso, R. Socher, and C. Xiong, “End-to-end dense video captioning with masked transformer,” in CVPR, 2018.
  * [211] D. Neimark, O. Bar, M. Zohar, and D. Asselmann, “Video transformer network,” arXiv preprint arXiv:2102.00719, 2021.
  * [212] A. Arnab, M. Dehghani, G. Heigold, C. Sun, M. Lučić, and C. Schmid, “Vivit: A video vision transformer,” arXiv preprint arXiv:2103.15691, 2021\.
  * [213] G. Bertasius, H. Wang, and L. Torresani, “Is space-time attention all you need for video understanding?,” in Proceedings of the International Conference on Machine Learning (ICML), July 2021.
  * [214] R. Krishna, K. Hata, F. Ren, L. Fei-Fei, and J. Carlos Niebles, “Dense-captioning events in videos,” in ICCV, pp. 706–715, 2017.
  * [215] L. Zhou, C. Xu, and J. Corso, “Towards automatic learning of procedures from web instructional videos,” in AAAI, vol. 32, 2018.
  * [216] C. Plizzari, M. Cannici, and M. Matteucci, “Spatial temporal transformer network for skeleton-based action recognition,” arXiv preprint arXiv:2008.07404, 2020.
  * [217] A. Shahroudy, J. Liu, T.-T. Ng, and G. Wang, “NTU RGB+D: A large scale dataset for 3d human activity analysis,” in CVPR, 2016.
  * [218] J. Liu, A. Shahroudy, M. Perez, G. Wang, L.-Y. Duan, and A. C. Kot, “NTU RGB+D 120: A large-scale benchmark for 3d human activity understanding,” TPAMI, 2019.
  * [219] H. Fan, B. Xiong, K. Mangalam, Y. Li, Z. Yan, J. Malik, and C. Feichtenhofer, “Multiscale vision transformers,” 2021.
  * [220] J. Wang, G. Bertasius, D. Tran, and L. Torresani, “Long-short temporal contrastive learning of video transformers,” arXiv preprint arXiv:2106.09212, 2021.
  * [221] L. Yang, Y. Fan, and N. Xu, “Video instance segmentation,” in ICCV, pp. 5188–5197, 2019.
  * [222] G. Bertasius and L. Torresani, “Classifying, segmenting, and tracking object instances in video with mask propagation,” in CVPR, pp. 9739–9748, 2020\.
  * [223] E. Triantafillou, T. Zhu, V. Dumoulin, P. Lamblin, U. Evci, K. Xu, R. Goroshin, C. Gelada, K. Swersky, P.-A. Manzagol, et al., “Meta-dataset: A dataset of datasets for learning to learn from few examples,” in ICLR, 2020\.
  * [224] T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” arXiv preprint arXiv:1609.02907, 2016.
  * [225] M. Zaheer, S. Kottur, S. Ravanbakhsh, B. Poczos, R. R. Salakhutdinov, and A. J. Smola, “Deep sets,” in NeurIPS, 2017.
  * [226] L. Liu, W. Hamilton, G. Long, J. Jiang, and H. Larochelle, “A universal representation transformer layer for few-shot image classification,” 2020.
  * [227] H. Edwards and A. Storkey, “Towards a neural statistician,” arXiv preprint arXiv:1606.02185, 2016.
  * [228] J. Lee, Y. Lee, J. Kim, A. Kosiorek, S. Choi, and Y. W. Teh, “Set transformer: A framework for attention-based permutation-invariant neural networks,” in ICML, 2019.
  * [229] J. Lee, Y. Lee, and Y. W. Teh, “Deep amortized clustering,” arXiv preprint arXiv:1909.13433, 2019.
  * [230] H. Zhao, L. Jiang, J. Jia, P. Torr, and V. Koltun, “Point transformer,” arXiv preprint arXiv:2012.09164, 2020.
  * [231] M.-H. Guo, J.-X. Cai, Z.-N. Liu, T.-J. Mu, R. R. Martin, and S.-M. Hu, “Pct: Point cloud transformer,” arXiv preprint arXiv:2012.09688, 2020.
  * [232] Z. Wu, S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang, and J. Xiao, “3D ShapeNets: A deep representation for volumetric shapes,” in CVPR, 2015\.
  * [233] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, J. Xiao, L. Yi, and F. Yu, “ShapeNet: An information-rich 3d model repository,” arXiv preprint arXiv:1512.03012, 2015.
  * [234] C. Ionescu, D. Papava, V. Olaru, and C. Sminchisescu, “Human3.6M: Large scale datasets and predictive methods for 3D human sensing in natural environments,” TPAMI, 2013.
  * [235] T. von Marcard, R. Henschel, M. J. Black, B. Rosenhahn, and G. Pons-Moll, “Recovering accurate 3d human pose in the wild using imus and a moving camera,” in ECCV, 2018.
  * [236] C. Zimmermann, D. Ceylan, J. Yang, B. Russell, M. Argus, and T. Brox, “FreiHAND: A dataset for markerless capture of hand pose and shape from single rgb images,” in ICCV, 2019.
  * [237] “OpenAI’s GPT-3 language model: A technical overview.” https://lambdalabs.com/blog/demystifying-gpt-3/.  Accessed: 2020-12-31.
  * [238] X. Zhai, A. Kolesnikov, N. Houlsby, and L. Beyer, “Scaling vision transformers,” 2021.
  * [239] P. Young, A. Lai, M. Hodosh, and J. Hockenmaier, “From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions,” TACL, 2014.
  * [240] Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh, “Making the v in vqa matter: Elevating the role of image understanding in visual question answering,” in CVPR, 2017.
  * [241] B. A. Plummer, L. Wang, C. M. Cervantes, J. C. Caicedo, J. Hockenmaier, and S. Lazebnik, “Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models,” in ICCV, 2015.
  * [242] C. R. Qi, L. Yi, H. Su, and L. J. Guibas, “PointNet++: Deep hierarchical feature learning on point sets in a metric space,” NeurIPS, 2017.
  * [243] H. Touvron, M. Cord, A. Sablayrolles, G. Synnaeve, and H. Jégou, “Going deeper with image transformers,” arXiv preprint arXiv:2103.17239, 2021\.
  * [244] S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He, “Aggregated residual transformations for deep neural networks,” in CVPR, 2017.
  * [245] R. Child, S. Gray, A. Radford, and I. Sutskever, “Generating long sequences with sparse transformers,” arXiv:1904.10509, 2019.
  * [246] N. Kitaev, Ł. Kaiser, and A. Levskaya, “Reformer: The efficient transformer,” in ICLR, 2020.
  * [247] I. Bello, “Lambdanetworks: Modeling long-range interactions without attention,” in International Conference on Learning Representations, 2021\.
  * [248] A. Vyas, A. Katharopoulos, and F. Fleuret, “Fast transformers with clustered attention,” NeurIPS, 2020.
  * [249] Y.-H. Wu, Y. Liu, X. Zhan, and M.-M. Cheng, “P2t: Pyramid pooling transformer for scene understanding,” arXiv preprint arXiv:2106.12011, 2021.
  * [250] A. Vaswani, P. Ramachandran, A. Srinivas, N. Parmar, B. Hechtman, and J. Shlens, “Scaling local self-attention for parameter efficient visual backbones,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12894–12904, 2021.
  * [251] X. Dong, J. Bao, D. Chen, W. Zhang, N. Yu, L. Yuan, D. Chen, and B. Guo, “Cswin transformer: A general vision transformer backbone with cross-shaped windows,” arXiv preprint arXiv:2107.00652, 2021.
  * [252] Y. Xiong, Z. Zeng, R. Chakraborty, M. Tan, G. Fung, Y. Li, and V. Singh, “Nystr$\backslash$” omformer: A nystr$\backslash$” om-based algorithm for approximating self-attention,” in AAAI, 2021.
  * [253] Y. Tay, D. Bahri, D. Metzler, D. Juan, Z. Zhao, and C. Zheng, “Synthesizer: Rethinking self-attention in transformer models,” in ICML, 2021.
  * [254] H. Peng, N. Pappas, D. Yogatama, R. Schwartz, N. A. Smith, and L. Kong, “Random feature attention,” in ICLR, 2021.
  * [255] K. Choromanski, V. Likhosherstov, D. Dohan, X. Song, A. Gane, T. Sarlos, P. Hawkins, J. Davis, A. Mohiuddin, L. Kaiser, et al., “Rethinking attention with performers,” in ICLR, 2021.
  * [256] Y. Tay, D. Bahri, L. Yang, D. Metzler, and D.-C. Juan, “Sparse sinkhorn attention,” in ICML, 2020.
  * [257] X. Chen, C.-J. Hsieh, and B. Gong, “When vision transformers outperform resnets without pretraining or strong data augmentations,” arXiv preprint arXiv:2106.01548, 2021.
  * [258] P. Foret, A. Kleiner, H. Mobahi, and B. Neyshabur, “Sharpness-aware minimization for efficiently improving generalization,” arXiv preprint arXiv:2010.01412, 2020.
  * [259] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818–2826, 2016.
  * [260] S. He, H. Luo, P. Wang, F. Wang, H. Li, and W. Jiang, “Transreid: Transformer-based object re-identification,” arXiv:2102.04378, 2021.
  * [261] D. R. So, C. Liang, and Q. V. Le, “The evolved transformer,” 2019.
  * [262] H. Wang, Z. Wu, Z. Liu, H. Cai, L. Zhu, C. Gan, and S. Han, “Hat: Hardware-aware transformers for efficient natural language processing,” 2020\.
  * [263] M. Chen, H. Peng, J. Fu, and H. Ling, “Autoformer: Searching transformers for visual recognition,” arXiv preprint arXiv:2107.00651, 2021.
  * [264] C. Li, T. Tang, G. Wang, J. Peng, B. Wang, X. Liang, and X. Chang, “Bossnas: Exploring hybrid cnn-transformers with block-wisely self-supervised neural architecture search,” arXiv preprint arXiv:2103.12424, 2021.
  * [265] B. Chen, P. Li, C. Li, B. Li, L. Bai, C. Lin, M. Sun, W. Ouyang, et al., “Glit: Neural architecture search for global and local image transformer,” arXiv preprint arXiv:2107.02960, 2021.
  * [266] M. Naseer, K. Ranasinghe, S. Khan, M. Hayat, F. S. Khan, and M.-H. Yang, “Intriguing properties of vision transformers,” arXiv preprint arXiv:2105.10497, 2021.
  * [267] E. Voita, D. Talbot, F. Moiseev, R. Sennrich, and I. Titov, “Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned,” arXiv preprint arXiv:1905.09418, 2019.
  * [268] S. Abnar and W. Zuidema, “Quantifying attention flow in transformers,” arXiv preprint arXiv:2005.00928, 2020.
  * [269] H. Chefer, S. Gur, and L. Wolf, “Transformer interpretability beyond attention visualization,” arXiv preprint arXiv:2012.09838, 2020.
  * [270] B. Li, S. Pandey, H. Fang, Y. Lyv, J. Li, J. Chen, M. Xie, L. Wan, H. Liu, and C. Ding, “FTRANS: energy-efficient acceleration of transformers using fpga,” in ISLPED, 2020.
  * [271] G. Bender, P.-J. Kindermans, B. Zoph, V. Vasudevan, and Q. Le, “Understanding and simplifying one-shot architecture search,” in ICML, 2018.
  * [272] Z. Guo, X. Zhang, H. Mu, W. Heng, Z. Liu, Y. Wei, and J. Sun, “Single path one-shot neural architecture search with uniform sampling,” arXiv preprint arXiv:1904.00420, 2019.
  * [273] H. Pham, M. Y. Guan, B. Zoph, Q. V. Le, and J. Dean, “Efficient neural architecture search via parameter sharing,” in ICML, 2018.
  * [274] A. Jaegle, F. Gimeno, A. Brock, A. Zisserman, O. Vinyals, and J. Carreira, “Perceiver: General perception with iterative attention,” arXiv preprint arXiv:2103.03206, 2021.
  * [275] A. Jaegle, S. Borgeaud, J.-B. Alayrac, C. Doersch, C. Ionescu, D. Ding, S. Koppula, D. Zoran, A. Brock, E. Shelhamer, et al., “Perceiver io: A general architecture for structured inputs & outputs,” arXiv preprint arXiv:2107.14795, 2021.