The core idea behind Vision Transformer is to transform an image into a sequence of visual tokens, and input this sequence into a Transformer encoder to generate a representation of the image. The Transformer encoder is standard, and so we will not discuss it here, given the introduction to Transformers we have presented so far in this chapter. Mapping a 2D image into a sequence of tokens needs some additional work. Suppose we have an image represented as an $H \times W \times C$ feature map, where $H$ is the height of the image, $W$ is the width of the image, and $C$ is the number of channels. The first step is to segment this image into a number of patches. Suppose all patches are squares of side length $P$. Then the resulting patches can be represented by feature maps of shape $P \times P \times C$. By ordering these patches in some way, we obtain a sequence of $\frac{HW}{P^2}$ patches, with each patch being treated as a “token”.

Given this patch sequence, the subsequent steps are straightforward. For the patch at each position, we obtain a $d$-dimensional embedding by a linear transformation of the input feature map. The input of the Transformer encoder is a sequence of $d$-dimensional vectors, each of which is the sum of the corresponding patch and positional embeddings. Figure <ref> illustrates the patching and embedding steps in Vision Transformer.

Illustration of Vision Transformer for image classification[Dosovitskiy et al., 2021]. There are three steps. In the first step, the input image is segmented into patches, which are then flattened and mapped into embeddings. In the second step, a Transformer encoder is employed to process the sequence of embeddings, representing the image as a real-valued vector (e.g., the output of the encoder at the first position). In the last step, a classifier is built on top of this image representation.

Once we have a sequence of vectors for representing the image, we can employ the Transformer encoder to encode the sequence. The encoding process is exactly the same as that in text encoding as discussed in Section <ref>. For classification problems, we need only a single representation of the input. It is convenient to take the output of the encoder at position 0 (denoted by $\mathbf{h}_0^{L}$) and feed it into a classifier. Given that the first token $[\mathrm{CLS}]$ serves as a special token that would be attended to by all other tokens, $\mathbf{h}_0^{L}$ provides an unbiased representation of the entire sequence.

Typically, a standard way to train Vision Transformer is to minimize some loss on labeled data, such as ImageNet. More recently, inspired by self-supervised learning in BERT-like models, there have been successful attempts to train Transformer-based image encoders on large-scale unlabeled data [Caron et al., 2021, Bao et al., 2021, He et al., 2022]. Note that one of the most significant contributions of Vision Transformer is that it unifies the representation models for different modalities. This suggests that if an object, whether an image or text, is represented as a sequence of embeddings, it can be easily modeled using the Transformer architecture.

§.§ Multimodal Models

The above discussion of Vision Transformer offers the possibility of unifying the representations from multiple modalities using the same Transformer architecture. In fact, many recent multimodal systems draw inspiration largely from Transformers [Xu et al., 2023]. Such systems convert objects from different modalities into vector sequences and feed these vectors into a single Transformer model. The output is a fused representation of all inputs, which can then be used in downstream systems.

As a simple example, consider the task of encoding a pair consisting of text and its corresponding image. First, we represent both the text and the image as sequences of embeddings that have the same dimensionality. This is a common step in sequence modeling, which we have confronted many times so far. We can do this by using either a simple embedding model (e.g., a word or patch embedding model) or a well-trained sequence model (e.g., a vision model). Then, these two sequences are concatenated into a long sequence involving both textual and visual embeddings. The follow-on step is standard: a Transformer encoder takes the concatenated sequence of embeddings as input and produces representations of the text and image as output. Note that concatenating textual and visual sequences is one of the simplest methods for vision-text modeling. There are several alternative ways to merge information from different modalities, for example, we can feed visual representations into the attention layers of a text encoder or decoder [Li et al., 2022, Alayrac et al., 2022].

The above multimodal encoder can be used in both encoder-only and encoder-decoder systems. For encoder-only systems, consider an example where, given an image and a description of it, we predict the class of the image using a classifier built on top of the encoder [Kim et al., 2021]. For encoder-decoder systems,  we pair the encoder with a decoder, as in sequence-to-sequence modeling [Cho et al., 2021]. For example, we might employ a Transformer decoder to generate text based on the output of the encoder. A common application of this architecture is visual question answering (VQA), where an image and a question about the image are provided, and the system is tasked with generating an answer [Antol et al., 2015]. The architectures of these models are illustrated in Figure <ref> (a-b).

Vision-text models. Blue boxes represent word+position embeddings, and red boxes represent image patch+position embeddings.

More recently, NLP has seen new advances by using large language models to deal with both textual and other forms of data, such as images, videos, and audio, leading to new breakthroughs in multimodal processing [Liu et al., 2023, Yin et al., 2023]. By representing all inputs as a sequence of token embeddings, the problem will be simple: we predict the next token given its context. This can be done by using decoder-only systems, as shown in Figure <ref> (c).

§ SUMMARY

Transformer models have achieved widespread use over the past few years since the concept of Transformer was proposed by Vaswani et al., 2017. This has accelerated the development of these models, leading to a great variety of new algorithms, systems and concepts. A thorough discussion of Transformers requires a broad scope, and so it is impossible to cover every problem and to provide a complete list of the corresponding references. While this chapter has presented a detailed introduction to Transformers, there are still topics that we did not mention, such as the theoretical aspects of these models. Figure <ref> shows an overview of Transformer models, where we attempt to give a big picture. Note that these models and related techniques can be classified in many different ways, and we just show one of them. To summarize, we would like to highlight the following points.

An overview of Transformers.

Foundations of Transformers. Although the impact of Transformers has been revolutionary, they are not completely "new" models. From a deep learning perspective, Transformers are composed of common building blocks, including word and positional embeddings [Bengio et al., 2003, Mikolov et al., 2013, Gehring et al., 2017], attention mechanisms [Bahdanau et al., 2014, Luong et al., 2015], residual connections [He et al., 2016], layer-normalization [Ba et al., 2016], and so on. Many of these components were presented in earlier systems, for example, similar ideas with QKV attention can be found in memory networks [Sukhbaatar et al., 2015] and hierarchical attention networks [Yang et al., 2016]. Transformers offer a novel approach to integrating these components, resulting in a unique architecture. For example, in Transformers, the combination of multi-head attention and dot-product QKV attention, along with the incorporation of layer-normalization and residual connections, gives rise to a distinctive neural network block, specifically a self-attention sub-layer. This design has since become a de facto standard in many follow-on sequence modeling systems.

Attention Models. The success of Transformers on NLP tasks has largely been attributed to the use of multi-head self-attention for sequence modeling. This has led to a surge of interest in enhancing the attention mechanisms within Transformers. While it is impossible to detail every attention model, there are several notable research directions. One prominent direction involves modifying the forms of QKV attention and multi-head attention for improved performance. The scope of this direction is vast, as there are numerous aspects to consider when enhancing Transformers [Lin et al., 2022]. For example, one may add new components to self-attention sub-layers to adapt them to specific tasks, resulting in various Transformer variants. A second direction is to incorporate prior knowledge into the design of attention models. This makes sense, because much of the emphasis in traditional NLP has been on using linguistic insights to guide system design, and we generally want NLP systems to be linguistically explainable. For example, many Transformer-based systems take syntactic parses as input in various forms and make use of syntax in sequence modeling. A third direction is to develop efficient attention models [Tay et al., 2020]. Self-attention has long been criticized for its quadratic time complexity and dependency on all previous  tokens for each new token. In response, many researchers have focused on simplifying the structure of self-attention, or on approximating it using sparse or recurrent models. This concern for efficiency also motivates the development of alternatives to self-attention, such as attention models with linear time complexity. In addition to exploring stronger and more efficient attention models, it is natural to examine what knowledge is learned by such models. Interestingly, researchers have found that the underlying structure of languages can be learned by multi-head self-attention models, although these models are not trained to represent such knowledge [Manning et al., 2020].

Word and Positional Embeddings. Transformers represent each input word as a word embedding, along with its positional embedding. Learning these word embeddings is not a specific problem for Transformers. We can either resort to well-trained word embeddings, such as the Word2Vec or GloVe embeddings, or treat them as learnable parameters of Transformers. A related issue is tokenization of the input sequences. In general, tokenization impacts the number of resulting tokens and the difficulty of learning the corresponding embeddings. In many applications, therefore, one needs to carefully choose a tokenization method. Furthermore, positional embedding plays an important role in Transformers, as the attention mechanisms are order-insensitive by design [Dufter et al., 2022]. Although positional embedding is a general problem, much of the research is focused on improving Transformers, leading to modifications to Transformer models [Shaw et al., 2018, Huang et al., 2018]. Additionally, studies show that, when we deal with sequences that are much longer than those in training data, extrapolation can be achieved by replacing sinusoidal positional embeddings with rotary positional embeddings or simply scaling attention weights with a positional scalar [Raffel et al., 2020, Su et al., 2021, Press et al., 2021].

Training and Model Scaling. In the era of deep learning, powerful systems are typically obtained by using large neural networks. A simple approach to increasing the model capacity of Transformers is to stack more layers and/or enlarge the size of each representation. We can see many cases where deep and wide Transformer models consistently outperform small models. However, challenges arise when we attempt to train extremely large Transformer models, especially when gradient descent is applied over vast amounts of data, demanding substantial computational resources. An engineering solution is to distribute the training across a cluster of computers [Lepikhin et al., 2021, Chowdhery et al., 2022]. While distributed training is a very general method and is not restricted to Transformers, it indeed influences the design of model architectures, for example, sparse expert models can ease the training with distributed parameters, serving as the foundation for many expansive Transformer-based systems. Scaling up the training of Transformers allows us to study the scaling law of large neural networks: how model performance relates to model size, training data size, and training cost [Hestness et al., 2017, Kaplan et al., 2020]. This is sometimes accompanied by an interesting behavior, known as emergence [Wei et al., 2022]. In recent NLP research, the acquisition of emergent abilities has been considered one of the prerequisites for developing strong language models.

Efficient Models. There are different goals for efficiency. For example, one may wish a system to be memory efficient when the problem is memory bound, or one may wish it to be speed efficient when latency is an important consideration. In general, we need to seek a balance between these goals, resulting in different efficiency optimizations. In the context of Transformers, many of these optimizations are achieved by modifying the attention models, as mentioned above. For example, several variants of the self-attention models are proposed to reduce the memory footprint when processing long sequences [Tay et al., 2020]. Similarly, other variants aim to reduce computation and thus give lower latency. Furthermore, being a type of neural network, Transformers can be optimized in ways independent of model architectures. Typical methods include but are not limited to conditional computation, knowledge distillation, structured pruning, and sequence compression. Efficiency optimizations can also be considered from the perspective of computer architecture [Kim et al., 2023]. For example, when applying Transformers to sequence-to-sequence problems, the encoding and decoding processes are generally compute-intensive and IO-intensive, respectively. Therefore, we can employ different optimization methods for different components of Transformers.

Inference. The inference problem is commonly discussed in sequence generation. In NLP, we often need to find the “best” hypothesis in a space involving sequences of tens or even hundreds of tokens over a vocabulary. Considering this an instance of the search problem in artificial intelligence, many algorithms can be applied, such as breadth-first search, depth-first search and A* search. In many practical applications of NLP, the efficiency of the search systems is an important consideration. As a result, optimized search algorithms are required. Most of these algorithms have been explored in machine translation and ASR, and are directly applicable to neural text generation models like Transformer. There are also optimizations of conventional decoding methods tailored to Transformers [Leviathan et al., 2023]. Moreover, the above-mentioned efficient approaches, such as the efficient attention models, are also in widespread use, with many successful examples in deploying neural machine translation systems and large language models [Heafield et al., 2021, Dao et al., 2023].

Applications. Applications of Transformers cover a wide variety of NLP problems. During the development of Transformers, they were at first used to build supervised models that perform particular tasks. Later, a greater success was achieved by using them as backbone networks for large scale self-supervised learning of foundation models [Bommasani et al., 2021]. This markedly changed the paradigm in NLP. We need only pre-train a model to obtain general knowledge of languages on huge amounts of text. Then, we adapt this model to downstream tasks using methods with little effort, such as fine-tuning or prompting. Over the past few years, we have also seen an explosion of applications for Transformers in fields other than NLP, such as computer vision, speech processing, and bioinformatics. The idea behind these applications is that we can represent any input data as a sequence of tokens and directly employ Transformers to model this sequence. This approach extends Transformers to general representation models across different modalities, making it easier to use Transformers for handling multi-modal data.

Large Language Models as Foundation Models. Transformers form the basis of recent large language models, such as the GPT series, which show surprising breakthroughs in NLP, and even in artificial general intelligence (AGI) [Bubeck et al., 2023, Yang et al., 2023]. Much of the research in large language models is more or less related to Transformers. For example, as discussed in Section <ref>, the problem of training these language models is the same as that of training Transformer decoders. And the modifications to Transformer decoders can be directly applied to large language models. On the other hand, the rapid development of large language models has also driven further improvements in various techniques for Transformers, such as efficient and low-cost adaptation of large Transformers to different tasks.

Theoretical Analysis. Although Transformers have shown strong empirical results in various fields, their theoretical aspects have received relatively less attention compared to the extensive research on model improvement and engineering. This is not a specific problem for Transformers, but a common problem for the NLP and machine learning communities. In response, researchers have made attempts to analyze Transformers more deeply. One way is to view Transformers as deep neural networks and interpret them via mathematical tools. For example, the residual networks in Transformers are mathematically equivalent to the Euler solvers for ODEs. This equivalence suggests that we can leverage insights from numerical ODE methods to inform model design. Another promising avenue of research aims to develop a theoretical understanding of the self-attention mechanism, which distinguishes Transformers from other deep learning models. For example, there have been studies on interpreting self-attention and Transformers from machine learning perspectives, such as data compression [Yu et al., 2023], optimization [Li et al., 2022], and function approximation [Yun et al., 2019]. Moreover, Transformers can also be related to formal systems, including Turing machines [Pérez et al., 2018], counter machines [Bhattamishra et al., 2020],  regular and context-free languages [Hahn, 2020], Boolean circuits [Hao et al., 2022, Merrill et al., 2022], programming languages [Weiss et al., 2021], first-order logic [Chiang et al., 2023], and so on. These provide tools to study the expressivity of Transformers. It is, however, worth noting that, while we can understand Transformers in several different ways, there are no general theories to explain the nature of these models. Perhaps this is a challenge for the field of machine learning, and many researchers are working on this issue. But it is indeed an important issue, as the development of the theories behind complex neural networks like Transformers can help develop systems with explainable and predictable behaviors.

§ ACKNOWLEDGEMENTS

We would like to thank Yongyu Mu, Chenglong Wang, Bei Li, Weiqiao Shan, Yuchun Fan, Kaiyan Chang, Tong Zheng, and Huiwen Bao for their suggestions on improving the early version of this work.

[Adi et al., 2016]
Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg.
Fine-grained analysis of sentence embeddings using auxiliary
prediction tasks.
In Proceedings of International Conference on Learning
Representations, 2016.

[Ainslie et al., 2020]
Joshua Ainslie, Santiago Ontanon, Chris Alberti, Vaclav Cvicek, Zachary Fisher,
Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, and Li Yang.
Etc: Encoding long and structured inputs in transformers.
In Proceedings of the 2020 Conference on Empirical Methods in
Natural Language Processing (EMNLP), pages 268–284, 2020.

[Alayrac et al., 2022]
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr,
Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds,
et al.
Flamingo: a visual language model for few-shot learning.
Advances in Neural Information Processing Systems,
35:0 23716–23736, 2022.

[Antol et al., 2015]
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra,
C Lawrence Zitnick, and Devi Parikh.
Vqa: Visual question answering.
In Proceedings of the IEEE international conference on computer
vision, pages 2425–2433, 2015.

[Åström and Wittenmark, 2013]
Karl J Åström and Björn Wittenmark.
Computer-controlled systems: theory and design.
Courier Corporation, 2013.

[Ba et al., 2016]
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.
Layer normalization.
arXiv preprint arXiv:1607.06450, 2016.

[Bachlechner et al., 2021]
Thomas Bachlechner, Bodhisattwa Prasad Majumder, Henry Mao, Gary Cottrell, and
Julian McAuley.
Rezero is all you need: Fast convergence at large depth.
In Proceedings of Uncertainty in Artificial Intelligence,
pages 1352–1361. PMLR, 2021.

[Bahdanau et al., 2014]
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio.
Neural machine translation by jointly learning to align and
arXiv preprint arXiv:1409.0473, 2014.

[Bai et al., 2021]
Jiangang Bai, Yujing Wang, Yiren Chen, Yaming Yang, Jing Bai, Jing Yu, and
Yunhai Tong.
Syntax-bert: Improving pre-trained transformers with syntax trees.
In Proceedings of the 16th Conference of the European Chapter
of the Association for Computational Linguistics: Main Volume, pages
3011–3020, 2021.

[Bao et al., 2021]
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei.
Beit: Bert pre-training of image transformers.
In Proceedings of International Conference on Learning
Representations, 2021.

[Barham et al., 2022]
Paul Barham, Aakanksha Chowdhery, Jeff Dean, Sanjay Ghemawat, Steven Hand,
Daniel Hurt, Michael Isard, Hyeontaek Lim, Ruoming Pang, Sudip Roy, et al.
Pathways: Asynchronous distributed dataflow for ml.
In Proceedings of Machine Learning and Systems, volume 4,
pages 430–449, 2022.

[Belinkov, 2022]
Yonatan Belinkov.
Probing classifiers: Promises, shortcomings, and advances.
Computational Linguistics, 480 (1):0 207–219,

[Bello, 2020]
Irwan Bello.
Lambdanetworks: Modeling long-range interactions without attention.
In Proceedings of International Conference on Learning
Representations, 2020.

[Beltagy et al., 2020]
Iz Beltagy, Matthew E. Peters, and Arman Cohan.
Longformer: The long-document transformer.
arXiv:2004.05150, 2020.

[Bengio et al., 2015]
Emmanuel Bengio, Pierre-Luc Bacon, Joelle Pineau, and Doina Precup.
Conditional computation in neural networks for faster models.
arXiv preprint arXiv:1511.06297, 2015.

[Bengio et al., 2003]
Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin.
A neural probabilistic language model.
Journal of Machine Learning Research, 3:0 1137–1155,

[Bengio et al., 2013]
Yoshua Bengio, Nicholas Léonard, and Aaron Courville.
Estimating or propagating gradients through stochastic neurons for
conditional computation.
arXiv preprint arXiv:1308.3432, 2013.

[Bhattamishra et al., 2020]
Satwik Bhattamishra, Kabir Ahuja, and Navin Goyal.
On the ability and limitations of transformers to recognize formal
In Proceedings of the 2020 Conference on Empirical Methods in
Natural Language Processing (EMNLP), pages 7096–7116, 2020.

[Bommasani et al., 2021]
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney
von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma
Brunskill, Erik Brynjolfsson, S. Buch, Dallas Card, Rodrigo Castellon,
Niladri S. Chatterji, Annie S. Chen, Kathleen A. Creel, Jared Davis, Dora
Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John
Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale,
Lauren E. Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha,
Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong,
Kyle Hsu, Jing Huang, Thomas F. Icard, Saahil Jain, Dan Jurafsky, Pratyusha
Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, O. Khattab,
Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar,
Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa
Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir P.
Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan,
Deepak Narayanan, Benjamin Newman, Allen Nie, Juan Carlos Niebles, Hamed
Nilforoshan, J. F. Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou,
Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi
Raghunathan, Robert Reich, Hongyu Ren, Frieda Rong, Yusuf H. Roohani, Camilo
Ruiz, Jack Ryan, Christopher R'e, Dorsa Sadigh, Shiori Sagawa, Keshav
Santhanam, Andy Shih, Krishna Parasuram Srinivasan, Alex Tamkin, Rohan Taori,
Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu,
Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You,
Matei A. Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang,
Lucia Zheng, Kaitlyn Zhou, and Percy Liang.
On the opportunities and risks of foundation models.
ArXiv, 2021.

[Bondarenko et al., 2021]
Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort.
Understanding and overcoming the challenges of efficient transformer
In Proceedings of the 2021 Conference on Empirical Methods in
Natural Language Processing, pages 7947–7969, 2021.

[Brown et al., 2020]
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla
Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell,
et al.
Language models are few-shot learners.
Advances in neural information processing systems,
33:0 1877–1901, 2020.

[Bubeck et al., 2023]
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric
Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg,
et al.
Sparks of artificial general intelligence: Early experiments with
arXiv preprint arXiv:2303.12712, 2023.

[Burchi and Vielzeuf, 2021]
Maxime Burchi and Valentin Vielzeuf.
Efficient conformer: Progressive downsampling and grouped attention
for automatic speech recognition.
In Proceedings of 2021 IEEE Automatic Speech Recognition and
Understanding Workshop (ASRU), pages 8–15. IEEE, 2021.

[Caron et al., 2021]
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal,
Piotr Bojanowski, and Armand Joulin.
Emerging properties in self-supervised vision transformers.
In Proceedings of the IEEE/CVF international conference on
computer vision, pages 9650–9660, 2021.

[Cer et al., 2018]
Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St
John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al.
Universal sentence encoder.
arXiv preprint arXiv:1803.11175, 2018.

[Chen et al., 2018]
Kehai Chen, Rui Wang, Masao Utiyama, Eiichiro Sumita, and Tiejun Zhao.
Syntax-directed attention for neural machine translation.
In Proceedings of the AAAI conference on artificial
intelligence, 2018a.

[Chen et al., 2018]
Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey,
George Foster, Llion Jones, Mike Schuster, Noam Shazeer, Niki Parmar, Ashish
Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Zhifeng Chen, Yonghui Wu, and
Macduff Hughes.
The best of both worlds: Combining recent advances in neural machine
In Proceedings of the 56th Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers), pages 76–86,

[Chen et al., 2018]
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud.
Neural ordinary differential equations.
Advances in neural information processing systems, 31,

[Chen et al., 2015]
Tianqi Chen, Ian Goodfellow, and Jonathon Shlens.
Net2net: Accelerating learning via knowledge transfer.
arXiv preprint arXiv:1511.05641, 2015.

[Chiang et al., 2023]
David Chiang, Peter Cholak, and Anand Pillay.
Tighter bounds on the expressivity of transformer encoders.
arXiv preprint arXiv:2301.10743, 2023.

[Child et al., 2019]
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever.
Generating long sequences with sparse transformers.
arXiv preprint arXiv:1904.10509, 2019.

[Cho et al., 2021]
Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal.
Unifying vision-and-language tasks via text generation.
In International Conference on Machine Learning, pages
1931–1942. PMLR, 2021.

[Choe and Charniak, 2016]
Do Kook Choe and Eugene Charniak.
Parsing as language modeling.
In Proceedings of the 2016 Conference on Empirical Methods in
Natural Language Processing, pages 2331–2336, 2016.

[Choromanski et al., 2020]
Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song,
Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz
Mohiuddin, Lukasz Kaiser, et al.
Rethinking attention with performers.
In Proceedings of International Conference on Learning
Representations, 2020.

[Chowdhery et al., 2022]
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra,
Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian
Gehrmann, et al.
Palm: Scaling language modeling with pathways.
arXiv preprint arXiv:2204.02311, 2022.

[Clark et al., 2019]
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning.
What does bert look at? an analysis of bert’s attention.
In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing
and Interpreting Neural Networks for NLP, pages 276–286, 2019.

[Conneau et al., 2018]
Alexis Conneau, Germán Kruszewski, Guillaume Lample, Loïc Barrault,
and Marco Baroni.
What you can cram into a single vector: Probing sentence embeddings
for linguistic properties.
In Proceedings of the 56th Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers), pages 2126–2136,

[Currey and Heafield, 2018]
Anna Currey and Kenneth Heafield.
Multi-source syntactic neural machine translation.
In Proceedings of the 2018 Conference on Empirical Methods in
Natural Language Processing, pages 2961–2966, 2018.

[Dai et al., 2019]
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G Carbonell, Quoc Le, and Ruslan
Transformer-xl: Attentive language models beyond a fixed-length
In Proceedings of the 57th Annual Meeting of the Association
for Computational Linguistics, pages 2978–2988, 2019.

[Dao et al., 2022]
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré.
Flashattention: Fast and memory-efficient exact attention with
Advances in Neural Information Processing Systems,
35:0 16344–16359, 2022.

[Dao et al., 2023]
Tri Dao, Daniel Haziza, Francisco Massa, and Grigory Sizov.
Flash-decoding for long-context inference.
<https://pytorch.org/blog/flash-decoding/>, 2023.
Retrieved 2023-10-23.

[Dehghani et al., 2018]
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Łukasz
Universal transformers.
arXiv preprint arXiv:1807.03819, 2018.

[Del Corro et al., 2023]
Luciano Del Corro, Allie Del Giorno, Sahaj Agarwal, Bin Yu, Ahmed Awadallah,
and Subhabrata Mukherjee.
Skipdecode: Autoregressive skip decoding with batching and caching
for efficient llm inference.
arXiv preprint arXiv:2307.02628, 2023.

[Devlin et al., 2019]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.
Bert: Pre-training of deep bidirectional transformers for language
In Proceedings of the 2019 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, 2019.

[Di Gangi et al., 2019]
Mattia Antonino Di Gangi, Matteo Negri, Roldano Cattoni, Roberto Dessi, and
Marco Turchi.
Enhancing transformer for end-to-end speech-to-text translation.
In Proceedings of Machine Translation Summit XVII: Research
Track, pages 21–31, 2019.

[Ding et al., 2021]
Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang
Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al.
Cogview: Mastering text-to-image generation via transformers.
Advances in Neural Information Processing Systems,
34:0 19822–19835, 2021.

[Dosovitskiy et al., 2021]
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn,
Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg
Heigold, Sylvain Gelly, et al.
An image is worth 16x16 words: Transformers for image recognition at
In Proceedings of ICLR 2021, 2021.

[Dufter et al., 2022]
Philipp Dufter, Martin Schmitt, and Hinrich Schütze.
Position information in transformers: An overview.
Computational Linguistics, 480 (3):0 733–763,

[Ee, 2017]
Weinan Ee.
A proposal on machine learning via dynamical systems.
Communications in Mathematics and Statistics, 5:0
1–11, 02 2017.

[Elbayad et al., 2020]
Maha Elbayad, Jiatao Gu, Edouard Grave, and Michael Auli.
Depth-adaptive transformer.
In Proceedings of International Conference on Learning
Representations, 2020.

[Elsken et al., 2019]
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter.
Neural architecture search: A survey.
The Journal of Machine Learning Research, 200
(1):0 1997–2017, 2019.

[Fan et al., 2021]
Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra
Malik, and Christoph Feichtenhofer.
Multiscale vision transformers.
In Proceedings of the IEEE/CVF International Conference on
Computer Vision, pages 6824–6835, 2021.

[Fan et al., 2020]
Yang Fan, Shufang Xie, Yingce Xia, Lijun Wu, Tao Qin, Xiang-Yang Li, and
Tie-Yan Liu.
Multi-branch attentive transformer.
arXiv preprint arXiv:2006.10270, 2020.

[Fedus et al., 2022]
William Fedus, Jeff Dean, and Barret Zoph.
A review of sparse expert models in deep learning.
arXiv preprint arXiv:2209.01667, 2022a.

[Fedus et al., 2022]
William Fedus, Barret Zoph, and Noam Shazeer.
Switch transformers: Scaling to trillion parameter models with simple
and efficient sparsity.
The Journal of Machine Learning Research, 230
(1):0 5232–5270, 2022b.

[Fu et al., 2022]
Daniel Y Fu, Tri Dao, Khaled Kamal Saab, Armin W Thomas, Atri Rudra, and
Christopher Re.
Hungry hungry hippos: Towards language modeling with state space
In Proceedings of The Eleventh International Conference on
Learning Representations, 2022.

[Gehring et al., 2017]
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin.
Convolutional sequence to sequence learning.
In International conference on machine learning, pages
1243–1252. PMLR, 2017.

[Gholami et al., 2022]
Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W Mahoney, and Kurt
A survey of quantization methods for efficient neural network
In Low-Power Computer Vision, pages 291–326. Chapman and
Hall/CRC, 2022.

[Glorot and Bengio, 2010]
Xavier Glorot and Yoshua Bengio.
Understanding the difficulty of training deep feedforward neural
In Proceedings of the thirteenth international conference on
artificial intelligence and statistics, pages 249–256. JMLR Workshop and
Conference Proceedings, 2010.

[Gomez et al., 2017]
Aidan N Gomez, Mengye Ren, Raquel Urtasun, and Roger B Grosse.
The reversible residual network: Backpropagation without storing
Advances in neural information processing systems, 30, 2017.

[Gou et al., 2021]
Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao.
Knowledge distillation: A survey.
International Journal of Computer Vision, 129:0
1789–1819, 2021.

[Gray, 1998]
Robert M. Gray.
IEEE transactions on information theory, 440
(6):0 2325–2383, 1998.

[Gu et al., 2021]
Albert Gu, Karan Goel, and Christopher Ré.
Efficiently modeling long sequences with structured state spaces.
In Proceedings of International Conference on Learning
Representations, 2021.

[Gu et al., 2022]
Albert Gu, Karan Goel, Ankit Gupta, and Christopher Ré.
On the parameterization and initialization of diagonal state space
Advances in Neural Information Processing Systems,
35:0 35971–35983, 2022a.

[Gu et al., 2022]
Albert Gu, Karan Goel, Khaled Saab, and Chris Ré.
Structured state spaces: Combining continuous-time, recurrent, and
convolutional models.
Retrieved 2022-01-14.

[Gulati et al., 2020]
Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu,
Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruoming Pang.
Conformer: Convolution-augmented transformer for speech recognition.
Proceedings of Interspeech 2020, pages 5036–5040, 2020.

[Guo et al., 2020]
Qipeng Guo, Xipeng Qiu, Pengfei Liu, Xiangyang Xue, and Zheng Zhang.
Multi-scale self-attention for text classification.
In Proceedings of the AAAI Conference on Artificial
Intelligence, volume 34, pages 7847–7854, 2020.

[Gupta et al., 2004]
Madan Gupta, Liang Jin, and Noriyasu Homma.
Static and dynamic neural networks: from fundamentals to
advanced theory.
John Wiley & Sons, 2004.

[Guu et al., 2020]
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang.
Retrieval augmented language model pre-training.
In Proceedings of International conference on machine
learning, pages 3929–3938. PMLR, 2020.

[Haber and Ruthotto, 2017]
Eldad Haber and Lars Ruthotto.
Stable architectures for deep neural networks.
Inverse problems, 340 (1):0 014004, 2017.

[Hahn, 2020]
Michael Hahn.
Theoretical limitations of self-attention in neural sequence models.
Transactions of the Association for Computational Linguistics,
8:0 156–171, 2020.

[Han et al., 2022]
Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu,
Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, et al.
A survey on vision transformer.
IEEE transactions on pattern analysis and machine
intelligence, 450 (1):0 87–110, 2022.

[Han et al., 2020]
Wei Han, Zhengdong Zhang, Yu Zhang, Jiahui Yu, Chung-Cheng Chiu, James Qin,
Anmol Gulati, Ruoming Pang, and Yonghui Wu.
Contextnet: Improving convolutional neural networks for automatic
speech recognition with global context.
In Proceedings of Interspeech 2020, pages 3610–3614, 2020.

[Han et al., 2021]
Yizeng Han, Gao Huang, Shiji Song, Le Yang, Honghui Wang, and Yulin Wang.
Dynamic neural networks: A survey.
IEEE Transactions on Pattern Analysis and Machine
Intelligence, 440 (11):0 7436–7456, 2021.

[Hao et al., 2019]
Jie Hao, Xing Wang, Shuming Shi, Jinfeng Zhang, and Zhaopeng Tu.
Multi-granularity self-attention for neural machine translation.
In Proceedings of the 2019 Conference on Empirical Methods in
Natural Language Processing and the 9th International Joint Conference on
Natural Language Processing (EMNLP-IJCNLP), pages 887–897, 2019.

[Hao et al., 2022]
Yiding Hao, Dana Angluin, and Robert Frank.
Formal language recognition by hard attention transformers:
Perspectives from circuit complexity.
Transactions of the Association for Computational Linguistics,
10:0 800–810, 2022.

[He et al., 2016]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Deep residual learning for image recognition.
In Proceedings of the IEEE conference on computer vision and
pattern recognition, pages 770–778, 2016a.

[He et al., 2016]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Identity mappings in deep residual networks.
In Proceedings of ECCV 2016, pages 630–645,

[He et al., 2022]
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross
Masked autoencoders are scalable vision learners.
In Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition, pages 16000–16009, 2022.

[He et al., 2021]
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen.
Deberta: Decoding-enhanced bert with disentangled attention.
In Proceedings of International Conference on Learning
Representations, 2021.

[Heafield et al., 2021]
Kenneth Heafield, Qianqian Zhu, and Roman Grundkiewicz.
Findings of the WMT 2021 shared task on efficient translation.
In Proceedings of the Sixth Conference on Machine Translation,
pages 639–651, 2021.

[Hestness et al., 2017]
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun,
Hassan Kianinejad, Md Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou.
Deep learning scaling is predictable, empirically.
arXiv preprint arXiv:1712.00409, 2017.

[Hewitt and Liang, 2019]
John Hewitt and Percy Liang.
Designing and interpreting probes with control tasks.
In Proceedings of the 2019 Conference on Empirical Methods in
Natural Language Processing and the 9th International Joint Conference on
Natural Language Processing (EMNLP-IJCNLP), pages 2733–2743, 2019.

[Hill et al., 2016]
Felix Hill, Kyunghyun Cho, and Anna Korhonen.
Learning distributed representations of sentences from unlabelled
In Proceedings of the 2016 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies, pages 1367–1377, 2016.

[Hinton et al., 2015]
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.
Distilling the knowledge in a neural network.
arXiv preprint arXiv:1503.02531, 2015.

[Hou et al., 2020]
Lu Hou, Zhiqi Huang, Lifeng Shang, Xin Jiang, Xiao Chen, and Qun Liu.
Dynabert: Dynamic bert with adaptive width and depth.
Advances in Neural Information Processing Systems,
33:0 9782–9793, 2020.

[Howard et al., 2019]
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing
Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al.
Searching for mobilenetv3.
In Proceedings of the IEEE/CVF international conference on
computer vision, pages 1314–1324, 2019.

[Hu et al., 2021]
Chi Hu, Chenglong Wang, Xiangnan Ma, Xia Meng, Yinqiao Li, Tong Xiao, Jingbo
Zhu, and Changliang Li.
Ranknas: Efficient neural architecture search by pairwise ranking.
In Proceedings of the 2021 Conference on Empirical Methods in
Natural Language Processing, pages 2469–2480, 2021.

[Huang et al., 2018]
Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Ian Simon, Curtis
Hawthorne, Noam Shazeer, Andrew M Dai, Matthew D Hoffman, Monica Dinculescu,
and Douglas Eck.
Music transformer: Generating music with long-term structure.
In Proceedings of International Conference on Learning
Representations, 2018.

[Huang et al., 2016]
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger.
Deep networks with stochastic depth.
In Proceedings of the 14th European Conference, pages
646–661. Springer, 2016.

[Huang et al., 2017]
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger.
Densely connected convolutional networks.
In Proceedings of the IEEE conference on computer vision and
pattern recognition, pages 4700–4708, 2017.

[Huang et al., 2020]
Zhiheng Huang, Davis Liang, Peng Xu, and Bing Xiang.
Improve transformer models with better relative position embeddings.
In Findings of the Association for Computational Linguistics:
EMNLP 2020, pages 3327–3335, 2020.

[Ivanov et al., 2021]
Andrei Ivanov, Nikoli Dryden, Tal Ben-Nun, Shigang Li, and Torsten Hoefler.
Data movement is all you need: A case study on optimizing
In Proceedings of Machine Learning and Systems, volume 3,
pages 711–732, 2021.

[Jacob et al., 2018]
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew
Howard, Hartwig Adam, and Dmitry Kalenichenko.
Quantization and training of neural networks for efficient
integer-arithmetic-only inference.
In Proceedings of the IEEE conference on computer vision and
pattern recognition, pages 2704–2713, 2018.

[Jaegle et al., 2021]
Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin
Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan
Shelhamer, et al.
Perceiver io: A general architecture for structured inputs &
In Proceedings of International Conference on Learning
Representations, 2021.

[Kaplan et al., 2020]
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon
Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.
Scaling laws for neural language models.
arXiv preprint arXiv:2001.08361, 2020.

[Katharopoulos et al., 2020]
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François
Transformers are rnns: Fast autoregressive transformers with linear
In International conference on machine learning, pages
5156–5165. PMLR, 2020.

[Kernes, 2021]
Jonathan Kernes.
Master positional encoding: Part i, 05 2021.

[Kidger, 2022]
Patrick Kidger.
On neural differential equations.
arXiv preprint arXiv:2202.02435, 2022.

[Kim and Cho, 2021]
Gyuwan Kim and Kyunghyun Cho.
Length-adaptive transformer: Train once with length drop, use anytime
with search.
In Proceedings of the 59th Annual Meeting of the Association
for Computational Linguistics and the 11th International Joint Conference on
Natural Language Processing (Volume 1: Long Papers), pages 6501–6511, 2021.

[Kim et al., 2019]
Najoung Kim, Roma Patel, Adam Poliak, Patrick Xia, Alex Wang, Tom McCoy, Ian
Tenney, Alexis Ross, Tal Linzen, Benjamin Van Durme, et al.
Probing what different nlp tasks teach machines about function word
In Proceedings of the Eighth Joint Conference on Lexical and
Computational Semantics (* SEM 2019), pages 235–249, 2019.

[Kim et al., 2023]
Sehoon Kim, Coleman Hooper, Thanakul Wattanawong, Minwoo Kang, Ruohan Yan,
Hasan Genc, Grace Dinh, Qijing Huang, Kurt Keutzer, Michael W Mahoney, et al.
Full stack optimization of transformer inference: a survey.
arXiv preprint arXiv:2302.14017, 2023.

[Kim et al., 2021]
Wonjae Kim, Bokyung Son, and Ildoo Kim.
Vilt: Vision-and-language transformer without convolution or region
In Proceedings of International Conference on Machine
Learning, pages 5583–5594. PMLR, 2021.

[Kim and Rush, 2016]
Yoon Kim and Alexander M Rush.
Sequence-level knowledge distillation.
In Proceedings of the 2016 Conference on Empirical Methods in
Natural Language Processing, pages 1317–1327, 2016.

[Kim and Awadalla, 2020]
Young Jin Kim and Hany Hassan Awadalla.
Fastformers: Highly efficient transformer models for natural language
In Proceedings of SustaiNLP: Workshop on Simple and Efficient
Natural Language Processing, pages 149–158, 2020.

[Kingma and Ba, 2014]
Diederik P Kingma and Jimmy Ba.
Adam: A method for stochastic optimization.
arXiv preprint arXiv:1412.6980, 2014.

[Kitaev et al., 2020]
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya.
Reformer: The efficient transformer.
In Proceedings of International Conference on Learning
Representations, 2020.

[Kudo, 2018]
Taku Kudo.
Subword regularization: Improving neural network translation models
with multiple subword candidates.
In Proceedings of the 56th Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers), pages 66–75, 2018.

[Kwon et al., 2023]
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu,
Joseph E Gonzalez, Hao Zhang, and Ion Stoica.
Efficient memory management for large language model serving with
arXiv preprint arXiv:2309.06180, 2023.

[Lagunas et al., 2021]
François Lagunas, Ella Charlaix, Victor Sanh, and Alexander M Rush.
Block pruning for faster transformers.
In Proceedings of the 2021 Conference on Empirical Methods in
Natural Language Processing, pages 10619–10629, 2021.

[Lample et al., 2019]
Guillaume Lample, Alexandre Sablayrolles, Marc'Aurelio Ranzato, Ludovic
Denoyer, and Hervé Jégou.
Large memory layers with product keys.
Advances in Neural Information Processing Systems, 32, 2019.

[Lepikhin et al., 2021]
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping
Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen.
Gshard: Scaling giant models with conditional computation and
automatic sharding.
In Proceedings of International Conference on Learning
Representations, 2021.

[Leviathan et al., 2023]
Yaniv Leviathan, Matan Kalman, and Yossi Matias.
Fast inference from transformers via speculative decoding.
In Proceedings of International Conference on Machine
Learning, pages 19274–19286. PMLR, 2023.

[Lewis et al., 2020]
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir
Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim
Rocktäschel, et al.
Retrieval-augmented generation for knowledge-intensive nlp tasks.
Advances in Neural Information Processing Systems,
33:0 9459–9474, 2020.

[Li et al., 2020]
Bei Li, Hui Liu, Ziyang Wang, Yufan Jiang, Tong Xiao, Jingbo Zhu, Tongran Liu,
and Changliang Li.
Does multi-encoder help? a case study on context-aware neural machine
In Proceedings of the 58th Annual Meeting of the Association
for Computational Linguistics, pages 3512–3518, 2020a.

[Li et al., 2020]
Bei Li, Ziyang Wang, Hui Liu, Yufan Jiang, Quan Du, Tong Xiao, Huizhen Wang,
and Jingbo Zhu.
Shallow-to-deep training for neural machine translation.
In Proceedings of the 2020 Conference on Empirical Methods in
Natural Language Processing (EMNLP), pages 995–1005, 2020b.

[Li et al., 2021]
Bei Li, Ziyang Wang, Hui Liu, Quan Du, Tong Xiao, Chunliang Zhang, and Jingbo
Learning light-weight translation models from deep transformer.
In Proceedings of the AAAI Conference on Artificial
Intelligence, volume 35, pages 13217–13225, 2021.

[Li et al., 2022]
Bei Li, Quan Du, Tao Zhou, Yi Jing, Shuhan Zhou, Xin Zeng, Tong Xiao, Jingbo
Zhu, Xuebo Liu, and Min Zhang.
Ode transformer: An ordinary differential equation-inspired model for
sequence generation.
In Proceedings of the 60th Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers), pages 8335–8351,

[Li et al., 2022]
Bei Li, Tong Zheng, Yi Jing, Chengbo Jiao, Tong Xiao, and Jingbo Zhu.
Learning multiscale transformer models for sequence generation.
In International Conference on Machine Learning, pages
13225–13241. PMLR, 2022b.

[Li et al., 2022]
Hongkang Li, Meng Wang, Sijia Liu, and Pin-Yu Chen.
A theoretical understanding of shallow vision transformers: Learning,
generalization, and sample complexity.
In The Eleventh International Conference on Learning
Representations, 2022c.

[Li et al., 2017]
Junhui Li, Deyi Xiong, Zhaopeng Tu, Muhua Zhu, Min Zhang, and Guodong Zhou.
Modeling source syntax for neural machine translation.
In Proceedings of the 55th Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers), pages 688–697, 2017.

[Li et al., 2022]
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.
Blip: Bootstrapping language-image pre-training for unified
vision-language understanding and generation.
In International Conference on Machine Learning, pages
12888–12900. PMLR, 2022d.

[Li et al., 2022]
Yanghao Li, Chao-Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra
Malik, and Christoph Feichtenhofer.
Mvitv2: Improved multiscale vision transformers for classification
and detection.
In Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition, pages 4804–4814, 2022e.

[Liao et al., 2021]
Kaiyuan Liao, Yi Zhang, Xuancheng Ren, Qi Su, Xu Sun, and Bin He.
A global past-future early exit method for accelerating inference of
pre-trained language models.
In Proceedings of the 2021 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies, pages 2013–2023, 2021.

[Likhomanenko et al., 2021]
Tatiana Likhomanenko, Qiantong Xu, Gabriel Synnaeve, Ronan Collobert, and Alex
Cape: Encoding relative positions with continuous augmented
positional embeddings.
Advances in Neural Information Processing Systems,
34:0 16079–16092, 2021.

[Lin et al., 2022]
Tianyang Lin, Yuxin Wang, Xiangyang Liu, and Xipeng Qiu.
A survey of transformers.
AI Open, 2022a.

[Lin et al., 2022]
Ye Lin, Shuhan Zhou, Yanyang Li, Anxiang Ma, Tong Xiao, and Jingbo Zhu.
Multi-path transformer is better: A case study on neural machine
In Findings of the Association for Computational Linguistics:
EMNLP 2022, pages 5646–5656, 2022b.

[Liu et al., 2020]
Fenglin Liu, Xuancheng Ren, Zhiyuan Zhang, Xu Sun, and Yuexian Zou.
Rethinking skip connection with layer normalization.
In Proceedings of the 28th international conference on
computational linguistics, pages 3586–3598, 2020a.

[Liu et al., 2023]
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.
Visual instruction tuning.
arXiv preprint arXiv:2304.08485, 2023a.

[Liu et al., 2020]
Liyuan Liu, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, and Jiawei Han.
Understanding the difficulty of training transformers.
In Proceedings of the 2020 Conference on Empirical Methods in
Natural Language Processing (EMNLP), pages 5747–5763, 2020b.

[Liu et al., 2018]
Peter J Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz
Kaiser, and Noam Shazeer.
Generating wikipedia by summarizing long sequences.
In Proceedings of International Conference on Learning
Representations, 2018.

[Liu et al., 2023]
Yang Liu, Yao Zhang, Yixin Wang, Feng Hou, Jin Yuan, Jiang Tian, Yang Zhang,
Zhongchao Shi, Jianping Fan, and Zhiqiang He.
A survey of visual transformers.
IEEE Transactions on Neural Networks and Learning Systems,

[Liu et al., 2020]
Yuchen Liu, Junnan Zhu, Jiajun Zhang, and Chengqing Zong.
Bridging the modality gap for speech-to-text translation.
arXiv preprint arXiv:2010.14920, 2020c.

[Luong et al., 2015]
Minh-Thang Luong, Hieu Pham, and Christopher D Manning.
Effective approaches to attention-based neural machine translation.
In Proceedings of the 2015 Conference on Empirical Methods in
Natural Language Processing, pages 1412–1421, 2015.

[Manning et al., 2020]
Christopher D Manning, Kevin Clark, John Hewitt, Urvashi Khandelwal, and Omer
Emergent linguistic structure in artificial neural networks trained
by self-supervision.
Proceedings of the National Academy of Sciences, 1170
(48):0 30046–30054, 2020.

[Martins et al., 2022]
Pedro Henrique Martins, Zita Marinho, and André FT Martins.
∞-former: Infinite memory transformer-former: Infinite memory
In Proceedings of the 60th Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers), pages 5468–5485,

[Masoudnia and Ebrahimpour, 2014]
Saeed Masoudnia and Reza Ebrahimpour.
Mixture of experts: a literature survey.
The Artificial Intelligence Review, 420 (2):0
275, 2014.

[McCarley et al., 2019]
JS McCarley, Rishav Chakravarti, and Avirup Sil.
Structured pruning of a bert-based question answering model.
arXiv preprint arXiv:1910.06360, 2019.

[Merrill et al., 2022]
William Merrill, Ashish Sabharwal, and Noah A Smith.
Saturated transformers are constant-depth threshold circuits.
Transactions of the Association for Computational Linguistics,
10:0 843–856, 2022.

[Michel et al., 2019]
Paul Michel, Omer Levy, and Graham Neubig.
Are sixteen heads really better than one?
Advances in neural information processing systems, 32, 2019.

[Mikolov et al., 2013]
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean.
Distributed representations of words and phrases and their
In Proceedings of the 26th International Conference on Neural
Information Processing Systems - Volume 2, page 3111–3119, 2013.

[Nagel et al., 2021]
Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysei Bondarenko, Mart
Van Baalen, and Tijmen Blankevoort.
A white paper on neural network quantization.
arXiv preprint arXiv:2106.08295, 2021.

[Oppenheim and Schafer, 1975]
Alan V Oppenheim and Ronald W Schafer.
Digital signal processing(book).
Prentice-Hall, 1975.

[Orvieto et al., 2023]
Antonio Orvieto, Samuel L Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre,
Razvan Pascanu, and Soham De.
Resurrecting recurrent neural networks for long sequences.
arXiv preprint arXiv:2303.06349, 2023.

[Ouyang et al., 2022]
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela
Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.
Training language models to follow instructions with human feedback.
Advances in Neural Information Processing Systems,
35:0 27730–27744, 2022.

[Park et al., 2019]
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho.
Relational knowledge distillation.
In Proceedings of the IEEE/CVF conference on computer vision
and pattern recognition, pages 3967–3976, 2019.

[Parmar et al., 2018]
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer,
Alexander Ku, and Dustin Tran.
Image transformer.
In International conference on machine learning, pages
4055–4064. PMLR, 2018.

[Peng et al., 2019]
Baoyun Peng, Xiao Jin, Jiaheng Liu, Dongsheng Li, Yichao Wu, Yu Liu, Shunfeng
Zhou, and Zhaoning Zhang.
Correlation congruence for knowledge distillation.
In Proceedings of the IEEE/CVF International Conference on
Computer Vision, pages 5007–5016, 2019.

[Peng et al., 2023]
Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Huanqi
Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran GV, et al.
Rwkv: Reinventing rnns for the transformer era.
arXiv preprint arXiv:2305.13048, 2023.

[Peng et al., 2021]
H Peng, N Pappas, D Yogatama, R Schwartz, N Smith, and L Kong.
Random feature attention.
In Proceedings of International Conference on Learning
Representations (ICLR 2021), 2021.

[Pérez et al., 2018]
Jorge Pérez, Javier Marinković, and Pablo Barceló.
On the turing completeness of modern neural network architectures.
In Proceedings of International Conference on Learning
Representations, 2018.

[Petroni et al., 2019]
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton
Bakhtin, Yuxiang Wu, and Alexander Miller.
Language models as knowledge bases?
In Proceedings of the 2019 Conference on Empirical Methods in
Natural Language Processing and the 9th International Joint Conference on
Natural Language Processing (EMNLP-IJCNLP), pages 2463–2473, 2019.

[Pham et al., 2019]
Ngoc-Quan Pham, Thai-Son Nguyen, Jan Niehues, Markus Müller, Sebastian
Stüker, and Alexander Waibel.
Very deep self-attention networks for end-to-end speech recognition.
arXiv preprint arXiv:1904.13377, 2019.

[Pires et al., 2023]
Telmo Pessoa Pires, António V Lopes, Yannick Assogba, and Hendra Setiawan.
One wide feedforward is all you need.
arXiv preprint arXiv:2309.01826, 2023.

[Pope et al., 2023]
Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury,
Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean.
Efficiently scaling transformer inference.
In Proceedings of Machine Learning and Systems, 2023.

[Press et al., 2021]
Ofir Press, Noah Smith, and Mike Lewis.
Train short, test long: Attention with linear biases enables input
length extrapolation.
In Proceedings of International Conference on Learning
Representations, 2021.

[Provilkov et al., 2020]
Ivan Provilkov, Dmitrii Emelianenko, and Elena Voita.
Bpe-dropout: Simple and effective subword regularization.
In Proceedings of the 58th Annual Meeting of the Association
for Computational Linguistics, pages 1882–1892, 2020.

[Qiu et al., 2020]
Jiezhong Qiu, Hao Ma, Omer Levy, Wen-tau Yih, Sinong Wang, and Jie Tang.
Blockwise self-attention for long document understanding.
In Findings of the Association for Computational Linguistics:
EMNLP 2020, pages 2555–2565, 2020.

[Rabiner and Gold, 1975]
Lawrence R Rabiner and Bernard Gold.
Theory and application of digital signal processing.
Prentice-Hall, 1975.

[Rae et al., 2019]
Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, Chloe Hillier, and Timothy P
Compressive transformers for long-range sequence modelling.
In Proceedings of International Conference on Learning
Representations, 2019.

[Raffel et al., 2020]
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael
Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al.
Exploring the limits of transfer learning with a unified text-to-text
Journal of Machine Learning Research, 210
(140):0 1–67, 2020.

[Reimers and Gurevych, 2019]
Nils Reimers and Iryna Gurevych.
Sentence-bert: Sentence embeddings using siamese bert-networks.
In Proceedings of the 2019 Conference on Empirical Methods in
Natural Language Processing and the 9th International Joint Conference on
Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992, 2019.

[Romero et al., 2014]
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo
Gatta, and Yoshua Bengio.
Fitnets: Hints for thin deep nets.
arXiv preprint arXiv:1412.6550, 2014.

[Roy et al., 2021]
Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier.
Efficient content-based sparse attention with routing transformers.
Transactions of the Association for Computational Linguistics,
9:0 53–68, 2021.

[Santacroce et al., 2023]
Michael Santacroce, Zixin Wen, Yelong Shen, and Yuanzhi Li.
What matters in the structured pruning of generative language models?
arXiv preprint arXiv:2302.03773, 2023.

[Schlag et al., 2021]
Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber.
Linear transformers are secretly fast weight programmers.
In Proceedings of International Conference on Machine
Learning, pages 9355–9366. PMLR, 2021.

[Schuster et al., 2022]
Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Tran,
Yi Tay, and Donald Metzler.
Confident adaptive language modeling.
Advances in Neural Information Processing Systems,
35:0 17456–17472, 2022.

[Schwartz et al., 2020]
Roy Schwartz, Gabriel Stanovsky, Swabha Swayamdipta, Jesse Dodge, and Noah A
The right tool for the job: Matching model and instance complexities.
In Proceedings of the 58th Annual Meeting of the Association
for Computational Linguistics, pages 6640–6651, 2020.

[See, 2018]
Abigail See.
Deep learning, structure and innate priors: A discussion between yann
lecun and christopher manning, 02 2018.

[Sennrich et al., 2016]
Rico Sennrich, Barry Haddow, and Alexandra Birch.
Neural machine translation of rare words with subword units.
In Proceedings of the 54th Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725,

[Shaw et al., 2018]
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani.
Self-attention with relative position representations.
In Proceedings of the 2018 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies, Volume 2 (Short Papers), pages 464–468, 2018.

[Shazeer, 2019]
Noam Shazeer.
Fast transformer decoding: One write-head is all you need.
arXiv preprint arXiv:1911.02150, 2019.

[Shazeer et al., 2017]
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le,
Geoffrey Hinton, and Jeff Dean.
Outrageously large neural networks: The sparsely-gated
mixture-of-experts layer.
In Proceedings of International Conference on Learning
Representations, 2017.

[Shen et al., 2020]
Dinghan Shen, Mingzhi Zheng, Yelong Shen, Yanru Qu, and Weizhu Chen.
A simple but tough-to-beat data augmentation approach for natural
language understanding and generation.
arXiv preprint arXiv:2009.13818, 2020.

[Shi et al., 2016]
Xing Shi, Inkit Padhi, and Kevin Knight.
Does string-based neural mt learn source syntax?
In Proceedings of the 2016 conference on empirical methods in
natural language processing, pages 1526–1534, 2016.

[Skorski et al., 2021]
Maciej Skorski, Alessandro Temperoni, and Martin Theobald.
Revisiting weight initialization of deep neural networks.
In Asian Conference on Machine Learning, pages 1192–1207.
PMLR, 2021.

[So et al., 2019]
David So, Quoc Le, and Chen Liang.
The evolved transformer.
In Proceedings of International conference on machine
learning, pages 5877–5886. PMLR, 2019.

[Sperber et al., 2018]
Matthias Sperber, Jan Niehues, Graham Neubig, Sebastian Stüker, and Alex
Self-attentional acoustic models.
In Proceedings of Interspeech 2018, pages 3723–3727, 2018.

[Srivastava et al., 2015]
Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber.
Highway networks.
arXiv preprint arXiv:1505.00387, 2015.

[Stock et al., 2021]
Pierre Stock, Angela Fan, Benjamin Graham, Edouard Grave, Rémi Gribonval,
Herve Jegou, and Armand Joulin.
Training with quantization noise for extreme model compression.
In Proceedings of International Conference on Learning
Representations, 2021.

[Strubell et al., 2018]
Emma Strubell, Patrick Verga, Daniel Andor, David Weiss, and Andrew McCallum.
Linguistically-informed self-attention for semantic role labeling.
In Proceedings of the 2018 Conference on Empirical Methods in
Natural Language Processing, pages 5027–5038, 2018.

[Su et al., 2021]
Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu.
Roformer: Enhanced transformer with rotary position embedding.
arXiv preprint arXiv:2104.09864, 2021.

[Sukhbaatar et al., 2015]
Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al.
End-to-end memory networks.
Advances in neural information processing systems, 28, 2015.

[Sukhbaatar et al., 2019]
Sainbayar Sukhbaatar, Édouard Grave, Piotr Bojanowski, and Armand Joulin.
Adaptive attention span in transformers.
In Proceedings of the 57th Annual Meeting of the Association
for Computational Linguistics, pages 331–335, 2019.

[Sun et al., 2023]
Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong
Wang, and Furu Wei.
Retentive network: A successor to transformer for large language
arXiv preprint arXiv:2307.08621, 2023.

[Szegedy et al., 2014]
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan,
Ian Goodfellow, and Rob Fergus.
Intriguing properties of neural networks.
In Proceedings of 2nd International Conference on Learning
Representations (ICLR 2014), 2014.

[Tan and Le, 2019]
Mingxing Tan and Quoc Le.
Efficientnet: Rethinking model scaling for convolutional neural
In International conference on machine learning, pages
6105–6114. PMLR, 2019.

[Tay et al., 2020]
Yi Tay, Dara Bahri, Liu Yang, Donald Metzler, and Da-Cheng Juan.
Sparse sinkhorn attention.
In Proceedings of International Conference on Machine
Learning, pages 9438–9447. PMLR, 2020a.

[Tay et al., 2020]
Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler.
Efficient transformers: A survey.
CoRR, abs/2009.06732, 2020b.

[Tenney et al., 2019]
Ian Tenney, Dipanjan Das, and Ellie Pavlick.
Bert rediscovers the classical nlp pipeline.
In Proceedings of the 57th Annual Meeting of the Association
for Computational Linguistics, pages 4593–4601, 2019a.

[Tenney et al., 2019]
Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy,
Najoung Kim, Benjamin Van Durme, Samuel R Bowman, Dipanjan Das, et al.
What do you learn from context? probing for sentence structure in
contextualized word representations.
In Proceedings of International Conference on Learning
Representations, 2019b.

[Touvron et al., 2023]
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne
Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric
Hambro, Faisal Azhar, et al.
Llama: Open and efficient foundation language models.
arXiv preprint arXiv:2302.13971, 2023a.

[Touvron et al., 2023]
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine
Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale,
et al.
Llama 2: Open foundation and fine-tuned chat models.
arXiv preprint arXiv:2307.09288, 2023b.

[Vaswani et al., 2017]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,
Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.
Attention is all you need.
In Proceedings of Advances in Neural Information Processing
Systems, volume 30, 2017.

[Vinyals et al., 2015]
Oriol Vinyals, Łukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, and
Geoffrey Hinton.
Grammar as a foreign language.
Advances in neural information processing systems, 28, 2015.

[Voita et al., 2018]
Elena Voita, Pavel Serdyukov, Rico Sennrich, and Ivan Titov.
Context-aware neural machine translation learns anaphora resolution.
In Proceedings of the 56th Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers), pages 1264–1274,

[Voita et al., 2019]
Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov.
Analyzing multi-head self-attention: Specialized heads do the heavy
lifting, the rest can be pruned.
In Proceedings of the 57th Annual Meeting of the Association
for Computational Linguistics, pages 5797–5808, 2019.

[Wallace et al., 2019]
Eric Wallace, Yizhong Wang, Sujian Li, Sameer Singh, and Matt Gardner.
Do nlp models know numbers? probing numeracy in embeddings.
In Proceedings of the 2019 Conference on Empirical Methods in
Natural Language Processing and the 9th International Joint Conference on
Natural Language Processing (EMNLP-IJCNLP), pages 5307–5315, 2019.

[Wang et al., 2020]
Hanrui Wang, Zhanghao Wu, Zhijian Liu, Han Cai, Ligeng Zhu, Chuang Gan, and
Song Han.
Hat: Hardware-aware transformers for efficient natural language
In Proceedings of the 58th Annual Meeting of the Association
for Computational Linguistics, pages 7675–7688, 2020a.

[Wang et al., 2022]
Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, and Furu Wei.
Deepnet: Scaling transformers to 1,000 layers.
arXiv preprint arXiv:2203.00555, 2022a.

[Wang et al., 2022]
Hongyu Wang, Shuming Ma, Shaohan Huang, Li Dong, Wenhui Wang, Zhiliang Peng,
Yu Wu, Payal Bajaj, Saksham Singhal, Alon Benhaim, et al.
Foundation transformers.
arXiv preprint arXiv:2210.06423, 2022b.

[Wang et al., 2022]
Jue Wang, Ke Chen, Gang Chen, Lidan Shou, and Julian McAuley.
Skipbert: Efficient inference with shallow layer skipping.
In Proceedings of the 60th Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers), pages 7287–7301,

[Wang and Yoon, 2021]
Lin Wang and Kuk-Jin Yoon.
Knowledge distillation and student-teacher learning for visual
intelligence: A review and new outlooks.
IEEE transactions on pattern analysis and machine
intelligence, 440 (6):0 3048–3068, 2021.

[Wang et al., 2023]
Peihao Wang, Rameswar Panda, Lucas Torroba Hennigen, Philip Greengard, Leonid
Karlinsky, Rogerio Feris, David Daniel Cox, Zhangyang Wang, and Yoon Kim.
Learning to grow pretrained models for efficient transformer
In Proceedings of The Eleventh International Conference on
Learning Representations, 2023.

[Wang et al., 2018]
Qiang Wang, Fuxue Li, Tong Xiao, Yanyang Li, Yinqiao Li, and Jingbo Zhu.
Multi-layer representation fusion for neural machine translation.
In Proceedings of the 27th International Conference on
Computational Linguistics, pages 3015–3026, 2018a.

[Wang et al., 2019]
Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F Wong, and
Lidia S Chao.
Learning deep transformer models for machine translation.
In Proceedings of the 57th Annual Meeting of the Association
for Computational Linguistics, pages 1810–1822, 2019.

[Wang et al., 2020]
Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma.
Linformer: Self-attention with linear complexity.
arXiv preprint arXiv:2006.04768, 2020b.

[Wang et al., 2018]
Xin Wang, Fisher Yu, Zi-Yi Dou, Trevor Darrell, and Joseph E Gonzalez.
Skipnet: Learning dynamic routing in convolutional networks.
In Proceedings of the European Conference on Computer Vision
(ECCV), pages 409–424, 2018b.

[Wang et al., 2020]
Ziheng Wang, Jeremy Wohlwend, and Tao Lei.
Structured pruning of large language models.
In Proceedings of the 2020 Conference on Empirical Methods in
Natural Language Processing (EMNLP), pages 6151–6162, 2020c.

[Wei et al., 2022]
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian
Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al.
Emergent abilities of large language models.
arXiv preprint arXiv:2206.07682, 2022.

[Weiss et al., 2021]
Gail Weiss, Yoav Goldberg, and Eran Yahav.
Thinking like transformers.
In Proceedings of International Conference on Machine
Learning, pages 11080–11090. PMLR, 2021.

[Wu et al., 2018]
Felix Wu, Angela Fan, Alexei Baevski, Yann Dauphin, and Michael Auli.
Pay less attention with lightweight and dynamic convolutions.
In Proceedings of International Conference on Learning
Representations, 2018a.

[Wu et al., 2019]
Felix Wu, Angela Fan, Alexei Baevski, Yann Dauphin, and Michael Auli.
Pay less attention with lightweight and dynamic convolutions.
In Proceedings of International Conference on Learning
Representations, 2019.

[Wu et al., 2021]
Yuhuai Wu, Markus Norman Rabe, DeLesley Hutchins, and Christian Szegedy.
Memorizing transformers.
In Proceedings of International Conference on Learning
Representations, 2021.

[Wu et al., 2020]
Zhanghao Wu, Zhijian Liu, Ji Lin, Yujun Lin, and Song Han.
Lite transformer with long-short range attention.
In Proceedings of International Conference on Learning
Representations (ICLR), 2020.

[Wu et al., 2018]
Zuxuan Wu, Tushar Nagarajan, Abhishek Kumar, Steven Rennie, Larry S Davis,
Kristen Grauman, and Rogerio Feris.
Blockdrop: Dynamic inference paths in residual networks.
In Proceedings of the IEEE conference on computer vision and
pattern recognition, pages 8817–8826, 2018b.

[Xiao et al., 2019]
Tong Xiao, Yinqiao Li, Jingbo Zhu, Zhengtao Yu, and Tongran Liu.
Sharing attention weights for fast transformer.
In Proceedings of the Twenty-Eighth International Joint
Conference on Artificial Intelligence (IJCAI-19), pages 5292–5298, 2019.

[Xie et al., 2017]
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He.
Aggregated residual transformations for deep neural networks.
In Proceedings of the IEEE conference on computer vision and
pattern recognition, pages 1492–1500, 2017.

[Xin et al., 2020]
Ji Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, and Jimmy Lin.
Deebert: Dynamic early exiting for accelerating bert inference.
In Proceedings of the 58th Annual Meeting of the Association
for Computational Linguistics, pages 2246–2251, 2020.

[Xiong et al., 2020]
Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing,
Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu.
On layer normalization in the transformer architecture.
In International Conference on Machine Learning, pages
10524–10533, 2020.

[Xu and Mcauley, 2023]
Canwen Xu and Julian Mcauley.
A survey on dynamic neural networks for natural language processing.
In Findings of the Association for Computational Linguistics:
EACL 2023, pages 2325–2336, 2023.

[Xu et al., 2021]
Chen Xu, Bojie Hu, Yanyang Li, Yuhao Zhang, Shen Huang, Qi Ju, Tong Xiao, and
Jingbo Zhu.
Stacked acoustic-and-textual encoding: Integrating the pre-trained
models into speech translation encoders.
In Proceedings of the 59th Annual Meeting of the Association
for Computational Linguistics and the 11th International Joint Conference on
Natural Language Processing (Volume 1: Long Papers), pages 2619–2630,

[Xu et al., 2023]
Chen Xu, Rong Ye, Qianqian Dong, Chengqi Zhao, Tom Ko, Mingxuan Wang, Tong
Xiao, and Jingbo Zhu.
Recent advances in direct speech-to-text translation.
In Proceedings of the Thirty-Second International Joint
Conference on Artificial Intelligence (IJCAI-23): Survey Track, pages
6796–6804, 2023a.

[Xu et al., 2023]
Chen Xu, Yuhao Zhang, Chengbo Jiao, Xiaoqian Liu, Chi Hu, Xin Zeng, Tong Xiao,
Anxiang Ma, Huizhen Wang, and Jingbo Zhu.
Bridging the granularity gap for acoustic modeling.
In Findings of the Association for Computational Linguistics:
ACL 2023, pages 10816–10833, 2023b.

[Xu et al., 2020]
Hongfei Xu, Qiuhui Liu, Josef van Genabith, Deyi Xiong, and Jingyi Zhang.
Lipschitz constrained parameter initialization for deep transformers.
In Proceedings of the 58th Annual Meeting of the Association
for Computational Linguistics, pages 397–402, July 2020.

[Xu et al., 2023]
Peng Xu, Xiatian Zhu, and David A Clifton.
Multimodal learning with transformers: A survey.
IEEE Transactions on Pattern Analysis and Machine
Intelligence, 2023c.

[Xu et al., 2021]
Zenan Xu, Daya Guo, Duyu Tang, Qinliang Su, Linjun Shou, Ming Gong, Wanjun
Zhong, Xiaojun Quan, Daxin Jiang, and Nan Duan.
Syntax-enhanced pre-trained model.
In Proceedings of the 59th Annual Meeting of the Association
for Computational Linguistics and the 11th International Joint Conference on
Natural Language Processing (Volume 1: Long Papers), pages 5412–5422,

[Yang et al., 2018]
Baosong Yang, Zhaopeng Tu, Derek F Wong, Fandong Meng, Lidia S Chao, and Tong
Modeling localness for self-attention networks.
In Proceedings of the 2018 Conference on Empirical Methods in
Natural Language Processing, pages 4449–4458, 2018.

[Yang et al., 2023]
Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng
Liu, and Lijuan Wang.
The dawn of lmms: Preliminary explorations with gpt-4v (ision).
arXiv preprint arXiv:2309.17421, 2023a.

[Yang et al., 2023]
Zi Yang, Samridhi Choudhary, Siegfried Kunzmann, and Zheng Zhang.
Quantization-aware and tensor-compressed training of transformers for
natural language understanding.
arXiv preprint arXiv:2306.01076, 2023b.

[Yang et al., 2016]
Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy.
Hierarchical attention networks for document classification.
In Proceedings of the 2016 conference of the North American
chapter of the association for computational linguistics: human language
technologies, pages 1480–1489, 2016.

[Ye et al., 2021]
Rong Ye, Mingxuan Wang, and Lei Li.
End-to-end speech translation via cross-modal progressive training.
arXiv preprint arXiv:2104.10380, 2021.

[Yin et al., 2023]
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen.
A survey on multimodal large language models.
arXiv preprint arXiv:2306.13549, 2023.

[Yu et al., 2023]
Yaodong Yu, Sam Buchanan, Druv Pai, Tianzhe Chu, Ziyang Wu, Shengbang Tong,
Benjamin D Haeffele, and Yi Ma.
White-box transformers via sparse rate reduction.
arXiv preprint arXiv:2306.01129, 2023.

[Yuksel et al., 2012]
Seniha Esen Yuksel, Joseph N Wilson, and Paul D Gader.
Twenty years of mixture of experts.
IEEE transactions on neural networks and learning systems,
230 (8):0 1177–1193, 2012.

[Yun et al., 2019]
Chulhee Yun, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank Reddi, and Sanjiv
Are transformers universal approximators of sequence-to-sequence
In Proceedings of International Conference on Learning
Representations, 2019.

[Zaheer et al., 2020]
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris
Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang,
et al.
Big bird: Transformers for longer sequences.
Advances in neural information processing systems,
33:0 17283–17297, 2020.

[Zhang et al., 2018]
Biao Zhang, Deyi Xiong, and Jinsong Su.
Accelerating neural transformer via an average attention network.
In Proceedings of the 56th Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers), pages 1789–1798,

[Zhang et al., 2019]
Biao Zhang, Ivan Titov, and Rico Sennrich.
Improving deep transformer with depth-scaled initialization and
merged attention.
In Proceedings of the 2019 Conference on Empirical Methods in
Natural Language Processing and the 9th International Joint Conference on
Natural Language Processing (EMNLP-IJCNLP), pages 898–909, 2019.

[Zhang et al., 2020]
Zhuosheng Zhang, Yuwei Wu, Junru Zhou, Sufeng Duan, Hai Zhao, and Rui Wang.
Sg-net: Syntax-guided machine reading comprehension.
In Proceedings of the AAAI Conference on Artificial
Intelligence, pages 9636–9643, 2020.

[Zhou et al., 2021]
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong,
and Wancai Zhang.
Informer: Beyond efficient transformer for long sequence time-series
In Proceedings of the AAAI conference on artificial
intelligence, volume 35, pages 11106–11115, 2021.

[Zhou et al., 2020]
Wangchunshu Zhou, Canwen Xu, Tao Ge, Julian McAuley, Ke Xu, and Furu Wei.
Bert loses patience: Fast and robust inference with early exit.
Advances in Neural Information Processing Systems,
33:0 18330–18341, 2020.

[Zhou, 2012]
Zhi-Hua Zhou.
Ensemble Methods: Foundations and Algorithms.
Chapman and Hall/CRC, 2012.

[Zoph and Le, 2016]
Barret Zoph and Quoc Le.
Neural architecture search with reinforcement learning.
In Proceedings of International Conference on Learning
Representations, 2016.