11institutetext: Carnegie Mellon University, Pittsburgh, USA 22institutetext:
Hong Kong University of Science and Technology, Hong Kong, China
33institutetext: Mohamed bin Zayed University of Artificial Intelligence, Abu
Dhabi, UAE 44institutetext: Reality Labs, Meta Inc. 44email:
zhiqiangshen@cse.ust.hk,zechunliu<EMAIL_ADDRESS>
# Sliced Recursive Transformer

Zhiqiang Shen 112233 Zechun Liu 2244 Eric Xing 1133

###### Abstract

We present a neat yet effective recursive operation on vision transformers
that can improve parameter utilization without involving additional
parameters. This is achieved by sharing weights across depth of transformer
networks. The proposed method can obtain a substantial gain ($\sim$2%) simply
using naïve recursive operation, requires no special or sophisticated
knowledge for designing principles of networks, and introduces minimal
computational overhead to the training procedure. To reduce the additional
computation caused by recursive operation while maintaining the superior
accuracy, we propose an approximating method through multiple sliced group
self-attentions across recursive layers which can reduce the cost consumption
by 10$\sim$30% without sacrificing performance. We call our model Sliced
Recursive Transformer (SReT), a novel and parameter-efficient vision
transformer design that is compatible with a broad range of other designs for
efficient ViT architectures. Our best model establishes significant
improvement on ImageNet-1K over state-of-the-art methods while containing
fewer parameters. The proposed weight sharing mechanism by sliced recursion
structure allows us to build a transformer with more than 100 or even 1000
shared layers with ease while keeping a compact size (13$\sim$15M), to avoid
optimization difficulties when the model is too large. The flexible
scalability has shown great potential for scaling up models and constructing
extremely deep vision transformers. Code is available at
https://github.com/szq0214/SReT.

##  1 Introduction

Figure 1: Params/FLOPs vs. ImageNet-1K Acc.

The architectures of transformer have achieved substantively breakthroughs
recently in the fields of natural language processing (NLP) [54], computer
vision (CV) [16] and speech [15, 56]. In the vision area, Dosovitskiy et al.
[16] introduced a vision transformer (ViT) model that splits a raw image into
a patch sequence as input, and they directly adopt transformer model [54] for
the image classification task. ViT achieved impressive results and has
inspired many follow-up works. However, the benefits of a transformer often
come with a large number of parameters and computational cost and it is always
of great challenge to achieve the optimal trade-off between the accuracy and
model complexity. In this work, we are motivated by the following question:
How can we improve the parameter utilization of a vision transformer, i.e.,
the representation ability without increasing the model size? We observe
recursive operation, as shown in Fig. 2, is a simple yet effective way to
achieve this purpose. Our recursion-based vision transformer models
significantly outperform state-of-the-art approaches while containing fewer
parameters and FLOPs, as illustrated in Fig. 1.

Intrinsically, the classifier requires high-level abstracted features from the
neural network to perform accurate classification, while the extraction of
these features often requires multiple layers and deeper networks. This
introduces parameter overhead into the model. Our motivation of this work
stems from an interesting phenomenon of latent representation visualization.
We observed that in the deep vision transformer network, the weights and
activations of adjacent layers are similar with not much difference (a similar
phenomenon is also discovered in [63]), which means they can be reused. The
transformer with a fixed stack of distinct layers loses the inductive bias in
the recurrent neural network which inspires us to share those weights in a
recursive manner, forming an iterative or recursive vision transformer.
Recursion can help extract stronger features without the need of increasing
the parameters, and further improve the accuracy. In addition, this weight
reuse or sharing strategy partially regularizes the training process by
reducing the number of parameters to avoid overfitting and ill-convergence
challenges, which will be discussed in the later sections.

Figure 2: Atomic Recursive Operation.

Table 1: Results using different numbers $N$ of naïve recursive operation on
ImageNet-1K dataset.

Method | Layers | #Params (M) | Top-1 Acc. (%)
---|---|---|---
DeiT-Tiny [52] | 12 | 5.7 | 72.2
\+ 1$\times$ naïve recursion | 24 | 5.7 | 74.0
\+ 2$\times$ naïve recursion | 36 | 5.7 | 74.1
\+ 3$\times$ naïve recursion | 48 | 5.7 | 73.6

Why do we need to introduce sliced recursion, i.e., the group self-attention,
into transformers? (advantages and drawbacks) We usually push towards
perfection on weight utilization of a network under a bounded range of
parameters, thus, it can be used practically in the resource-limited
circumstances like embedded devices. Recursion is a straightforward way to
compress the feature representation in a cyclic scheme. The recursive neural
networks also allow the branching of connections and structures with
hierarchies. We found that it is intriguingly crucial for learning better
representations on vision data in a hierarchical manner, as we will introduce
in Fig. 10 of our experiments. Also, even the most simplistic recursive
operation still improves the compactness of utilizing parameters without
requiring to modify the transformer block structure, unlike others [50, 61,
24, 55, 57, 37, 31, 59], that add more parameters or involve additional fine-
grained information from input [19]. However, such a recursion will incur more
computational cost by its loops, namely, it sacrifices the executing
efficiency for better parameter representation utilization. To address this
shortcoming, we propose an approximating method for global self-attention
through decomposing into multiple sliced group self-attentions across
recursive layers, meanwhile, enjoying similar FLOPs and better
representations, we also apply the spatial pyramid design to reduce the
complexity of the network.

Feed-forward Networks, Recurrent Neural Networks and Recursive Neural
Networks. Feed-forward networks, such as CNNs and transformers, are directed
acyclic graphs (DAG), so the information path in the feed-forward processing
is unidirectional. Recurrent networks (RNNs) are usually developed to process
the time-series and other sequential data, and predict using current input and
past memory. Recursive network is a less frequently used term compared to
other two counterparts. Recursive refers to repeating or reusing a certain
piece of a network111In a broader sense, the recurrent neural network is a
type of recursive neural network.. Different from RNNs that repeat the same
block throughout the whole network, recursive network selectively repeats
critical blocks for particular purposes. The recursive transformer iteratively
refines its representations for all patches in the sequence. We found that,
through the designed recursion into the feed-forward transformer, we can
dramatically enhance feature representation especially for structured data
without including additional parameters.

The strong experimental results show that integrating the proposed sliced
recursive operation in the transformer strike a competitive trade-off among
accuracy, model size and complexity. To the best of our knowledge, there are
barely existing works studying the effectiveness of recursive operation in
vision transformers and proposing the approximation of self-attention method
for reducing the complexity of recursive operation. We have done extensive
experiments to derive a set of guidelines for the new design on vision task,
and hope it is useful for future research. Moreover, since our method does not
involve the sophisticated knowledge for modification of transformer block or
additional input information, it is orthogonal and friendly to most of
existing ViT designs and approaches.

Our contributions.

\- We investigate the feasibility of leveraging recursive operation with
sliced group self-attention in the vision transformers, which is a promising
direction for establishing efficient transformers and has not been well-
explored before. We conducted in-depth studies on the roles of recursion in
transformers and conclude an effective scheme to use them for better parameter
utilization.

\- We provide design principles, including the concrete format and
comprehensive comparison to variants of SReT architectures, computational
equivalency analysis, modified distillation, etc., in hope of enlightening
future studies in compact transformer design and optimization.

\- We verify our method across a variety of scenarios, including vision
transformer, all-MLP architecture of transformer variant, and neural machine
translation (NMT) using transformers. Our model outperforms the state-of-the-
art methods by a significant margin with fewer parameters.

##  2 Related Work

(i) Transformer [54] was originally designed for natural language processing
tasks and has been the dominant approach [14, 60, 41, 9, 35] in this field.
Recently, Vision Transformer (ViT) [16] demonstrates that such multi-head self
attention blocks can completely replace convolutions and achieve competitive
performance on image classification. While it relied on pre-training on large
amounts of data and transferring to downstream datasets. DeiT [52] explored
the training strategies and various data augmentation on ViT models, to train
them on ImageNet-1K directly. Basically, DeiT can be regarded as a framework
of ViT backbone + massive data augmentation + hyper-parameter tuning + hard
distillation with tokens. After that, many extensions and variants of ViT
models have emerged on image classification task, such as Bottleneck
Transformer [50], Multimodal Transformer [22], Tokens-to-Token Transformer
[61], Spatial Pyramid Transformer [24, 55], Class-Attention Transformer [53],
Transformer in Transformer [19], Convolution Transformer [57], Shifted Windows
Transformer [37], Co-Scale Conv-Attentional Transformer [59], etc. (ii)
Recursive operation has been explored in NLP [34, 12, 6, 5, 7, 28, 11] and
vision [32, 26, 18, 36] areas. In particular, DEQ [5] proposed to find
equilibrium points via root-finding in the weight-tied feedforward models like
transformers and trellis for constant memory. UT [12] presented the
transformer with recurrent inductive bias of RNNs which is similar to our SReT
format. However, these works ignored the complexity increased by recursive
operation in designing networks. In this paper, we focus on utilizing
recursion properly by approximating self-attention through multiple group
self-attentions for building compact and efficient vision transformers.

##  3 Recursive Transformer

Vanilla Transformer Block. A basic transformer block $\mathcal{F}$ consists of
a Multi-head Self Attention (MHSA), Layer Normalization (LN), Feed-forward
Network (FFN), and Residual Connections (RC). It can be formulated as:

$\displaystyle\mathbf{z}_{\ell}^{\prime}=\text{MHSA}\left(\text{LN}\left(\mathbf{z}_{\ell-1}\right)\right)+\mathbf{z}_{\ell-1};\mathbf{z}_{\ell}=\text{FFN}\left(\text{LN}\left(\mathbf{z}_{\ell}^{\prime}\right)\right)+\mathbf{z}_{\ell}^{\prime};i.e.,\mathbf{z}_{\ell}=\mathcal{F}_{\ell-1}(\mathbf{z}_{\ell-1})$
(1)

where $\mathbf{z}_{\ell}^{\prime}$ and $\mathbf{z}_{\ell-1}$ are the
intermediate representations. $\mathcal{F}_{\ell}$ indicates the transformer
block at $\ell$-th layer. $\ell\in\\{0,1,\dots,L\\}$ is the layer index and
$L$ is the number of hidden layers. The self-attention module is realized by
the inner products with a scaling factor and a softmax operation, which is
written as:

$\operatorname{Attention}(Q,K,V)=\operatorname{Softmax}\left(QK^{\top}/{\sqrt{d_{k}}}\right)V$
(2)

where $Q,K,V$ are query, key and value vectors, respectively. $1/\sqrt{d_{k}}$
is the scaling factor for normalization. Multi-head self attention further
concatenates the parallel attention layers to increase the representation
ability:
$\text{MHSA}(Q,K,V)=\operatorname{Concat}\left(\operatorname{head}_{1},\ldots,\operatorname{head}_{\mathrm{h}}\right)W^{O}$,
where $W^{O}\in\mathbb{R}^{hd_{v}\times d_{\text{model }}}$.
$\text{head}_{\mathrm{i}}=\operatorname{Attention}\left(QW_{i}^{Q},KW_{i}^{K},VW_{i}^{V}\right)$
are the projections with parameter matrices
$W_{i}^{Q}\in\mathbb{R}^{d_{\text{model }}\times
d_{k}},W_{i}^{K}\in\mathbb{R}^{d_{\text{model }}\times
d_{k}},W_{i}^{V}\in\mathbb{R}^{d_{\text{model }}\times d_{v}}$. The FFN
contains two linear layers with a GELU non-linearity [23] in between

$\mathrm{FFN}(x)=(\text{GELU}\left(\mathbf{z}W_{1}+b_{1}\right))W_{2}+b_{2}$
(3)

where $\mathbf{z}$ is the input. $W_{1},b_{1},W_{2},b_{2}$ are the two linear
layers’ weights and biases.

Recursive Operation. In the original recursive module [49] for the language
modality, the shared weights are recursively applied on a structured input
which is among the complex inherent chains, so it is capable of learning deep
structured knowledge. Recursive neural networks are made of architectural data
and class, which is majorly proposed for model compositionality on NLP tasks.
Here, we still use the sequence of patch tokens from the images as the inputs
following the ViT model [16]. And, there are no additional inputs used for
feeding into each recursive loop of recursive block as used on structured
data. Take two loops as an example for building the network, the recursive
operation can be simplified:

$\mathbf{z}_{\ell}=\mathcal{F}_{\ell-1}(\mathcal{F}_{\ell-1}(\mathbf{z}_{\ell-1}))$
(4)

The naïve recursive operation tends to learn a simple and trivial solution
like the identity mapping by the optimizer, since the $\mathcal{F}_{\ell-1}$’s
output and input are identical at the adjacent two depths (layers).

Non-linear Projection Layer (NLL). NLL is placed between two recursive
operations to enable the non-linear transformation between each block’s output
and input, to avoid learning trivial status for these recursive blocks by
forcing nonequivalence on neighboring output and input. NLL can be formulated
as:

$\text{NLL}(\mathbf{z}_{\ell-1})=\text{MLP}\left(\text{LN}\left(\mathbf{z}_{\ell-1}^{\prime}\right)\right)+\mathbf{z}_{\ell-1}^{\prime}$
(5)

where MLP is a multi-layer projection as FFN, but has different mlp ratio for
hidden features. We also use residual connection in it for better
representation. As shown in Table 1, more recursions will not improve accuracy
without NLL.

Recursive Transformer. A recursive transformer with two loops in every block
is:

$\mathbf{z}_{\ell}=\text{NLL}_{2}(\mathcal{F}_{\ell-1}(\text{NLL}_{1}(\mathcal{F}_{\ell-1}(\mathbf{z}_{\ell-1}))))$
(6)

where $\mathbf{z}_{\ell-1}$ and $\mathbf{z}_{\ell}$ are each recursive block’s
input and output. Different from MHSA and FFN that share parameters across all
recursive operations within a block, $\text{NLL}_{1}$ and $\text{NLL}_{2}$ use
the non-shared weights independently regardless of positioning within or
outside the recursive blocks.

Recursive All-MLP [51] (an extension). We can formulate it as:

$\displaystyle\mathbf{U}_{*,i}$
$\displaystyle=\mathbf{X}_{*,i}+\mathbf{W}_{2}*\text{GELU}\left(\mathbf{W}_{1}*\text{
LN }(\mathbf{X})_{*,i}\right),$ (7) $\displaystyle\mathbf{Y}_{j,*}$
$\displaystyle=\mathbf{U}_{j,*}+\mathbf{W}_{4}*\text{GELU}\left(\mathbf{W}_{3}*\text{
LN }(\mathbf{U})_{j,*}\right),$ $\displaystyle\mathbf{Y}_{j,*}$
$\displaystyle=\mathcal{M}_{\ell-1}(\mathcal{M}_{\ell-1}(\mathbf{X}_{*,i}))$

where the first and second lines are token-mixing and channel-mixing from
[51]. $\mathcal{M}_{\ell-1}$ is a MLP block, $C$ is the hidden dimension and
$S$ is the number of non-overlapping image patches. NLL is not used here for
simplicity.

Gradients in A Recursive Block. Here, we simply use explicit backpropagation
through the exact operations in the forward pass like gradient descent method
since SReT has no constraint to obtain the equilibrium of input-output in
recursions like DEQ [5] and the number of loops can be small to control the
network computation and depth. Our backward pass is more like UT [12]. In
general, the gradient of the parameters in each recursive block can be:

$\displaystyle\frac{\partial\mathcal{L}}{\partial\mathbf{W}_{\mathcal{F}}}$
$\displaystyle=\frac{\partial\mathcal{L}}{\partial\mathbf{z}^{N}}\frac{\partial\mathbf{z}^{N}}{\partial\mathbf{W}_{\mathcal{F}}}+\frac{\partial\mathcal{L}}{\partial\mathbf{z}^{N}}\frac{\partial\mathbf{z}^{N}}{\partial\mathbf{z}^{N-1}}\frac{\partial\mathbf{z}^{N-1}}{\partial\mathbf{W}_{\mathcal{F}}}+\ldots\frac{\partial\mathcal{L}}{\partial\mathbf{z}^{N}}\frac{\partial\mathbf{z}^{N}}{\partial\mathbf{z}^{N-1}}\ldots\frac{\partial\mathbf{z}^{2}}{\partial\mathbf{z}^{1}}\frac{\partial\mathbf{z}^{1}}{\partial\mathbf{W}_{\mathcal{F}}}$
(8)
$\displaystyle=\sum_{i=1}^{N}\frac{\partial\mathcal{L}}{\partial\mathbf{z}^{N}}\left(\prod_{j=i}^{N-1}\frac{\partial\mathbf{z}^{j+1}}{\partial\mathbf{z}^{j}}\right)\frac{\partial\mathbf{z}^{i}}{\partial\mathbf{W}_{\mathcal{F}}}$

where $\mathbf{W}_{\mathcal{F}}$ is the parameters of recursive block.
$\mathcal{L}$ is the objective function.

Learnable Residual Connection (LRC) for Recursive Vision Transformers. He et
al. [21] studied various strategies of shortcut connections on CNNs and found
that the original residual design with pre-activation performs best. Here, we
found simply adding learnable coefficients on each branch of residual
connection can benefit to the performance of ViT following the similar
discovery of literature [33]. Formally, Eq. 1 and Eq. 5 can be reformulated
as:

$\displaystyle\mathbf{z}_{\ell}^{\prime}$
$\displaystyle=\alpha*\text{MHSA}\left(\text{LN}\left(\mathbf{z}_{\ell-1}\right)\right)+\beta*\mathbf{z}_{\ell-1};$
(9) $\displaystyle\mathbf{z}_{\ell}$
$\displaystyle=\gamma*\text{FFN}\left(\text{LN}\left(\mathbf{z}_{\ell}^{\prime}\right)\right)+\delta*\mathbf{z}_{\ell}^{\prime};$
$\text{NLL}(\mathbf{z}_{\ell-1})=\zeta*\text{MLP}\left(\text{LN}\left(\mathbf{z}_{\ell-1}^{\prime}\right)\right)+\theta*\mathbf{z}_{\ell-1}^{\prime}$
(10)

where $\alpha,\beta,\gamma,\delta,\zeta,\theta$ are the learnable
coefficients. They are initialized as 1 and trained with other model’s
parameters simultaneously without restrictions.

Extremely Deep Transformers. Weight-sharing mechanism allows us to build a
transformer with more than 100 layers still keeping a small model. We
demonstrate empirically that the proposed method can significantly simplify
the optimization when the transformer is scaled up to an exaggerated number of
layers.

Figure 3: Approximating global MHSA via sliced group MHSA with permutation.

##  4 Approximating Global MHSA via Multi-Group MHSA

Though recursive operation is adequate to provide better representation using
the same number of parameters, the additional forward loop makes the overhead
in training and inference increasing unnegligibly. To address the extra
computational cost caused by recursion while maintaining the improved
accuracy, we introduce an approximating method through multiple group self-
attentions which is surprisingly effective in reducing FLOPs without
compromising accuracy.

Approximating Global Self-Attention in SReT. As shown in Fig. 3, a regular
self-attention layer can be decoupled through multiple group self-attentions
in a recursion manner with similar or even smaller computational cost. In
general, the number of groups in different recursion can be the same or
different depending on the requirements of FLOPs and accuracy trade-off. Such
a strategy will not change the number of parameters while more groups can
enjoy lower FLOPs but slightly inferior performance. We empirically verified
that the decoupling scheme can achieve similar performance with significantly
fewer FLOPs if using proper splitting of self-attention in a tolerable scope,
as shown in Appendix.

Computational Equivalency Analysis. In this subsection, we analyze the
complexity of global (i.e., original) and sliced group self-attentions and
compare with different values of groups in a vision transformer.

###### Theorem 4.1

(Equivalency of global and multiple group self-attentions with recursion on
FLOPs.) Let $\\{\textbf{N}_{\ell},\textbf{G}_{\ell}\\}\in\mathbb{R}^{1}$, when
$\textbf{N}_{\ell}\\!=\\!\textbf{G}_{\ell}$, FLOPs(1 V-SA)=
FLOPs($\textbf{N}_{\ell}\times\\!$ Recursion with
$\textbf{G}_{\ell}\times\textbf{G-SAs}$). The complexity C of global and group
self-attentions can be calculated as: (For simplicity, here we assume #groups
and vector dimensions in each recursive operation are the same.)

${\bm{\textbf{C}_{\text{G-SA}}}=\frac{\textbf{N}_{\ell}}{\textbf{G}_{\ell}}\times\bm{\textbf{C}_{\text{V-SA}}}}$
(11)

where $\textbf{N}_{\ell}$ and $\textbf{G}_{\ell}$ are the numbers of recursion
and group MHSA in layer $\ell$, i.e., $\ell$-th recursive block. V-SA and G-SA
represent the vanilla and group MHSA.

The proof is provided in Appendix. The insight provided by Theorem 4.1 is at
the core of our method to control the complexity and its various benefits on
better representations. Importantly, the computation of self-attention through
the “slice” paralleling is equal to the vanilla self-attention. We can observe
that when ${\textbf{N}_{\ell}=\textbf{G}_{\ell}}$,
${\textbf{C}_{\text{V-SA}}\approx\textbf{C}_{\text{G-SA}}}$222In practice, the
FLOPs of the two forms are not identical as self-attention module includes
extra operations like softmax, multiplication with scale and attention values,
which will be multiples by the recursive operation. and if
${\textbf{N}_{\ell}\\!<\\!\textbf{G}_{\ell}}$,
$\bm{\bm{\textbf{missing}}{C}_{\text{G-SA}}\\!<\\!\textbf{C}_{\text{V-SA}}}$,
we can use this property to reduce the FLOPs in designing ViT.

Empirical observation: When FLOPs(recursion + G-SA) $\\!\approx\\!$
FLOPs(V-SA), Acc.(recursion + G-SAs) $>$ Acc.(V-SA).

Table 2: Representation ability with global/group self-attentions.

Method | #Params (M) | FLOPs (B) | Top-1 Acc. (%)
---|---|---|---
Baseline (PiT [24]) | 4.9 | 0.7 | 73.0
SReT (global self-attention w/o loop) | 4.0 | 0.7 | 73.6
SReT (group self-attentions w/ loops) | 4.0 | 0.7 | 74.0

We employ ex-tiny model to evaluate the performance of global self-attention
and sliced group self-attention with recursion. As shown in Table 2, we
empirically verify that, with the similar computation, group self-attention
with recursion can obtain better accuracy than vanilla self-attention.

Analysis: Where is the benefit from in SReT? Theoretical analysis on recursion
could further help understand the advantage behind, while it is difficult and
prior literature on this always proves it empirically. Here, we provide some
basic theoretical explanations from the optimization angle for better
understanding this approach. One is the enhanced gradients accumulation. Let
$g_{t}$=$\nabla_{\theta}f_{t}(\theta)$ denote the gradient, we take Adam
optimizer [27] as an example, naïve parameter update is
$\theta_{t}\\!\leftarrow\\!\theta_{t-1}\\!-\\!\alpha\cdot\widehat{m}_{t}/\left(\sqrt{\widehat{v}_{t}}+\epsilon\right)$
where the gradients w.r.t. stochastic objective at timestep $t$ is
$g_{t}\\!\leftarrow\\!\nabla_{\theta}f_{t}\left(\theta_{t-1}\right)$, here we
omit first and second moment estimate formulae. After involving recursion
(here NLL guarantees $\widehat{m}^{i}_{t}$, ${\widehat{v}^{i}_{t}}$’s
discrepancy), the new updating is:
$\theta_{t}\\!\leftarrow\\!\theta_{t-1}\\!-\\!\sum_{i=1}^{\bm{N}}\alpha\cdot\widehat{m}^{i}_{t}/\left(\sqrt{\widehat{v}^{i}_{t}}+\epsilon\right)$
where $\bm{N}$ is the number of recursion loops. Basically, recursion enables
more updating/tuning of parameters in the same iteration, so that the learned
weights are more aligned to the loss function, and the performance is
naturally better.

##  5 Experiments

In this section, we first empirically verify the proposed SReT on image
classification task with self-attention [54] and all-MLP [51] architectures,
respectively. We also perform detailed ablation studies to explore the optimal
hyper-parameters of our proposed network. Then, we extend it to the neural
machine translation (NMT) task to further verify the generalization ability of
the proposed approach. Finally, we visualize the evolution of learned
coefficients in LRC and intermediate activation maps to better understand the
behaviors and properties of our proposed model. Our experiments are conducted
on CIAI cluster.

###  5.1 Datasets and Experimental Settings

(i) ImageNet-1K [13]: ImageNet-1K is a standard image classification dataset,
which contains 1K classes with a total number of 1.2 million training images
and 50K validation images. Our models are trained on this dataset solely
without additional images; (ii) IWSLT’14 German to English (De-En) dataset
[2]: It contains about 160K sentence pairs as the training set. We train and
evaluate models following the protocol [1]; (iii) WMT’14 English to German
(En-De) dataset [3]: The WMT’14 training data consists of 4.5M sentences pairs
(116M English words, 110M German words). We use the same setup as [38].

Settings: Our detailed training settings and hyper-parameters are shown in
Appendix. On ImageNet-1K, our backbone network is a spatial pyramid [24]
architecture with stem structure following [47].

Soft distillation strategy. On vision transformer, DeiT [52] proposed to
distill tokens together with hard predictions from the teacher. They stated
that using one-hot label with hard distillation can achieve the best accuracy.
This seems counterintuitive since soft labels can provide more subtle
differences and fine-grained information of the input. In this work, through a
proper distillation design, our soft label based distillation framework (one-
hot label is not used) consistently obtained better performance than DeiT333We
observed a minor issue of soft distillation implementation in DeiT
(https://github.com/facebookresearch/deit/blob/main/losses.py#L56). Basically,
it is unnecessary to use logarithm for teacher’s output (logits) according to
the formulation of KL-divergence or cross-entropy. Adding log on both teacher
and student’s logits will make the results of KL to be extremely small and
intrinsically negligible. We argue that soft labels can provide fine-grained
information for distillation, and consistently achieve better results using
soft labels in a proper way than one-hot label + hard distillation, as shown
in Sec. 5.3.. Our loss is a soft version of cross-entropy between teacher and
student’s outputs as used in [46, 44, 45, 4]:
${{\mathcal{L}}_{CE}}({\mathcal{S}_{\mathbf{W}}})=-\frac{1}{N}{\sum_{i=1}^{N}{{\bf
P}_{{\mathcal{T}_{\mathbf{W}}}}({\mathbf{z}})\log}}{\bf
P}_{{\mathcal{S}_{\mathbf{W}}}}({\mathbf{z}}),$ where ${\bf
P}_{\mathcal{T}_{\mathbf{W}}}$ and ${\bf P}_{\mathcal{S}_{\mathbf{W}}}$ are
the outputs of teacher and student, respectively. More details can be referred
to Appendix.

###  5.2 Naïve Recursion on Transformer

Figure 4: Paradigms of recursive designs in transformer.

In this section, we examine the effectiveness of proposed recursion using DeiT
training strategies. We verify the following two fashions of recursion.

Internal and external loops. As illustrated in Fig. 4, there are two possible
recursion designs on transformer networks. One is the internal loop that
repeats every block separately. Another one is the external loop that
cyclically executes all blocks together. Although external loop design can
force the model being more compact as it shares parameters across all blocks
with fewer non-shared NLL layers, we found such a structure is inflexible with
limited representation ability. We conducted a comparison with 12 layers of
basic transformers with 2$\times$ recursive operation and the results are:
external 67.0% (3.2M) vs. internal 67.6% (3.0M) $|$ 70.3% (3.9M). In the
following experiments, we use the internal recursive design as our default
setting.

| #Params (M) | Top-1 (%)
---|---|---
Baseline | 5.7 | 72.2
Recursion ${}_{\text{w/o NLL}}$ | 3.8 | 72.5
Recursion + NLL | 5.0 | 74.7
Recursion + NLL - Class Token | 5.0 | 75.0
Recursion + NLL + LRC | 5.0 | 75.2
Recursion + NLL + Stem | 5.0 | 76.0
Recursion (Full Components) | 5.0 | 76.8
GT+Hard Distill [52] | 5.0 | 77.5
Soft Distill (Ours) | 5.0 | 77.9

Table 3: Effectiveness of various designs on ImageNet-1K val set. Please refer
to Sec. 5.3 and our Appendix for more details. In this ablation study, the
backbone is SReT-TL model using spatial pyramid architecture.

###  5.3 Ablation Studies

The overview of our ablation studies is shown in Table 3. The first row
presents the baseline, the second group is the different structures indicated
by the used factors. The last is the comparison of KD. We also verify the
following designs.

Architecture configuration. As in Table 5, SReT-T is our tiny model which has
mlp ratio = 3.6 in FFN and 4.0 for SReT-TL. More details about these
architectures are provided in our Appendix. To examine the effectiveness of
recursive operation, we conduct different loops of naïve recursion on DeiT-T.
The results of accuracy curves on validation data are shown in Fig. 5 (1), we
can see $2\times$ is slightly better than $1\times$ and the further boost is
marginal, while the $1\times$ is much faster for executing. Thus, we use this
in the following experiments.

NLL configuration. NLL is a crucial factor for size and performance since the
weights in it are not shared. To find an optimal trade-off between model
compactness and accuracy, we explore the NLL ratios in Fig. 5 (2, 3).
Generally, a larger NLL ratio can achieve better performance but the model
size increases accordingly. We use 1.0 in our SReT-T and SReT-TL, and 2.0 in
our SReT-S.

Different permutation designs and groups numbers. We explore the different
permutation designs and the principle of choosing group numbers for better
accuracy-FLOPs trade-off. We propose to insert permutation and inverse
permutation layers to preserve the input’s order information after the sliced
group self-attention operation. The detailed formulation of this module,
together with recursions and their result analyses are given in our Appendix.

Figure 5: A comprehensive ablation study on different design factors.

Distillation. To examine the effectiveness of our proposed soft distillation
method, we conduct the comparison of one-hot label + hard distillation and
soft distillation only. The backbone network is SReT-T, all hyper-parameters
are the same except the loss functions. The accuracy curves are shown in our
Appendix. Our result 77.7% is significantly better than the baseline 77.1%.

Throughput evaluation. In Table 4, we provide the throughput comparisons with
DeiT and Swin on one NVIDIA GeForce RTX 3090 which can directly reflect the
real inference speed and time consumption. We highlight that our method
obtains significantly fewer params and FLOPs with better throughput.

Table 4: Throughput evaluation of SReT and baselines.

DeiT-T | FLOPs: 1.3B | #Params: 5.7M | Acc.: 72.2% | Throughput: 3283.49 img/s
---|---|---|---|---
SReT-ExT | FLOPs: 0.7B↓46.2% | #Params: 4.0M↓29.8% | Acc.: 74.0%↑1.8% | Throughput: 3473.43 img/s
Swin-T | FLOPs: 4.5B | #Params: 29.0M | Acc.: 81.3% | Throughput: 1071.43 img/s
SReT-S | FLOPs: 4.2B↓6.7% | #Params: 20.9M↓27.9% | Acc.: 81.9%↑0.6% | Throughput: 1101.84 img/s

Table 5: Comparison of Top-1 (%) on ImageNet-1K with state-of-the-art methods.
$\divideontimes$ denotes the model is trained without the proposed group self-
attention approximation. Fine-tuning on large resolution is highlighted by
gray color.

Method | Resolution | #Params (M) | FLOPs (B) | Top-1 (%)
---|---|---|---|---
DeiT-T [52] | 224 | 5.7 | 1.3 | 72.2
PiT-T [24] | 224 | 4.9 | 0.7 | 73.0
SReT-ExT (Ours) | 224 | 4.0 | 0.7 | 74.0
DeiT-T [52] | 224 | 5.7 | 1.3 | 72.2
SReT-$\divideontimes$T (Ours) | 224 |  4.8↓15.8% | 1.4 | 76.1
SReT-T (Ours) | 224 | 4.8 |  1.1↓21.4% | 76.0
DeiT-TDistill [52] | 224 | 5.7 | 1.3 | 74.5
SReT-$\divideontimes$TDistill (Ours) | 224 | 4.8 | 1.4 | 77.7
SReT-TDistill (Ours) | 224 | 4.8 |  1.1↓21.4% | 77.6
SReT-$\divideontimes$TDistill&384↑ (Ours) | 384 | 4.9 | 6.4 | 79.7
SReT-TDistill&384↑ (Ours) | 384 | 4.9 |  4.3↓32.8% | 79.6
DeiT-T [52] | 224 | 5.7 | 1.3 | 72.2
AutoFormer-Tiny [10] | 224 | 5.7 | 1.3 | 74.7
CoaT-Lite Tiny [59] | 224 | 5.7 | 1.6 | 76.6
SReT-$\divideontimes$TL (Ours) | 224 |  5.0↓12.3% | 1.4 | 76.8
SReT-TL (Ours) | 224 | 5.0 |  1.2↓14.3% | 76.7
SReT-$\divideontimes$TLDistill (Ours) | 224 | 5.0 | 1.4 | 77.9
SReT-TLDistill (Ours) | 224 | 5.0 | 1.2 | 77.7
SReT-$\divideontimes$TLDistill&384↑ (Ours) | 384 | 5.1 | 6.6 | 80.0
SReT-TLDistill&384↑ (Ours) | 384 | 5.1 |  4.4↓33.3% | 79.8
ViT-B/16 [16] | 384 | 86.0 | 55.4 | 77.9
DeiT-S [52] | 224 | 22.1 | 4.6 | 79.8
PVT-S [55] | 224 | 24.5 | 3.8 | 79.8
PiT-S [24] | 224 | 23.5 | 2.9 | 80.9
T2T-ViTt-14 [61] | 224 | 21.5 | 5.2 | 80.7
TNT-S [19] | 224 | 23.8 | 5.2 | 81.3
Swin-T [37] | 224 | 29.0 | 4.5 | 81.3
SReT-$\divideontimes$S (Ours) | 224 |  20.9↓27.9% | 4.7 | 82.0
SReT-S (Ours) | 224 | 20.9 |  4.2↓10.6% | 81.9
PiT-SDistill [24] | 224 | 23.5 | 2.9 | 81.9
DeiT-SDistill [52] | 224 | 22.1 | 4.6 | 81.2
T2T-ViTt-14Distill [61] | 224 | 21.5 | 5.2 | 81.7
SReT-$\divideontimes$SDistill (Ours) | 224 | 20.9 | 4.7 | 82.8
SReT-SDistill (Ours) | 224 | 20.9 |  4.2↓10.6% | 82.7
SReT-$\divideontimes$SDistill&384↑ (Ours) | 384 | 21.0 | 18.5 | 83.8
SReT-$\divideontimes$SDistill&512↑ (Ours) | 512 | 21.3 | 42.8 | 84.3

Figure 6: (1) ImageNet-1K results on All-MLP. (2) Evolution of coefficients.
Figure 7: Comparison of BLEU, training loss and val loss on WMT14 En-De.

###  5.4 Comparison with State-of-the-art Approaches

A summary of our main results is shown in Table 5, our SReT-ExT is better than
PiT-T by 1.0% with 18.4%$\downarrow$ parameters. SReT-T also outperforms
DeiT-T by 3.8% with 15.8%$\downarrow$ parameters and 15.4%$\downarrow$ FLOPs.
Distillation can help improve the accuracy by 1.6% and fine-tuning on large
resolution further boosts to 79.6%. Moreover, our SReT-S is consistently
better than state-of-the-art Swin-T, T2T, etc., on accuracy, model size and
FLOPs, which demonstrates the superiority and potential of our architectures
in practice.

###  5.5 All-MLP Architecture

MLP-Mixer [51] (Baseline), MLP-Mixer+Recursion and MLP-Mixer+Recursion +LRC:
Mixer is a recently proposed plain design that is based entirely on multi-
layer perceptrons (MLPs). We apply our recursive operation and LRC on MLP-
Mixer to verify their generalization. Results are shown in Fig. 6 (1), our
method is consistently better than the baseline using the same training
protocol.

Figure 8: Illustration of recursive transformer with different designs. Figure
9: The actual optimization landscape from DeiT-108, our SReT-108 and SReT-108
mixed-depth models.

###  5.6 Neural Machine Translation

In this section, we compare the BLEU scores [39] of vanilla transformer [54]
and ours on the WMT14 En-De and IWSLT’14 De-En (Appendix) using fairseq
toolkit [17]. IWSLT’14 De-En is a relatively small dataset so the improvement
is not as significant as on WMT14 En-De. The results are shown in Fig. 7, we
can see our method is favorably better than the baseline. Without LRC, the
model slightly converges faster, but the final accuracy is inferior to using
LRC. Also, LRC makes the training process more stable, as shown in the red
dashed box.

###  5.7 Landscape Visualizations of DeiT and Our Mixed-depth SReT

Explicit mixed-depth training. The recursive neural network enables to train
the model in a mixed-depth scheme. As shown in Fig. 8 (d), the left branch is
the subnetwork containing recursive blocks, while the right is the blocks
without sharing the weights on depth, but their weights are re-used with the
left branch. In this structure, the two branches take inputs from the same
stem block. Mixed-depth training offers simplified optimization by performing
operations parallelly and prevents under-optimizing when the network is
extremely deep.

Benefits of mixed-depth training. The spin-off benefit of sliced recursion is
the feasibility of mixed-depth training, which essentially is an explicit deep
supervision scheme as the shallow branch receives stronger supervision that is
closer to the final loss layer, meanwhile, weights are shared with the deep
branch.

Inspired by [30], we visualize the landscape of baseline DeiT-108 and our
SReT-108 & SReT-108 mixed-depth models to examine and analyze the difficulty
of optimization on these three architectures. The results are illustrated in
Fig. 9, we can observe that DeiT-108 is more chaotic and harder for
optimization with a deeper local minimum than our mixed-depth network. This
verifies the advantage of our proposed network structure for simpler
optimization.

###  5.8 Analysis and Understanding

Here, we provide two visualizations regarding LRC and learned response maps.
Evolution of LRC coefficients. As shown in Fig. 6 (2), we plot the evolution
of learned coefficients in the first block. We can observe that the
coefficients on the identity mapping ($\alpha,\gamma,\zeta$) first go up and
then down as the training continues. This phenomenon indicates that, at the
beginning of model training, the identify mapping plays a major role in the
representations. After $\sim$50 epochs of training, the main branch is
becoming increasingly important. Once the training is complete, in FFN and
NLL, the main branch exceeds the residual connection branch while on MHSA it
is the opposite. We believe this phenomenon can inspire us to design a more
reasonable residual connection structure in ViT.

Learned response maps. We visualize the activation maps of DeiT-T and our
SReT-T model at shallow and deep layers. As shown in Fig. 10, DeiT is a
network with uniform resolution of feature maps (14$\times$14). While, our
spatial pyramid structure has different sizes of feature maps along with the
depth of the network, i.e., the resolution of feature maps decreases when the
depth increases. More interesting observations are discussed in Appendix.

Figure 10: Illustration of activation distributions on shallow, middle and
deep layers of DeiT-Tiny and our SReT-T networks. Under each subfigure,
$14\times 14$, $28\times 28$ and $7\times 7$ are the resolutions of feature
maps. “R1/2” indicates the index of recursive operations in each block.

##  6 Conclusion

It is worthwhile considering how to improve the efficiency of parameter
utilization for a vision transformer with minimum overhead. In this work, we
have summarized and explained several behaviors observed while training such
networks. We focused on building an efficient vision transformer with a
compact model size through the recursive operation, and the proposed group
self-attention approximation method allows us to train in a more efficient
manner with recursive transformers. We highlight such a training scheme has
not been well-explored yet in previous literature. We attributed the superior
performance of sliced recursive transformer to its ability of intensifying the
representation quality of intermediate features. We conducted comprehensive
experiments to establish the success of our method on the image classification
and neural machine translation tasks, not just verifying it in the vision
domain, but proving the capability to generalize for multiple modalities and
architectures, such as MLP-Mixer.

## Appendix

In this appendix, we provide details omitted in the main text, including:

• Section 0.A: Proof for equivalency of global self-attention and sliced group
self-attention with recursive operation on FLOPs. (Sec. 4 “Approximating
Global Self-Attention via Permutation of Group/Local Self-Attentions” of the
main paper.)

• Section 0.B: Results of SReT on ImageNet ReaL [8] and ImageNetV2 [43]
datasets. (Sec. 5 “Experiments and Analysis” of the main paper.)

• Section 0.C: More ablation results on different permutation designs and
numbers of groups when approximating global self-attention on ImageNet-1K.
(Sec. 5.3 “Ablation Studies” of the main paper.)

• Section 0.D: Pseudocode for implementing sliced group self-attention. (Sec.
4 “Approximating Global Self-Attention via Permutation of Group/Local Self-
Attentions” of the main paper.)

• Section 0.E: Implementation details of training on ImageNet-1K. (Sec. 5.1
“Datasets and Experimental Settings” of the main paper.)

• Section 0.F: Hyper-parameters setting for training language models on WMT14
En-De and IWSLT14 De-En datasets. (Sec. 5.1 “Datasets and Experimental
Settings” and Sec. 5.6 “Neural Machine Translation” of the main paper.)

• Section 0.G: Details of our SReT-T, SReT-TL, SReT-S and SReT-B
architectures. (Sec. 3 “Recursive Transformer” and Sec. 5.3. “Ablation
Studies” of the main paper.)

• Section 0.H: Details of All-MLP structure. (Sec. 5.5 “All-MLP Architecture”
of the main paper.)

• Section 0.I: Ablation study on different LRC designs. (Sec. 3 “Recursive
Transformer” and Sec. 5.8 “Analysis and Understanding” of the main paper.)

• Section 0.J: Observations of Response Maps. (Sec. 5.8 “Analysis and
Understanding” of the main paper.)

• Section 0.K: More evolution visualization of LRC coefficients on ImageNet-1K
dataset. (Sec. 5.8 “Analysis and Understanding” of the main paper.)

• Section 0.L: Evolution visualization of LRC coefficients in language model
on WMT14 En-De dataset. (Sec. 5.6 “Neural Machine Translation” and Sec. 5.8
“Analysis and Understanding” of the main paper.)

• Section 0.M: More ablation results on directly expanding the depth of
baseline DeiT model on ImageNet-1K dataset. (Sec. 5.8 “Analysis and
Understanding” of the main paper.)

• Section 0.N: More definitions of “Feed-forward Networks, Recurrent Neural
Networks and Recursive Neural Networks” and explanations of difference to
prior arts. (Sec. 1 “Introduction” and Sec. 2 “Related Work” of the main
paper.)

##  Appendix 0.A FLOPs Analysis

One of the key benefits of our SReT is to control the complexity of a
recursive network. We analyze the FLOPs of global (i.e., original) and sliced
group self-attentions and compare them with different circumstances of groups
in a vision transformer. In this section, we provide a proof to Theorem 1
which we restate below.

Theorem 1. (Equivalency of global self-attention and group self-attention with
recursive operation on FLOPs.) Let
$\\{\textbf{N}_{\ell},\textbf{G}_{\ell}\\}\in\mathbb{R}^{1}$, when
$\textbf{N}_{\ell}=\textbf{G}_{\ell}$, FLOPs(1 V-SA) =
FLOPs($\textbf{N}_{\ell}\times$ Recursive with $\textbf{G}_{\ell}\times$
G-SAs). The complexity of regular and group self-attentions can be calculated
as: (For simplicity, here we assume #groups and vector dimensions in each
recursive operation are the same.)

${\bm{\textbf{C}_{\text{G-SA}}}=\frac{\textbf{N}_{\ell}}{\textbf{G}_{\ell}}\times\bm{\textbf{C}_{\text{V-SA}}}}$
(12)

where $\textbf{N}_{\ell}$ is the number of recursive operation and
$\textbf{G}_{\ell}$ is the number of group self-attentions in layer $\ell$,
i.e., $\ell$-th recursive block. V-SA and G-SA represent the vanilla and group
self-attentions, respectively.

###### Proof

(Theorem 1) The complexity C of regular self-attention can be calculated as:

${\bm{\textbf{C}_{\text{V-SA}}}=\mathcal{O}(\textbf{L}_{\ell}^{2}\times\textbf{D}_{\ell})}$
(13)

where $\textbf{L}_{\ell}$ is the sequence length and $\textbf{D}_{\ell}$ is
the dimensionality of the latent representations.

The complexity of simple recursive operation without group will be:

${\bm{\textbf{C}_{\text{recursive}}}=\mathcal{O}(\textbf{N}_{\ell}\times\textbf{L}_{\ell}^{2}\times\textbf{D}_{\ell})}$
(14)

where $\textbf{N}_{\ell}$ is the number of recursive operation.

The complexity of sliced group self-attentions with a recursive block can be
calculated as:

$\displaystyle\bm{\textbf{C}_{\text{G-SA}}}=\mathcal{O}(\sum_{i}^{\textbf{N}_{\ell}}(\textbf{g}_{\ell}^{i}\times(\frac{\textbf{L}_{\ell}}{\textbf{g}_{\ell}^{i}})^{2}\times\textbf{d}_{\ell}^{i}))$
(15)
$\displaystyle=\mathcal{O}(\sum_{i}^{\textbf{N}_{\ell}}(\frac{\textbf{L}_{\ell}^{2}}{\textbf{g}_{\ell}^{i}}\times\textbf{d}_{\ell}^{i}))$

where $\textbf{g}_{\ell}^{i}\in\\{\textbf{G}_{\ell}\\}$,
$\textbf{d}_{\ell}^{i}\in\\{\textbf{D}_{\ell}\\}$, $i=1,\dots,\textbf{\em
N}_{\ell}$.

Consider the condition of #groups $\textbf{\em g}_{\ell}^{i}$ and vector
dimension $\textbf{\em d}_{\ell}^{i}$ in each recursive operation are the
same. The complexity of group self-attentions can be re-formulated as:

${\bm{\textbf{C}_{\text{G-SA}}}=\mathcal{O}(\textbf{N}_{\ell}\times\frac{\textbf{L}_{\ell}^{2}}{\textbf{G}_{\ell}}\times\textbf{D}_{\ell})=\frac{\textbf{N}_{\ell}}{\textbf{G}_{\ell}}\times\bm{\textbf{C}_{\text{V-SA}}}}$
(16)

where $\textbf{\em G}_{\ell}$ is the number of group self-attentions. When
${\textbf{N}_{\ell}=\textbf{G}_{\ell}}$,
${\textbf{C}_{\text{V-SA}}=\textbf{C}_{\text{G-SA}}}$ and if
${\textbf{N}_{\ell}\\!<\\!\textbf{G}_{\ell}}$,
${\textbf{C}_{\text{G-SA}}\\!<\\!\textbf{C}_{\text{V-SA}}}$.

##  Appendix 0.B More Results and Comparisons on ImageNet ReaL [8] and
ImageNetV2 [43] Datasets

In this section, we provide results on ImageNet ReaL [8] and ImageNetV2 [43]
datasets. On ImageNetV2 [43], we verify our SReT models on three metrics “Top-
Images”, “Matched Frequency”, and “Threshold 0.7”. The results are shown in
Table 6, we achieve consistent improvement over DeiT on various network
architectures.

Table 6: More Comparison of SReT on ReaL [8] and ImageNetV2 [43] datasets.

Method | Network | #Parames | FLOPs | ImageNet | ReaL | ImageNetV2 | ImageNetV2 | ImageNetV2
---|---|---|---|---|---|---|---|---
Top-images | Matched-frequency | Threshold-0.7
DeiT [52] | Tiny | 5.7 | 1.3 | 72.2 | 80.1 | 74.4 | 59.9 | 68.5
SReT | Tiny | 4.8 | 1.1 | 76.0 | 83.1 | 77.9 | 64.0 | 72.8
DeiT [52] | Tiny+Distill | 5.7 | 1.3 | 74.5 | 82.1 | 77.0 | 62.3 | 71.1
SReT | Tiny+Distill | 4.8 | 1.1 | 77.6 | 84.4 | 79.6 | 65.7 | 74.2
DeiT [52] | Small | 22.1 | 4.6 | 79.8 | 85.7 | 81.0 | 68.1 | 76.4
SReT | Small | 20.9 | 4.2 | 81.9 | 86.7 | 82.8 | 70.3 | 78.1
DeiT [52] | Small+Distill | 22.1 | 4.6 | 81.2 | 86.8 | 82.5 | 69.7 | 77.5
SReT | Small+Distill | 20.9 | 4.2 | 82.7 | 88.1 | 84.0 | 72.3 | 79.9

##  Appendix 0.C Ablation Results on Different Permutation Designs and Groups
Numbers

In this section, we explore the different permutation designs and the
principle of choosing group numbers for the best accuracy-FLOPs trade-off. We
propose to insert an inverse permutation layer to preserve the input order
information after the sliced group self-attention operation. The formulation
of this operation is shown in Fig. 11 and the ablation results for this design
are given in Table 7 of the first group. In the table, “P” represents the
permutation layer, “I” represents the inverse permutation layer and “L”
indicates that we did not involve permutation and inverse permutation in the
last stage of models when the number of groups equals 1. We use SReT-T and
SReT-TL as the base structures for the ablation of different groups. In the
Groups column of the table, we applied two loops of recursion in each
recursive block according to the ablation study in Table 1 of our main text.
In each pair of the square brackets, the values denote the number of groups
for each recursion, and each pair of square brackets represents one stage of
blocks in the spatial pyramid based backbone network. We use [8,2][4,1][1,1]
as our final SReT structure design since it has the best trade-off on accuracy
and computational cost.

Figure 11: Details of group self-attention with permutation designs. Table 7:
Ablation results of SReT-T and SReT-TL with different group designs.

Groups | Net | Layers | Params (M) | #FLOPs (B) | Top-1 (%)
---|---|---|---|---|---
[8,8][4,4][1,1] | P | 20 | 4.99 | 1.08 | 75.41
[8,8][4,4][1,1] | P+I | 20 | 4.99 | 1.08 | 75.94
[8,8][4,4][1,1] | P+I-L | 20 | 4.99 | 1.08 | 76.06
[1,1][1,1][1,1] | SReT-$\divideontimes$T | 20 | 4.76 | 1.38 | 76.07
[8,8][4,4][1,1] | SReT-T | 20 | 4.76 | 1.03 | 75.73
[16,2][4,2][1,1] | SReT-T | 20 | 4.76 | 1.01 | 75.79
[8,2][4,1][1,1] | SReT-T | 20 | 4.76 | 1.12 | 75.97
[1,1][1,1][1,1] | SReT-$\divideontimes$TL | 20 | 4.99 | 1.43 | 76.78
[8,8][4,4][1,1] | SReT-TL | 20 | 4.99 | 1.08 | 76.06
[8,4][4,2][1,1] | SReT-TL | 20 | 4.99 | 1.14 | 76.16
[8,2][4,1][1,1] | SReT-TL | 20 | 4.99 | 1.18 | 76.65
[8,1][4,1][1,1] | SReT-TL | 20 | 4.99 | 1.25 | 76.72
[16,1][14,1][1,1] | SReT-TL | 20 | 4.99 | 1.24 | 76.56
[49,1][28,1][1,1] | SReT-TL | 20 | 4.99 | 1.23 | 76.30

##  Appendix 0.D Pseudocode for Sliced Group Self-attention

The PyTorch pseudocode for implementation of our sliced group self-attention
is shown in Algorithm 1.

Algorithm 1 PyTorch-like Code for Sliced Group MHSA with 2$\times$ Recursion.

⬇

# num_groups1 and num_groups2: numbers of groups in different recursions

# recursion: recursive indicator

class SG_Attention(nn.Module):

def __init__(self, dim, num_groups1=8, num_groups2=4, num_heads=8,
qkv_bias=False, qk_scale=None, attn_drop=0., proj_drop=0.):

super().__init__()

self.num_heads = num_heads

# numbers of groups in different recursions

self.num_groups1 = num_groups1

self.num_groups2 = num_groups2

head_dim = dim // num_heads

self.scale = qk_scale or head_dim ** -0.5

self.qkv = nn.Linear(dim, dim * 3, bias=qkv_bias)

self.attn_drop = nn.Dropout(attn_drop)

self.proj = nn.Linear(dim, dim)

self.proj_drop = nn.Dropout(proj_drop)

def forward(self, x, recursion):

B, N, C = x.shape

if recursion == False:

num_groups = self.num_groups1

else:

num_groups = self.num_groups2

# we will not do permutation and inverse permutation if #group=1

if num_groups != 1:

idx = torch.randperm(N)

# perform permutation

x = x[:,idx,:]

# prepare for inverse permutation

inverse = torch.argsort(idx)

qkv = self.qkv(x).reshape(B, num_groups, N // num_groups, 3, self.num_heads, C
// self.num_heads).permute(3, 0, 1, 4, 2, 5)

q, k, v = qkv[0], qkv[1], qkv[2] # make torchscript happy (cannot use tensor
as tuple)

attn = (q k.transpose(-2, -1)) * self.scale

attn = attn.softmax(dim=-1)

attn = self.attn_drop(attn)

x = (attn v).transpose(2, 3).reshape(B, num_groups, N // num_groups, C)

x = x.permute(0, 3, 1, 2).reshape(B, C, N).transpose(1, 2)

if recursion == True and num_groups != 1:

# perform inverse permutation

x = x[:,inverse,:]

x = self.proj(x)

x = self.proj_drop(x)

return x

...

##  Appendix 0.E Training Details on ImageNet-1K

On ImageNet-1K, we conduct experiments on three training schemes: (1)
conventional training with one-hot labels; (2) distillation with soft labels
from a pre-trained teacher; (3) finetuning from distilled parameters with
higher resolution. Our training settings and hyper-parameters mainly follow
the designs of DeiT [52]. A detailed introduction of these settings is shown
in Table 10, 10 and 10 with an item-by-item comparison.

Table 8: Hyper-parameter details of conventional training.

Method | SReT-T | SReT-TL | SReT-S
---|---|---|---
Epoch | 300 | 300 | 300
Batch size | 1024 | 1024 | 512
Optimizer | AdamW | AdamW | AdamW
Learning rate | 0.001 | 0.001 | 0.001
Weight decay | 0.05 | 0.05 | 0.05
Warmup epochs | 5 | 5 | 5
Label smoothing | 0.1 | 0.1 | 0.1
Stoch. Depth | 0.1 | 0.1 | 0.2

Table 9: Hyper-parameter details of soft distillation training.

Method | DeiT | SReT
---|---|---
Label | one-hot+hard distillation | soft distillation
Epoch | 300 | 300
Batch size | 1024 | 1024
Optimizer | AdamW | AdamW
Learning rate | 0.001 | 0.001

Table 10: Hyper-parameter details of higher-resolution finetuning.

Method | DeiT | SReT
---|---|---
Resolution | 384 | 384
Weight decay | 1e-8 | 0.0
Learning rate | 5e-6 | 5e-6

Conventional Training from Scratch with One-hot Label. As shown in Table 10,
we use batch-sizes of 512/1024 for training our models and the default initial
learning rate is $1e$-3, while from our experiments, larger initial lr of
$2e$-3 with more warmup epochs of 30 can favorably improve the accuracy. Other
settings are following [52].

Figure 12: Comprehensive ablation study on different design factors.

Distillation Strategy. Knowledge distillation [25] is a popular way to boost
the performance of a student network. Recently, many promising results [40,
48, 58] have been achieved using this technique. On vision transformer, DeiT
[52] proposed to distill tokens together with hard predictions from the
teacher, and it claimed that using one-hot label with hard distillation can
achieve the best accuracy. This seems counterintuitive since soft labels can
provide more subtle differences and fine-grained information of the input. In
this work, through a proper distillation scheme, our soft label based
distillation framework (one-hot label is not used) consistently obtained
better performance than DeiT. Our loss is a soft version of cross-entropy
between teacher and student’s outputs as used in [44, 4, 46]:

${{\mathcal{L}}_{CE}}({\mathcal{S}_{\mathbf{W}}})=-\frac{1}{N}{\sum\limits_{i=1}^{N}{{\bf
P}_{{\mathcal{T}_{\mathbf{W}}}}({\mathbf{z}})\log}}{\bf
P}_{{\mathcal{S}_{\mathbf{W}}}}({\mathbf{z}})$ (17)

where ${\bf P}_{\mathcal{T}_{\mathbf{W}}}$ and ${\bf
P}_{\mathcal{S}_{\mathbf{W}}}$ are the outputs of teacher and student,
respectively.

Distillation from Scratch. As shown in Table 10, we use soft predictions
solely from RegNetY-16GF [42] as a teacher instead of one-hot label + hard
distillation used in [52]. The ablation study on this point is provided in
Fig. 12 (1) with SReT-T.

Spatial Pyramid (SP) Design. Pyramids [29, 20] are an effective design in
conventional vision tasks. The resolution of the shallow stage in a network is
usually large, SP can help to redistribute the computation from shallow to
deep stages of a network according to their representation ability. Here, we
follow the construction principles [24] but replacing the first patch
embedding layer with a Stem block (i.e., a stack of three 3$\times$3
convolution layers with stride = 2) following [47].

Figure 13: Our modifications by removing class token and distillation token.

Other Small Modifications. Considering the unique properties of vision
modality compared to the language, we further apply some minor modifications
on our network design, some of them have been proven useful on CNNs in the
vision domain, including: (i) We remove the class token and replace with a
global average pooling (GAP) on the last output together with a fully-
connected layer; (ii) We also remove the distillation token if the training
process involves KD, which means we use the same feature embedding for both
the ground-truth labels in standard training, and distillation with soft
labels from the teacher. (iii) When fine-tuning from low resolution
(224$\times$224) to high resolution (384$\times$384) [52], following the
perspective of [48] that to increase the capacity of a model, we do not apply
weight decay (set it as 0) during fine-tuning. Generally, the above
modifications can slightly save parameters, boost the performance and
significantly improve the simplicity of the whole framework. The illustration
of these modifications is shown in Fig. 13.

Figure 14: Comparison of BLEU, training loss and val loss on WMT14 En-De (top)
and IWSLT14 De-En datasets (bottom). The red dashed box indicates that LRC
makes training more stable.

##  Appendix 0.F Hyper-parameter Settings of Language Models

We test our proposed method on two public language datasets: IWSLT14 De-En and
WMT14 En-De translation tasks. We describe experimental settings in detail in
Table 11.

Network Configurations. We use the Transformer [54] implemented in Fairseq
[17] that shares the decoder input and output embedding as the basic NMT
model.

Table 11: Training details of our language models. The architectures we used
are in Fairseq [17].

Method | IWSLT14 De-En | WMT14 En-De
---|---|---
arch | transformer_iwslt_de_en | transformer_wmt_en_de
share decoder input output embed | True | True
optimizer | Adam | Adam
adam-betas | (0.9, 0.98) | (0.9, 0.98)
clip-norm | 0.0 | 0.0
learning rate | 5e-4 | 5e-4
lr scheduler | inverse sqrt | inverse sqrt
warmup updates | 4K | 4K
dropout | 0.3 | 0.3
weight decay | 0.0001 | 0.0001
criterion | label smoothed cross-entropy | label smoothed cross-entropy
label smoothing | 0.1 | 0.1
max tokens | 4096 | 4096

Figure 15: Ablation study on different LRC designs.

##  Appendix 0.G Details of Our SReT Architectures

The details of our SReT-T, SReT-TL, SReT-S and SReT-B architectures are shown
in Table 12. In each recursive transformer block $[[.]\times A]\times B$, $A$
is the number of blocks with self-contained (non-shared) parameters, $B$ is
the number of recursive operations for each block. For $C\times\text{FFN}$ and
$D\times\text{NLL}$, $C$ and $D$ are the dimensions (ratios) of hidden
features between the two fully-connected layers.

Table 12: SReT architectures (Input size is 3$\times 224\times 224$, sliced
group self-attention is not included for simplicity.)

Layers | Output Size | SReT-T | SReT-TL
---|---|---|---
Stem | Conv-BN-ReLU | 32$\times$112$\times$112 | 3$\times$3 conv, stride 2 | 3$\times$3 conv, stride 2
Conv-BN-ReLU | 64$\times$56$\times$56 | 3$\times$3 conv, stride 2 | 3$\times$3 conv, stride 2
Conv-BN-ReLU | 64$\times$28$\times$28 | 3$\times$3 conv, stride 2 | 3$\times$3 conv, stride 2
| Recursive T Block
---
(1)
64$\times$28$\times$28 | $\left[\left[\begin{array}[]{ll}\text{ \ \ \ 64-dim}\text{ MHSA}\\\ \text{3.6$\times$}\text{FFN}/\text{1.0$\times$}\text{NLL}\end{array}\right]\times\text{2}\right]\times\text{2}$ | $\left[\left[\begin{array}[]{ll}\text{ \ \ \ 64-dim}\text{ MHSA}\\\ \text{4.0$\times$}\text{FFN}/\text{1.0$\times$}\text{NLL}\end{array}\right]\times\text{2}\right]\times\text{2}$
Conv-Pooling Layer (1) | 128$\times$14$\times$14 | 3$\times$3 conv, stride 2, group 64 | 3$\times$3 conv, stride 2, group 64
| Recursive T Block
---
(2)
128$\times$14$\times$14 | $\left[\left[\begin{array}[]{ll}\text{ \ \ \ 128-dim}\text{ MHSA}\\\ \text{3.6$\times$}\text{FFN}/\text{1.0$\times$}\text{NLL}\end{array}\right]\times\text{5}\right]\times\text{2}$ | $\left[\left[\begin{array}[]{ll}\text{ \ \ \ 128-dim}\text{ MHSA}\\\ \text{4.0$\times$}\text{FFN}/\text{1.0$\times$}\text{NLL}\end{array}\right]\times\text{5}\right]\times\text{2}$
Conv-Pooling Layer (2) | 256$\times$7$\times$7 | 3$\times$3 conv, stride 2, group 128 | 3$\times$3 conv, stride 2, group 128
| Recursive T Block
---
(3)
256$\times$7$\times$7 | $\left[\left[\begin{array}[]{ll}\text{ \ \ \ 256-dim}\text{ MHSA}\\\ \text{3.6$\times$}\text{FFN}/\text{1.0$\times$}\text{NLL}\end{array}\right]\times\text{3}\right]\times\text{2}$ | $\left[\left[\begin{array}[]{ll}\text{ \ \ \ 256-dim}\text{ MHSA}\\\ \text{4.0$\times$}\text{FFN}/\text{1.0$\times$}\text{NLL}\end{array}\right]\times\text{3}\right]\times\text{2}$
Global Average Pooling | 256$\times$1$\times$1 | AdaptiveAvgPool | AdaptiveAvgPool
Linear Layer | 1000
#Params (M) |  | 4.8 M | 5.0 M
Accuracy (%) |  | 76.1 | 76.8
Distilled Accuracy (%) |  | 77.7 | 77.9
Finetuning Accuracy $\uparrow$384 (%) |  | 79.7 | 80.0

Layers | Output Size | SReT-S | Output Size | SReT-B
---|---|---|---|---
Stem | Conv-BN-ReLU | 63$\times$112$\times$112 | 3$\times$3 conv, stride 2 | 96$\times$112$\times$112 | 3$\times$3 conv, stride 2
Conv-BN-ReLU | 126$\times$56$\times$56 | 3$\times$3 conv, stride 2 | 168$\times$56$\times$56 | 3$\times$3 conv, stride 2
Conv-BN-ReLU | 126$\times$28$\times$28 | 3$\times$3 conv, stride 2 | 336$\times$28$\times$28 | 3$\times$3 conv, stride 2
| Recursive T Block
---
(1)
126$\times$28$\times$28 | $\left[\left[\begin{array}[]{ll}\text{ \ \ \ 126-dim}\text{ MHSA}\\\ \text{3.0$\times$}\text{FFN}/\text{2.0$\times$}\text{NLL}\end{array}\right]\times\text{2}\right]\times\text{2}$ | 336$\times$28$\times$28 | $\left[\left[\begin{array}[]{ll}\text{ \ \ \ 336-dim}\text{ MHSA}\\\ \text{3.0$\times$}\text{FFN}/\text{2.0$\times$}\text{NLL}\end{array}\right]\times\text{2}\right]\times\text{2}$
Conv-Pooling Layer (1) | 252$\times$14$\times$14 | 3$\times$3 conv, stride 2, group 126 | 672$\times$14$\times$14 | 3$\times$3 conv, stride 2, group 336
| Recursive T Block
---
(2)
252$\times$14$\times$14 | $\left[\left[\begin{array}[]{ll}\text{ \ \ \ 252-dim}\text{ MHSA}\\\ \text{3.0$\times$}\text{FFN}/\text{2.0$\times$}\text{NLL}\end{array}\right]\times\text{5}\right]\times\text{2}$ | 672$\times$14$\times$14 | $\left[\left[\begin{array}[]{ll}\text{ \ \ \ 672-dim}\text{ MHSA}\\\ \text{3.0$\times$}\text{FFN}/\text{2.0$\times$}\text{NLL}\end{array}\right]\times\text{5}\right]\times\text{2}$
Conv-Pooling Layer (2) | 504$\times$7$\times$7 | 3$\times$3 conv, stride 2, group 252 | 1344$\times$7$\times$7 | 3$\times$3 conv, stride 2, group 672
| Recursive T Block
---
(3)
504$\times$7$\times$7 | $\left[\left[\begin{array}[]{ll}\text{ \ \ \ 504-dim}\text{ MHSA}\\\ \text{3.0$\times$}\text{FFN}/\text{2.0$\times$}\text{NLL}\end{array}\right]\times\text{3}\right]\times\text{2}$ | 1344$\times$7$\times$7 | $\left[\left[\begin{array}[]{ll}\text{ \ \ \ 1344-dim}\text{ MHSA}\\\ \text{3.0$\times$}\text{FFN}/\text{2.0$\times$}\text{NLL}\end{array}\right]\times\text{3}\right]\times\text{2}$
Global Average Pooling | 504$\times$1$\times$1 | AdaptiveAvgPool | 1344$\times$1$\times$1 | AdaptiveAvgPool
Linear Layer | 1000
#Params (M) |  | 20.9 M |  | 71.2 M
Accuracy (%) |  | 82.0 |  | 82.7
Distilled Accuracy (%) |  | 82.8 |  | 83.7
Finetuning Accuracy $\uparrow$384 (%) |  | 83.8 |  | 84.8

##  Appendix 0.H All-MLP Structure

We use B/16 in Mixer architectures [51] as our backbone network. In
particular, it contains 12 layers, the patch resolution is $16\times 16$, the
hidden size $C$ is 768, the sequence length $S$ is 196, the MLP dimension
$D_{C}$ and $D_{S}$ are 3072 and 384, respectively.

##  Appendix 0.I Ablation Study on Different LRC Designs

In this section, we verify the effectiveness of different LRC designs as shown
in Fig. 15, including: (1) learnable coefficients on the identity mapping
branch; (2) learnable coefficients on the main self-attention/MLP branch; (3)
our used design in the main text, i.e., including learnable coefficients on
both branches.

The quantitative results of different LRC designs are shown in Table 13, we
can observe that strategy (1) is slightly better than (2), while, (3) can
achieve consistent improvement over (1) and (2), and it is applied in our main
text. We further visualize more evolution visualizations on various
layers/depths of our SReT-TL architecture. The results are shown in Fig. 16
and the analysis is provided in Sec. 0.K.

Table 13: Ablation study on different LRC designs.

Method | #Params (M) | Top-1 Acc. (%)
---|---|---
Baseline (SReT-TL w/o LRC) | 5.0 | 74.7
on $x$ branch (1) | 5.0 | 75.0
on $f$ branch (2) | 5.0 | 74.9
on both (3) | 5.0 | 75.2

##  Appendix 0.J Observations of Response Maps

We have a few interesting observations on the visualizations of Fig. 8 (main
text): (1) In the uniform size of transformer DeiT, information in the shallow
layers is basically vague, blurry and lacks details. In contrast, the high-
level layers contain stronger semantic information and are more aligned with
the input. However, our model has a completely different behavior: first, in
the same block but with different recursive operations, we can observe that
the features are hierarchical (in Fig. 8 of main text (2)). Taken as a whole,
shallow layers can capture more details like edges, shapes and contours and
deep layers focus on the high-level semantic information, which is similar to
CNNs. We emphasize such hierarchical representation enabled by recursion and
spatial pyramid is critical for vision modality like images.

##  Appendix 0.K More Evolution Visualization of LRC Coefficients on
ImageNet-1K Dataset

The visualizations of coefficients evolution at different recursive blocks and
layers are shown in Fig. 15. Intriguingly, we can observe in the deep layers
of recursive blocks, $\alpha$ tends to be one stably during the whole
training. Other coefficients on the identity mapping ($\gamma$ and $\zeta$)
are holding fixed values that are also close to one during the training. This
phenomenon indicates that the identity mapping branch tends to pass the
original signal with small scaling. Moreover, it seems the contributions of
the two branches have a particular proportion for the particular depth of
layers.

Figure 16: Evolution of coefficients at different recursive blocks and layers.

##  Appendix 0.L Evolution Visualization of LRC Coefficients on Language Model

The visualization of coefficients evolution on the language model is shown in
Fig. 17. Different from the evolution in vision transformer models, the
coefficients in language model are more stable during training with small
variance. Also, they are symmetrical with value one.

Figure 17: Evolution of coefficients on language of WMT14 En-De dataset.

##  Appendix 0.M More Ablation Results on Directly Enlarging Depth of Baseline
DeiT Model

In this section, we provide the results by directly expanding the depth of
baseline DeiT model, as shown in Table 14. We can see deeper naïve DeiT could
not bring additional gain on performance since the deeper and heavier network
is usually more difficult to learn meaningful and diverse intermediate
features, while our recursive operation through sharing/reusing parameters is
an effective way to enlarge the depth of a transformer, meanwhile, obtaining
extra improvement.

Table 14: More ablation results on directly expanding depth of baseline DeiT
model. * indicates that the total number layers of our network is 20
(recursive transformer blocks) + 10 (NLL) + 3 (image patch embeddings).
Permutation and inverse permutation layers are not included.

Method | #Layers | #Params (M) | Top-1 Acc. (%)
---|---|---|---
DeiT-Tiny [52] | 12 | 5.7 | 72.20
\+ extend depth | 24 | 11.55 | 77.35
\+ extend depth | 36 | 16.39 | 77.18
\+ extend depth | 48 | 21.73 | 75.89
Ours (SReT-S) | 33* | 20.90 | 81.90

##  Appendix 0.N More Definitions and Explanations to Prior Arts

Feed-forward Networks, Recurrent Neural Networks and Recursive Neural
Networks. To clarify the definition of proposed recursive operation, we
distinct recursive neural networks from feed-forward networks and recurrent
neural networks. Feed-forward networks, such as CNNs and transformers, are
directed acyclic graphs (DAG). The information path in the feed-forward
processing is unidirectional, making the feed-forward networks hard to tackle
the structured data with long-span correlations. Recurrent neural networks
(RNNs) are usually developed to process the time-series and other sequential
data. They output predictions based on the current input and past memory, so
they are capable of processing data that contains long-term interdependent
compounds like language. Recursive network is a less frequently used term
compared to other two counterparts. Recursive refers to repeating or reusing a
certain piece of a network. Different from RNNs that repeat the same block
throughout the whole network, recursive neural network selectively repeats
critical blocks for particular purposes. The recursive transformer iteratively
refines its representations for all image patches in the sequence.

Difference to Prior Arts: On CNNs, ShuffleNet [62] uses inerratic shuffle for
efficient design while it is not truly random. Thus, there is no inverse
operation involved. In contrast, our permutation is entirely stochastic and
inverse is crucial since self-attention is sensitive to tokens’ order. The
naïve group self-attention only has interaction within the window, Swin [37]
addresses this using shifted windows across different layers. While, we solve
it by integrating “slice+permutation+recursion” on the same layer’s
parameters, so each layer enables to interact with all other windows, not
across layers as Swin.

## References

  * [1] https://github.com/pytorch/fairseq/blob/master/examples/translation/README.md
  * [2] https://workshop2014.iwslt.org/downloads/proceeding.pdf
  * [3] https://www.statmt.org/wmt14/translation-task.html
  * [4] Bagherinezhad, H., Horton, M., Rastegari, M., Farhadi, A.: Label refinery: Improving imagenet classification through label progression. arXiv preprint arXiv:1805.02641 (2018)
  * [5] Bai, S., Kolter, J.Z., Koltun, V.: Deep equilibrium models. In: Proceedings of the International Conference on Neural Information Processing Systems (2019)
  * [6] Bai, S., Kolter, J.Z., Koltun, V.: Trellis networks for sequence modeling. In: ICLR (2019)
  * [7] Bai, S., Koltun, V., Kolter, J.Z.: Multiscale deep equilibrium models. In: Proceedings of the International Conference on Neural Information Processing Systems (2020)
  * [8] Beyer, L., Hénaff, O.J., Kolesnikov, A., Zhai, X., Oord, A.v.d.: Are we done with imagenet? arXiv preprint arXiv:2006.07159 (2020)
  * [9] Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. arXiv preprint arXiv:2005.14165 (2020)
  * [10] Chen, M., Peng, H., Fu, J., Ling, H.: Autoformer: Searching transformers for visual recognition. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (2021)
  * [11] Chowdhury, J.R., Caragea, C.: Modeling hierarchical structures with continuous recursive neural networks. In: Proceedings of the 38th International Conference on Machine Learning. pp. 1975–1988 (2021)
  * [12] Dehghani, M., Gouws, S., Vinyals, O., Uszkoreit, J., Kaiser, L.: Universal transformers. In: International Conference on Learning Representations (2018)
  * [13] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition. pp. 248–255. Ieee (2009)
  * [14] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pp. 4171–4186 (2019)
  * [15] Dong, L., Xu, S., Xu, B.: Speech-transformer: A no-recurrence sequence-to-sequence model for speech recognition. In: 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 5884–5888 (2018)
  * [16] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale. In: International Conference on Learning Representations (2021)
  * [17] FAIR: https://github.com/pytorch/fairseq
  * [18] Guo, Q., Yu, Z., Wu, Y., Liang, D., Qin, H., Yan, J.: Dynamic recursive neural network. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5147–5156 (2019)
  * [19] Han, K., Xiao, A., Wu, E., Guo, J., Xu, C., Wang, Y.: Transformer in transformer. arXiv preprint arXiv:2103.00112 (2021)
  * [20] He, K., Zhang, X., Ren, S., Sun, J.: Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE transactions on pattern analysis and machine intelligence 37(9), 1904–1916 (2015)
  * [21] He, K., Zhang, X., Ren, S., Sun, J.: Identity mappings in deep residual networks. In: European conference on computer vision. pp. 630–645 (2016)
  * [22] Hendricks, L.A., Mellor, J., Schneider, R., Alayrac, J.B., Nematzadeh, A.: Decoupling the role of data, attention, and losses in multimodal transformers. arXiv preprint arXiv:2102.00529 (2021)
  * [23] Hendrycks, D., Gimpel, K.: Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415 (2016)
  * [24] Heo, B., Yun, S., Han, D., Chun, S., Choe, J., Oh, S.J.: Rethinking spatial dimensions of vision transformers. arXiv preprint arXiv:2103.16302 (2021)
  * [25] Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015)
  * [26] Kim, J., Lee, J.K., Lee, K.M.: Deeply-recursive convolutional network for image super-resolution. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1637–1645 (2016)
  * [27] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  * [28] Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., Soricut, R.: Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942 (2019)
  * [29] Lazebnik, S., Schmid, C., Ponce, J.: Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. In: 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06). vol. 2, pp. 2169–2178 (2006)
  * [30] Li, H., Xu, Z., Taylor, G., Studer, C., Goldstein, T.: Visualizing the loss landscape of neural nets. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems. pp. 6391–6401 (2018)
  * [31] Li, Y., Zhang, K., Cao, J., Timofte, R., Van Gool, L.: Localvit: Bringing locality to vision transformers. arXiv preprint arXiv:2104.05707 (2021)
  * [32] Liang, M., Hu, X.: Recurrent convolutional neural network for object recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3367–3375 (2015)
  * [33] Liu, F., Gao, M., Liu, Y., Lei, K.: Self-adaptive scaling for learnable residual structure. In: Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL) (2019)
  * [34] Liu, S., Yang, N., Li, M., Zhou, M.: A recursive recurrent neural network for statistical machine translation. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 1491–1500 (2014)
  * [35] Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V.: Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019)
  * [36] Liu, Y., Wang, Y., Wang, S., Liang, T., Zhao, Q., Tang, Z., Ling, H.: Cbnet: A novel composite backbone network architecture for object detection. In: Proceedings of the AAAI conference on artificial intelligence. vol. 34, pp. 11653–11660 (2020)
  * [37] Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103.14030 (2021)
  * [38] Luong, M.T., Pham, H., Manning, C.D.: Effective approaches to attention-based neural machine translation. In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. pp. 1412–1421 (2015)
  * [39] Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (2002)
  * [40] Pham, H., Dai, Z., Xie, Q., Luong, M.T., Le, Q.V.: Meta pseudo labels. arXiv preprint arXiv:2003.10580 (2020)
  * [41] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I.: Language models are unsupervised multitask learners. OpenAI blog 1(8), 9 (2019)
  * [42] Radosavovic, I., Kosaraju, R.P., Girshick, R., He, K., Dollár, P.: Designing network design spaces. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10428–10436 (2020)
  * [43] Recht, B., Roelofs, R., Schmidt, L., Shankar, V.: Do imagenet classifiers generalize to imagenet? In: International Conference on Machine Learning. pp. 5389–5400. PMLR (2019)
  * [44] Romero, A., Ballas, N., Kahou, S.E., Chassang, A., Gatta, C., Bengio, Y.: Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550 (2014)
  * [45] Shen, Z., He, Z., Xue, X.: Meal: Multi-model ensemble via adversarial learning. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 33, pp. 4886–4893 (2019)
  * [46] Shen, Z., Liu, Z., Xu, D., Chen, Z., Cheng, K.T., Savvides, M.: Is label smoothing truly incompatible with knowledge distillation: An empirical study. In: International Conference on Learning Representations (2021)
  * [47] Shen, Z., Liu, Z., Li, J., Jiang, Y.G., Chen, Y., Xue, X.: Dsod: Learning deeply supervised object detectors from scratch. In: Proceedings of the IEEE international conference on computer vision. pp. 1919–1927 (2017)
  * [48] Shen, Z., Savvides, M.: Meal v2: Boosting vanilla resnet-50 to 80%+ top-1 accuracy on imagenet without tricks. In: NeurIPS Workshop (2020)
  * [49] Sperduti, A., Starita, A.: Supervised neural networks for the classification of structures. IEEE Transactions on Neural Networks 8(3), 714–735 (1997)
  * [50] Srinivas, A., Lin, T.Y., Parmar, N., Shlens, J., Abbeel, P., Vaswani, A.: Bottleneck transformers for visual recognition. arXiv preprint arXiv:2101.11605 (2021)
  * [51] Tolstikhin, I., Houlsby, N., Kolesnikov, A., Beyer, L., Zhai, X., Unterthiner, T., Yung, J., Keysers, D., Uszkoreit, J., Lucic, M., Dosovitskiy, A.: Mlp-mixer: An all-mlp architecture for vision. arXiv preprint arXiv:2105.01601 (2021)
  * [52] Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., Jégou, H.: Training data-efficient image transformers & distillation through attention. arXiv preprint arXiv:2012.12877 (2020)
  * [53] Touvron, H., Cord, M., Sablayrolles, A., Synnaeve, G., Jégou, H.: Going deeper with image transformers. arXiv preprint arXiv:2103.17239 (2021)
  * [54] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. In: NIPS (2017)
  * [55] Wang, W., Xie, E., Li, X., Fan, D.P., Song, K., Liang, D., Lu, T., Luo, P., Shao, L.: Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. arXiv preprint arXiv:2102.12122 (2021)
  * [56] Wang, Y., Shi, Y., Zhang, F., Wu, C., Chan, J., Yeh, C.F., Xiao, A.: Transformer in action: a comparative study of transformer-based acoustic models for large scale speech recognition applications. In: ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 6778–6782. IEEE (2021)
  * [57] Wu, H., Xiao, B., Codella, N., Liu, M., Dai, X., Yuan, L., Zhang, L.: Cvt: Introducing convolutions to vision transformers. arXiv preprint arXiv:2103.15808 (2021)
  * [58] Xie, Q., Luong, M.T., Hovy, E., Le, Q.V.: Self-training with noisy student improves imagenet classification. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10687–10698 (2020)
  * [59] Xu, W., Xu, Y., Chang, T., Tu, Z.: Co-scale conv-attentional image transformers. arXiv preprint arXiv:2104.06399 (2021)
  * [60] Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R.R., Le, Q.V.: Xlnet: Generalized autoregressive pretraining for language understanding. Advances in Neural Information Processing Systems (2019)
  * [61] Yuan, L., Chen, Y., Wang, T., Yu, W., Shi, Y., Tay, F.E., Feng, J., Yan, S.: Tokens-to-token vit: Training vision transformers from scratch on imagenet. arXiv preprint arXiv:2101.11986 (2021)
  * [62] Zhang, X., Zhou, X., Lin, M., Sun, J.: Shufflenet: An extremely efficient convolutional neural network for mobile devices. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 6848–6856 (2018)
  * [63] Zhou, D., Kang, B., Jin, X., Yang, L., Lian, X., Jiang, Z., Hou, Q., Feng, J.: Deepvit: Towards deeper vision transformer. arXiv preprint arXiv:2103.11886 (2021)