# When Training-Free NAS Meets Vision Transformers: A Neural Tangent Kernel
Perspective

###### Abstract

This paper investigates the Neural Tangent Kernel (NTK) to search vision
transformers without training. In contrast with the previous observation that
NTK-based metrics can effectively predict CNN’s performance at initialization,
we empirically show their inefficacy in the ViT search space. We hypothesize
that the fundamental feature learning preference within ViT contributes to the
ineffectiveness of applying NTK to NAS for ViT. We both theoretically and
empirically validate that NTK essentially estimates the ability of neural
networks that learn low-frequency signals, completely ignoring the impact of
high-frequency signals in feature learning. To address this limitation, we
propose a new method called ViNTK that generalizes the standard NTK to the
high-frequency domain by integrating the Fourier features from inputs.
Experiments with multiple ViT search spaces on image classification and
semantic segmentation tasks show that our method can significantly speed up
search costs over prior state-of-the-art NAS for ViT while maintaining similar
performance on searched architectures.

Index Terms—  Neural Architecture Search; Vision Transformer

##  1 Introduction

Vision Transformer (ViT) is increasingly matching CNNs in various vision tasks
in various domains such as VQA and robotics [1, 2, 3, 4]. Prior studies have
presented many ways to accelerate vision transformer, i.e., distillation [5,
6, 7], dynamic networks [8] Neural architecture search (NAS). The latter
aiming for hands-off architectural design, has successfully influenced CNNs
across tasks. Yet, traditional training-based NAS methods are resource-
draining. There’s a rising interest in training-free NAS, which assesses
untrained architectures using performance predictors. Given ViT’s complex
structure, including elements like multi-head self-attention (MSAs) and feed-
forward networks (FFN), the reliability of training-free metrics tailored for
CNNs in the ViT context is uncharted.

Fig. 1: The correlation analysis of NTK-based metrics on CNN and ViT search
spaces. Left: Kendall-Tau value on NAS-Bench-201 [9]. Right: Kendall-Tau value
on two ViT search spaces.

The shortcomings of NTK-based metrics for ViT are intriguing. We hypothesize
that this is due to ViT’s mixed feature learning methods. Though its self-
attention mechanism functions as a low-pass filter, ViT still leans on high-
frequency signals for discerning images. This duality can skew metrics. Our
study shows that NTK primarily reflects networks’ low-frequency learning
prowess. Tests focusing on ViT’s MSAs affirm that NTK can adeptly pinpoint
potent ViT architectures, reinforcing MSAs’ role as established low-pass
filters.

This makes us wonder what causes the inefficacy of NTK-based metrics on ViT
search spaces. We posit that this observation is attributed to the conflict in
feature learning of ViT. Recall that the self-attention mechanism acts as a
low-pass filter. The ViT still relies on high-frequency signals to
discriminate images. As such, the conflict feature learning capacity between
different components confuses the metrics. We give a theoretical analysis to
understand this phenomenon by proving an approximation error of NTK’s
predictive ability on deep neural networks (DNNs) under mild conditions.
Specifically, our analysis shows that NTK-based metrics are strongly
correlated to the dimension of low-frequency signals. In other words, NTK can
reasonably approximate the strength of low-frequency signals learned by neural
networks. As proof of concept, we again assess the NTK-based metrics on ViT
search space, searching multi-head self-attention (MSAs) only. Surprisingly,
the experimental results show that NTK can find promising ViT when only MSAs,
provable low-pass filters, are involved in the search process.

While current analyses of NTK on ViT seem adequate, they reveal that NTK-based
training-free metrics aren’t ideal for finding top-tier ViT. We introduce the
Fourier Neural Tangent Kernel, transforming NTK metrics using Fourier feature
mapping. This upgrade enhances NTK’s capacity to predict high-frequency
signals, mitigating the conflict between MSAs and other high-frequency
modules.

##  2 Analysis of Using NTK-Based Metrics to Search ViT

###  2.1 Preliminary

Suppose there are $D$ training points denoted by $\\{(x,y)_{d=1}^{D}\\}$,
where input $X=(x_{1},\dots,x_{D})\in\mathbb{R}^{n_{0}\times D}$, and label
$Y=(y_{1},y_{D})\in\mathbb{R}^{n_{L}\times D}$. We consider either a fully
connected network or a convolutional neural network. Since our analysis does
not involve these two networks, We omit the detailed descriptions for
simplicity.

A large body of recent works [10] has shown that when training deep networks
in a certain over-parameterized regime, the dynamic of gradient descent
behaves like those of linear models on non-linear features determined at
initialization. With sufficient over-parameterization, the optimization
dynamics converge to a certain gram matrix in the reproducing kernel Hilbert
space (RKHS) known as the neural tangent kernel
$\Theta_{ntk}(x,x^{\prime}):=\langle\nabla_{\theta}f(x,\theta),\nabla_{\theta}f(x^{\prime},\theta))\rangle$.
Where function $f$ is the output of the network at training time, i.e.,
$f(x,\theta)=h_{t}^{L}(\theta,X)\in\mathbb{R}^{D\times n_{L}}$, and
$\nabla_{\theta}f(\theta,X)=vec([\nabla_{\theta}f(x,\theta)]_{x\in
X})\in\mathbb{R}^{D_{n_{L}}}$.

In this limit and under some assumptions, one can show that weights barely
move, and the kernel remains fixed during training. Such property motivates
recent training-free NAS to directly calculate some NTK-based metrics for
random initialized CNNs and use the metrics as a performance proxy.

###  2.2 Rank Correlation Analysis

In this section, we evaluate how reliable NTK-based metrics are for training-
free NAS. We compare with four variants of NTK-based metrics and refer readers
to more detailed descriptions of their papers. These methods include F-Norm
NTK [11], Mean NTK [11], NCN NTK [12], and ReLU NTK [13]. We conduct
experiments on a pure ViT search space and a hybrid ViT-CNN search space. For
each ViT-based search space, we randomly sample 100 architectures. All of them
are trained to convergence, and the corresponding metrics are reported.
Finally, we briefly introduce two ViT-based search spaces, where the
comprehensive description of the listed search space is presented in the
Appendix.

ViT search space. We construct a simple, pure ViT search space that is
composed of kernel size (used to split image patches) and expansion number for
feed-forward networks. We follow the Twins [14] to build a four-stage ViT, a
down-sample module at the top of each stage to reduce the spatial dimension of
feature maps gradually.

ViT-CNN hybrid search space. We follow the NASViT [15] to construct a ViT-CNN
hybrid search space to verify our claim. The search space consists of two CNN
blocks followed by two transformer blocks. A downsampling module is inserted
prior to each building block. We search for the kernel size and SE module in
the CNN block, as well as the expansion number in the Transformer block.

In Figure 1, we report the Kendall-tau correlations $\mathbf{\tau}$ between
these metrics and the model’s ground-truth accuracies. On both ViT search
spaces, the Kendall-Tau correlation on all four metrics is low: on the pure
ViT search space, the highest Kendall-tau is 0.07, and the lowest Kendall-Tau
is -0.05. Similar results are obtained on hybrid ViT-CNN search space, with a
slightly better highest Tau value $\tau=0.14$. These are counter-intuitive
results since NTK is very effective on NAS-Bench-201 [9]. For instance, NCN-
NTK achieves over $\tau$ value over 0.5 on NAS-Bench-201; and even the worse
metric, F-Norm, obtains $\tau=0.28$. Based on our observation, it seems that
the magic of NTK-based metrics is somehow gone in the ViT search space. The
following section sheds light on this question.

###  2.3 Why NTK Fails to Find Good ViT?

In this section, we provide a theoretical justification to answer why NTK
fails to find good ViT. First, we provide an approximation error for NTK on
deep neural networks (DNNs), which is extended from Ghorban et al. [16]. We
show that NTK is a better approximator for DNN that relies on low-frequency
features. We further provide a proof of concept to show our justification can
indeed explain such phenomena.

We start our analysis by setting up a simple proxy for Fourier decomposition,
the decomposition of covariate vector $\mathbf{x}$, where its low-dimensional
dominant component and high-dimensional dominant component represent low-
frequency and high-frequency features, respectively. Then, we introduce the
spiked covariates model that approximates low-dimensional covariates. That is,
we not only expect the target function $f^{*}$ to depend predominantly on the
low-frequency components of image $\mathbf{x}$, but the image $\mathbf{x}$
itself to have most of its spectrum concentrated on low-frequency components.
As such, our expected target function would be the neural network that can
approximate the low-frequency features. Specifically, we consider
$\mathbf{x}=\mathbf{U}\mathbf{z}_{1}+\mathbf{U}^{\perp}\mathbf{z}_{2}$,
$\mathbf{U}^{\perp}\in\mathbb{R}^{d\times d_{0}}$,
$\mathbf{U}\in\mathbb{R}^{d\times(d-d_{0})}$,
$[\mathbf{U}|\mathbf{U}^{\perp}]\in\mathbb{R}^{d\times d_{0}}$ is an
orthogonal matrix. We assume that $\mathbf{z}_{1}$ and $\mathbf{z}_{2}$ are
uniformly distributed over the sphere $\mathbf{z}_{1}\sim
U(S^{d_{0}-1}(r_{1}\sqrt{d_{0}}))$, $\mathbf{z}_{2}\sim
U(S^{d-d_{0}-1}(r_{1}\sqrt{d-d_{0}}))$, and $r_{1}\geq r_{2}$. The $d_{0}$ is
signal dimension, $d$ is ambient dimension, and $\frac{r_{1}}{r_{2}}$ is
covariate signal-to-noise ratio. We can have the next theorem that
characterizes the asymptotics of the approximation error for NTK on NN:

Theorem 1. (Approximation of NTK for NN) Without loss of generality, we assume
the target function $f_{*}(\mathbf{x})=\sigma(\mathbf{U}^{T}\mathbf{x})$.
Assume that $\sigma\in\mathcal{C}^{\infty}(\mathbf{R}),\sigma^{k}(x)^{2}\leq
c_{0,k}e^{c_{1,k}\frac{x^{2}}{2}}$, for $\forall k,\forall
x\in\mathcal{R},c_{0,k}>0$ and $c_{1,k}<1$, then we have

$\begin{split}|\mathbf{R}_{NN,N}(f_{*};\mathbf{W})-\mathbf{R}_{NTK,N}(f_{*};\mathbf{W})|\leq\\\
o_{d}(1)\cdot||P_{>l+1}f_{*}||_{L^{2}}(1-||f_{*}||^{2}_{L^{2}}+\tau^{2})\end{split}$
(1)

The $P_{>k}=I-P_{\leq k}$ and $P_{\leq k}:L^{2}\rightarrow L^{2}$ is a mapping
function to the space of degree k orthogonal polynomials. The
$\mathbf{R}_{NN}$ and $\mathbf{R}_{NTK}$ denote the prediction error of neural
networks and NTK with respect to the data. Remarkably, the above theorem shows
that the effective signal dimension controls the sample complexity N for NTK
in approximating an NN to learn a degree l polynomial. Based on our initial
setting, the low-frequency components effectively contribute to the capacity
of NTK on the ability of performance estimation of NN. Theorem 1 explains that
NTK can achieve remarkably better approximation on NN because ViT extracts
low-frequency signals to recognize the object.

Fig. 2: The Kendall-Tau correlation for MSAs only in two ViT search spaces.
The value of $\tau$ clearly improves when only MSAs are involved in the search
space. Table 1: Image Classification on ImageNet-1k with AutoFormer [17]
search space.

Method | Param (M) | FLOPs (B) | Top-1 (%) | Top-5 (%) | Search Cost
---|---|---|---|---|---
DeiT-T [18] | 5.7 | 1.2 | 72.2 | 91.1 | -
ViTAS-C [19] | 5.6 | 1.3 | 74.7 | 91.6 | 32
AutoFormer-T [17] | 5.7 | 1.4 | 74.7 | 92.6 | 24
TF-TAS-Ti [20] | 5.9 | 1.4 | 75.3 | 92.8 | 0.5
ViNTK-T | 5.8 | 1.3 | 75.6 | 93.1 | 0.8
DeiT-S [18] | 22.1 | 4.7 | 79.9 | 95.0 | -
Swin-T [21] | 29.0 | 4.5 | 81.3 | - | -
ViTAS-F [19] | 27.6 | 6.0 | 80.5 | 95.1 | 32
AutoFormer-S [17] | 22.9 | 5.1 | 81.7 | 95.7 | 24
TF-TAS-S [20] | 22.8 | 5.0 | 81.9 | 95.8 | 0.5
ViNTK-S | 22.7 | 4.9 | 82.3 | 96.2 | 0.8
DeiT-B [18] | 86.0 | 18.0 | 81.8 | 95.6 | -
Swin-B [21] | 88.0 | 15.4 | 83.5 | - | -
AutoFormer-B [17] | 54.0 | 11.0 | 82.4 | 95.7 | 24
TF-TAS-B [20] | 54.0 | 12.0 | 82.2 | 95.6 | 0.5
ViNTK-B | 53.3 | 10.8 | 82.3 | 95.6 | 0.8
ViNTK-L | 53.9 | 12.0 | 83.0 | 96.3 | 0.8

Proof of concept. Our theory suggests that NTK effectively approximates
networks learning low-frequency signals. Thus, we divide the search space:
MSAs represent low-frequency components, while convolutional layers, patch
embedding, and feed-forward networks represent high-frequency. Only MSAs are
available for search, ensuring other architectural capabilities remain
constant. This means NTK-based metrics should mainly reflect the strength of
low-frequency signals.

Figure 2 showcases the correlation between NTK metrics and accuracy when only
MSAs are considered. The correlation is significantly stronger than in a full
search space. Specifically, the correlation values in pure ViT and hybrid ViT-
CNN spaces increase by 0.28 and 0.24, respectively, confirming our theoretical
insights and highlighting potential shortcomings in NTK-based metrics.

##  3 Fourier Neural Tangent Kernel

Drawing inspiration from the Fourier features, we propose to align the Fourier
features with NTK. Specifically, consider optimizing a parametric model with
some bounded loss $l(y,\hat{y})$, where $\hat{y}$ is the prediction of the
models on the training dataset. We can rewrite the NTK by taking the
eigendecomposition of NTK matrix $\Theta_{ntk}$ into $\Theta_{ntk}=QAQ^{T}$,
where $Q$ is an orthogonal matrix and $A$ is a diagonal matrix whose entries
are the eigenvalues $\lambda_{i}\leq 0$ of $\Theta_{ntk}$, given
$\Theta_{ntk}$ is a symmetric positive semi-definite matrix. As a result, we
reformulate the NTK matrix:

$\Theta_{ntk}=-e^{-\eta At}Q^{T}y$ (2)

The above equation represents that in the eigenbasis of the NTK, the
convergence rate of $i^{th}$ component of NTK will decay approximately
exponentially at the rate of $\eta\lambda$ [22]. Consequently, the convergence
of high-frequency signals is slower than low-frequency signals. We would like
to fix this issue by balancing the convergence rate on the Fourier spectrum.

We leverage random Fourier features [23], introduced to approximate an
arbitrary stationary kernel function via Bochner’s theorem. We use their
method to feature input on the Fourier domain and combine it with NTK.
Specifically, a Fourier feature mapping function $\gamma$ transforms input to
the surface of a higher dimensional hyper-sphere and obtains the Fourier
kernel function as

$\Theta_{fourier}(x,x^{\prime})=\gamma(x)^{T}\gamma(x^{\prime})=\sum^{m}_{j=1}\alpha^{2}_{j}cos(2\pi
b^{T}_{j}(x-x^{\prime}))$ (3)

where $\gamma(\cdot)$ is a set of sinusoids. Combined with NTK, we now have

$\Theta_{vintk}:=\Theta_{ntk}(x,x^{\prime})\circ\Theta_{fourier}(x,x^{\prime})$
(4)

In practice, the $b_{j}=j$ is set as full Fourier basis,
$\alpha_{j}=\frac{1}{j_{p}}$ and $p=2$ for $j=1,\dots,\frac{n}{2}$. For NTK,
we directly adopt Mean-NTK. The extra computational cost of calculating ViNTK
is negligible, making our method ultra-fast at the search stage.

To verify the effectiveness of our proposed metric, we conduct a Kendall-Tau
$\tau$ correlation test on ImageNet-1K, as shown in Figure 3. Note that all
p-values for the tau correlation are less than 0.005 to ensure the results are
statistically significant. We demonstrate a strong correlation between the
accuracy and the ViNTK metric, showing the efficacy of our proposed metric in
predicting ViT’s performance.

Fig. 3: The correlation of our proposed ViNTK. Table 2: Image Classification
on ImageNet-1k with NASViT search space. The model size range from 300M to 2G.

Size | Method | FLOPs (M) | Top-1 ($\%$) | Search Cost
---|---|---|---|---
200M - 300M | AlphaNet-A0 [24] | 203 | 77.9 | 45
NASViT-A0 [15] | 208 | 78.2 | 27
ViNTK-A0 | 210 | 78.3 | 1.0
300M - 400M | LeViT [25] | 300 | 76.6 | -
AlphaNet-A1 [24] | 317 | 79.4 | 75
NASViT-A1 [15] | 309 | 79.7 | 27
ViNTK-A1 | 310 | 79.7 | 1.0
400M - 500M | LeViT [25] | 406 | 78.6 | -
AlphaNet-A4 [24] | 444 | 80.4 | -
NASViT-A2 [15] | 421 | 80.5 | 27
ViNTK-A2 | 417 | 80.7 | 1.0
600M - 1G | LeViT [25] | 658 | 80.0 | -
NASViT-A5 [15] | 757 | 81.8 | 27
ViNTK-A5 | 772 | 82.0 | 1.0
1G - 2G | PiT-XS [26] | 1,400 | 79.1 | -
NASViT-A6 [15] | 1,881 | 82.9 | 27
ViNTK-A6 | 1,910 | 82.9 | 1.0

##  4 Experiments

We validate the effectiveness of ImageNet-1K classification [27] with two
search spaces and semantic segmentation on both Cityscapes [28] and ADE20K
[29].

Fig. 4: Experiments on semantics segmentation.

###  4.1 Image Classification on ImageNet-1K

Searching Width and Depth on AutoFormer Search Space. We tested our ViNTK in
AutoFormer search space [17], comparing it to manual designs like DeiT [18],
Swin [21], and NAS approaches such as ViTAS, AutoFormer, and TF-TAS. As shown
in Table 1, ViNTK-identified architectures surpass manually-designed ViTs.
Importantly, while our designs perform similarly to AutoFormer, our algorithm
is 30x more efficient in search cost. Against TF-TAS, a training-free NAS, our
approach either achieves better accuracy or reduced FLOPs, albeit at a
slightly increased search cost.

Searching Width + Depth + Operations on NASViT Search Space. We adopted NASViT
search space for models ranging from 300M to 2G FLOPs, benchmarking against
NASViT. As depicted in Table 2, our method achieves similar or slightly
superior results to NASViT but at 1/27th of the search cost. Traditional
methods like NASViT and AlphaNet necessitate extended training on supernets,
making them less suitable for resource-limited settings. Our approach proves
to be significantly more efficient than leading NAS techniques.

###  4.2 Semantic Segmentation

We further evaluate our method of semantic segmentation. Instead of directly
finetuning our discovered architectures from ImageNet, we directly search the
architecture via the target dataset. In particular, we search for four
architectures with varying model sizes corresponding to the NASViT. On two
benchmarks, Cityscapes [28] and ADE20K [29], we conduct separate search
processes in order to find the optimal model. We set a hard threshold during
our search process to ensure all architectures have similar FLOPs compared to
NASViT. We also compare our method with SegFormer [30], MobileNetv3 [31], and
AlphaNet [24]. Similar to NASViT, we adopt the lightweight head from SegFormer
as the decoder head for all backbones. Figure 4 presents the experimental
results. By directly searching the target dataset, ViNTK achieved better
performance than NASViT. For instance, on Cityscapes, among four different
architecture, two of them achieve better performance than NASViT. As for
ADE20K, all four data points obtain higher mIoU than NASViT. These results
indicate that direct searching can be beneficial for training-free NAS.

##  5 Conclusions

Training-free NAS requires a deep understanding of deep neural networks. Our
work thoroughly discusses the effectiveness of NTK-based training-free metrics
in searching vision transformers from empirical study and theoretical
analysis. To improve the inefficacy of NTK in approximating networks that
learn high-frequency signals, we present a Fourier-NTK to resolve this issue.
Moreover, we propose Fourier NTK to enhance the estimation power of training-
free metrics. Our method significantly reduces the search cost of conventional
training-based NAS for ViT.

## References

  * [1] Yichen Zhu, Minjie Zhu, Ning Liu, Zhicai Ou, Xiaofeng Mou, and Jian Tang,  “Llava-phi: Efficient multi-modal assistant with small language model,”  arXiv preprint arXiv:2401.02330, 2024.
  * [2] Xin Liu, Yichen Zhu, Yunshi Lan, Chao Yang, and Yu Qiao,  “Query-relevant images jailbreak large multi-modal models,”  arXiv preprint arXiv:2311.17600, 2023.
  * [3] Junjie Wen, Yichen Zhu, Minjie Zhu, Jinming Li, Zhiyuan Xu, Zhengping Che, Chaomin Shen, Yaxin Peng, Dong Liu, Feifei Feng, et al.,  “Object-centric instruction augmentation for robotic manipulation,”  arXiv preprint arXiv:2401.02814, 2024.
  * [4] Minjie Zhu, Yichen Zhu, Jinming Li, Junjie Wen, Zhiyuan Xu, Zhengping Che, Chaomin Shen, Yaxin Peng, Dong Liu, Feifei Feng, et al.,  “Language-conditioned robotic manipulation with fast and slow thinking,”  arXiv preprint arXiv:2401.04181, 2024.
  * [5] Yichen Zhu and Yi Wang,  “Student customized knowledge distillation: Bridging the gap between student and teacher,”  in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 5057–5066.
  * [6] Yichen Zhu, Ning Liu, Zhiyuan Xu, Xin Liu, Weibin Meng, Louis Wang, Zhicai Ou, and Jian Tang,  “Teach less, learn more: On the undistillable classes in knowledge distillation,”  Advances in Neural Information Processing Systems, vol. 35, pp. 32011–32024, 2022.
  * [7] Yichen Zhu, Qiqi Zhou, Ning Liu, Zhiyuan Xu, Zhicai Ou, Xiaofeng Mou, and Jian Tang,  “Scalekd: Distilling scale-aware knowledge in small object detector,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 19723–19733.
  * [8] Qiqi Zhou and Yichen Zhu,  “Make a long image short: Adaptive token length for vision transformers,”  in Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, 2023, pp. 69–85.
  * [9] Xuanyi Dong and Yi Yang,  “Nas-bench-201: Extending the scope of reproducible neural architecture search,”  arXiv preprint arXiv:2001.00326, 2020.
  * [10] Alberto Bietti and Julien Mairal,  “On the inductive bias of neural tangent kernels,”  Advances in Neural Information Processing Systems, vol. 32, 2019.
  * [11] Jingjing Xu, Liang Zhao, Junyang Lin, Rundong Gao, Xu Sun, and Hongxia Yang,  “Knas: green neural architecture search,”  in International Conference on Machine Learning. PMLR, 2021, pp. 11613–11625.
  * [12] Minghao Chen, Kan Wu, Bolin Ni, Houwen Peng, Bei Liu, Jianlong Fu, Hongyang Chao, and Haibin Ling,  “Searching the search space of vision transformer,”  Advances in Neural Information Processing Systems, vol. 34, 2021.
  * [13] Youngmin Cho and Lawrence Saul,  “Kernel methods for deep learning,”  Advances in neural information processing systems, vol. 22, 2009.
  * [14] Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen,  “Twins: Revisiting the design of spatial attention in vision transformers,”  Advances in Neural Information Processing Systems, vol. 34, 2021.
  * [15] Chengyue Gong et al.,  “NASVit: Neural architecture search for efficient vision transformers with gradient conflict aware supernet training,”  in International Conference on Learning Representations, 2022.
  * [16] Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari,  “When do neural networks outperform kernel methods?,”  Advances in Neural Information Processing Systems, vol. 33, pp. 14820–14830, 2020.
  * [17] Minghao Chen et al.,  “Autoformer: Searching transformers for visual recognition,”  in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 12270–12280.
  * [18] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou,  “Training data-efficient image transformers & distillation through attention,”  in International Conference on Machine Learning. PMLR, 2021, pp. 10347–10357.
  * [19] Xiu Su, Shan You, Jiyang Xie, Mingkai Zheng, Fei Wang, Chen Qian, Changshui Zhang, Xiaogang Wang, and Chang Xu,  “Vision transformer architecture search,”  arXiv preprint arXiv:2106.13700, 2021.
  * [20] Qinqin Zhou et al.,  “Training-free transformer architecture search,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 10894–10903.
  * [21] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo,  “Swin transformer: Hierarchical vision transformer using shifted windows,”  in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 10012–10022.
  * [22] Basri Ronen, David Jacobs, Yoni Kasten, and Shira Kritchman,  “The convergence rate of neural networks for learned functions of different frequencies,”  Advances in Neural Information Processing Systems, vol. 32, 2019.
  * [23] Ali Rahimi and Benjamin Recht,  “Weighted sums of random kitchen sinks: Replacing minimization with randomization in learning,”  Advances in neural information processing systems, vol. 21, 2008.
  * [24] Dilin Wang, Chengyue Gong, Meng Li, Qiang Liu, and Vikas Chandra,  “Alphanet: Improved training of supernets with alpha-divergence,”  in International Conference on Machine Learning. PMLR, 2021, pp. 10760–10771.
  * [25] Sachin Mehta and Mohammad Rastegari,  “Mobilevit: Light-weight, general-purpose, and mobile-friendly vision transformer,”  in International Conference on Learning Representations, 2022.
  * [26] Byeongho Heo, Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Junsuk Choe, and Seong Joon Oh,  “Rethinking spatial dimensions of vision transformers,”  in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 11936–11945.
  * [27] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton,  “Imagenet classification with deep convolutional neural networks,”  Advances in neural information processing systems, vol. 25, 2012.
  * [28] Marius Cordts et al.,  “The cityscapes dataset for semantic urban scene understanding,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 3213–3223.
  * [29] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba,  “Scene parsing through ade20k dataset,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 633–641.
  * [30] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo,  “Segformer: Simple and efficient design for semantic segmentation with transformers,”  Advances in Neural Information Processing Systems, vol. 34, 2021.
  * [31] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al.,  “Searching for mobilenetv3,”  in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 1314–1324.