# Are Neural Architecture Search Benchmarks Well Designed? A Deeper Look Into
Operation Importance

Vasco Lopes Bruno Degardin Luís A. Alexandre This work was supported by ‘FCT -
Fundação para a Ciência e Tecnologia’ through the research grant
‘2020.04588.BD’, and partially supported by NOVA LINCS (UIDB/04516/2020) with
the financial support of FCT, through national funds and by
CENTRO-01-0247-FEDER-113023 - DeepNeuronic. V. Lopes and L. A. Alexandre are
with NOVA Lincs, Universidade da Beira Interior (email: vasco.lopes@ubi.pt).
B. Degardin is with Universidade da Beira Interior.

###### Abstract

Neural Architecture Search (NAS) benchmarks significantly improved the
capability of developing and comparing NAS methods while at the same time
drastically reduced the computational overhead by providing meta-information
about thousands of trained neural networks. However, tabular benchmarks have
several drawbacks that can hinder fair comparisons and provide unreliable
results. These usually focus on providing a small pool of operations in
heavily constrained search spaces – usually cell-based neural networks with
pre-defined outer-skeletons. In this work, we conducted an empirical analysis
of the widely used NAS-Bench-101, NAS-Bench-201 and TransNAS-Bench-101
benchmarks in terms of their generability and how different operations
influence the performance of the generated architectures. We found that only a
subset of the operation pool is required to generate architectures close to
the upper-bound of the performance range. Also, the performance distribution
is negatively skewed, having a higher density of architectures in the upper-
bound range. We consistently found convolution layers to have the highest
impact on the architecture’s performance, and that specific combination of
operations favors top-scoring architectures. These findings shed insights on
the correct evaluation and comparison of NAS methods using NAS benchmarks,
showing that directly searching on NAS-Bench-201, ImageNet16-120 and TransNAS-
Bench-101 produces more reliable results than searching only on CIFAR-10.
Furthermore, with this work we provide suggestions for future benchmark
evaluations and design. The code used to conduct the evaluations is available
at https://github.com/VascoLopes/NAS-Benchmark-Evaluation.

###### Index Terms:

Neural Architecture Search, AutoML, Convolutional Neural Networks, Benchmarks,
Cell-search, Operation Importance

##  I Introduction

Advances in deep learning algorithms obtained remarkable progress in various
problems, mainly due to human experts’ ingenuity and engineering efforts that
exhaustively designed and engineered high performance architectures. Amongst
deep learning algorithms, Convolutional Neural Networks (CNNs) are within the
most used ones, having been applied with great success to a wide range of
tasks with unprecedented results, from image classification [7, 20], to
semantic segmentation [18], text analysis [6], amongst many others [28, 49,
24].

CNN’s inherent capability of feature extraction automated the process of
feature engineering and provided a mechanism to ease transferability between
different problems. Over the years, several proposals, such as skip and
residual connections, resulted in improving CNN’s capabilities to learn more
efficiently while at the same time reducing their inference time and
associated problems [27, 51, 21, 22, 52, 12]. However, such proposals are the
result of years of expertise and trial and error, as designing CNNs is an
endeavor that heavily relies on human expertise. Design choices intrinsic to
the architectures, layer combination, and training require extensive
architecture engineering, are heavily dependent on expertize and require heavy
computation for a systematic and exhaustive evaluation. Thus, automating this
process became logical, creating a growing interest in Neural Architecture
Search (NAS) [23]. NAS methods have successfully been applied to a panoply of
problems, ranging from image classification tasks [55, 35, 65, 60, 37, 36],
semantic segmentation [33, 34], object detection [4, 59] and image generation
[19, 17], by consistently improving the search mechanisms [35, 46, 39, 40] and
the performance estimation mechanisms [56, 43, 38, 3]. NAS methods are usually
comprised of three components. The search space defines the possible
operations to be sampled and their connections, thus defining the type of
architecture the search method can generate. The search method is the approach
used to explore the search space and generate architectures. Common NAS
methods use gradient-based, evolution strategies or Bayesian optimization to
perform the search [57, 15, 48]. Finally, the third component is the
performance estimation strategy, which is the method used to evaluate the
generated architectures. In a nutshell, the goal of a NAS method is to, based
on the search method, efficiently search the space of possible networks to
find an optimal architecture for a given problem, with minimal user
interaction [15].

Evaluating and comparing NAS methods is challenging, as different search
spaces and training protocols difficult a fair comparison and hinder a true
evaluation of how well a NAS method behaves. Researches have continuously
focused on analyzing different NAS baselines to propose proper comparison
metrics [32, 16, 64]. Li and Talwaker extensively studied random-search in NAS
and found that it presents a strong baseline for comparison, outperforming
several NAS proposals [31]. Then, Yang et. al. conducted an extensive study to
evaluate the impact of the training protocols on the results of a generated
architecture, showing that well-engineered training protocols usually have a
greater impact on the final performance of a NAS method than the search
strategy, thus suggesting that NAS methods should provide results using the
same protocols to ensure fair comparisons [62]. In [54], the authors analyzed
popular cell-based search spaces. They found that existing search spaces
contain a high degree of redundancy and generated architectures from distinct
NAS methods have similar patterns. Studies to evaluate NAS performance
predictors in terms of their correlation to the final architecture’s
performance, inference and initialization time have also been conducted [44,
58, 45]. However, evaluating different NAS methods is still a challenge due to
the unconstrained search spaces and training protocols. To mitigate the
difficulty in designing and evaluating NAS methods, several benchmarks have
been proposed to smooth the design and evaluation of NAS methods by providing
fixed search spaces and hyper-parameters [42]. Nonetheless, little attention
has been devoted to the analysis of the benchmarks themselves to ensure that
these provide fair and competitive settings that can and should be used as
common ground for NAS methods’ comparisons.

In this work, we set out to evaluate the impact that the operations, their
combination, and their occurrences have in the final validation accuracy of an
architecture in NAS-Bench-201, as well as evaluating the search space
distribution in terms of the architecture’s accuracy and their correlation
between data sets. With this, we extend prior works that looked into proposing
best practices for NAS methods, as well as evaluating search spaces and
benchmarks [32, 62, 54, 42].

The main contributions of this paper are:

  * •

A detailed analysis of the importance of different operations in the final
performance of a generated architecture, as well as their positioning,
combinations, and occurrences. We found that convolutional layers and skip
connections are the most important in all benchmarks evaluated.

  * •

We study the inter-data set ranking of top-scoring architectures, showing that
TransNAS-Bench-201 presents more heterogeneity than NAS-Bench-201, and that in
all data sets, specific operations are present in key edges of the cell.

  * •

We suggest a set of good practices for the design of future benchmarks and
comparisons between NAS methods, when using different NAS benchmarks.

##  II Neural Architecture Search Benchmarks Overview

TABLE I: Overview of NAS benchmarks and categorization based on their
properties. Acronyms used in the table: Image Classification (IC), Computer
Vision (CV), Automatic Speech Recognition (ASR), Natural Language Processing
(NLP).

Benchmark | Size | Search | Type | #Ops | Data | Task | License | Year | Venue
---|---|---|---|---|---|---|---|---|---
Scheme | Tab. | Surr. | sets
NAS-Bench-101 [63] |  $423$k | cell | ✓ |  | $3$ | $1$ | IC | A2.0 | 2019 | ICML
NDS [47] |  $139$M | cell | ✓$\dagger$ |  |  $5$-$8$ | $2$ | IC | MIT | 2019 | ICCV
NAS-Bench-1Shot1 [66] |  $364$k | cell | ✓ |  | $3$ | $1$ | IC | A2.0 | 2020 | ICLR
| NAS-Bench-201 [11]
---
NATS-Bench [10]
|  $15$k
---
$32$k
| cell
---
macro
✓ |  | $5$ | $3$ | IC | MIT |  | 2020
---
2021
| ICML
---
TPAMI
LatBench [14] |  $15$k | cell | ✓ |  | $5$ | $3$ | IC | A2.0 | 2020 | NeurIPS
TransNAS-Bench-101 [13] |  |  $4$k
---
$3$k
| cell
---
macro
✓ |  | $4$ | $7$ | CV | MIT | 2021 | CVPR
NAS-Bench-Macro [50] |  $6$k | macro | ✓ |  | $3$ | $1$ | IC | A2.0 | 2021 | CVPR
NAS-Bench-ASR [41] |  $8$k | cell | ✓ |  | $8$ | $1$ | ASR | A2.0 | 2021 | ICLR
NAS-Bench-111 [61] |  $423$k | cell |  | ✓ | $3$ | $1$ | IC | A2.0 | 2021 | NeurIPS
NAS-Bench-311 [61] | $10^{18}$ | cell |  | ✓ | $8$ | $1$ | IC | A2.0 | 2021 | NeurIPS
NAS-Bench-NLP11 [61] | $10^{53}$ | cell |  | ✓ | $7$ | $1$ | NLP | A2.0 | 2021 | NeurIPS
HW-NAS-Bench-201 [29] |  $15$k | cell | ✓ |  | $3$ | $3$ | IC | MIT | 2021 | ICLR
HW-NAS-Bench-FBNet [29] | $10^{21}$ | chain | ✓ |  | $9$ | $1$ | IC | MIT | 2021 | ICLR
BLOX [2] |  $91$k | macro | ✓ |  | $5$ | $1$ | IC | CC | 2022 | NeurIPS
JAHS-Bench-201 [1] |  $270$k | cell |  | ✓ | $5$ | $3$ | IC | MIT | 2022 | NeurIPS
NAS-Bench-NLP [25] | $10^{53}$ | cell | ✓$\dagger$ |  | $7$ | $1$ | NLP | A2.0 | 2022 |  | IEEE
---
Access
| NAS-Bench-301 [67]
---
(Surr-NAS-Bench-DARTS)
$10^{18}$ | cell |  | ✓ | $8$ | $1$ | IC | A2.0 | 2022 | ICLR
Surr-NAS-Bench-FBNet [67] | $10^{21}$ | chain |  | ✓ | $9$ | $1$ | IC | A2.0 | 2022 | ICLR
NAS-Bench-MR [9] | $10^{23}$ | cell |  | ✓ | - | $9$ | CV | BSD | 2022 | ICLR

$\dagger$ only a small subset of the entire search space was fully trained and
provided as a tabular benchmark.

Evaluating and comparing NAS methods can be a laborious task. Different NAS
methods often employ tailored training protocols, use search spaces with
different operation pools, and report different evaluation metrics, making it
difficult to make optimal comparisons. This often leads to sub-optimal
comparisons and necessitates high computational costs to ensure fair
comparisons in similar settings. To address this issue, several benchmarks
have been proposed to facilitate the design and evaluation of NAS methods.
These benchmarks allow NAS methods to access information about a unified
search space and force fairer comparisons between methods by fixing hyper-
parameters and training protocols [32]. There are two types of benchmarks:
tabular and surrogate. Tabular benchmarks provide precomputed information on
all possible architectures trained in one or more data sets, while surrogate
benchmarks provide a model capable of predicting the performance of an
architecture, along with precomputed information about some of the
architectures. By providing access to information, either through a tabular
setting or a surrogate model, benchmarks can reduce the need for computational
resources, enabling quick and efficient evaluations.

Table I provides a detailed overview of the existing NAS benchmarks. NAS-
Bench-101 [63] was the first tabular benchmark proposed and provides
information about the validation and test accuracy of the 423,624 cell-based
architectures that make up the search space. Each architecture was trained on
CIFAR-10 with different initialization random seeds. NAS-Bench-1Shot1 [66]
leverages the information in NAS-Bench-101 by defining subsets of the search
space with a fixed number of nodes, enabling one-shot NAS methods. The NDS
benchmark [47], on the other hand, focuses on evaluating different randomly
sampled architectures from five search spaces.

Another tabular benchmark, NAS-Bench-201 [11], proposes a smaller search space
but with added information in different data sets. This benchmark includes
15,625 trained architectures on CIFAR-10, CIFAR-100, and ImageNet16-120.
Additionally, LatBench augments NAS-Bench-201 by providing latency information
about all the architectures in six different devices. Similarly, HW-NAS-Bench
[29] proposes two benchmarks: HW-NAS-Bench-201, an extension of NAS-Bench-201,
and HW-NAS-Bench-FBNet, which uses the FBNet search space. The focus of these
benchmarks is to measure and estimate the latency and energy consumption of
all architectures in the initial search spaces in six different devices. The
authors of NAS-Bench-201 also proposed NATS-Bench [10], which provides a macro
search space with tabular information for 32,768 architectures. More, JAHS-
Bench-201 [1] also extends the search space proposed by NAS-Bench-201 to a
joint optimization of the architecture design and hyper-parameters. The
authors designed a combination of the 15,625 architectures with different
hyper-parameters, totaling 270,000 configurations for which 20 performance
metrics in 3 data sets were evaluated, thus providing more than 161M surrogate
data points.

Benchmarks that provide a macro search space enable the evaluation of NAS
methods beyond traditional operation sampling, extending to decisions about
node connections, operation parameters, or the architecture skeleton. One such
benchmark is NAS-Bench-Macro [50], which proposes a search space with 6,561
architectures. The goal of this benchmark is to search for 8 layers with a
pool of 3 possible blocks. Another macro-based benchmark is Blox [2], which
provides tabular information about $91,125$ unique architectures. These
architectures are designed by fixing an outer-skeleton and searching for 3
blocks with diverse connectivities.

The original focus of NAS benchmark was on evaluating architectures for image
classification (IC) tasks. However, recent benchmarks have expanded to allow
the evaluation of NAS methods in terms of generability and transferability to
different tasks. One example is TransNAS-Bench-101, which offers both a cell-
based and macro-based search space, and evaluates architectures over seven
different computer vision (CV) problems, with the aim of providing a framework
for evaluating NAS transferability across tasks [13]. Another example is NAS-
Bench-MR, which offers a surrogate benchmark for four different CV tasks,
including segmentation [8]. To further expand the scope of NAS applications,
NAS-Bench-ASR proposes a cell-based search space with $8,242$ trained
architectures for automatic speech recognition [41], while NAS-Bench-NLP
focuses on proposing a benchmark for natural language processing, providing
tabular information for a subset of the entire search space [25].

Surrogate benchmarks have been proposed to enable the design and evaluation of
larger search spaces by training a set of architectures and then fitting a
surrogate model capable of inferring the performance of newly generated
architectures. Two such benchmarks are NAS-Bench-301 and Surr-NAS-Bench-FBNet,
which fully train a set of architectures and fit surrogate models to estimate
the final performance of the remaining architectures [67]. However, the use of
surrogate benchmarks often only provides the final surrogate performance
estimation, which does not support the development of multi-fidelity NAS
methods. To address this issue, the authors of [61] propose NAS-Bench-x11,
which provides a surrogate method for predicting the full learning curve of an
architecture. By using this method, the authors extend three existing
benchmarks (NAS-Bench-101, NAS-Bench-301, and NAS-Bench-NLP) to include the
full learning curve, enabling the development and evaluation of NAS methods
that rely on the training curve.

NAS benchmark suites aim to unify various benchmarks by providing a common
interface to query them, allowing for easy transferability and evaluation of
NAS methods. The first suite, NAS-Bench-360 [53], introduced 2D and 1D data
sets. NAS-Bench-Suite [42] is a more comprehensive suite, consisting of 28
different tasks. To further improve NAS-Bench-Suite, NAS-Bench-Suite-Zero [26]
provides precomputed information about several zero-cost proxy estimators.
These suite-benchmarks make it easier for researchers to evaluate and compare
NAS methods across different tasks and data sets, ultimately aiding in the
development of more effective and generalizable NAS methods.

In this study, our main objective is to assess the significance of operations
in a search space across various benchmarks. We hypothesize that the benchmark
design plays a crucial role in determining the performance of the architecture
pool, and thus indirectly influences the success of NAS methods that learn the
underlying operation preference. To test this hypothesis, we limit our focus
to tabular benchmarks that provide architecture performance data after being
trained under similar settings. Specifically, we examine the importance of
operations in NAS-Bench-101, NAS-Bench-201, and TransNAS-Bench-101 as they are
arguably the most commonly used tabular benchmarks for evaluating search
methods.

(a) Cell Structure

(b) Outer-skeleton Structure

Figure 1: Representation of the NAS-Bench-201 and TransNAS-Bench-201 cell
structure (a), and the pre-defined outer-skeleton (b). A cell is composed of 4
nodes and 6 edges, where edges perform operations from an input node and add
them to a posterior node. There are 5 possible operations to be used as edges
in NAS-Bench-201 and 4 in TransNAS-Bench-101. A cell is used as the building
block of the outer-skeleton.

##  III Results

###  III-A Search Spaces

NAS-Bench-101 is a tabular benchmark used for evaluating the performance of
different NAS methods. It consists of a cell-based search space comprising
$423,624$ neural networks trained on CIFAR-10 for 108 epochs with 3 different
weight initializations. All the architectures in this benchmark share a common
outer-skeleton, and the differences between them lie in the cells placed in
the outer-skeleton. The cells in NAS-Bench-101 are defined as directed acyclic
graphs with a maximum of 7 nodes and 9 edges and are encoded as a $7\times 7$
upper-triangular matrix. The operation pool consists of three possible layers:
convolutional $1\times 1$ ($C_{1\times 1}$), convolution $3\times 3$
($C_{3\times 3}$), and $3\times 3$ max pooling ($MP_{3\times 3}$).

NAS-Bench-201 is a tabular cell-based search space with 15625 possible
architectures, each composed of a fixed cell-based design with 5 possible
operations: None, Skip Connection ($SC$), $C_{1\times 1}$, $C_{3\times 3}$,
and Average Pooling $3\times 3$ ($AP_{3\times 3}$). The structure of a cell is
represented as a directed acyclic graph with 6 edges and 4 nodes, where edges
represent operations. All architectures share a common outer-skeleton and are
designed by interleaving searched cells with residual down-sampling blocks.
The benchmark provides information about the learning curve and final
performance of the architectures on CIFAR-10, CIFAR-100, and ImageNet16-120,
recorded and provided as a tabular benchmark. Figure 1 depicts the
representation of the cell structure and the outer-skeleton.

TransNAS-Bench-101 benchmark provides the performance of architectures across
seven vision tasks, including classification, regression, pixel-level
prediction, and self-supervised tasks. By having multiple tasks that can be
evaluated using the same input, TransNAS-Bench-101 provides an easy setup to
evaluate the generality and transferability of NAS across different tasks.
There are two types of search spaces in this benchmark: a cell-based search
space containing $4096$ possible cells and a macro skeleton search space based
on residual blocks containing $3256$ architectures. For both search spaces,
the pool of operations has four possible candidates: zeroize, $SC$,
$C_{1\times 1}$, and $C_{3\times 3}$. TransNAS-Bench-101 provides tabular
information about the training and performance of all architectures in the
search space using the same training protocols and hyper-parameters within
each task. In this study, we focus on the cell search space. The
representation of the cell structure is depicted in Figure 1.

Figure 2: Distribution and density of all architectures, based on their
validation accuracy (%) in NAS-Bench-101.

Figure 3: Distribution and density of all architectures, based on their
validation accuracy (%), for all the data sets present in NAS-Bench-201.

Figure 4: Distribution and density of all architectures, based on their
performance on all the data sets present in TransNAS-Bench-101. Metrics were
scaled to positive values to match accuracy ranges.

###  III-B Evaluation

####  III-B1 Performance Distribution

By providing tabular information about trained architectures, NAS benchmarks
allow direct querying of an architecture performance while searching. Given
the information on all architectures, we evaluated the distribution of the
architecture’s performance in NAS-Bench-101, NAS-Bench-201, and TransNAS-
Bench-201. For NAS-Bench-101, Fig. 2 shows a negatively skewed distribution of
architecture performances, with a higher density of architectures closer to
the upper bound of validation accuracy. This indicates that there are many
optimal architectures with small differences, making it challenging to compare
different NAS methods. Similarly, NAS-Bench-201 also shows a negatively skewed
distribution, as seen in Fig. 3. However, the skewed distributions are more
evident on CIFAR data sets, suggesting that ImageNet16-120 is a more
competitive data set with more likelihood of showing more differences between
different NAS methods. Regarding TransNAS-Bench-201, as shown in Figure 4,
there is a higher density of architectures closer to the optimal performance
in all tasks, with semantic segmentation and room layout being the tasks with
the sparsest distribution. This observation suggests that it is important for
NAS methods to indicate the results of Random Search (RS) when comparing on a
specific benchmark to demonstrate how well an RS method can sample the search
space. Additionally, when comparing gains in performance, it is important to
compare with the most optimal architecture in each task to ensure that small
increments in performance are significant if closer to the upper-bound.

Figure 5: Mean accuracy (%) of all the architectures that have at least one
operation of a given type in NAS-Bench-101.

(a) CIFAR-10

(b) CIFAR-100

(c) ImageNet16-120

Figure 6: Mean accuracy (%) of all the architectures that have at least one
operation of a given type in NAS-Bench-201. Architectures with at least 1
convolutional layer show higher mean validation accuracies (%) in all data
sets.

(a) Cls. Object

(b) Cls. Scene

(c) Autoencoding

(d) Surf. Normal

(e) Sem. Segment.

(f) Room Layout

(g) Jigsaw

Figure 7: Mean performance of all the architectures that have at least one
operation of a given type in TransNAS-Bench-101. Architectures with at least 1
convolutional layer show higher mean performance in all data sets except in
Autoencoding, where $SC$ has the best mean performance.

####  III-B2 Operation Presence

Based on the architecture’s distribution, we now focus on evaluating the
impact of each operation on the final architecture accuracy. First, we look at
the mean accuracy (%) of all architectures that contain at least 1 operation
of a given type. Practically, this means that we evaluated all architectures
that contain 1 or more $C_{3\times 3}$ independently of the rest, and so on
and so forth for all operations. What we found, is that on all benchmarks and
all data sets, convolutional layers yield the best mean performances, with the
exception of the TransNAS-Bench-101 Autoencoding task, where skip connections
have the best mean results. Radar graphs are depicted in Figs. 5, 6 and 7 for
NAS-Bench-101, NAS-Bench-201 and TransNas-Bench-101, respectively. We justify
the importance of convolutional layers due to the fact that cells were
carefully designed and have fixed operations that perform pooling and residual
connections, which benefit from receiving rich feature maps that can be
obtained by performing convolutional operations. On the opposite spectrum,
architectures with the operation $None$ have the worst results, which
indicates that all edges and nodes on a small cell are important and benefit
from having an operation that computes features.

####  III-B3 Operation Occurrence

Based on the mean performance of architectures that rely on the presence of
different operations, we now evaluated how architectures behave based on the
presence of different occurrences of a given operation. Practically, in NAS-
Bench-101 we evaluated architectures based on a cell having 0 to 5 occurrences
of a given operation, and from 0 to 6 on NAS-Bench-201 and TransNas-Bench-101,
as it is the maximum number of operations that each cell accommodates. Boxplot
graphs shown in Figs. 8, 9 and 10 depict the minimum, first quartile, median,
third quartile, maximum, and outlier values for all possible occurrences of
all possible operations for NAS-Bench-101, NAS-Bench-201, and TransNAS-
Bench-101 respectively. The results corroborate the hypothesis that
convolutional layers yield better cells, as increasing the number of
occurrences of such operations consistently results in increases in mean
performance. In all benchmarks, mean performances show that $C_{3\times 3}$ is
the optimal layer, followed by $C_{1\times 1}$, and when present, $SC$. More,
increasing occurrences of $None$ or pooling operations have a negative impact
on the architecture’s performance. This negative impact on the performance
further corroborates the hypothesis that the structure of the cells is
dependent on operations that are capable of creating rich features and that
the fixed outer-skeleton and pre-defined operations perform enough data
normalization for the data sets analyzed. These results indicate that NAS
methods that exhaustively search only for convolutional layers will yield
satisfactory results. Therefore, researchers should promote the indication of
the performance of NAS methods over time, and if possible, representations of
the designed cells, therefore promoting discussion if the method is indeed
capable of searching or quickly falls into looking only for convolutional
layers.

Figure 8: Boxplot analysis of the mean accuracy (%) of all architectures based
on the number of occurrences of the different operations on a cell. nas-
bench-101.

Figure 9: Boxplot analysis of the mean accuracy (%) of all architectures based
on the number of occurrences of the different operations on a cell. The top
figure presents results on CIFAR10, the middle one on C100 and the bottom one
on ImageNet16-120.

Figure 10: Boxplot analysis of the mean performance of all architectures based
on the number of occurrences of the different operations on a cell in
TransNAS-Bench-101. Tasks from top to bottom: object classification, scene
classification, autoenconding, surface normal, semantic segmentation, room
layout and jigsaw.

####  III-B4 Operation Positioning

Given that convolutional layers, especially the ones with larger kernel sizes,
lead on average to better results, we now focus on evaluating the performance
of the architectures based on having an operation on specific edge positions.
For this, we show the mean performance and standard deviation of all
architectures that contain an operation on a given edge position $i$, for
$i=1,...,I$, where $I$ is the maximum number of edges in a given benchmark.
The rationale behind this evaluation is to further corroborate that the pre-
defined structure of the cells and the outer-skeletons have a significant
impact on the decisions that a NAS method performs while searching. NAS-
Bench-101 allows operations in 5 out of the 7 positions, where the remaining 2
are reserved for input and output operations. The results in Table II show
that architectures with $C_{3\times 3}$ in any position yield the highest mean
validation accuracies, followed by $C_{1\times 1}$ with an exception for the
second position, where $MP_{3\times 3}$ has a higher mean validation accuracy,
which is justified by the fact that the second position is the first to
receive the input values from previous cells, thus operations that reduce the
complexity of the input feature maps can have a direct impact in the final
performance. Results for NAS-Bench-201, shown in Table III, further promote
that convolutional layers are preferable, yielding the best results in all
possible positions for all data sets. Results also show that $SC$ layers show
improved importance in positions that connect earlier nodes to later nodes in
the cell, which benefit from residual connections. Also, $None$ operations
tend to be better in the middle of the cell, and $AP_{3\times 3}$ stays
consistent throughout. These findings suggest that convolutions are preferable
throughout the cell structure, but other operations, such as $SC$, can have
importance if placed strategically in specific positions of the cell.
Similarly, in all data sets of TransNAS-Bench-101 with exception of
Autoencoding, $C_{3\times 3}$ yields the best results, whereas in
Autoencoding, $SC$ is the best operation to use in all positions (Table IV).
$SC$ and $C_{1\times 1}$ behave similarly in TransNAS-Bench-101, achieving
better results earlier in the cell, whereas $C_{3\times 3}$ consistently
performs throughout all possible positions in the cell. $None$ operation has
the worst mean performances in all data sets, further promoting the idea that
all cell operations have an impact on the final architecture’s performance.
Results in all data sets show that convolutional layers have the most impact
on an architecture’s performance and that in some specific cases, the presence
of a skip connection interleaved with convolutional layers can be fruitful.

####  III-B5 Operation Combination

We further extend the evaluation of the operation’s importance by looking at
two consecutive combinations of operations, as subsequent operations in the
cell’s structure. This analysis evaluates not only the importance of a single
operation, but how they behave if combined. For this, we looked at all
possible combinations of two operations and calculated the mean performance
and standard deviation of all architectures that contain such combinations.
Results for NAS-Bench-101, shown in Table V, further promote that $C_{3\times
3}$ operations yield the best results, even when combined with any other
operation. Furthermore, $C_{1\times 1}$ follows the pattern, by achieving the
next higher mean performances when combined with any convolutional operation,
followed by $MP_{3\times 3}$. In NAS-Bench-201, we see the same behaviour.
Results in Table VI show that combining convolutional layers results, on
average, in the best performant architectures. From the table it is also
possible to see that combining operations with $None$ and $AP_{3\times 3}$
leads to worse performant architectures, and that a $SC$ combined with
convolutional layers, achieves results close to the best combination, further
suggesting that in some specific cases, $SC$ layers associated with
convolutional layers result in good cell designs. Finally, the evaluation of
the operation combinations on TransNAS-Bench-101 is shown in Table VII, where
in all tasks, the combination of two $C_{3\times 3}$ achieves the highest mean
performances, with exception for Autoencoding, where combining two $SC$ has
the best results. From the table, it is possible to see that for all tasks,
with exception for Autoencoding, combining an operation with $None$ always
yields the worst results, whereas combining any operation with $C_{3\times 3}$
consistently achieves better results when compared with combining with other
operation. Furthermore, the results show that in TransNAS-Bench-201,
$C_{1\times 1}$ and $SC$ operations have similar results when combined with
other operations, further suggesting that operations that create rich features
are preferred, but when not present, operations that help reducing the
complexity of the feature maps are good trade-offs, as they achieve good
results by having a significant impact on the flatness of the loss landscape,
as shown in previous works [30]. Results across all benchmarks corroborate the
hypothesis that the operation pool and the structure of the cells (search
space) have a high influence on the quality of the final architectures, which
is shown by the consistent results that convolutional and skip connection
layers have in all three benchmarks.

TABLE II: Mean validation accuracy (%) and standard deviation of all
architectures that contain an operation on a specific edge in NAS-Bench-101.

Operation | Position
---|---
2 | 3 | 4 | 5 | 6
CIFAR-10
$C_{1\times 1}$ | 89.64 $\pm$ 7.67 | 89.99 $\pm$ 6.36 | 90.13 $\pm$ 5.71 | 90.30 $\pm$ 5.00 | 90.58 $\pm$ 4.43
$C_{3\times 3}$ |  90.92 $\pm$ 7.92 |  91.19 $\pm$ 5.63 |  91.27 $\pm$ 4.82 |  91.45 $\pm$ 3.89 |  91.70 $\pm$ 3.13
$MP_{3\times 3}$ | 90.18 $\pm$ 4.96 | 89.56 $\pm$ 8.62 | 89.34 $\pm$ 9.48 | 89.01 $\pm$ 10.22 | 88.22 $\pm$ 10.66

TABLE III: Mean validation accuracy (%) and standard deviation of all
architectures that contain an operation on a specific edge in NAS-Bench-201.

Operation | Position
---|---
1 | 2 | 3 | 4 | 5 | 6
CIFAR-10
$None$ | 79.02 $\pm$ 20.17 | 81.02 $\pm$ 17.68 | 83.40 $\pm$ 13.34 | 75.04 $\pm$ 24.02 | 81.18 $\pm$ 17.68 | 78.99 $\pm$ 20.17
$C_{1\times 1}$ | 86.62 $\pm$ 7.93 | 85.59 $\pm$ 9.93 | 84.51 $\pm$ 11.87 | 87.33 $\pm$ 2.41 | 85.37 $\pm$ 10.12 | 86.34 $\pm$ 8.24
$C_{3\times 3}$ |  87.61 $\pm$ 7.98 |  86.31 $\pm$ 10.00 |  84.93 $\pm$ 11.93 |  88.31 $\pm$ 1.99 |  85.99 $\pm$ 10.09 |  87.15 $\pm$ 8.13
$AP_{3\times 3}$ | 81.32 $\pm$ 11.08 | 81.91 $\pm$ 12.15 | 82.58 $\pm$ 13.38 | 81.46 $\pm$ 8.74 | 82.32 $\pm$ 12.09 | 82.11 $\pm$ 11.08
$SC$ | 83.81 $\pm$ 10.59 | 83.54 $\pm$ 11.85 | 82.95 $\pm$ 13.26 | 85.33 $\pm$ 7.38 | 83.50 $\pm$ 11.85 | 83.77 $\pm$ 10.62
CIFAR-100
$None$ | 56.83 $\pm$ 17.16 | 58.90 $\pm$ 15.43 | 61.05 $\pm$ 12.23 | 54.47 $\pm$ 20.13 | 59.03 $\pm$ 15.41 | 56.80 $\pm$ 17.13
$C_{1\times 1}$ | 64.95 $\pm$ 7.53 | 63.51 $\pm$ 9.47 | 62.37 $\pm$ 11.11 | 65.27 $\pm$ 4.30 | 63.34 $\pm$ 9.74 | 64.56 $\pm$ 8.00
$C_{3\times 3}$ |  66.35 $\pm$ 7.43 |  64.58 $\pm$ 9.48 |  63.03 $\pm$ 11.11 |  66.87 $\pm$ 3.54 |  64.35 $\pm$ 9.57 |  65.85 $\pm$ 7.67
$AP_{3\times 3}$ | 57.81 $\pm$ 11.64 | 58.82 $\pm$ 12.23 | 59.76 $\pm$ 12.97 | 57.96 $\pm$ 10.38 | 59.17 $\pm$ 12.16 | 58.61 $\pm$ 11.63
$SC$ | 60.47 $\pm$ 11.13 | 60.60 $\pm$ 11.91 | 60.20 $\pm$ 12.84 | 61.84 $\pm$ 9.40 | 60.52 $\pm$ 11.89 | 60.58 $\pm$ 11.24
ImageNet16-120
$None$ | 29.77 $\pm$ 11.43 | 31.47 $\pm$ 10.86 | 33.22 $\pm$ 9.43 | 27.42 $\pm$ 12.38 | 31.74 $\pm$ 10.81 | 29.75 $\pm$ 11.38
$C_{1\times 1}$ | 37.32 $\pm$ 6.65 | 35.75 $\pm$ 7.70 | 34.59 $\pm$ 8.87 | 37.83 $\pm$ 4.79 | 35.46 $\pm$ 8.18 | 36.54 $\pm$ 7.43
$C_{3\times 3}$ |  38.29 $\pm$ 6.68 |  36.77 $\pm$ 7.75 |  35.38 $\pm$ 8.98 |  39.19 $\pm$ 2.98 |  36.36 $\pm$ 8.13 |  37.76 $\pm$ 7.31
$AP_{3\times 3}$ | 30.10 $\pm$ 8.97 | 31.26 $\pm$ 9.33 | 32.62 $\pm$ 9.37 | 29.78 $\pm$ 8.93 | 31.82 $\pm$ 9.05 | 31.17 $\pm$ 8.57
$SC$ | 33.77 $\pm$ 8.17 | 33.70 $\pm$ 8.84 | 33.15 $\pm$ 9.25 | 34.74 $\pm$ 7.69 | 33.59 $\pm$ 8.82 | 33.75 $\pm$ 8.30

TABLE IV: Mean performance and standard deviation of all architectures that
contain an operation on a specific edge in TransNAS-Bench-101.

Operation | Position
---|---
1 | 2 | 3 | 4 | 5 | 6
Cls. Object (Acc. (%) $\uparrow$)
$None$ | 36.45 $\pm$ 6.89 | 37.95 $\pm$ 6.67 | 39.07 $\pm$ 6.24 | 37.10 $\pm$ 6.88 | 38.05 $\pm$ 6.66 | 36.50 $\pm$ 6.91
$SC$ | 40.36 $\pm$ 4.87 | 39.95 $\pm$ 5.33 | 39.57 $\pm$ 5.93 | 39.87 $\pm$ 5.36 | 39.67 $\pm$ 5.90 | 40.10 $\pm$ 5.60
$C_{1\times 1}$ | 39.47 $\pm$ 5.99 | 39.17 $\pm$ 6.15 | 39.66 $\pm$ 5.90 | 38.36 $\pm$ 6.38 | 39.66 $\pm$ 5.85 | 40.06 $\pm$ 5.56
$C_{3\times 3}$ |  42.57 $\pm$ 3.73 |  41.77 $\pm$ 4.68 |  40.54 $\pm$ 5.51 |  43.52 $\pm$ 0.64 |  41.46 $\pm$ 4.60 |  42.19 $\pm$ 3.69
Cls. Scene (Acc. (%) $\uparrow$)
$None$ | 38.52 $\pm$ 14.57 | 41.67 $\pm$ 14.12 | 43.84 $\pm$ 12.68 | 40.30 $\pm$ 15.49 | 41.97 $\pm$ 13.81 | 38.69 $\pm$ 14.36
$SC$ | 46.27 $\pm$ 9.93 | 45.43 $\pm$ 10.86 | 44.92 $\pm$ 12.08 | 44.89 $\pm$ 9.79 | 44.74 $\pm$ 11.75 | 45.47 $\pm$ 11.16
$C_{1\times 1}$ | 44.86 $\pm$ 11.85 | 44.33 $\pm$ 12.28 | 45.05 $\pm$ 12.16 | 42.79 $\pm$ 12.83 | 45.06 $\pm$ 11.90 | 45.97 $\pm$ 11.23
$C_{3\times 3}$ |  51.26 $\pm$ 7.54 |  49.49 $\pm$ 9.54 |  47.11 $\pm$ 11.43 |  52.93 $\pm$ 0.84 |  49.14 $\pm$ 9.71 |  50.78 $\pm$ 7.73
Autoencoding (SSIM $\uparrow$)
$None$ | 0.42 $\pm$ 0.14 | 0.43 $\pm$ 0.13 | 0.46 $\pm$ 0.11 | 0.38 $\pm$ 0.15 | 0.43 $\pm$ 0.13 | 0.42 $\pm$ 0.14
$SC$ |  0.49 $\pm$ 0.07 |  0.48 $\pm$ 0.09 | 0.46 $\pm$ 0.10 |  0.53 $\pm$ 0.01 |  0.48 $\pm$ 0.09 |  0.48 $\pm$ 0.08
$C_{1\times 1}$ | 0.46 $\pm$ 0.09 | 0.45 $\pm$ 0.09 | 0.46 $\pm$ 0.10 | 0.44 $\pm$ 0.06 | 0.46 $\pm$ 0.09 | 0.46 $\pm$ 0.08
$C_{3\times 3}$ | 0.47 $\pm$ 0.07 | 0.47 $\pm$ 0.08 | 0.46 $\pm$ 0.10 | 0.48 $\pm$ 0.03 | 0.47 $\pm$ 0.08 | 0.47 $\pm$ 0.07
Surf. Normal (SSIM $\uparrow$)
$None$ | 0.49 $\pm$ 0.07 | 0.50 $\pm$ 0.07 | 0.51 $\pm$ 0.06 | 0.49 $\pm$ 0.08 | 0.50 $\pm$ 0.07 | 0.49 $\pm$ 0.08
$SC$ | 0.53 $\pm$ 0.04 | 0.52 $\pm$ 0.05 | 0.52 $\pm$ 0.06 | 0.53 $\pm$ 0.03 | 0.52 $\pm$ 0.06 | 0.52 $\pm$ 0.05
$C_{1\times 1}$ | 0.52 $\pm$ 0.06 | 0.51 $\pm$ 0.06 | 0.52 $\pm$ 0.06 | 0.51 $\pm$ 0.06 | 0.52 $\pm$ 0.06 | 0.52 $\pm$ 0.05
$C_{3\times 3}$ |  0.55 $\pm$ 0.04 |  0.54 $\pm$ 0.05 |  0.53 $\pm$ 0.06 |  0.55 $\pm$ 0.01 |  0.54 $\pm$ 0.05 |  0.54 $\pm$ 0.04
Sem. Segment. (mIoU $\uparrow$)
$None$ | 17.10 $\pm$ 6.68 | 18.48 $\pm$ 6.44 | 19.36 $\pm$ 5.78 | 17.81 $\pm$ 7.25 | 18.55 $\pm$ 6.38 | 17.12 $\pm$ 6.69
$SC$ | 20.32 $\pm$ 4.08 | 20.00 $\pm$ 4.64 | 19.81 $\pm$ 5.48 | 20.00 $\pm$ 3.31 | 19.67 $\pm$ 5.24 | 20.01 $\pm$ 4.91
$C_{1\times 1}$ | 19.45 $\pm$ 5.52 | 19.36 $\pm$ 5.73 | 19.88 $\pm$ 5.53 | 18.44 $\pm$ 6.16 | 19.90 $\pm$ 5.44 | 20.24 $\pm$ 5.14
$C_{3\times 3}$ |  22.92 $\pm$ 3.57 |  21.95 $\pm$ 4.42 |  20.75 $\pm$ 5.18 |  23.54 $\pm$ 0.88 |  21.67 $\pm$ 4.38 |  22.42 $\pm$ 3.52
Room Layout (L2 Loss $\downarrow$)
$None$ | 0.79 $\pm$ 0.15 | 0.76 $\pm$ 0.14 | 0.74 $\pm$ 0.12 | 0.79 $\pm$ 0.16 | 0.76 $\pm$ 0.14 | 0.79 $\pm$ 0.15
$SC$ | 0.71 $\pm$ 0.10 | 0.72 $\pm$ 0.11 | 0.73 $\pm$ 0.12 | 0.72 $\pm$ 0.10 | 0.72 $\pm$ 0.11 | 0.71 $\pm$ 0.11
$C_{1\times 1}$ | 0.73 $\pm$ 0.11 | 0.73 $\pm$ 0.11 | 0.73 $\pm$ 0.11 | 0.75 $\pm$ 0.10 | 0.73 $\pm$ 0.11 | 0.72 $\pm$ 0.10
$C_{3\times 3}$ |  0.68 $\pm$ 0.08 |  0.69 $\pm$ 0.10 |  0.71 $\pm$ 0.11 |  0.66 $\pm$ 0.02 |  0.70 $\pm$ 0.09 |  0.69 $\pm$ 0.08
Jigsaw (Acc. (%) $\uparrow$)
$None$ | 62.08 $\pm$ 35.94 | 68.13 $\pm$ 33.58 | 74.14 $\pm$ 30.10 | 62.40 $\pm$ 36.16 | 69.08 $\pm$ 33.38 | 62.27 $\pm$ 35.98
$SC$ | 83.16 $\pm$ 18.51 | 80.60 $\pm$ 22.42 | 76.46 $\pm$ 28.29 | 84.10 $\pm$ 11.54 | 77.31 $\pm$ 27.62 | 79.41 $\pm$ 25.60
$C_{1\times 1}$ | 73.84 $\pm$ 29.95 | 73.30 $\pm$ 30.57 | 76.42 $\pm$ 28.18 | 67.65 $\pm$ 34.49 | 77.18 $\pm$ 27.43 | 79.12 $\pm$ 25.49
$C_{3\times 3}$ |  87.21 $\pm$ 17.53 |  84.26 $\pm$ 22.14 |  79.27 $\pm$ 26.43 |  92.14 $\pm$ 1.77 |  82.71 $\pm$ 22.09 |  85.50 $\pm$ 17.72

TABLE V: Mean validation accuracy (%) and standard deviation of all
architectures, evaluated based on the presence of the combination of 2
operations in NAS-Bench-101.

Operation | $Input$ | $C_{1\times 1}$ | $C_{3\times 3}$ | $MP_{3\times 3}$ | $Output$
---|---|---|---|---|---
$Input$ | – | 89.65 $\pm$ 7.60 | 90.92 $\pm$ 7.90 | 90.19 $\pm$ 4.83 | 83.16 $\pm$ 0.00
$C_{1\times 1}$ | – | 89.24 $\pm$ 6.59 | 91.46 $\pm$ 3.77 | 89.29 $\pm$ 7.58 | 90.64 $\pm$ 4.40
$C_{3\times 3}$ | – | 91.48 $\pm$ 3.98 | 91.66 $\pm$ 4.10 | 90.32 $\pm$ 8.36 |  91.83 $\pm$ 3.00
$MP_{3\times 3}$ | – | 90.00 $\pm$ 5.19 | 91.06 $\pm$ 5.47 | 87.67 $\pm$ 12.22 | 88.36 $\pm$ 10.46
$Output$ | – | – | – | – | –

TABLE VI: Mean validation accuracy (%) and standard deviation of all
architectures, evaluated based on the presence of the combination of 2
operations in NAS-Bench-201.

Operation | $None$ | $C_{1\times 1}$ | $C_{3\times 3}$ | $AP_{3\times 3}$ | $SC$
---|---|---|---|---|---
CIFAR-10
$None$ | 70.78 $\pm$ 28.43 | 83.63 $\pm$ 14.53 | 84.65 $\pm$ 14.65 | 79.38 $\pm$ 15.28 | 82.10 $\pm$ 15.05
$C_{1\times 1}$ | 83.73 $\pm$ 14.52 | 87.00 $\pm$ 6.60 | 87.80 $\pm$ 6.59 | 84.77 $\pm$ 7.58 | 86.12 $\pm$ 7.05
$C_{3\times 3}$ | 84.80 $\pm$ 14.66 | 87.84 $\pm$ 6.59 |  88.10 $\pm$ 6.56 | 85.55 $\pm$ 7.65 | 86.87 $\pm$ 7.11
$AP_{3\times 3}$ | 79.10 $\pm$ 15.22 | 84.57 $\pm$ 7.69 | 85.36 $\pm$ 7.67 | 79.31 $\pm$ 12.09 | 81.25 $\pm$ 11.96
$SC$ | 82.11 $\pm$ 15.01 | 86.10 $\pm$ 7.14 | 86.78 $\pm$ 7.19 | 81.38 $\pm$ 11.93 | 82.76 $\pm$ 11.50
CIFAR-100
$None$ | 49.70 $\pm$ 23.23 | 61.65 $\pm$ 12.68 | 63.24 $\pm$ 12.75 | 56.44 $\pm$ 13.81 | 59.26 $\pm$ 13.51
$C_{1\times 1}$ | 61.81 $\pm$ 12.63 | 65.42 $\pm$ 6.45 | 66.62 $\pm$ 6.33 | 62.04 $\pm$ 8.37 | 63.55 $\pm$ 7.75
$C_{3\times 3}$ | 63.42 $\pm$ 12.77 | 66.65 $\pm$ 6.35 |  67.12 $\pm$ 6.25 | 63.25 $\pm$ 8.29 | 64.73 $\pm$ 7.66
$AP_{3\times 3}$ | 56.17 $\pm$ 13.75 | 61.83 $\pm$ 8.49 | 63.03 $\pm$ 8.35 | 55.20 $\pm$ 12.89 | 57.30 $\pm$ 12.87
$SC$ | 59.15 $\pm$ 13.42 | 63.51 $\pm$ 7.84 | 64.66 $\pm$ 7.72 | 57.41 $\pm$ 12.82 | 58.90 $\pm$ 12.50
ImageNet16-120
$None$ | 24.91 $\pm$ 13.43 | 33.68 $\pm$ 9.34 | 34.94 $\pm$ 9.60 | 28.17 $\pm$ 10.10 | 31.91 $\pm$ 9.86
$C_{1\times 1}$ | 33.89 $\pm$ 9.23 | 37.67 $\pm$ 6.10 |  39.11 $\pm$ 5.85 | 34.09 $\pm$ 7.46 | 35.90 $\pm$ 6.78
$C_{3\times 3}$ | 35.38 $\pm$ 9.44 | 39.02 $\pm$ 5.93 | 38.91 $\pm$ 5.78 | 35.25 $\pm$ 7.36 | 37.44 $\pm$ 6.66
$AP_{3\times 3}$ | 27.74 $\pm$ 10.10 | 33.82 $\pm$ 7.69 | 35.04 $\pm$ 7.51 | 28.10 $\pm$ 8.96 | 30.89 $\pm$ 9.02
$SC$ | 31.69 $\pm$ 9.70 | 35.98 $\pm$ 6.95 | 37.46 $\pm$ 6.71 | 31.04 $\pm$ 8.88 | 32.80 $\pm$ 8.44

TABLE VII: Mean performance metric and standard deviation of all
architectures, evaluated based on the presence of the combination of 2
operations in TransNAS-Bench-101.

Operation | $None$ | $SC$ | $C_{1\times 1}$ | $C_{3\times 3}$
---|---|---|---|---
Cls. Object (Acc. (%) $\uparrow$)
$None$ | 34.43 $\pm$ 7.07 | 38.11 $\pm$ 6.23 | 37.42 $\pm$ 6.69 | 40.94 $\pm$ 5.11
$SC$ | 38.05 $\pm$ 6.12 | 39.69 $\pm$ 5.29 | 39.60 $\pm$ 5.51 | 42.20 $\pm$ 4.03
$C_{1\times 1}$ | 37.29 $\pm$ 6.71 | 39.34 $\pm$ 5.80 | 38.72 $\pm$ 6.21 | 41.71 $\pm$ 4.51
$C_{3\times 3}$ | 41.18 $\pm$ 5.09 | 42.20 $\pm$ 4.20 | 41.79 $\pm$ 4.49 |  42.74 $\pm$ 3.05
Cls. Scene (Acc. (%) $\uparrow$)
$None$ | 34.30 $\pm$ 15.69 | 41.74 $\pm$ 12.63 | 40.86 $\pm$ 13.75 | 48.15 $\pm$ 11.15
$SC$ | 41.79 $\pm$ 12.44 | 44.58 $\pm$ 10.25 | 44.60 $\pm$ 10.98 | 50.03 $\pm$ 7.90
$C_{1\times 1}$ | 40.56 $\pm$ 13.72 | 44.12 $\pm$ 11.43 | 43.40 $\pm$ 12.44 | 49.59 $\pm$ 9.11
$C_{3\times 3}$ | 48.52 $\pm$ 10.89 | 49.93 $\pm$ 8.19 | 49.70 $\pm$ 9.15 |  51.79 $\pm$ 6.29
Autoencoding (SSIM $\uparrow$)
$None$ | 0.36 $\pm$ 0.17 | 0.47 $\pm$ 0.11 | 0.43 $\pm$ 0.11 | 0.45 $\pm$ 0.10
$SC$ | 0.47 $\pm$ 0.12 |  0.51 $\pm$ 0.06 | 0.49 $\pm$ 0.07 | 0.50 $\pm$ 0.06
$C_{1\times 1}$ | 0.42 $\pm$ 0.11 | 0.48 $\pm$ 0.08 | 0.45 $\pm$ 0.08 | 0.46 $\pm$ 0.07
$C_{3\times 3}$ | 0.45 $\pm$ 0.10 | 0.50 $\pm$ 0.07 | 0.46 $\pm$ 0.07 | 0.47 $\pm$ 0.06
Surf. Normal (SSIM $\uparrow$)
$None$ | 0.46 $\pm$ 0.09 | 0.51 $\pm$ 0.06 | 0.50 $\pm$ 0.07 | 0.53 $\pm$ 0.06
$SC$ | 0.51 $\pm$ 0.06 | 0.53 $\pm$ 0.04 | 0.52 $\pm$ 0.05 | 0.54 $\pm$ 0.03
$C_{1\times 1}$ | 0.49 $\pm$ 0.07 | 0.53 $\pm$ 0.05 | 0.51 $\pm$ 0.06 | 0.54 $\pm$ 0.05
$C_{3\times 3}$ | 0.53 $\pm$ 0.06 | 0.54 $\pm$ 0.04 | 0.54 $\pm$ 0.04 |  0.55 $\pm$ 0.03
Sem. Segment. (mIoU $\uparrow$)
$None$ | 15.18 $\pm$ 7.36 | 18.55 $\pm$ 5.55 | 17.91 $\pm$ 6.48 | 21.41 $\pm$ 5.09
$SC$ | 18.63 $\pm$ 5.39 | 19.71 $\pm$ 4.03 | 19.58 $\pm$ 4.79 | 21.93 $\pm$ 3.38
$C_{1\times 1}$ | 17.66 $\pm$ 6.50 | 19.36 $\pm$ 5.06 | 18.81 $\pm$ 5.87 | 21.80 $\pm$ 4.33
$C_{3\times 3}$ | 21.60 $\pm$ 5.09 | 21.99 $\pm$ 3.49 | 21.95 $\pm$ 4.36 |  23.12 $\pm$ 2.95
Room Layout (L2 Loss $\downarrow$)
$None$ | 0.84 $\pm$ 0.17 | 0.75 $\pm$ 0.13 | 0.77 $\pm$ 0.13 | 0.71 $\pm$ 0.11
$SC$ | 0.75 $\pm$ 0.13 | 0.72 $\pm$ 0.10 | 0.72 $\pm$ 0.10 | 0.68 $\pm$ 0.08
$C_{1\times 1}$ | 0.77 $\pm$ 0.13 | 0.73 $\pm$ 0.10 | 0.74 $\pm$ 0.11 | 0.69 $\pm$ 0.08
$C_{3\times 3}$ | 0.71 $\pm$ 0.11 | 0.68 $\pm$ 0.08 | 0.69 $\pm$ 0.08 |  0.68 $\pm$ 0.06
Jigsaw (Acc. (%) $\uparrow$)
$None$ | 50.57 $\pm$ 38.75 | 72.82 $\pm$ 29.02 | 65.32 $\pm$ 34.60 | 79.96 $\pm$ 25.76
$SC$ | 73.50 $\pm$ 27.72 | 81.67 $\pm$ 19.14 | 78.76 $\pm$ 24.37 | 87.38 $\pm$ 15.53
$C_{1\times 1}$ | 63.77 $\pm$ 35.21 | 76.62 $\pm$ 26.85 | 70.90 $\pm$ 31.92 | 83.43 $\pm$ 22.59
$C_{3\times 3}$ | 80.98 $\pm$ 25.54 | 87.20 $\pm$ 16.32 | 83.96 $\pm$ 22.50 |  88.34 $\pm$ 14.39

####  III-B6 Cell’s Ranking and Inter-data set Correlation

Lastly, we evaluate cells based on their ranking by sorting them using the
performance on the validation set. The intuition behind looking at the ranking
of the top cells is that it allows a direct analysis of the top cells based on
their operation diversity, and on benchmarks with more than 1 data set or
task, allows an evaluation of how the different data sets correlate between
each other. First, we look at the top 10 performant architectures in NAS-
Bench-101. The cells are shown in Table VIII, where on the left the operations
in each position are shown, and on the right the associated rank and
validation accuracy is depicted. From the table, we clearly see that
convolutional layers are the preferred operation on all positions, with the
optimal cell entirely designed with $C_{3\times 3}$ operations. Overall,
convolutional layers represent 84% of the operations present in the top-10
cells in NAS-Bench-101 (not considering the fixed input and output
operations), thus suggesting that operations other than convolutional ones
have a small impact on the final performance of the cell. More, this indicates
that a NAS method that searches solely for convolutional layers will in fact
achieve high results without requiring exploring the search space for other
operations. By looking at NAS-Bench-201 in Table IX, we see the same
behaviour, where convolutional layers represent 77% of the top cells
considering all three data sets, having a representation of 75% in CIFAR-10,
80% in CIFAR-100 and 77% in ImageNet16-120. This further indicates that NAS-
Bench-201 is heavily dependant on convolutional layers in top-scoring
architectures. More, in all three data sets, it is possible to see patterns of
combination between the possible two convolutional layers ($C_{1\times 1}$ and
$C_{3\times 3}$), as well as skip connections on the 4th position of the cell
structure. These skip connections allow residual connections from the input
node to the output node, which, combined with different position permutations
of the convolutional layers, yield the best results. We note that most of the
best architectures for all three data sets contain convolutional layers as the
first and fifth operations and an $SC$ as the fourth. Hence, one could
envision a NAS procedure where these three operations would be fixed and only
the remaining ones would be the target of the search, thus achieving excellent
results across the three data sets, considering only a subspace related to
half of the original number of operations. Furthermore, the cell’s ranking
between data sets suggests that top-scoring architectures on one data set will
have good results on other data sets if transferred. This is more explicit
between CIFAR-10 and CIFAR-100, as these two data sets have similarities at
the level of the problem itself. To further study this, we evaluated the
inter-data set ranking and cross-correlation between data sets on Figure 11.
On the left (a), we present the top-50 architectures on CIFAR-10 and their
rankings on CIFAR-100 and ImageNet16-120, and on the right (b), the
correlation between data sets in terms of the performance of all
architectures. It is clear that there is a high positive correlation between
all data sets, in particular, between the CIFAR data sets. These results
suggest that NAS-Bench-201 should not be used alone when evaluating the
generability of a NAS method in different data sets. More, when using NAS-
Bench-201, it is paramount to show results on directly searching in each data
set, as ImageNet16-120 has the lowest correlation with other data sets, it
allows a better understanding of how good a NAS method is. As for the position
of the operations in the top cells of TransNAS-Bench-101, we see in Table X
that convolutional layers have a smaller representation across all tasks, with
58% of the possible operations being convolutional layers, 31% $SC$ and 11%
$None$. Looking at specific tasks, semantic segmentation has the highest
percentage of convolutional layers, with 80% and 66% respectively, while
autoencoding, room layout and object classification have the lowest, with 42%,
50% and 51% respectively. More, in those that the percentage of convolutional
layers is lower, $SC$ represents a larger part, with approximately 42% in all
three. As TransNAS-Bench-101 uses the same cell structure as NAS-Bench-201, we
see the same pattern in terms of operation position, where $SC$ is frequently
present in the 4th position to create residual connections between the input
and output nodes, and convolutional layers are the preferred operation in the
first positions. However, the evaluated tasks allow for a more diverse use of
operations, requiring NAS methods to be more general and capable of learning
which patterns are important for each task, which is further justified by the
inter-task ranking, where we see that a good architecture in a specific task
will not guarantee a good result when transferred to other tasks. This is
further studied in Fig. 12, where on the left, the top 50 architectures on
object classification do appear as top architectures in other tasks. More, on
the right figure, the Kendall Tau correlation suggests that the tasks are not
as correlated as NAS-Bench-201, and Autoencoding is the task with the lowest
correlation with any other. By having a lower correlation between tasks,
TransNAS-Bench-101 becomes an interesting benchmark to evaluate the
generability of NAS methods, as transferring architectures searched on a
single task to others does directly translate to good results.

TABLE VIII: Top 10 cells on NAS-Bench-101 based on the validation accuracy
(%).

Operations | CIFAR-10
---|---
1 | 2 | 3 | 4 | 5 | 6 | 7 | Rank | Val. Acc (%)
$Input$ | $C_{3\times 3}$ | $C_{3\times 3}$ | $C_{3\times 3}$ | $C_{3\times 3}$ | $Output$ | - | 1 | 95.18%
$Input$ | $C_{1\times 1}$ | $MP_{3\times 3}$ | $C_{3\times 3}$ | $C_{3\times 3}$ | $C_{1\times 1}$ | $Output$ | 2 | 95.11%
$Input$ | $C_{1\times 1}$ | $C_{3\times 3}$ | $C_{3\times 3}$ | $C_{1\times 1}$ | $Output$ | - | 3 | 95.11%
$Input$ | $C_{3\times 3}$ | $C_{3\times 3}$ | $C_{1\times 1}$ | $C_{3\times 3}$ | $Output$ | - | 4 | 95.07%
$Input$ | $MP_{3\times 3}$ | $C_{3\times 3}$ | $C_{1\times 1}$ | $C_{3\times 3}$ | $C_{1\times 1}$ | $Output$ | 5 | 95.06%
$Input$ | $C_{3\times 3}$ | $MP_{3\times 3}$ | $C_{3\times 3}$ | $C_{1\times 1}$ | $C_{3\times 3}$ | $Output$ | 6 | 95.04%
$Input$ | $C_{3\times 3}$ | $MP_{3\times 3}$ | $C_{3\times 3}$ | $C_{1\times 1}$ | $Output$ | - | 7 | 95.03%
$Input$ | $C_{3\times 3}$ | $MP_{3\times 3}$ | $C_{3\times 3}$ | $C_{3\times 3}$ | $Output$ | - | 8 | 94.99%
$Input$ | $MP_{3\times 3}$ | $C_{3\times 3}$ | $C_{1\times 1}$ | $C_{3\times 3}$ | $Output$ | - | 9 | 94.95%
$Input$ | $MP_{3\times 3}$ | $C_{3\times 3}$ | $C_{1\times 1}$ | $C_{3\times 3}$ | $Output$ | - | 10 | 94.94 %

TABLE IX: Top 10 cells for each one of the data sets on NAS-Bench-201.
Validation accuracy (%) is given only for the top-10 cells in each data set,
while ranking is also presented for the architectures outside the 10 best,
allowing inter-data set comparison.

Operations |  Rank / Val. Acc. ($\%)$
---|---
1 | 2 | 3 | 4 | 5 | 6 | C10 | C100 | IN16-120
$C_{3\times 3}$ | $C_{3\times 3}$ | $C_{3\times 3}$ | $SC$ | $C_{3\times 3}$ | $C_{1\times 1}$ | 1 | 91.61% | 11 |  | 74 |
$AP_{3\times 3}$ | $C_{3\times 3}$ | $None$ | $SC$ | $C_{1\times 1}$ | $C_{3\times 3}$ | 2 | 91.57% | 202 |  | 1083 |
$C_{3\times 3}$ | $C_{3\times 3}$ | $C_{3\times 3}$ | $SC$ | $C_{3\times 3}$ | $C_{3\times 3}$ | 3 | 91.55% | 1 | 73.49% | 12 |
$C_{3\times 3}$ | $None$ | $C_{3\times 3}$ | $SC$ | $C_{1\times 1}$ | $C_{1\times 1}$ | 4 | 91.55% | 52 |  | 87 |
$C_{3\times 3}$ | $C_{1\times 1}$ | $SC$ | $SC$ | $C_{3\times 3}$ | $C_{1\times 1}$ | 5 | 91.54% | 219 |  | 112 |
$C_{3\times 3}$ | $C_{1\times 1}$ | $C_{3\times 3}$ | $SC$ | $C_{3\times 3}$ | $C_{3\times 3}$ | 6 | 91.53% | 3 | 73.13% | 14 |
$C_{3\times 3}$ | $C_{3\times 3}$ | $C_{1\times 1}$ | $SC$ | $C_{3\times 3}$ | $C_{1\times 1}$ | 7 | 91.52% | 31 |  | 20 |
$C_{3\times 3}$ | $C_{3\times 3}$ | $None$ | $SC$ | $C_{3\times 3}$ | $C_{1\times 1}$ | 8 | 91.51% | 38 |  | 188 |
$C_{3\times 3}$ | $C_{3\times 3}$ | $C_{3\times 3}$ | $SC$ | $C_{1\times 1}$ | $C_{3\times 3}$ | 9 | 91.50% | 2 | 73.31% | 66 |
$C_{3\times 3}$ | $C_{1\times 1}$ | $C_{3\times 3}$ | $SC$ | $C_{1\times 1}$ | $C_{3\times 3}$ | 10 | 91.48% | 21 |  | 16 |
$C_{3\times 3}$ | $C_{3\times 3}$ | $None$ | $SC$ | $C_{1\times 1}$ | $C_{3\times 3}$ | 20 |  | 4 | 73.09% | 94 |
$C_{3\times 3}$ | $C_{3\times 3}$ | $SC$ | $SC$ | $C_{3\times 3}$ | $C_{1\times 1}$ | 128 |  | 5 | 73.02% | 253 |
$C_{1\times 1}$ | $C_{3\times 3}$ | $C_{1\times 1}$ | $SC$ | $C_{3\times 3}$ | $C_{3\times 3}$ | 27 |  | 6 | 72.98% | 134 |
$C_{3\times 3}$ | $C_{3\times 3}$ | $C_{1\times 1}$ | $SC$ | $C_{1\times 1}$ | $C_{3\times 3}$ | 31 |  | 7 | 72.96% | 142 |
$C_{3\times 3}$ | $C_{3\times 3}$ | $C_{3\times 3}$ | $SC$ | $C_{1\times 1}$ | $C_{1\times 1}$ | 12 |  | 8 | 72.95% | 18 |
$C_{1\times 1}$ | $C_{3\times 3}$ | $C_{3\times 3}$ | $SC$ | $C_{1\times 1}$ | $C_{3\times 3}$ | 18 |  | 9 | 72.86% | 85 |
$C_{1\times 1}$ | $C_{3\times 3}$ | $C_{3\times 3}$ | $SC$ | $C_{3\times 3}$ | $C_{3\times 3}$ | 23 |  | 10 | 72.77% | 77 |
$C_{3\times 3}$ | $C_{1\times 1}$ | $C_{1\times 1}$ | $SC$ | $C_{3\times 3}$ | $C_{3\times 3}$ | 32 |  | 19 |  | 1 | 46.73%
$C_{3\times 3}$ | $C_{1\times 1}$ | $C_{3\times 3}$ | $SC$ | $C_{3\times 3}$ | $C_{1\times 1}$ | 11 |  | 12 |  | 2 | 46.56%
$C_{3\times 3}$ | $C_{1\times 1}$ | $C_{3\times 3}$ | $C_{1\times 1}$ | $None$ | $C_{3\times 3}$ | 414 |  | 270 |  | 3 | 46.52%
$C_{3\times 3}$ | $None$ | $C_{3\times 3}$ | $SC$ | $C_{3\times 3}$ | $C_{1\times 1}$ | 48 |  | 40 |  | 4 | 46.50%
$C_{1\times 1}$ | $C_{3\times 3}$ | $C_{3\times 3}$ | $SC$ | $C_{3\times 3}$ | $C_{1\times 1}$ | 15 |  | 26 |  | 5 | 46.50%
$C_{3\times 3}$ | $C_{1\times 1}$ | $C_{3\times 3}$ | $SC$ | $C_{1\times 1}$ | $C_{1\times 1}$ | 29 |  | 75 |  | 6 | 46.49%
$C_{3\times 3}$ | $C_{1\times 1}$ | $C_{3\times 3}$ | $C_{1\times 1}$ | $SC$ | $C_{1\times 1}$ | 753 |  | 728 |  | 7 | 46.47%
$C_{1\times 1}$ | $C_{3\times 3}$ | $C_{3\times 3}$ | $SC$ | $C_{1\times 1}$ | $C_{1\times 1}$ | 280 |  | 55 |  | 8 | 46.45%
$C_{3\times 3}$ | $SC$ | $C_{3\times 3}$ | $SC$ | $C_{1\times 1}$ | $C_{3\times 3}$ | 39 |  | 53 |  | 9 | 46.40%
$C_{1\times 1}$ | $C_{1\times 1}$ | $C_{3\times 3}$ | $SC$ | $C_{3\times 3}$ | $C_{3\times 3}$ | 83 |  | 22 |  | 10 | 46.38%

TABLE X: Top 10 cells for each task on NAS-Bench-201. Performance is given
only for the top-10 cells in each task while ranking is also presented for
architectures outside the top 10, promoting inter-task comparison.

Operations | Rank / Performance
---|---
1 | 2 | 3 | 4 | 5 | 6 |  | Cls.
---
Object
| Cls.
---
Scene
Autoencoding |  | Surf.
---
Normal
| Sem.
---
Segment.
| Room
---
Layout
Jigsaw
$C_{3\times 3}$ | $None$ | $C_{3\times 3}$ | $SC$ | $C_{1\times 1}$ | $C_{1\times 1}$ | 1 | 46.32 | 534 |  | 791 |  | 172 |  | 582 |  | 2072 |  | 126 |
$C_{3\times 3}$ | $SC$ | $C_{3\times 3}$ | $SC$ | $SC$ | $C_{3\times 3}$ | 2 | 46.23 | 469 |  | 1124 |  | 495 |  | 1645 |  | 1057 |  | 1460 |
$C_{3\times 3}$ | $SC$ | $SC$ | $C_{1\times 1}$ | $C_{3\times 3}$ | $SC$ | 3 | 46.06 | 920 |  | 116 |  | 310 |  | 1191 |  | 736 |  | 111 |
$C_{3\times 3}$ | $SC$ | $C_{1\times 1}$ | $C_{1\times 1}$ | $None$ | $SC$ | 4 | 45.98 | 369 |  | 1367 |  | 1331 |  | 2153 |  | 316 |  | 477 |
$C_{3\times 3}$ | $SC$ | $SC$ | $SC$ | $C_{3\times 3}$ | $SC$ | 5 | 45.98 | 292 |  | 390 |  | 701 |  | 2259 |  | 1 | 0.59 | 146 |
$SC$ | $C_{1\times 1}$ | $C_{3\times 3}$ | $C_{3\times 3}$ | $SC$ | $C_{3\times 3}$ | 6 | 45.88 | 115 |  | 768 |  | 332 |  | 519 |  | 752 |  | 1567 |
$C_{3\times 3}$ | $SC$ | $C_{3\times 3}$ | $SC$ | $None$ | $SC$ | 7 | 45.86 | 543 |  | 49 |  | 234 |  | 2504 |  | 74 |  | 150 |
$C_{3\times 3}$ | $C_{3\times 3}$ | $C_{1\times 1}$ | $SC$ | $C_{3\times 3}$ | $SC$ | 8 | 45.86 | 331 |  | 85 |  | 437 |  | 553 |  | 748 |  | 534 |
$SC$ | $C_{3\times 3}$ | $None$ | $C_{1\times 1}$ | $SC$ | $C_{3\times 3}$ | 9 | 45.86 | 178 |  | 344 |  | 192 |  | 1442 |  | 1595 |  | 240 |
$C_{3\times 3}$ | $C_{3\times 3}$ | $SC$ | $SC$ | $C_{3\times 3}$ | $SC$ | 10 | 45.81 | 297 |  | 1206 |  | 902 |  | 1536 |  | 115 |  | 778 |
$C_{3\times 3}$ | $C_{3\times 3}$ | $C_{1\times 1}$ | $SC$ | $C_{3\times 3}$ | $C_{1\times 1}$ | 61 |  | 1 | 54.94 | 218 |  | 52 |  | 800 |  | 1647 |  | 423 |
$C_{3\times 3}$ | $SC$ | $C_{3\times 3}$ | $SC$ | $C_{1\times 1}$ | $C_{1\times 1}$ | 148 |  | 2 | 54.93 | 958 |  | 1728 |  | 2269 |  | 13 |  | 974 |
$C_{3\times 3}$ | $C_{3\times 3}$ | $C_{3\times 3}$ | $SC$ | $None$ | $C_{1\times 1}$ | 213 |  | 3 | 54.87 | 565 |  | 307 |  | 1871 |  | 1836 |  | 344 |
$C_{3\times 3}$ | $C_{3\times 3}$ | $C_{3\times 3}$ | $SC$ | $C_{3\times 3}$ | $C_{3\times 3}$ | 572 |  | 4 | 54.87 | 1411 |  | 266 |  | 1274 |  | 2191 |  | 32 |
$C_{3\times 3}$ | $None$ | $SC$ | $C_{1\times 1}$ | $C_{3\times 3}$ | $SC$ | 1129 |  | 5 | 54.86 | 3019 |  | 187 |  | 117 |  | 2214 |  | 790 |
$C_{3\times 3}$ | $C_{1\times 1}$ | $C_{3\times 3}$ | $None$ | $SC$ | $SC$ | 1132 |  | 6 | 54.86 | 3020 |  | 188 |  | 118 |  | 2215 |  | 791 |
$C_{3\times 3}$ | $C_{1\times 1}$ | $C_{3\times 3}$ | $C_{1\times 1}$ | $SC$ | $SC$ | 1490 |  | 7 | 54.85 | 2724 |  | 547 |  | 796 |  | 1772 |  | 153 |
$C_{3\times 3}$ | $SC$ | $C_{1\times 1}$ | $C_{1\times 1}$ | $SC$ | $C_{1\times 1}$ | 93 |  | 8 | 54.80 | 2296 |  | 1834 |  | 229 |  | 1298 |  | 197 |
$C_{3\times 3}$ | $C_{3\times 3}$ | $None$ | $C_{1\times 1}$ | $SC$ | $C_{3\times 3}$ | 928 |  | 9 | 54.79 | 1906 |  | 339 |  | 23 |  | 1058 |  | 1323 |
$C_{3\times 3}$ | $SC$ | $C_{1\times 1}$ | $C_{3\times 3}$ | $C_{3\times 3}$ | $SC$ | 381 |  | 10 | 54.77 | 127 |  | 361 |  | 1287 |  | 310 |  | 123 |
$SC$ | $C_{1\times 1}$ | $None$ | $C_{3\times 3}$ | $SC$ | $None$ | 529 |  | 2545 |  | 1 | 0.58 | 1355 |  | 2367 |  | 995 |  | 1839 |
$SC$ | $C_{1\times 1}$ | $SC$ | $C_{3\times 3}$ | $SC$ | $None$ | 530 |  | 2546 |  | 2 | 0.58 | 1356 |  | 2368 |  | 996 |  | 1840 |
$SC$ | $C_{3\times 3}$ | $C_{1\times 1}$ | $SC$ | $None$ | $SC$ | 1704 |  | 2619 |  | 3 | 0.57 | 1192 |  | 2454 |  | 380 |  | 502 |
$C_{1\times 1}$ | $SC$ | $C_{3\times 3}$ | $C_{3\times 3}$ | $C_{1\times 1}$ | $SC$ | 390 |  | 1208 |  | 4 | 0.57 | 607 |  | 2598 |  | 11 |  | 1119 |
$C_{3\times 3}$ | $SC$ | $SC$ | $SC$ | $None$ | $SC$ | 1042 |  | 2754 |  | 5 | 0.57 | 2432 |  | 2535 |  | 365 |  | 2312 |
$C_{1\times 1}$ | $SC$ | $C_{3\times 3}$ | $SC$ | $C_{3\times 3}$ | $None$ | 2314 |  | 2159 |  | 6 | 0.57 | 2205 |  | 1612 |  | 1161 |  | 836 |
$SC$ | $SC$ | $C_{3\times 3}$ | $SC$ | $None$ | $SC$ | 1445 |  | 2821 |  | 7 | 0.57 | 2549 |  | 2789 |  | 234 |  | 953 |
$C_{3\times 3}$ | $C_{1\times 1}$ | $C_{3\times 3}$ | $SC$ | $C_{1\times 1}$ | $None$ | 313 |  | 348 |  | 8 | 0.57 | 906 |  | 2042 |  | 1241 |  | 30 |
$SC$ | $C_{1\times 1}$ | $C_{3\times 3}$ | $SC$ | $SC$ | $C_{3\times 3}$ | 145 |  | 790 |  | 9 | 0.57 | 1894 |  | 2647 |  | 21 |  | 431 |
$SC$ | $C_{3\times 3}$ | $C_{3\times 3}$ | $None$ | $SC$ | $C_{1\times 1}$ | 468 |  | 404 |  | 10 | 0.57 | 1431 |  | 1781 |  | 1406 |  | 827 |
$C_{3\times 3}$ | $C_{3\times 3}$ | $SC$ | $C_{3\times 3}$ | $C_{3\times 3}$ | $C_{1\times 1}$ | 1011 |  | 203 |  | 2933 |  | 1 | 0.60 | 97 |  | 2643 |  | 1089 |
$C_{3\times 3}$ | $C_{3\times 3}$ | $None$ | $SC$ | $SC$ | $C_{1\times 1}$ | 173 |  | 206 |  | 384 |  | 2 | 0.59 | 685 |  | 242 |  | 503 |
$SC$ | $C_{3\times 3}$ | $None$ | $SC$ | $C_{1\times 1}$ | $C_{3\times 3}$ | 96 |  | 96 |  | 1102 |  | 3 | 0.59 | 1847 |  | 38 |  | 1099 |
$C_{3\times 3}$ | $SC$ | $C_{3\times 3}$ | $C_{1\times 1}$ | $SC$ | $SC$ | 20 |  | 76 |  | 246 |  | 4 | 0.58 | 1539 |  | 364 |  | 871 |
$C_{3\times 3}$ | $C_{1\times 1}$ | $SC$ | $SC$ | $C_{1\times 1}$ | $C_{1\times 1}$ | 598 |  | 766 |  | 141 |  | 5 | 0.58 | 1937 |  | 1069 |  | 625 |
$C_{3\times 3}$ | $SC$ | $C_{3\times 3}$ | $SC$ | $None$ | $C_{1\times 1}$ | 672 |  | 1203 |  | 1352 |  | 6 | 0.58 | 1385 |  | 1221 |  | 275 |
$C_{3\times 3}$ | $C_{1\times 1}$ | $C_{3\times 3}$ | $C_{3\times 3}$ | $C_{1\times 1}$ | $C_{1\times 1}$ | 2488 |  | 51 |  | 2304 |  | 7 | 0.58 | 12 |  | 2368 |  | 1471 |
$SC$ | $C_{3\times 3}$ | $None$ | $SC$ | $None$ | $C_{3\times 3}$ | 663 |  | 15 |  | 364 |  | 8 | 0.58 | 864 |  | 240 |  | 9 | 94.90
$SC$ | $C_{3\times 3}$ | $C_{1\times 1}$ | $C_{3\times 3}$ | $SC$ | $SC$ | 614 |  | 2518 |  | 825 |  | 9 | 0.57 | 629 |  | 610 |  | 1129 |
$C_{3\times 3}$ | $None$ | $C_{3\times 3}$ | $C_{1\times 1}$ | $C_{3\times 3}$ | $SC$ | 953 |  | 785 |  | 2451 |  | 10 | 0.57 | 414 |  | 2456 |  | 2023 |
$C_{3\times 3}$ | $C_{3\times 3}$ | $C_{3\times 3}$ | $C_{1\times 1}$ | $SC$ | $C_{3\times 3}$ | 845 |  | 24 |  | 1782 |  | 101 |  | 1 | 26.27 | 2250 |  | 781 |
$C_{1\times 1}$ | $None$ | $C_{3\times 3}$ | $C_{3\times 3}$ | $C_{3\times 3}$ | $C_{3\times 3}$ | 1448 |  | 1324 |  | 2708 |  | 1187 |  | 2 | 26.10 | 2622 |  | 1910 |
$C_{3\times 3}$ | $None$ | $C_{3\times 3}$ | $SC$ | $C_{1\times 1}$ | $C_{3\times 3}$ | 30 |  | 224 |  | 912 |  | 342 |  | 3 | 25.95 | 2184 |  | 175 |
$C_{3\times 3}$ | $SC$ | $C_{3\times 3}$ | $C_{3\times 3}$ | $C_{3\times 3}$ | $C_{1\times 1}$ | 658 |  | 210 |  | 3139 |  | 44 |  | 4 | 25.91 | 1464 |  | 136 |
$C_{3\times 3}$ | $C_{1\times 1}$ | $C_{3\times 3}$ | $C_{3\times 3}$ | $None$ | $SC$ | 362 |  | 540 |  | 1928 |  | 206 |  | 5 | 25.83 | 1693 |  | 1957 |
$C_{3\times 3}$ | $None$ | $C_{3\times 3}$ | $C_{3\times 3}$ | $C_{1\times 1}$ | $C_{3\times 3}$ | 2081 |  | 788 |  | 2778 |  | 197 |  | 6 | 25.82 | 1923 |  | 2194 |
$C_{3\times 3}$ | $C_{3\times 3}$ | $C_{3\times 3}$ | $C_{1\times 1}$ | $C_{3\times 3}$ | $C_{1\times 1}$ | 1802 |  | 620 |  | 2358 |  | 36 |  | 7 | 25.80 | 2713 |  | 2172 |
$C_{3\times 3}$ | $C_{3\times 3}$ | $C_{3\times 3}$ | $C_{1\times 1}$ | $C_{3\times 3}$ | $SC$ | 703 |  | 507 |  | 2552 |  | 367 |  | 8 | 25.79 | 2617 |  | 1598 |
$C_{3\times 3}$ | $None$ | $C_{3\times 3}$ | $None$ | $C_{1\times 1}$ | $C_{1\times 1}$ | 2848 |  | 2103 |  | 3500 |  | 1208 |  | 9 | 25.78 | 2697 |  | 2822 |
$C_{3\times 3}$ | $C_{3\times 3}$ | $C_{3\times 3}$ | $None$ | $C_{1\times 1}$ | $C_{1\times 1}$ | 2707 |  | 1878 |  | 3260 |  | 971 |  | 10 | 25.76 | 2573 |  | 2821 |
$C_{3\times 3}$ | $C_{1\times 1}$ | $None$ | $SC$ | $C_{3\times 3}$ | $None$ | 344 |  | 59 |  | 510 |  | 252 |  | 1391 |  | 2 | 0.60 | 1263 |
$C_{3\times 3}$ | $C_{1\times 1}$ | $SC$ | $SC$ | $C_{3\times 3}$ | $None$ | 345 |  | 60 |  | 511 |  | 253 |  | 1392 |  | 3 | 0.60 | 1264 |
$SC$ | $C_{3\times 3}$ | $C_{3\times 3}$ | $SC$ | $SC$ | $C_{1\times 1}$ | 241 |  | 1246 |  | 103 |  | 1927 |  | 2559 |  | 4 | 0.60 | 938 |
$SC$ | $C_{3\times 3}$ | $C_{3\times 3}$ | $SC$ | $C_{1\times 1}$ | $SC$ | 2058 |  | 2406 |  | 552 |  | 291 |  | 1383 |  | 5 | 0.60 | 292 |
$C_{3\times 3}$ | $SC$ | $SC$ | $SC$ | $C_{3\times 3}$ | $C_{3\times 3}$ | 461 |  | 70 |  | 1118 |  | 15 |  | 2081 |  | 6 | 0.60 | 684 |
$C_{3\times 3}$ | $C_{1\times 1}$ | $None$ | $C_{1\times 1}$ | $C_{1\times 1}$ | $C_{1\times 1}$ | 862 |  | 1389 |  | 2117 |  | 1659 |  | 370 |  | 7 | 0.60 | 1148 |
$SC$ | $C_{3\times 3}$ | $C_{3\times 3}$ | $SC$ | $SC$ | $C_{3\times 3}$ | 15 |  | 529 |  | 1306 |  | 512 |  | 1805 |  | 8 | 0.60 | 599 |
$C_{3\times 3}$ | $SC$ | $None$ | $SC$ | $C_{3\times 3}$ | $SC$ | 125 |  | 384 |  | 25 |  | 53 |  | 2323 |  | 9 | 0.60 | 346 |
$SC$ | $C_{3\times 3}$ | $SC$ | $SC$ | $C_{3\times 3}$ | $C_{3\times 3}$ | 307 |  | 807 |  | 1072 |  | 54 |  | 2595 |  | 10 | 0.60 | 1709 |
$C_{3\times 3}$ | $C_{3\times 3}$ | $None$ | $SC$ | $C_{3\times 3}$ | $C_{1\times 1}$ | 428 |  | 95 |  | 1241 |  | 186 |  | 613 |  | 887 |  | 1 | 95.37
$C_{3\times 3}$ | $C_{1\times 1}$ | $SC$ | $SC$ | $None$ | $C_{1\times 1}$ | 1431 |  | 1059 |  | 402 |  | 1357 |  | 2361 |  | 144 |  | 2 | 95.22
$C_{3\times 3}$ | $C_{3\times 3}$ | $None$ | $SC$ | $SC$ | $None$ | 1405 |  | 2652 |  | 1309 |  | 1761 |  | 2619 |  | 213 |  | 3 | 95.00
$C_{3\times 3}$ | $C_{3\times 3}$ | $SC$ | $SC$ | $SC$ | $None$ | 1407 |  | 2653 |  | 1310 |  | 1762 |  | 2620 |  | 214 |  | 4 | 95.00
$None$ | $C_{3\times 3}$ | $C_{1\times 1}$ | $C_{1\times 1}$ | $C_{1\times 1}$ | $C_{3\times 3}$ | 2671 |  | 321 |  | 2855 |  | 352 |  | 1529 |  | 2302 |  | 5 | 95.00
$C_{3\times 3}$ | $C_{1\times 1}$ | $C_{1\times 1}$ | $C_{3\times 3}$ | $SC$ | $SC$ | 1275 |  | 361 |  | 1404 |  | 311 |  | 958 |  | 1245 |  | 6 | 94.99
$C_{3\times 3}$ | $None$ | $None$ | $SC$ | $C_{1\times 1}$ | $C_{1\times 1}$ | 328 |  | 1158 |  | 1172 |  | 1681 |  | 2442 |  | 649 |  | 7 | 94.96
$C_{3\times 3}$ | $C_{3\times 3}$ | $C_{1\times 1}$ | $C_{1\times 1}$ | $SC$ | $C_{1\times 1}$ | 2146 |  | 493 |  | 1975 |  | 169 |  | 209 |  | 2041 |  | 8 | 94.91
$C_{1\times 1}$ | $SC$ | $C_{1\times 1}$ | $C_{3\times 3}$ | $SC$ | $SC$ | 922 |  | 2121 |  | 196 |  | 1249 |  | 2145 |  | 421 |  | 10 | 94.88

(a) Top-50 Architectures inter-rank.

(b) Data set Kendall’s tau correlation.

Figure 11: Evaluation of NAS-Bench-201 architectures in terms of their inter-
data set ranking and the correlation between data sets. On (a) we show the
ranking of the top-50 architectures from CIFAR-10 when transferred and
evaluated to CIFAR-100 and ImageNet16-120; and (b) the Kendall’s Tau
correlation between data sets.

(a) Top-50 Architectures inter-rank.

(b) Task Kendall’s tau correlation.

Figure 12: Evaluation of TransNAS-Bench-101 architectures in terms of their
inter-task ranking and task correlation. On (a) we show the ranking of the
top-50 architectures from Object Classification when transferred and evaluated
to other tasks; and (b) the Kendall’s Tau correlation between tasks.

##  IV Suggestions for Future NAS Work

The presented results shed insights on the importance of the different
operations in a NAS search space, how they are influenced by their
positioning, their combinations and how this influences the performance of an
architecture. Based on those, we consider that they have a two-fold
application: 1) on how NAS methods should be evaluated; and 2) what
researchers should take into consideration when designing and creating new NAS
benchmarks.

###  IV-A Evaluating NAS Methods

Our findings suggest that specific operations, such as $C_{3\times 3}$, have a
considerable impact on the final performance of an architecture. To promote a
thorough evaluation of NAS methods, we recommend that a NAS evaluation method
should consider:

  * •

Evaluating NAS methods over time, reporting performance as an average and
standard deviation to allow analysis of precision and trainability of the
method.

  * •

Analyzing the cell’s design to determine if a NAS method focuses only on a
small subset of the operation pool, which can hinder optimal results. If this
is done over time, it can provide information about how well a NAS method
generalizes and learns.

  * •

Evaluating NAS methods on multiple benchmarks, not just a single one,
especially if the analyzed task allows most architectures to attain high
performance, such as the CIFAR-10 data set. This corroborates past suggestions
and findings [32, 62, 54, 42]. This would allow a proper evaluation of the
method’s generability, as it could be easily overfitting a setting from a
small search space on a given benchmark.

  * •

Evaluating NAS methods by both searching directly on all data sets of a
benchmark and also by transferring searched architectures to all other data
sets. This will allow an evaluation of the method’s generability and the
capability of generating architectures in different scenarios and constraints.

###  IV-B Designing NAS benchmarks

Based on the findings presented in this paper, we propose several suggestions
for researchers interested in designing and creating new NAS benchmarks.
Firstly, it is important to note that within the search space of all three
benchmarks evaluated, there are operations that have the most importance –
convolutional layers and $SC$. This suggests that a subset of the operations
pool is sufficient to generate the best scoring architectures. Additionally,
our results show a high correlation between data sets within a benchmark,
hindering the evaluation of NAS methods in terms of generability and
transferability. To address this, researchers should consider the following
when designing NAS benchmarks:

  * •

The cell structure has a high impact on the final architecture performance.
This is seen in NAS-Bench-201 and TransNAS-Bench-101 when $SC$ layers are
often placed in the 4th position. Therefore, researchers should take this into
account when designing their benchmarks by progressively analyzing the
architectures as the benchmark is developed. This can be done by looking at a
small set of architectures and partially training them. In this, looking the
learning curve and zero-proxy estimators can help in obtaining quicker
estimates of how the operations and the cell structure impacts the final
performance.

  * •

Small search spaces could lead to operation overfitting, suggesting that if
possible, authors should create an operation space that contains more
operations than existing ones. This could be manageable by reducing the cell
structure, fixing specific operations, and increasing the operation pool to
include more operations such as activation layers and a wider range of pooling
operations.

  * •

New benchmarks should strive to increase the number of architectures. While a
small number of architectures is required due to the extreme computational
costs of training thousands of architectures, benchmarks could be designed
with a larger search space and similar computational costs by training the
architectures with a smaller number of epochs or simplifying the
architecture’s outer-skeleton to reduce the number of parameters. In the
studied benchmarks, NAS-Bench-101 is the only one with a large set of possible
architectures, with over 423k candidates. In contrast, NAS-Bench-201 only has
15k architectures and TransNAS-Bench-101 cell space has approximately 4k
possible architectures.

  * •

NAS benchmarks should include different types of constraints and information
regarding the architectures, such as latency, power consumption, and model
size, among others. This will allow the evaluation of NAS methods that search
architectures that meet different practical constraints.

  * •

Benchmarks should provide a comprehensive evaluation metric that takes into
consideration both accuracy and practical constraints to evaluate the overall
performance of the searched architectures. This will ensure that the searched
architectures are not only accurate but also practical and feasible for real-
world applications.

Furthermore, as architectures become less dependent on convolutional layers,
their training time will reduce. Convolutional layers are known for their
excellent feature extraction capabilities but come with the cost of more
parameters. To reduce inference time and the number of parameters, researchers
can use convolutional layers that reduce computational costs, such as depth-
wise separable convolutions [5]. These layers have been used extensively in
the DARTS search space [35].

##  V Conclusions

In this work, we present an analysis of three popular NAS benchmarks in terms
of their operations. We found that convolutional layers are essential to have
architectures with high performance, and also found that the distribution of
the accuracy ranges are skewed, suggesting that finding architectures with
accuracies closer to the upper bound is not hard, as several operation
patterns, such as their positioning in a cell and their combinations, tend to
yield top-scoring architectures. From the analysis conducted, it is clear that
specific patterns tend to yield better architectures, such as having specific
operations in specific positions of the cell’s structure. These findings were
also supported by showing that two consecutive combination of operations can
have tremendous impact on the architecture’s performance, suggesting that
convolutional layers and skip connections are more important to generate top-
scoring architectures in the evaluated benchmarks. Moreover, TransNAS-
Bench-101 presents more heterogeneity in terms of the top-scoring
architectures, suggesting that NAS methods should refrain from searching only
on a single data set and then transferring the top-scoring architecture, to
directly search on the three data sets independently. This was further
supported by finding that inter-data set ranking has a high Kendall tau
correlation. Finally, we discuss how NAS methods should take the findings of
this paper into consideration to promote fair comparisons and to design new
benchmarks.

Notwithstanding that the findings of this work show that some operations can
be essential for having good architectures and that there is a high
correlation between operations and the final performance of the architectures,
we believe that benchmarks are a key part of NAS research. In summary, we
believe that the suggestions presented in this paper can guide future research
in NAS, both in terms of evaluating NAS methods and designing new NAS
benchmarks. By considering the importance of different operations in a search
space and their impact on the final performance of the searched architectures,
researchers can better design and evaluate NAS methods and benchmarks, leading
to more efficient and effective NAS approaches.

## References

  * [1] Archit Bansal, Danny Stoll, Maciej Janowski, Arber Zela, and Frank Hutter.  JAHS-bench-201: A foundation for research on joint architecture and hyperparameter search.  In NeurIPS Datasets and Benchmarks Track, 2022.
  * [2] Thomas Chun Pong Chau, Łukasz Dudziak, Hongkai Wen, Nicholas Donald Lane, and Mohamed S Abdelfattah.  BLOX: Macro neural architecture search benchmark and algorithms.  In NeurIPS Datasets and Benchmarks Track, 2022.
  * [3] Wuyang Chen, Xinyu Gong, and Zhangyang Wang.  Neural architecture search on imagenet in four GPU hours: A theoretically inspired perspective.  In ICLR, 2021.
  * [4] Yukang Chen, Tong Yang, Xiangyu Zhang, Gaofeng Meng, Chunhong Pan, and Jian Sun.  Detnas: Neural architecture search on object detection.  arXiv preprint arXiv:1903.10979, 2019.
  * [5] François Chollet.  Xception: Deep learning with depthwise separable convolutions.  In CVPR, 2017.
  * [6] Alexis Conneau, Holger Schwenk, Loïc Barrault, and Yann LeCun.  Very deep convolutional networks for text classification.  In Mirella Lapata, Phil Blunsom, and Alexander Koller, editors, EACL, 2017.
  * [7] Li Deng, Dong Yu, et al.  Deep learning: methods and applications.  Foundations and Trends in Signal Processing, 2014.
  * [8] Mingyu Ding, Yuqi Huo, Haoyu Lu, Linjie Yang, Zhe Wang, Zhiwu Lu, Jingdong Wang, and Ping Luo.  Learning versatile neural architectures by propagating network codes.  CoRR, abs/2103.13253, 2021.
  * [9] Mingyu Ding, Yuqi Huo, Haoyu Lu, Linjie Yang, Zhe Wang, Zhiwu Lu, Jingdong Wang, and Ping Luo.  Learning versatile neural architectures by propagating network codes.  In ICLR, 2022.
  * [10] Xuanyi Dong, Lu Liu, Katarzyna Musial, and Bogdan Gabrys.  NATS-Bench: Benchmarking nas algorithms for architecture topology and size.  TPAMI, 2021.
  * [11] Xuanyi Dong and Yi Yang.  NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search.  In ICLR, 2020.
  * [12] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In ICLR, 2021.
  * [13] Yawen Duan, Xin Chen, Hang Xu, Zewei Chen, Xiaodan Liang, Tong Zhang, and Zhenguo Li.  Transnas-bench-101: Improving transferability and generalizability of cross-task neural architecture search.  In CVPR, 2021.
  * [14] Lukasz Dudziak, Thomas Chau, Mohamed S. Abdelfattah, Royson Lee, Hyeji Kim, and Nicholas D. Lane.  BRP-NAS: prediction-based NAS using gcns.  In NeurIPS, 2020.
  * [15] Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter.  Neural architecture search: A survey.  JMLR, 2019.
  * [16] Thomas Elsken, Benedikt Staffler, Arber Zela, Jan Hendrik Metzen, and Frank Hutter.  Bag of tricks for neural architecture search.  CoRR, abs/2107.03719, 2021.
  * [17] Chen Gao, Yunpeng Chen, Si Liu, Zhenxiong Tan, and Shuicheng Yan.  Adversarialnas: Adversarial neural architecture search for gans.  In CVPR, 2020.
  * [18] Alberto Garcia-Garcia, Sergio Orts-Escolano, Sergiu Oprea, Victor Villena-Martinez, Pablo Martinez-Gonzalez, and Jose Garcia-Rodriguez.  A survey on deep learning techniques for image and video semantic segmentation.  Applied Soft Computing, 70:41–65, 2018.
  * [19] Xinyu Gong, Shiyu Chang, Yifan Jiang, and Zhangyang Wang.  Autogan: Neural architecture search for generative adversarial networks.  In ICCV, 2019.
  * [20] Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio.  Deep learning, volume 1.  MIT press Cambridge, 2016.
  * [21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, 2016.
  * [22] Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger.  Densely connected convolutional networks.  In CVPR, 2017.
  * [23] Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren.  Automated Machine Learning.  Springer, 2019.
  * [24] Asifullah Khan, Anabia Sohail, Umme Zahoora, and Aqsa Saeed Qureshi.  A survey of the recent architectures of deep convolutional neural networks.  Artificial Intelligence Review, 2020.
  * [25] Nikita Klyuchnikov, Ilya Trofimov, Ekaterina Artemova, Mikhail Salnikov, Maxim V. Fedorov, Alexander Filippov, and Evgeny Burnaev.  Nas-bench-nlp: Neural architecture search benchmark for natural language processing.  IEEE Access, 10:45736–45747, 2022.
  * [26] Arjun Krishnakumar, Colin White, Arber Zela, Renbo Tu, Mahmoud Safari, and Frank Hutter.  Nas-bench-suite-zero: Accelerating research on zero cost proxies.  CoRR, 2022.
  * [27] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton.  Imagenet classification with deep convolutional neural networks.  In NeurIPS, 2012.
  * [28] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton.  Deep learning.  Nature, 2015.
  * [29] Chaojian Li, Zhongzhi Yu, Yonggan Fu, Yongan Zhang, Yang Zhao, Haoran You, Qixuan Yu, Yue Wang, Cong Hao, and Yingyan Lin.  Hw-nas-bench: Hardware-aware neural architecture search benchmark.  In ICLR, 2021.
  * [30] Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein.  Visualizing the loss landscape of neural nets.  In NeurIPS, 2018.
  * [31] Liam Li and Ameet Talwalkar.  Random search and reproducibility for neural architecture search.  In UAI, pages 367–377. PMLR, 2020.
  * [32] Marius Lindauer and Frank Hutter.  Best practices for scientific research on neural architecture search.  JMLR, 21(243):1–18, 2020.
  * [33] Chenxi Liu, Liang-Chieh Chen, Florian Schroff, Hartwig Adam, Wei Hua, Alan L Yuille, and Li Fei-Fei.  Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation.  In CVPR, 2019.
  * [34] Chenxi Liu, Piotr Dollár, Kaiming He, Ross B. Girshick, Alan L. Yuille, and Saining Xie.  Are labels necessary for neural architecture search?  In ECCV, 2020.
  * [35] Hanxiao Liu, Karen Simonyan, and Yiming Yang.  DARTS: Differentiable Architecture Search.  In ICLR, 2019.
  * [36] Vasco Lopes and Luís A Alexandre.  Auto-classifier: A robust defect detector based on an automl head.  In ICONIP, 2020.
  * [37] Vasco Lopes and Luís A. Alexandre.  Towards less constrained macro-neural architecture search.  CoRR, abs/2203.05508, 2022.
  * [38] Vasco Lopes, Saeid Alirezazadeh, and Luís A Alexandre.  EPE-NAS: Efficient Performance Estimation Without Training for Neural Architecture Search.  In ICANN, 2021.
  * [39] Vasco Lopes, Fabio Maria Carlucci, Pedro M Esperança, Marco Singh, Victor Gabillon, Antoine Yang, Hang Xu, Zewei Chen, and Jun Wang.  Manas: Multi-agent neural architecture search.  arXiv preprint arXiv:1909.01051, 2023.
  * [40] Vasco Lopes, Miguel Santos, Bruno Degardin, and Luís A. Alexandre.  Efficient guided evolution for neural architecture search.  In GECCO, 2022.
  * [41] Abhinav Mehrotra, Alberto Gil C. P. Ramos, Sourav Bhattacharya, Lukasz Dudziak, Ravichander Vipperla, Thomas C. P. Chau, Mohamed S. Abdelfattah, Samin Ishtiaq, and Nicholas Donald Lane.  Nas-bench-asr: Reproducible neural architecture search for speech recognition.  In ICLR, 2021.
  * [42] Yash Mehta, Colin White, Arber Zela, Arjun Krishnakumar, Guri Zabergja, Shakiba Moradian, Mahmoud Safari, Kaicheng Yu, and Frank Hutter.  NAS-bench-suite: NAS evaluation is (now) surprisingly easy.  In ICLR, 2022.
  * [43] Joseph Mellor, Jack Turner, Amos J. Storkey, and Elliot J. Crowley.  Neural Architecture Search without Training.  In ICML, 2021.
  * [44] Xuefei Ning, Wenshuo Li, Zixuan Zhou, Tianchen Zhao, Yin Zheng, Shuang Liang, Huazhong Yang, and Yu Wang.  A surgery of the neural architecture evaluators.  CoRR, abs/2008.03064, 2020.
  * [45] Xuefei Ning, Changcheng Tang, Wenshuo Li, Zixuan Zhou, Shuang Liang, Huazhong Yang, and Yu Wang.  Evaluating efficient performance estimators of neural architectures.  NeurIPS, 2021.
  * [46] Hieu Pham, Melody Y. Guan, Barret Zoph, Quoc V. Le, and Jeff Dean.  Efficient neural architecture search via parameter sharing.  In Jennifer G. Dy and Andreas Krause, editors, ICML, 2018.
  * [47] Ilija Radosavovic, Justin Johnson, Saining Xie, Wan-Yen Lo, and Piotr Dollár.  On network design spaces for visual recognition.  In ICCV, 2019.
  * [48] Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-Yao Huang, Zhihui Li, Xiaojiang Chen, and Xin Wang.  A comprehensive survey of neural architecture search: Challenges and solutions.  ACM Computing Surveys, 54(4), 2021.
  * [49] Jürgen Schmidhuber.  Deep Learning in Neural Networks: An Overview.  Neural networks, 2015.
  * [50] Xiu Su, Tao Huang, Yanxi Li, Shan You, Fei Wang, Chen Qian, Changshui Zhang, and Chang Xu.  Prioritized architecture sampling with monto-carlo tree search.  In CVPR, 2021.
  * [51] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich.  Going deeper with convolutions.  In CVPR, 2015.
  * [52] Mingxing Tan and Quoc V. Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, ICML, 2019.
  * [53] Renbo Tu, Mikhail Khodak, Nicholas Roberts, and Ameet Talwalkar.  Nas-bench-360: Benchmarking diverse tasks for neural architecture search.  CoRR, abs/2110.05668, 2021.
  * [54] Xingchen Wan, Binxin Ru, Pedro M Esperança, and Zhenguo Li.  On redundancy and diversity in cell-based neural architecture search.  In ICLR, 2022.
  * [55] Chen Wei, Yiping Tang, Chuang Niu Chuang Niu, Haihong Hu, Yue Wang, and Jimin Liang.  Self-supervised representation learning for evolutionary neural architecture search.  IEEE Computational Intelligence Magazine, 16(3):33–49, 2021.
  * [56] Colin White, Willie Neiswanger, and Yash Savani.  BANANAS: bayesian optimization with neural architectures for neural architecture search.  In AAAI, 2021.
  * [57] Colin White, Mahmoud Safari, Rhea Sukthanker, Binxin Ru, Thomas Elsken, Arber Zela, Debadeepta Dey, and Frank Hutter.  Neural architecture search: Insights from 1000 papers.  CoRR, abs/2301.08727, 2023.
  * [58] Colin White, Arber Zela, Binxin Ru, Yang Liu, and Frank Hutter.  How Powerful are Performance Predictors in Neural Architecture Search?  arXiv preprint arXiv:2104.01177, 2021.
  * [59] Song Xue, Hanlin Chen, Chunyu Xie, Baochang Zhang, Xuan Gong, and David Doermann.  Fast and unsupervised neural architecture evolution for visual representation learning.  IEEE Computational Intelligence Magazine, 16(3):22–32, 2021.
  * [60] Yu Xue, Yankang Wang, Jiayu Liang, and Adam Slowik.  A self-adaptive mutation neural architecture search algorithm based on blocks.  IEEE Computational Intelligence Magazine, 16(3):67–78, 2021.
  * [61] Shen Yan, Colin White, Yash Savani, and Frank Hutter.  Nas-bench-x11 and the power of learning curves.  NeurIPS, 34, 2021.
  * [62] Antoine Yang, Pedro M. Esperança, and Fabio M. Carlucci.  Nas evaluation is frustratingly hard.  In ICLR, 2020.
  * [63] Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter.  NAS-Bench-101: Towards Reproducible Neural Architecture Search.  In ICML, 2019.
  * [64] Kaicheng Yu, Christian Sciuto, Martin Jaggi, Claudiu Musat, and Mathieu Salzmann.  Evaluating the search phase of neural architecture search.  In ICLR, 2020.
  * [65] Arber Zela, Thomas Elsken, Tonmoy Saikia, Yassine Marrakchi, Thomas Brox, and Frank Hutter.  Understanding and robustifying differentiable architecture search.  In ICLR, 2020.
  * [66] Arber Zela, Julien Siems, and Frank Hutter.  Nas-bench-1shot1: Benchmarking and dissecting one-shot neural architecture search.  In ICLR, 2020.
  * [67] Arber Zela, Julien Niklas Siems, Lucas Zimmer, Jovita Lukasik, Margret Keuper, and Frank Hutter.  Surrogate NAS benchmarks: Going beyond the limited search spaces of tabular NAS benchmarks.  In ICLR, 2022.


defined as in (4.54)-(4.60). Instead, the term $\mathcal{I}_{3}^{w}$ is given
by

$\mathcal{I}^{w}_{3}=-\frac{3}{4}\frac{\partial_{t}w}{w}|Z_{3}^{w}|^{2}+2\nu
M^{2}\frac{k^{2}}{p}|Z_{3}^{w}|^{2}.$ (4.115)

In particular, with respect to (4.57), there is the great advantage that
$\partial_{t}w/w\geq 0$, meaning that we can bound $\mathcal{I}^{w}_{3}$ just
with the last term in the right-hand side of (4.115). Therefore, thanks to
(4.115), by making the same estimates given in (4.63)-(4.66) and (4.69)-(4.73)
we infer

$\displaystyle\frac{\mathrm{d}}{\mathrm{d}t}E^{w}(t)\leq$
$\displaystyle-\left(\frac{1}{16}(\frac{\partial_{t}m}{m}+\mu
p)+\frac{3}{4}(\frac{\partial_{t}w}{w}-\frac{\partial_{t}p}{p})\right)|Z^{w}_{2}|^{2}$
(4.116)
$\displaystyle-\left(\frac{1}{16}\left(\frac{\partial_{t}m}{m}+\nu^{\frac{1}{3}}\right)+\frac{3}{4}(\frac{\partial_{t}w}{w}-\frac{\partial_{t}p}{p})\right)|Z_{1}^{w}|^{2}$
(4.117)
$\displaystyle-\frac{\nu^{\frac{1}{3}}}{8}\frac{M^{2}(\partial_{t}p)^{2}}{p^{3}}|Z_{1}^{w}|^{2}$
(4.118) $\displaystyle-\frac{1}{16}\left(\frac{\partial_{t}m}{m}+\nu
p\right)|Z^{w}_{3}|^{2}+4\left(C_{M}\frac{k^{2}}{p}+\frac{1}{2}\frac{\partial_{t}m}{m}\right)E^{w}(t),$
(4.119)

where to obtain (4.118) we have used (4.76), whereas to get the last term in
(4.119) we have used the coercivity properties of the functional, see
(4.24)-(4.25). We now have to exploit the properties (4.102)-(4.103). In
particular, in our case we have $\delta_{\beta}=1/12$, hence choosing
$\beta>4$ in the definition of $w$, see (4.99), we get

$\displaystyle\frac{\mathrm{d}}{\mathrm{d}t}E^{w}(t)\leq$
$\displaystyle-\frac{1}{16}\nu^{\frac{1}{3}}|Z^{w}_{2}|^{2}-\frac{1}{32}\nu^{\frac{1}{3}}|Z_{1}^{w}|^{2}-\frac{\nu^{\frac{1}{3}}}{8}\frac{M^{2}(\partial_{t}p)^{2}}{p^{3}}|Z_{1}^{w}|^{2}$
$\displaystyle-\frac{1}{16}\nu^{\frac{1}{3}}|Z^{w}_{3}|^{2}+4\left(C_{M}\frac{k^{2}}{p}+\frac{1}{2}\frac{\partial_{t}m}{m}\right)E^{w}(t).$
$\displaystyle\leq$
$\displaystyle-\frac{1}{32}\nu^{\frac{1}{3}}E^{w}(t)+4\left(C_{M}\frac{k^{2}}{p}+\frac{1}{2}\frac{\partial_{t}m}{m}\right)E^{w}(t),$

where in the last line we have used (4.25). Therefore, by applying Grönwall’s
Lemma we obtain

$E^{w}(t)\lesssim e^{-\frac{1}{32}\nu^{\frac{1}{3}}t}E^{w}(0).$ (4.120)

Then, in view of the definition (4.108) and the coercivity of the functional,
we also know that

$\displaystyle\sum_{k}\int E^{w}(t)d\eta\approx$
$\displaystyle\frac{1}{M}\left\lVert(w^{-\frac{3}{4}}p^{\frac{1}{2}}{\widehat{R}})(t)\right\rVert_{H^{s}}^{2}+\left\lVert(w^{-\frac{3}{4}}{\widehat{A}})(t)\right\rVert_{H^{s}}^{2}$
$\displaystyle+\left\lVert(w^{-\frac{3}{4}}(\widehat{\Xi}-\nu
M^{2}\widehat{A}))(t)\right\rVert_{H^{s}}^{2},$

hence, thanks to (4.120), the proof is over. ∎

### Acknowledgements

We would like to thank J. Bedrossian, M. Coti Zelati and T. Gallay for useful
comments and suggestions about this work.

PA, MD were partially supported through the INdAM-GNAMPA project “Esistenza,
limiti singolari e comportamento asintotico per equazioni
Eulero/Navier–Stokes–Korteweg”. MD was partially supported by the Royal
Society through the (URF/R1/191492). PM acknowledges partial support by the
PRIN-MIUR project 2015YCJY3A_003 “Hyperbolic Systems of Conservation Laws and
Fluid Dynamics: Analysis and Applications”.

## References