bar incomplete/.append style=fill=red!40, group/.append style=draw=black,
fill=red,

# ICDARTS: Improving the Stability and Performance of Cyclic DARTS

Emily Herron<EMAIL_ADDRESS>
Bredesen Center for Interdisciplinary
Research and Graduate Education
University of Tennessee
Knoville, TN 37996-3394, USA Derek Rose<EMAIL_ADDRESS>
Electrification and Energy Infrastructure Division
Oak Ridge National Laboratory
Oak Ridge, TN 37831-6075, USA Steven Young<EMAIL_ADDRESS>
Computer Science and Mathematics Division
Oak Ridge National Laboratory
Oak Ridge, TN 37831-6164, USA

###### Abstract

NOTE: This is an expanded version of a previously published conference paper
(Herron et al., 2022). This paper includes an expanded study of the importance
of each algorithm change, an ablation study of the importance of each layer
choice, a study of the effect of different layer choices, and a study of
performing ICDARTS NAS on a dynamic search space.

This work introduces improvements to the stability and generalizability of
Cyclic DARTS (CDARTS). CDARTS is a Differentiable Architecture Search
(DARTS)-based approach to neural architecture search (NAS) that uses a cyclic
feedback mechanism to train search and evaluation networks concurrently. This
training protocol aims to optimize the search process by enforcing that the
search and evaluation networks produce similar outputs. However, CDARTS
introduces a loss function for the evaluation network that is dependent on the
search network. The dissimilarity between the loss functions used by the
evaluation networks during the search and retraining phases results in a
search-phase evaluation network that is a sub-optimal proxy for the final
evaluation network that is utilized during retraining. We present ICDARTS, a
revised approach that eliminates the dependency of the evaluation network
weights upon those of the search network, along with a modified process for
discretizing the search network’s zero operations that allows these operations
to be retained in the final evaluation networks. We pair the results of these
changes with ablation studies on ICDARTS’ algorithm and network template.
Finally, we explore methods for expanding the search space of ICDARTS by
expanding its operation set and exploring alternate methods for discretizing
its continuous search cells. These experiments resulted in networks with
improved generalizability and the implementation of a novel method for
incorporating a dynamic search space into ICDARTS.

111Notice: This manuscript has been authored by UT-Battelle, LLC, under
contract DE-AC05-00OR22725 with the US Department of Energy (DOE). The US
government retains and the publisher, by accepting the article for
publication, acknowledges that the US government retains a nonexclusive, paid-
up, irrevocable, worldwide license to publish or reproduce the published form
of this manuscript, or allow others to do so, for US government purposes. DOE
will provide public access to these results of federally sponsored research in
accordance with the DOE Public Access Plan (https://www.energy.gov/doe-public-
access-plan).

Keywords: Neural Architecture Search, AutoML, Deep Learning, Neural Networks

##  1 Introduction

In deep learning, deep neural networks are applied to a range of vision tasks,
including image recognition (Tan and Le, 2021), object detection (Redmon and
Farhadi, 2017; He et al., 2017; Wang et al., 2020), and semantic segmentation
(Yu et al., 2018; Nekrasov et al., 2019). When encountering a new task,
machine learning researchers often deploy a state-of-the-art deep learning
architecture designed for and evaluated on a limited number of popular
benchmark datasets. Since image datasets can vary widely in size, resolution,
and subject matter, a single state-of-the-art network architecture that
generalizes well to one task may perform poorly on a different dataset. NAS
algorithms strive to design optimal network architectures for new tasks and
datasets efficiently. Some popular neural architecture search algorithms rely
on reinforcement learning (Zoph and Le, 2017; Zoph et al., 2018), evolutionary
algorithms (Xie and Yuille, 2017; Real et al., 2019; Elsken et al., 2018a;
Yang et al., 2020), and gradient optimization (Kandasamy et al., 2018; Liu et
al., 2018b). One gradient-based algorithm that has become prominent in NAS is
DARTS (Liu et al., 2019). In a departure from previous methods that rely on
discrete search spaces, DARTS represents its search space as a set of
continuous and differentiable directed acyclic graph structures called cells.
Architectures are learned in the algorithm’s initial search phase by
optimizing the continuous connections between cell nodes and their associated
operations using a gradient-based approach. Then, in a separate evaluation
phase, the cells are discretized by removing all but the most optimal
connections at each depth. A final, deeper architecture is finally constructed
from these discretized cells and evaluated on the full dataset.

The popularity of the original DARTS paper inspired multiple follow-up
publications. One presents the Progressive DARTS algorithm (P-DARTS) (Chen et
al., 2019), which addresses the discrepancy between the networks’ depths
during DARTS’ search and evaluation phases by gradually increasing the depth
of the network during the search phase. Another publication introduces the
CDARTS (Yu and Peng, 2020) method, which builds upon P-DARTS by proposing a
cyclic feedback mechanism between the search and evaluation networks that
allows the networks to be trained jointly in a cyclic manner. This joint
learning strategy ensures that the search and evaluation networks are each
optimized while learning similar features and producing similar results.

Although CDARTS begins to address the issue of DARTS’ continuous search
network by imperfectly predicting the performance of the discretized network,
we find that its current method for training its evaluation network in the
search phase is inconsistent with the method that was used for retraining this
network in the evaluation phase. This discrepancy decreases the value of this
network as a proxy for the final discretized network. We contribute
improvements to the algorithm and search space of CDARTS that result in
improved stability and generalization ability. Our enhanced version of the
CDARTS algorithm, ICDARTS, presents a training procedure for the evaluation
network at search time that better resembles its retraining process. We pair
the results of these improvements with two ablation studies: the first on the
elements of the CDARTS template, including the layer type options, reduce
cells, and stemming layers, and the second on two routes of incremental
changes that transform the CDARTS algorithm to ICDARTS. In addition to our
algorithmic changes, we found that modifying and expanding the default search
space of ICDARTS further improved the algorithm’s stability and generalization
ability. The first of these changes involved eliminating the zero operation
from the operation of ICDARTS since this operation is removed upon
discretization in the original version of CDARTS and is thus irrelevant to the
final network. Next, we explored alternative sets of operation choices of
varying complexity for ICDARTS. These experiments culminated in developing an
efficient, tournament-style method for incorporating dynamic search spaces
into ICDARTS. The final set of improvements to the search space consisted of
new cell discretization protocols that expanded the space of possible cells
that could be discovered by ICDARTS and allowed fairer comparisons between
incoming edges.

InputNormal CellNormal CellReduction CellNormal CellNormal CellReduction
CellNormal CellNormal CellOutputJoint LossptptptptInputNormal CellNormal
CellNormal CellNormal CellNormal CellNormal CellReduction CellNormal
CellNormal CellNormal CellNormal CellNormal CellNormal CellReduction
CellNormal CellNormal CellNormal CellNormal CellNormal CellNormal
CellOutputptptptptptpt$n_{0}$pt$n_{1}$pt$n_{2}$pt$n_{3}$ptpt$n_{0}$pt$n_{1}$pt$n_{2}$pt$n_{3}$ptSearch
NetworkptEvaluationNetworkptContinuousCellptDiscetizedCellDiscretizeNetwork

Figure 1: Overview of the CDARTS NAS algorithm, including search (left) and
evaluation (right) networks with examples of continuous and discretized cells.

##  2 Related Work

Early automatic neural architecture search algorithms achieved remarkable
performance on various image processing benchmarks (Elsken et al., 2019).
These methods’ search strategies relied on various techniques, including
reinforcement learning (Zoph and Le, 2017; Zoph et al., 2018) and evolutionary
algorithms (Xie and Yuille, 2017; Real et al., 2019; Elsken et al., 2018a;
Yang et al., 2020) to discover optimal network architectures given a large
search space. Although these algorithms perform well, they involve evaluating
many candidate architectures and can be computationally expensive. Weight
sharing one-shot methods have been proposed as one solution to this issue
(Brock et al., 2018; Li and Talwalkar, 2020; Pham et al., 2018). These methods
involve training one model, typically represented as one over-parameterized
network graph, and iteratively sampling child models, or paths within the
network graph, that share weights throughout the search process. Following
training, the optimal model is selected by ranking the performance of the
child models (Cai et al., 2019a; Guo et al., 2020). By harnessing the weight-
sharing mechanism, one-shot family models can evaluate high-capacity
architectures within a few GPU days. The DARTS algorithm introduced a new
variant of the weight-sharing model. Rather than searching over discrete
architectures within the main search graph, DARTS searches over a continuous
space of architectures by using stochastic gradient descent to optimize a
mixture of weights corresponding to each edge of the main search graph. Then,
in a separate retraining phase, the continuous architectures are ”discretized”
by selecting the set of k edges at each node corresponding to the weights with
the highest probabilities. By applying this approach, DARTS finds high-quality
architectures with reduced computational cost (Liu et al., 2019; Elsken et
al., 2018b).

The DARTS algorithm has its shortcomings. One is that the different search and
retraining phases may result in two independent networks with limited
correlation (Cai et al., 2019b; Yu and Peng, 2020). One follow-up publication
(Xie et al., 2019), argues that the inconsistency between the search and
evaluation networks results from removing operations between the search and
evaluation networks that adds bias to the loss function and results in the
need to retrain the network. The authors’ solution is Stochastic Neural
Architecture Search (SNAS), a framework that trains network and architecture
weights on the same round of backpropagation. As a part of this process, one-
hot random variables are used as masks to select operations in the search
graph. The resulting method has improved stability and efficiency. The
ProxylessNAS algorithm (Cai et al., 2019b) was also designed to address this
discrepancy by searching for a final, deep target network instead of a
shallow, intermediate (or ”proxy”) network. This model minimizes computational
expense by binarizing its architecture parameters so that only a single path
is active at a time. Another approach, Progressive DARTS (P-DARTS) (Chen et
al., 2019, 2021), addresses the disparity between the networks of the search
and retraining phases by progressively increasing the depth of the search
network until it reaches that of the evaluation network. In addition, this
method further avoids high computational expense by gradually dropping lower-
scoring candidate operations as the search process progresses.

Cyclic DARTS (CDARTS) (Yu and Peng, 2020) addresses the discrepancy between
the over-parameterized search network and the discretized retraining network
by introducing a cyclic feedback mechanism into the search phase. In each
iteration of its search phase, its search network and an intermediate
evaluation network are optimized using a joint learning process. The
evaluation network is generated at the beginning of each search iteration by
discretizing the search network given its connection weights at that stage in
the search process. By providing feedback to the search network cyclically,
the evaluation network serves as an effective proxy for the final network
architecture.

Our approach introduces critical improvements to the original CDARTS
algorithm, which further minimizes discrepancies between the search and
evaluation networks by modifying the loss functions of each. We first correct
the inconsistency between the evaluation network loss functions in the search
and retraining phases by eliminating the feature distillation loss term, which
measures the distance between the logits of the search and evaluation
networks, from the search phase loss function. This change removes the
dependency of the search phase evaluation network’s weights on the search
network’s loss and results in an evaluation network loss function that is
consistent across the search and retraining phases.

Next, we modified CDARTS to optimize its search and evaluation network weights
on the same dataset. In the original version of CDARTS, these weights are
trained on separate splits of the training dataset, which introduces potential
bias in the joint learning phase.

Finally, we incorporate the feature distillation loss term into the search
network’s loss function so that this network’s weights are updated based on
feedback supplied by the evaluation network rather than the other way around.
The culmination of these changes is an updated version of the CDARTS algorithm
that improves upon the stability of the original algorithm by incorporating
modifications that ensure the search and evaluation networks more closely
resemble each other.

##  3 Methods

This section details our improvements to the CDARTS algorithm. We first
describe the original DARTS and CDARTS architectures, from the directed
acyclic graph structures representing their cell motifs to the network
templates that dictate how they are arranged within the search and evaluation
networks. We then summarize the CDARTS algorithm’s objective and loss
functions. The remainder of this section introduces improvements to the CDARTS
algorithm’s joint optimization process that result in processes for training
the search and evaluation networks that more closely resemble each other. We
propose alternatives to the $zero$ layer option from the DARTS (and CDARTS)
search space alongside these algorithmic changes. Then, we propose ablation
studies on the ICDARTS algorithm and network structure. Afterward, we propose
methods for expanding the search space of ICDARTS, including exploring
alternate operation spaces and higher capacity cell discretization approaches.
With these methods, we propose a novel algorithm for incorporating a dynamic
search space into ICDARTS.

###  3.1 CDARTS

In this subsection, we will review the CDARTS algorithm as presented in (Xu et
al., 2020).

####  3.1.1 CDARTS Architecture

Figure 1 gives an overview of the CDARTS algorithm, illustrates the search and
evaluation network templates, and shows examples of continuous and discretized
cell structures. In both DARTS and CDARTS, the architectures of the search and
evaluation networks are composed of cell motifs represented by directed
acyclic graphs (DAG), which serve as the building blocks of both networks. As
discussed in (Liu et al., 2018a) and (Liu et al., 2019), each cell graph
consists of $N$ nodes, each of which denote some feature representation. Each
directed edge $i,j$ within the graph represents a particular operation,
$o^{(i,j)}$, that is applied to a node, $x_{i}$, to produce $x_{j}$. The
operation space of DARTS consists of operations with (e.g. convolution) and
without (e.g. maximum and average pooling operations, skip connections, and
$zero$ operations) learned weights.  DARTS achieves a continuous search space
by weighting the candidate operations at each edge by their corresponding
$\alpha$ values:

$\displaystyle\bar{o}^{(i,j)}(x_{i})=\sum_{o\in
O}\frac{\exp(\alpha_{o}^{(i,j)})}{\sum_{o^{\prime}\in
O}\exp(\alpha_{o^{\prime}}^{(i,j)})}o(x_{i})$ (1)

These $\alpha^{(i,j)}$ values consist of vectors of size $\lvert O\rvert$ that
parameterize the candidate operation strength for each edge in the graph. Each
cell accepts inputs from two previous cells, $c_{k-2}$ and $c_{k-1}$. Each
node’s output is computed by taking the weighted sum of the outputs of the
preceding nodes and outputs of the two previous cells:
$x_{j}=\sum\hat{o}^{(i,j)(x_{i})}$. The outputs of each node in a cell are
concatenated to produce the final output of the cell. Two types of cells are
optimized during the search phase: normal and reduction. The normal cells
produce outputs with the same spatial dimensions as their inputs. In contrast,
the reduction cells produce outputs with spatial dimensions that have been
reduced by a factor of two by applying a stride of 2 two in its operations
(Zoph et al., 2018).

To construct the evaluation networks, the continuous search cells are
discretized by removing all but the edges with top $k=2$ alpha values for each
node, such that $o^{(i,j)}=\operatorname*{arg\,max}_{o\in
O}\alpha_{o}^{(i,j)}$. These cells are then stacked to form a deeper network
specified by a template (again, see Figure 1) (Zoph et al., 2018; Real et al.,
2019; Liu et al., 2018a).

In addition to the network structure pictured in Figure 1, both networks
include an auxiliary head structure, which combines the output of the final
normal cell with that of both reduction cells to get the network’s final
output (Yu and Peng, 2020).

####  3.1.2 CDARTS Algorithm

As discussed in (Liu et al., 2019), the DARTS algorithm searches optimal cell
structures for the full evaluation network by using stochastic gradient
descent to optimize connection weights within the continuous cells that form
the search network. The objective of CDARTS’ search phase is to identify the
connection weights, $\alpha$, and search network weights, $w_{s}$ that satisfy
the following bilevel optimization problem:

$\displaystyle\min_{\alpha}L_{val}(w^{*}_{s},\alpha)$ (2) $\displaystyle
s.t.\;w^{*}_{s}(\alpha)=\operatorname*{arg\,min}_{w_{s}}L_{train}(w_{s},\alpha)$

The connection weights, $\alpha$, are optimized given a validation dataset,
but the search network weights, $w_{s}$, are learned given a separate training
dataset. Afterward, a retraining phase takes place, in which a deeper
evaluation network is generated by discretizing the cells discovered in the
search phase. The network is retrained from scratch and evaluated on a new
test dataset (Liu et al., 2019; Yu and Peng, 2020)

CDARTS expands upon DARTS by introducing a cyclic feedback mechanism between
the search network and an intermediate version of its evaluation network
during its search phase. The intermediate evaluation network is generated at
the beginning of each iteration of the algorithm and is intended to resemble
the final evaluation network of the retraining phase. This innovation allows
for the search and evaluation networks to be optimized jointly and enables
them to better mirror each other in terms of performance and learned features.
The following equation represents the joint optimization process, given the
search and intermediate evaluation network weights, $w_{S}$ and $w_{E}$ and
connection weights, $\alpha$:

$\displaystyle\min_{\alpha}L_{val}(w_{E}^{*},w_{S}^{*},\alpha)$ (3)
$\displaystyle s.t.\leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\
w^{*}_{S}=\operatorname*{arg\,min}_{w_{S}}L_{train}(w_{S},\alpha)$
$\displaystyle
w^{*}_{E}=\operatorname*{arg\,min}_{w_{E}}L_{train}(w_{E},\alpha)$

CDARTS consists of two phases: pre-training and joint learning, which are
repeated cyclically until convergence. In the pre-training phase, the weights
of the search and intermediate evaluation networks are trained for a limited
number of epochs. The search network weights $w_{S}$ are optimized according
to the following equation:

$w^{*}_{S}=\operatorname*{arg\,min}_{w_{S}}L_{train}^{S}(w_{S},\alpha)$ (4)

where $L_{train}^{S}$ denotes the cross entropy loss function. The
intermediate evaluation network is then generated given the learned $\alpha$
weights by following the same discretization procedure used in DARTS. The
intermediate evaluation network weights $w_{E}$ are finally optimized on the
validation set according to the loss function:

$w^{*}_{E}=\operatorname*{arg\,min}_{w_{E}}L_{val}^{E}(w_{E},\bar{\alpha})$
(5)

where $\bar{\alpha}$ represents the discretized cell architetures resulting
from $\alpha$.

During the joint training phase, the $\alpha$ and $w_{E}$ weights are updated
based on the cross entropy losses of the search and evaluation networks and a
soft-target cross-entropy loss, which measures the distance between the logits
of the search and evaluation networks. The following equation represents this
joint optimization task:

$\displaystyle\alpha^{*},w^{*}_{E}=\operatorname*{arg\,min}_{\alpha,w_{E}}L_{val}^{S}(w_{S}^{*},\alpha)+L_{val}^{E}(w_{E},\bar{\alpha})$
(6) $\displaystyle+\lambda L_{val}^{S,E}(w_{S}^{*},\alpha,w_{E},\bar{\alpha})$

By minimizing the $L_{val}^{S}(w_{S}^{*},\alpha)$ term, the $\alpha$ parameter
is optimized given the fixed search network weights $w_{S}^{*}$. The
$L_{val}^{E}(w_{E},\bar{\alpha})$ variable serves to optimize the evaluation
network weights given $\bar{\alpha}$ as a fixed parameter. Finally, the soft-
target cross-entropy loss term,
$L_{val}^{S,E}(w_{S}^{*},\alpha,w_{E},\bar{\alpha})$ allows for the transfer
of knowledge from the evaluation network to the search network by applying the
features learned from the evaluation network to the $\alpha$ parameter of the
search network. This term is formulated as:

$L_{val}^{S,E}(w_{S}^{*},\alpha,w_{E},\bar{\alpha})=\frac{T^{2}}{N}\sum_{i=1}^{N}(p(w_{E},\bar{\alpha})log(\frac{p(w_{E},\bar{\alpha})}{q(w_{S}^{*},\alpha)}))$
(7)

where $N$ is the number of training samples, $T$ is the temperature
coefficient, and $p$ and $q$ are the feature logits of the search and
evaluation networks, respectively. $p$ and $q$ are calculated given the
features of the search and evaluation networks $f_{i}^{S}$ and $f_{i}^{E}$:

$\displaystyle
p(w_{E},\bar{\alpha})=\frac{\exp(f_{i}^{E}/T)}{\sum_{j}\exp(f_{j}^{E}/T)},$
(8) $\displaystyle
q(w_{S}^{*},\alpha)=\frac{\exp(f_{i}^{S}/T)}{\sum_{j}\exp(f_{j}^{S}/T)}$

The result of this joint training phase is sufficient knowledge transfer
between the search and evaluation networks that ensures both perform similarly
and learn similar features (Yu and Peng, 2020).

Algorithm 1 ICDARTS Search Phase

Input: Datasets $train$ and $val$, search and evaluation iterations $S_{S}$
and $S_{E}$, update iterations $S_{U}$, architecture hyperparameter $\alpha$,
and weights $w_{S}$ and $w_{E}$ for search $S$ and evaluation $E$ networks

Output: Evaluation network $E$

Initialize $\alpha$ randomly

Initialize $w_{S}$

for each search step $i\in[0,S_{S}]$ do

if $iModS_{U}$ then

Discretize $\alpha$ to $\bar{\alpha}$ by selecting the top $k$

Generate $E$ with $\bar{\alpha}$

for each evaluation step $j\in[0,S_{E}]$ do

Calculate $L^{E}_{val}$

Update $w_{E}$ according to Eq. 5

end for

end if

Calculate $L^{S}_{val}$, $L^{E}_{val}$, and $L^{S,E}_{val}$

Update $\alpha$ according to Eq. 9

Calculate $L^{S}_{train}$ and $L^{S,E}_{train}$

Update $w_{S}$ according to Eq. 10

Calculate $L^{E}_{train}$

Update $w_{E}$ according to Eq. 11

end for

###  3.2 ICDARTS

####  3.2.1 Algorithm Updates

Table 1: ICDARTS ’Zero’ Operation Return Type Configurations Zero Return | Zero Return Type
---|---
Configuration | Search | Evaluation | Retraining
V0 | zero | not included | not included
V1 | not included | not included | not included
V2 | random | random | random
V3 | random | random | zero
V4 | random | zero | zero

CDARTS strives to ensure that the continuous search cells optimized within its
search network effectively translate to discretized cells within a deeper
evaluation network that performs well and learns similar features. However,
when reviewing the CDARTS algorithm and its loss functions, we identified
multiple issues that limited correlation between the networks optimized in the
search and evaluation phases and introduced changes that address each.

First, we noted that the loss function used for updating the intermediate
evaluation network during the search phase (see Equation 6) was formulated so
that the network’s weights are dependent on both the search network’s loss and
the soft-target cross-entropy loss. This was in contrast to the loss function
used for training the evaluation network during the retraining phase, which
depended only on the evaluation network’s loss. In order to remove the
dependence of the search phase evaluation network on the additional terms, we
modify CDARTS’ joint optimization approach so that the weights of the
evaluation network are no longer dependent on the loss of the search network
but only that of the evaluation network.

Next, we shift the soft-target cross-entropy loss term, $\lambda
L_{val}^{S,E}(w_{S},\alpha,w_{E},\bar{\alpha})$, to the equation for updating
the search network weights. This change allows this term to be retained in the
joint learning phase and better serves the purpose of facilitating the
transfer of knowledge from the evaluation network’s weights to those of the
search network.

Finally, we noted that CDARTS optimizes the search and evaluation networks’
weights on two separate datasets. This approach injects unnecessary bias into
the joint learning phase, particularly in the case of the soft-target cross-
entropy loss term, where CDARTS is attempting to get the two networks to
produce the same output even though they are trained on separate datasets.
With this in mind, we modify the function for updating the evaluation network
weights so that it is dependent on the evaluation network’s loss on the
training rather than the validation dataset.

The loss functions for training the $\alpha$, $w_{S}$, and $w_{E}$ weights are
now:

$\alpha^{*}=\operatorname*{arg\,min}_{\alpha}L_{val}^{S}(w_{S}^{*},\alpha)+\lambda
L_{val}^{S,E}(w_{S}^{*},\alpha,w_{E}^{*},\bar{\alpha})$ (9)

$w^{*}_{S}=\operatorname*{arg\,min}_{w_{S}}L_{train}^{S}(w_{S},\alpha)+\lambda
L_{train}^{S,E}(w_{S},\alpha,w_{E},\bar{\alpha})$ (10)

$w^{*}_{E}=\operatorname*{arg\,min}_{w_{E}}L_{train}^{E}(w_{E},\bar{\alpha})$
(11)

Algorithm 1 presents the algorithm for the ICDARTS search phase and shows how
it incorporates our reformulations to the CDARTS loss functions as first
introduced in our conference paper (Herron et al., 2022). The result of our
changes is a search phase evaluation network that is a better proxy for that
of the retraining phase. The search phase evaluation network no longer depends
on a loss term that will be unavailable during retraining. Additionally, the
soft target cross-entropy loss term is now used for updating the search
network’s weights so that feedback from the evaluation network is supplied to
the search network, which better aligns with the original intended purpose of
this term. Finally, training the networks’ weights on the same dataset ensures
that the networks perform similarly rather than forcing them to produce the
same output given two different datasets.

Table 2: ICDARTS Template Ablations and Replacements

Ablation | Replacement
---|---
Pooling Operations | -
Identity Operation | -
Dilated Convolution Operations | -
Seperable Convolution Operations | -
Auxilary Heads | -
Stemming Layer | Identity
Convolution (k=1), Batch Norm
Concatenated Inputs
Reduce Cells | Average Pooling
Max Pooling
Convolution (k=1, stride=2)

Table 3: ICDARTS Algorithmic Ablation Study Routes with Loss Function Updates

Algorithm |  Loss Function
---|---
Route A |  Route B
CDARTS |  $\alpha\rightarrow L_{val}+L_{val}(J)$ $w_{e}\rightarrow L_{val}+L_{val}(J)$ $w_{s}\rightarrow L_{trn}$
Ablation 1 |  $\alpha\rightarrow$ No Change $w_{e}\rightarrow$ No Change $w_{s}\rightarrow L_{trn}+L_{trn}(J)$ |  $\alpha\rightarrow$ No Change $w_{e}\rightarrow L_{trn}+L_{trn}(J)$ $w_{s}\rightarrow$ No Change
Ablation 2 |  $\alpha\rightarrow$ No Change $w_{e}\rightarrow L_{val}$ $w_{s}\rightarrow$ No Change |  $\alpha\rightarrow$ No Change $w_{e}\rightarrow L_{trn}$ $w_{s}\rightarrow$ No Change
ICDARTS |  $\alpha\rightarrow$ No Change $w_{e}\rightarrow L_{trn}(J)$ $w_{s}\rightarrow$ No Change |  $\alpha\rightarrow$ No Change $w_{e}\rightarrow$ No Change $w_{s}\rightarrow L_{trn}+L_{trn}(J)$

Table 4: ICDARTS Operation Search Spaces

Search Space | Category | Description | Operation | Parameters
---|---|---|---|---
1 | Basic Operations | Basic building blocks of more complex operations. | Convolution | $k={3,5}$
Depthwise Convolution | $k={3,5}$
ReLU |
Leaky ReLU |
BatchNorm |
2 | Simple Operations | Operations from early NAS literature including NASNet Zoph et al. (2017). | Minimum Convolution | $k={3,5}$
Standard Convolution | $k={3,5}$
Factorized Convolution | $k={7,9}$
3 | ICDARTS Search Space | Widely-adopted search space used by DARTS and its derivatives. | Identity* |
Max Pooling* | $k={3}$
Average Pooling* | $k={3}$
Separable Convolution | $k={3,5}$
Dilated Convolution | $k={3,5}$
4 | MBConv Blocks | State-of-the-art mobile convolution blocks. |  MBConv Sandler et al. (2018) | $k={3}$
MBConv Tan and Le (2019) |  $k={3}$; $g={1}$
Fused-MBConv Tan and Le (2021) |  $k={3}$; $g={1}$
Note: Operations marked with * are included in all search spaces.

Table 5: Combined Search Space used by Dynamic Search Algorithm Operation | Parameters | Operation | Parameters
---|---|---|---
ReLU | - | Leaky ReLU | -
Sigmoid | - | Tanh | -
Batch Norm | - | Identity | -
Convolution | $k={3,5}$ | Max Pooling | $k={3,5}$
Avg Pooling | $k={3,5}$ | Minimum Convolution | $k={3,5}$
Standard Convolution | $k={3,5}$ | Factorized Convolution | $k={7,9}$
Separable Convolution | $k={3,5}$ | Dilated Convolution | $k={3,5}$
MBConv | $k={3,5}$ | MBConvV2 |  $k={3,5}$ $g={1,4,6}$
Fused MBConv |  $k={3}$ $g={1,4,6}$ |  |
Algorithm 2 ICDARTS Dynamic Search Space Algorithm

Input: Number of Tiers $T$, Pool of Layer Operations $O$, Maximum Operations
per Set of Edges $O_{max}$

Output: Optimal Operations, $o$ for Each Set of Edges

for each $t\in{T,...1}$ do

for each $r\in{1,...,2t}$ do

if $t=3$ then

Randomly select $O_{max}$ layer options $o_{t,r}$ for each set of edges given
$O$

else

Set $o_{t,r}$ to $o_{t-1,2r}+o_{t-1,2r+1}$ for each set of edges

end if

Run ICDARTS Search Phase to optimize $o_{t,r}$ for each set of edges based on
learned $\alpha$ values

if $t=1$ then

Return $o_{t,r}$ for each set of edges

else

Update $o_{t,r}$ to top $50\%$ of $o_{t,r}$ by $\alpha$ value for each set of
edges

end if

end for

end for

Figure 2: Tournament-Style Dynamic Search Space Algorithm for ICDARTS Overview
with Tiers

####  3.2.2 Enabling None Layers in Discretized Networks

The original DARTS method (Liu et al., 2019) includes zero as a layer choice
candidate operation. This option allows the NAS method to choose a layer that
will produce no output. However, in DARTS and its derivatives, (Chen et al.,
2019; Yu and Peng, 2020), the zero layer choice is not allowed when the
network candidate operations are discretized as this option is constant and
has a gradient of zero. Thus, its corresponding $\alpha$ weights cannot
properly ascertain the importance of this operation. In practice, a large
$\alpha$ value and small $\alpha$ value would produce the same output for the
zero layer. Since the corresponding $\alpha$ value is unlearnable, it is
unclear why this operation choice is included during the search phase. In the
following section, experimental results demonstrate comparable performance
without including this operation during the search phase. Also explored is an
alternative formulation of this operation that outputs randomly generated
activations during search and is replaced with a no-op when the evaluation
network is generated. The reformulation allows the alpha weight corresponding
to the zero operation to be learned without contributing to extracting
features from previous layers. This approach assumes that if a layer
contributes less than random noise, it should not be included in the final
evaluation network.

To access the effectiveness of ICDARTS both with and without the zero
operation choice, we carried out five different experiments, each using a
different configuration of $zero$ operation return types. These configurations
are all listed in Table 1. The V0 configuration is the same as the one used in
the original CDARTS paper, in which the $zero$ option is only included in the
search network but ignored at discretization so that the deep evaluation
networks and the final network used in retraining have no zero layers.
However, configurations $V1-V4$ use the same number of layer options for the
continuous search network and the discrete evaluation network. The traditional
$zero$ option returns a tensor of zeroes the size and shape of the input, and
the $random$ option similarly returns a tensor of uniform random values for
each input. In addition to the experiments run using the CDARTS and ICDARTS
search procedures, a separate set of experiments is run in which network cell
motifs are randomly initialized eight times for each $zero$ operation return
protocol. These experiments aim to provide baseline performance results to
contrast those of architectures discovered by both algorithms.

###  3.3 Ablation Studies

After making our initial set of improvements to the CDARTS algorithm, we
conducted two ablation studies on the ICDARTS search space and algorithm. The
first was conducted on the search space template of ICDARTS, including its
operation choices, auxiliary heads, stemming layers, and reduce cells. Table 2
contains a comprehensive listing of each ablation and any operations used to
replace the ablation. Note that the none operation is left out of the set of
operation choices by default.

The second ablation study was of the algorithmic changes to the CDARTS
algorithm that resulted in ICDARTS. We considered two routes of incremental
improvements to the original algorithm. The details of both are listed in
Table 9.

###  3.4 Alternate Search Spaces

Upon completing the ablation study, we explored additional methods for
expanding the limited search space of ICDARTS. This process involved exploring
alternative sets of operation choices with varying complexities and cell
discretization approaches that expanded the space of architectures that could
be discovered during the search phase. We developed a novel tournament-based
approach for incorporating dynamic search spaces in ICDARTS in conjunction
with these experiments.

####  3.4.1 Alternate Operation Spaces

We first explored alternative operation search spaces for ICDARTS. We began by
curating three additional operation search spaces for ICDARTS. Each search
space was comprised of operations of varying complexities, ranging from the
most basic operations to mobile convolution blocks from current state-of-the-
art vision models. Each operation space and a brief description are listed in
Table 4. Note that pooling and identity operations, marked by the *, were
included in each of the four search spaces, as they are all simple operations,
yet have been included alongside more complex operations, such as in the
widely adopted search space 3. Also, note that the minimum convolution
operation is defined by a convolution followed by a batch norm, and the
standard convolution operation consists of a ReLU activation function followed
by a convolution and batch norm.

After running ICDARTS with operation search space, we combined the operations
of all four search spaces and additional operations left out of the previous
search spaces due to memory constraints into one master search space (see
Table 5). In order to ensure that ICDARTS could traverse this large search
space efficiently, we implemented an algorithm for incorporating dynamic
search spaces into ICDARTS. Previous dynamic search algorithms partially
inspired this novel, tournament-style dynamic search algorithm, including (Li
et al., 2021) and (Shaw et al., 2019). As shown in Algorithm 2 and Figure 2,
our algorithm begins by randomly selecting $O_{max}$ operations from the set
of all operations, $O$, for each set of cell edges. After running the search
phase of ICDARTS to optimize the alpha weights corresponding to each edge, the
operations corresponding to the top 50% of alpha values are retained. The
algorithm then proceeds to the next tier, forming its per-node operation set
by combining the top operations from 2 runs in the previous tier. The process
repeats until it reaches the topmost tier, after which it yields the final
evaluation network to be evaluated in the retraining phase.

####  3.4.2 Alternate Search Cells

(a) Original DARTS (Liu et al., 2019)

(b) I-DARTS (Jiang et al., 2019)

(c) XDARTS

Figure 3: DARTS Cell Discretization Approaches

Next, we explored expanding the capacity of ICDARTS’ cell search space by
implementing new methods for discretization and incorporating them into the
ICDARTS algorithm. As previously discussed, the search cells optimized in
CDARTS and ICDARTS were initially introduced in DARTS. In these cells, a
softmax operation is applied only to edges coming from the same node, and of
these edges, only the one with the highest softmax values is eligible to be
selected as an input to a cell (see Figure 3(a)). Allowing a maximum of one
output of a previous node to be input to a current node eliminates any
potential edge candidates from the same cell that may be more suitable than
those from a different cell, thereby limiting the number of cell structures
the search algorithm can discover. Based on this observation, (Jiang et al.,
2019) presented the I-DARTS approach, in which all incoming edges to a node
are compared equally by applying one softmax operation across all their
weights. $k$ edges are then selected from among these with equal weight,
regardless of whether or not they come from different preceding nodes (see
Figure 3(b)). The advantages of this approach are that it increases the amount
of cell architectures that can be discovered and allows all incoming edges to
be compared fairly. After implementing a version of the I-DARTS cells and
incorporating it into ICDARTS, we also designed and implemented a novel
variant of this discretization approach, XDARTS, which further expands the
search space of the cells. This XDARTS aims to compensate for any instability
caused by selecting from a large set of edges at each depth. Following this
discretization approach, the number of edges selected as inputs to a node
equals the number of nodes preceding it. Thus, the number of edges selected at
each node grows alongside its number of prospective input edges (see Figure
3(c)).

##  4 Experiments & Results

In this section, we will describe the experiments and results performed to
evaluate the ICDARTS algorithm developed in this work.

Figure 4: CDARTS (left) and ICDARTS (right) Search Phase Evaluation Network
CIFAR-10 Test Accuracy Curves Given Different Zero Operation Configurations
(see Table 1).

pt$c_{k-1}$pt$c_{k-2}$pt$n_{0}$pt$n_{1}$pt$n_{2}$pt$n_{3}$pt$c_{k}$sep_conv_3x3sep_conv_3x3sep_conv_3x3sep_conv_3x3sep_conv_3x3sep_conv_3x3sep_conv_3x3ptsep_conv_3x3

pt$c_{k-1}$pt$c_{k-2}$pt$n_{0}$pt$n_{1}$pt$n_{2}$pt$n_{3}$pt$c_{k}$ptavg_pool_3x3ptidentitysep_conv_3x3sep_conv_5x5ptdil_conv_5x5sep_conv_3x3ptdil_conv_5x5ptdil_conv_5x5

Figure 5: CDARTS V0 normal (left) and reduction (right) cells from the network
with the best overall test accuracy.

pt$c_{k-1}$pt$c_{k-2}$pt$n_{0}$pt$n_{1}$pt$n_{2}$pt$n_{3}$pt$c_{k}$sep_conv_3x3sep_conv_3x3sep_conv_3x3sep_conv_3x3sep_conv_3x3sep_conv_3x3sep_conv_3x3ptsep_conv_5x5

pt$c_{k-1}$pt$c_{k-2}$pt$n_{0}$pt$n_{1}$pt$n_{2}$pt$n_{3}$pt$c_{k}$ptsep_conv_3x3sep_conv_3x3sep_conv_3x3ptsep_conv_5x5ptdil_conv_5x5ptdil_conv_5x5sep_conv_5x5sep_conv_3x3

Figure 6: ICDARTS V1 normal (left) and reduction (right) cells from the network with the best overall test accuracy. Table 6: CIFAR-10 Retraining Test Set Accuracies of Networks Produced by CDARTS, ICDARTS, and Random Initialization using Different Zero Return Configurations (see Table 1). Zero Return | Algorithm
---|---
Configuration | Random | CDARTS | ICDARTS
V0 | 96.78 (0.39) | 97.17 (0.21) |  96.99 (0.16)
V1 | 96.57 (0.32) | 97.01 (0.24) |  97.10 (0.19)
V2 | 96.52 (0.37) | 96.97 (0.27) |  96.92 (0.15)
V3 | 96.58 (0.31) | 96.96 (0.28) |  96.93 (0.14)
V4 | 96.58 (0.31) | 23.68 (29.71) |  90.00 (12.72)
Table 7: CIFAR-100 Retraining Test Set Accuracies of Networks Produced by CDARTS, ICDARTS, and Random Initialization using Different Zero Return Configurations (see Table 1). Zero Return | Algorithm
---|---
Configuration | Random | CDARTS | ICDARTS
V0 | 81.11 (1.39) |  83.66 (0.29) | 83.10 (0.65)
V1 | 80.54 (1.05) | 83.13 (0.71) |  83.33 (0.19)
V2 | 80.63 (1.26) | 83.19 (0.68) |  83.19 (0.21)
V3 | 80.84 (1.13) | 83.05 (0.72) |  83.15 (0.36)
V4 | 80.84 (1.13) |  10.98 (25.91) | 63.13 (28.35)

###  4.1 Datasets

For each change to the ICDARTS algorithm and search space, the architecture
search phase is conducted on the CIFAR-10 dataset and the resulting network
architecture is retrained on CIFAR-10. The networks produced before and after
the initial algorithmic changes that resulted in the ICDARTS algorithm, as
well as the experiments on the zero operation, were additionally retrained and
evaluated on the CIFAR-100 dataset. The CIFAR-10 and CIFAR-100 image
classification benchmarks consist of $10$ and $100$ classes, respectively, and
both are comprised of $50K$ training and $10K$ testing images of resolution
$32\times 32$.

###  4.2 Search & Evaluation Settings

At the beginning of the search phase, the original training set is divided
into two datasets of equal size, denoted by train and val, as in (Liu et al.,
2019) and (Yu and Peng, 2020). As previously discussed, in the ICDARTS
algorithm, the train partition is used for updating the weights of both the
search and evaluation networks, while val is reserved for updating the
$\alpha$ weights. The search phase runs for 30 epochs, not including 2 epochs
for pre-training the search network and 1 for warming up the intermediate
evaluation network each time it is generated. The search and evaluation
weights, $w_{S}$ and $w_{E}$ are updated using separate SGD optimizers with
learning rates of $0.08$, decay rates of $3\times 10^{-4}$, and momentum
settings of 0.9. The $\alpha$ weights are updated using an Adam optimizer
(Kingma and Ba, 2015) with a learning rate of $3\times 10^{-4}$, decay rate of
$0$, and momentum $\beta$s of ${0.5,0.999}$.

The evaluation stage involves retraining the discovered architectures for 600
epochs on the full training dataset and evaluating the retrained network on
the test dataset. Our retraining procedure closely resembles that of the
original CDARTS paper. The batch size is set to 128, and an SGD optimizer is
employed with a learning rate of $0.025$, momentum of $0.9$, and weight decay
of $5\times 10^{-4}$. As in the search phase, this optimizer is paired with a
cosine annealing learning rate scheduler. Following the approach of (Pham et
al., 2018; Zoph et al., 2018; Liu et al., 2018a), the training dataset is
augmented with a cutout regulation length of 16 (DeVries and Taylor, 2017),
the drop path rate of the full evaluation network is set to $0.3$, and the
auxiliary towers to $0.4$. The search and evaluation phases of each experiment
discussed in this publication were each run 8 times unless indicated otherwise
in their result tables.

###  4.3 ICDARTS

Figure 4 depicts the test set accuracies of the evaluation networks throughout
the search phases of CDARTS and ICDARTS. Each curve has been plotted with a
$95\%$ confidence interval across the runs of each configuration. Search
network pre-training epochs are not included in these graphs. The lower
standard deviation of the revised algorithm is evidence that our approach
gives results with improved consistency and stability across different
initializations. The accuracy disparity between the search and retraining
phases is expected due to the differences in data augmentation and the number
of epochs trained.

The evaluation network curve for the V4 search space configuration, which
includes the zero operation in the search phase evaluation network but not the
search network, shows particularly poor stability and difficulty learning.
This outcome is likely the product of the difference in behavior between the
search and evaluation networks during the search process, which results in a
feedback loop that produces an increasing number of $zero$ operations in the
discretized evaluation network.

Tables 6 and 7 list the average and standard deviation test set accuracies of
the evaluation networks on CIFAR-10 and CIFAR-100. Note that the values we
report were obtained at the end of the full retraining cycle rather than the
best test performance obtained at any point during retraining( as was reported
in (Liu et al., 2019) and later (Yu and Peng, 2020)). Hence, our results
measure a typical run’s performance rather than report our best outlier’s
performance. The accuracy results of the architecture produced randomly and by
the CDARTS and ICDARTS algorithms show that the ICDARTS networks achieved
similar mean accuracies to that of the CDARTS network but with much smaller
variation in performance, demonstrating stability improvements and results
that are potentially more reproducible.

Table 8: ICDARTS Template Ablation Study Retraining CIFAR-10 Test Set Accuracies and Inference Latencies Ablation | Results
---|---
| Retraining Accuracy | Inference Latency (batch/s)
Original | 97.13 (0.14) | 0.10 (0.01)
Pooling | 97.19 (0.23) | 0.09 (0.02)
Identity | 96.96 (0.35) | 0.11 (0.02)
Dilated Convolutions | 97.02 (0.26) | 0.10 (0.01)
Separable Convolutions | 96.15 (0.27) | 0.07 (0.00)
Auxilary Heads ∗ | 96.86 (0.14) | 0.11 (0.00)
Stem Identity ∗ | 80.01 (2.18) | 0.09 (0.02)
Stem Conv BN ∗ | 96.81 (0.38) | 0.11 (0.03)
Stem Concat ∗ | 92.40 (0.33) | 0.16 (0.00)
Reduce AvgPool | 96.94 (0.19) | 0.12 (0.07)
Reduce MaxPool | 96.87 (0.06) | 0.10 (0.03)
Reduce Convolution | 96.77 (0.12) | 0.09 (0.01)
Note: * indicates results that are averages of 4 runs.

Figure 7: ICDARTS Template Ablation Per Network Layer Type Frequencies

Figure 8: ICDARTS Template Ablation Per Network Cell Depth Frequencies

###  4.4 Ablation Studies

Table 2 and Figures 7 and 8 show the results of the ablation study of the
ICDARTS template. Note the inference latencies were obtained by calculating
the average per batch inference times on the test dataset prior to re-
training. Additionally, the operation type and cell depth frequencies shown in
Figures 7 and 8 are represented as the totals in each evaluation network in
order to account for the difference in the number of normal and reduce cells
in each network. For the cell operation choice ablations, only the ablation of
the pooling operations improved the average retraining accuracy, likely
because this resulted in less competition with better-performing operations
and the selection of more identity operations (see Figure 7). The retraining
accuracies for the rest of the ablations fell below that of the original, with
the ablation of the separable convolutions yielding the lowest accuracies for
this category of ablation. On the other hand, each cell operation choice
ablation improved average latencies, except for the dilated convolution
ablation. This trend likely occurred because removing this option caused the
algorithm to favor the computationally expensive separable convolutions, as
shown in Figure 7. The ablation of the separable convolution operations
improved the retraining latencies by the most significant margin, although it
also produced the worst average retraining accuracy.

The ablation of the auxiliary heads also resulted in lower retraining
accuracies. However, this ablation has higher latencies, a finding that might
be explained by this ablation resulting in ICDARTS favoring a deeper cell
structure (see Figure 8).

The stemming layer ablations also offered no improvement in retraining
accuracies and worse inference latencies. The only exception was when the
layer was replaced with an identity operation so that the number of input
channels was equal to that of the input images, which resulted in a network
with fewer parameters and, hence, a lower latency.

Finally, the accuracies from the reduce cell ablations were also worse than
that of ICDARTS. On the other hand, the average inference latencies varied,
with the case in which reduce cells were replaced with convolutions with
stride two providing the only latency improvement.

The results of the algorithmic ablation study listed in Table 9 generally
demonstrate stability improvements with the addition of each modification to
the original algorithm on both routes. The inference latencies tend to slow
with each change. The exception to this pattern was route A, in which the
switch to updating both network weights on the training set was not made until
the final step. However, as shown in the tables, the slower latencies were
eventually rectified with the final modification to the algorithm on both
routes.

Table 9: ICDARTS Algorithmic Ablation CIFAR-10 Retraining Test Set Accuracies
and Inference Latencies. The details of each ablation are listed in Table 3.

Ablation | Retraining Accuracy | Inference Latency (batch/s)
---|---|---
CDARTS (No Zero Operation) | 96.94 (0.30) | 0.09 (0.01)
A1 | 96.35 (0.48) | 0.08 (0.00)
B1 | 96.99 (0.22) | 0.09 (0.01)
A2 | 96.69 (0.22) | 0.08 (0.00)
B2 | 97.05 (0.18) | 0.11 (0.00)
ICDARTS | 97.13 (0.14) | 0.10 (0.01)

Figure 9: ICDARTS Per Network Layer Type Frequencies for Algorithmic Ablation
Study Routes A (left) and B (right). The details of each route are listed in
Table 3.

Figure 10: ICDARTS Per Network Cell Depth Frequencies for Algorithmic Ablation
Study Routes A (left) and B (right). The detail of each route are found in
Table 3.

###  4.5 Alternate Search Spaces

The results from running ICDARTS with different operation search spaces are
shown in Table 4 and Figures 11 and 12. Search space 2 achieved the highest
average retraining accuracy despite using simpler operations than the default
search space 3, yet had slightly more variation in accuracy. This outcome
might be explained by ICDARTS’ tendency to favor slightly deeper cell
architectures with this search space than with search space 3 or by this
search space’s diverse, yet competitive selection of operation choices that
appears to balance operation complexity and performance, resulting in the
search algorithm preferring cells composed of a diverse selection of simpler
operations. By contrast, the cells produced by the other search spaces tended
to favor one operation choice above all others. The networks produced with
search space 1, favored the convolution with kernel size 5 operation and
slightly deeper cell architectures than those produced by the original search
space, although these architectures yielded the worst accuracies of any search
space. Search space 4, which used the most complex operations and favored the
MBConvV1 operation, produced the second lowest accuracies. However, this
search space’s poor performance and inference latency could be explained by
the limitations placed on this search space due to the high computational
demands of its operations.

Table 10: ICDARTS Expanded Search CIFAR-10 Retraining Test Set Accuracies and Inference Latencies Search Space | Retraining Accuracy | Inference Latency (s/batch)
---|---|---
1* | 95.08 (0.39) | 0.11 (0.02)
2* | 97.29 (0.29) | 0.10 (0.02)
3 | 97.13 (0.14) | 0.10 (0.01)
4* | 96.55 (0.21) | 0.21 (0.01)
Dynamic* | 97.25 (0.06) | 0.10 (0.01)
Note: Results from 6 runs are indicated by *.

Figure 11: ICDARTS Alternative Operation Search Spaces Per Network Layer Type
Frequencies. The details of each search space are listed in Table 4.

Figure 12: ICDARTS Alternative Search Spaces Per Network Cell Depth
Frequencies. The details of each search space are listed in Table 4.

Figure 13: ICDARTS Dynamic Search Space Algorithm Tiers Per Network Cell Depth
Frequencies

The average accuracy and inference latency of ICDARTS using the dynamic search
space algorithm are listed alongside those of the other search spaces in Table
10. Additional results from each tier of the search algorithm are displayed in
Figures 14 and 13. The tournament-style dynamic search space algorithm applied
to the master operation space of Table 5 achieved an accuracy second only to
that of search space 2, but with one of lowest standard deviations of any
experiment performed on ICDARTS and an inference latency on par with that of
ICDARTS given the 2nd and 3rd search spaces. As shown in Figure 14, the cells
produced by the final tier of the algorithm most favored the tanh, MBConv, and
simple, standard, and separable convolution operations. All of these
operations were among the top in their derivative search spaces, with the
exception of the tanh activation and the standard convolutions, possibly due
to the ability of these operations to pair well with others. Figure 13 reveals
that deeper cell architectures were favored in the lower tiers, in which the
cell operations varied most due to the operation spaces being randomly spawned
for each cell at this level. However, as the architectures converged towards a
selection of the most optimal layer operations in the higher tiers, shallower
architectures were favored. This outcome suggests that, when given a large and
diverse search space of candidate operations, the effectiveness of layer
choices may be more important than cell depth for designing an optimal network
architecture. It may also explain how this search space produced networks with
relatively low inference latencies and suggests that this algorithm may prove
helpful for discovering networks with both low inference latencies and high
generalization abilities.

Figure 14: ICDARTS Dynamic Search Space Algorithm Tiers Per Network Layer Type Frequencies Table 11: ICDARTS Alternative Cells CIFAR-10 Retraining Test Set Accuracies and Inference Latencies Ablation | Retraining Accuracy | Inference Latency (s/batch)
---|---|---
DARTS Cells | 97.13 (0.14) | 0.10 (0.01)
I-DARTS Cells | 96.87 (0.27) | 0.10 (0.01)
XDARTS Cells | 97.21 (0.09) | 0.17 (0.01)

Figure 15: ICDARTS Alternative Cell Discretization Approaches: Per Network
Layer Type Frequencies (left) and Cell Depth Frequencies (right)

Table 11 and Figure 15 show that the I-DARTS cells produced results
significantly worse than the IDARTS cells in terms of accuracy and stability,
while the opposite was the case with the XDARTS cells. Since the depths and
operation choices of the searched DARTS and I-DARTS cells are similar, with
the I-DARTS cells even tending to be slightly deeper than the DARTS cells, the
reason for the poor performance and instability of I-DARTS cells is not
immediately apparent. However, we believe that the poor performance of the
I-DARTS cells is due to the softmax being applied across all incoming edges to
a node, the number of which increased based on the depth of the node in the
cell graph, rather than to a constant number of edges, as was the protocol
with the DARTS cells. The results from the XDARTS cells further support this
interpretation and show that the stability and performance reductions caused
by applying softmax functions over increasing numbers of edges can be
rectified by allowing the selection of additional edges based on a node’s
depth in the cell. Although the XDARTS cells produced significantly more
stable and accurate results than the DARTS cells, a severe drawback was the
high latencies of these networks. To address this problem, we are working on a
multi-objective version of ICDARTS that optimizes networks for latency and
performance.

##  5 Conclusion

In this work, we presented improvements to the original CDARTS algorithm and
search space, yielding results with superior stability and generalization
performance. The optimized networks produced by our initial set of
improvements obtained similar accuracies to those produced by CDARTS while
maintaining a minor variance (typically at least 2$\times$ smaller).
Additionally, we evaluated alternative zero operation schemes and demonstrated
that the best performance and stability results could be achieved by
completely excluding the zero operation from the search phase. Alternatively,
if it is desired to include zero operations in the discretized networks, we
found that this could be accomplished by replacing the zero operation with a
random operation during the search phase. We then conducted ablation studies
on the improved algorithm and its network template. The results showed that
improved accuracies could be obtained by removing pooling operations from its
search space. The search, inference, and retraining latencies were minimized
most by eliminating the separable convolution operations and the reduce cells,
although both changes negatively impacted the retraining accuracies. The
algorithmic ablation study generally showed that each change offered stability
improvements over the original algorithm. Finally, we experimented with
expanding the search space of our algorithm by introducing new operation
search spaces and new search cell discretization methods that increased the
diversity of cells that could be discovered. These experiments spawned the
creation of a novel algorithm for efficiently traversing large search spaces
using the improved search method.

Acknowledgments and Disclosure of Funding

This research used resources of the Compute and Data Environment for Science
(CADES) at the Oak Ridge National Laboratory, which is supported by the Office
of Science of the U.S. Department of Energy under Contract No. DE-
AC05-00OR22725

This research used resources of the Oak Ridge Leadership Computing Facility at
the Oak Ridge National Laboratory, which is supported by the Office of Science
of the U.S. Department of Energy under Contract No. DE-AC05-00OR22725.

## References

  * Brock et al. (2018) Andrew Brock, Theo Lim, JM Ritchie, and Nick Weston.  Smash: One-shot model architecture search through hypernetworks.  In _International Conference on Learning Representations_ , 2018.
  * Cai et al. (2019a) Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han.  Once-for-all: Train one network and specialize it for efficient deployment.  In _International Conference on Learning Representations_ , 2019a.
  * Cai et al. (2019b) Han Cai, Ligeng Zhu, and Song Han.  ProxylessNAS: Direct neural architecture search on target task and hardware.  In _International Conference on Learning Representations_ , 2019b.  URL https://arxiv.org/pdf/1812.00332.pdf.
  * Chen et al. (2019) Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian.  Progressive differentiable architecture search: Bridging the depth gap between search and evaluation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 1294–1303, 2019.
  * Chen et al. (2021) Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian.  Progressive darts: Bridging the optimization gap for nas in the wild.  _International Journal of Computer Vision_ , 129(3):638–655, 2021.
  * DeVries and Taylor (2017) Terrance DeVries and Graham W. Taylor.  Improved regularization of convolutional neural networks with cutout, 2017\.
  * Elsken et al. (2018a) Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter.  Efficient multi-objective neural architecture search via lamarckian evolution.  In _International Conference on Learning Representations_ , 2018a.
  * Elsken et al. (2018b) Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter.  Neural architecture search: A survey.  2018b.  doi: 10.48550/ARXIV.1808.05377.  URL https://arxiv.org/abs/1808.05377.
  * Elsken et al. (2019) Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter.  Neural architecture search: A survey.  _The Journal of Machine Learning Research_ , 20(1):1997–2017, 2019.
  * Guo et al. (2020) Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun.  Single path one-shot neural architecture search with uniform sampling.  In _European Conference on Computer Vision_ , pages 544–560. Springer, 2020.
  * He et al. (2017) Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross Girshick.  Mask r-cnn.  In _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , Oct 2017.
  * Herron et al. (2022) Emily Herron, Steven R Young, and Derek Rose.  Icdarts: Improving the stability of cyclic darts.  In _2022 21st IEEE International Conference on Machine Learning and Applications (ICMLA)_ , pages 1055–1062. IEEE, 2022.
  * Jiang et al. (2019) Yufan Jiang, Chi Hu, Tong Xiao, Chunliang Zhang, and Jingbo Zhu.  Improved differentiable architecture search for language modeling and named entity recognition.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3585–3590, Hong Kong, China, November 2019. Association for Computational Linguistics.  doi: 10.18653/v1/D19-1367.  URL https://aclanthology.org/D19-1367.
  * Kandasamy et al. (2018) Kirthevasan Kandasamy, Willie Neiswanger, Jeff Schneider, Barnabas Poczos, and Eric P Xing.  Neural architecture search with bayesian optimisation and optimal transport.  _Advances in neural information processing systems_ , 31, 2018.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In _ICLR (Poster)_ , 2015.  URL http://arxiv.org/abs/1412.6980.
  * Li and Talwalkar (2020) Liam Li and Ameet Talwalkar.  Random search and reproducibility for neural architecture search.  In _Uncertainty in artificial intelligence_ , pages 367–377. PMLR, 2020.
  * Li et al. (2021) Yanxi Li, Zean Wen, Yunhe Wang, and Chang Xu.  One-shot graph neural architecture search with dynamic search space.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 35(10):8510–8517, May 2021.  URL https://ojs.aaai.org/index.php/AAAI/article/view/17033.
  * Liu et al. (2018a) Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy.  Progressive neural architecture search.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 19–34, 2018a.
  * Liu et al. (2018b) Hanxiao Liu, Karen Simonyan, Oriol Vinyals, Chrisantha Fernando, and Koray Kavukcuoglu.  Hierarchical representations for efficient architecture search.  In _International Conference on Learning Representations_ , 2018b.
  * Liu et al. (2019) Hanxiao Liu, Karen Simonyan, and Yiming Yang.  DARTS: Differentiable architecture search.  In _International Conference on Learning Representations_ , 2019.  URL https://openreview.net/forum?id=S1eYHoC5FX.
  * Nekrasov et al. (2019) Vladimir Nekrasov, Hao Chen, Chunhua Shen, and Ian Reid.  Fast neural architecture search of compact semantic segmentation models via auxiliary cells.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 9126–9135, 2019.
  * Pham et al. (2018) Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean.  Efficient neural architecture search via parameters sharing.  In _International conference on machine learning_ , pages 4095–4104. PMLR, 2018.
  * Real et al. (2019) Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le.  Regularized evolution for image classifier architecture search.  In _Proceedings of the aaai conference on artificial intelligence_ , volume 33, pages 4780–4789, 2019.
  * Redmon and Farhadi (2017) Joseph Redmon and Ali Farhadi.  Yolo9000: Better, faster, stronger.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , July 2017.
  * Sandler et al. (2018) Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen.  Mobilenetv2: Inverted residuals and linear bottlenecks.  2018\.  doi: 10.48550/ARXIV.1801.04381.  URL https://arxiv.org/abs/1801.04381.
  * Shaw et al. (2019) Albert Shaw, Daniel Hunter, Forrest Iandola, and Sammy Sidhu.  Squeezenas: Fast neural architecture search for faster semantic segmentation, 2019.
  * Tan and Le (2021) Mingxing Tan and Quoc Le.  Efficientnetv2: Smaller models and faster training.  In _International Conference on Machine Learning_ , pages 10096–10106. PMLR, 2021.
  * Tan and Le (2019) Mingxing Tan and Quoc V. Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  2019\.  doi: 10.48550/ARXIV.1905.11946.  URL https://arxiv.org/abs/1905.11946.
  * Wang et al. (2020) Ning Wang, Yang Gao, Hao Chen, Peng Wang, Zhi Tian, Chunhua Shen, and Yanning Zhang.  Nas-fcos: Fast neural architecture search for object detection.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2020.
  * Xie and Yuille (2017) Lingxi Xie and Alan Yuille.  Genetic cnn.  In _Proceedings of the IEEE international conference on computer vision_ , pages 1379–1388, 2017.
  * Xie et al. (2019) Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin.  SNAS: stochastic neural architecture search.  In _International Conference on Learning Representations_ , 2019.  URL https://openreview.net/forum?id=rylqooRqK7.
  * Xu et al. (2020) Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi, Qi Tian, and Hongkai Xiong.  Pc-darts: Partial channel connections for memory-efficient architecture search, 2020.
  * Yang et al. (2020) Zhaohui Yang, Yunhe Wang, Xinghao Chen, Boxin Shi, Chao Xu, Chunjing Xu, Qi Tian, and Chang Xu.  Cars: Continuous evolution for efficient neural architecture search.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1829–1838, 2020.
  * Yu et al. (2018) Hongshan Yu, Zhengeng Yang, Lei Tan, Yaonan Wang, Wei Sun, Mingui Sun, and Yandong Tang.  Methods and datasets on semantic segmentation: A review.  _Neurocomputing_ , 304:82–103, 2018.
  * Yu and Peng (2020) Hongyuan Yu and Houwen Peng.  Cyclic differentiable architecture search, 2020.
  * Zoph and Le (2017) Barret Zoph and Quoc V. Le.  Neural architecture search with reinforcement learning, 2017.
  * Zoph et al. (2017) Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V. Le.  Learning transferable architectures for scalable image recognition, 2017\.  URL https://arxiv.org/abs/1707.07012.
  * Zoph et al. (2018) Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le.  Learning transferable architectures for scalable image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 8697–8710, 2018.