11institutetext:  Hector Reyes, Saeid Tizpaz-Niari, and Vladik Kreinovich
22institutetext: Department of Computer Science, University of Texas at El
Paso
500 W. University, El Paso, Texas 79968, USA
22email<EMAIL_ADDRESS>saied<EMAIL_ADDRESS>
# Over-Measurement Paradox: Suspension of Thermonuclear Research Center and
Need to Update Standards

Hector Reyes  Saeid Tizpaz-Niari  and Vladik Kreinovich

###### Abstract

In general, the more measurements we perform, the more information we gain
about the system and thus, the more adequate decisions we will be able to
make. However, in situations when we perform measurements to check for safety,
the situation is sometimes opposite: the more additional measurements we
perform beyond what is required, the worse the decisions will be: namely, the
higher the chance that a perfectly safe system will be erroneously classified
as unsafe and therefore, unnecessary additional features will be added to the
system design. This is not just a theoretical possibility: exactly this
phenomenon is one of the reasons why the construction of a world-wide
thermonuclear research center has been suspended. In this paper, we show that
the reason for this paradox is in the way the safety standards are formulated
now – what was a right formulation when sensors were much more expensive is no
longer adequate now when sensors and measurements are much cheaper. We also
propose how to modify the safety standards so as to avoid this paradox and
make sure that additional measurements always lead to better solutions.

##  1 What Is Over-Measurement Paradox

General case: the more measurements, the better. Most of our knowledge about
the world comes from measurements; see, e.g., Rabinovich 2005 . Each
measurement provides is with an additional information about the world – and
once we have a sufficient number of measurements of the same system, we may be
able to find the equations that describe the dynamics of this system and thus,
to get even more additional information that was hidden in the original
measurements.

The more measurements we perform, the more information we gain about the
system, the more accurate our estimates, and thus, the better will be our
decisions. From this viewpoint:

  * •

the more measurements we perform,

  * •

the better.

We only expect one limitation on the number of measurements – the financial
one. Indeed, at some point, after we have performed a large number of
measurements, we get a very accurate picture of the measured system. Decisions
based on this picture are close to optimal, and a very small expected increase
in optimality may not be worth spending money on additional measurements.

Over-measurement paradox: case study. Most of our energy comes from the Sun.
In the Sun, as in most stars, energy is generated by the thermonuclear
synthesis, when protons – i.e., nuclei of Hydrogen (H) – combine together to
form nuclei of Helium (He). This is a very efficient way of generating energy,
a way that does not lead to pollution or other side effects. The majority of
physicists believe that this is a way to get energy for our civilization:
instead of relying on direct or indirect energy from the thermonuclear
reaction inside the Sun, why not use the same reactions ourselves – this will
be a very effective and clean idea.

The idea is theoretically feasible, but technically, this is a very difficult
task. Researchers and engineers all over the world have been working on it
since the 1950s. To speed up the process, researchers from 35 major world
countries decided to join efforts, and allocated $65 billion dollars to build
an international research center where specialists from all the world will
work on this topic. This project is named ITER – this is both:

  * •

an abbreviation of International Thermonuclear Experimental Reactor and

  * •

the Latin word meaning “the way”; see, e.g., ITER .

The problem is that as of now, this project is suspended, and one of the main
reasons for this suspension is over-measurement; see, e.g., Kramer 2022 . In a
nutshell, the requirement was that, to guarantee safety, the level of danger –
e.g., the level of radiation – was supposed to be below the safety threshold
at a certain number of locations and scenarios.

  * •

The current design does satisfy this criterion.

  * •

However, the designers decided to be thorough and simulated more measurement
situations.

Unfortunately, some of the expected measurement results exceed the threshold.
As a result, the whole project is in suspension. Making sure that all future
measurements satisfy the criterion would require a drastic redesign and a
drastic further increase in the cost of the whole project – so drastic that it
is doubtful that this additional funding will appear, especially in the
current economic situation.

Why is it a paradox? If the designers did not perform these additional
measurements, the design would have been approved and the project would have
started. So in this case, additional measurements made the situation much
worse – not only for the researchers, but for the humankind as a whole. This
is a clear situation where additional measurements do not help at all.

But is it really a paradox? Maybe it is good that the project stopped – maybe
additional measurements revealed that the original design was unsafe?

What we do in this paper. In this paper, we analyze the situation from the
general measurement viewpoint and come up with several conclusions.

  * •

first, we show that this situation is, in principle, ubiquitous: a similar
problem will surface in many other projects, including those that have already
been approved and designed and seem to function OK;

  * •

second, although it may look that the problem is caused by insufficient safety
of the original design, we show that this is not the case: practically any
design, no matter how safe, will fail the currently used criteria if we
perform sufficiently many measurements;

  * •

finally, we propose a natural suggestion on now to change the corresponding
standards so as to avoid such unfortunate situations.

##  2 Analysis of the Problem

Let us formulate the situation in precise terms. We are interested in studying
states of different systems. A usual way to describe each state is by
describing the values of the corresponding quantities at different locations
and at different moments of time.

Usually, specifications include constraints on the values of some of these
quantities. These may be constrains on the radioactivity level, constraints on
concentration of potentially harmless chemicals, on the temperature, etc.

In all these cases, a typical constraint is that the value of some quantity
$q$ should not exceed some threshold $q_{0}$: $q\leq q_{0}$.

How can we check this constraint: seemingly natural idea. In the ideal world,
we should be able to measure the value $q(x,t)$ at all possible spatial
locations $x$ and for all possible moments of time $t$, and check that all
these values do not exceed $q_{0}$.

Of course, in real life, we can only perform finitely many measurements. So, a
seemingly natural idea is to perform several measurements, and to check that
all measurement results $q_{1},\ldots,q_{n}$ do not exceed $q_{0}$. However,
it is known that this seemingly natural idea can lead to dangerous
consequences; see, e.g., Rabinovich 2005 . Let us explain why.

Why the above seemingly natural idea is dangerous. The actual value of the
quantity $q$ depends on many factors which are beyond our control. For
example, the actual radioactivity level at a given location is affected by the
natural radioactivity level at this location – the level that can change
based, e.g., on weather conditions, when wind brings matter from neighboring
areas where this natural level is somewhat higher. There are many small
independent factors affecting the actual value of the quantity $q$.

In addition, the measurement result is somewhat different from the actual
value of the measured quantity; see, e.g., Rabinovich 2005 . We may be able to
get rid of major sources of such measurement errors, but there are always a
lot of small independent factors that lead to small changes of the measurement
results.

Because of both types of random factors, the measured value differs from its
locally-average level, and this difference is the result of a joint effort of
a large number of small independent factors. It is known (see, e.g., Sheskin
2011 ) that such a joint effect is usually well described by a normal
(Gaussian) distribution. To be more precise:

  * •

What is known is that in the limit, when the number $N$ of small independent
random factors increases (and the size of each factor appropriately
decreases), the probability distribution of the joint effect of all these
factors tends to the normal distribution – which thus appears as the limit of
the actual distributions when $N$ increases.

  * •

By definition of the limit, this means exactly that when the number $N$ of
factors is large – and in many practical situations it is large – the actual
distribution is very close to normal.

So, with high accuracy, we can safely assume that this distribution is normal.

This assumption explains why the above seemingly natural idea is dangerous.
Indeed, what we have is several measurement results $q_{1},\ldots,q_{n}$,
i.e., in effect, several samples from the normal distribution. Usually,
measurement errors corresponding to different measurements are practically
independent – and the same can be said about the random factors affecting the
value of the quantity $q$ at different spatial locations and at different
moments of time. From this viewpoint, what we observe are $n$ independent
samples from a normal distribution.

If we only require that $q_{i}\leq q_{0}$, we thus require that
$\max(q_{1},\ldots,q_{n})\leq q_{0}$. Usually, our resources are limited, so
we try to make the minimal effort to satisfy the requirements. In other words,
when we institute more and more efficient filters – thus slowly decreasing the
value $q_{i}$ – and finally, reach the condition $\max(q_{1},\ldots,q_{n})\leq
q_{0}$, we stop and declare this design to be safe.

  * •

We start with the design for which $\max(q_{1},\ldots,q_{n})>q_{0}$.

  * •

So the first time when we satisfy the desired constraint
$\max(q_{1},\ldots,q_{n})\leq q_{0}$ is when we get

$\max(q_{1},\ldots,q_{n})=q_{0}.$

This again may sound reasonable, but it is known that the probability that the
next random variable will exceed the maximum $\max(q_{1},\ldots,q_{n})$ is
proportional to $1/(n+1)$. So:

  * •

even if we perform 40 measurements – and this is, e.g., what measurement
theory requires for a thorough analysis of a measuring instrument (see, e.g.,
Rabinovich 2005 ),

  * •

we get a $1/40\approx 2.5\%$ probability that next time, we will go beyond the
safety threshold.

This is clearly not an acceptable level of safety – especially when we talk
about serious, potentially deadly dangers like radioactivity or dangerous
chemicals.

So what can be done to avoid this danger. To simplify our analysis, let us
suppose that the mean value of $q$ is 0. This can always be achieved if we
simply subtract the actual mean value from all the measurements result, i.e.,
for example, consider not the actual radioactivity level, but the excess
radioactivity over the average value of the natural radioactivity background.

In this case, measurement results $q_{1},\ldots,q_{n}$ form a sample from a
normal distribution with 0 mean and some standard deviation $\sigma$.

  * •

Of course, no matter how small $\sigma$, the normally distributed random
variable always has a non-zero probability of being as large as possible –
since the probability density function of a normal distribution is always
positive, and never reaches 0.

  * •

So, we cannot absolutely guarantee that all future values of $q$ will be
smaller than or equal to $q_{0}$.

  * •

We can only guarantee that the probability of this happening is smaller than
some given probability $p_{0}$, i.e., that

${\rm Prob}(q>q_{0})\leq p_{0}.$

So, to drastically decrease the probability of a possible disaster – from the
unsafe $2.5\%$ to the much smaller safety level $p_{0}\ll 2.5\%$:

  * •

instead of the original threshold $q_{0}$,

  * •

we select a smaller threshold $\widetilde{q}_{0}<q_{0}$ that guarantees that
the conditional probability of exceeding $q_{0}$ is small:

${\rm Prob}(q>q_{0}\,|\,\max(q_{1},\ldots,q_{n})\leq\widetilde{q}_{0})\leq
p_{0}.$

In this case:

  * •

if we have $n$ measurement $q_{1},\ldots,q_{n}$ all below $\widetilde{q}_{0}$,

  * •

then we guarantee, with almost-1 probability $1-p_{0}$, that the next value
will be below the actual danger threshold $q_{0}$.

This value $\widetilde{q}_{0}$ depends on $q_{0}$ and on the number of
measurements $n$:

  * •

the larger $n$,

  * •

the larger the value $\widetilde{q}_{0}$.

When $n$ increases, this value tends to $q_{0}$.

So what is included in the safety standard. When safety standards are
designed, one of the objectives is to make them easy to follow:

  * •

We do not want practitioners – who need to follow these standards – to perform
complex computations of conditional probabilities.

  * •

We need to give them clear simple recommendations.

From this viewpoint, the easiest to check if whether the measurement result
satisfies a given inequality.

So, a reasonable way to set up the corresponding standard is to set up:

  * •

the new threshold $\widetilde{q}_{0}$ and

  * •

the minimal necessary number of measurements $n$.

The standard then says that:

  * •

if we perform $n$ measurements, and the results $q_{1},\ldots,q_{n}$ of all
these $n$ measurements do not exceed this threshold $\widetilde{q}_{0}$, then
the situation is safe;

  * •

otherwise, the situation is not safe, and additional measures need to be
undertaken to make this situation safer.

Resulting common misunderstanding. The fact that safety standards provide such
a simplified description – and rarely mention actual threshold
$q_{0}>\widetilde{q}_{0}$ – makes most people assume that the critical value
$\widetilde{q}_{0}$ provided by a standard is the actual danger level, so any
situation in which a measured value exceeds $\widetilde{q}_{0}$ is
unacceptable. This is exactly what happened in the above case study.

And this is wrong conclusion:

  * •

if we perform a sufficiently large number of measurements,

  * •

we will eventually get beyond any threshold.

Indeed, according to the extreme value theory (see, e.g., Beirlant 2004 ; de
Haan 2006 ; Embrechts 2012 ; Gumbel 2004 ; Lorkowski 2015 ), for normal
distribution with mean 0 and standard deviation $\sigma$, the average value
$A_{n}$ of the maximum $\max(q_{1},\ldots,q_{n})$ grows with $n$ as

$A_{n}\sim\gamma\cdot\sqrt{2\ln(n)}\cdot\sigma,$

where $\gamma\approx 0.5772$ is the Euler’s constant

$\gamma\stackrel{{\scriptstyle\rm
def}}{{=}}\lim_{n\to\infty}\left(\sum_{k=1}^{n}\frac{1}{k}-\ln(n)\right).$

So, this mean value indeed grows with $n$.

Why this problem surfaces only now? Gaussian distributions was invented by
Gauss in early 19 century, measurements have been performed since antiquity,
so why is this problem surfacing only now? Why did not it surface much
earlier?

The main reason, in our opinion, is that, until recently:

  * •

sensors were reasonably expensive – especially accurate ones – and the cost of
measurements was non-negligible;

  * •

in this case, while in principle, it was possible to perform more measurement
than requires for safety testing, this would have led to useless costs.

Lately, however:

  * •

sensors have become very cheap: kids buy them to make robots, the cheapest
cell phones have very accurate sensors of positions, acceleration, etc.;

  * •

as a result, it is reasonably inexpensive to perform many more measurements
than required;

  * •

and, as we have mentioned, as a result, in situations that would previously –
based on only the required number of measurements – would be classified as
safe, now we get values exceeding the threshold $\widetilde{q}_{0}$ provided
by the standard – and thus, we end up classifying perfectly safe situations as
unsafe.

##  3 So What Do We Propose

What is the problem now: summarizing our findings. The reason why we have the
over-measurement paradox is that current safety standards usually list only
two numbers:

  * •

the recommended threshold $\widetilde{q}_{0}$ and

  * •

the recommended number of measurements $n$.

The idea that the results of all the measurements must not exceed
$\widetilde{q}_{0}$ for the situation to be considered safe.

The problem is that the recommended threshold $\widetilde{q}_{0}$ is actually
not the safety threshold $q_{0}$, it is smaller than the safety threshold –
smaller so that for the prescribed number of measurements $n$, we would
guarantee that:

  * •

for all future values,

  * •

the probability to exceed the real safety threshold $q_{0}$ should be smaller
than the desired small value $p_{0}$.

When, in an actually safe situation, in which the probability to exceed
$q_{0}$ does not exceed $p_{0}$, we perform more measurements than
recommended, then it is eventually inevitable that some of them will be larger
than the recommended threshold $\widetilde{q}_{0}$ – even though they will
still, with almost-1 probability, be not larger than the actual danger
threshold $q_{0}$. This leads to the following natural solution to the over-
measurement problem.

Proposed solution: we need to change the standards. In addition to providing
the two numbers $\widetilde{q}_{0}$ and $n$, we should provide the formula
describing the dependence of the testing safety threshold $t(n^{\prime})$ for
different numbers $n^{\prime}\geq n$ of actual measurements, so that for all
$n^{\prime}$, we should have

${\rm Prob}(q>q_{0}\,|\,\max(q_{1},\ldots,q_{n^{\prime}})\leq
t(n^{\prime}))\leq p_{0}.$

At least we should provide the value $t(n^{\prime})$ for several different
values $n^{\prime}$, thus taking care of the cases when, due to thoroughness,
practitioners will provide more measurements.

## Acknowledgments

This work was supported in part by the National Science Foundation grants
1623190 (A Model of Change for Preparing a New Generation for Professional
Practice in Computer Science), and HRD-1834620 and HRD-2034030 (CAHSI
Includes), and by the AT&T Fellowship in Information Technology.

It was also supported by the program of the development of the Scientific-
Educational Mathematical Center of Volga Federal District No.
075-02-2020-1478, and by a grant from the Hungarian National Research,
Development and Innovation Office (NRDI).

## References

  * (1) J. Beirlant, Y. Goegevuer, J. Teugels, and J. Segers, Statistics of Extremes: Theory and Applications, Wiley, Chichester, 2004.
  * (2) L. de Haan and A. Ferreira, Extreme Value Theory: An Introduction, Springer Verlag, Berlin, Hiedelberg, New York, 2006.
  * (3) P. Embrechts, C. Klüppelberg, and T. Mikosch, Modelling Extremal Events for Insurance and Finance, Springer Verlag, Berlin, Heidelberg, New York, 2012.
  * (4) E. J. Gumbel, Statistics of Extremes, Dover Publ., New York, 2004.
  * (5) ITER Project, https://www.iter.org/
  * (6) D. Kramer, “Further delays at ITER are certain, but their duration isn’t clear”, Physics Today, May 2022, pp. 20–22.
  * (7) J. Lorkowski, O. Kosheleva, V. Kreinovich, and S. Soloviev, “How design quality improves with increasing computational abilities: general formulas and case study of aircraft fuel efficiency”, Journal of Advanced Computational Intelligence and Intelligent Informatics (JACIII), 2015, Vol. 19, No. 5, pp. 581–584.
  * (8) S. G. Rabinovich, Measurement Errors and Uncertainty: Theory and Practice, Springer Verlag, New York, 2005.
  * (9) D. J. Sheskin, Handbook of Parametric and Nonparametric Statistical Procedures, Chapman and Hall/CRC, Boca Raton, Florida, 2011.


capbtabboxtable[][]

# MS-DARTS: Mean-Shift Based Differentiable Architecture Search

Jun-Wei Hsieh
College of Artificial Intelligence and Green Energy
National Yang Ming Chiao Tung University
<EMAIL_ADDRESS>
&Ming-Ching Chang
Department of Computer Science
University at Albany - SUNY
<EMAIL_ADDRESS>
&Ping-Yang Chen
Department of Computer Science
National Yang Ming Chiao Tung University
<EMAIL_ADDRESS>
&Santanu Santra
Department of Computer Science and Engineering
Yuan Ze University
<EMAIL_ADDRESS>
&Cheng-Han Chou*
College of Artificial Intelligence and Green Energy
National Yang Ming Chiao Tung University
<EMAIL_ADDRESS>
&Chih-Sheng Huang
Elan Microelectronics Corp. and
College of Artificial Intelligence and Green Energy
National Yang Ming Chiao Tung University
<EMAIL_ADDRESS>

###### Abstract

Differentiable Architecture Search (DARTS) is an effective continuous
relaxation-based network architecture search (NAS) method with low search
cost. It has attracted significant attentions in Auto-ML research and becomes
one of the most useful paradigms in NAS. Although DARTS can produce superior
efficiency over traditional NAS approaches with better control of complex
parameters, oftentimes it suffers from stabilization issues in producing
deteriorating architectures when discretizing the continuous architecture. We
observed considerable loss of validity causing dramatic decline in performance
at this final discretization step of DARTS. To address this issue, we propose
a Mean-Shift based DARTS (MS-DARTS) to improve stability based on sampling and
perturbation. Our approach can improve bot the stability and accuracy of
DARTS, by smoothing the loss landscape and sampling architecture parameters
within a suitable bandwidth. We investigate the convergence of our mean-shift
approach, together with the effects of bandwidth selection that affects
stability and accuracy. Evaluations performed on CIFAR-10, CIFAR-100, and
ImageNet show that MS-DARTS archives higher performance over other state-of-
the-art NAS methods with reduced search cost. Code is available at
https://github.com/aaron851113/MS-DARTS

##  1 Introduction

Recent development of convolutional neural network (CNN) architectures has
advanced substantially in several fields including computer vision and
language models. Manual CNN architecture design is still a common practice
nowadays; However, it can take substantial amount of time and efforts.
Alternatively, network architecture design process can be automated, which
might lead to improved models with lower costs and fewer computational time.
Neural Architecture Search (NAS) is a technique for automating this network
design process, where a large set of possible architectures are explored and
optimized. NAS has growing popularity that can potentially replace the manual,
trial-and-error paradigm of CNN architecture design in various fields. NAS
techniques [1, 2, 3] can automatically find suitable network architectures
depending on the application needs. In some cases, the resulting model can
outperform networks designed by human experts. This automatic architecture
search is performed upon three conceptual components:

(1) The search space defines the possible architectures as a principal
representation for design optimization. (2) The search strategy defines the
exploration technique to be performed in the search space. (3) The performance
estimation strategy evaluates the predictive performance of a given
architecture on unseen data. NAS approaches generally fall into two paradigms:
heuristic search and differentiable search.

Figure 1:  Overview of the MS-DARTS Network Architecture Search (NAS)
pipeline. The subfigure on the left is adopted from [3].

Heuristic NAS methods contain two sub-categories: (i) evolution based e.g.,
AmoebaNet [4] and hierarchical evolution [5] and (ii) reinforcement learning
(RL) based e.g., NAS [2], NASNet [6], and ENAS [7]. The bottleneck of
heuristic search algorithms is typically the high computational cost in
validating the accuracy of various architecture candidates during the model
generation and optimization process. RL-based methods take accuracy as the
reward to decide if a candidate model should be removed in the process.
Despite their effectiveness in finding high-quality architectures, these
approaches often require high computational cost (many GPU days) [2]. These
methods are often impractical due to computational resource limitation.

Differentiable NAS methods such as DARTS [3, 8, 9] build a supernet with a set
of architectural parameters. Based on the supernet, the search process
optimizes both network parameters and architectural parameters in a single
training framework. The best architecture is generated with optimized
parameters [10]. DARTS methods are efficient when running with limited
computational resources. Despite of the computational efficiency, DARTS faces
stability and generalizability issues of the obtained network architectures
[SantraSurvey]. Although the validation accuracy of the mixture architecture
can be improved, performance of the derived architecture often collapses at
the final evaluation stage, when discretizing the continuous architecture
representation into a discrete one (the actual network architecture) [11, 12].
Such instability makes DARTS producing unwanted architectures. For example,
parameter-free operations such as skip connections usually dominate the
generated architecture [11]. DARTS algorithms often prefer wide and shallow
structures [13]. In R-DARTS [11], early-stopping based on hand generated
criteria is proposed to address these issues. However, this solution does not
always work, since the intrinsic instability can occur from the beginning of
the search process. The key discretization step at the end of DARTS is
performed by projecting a continuous architecture onto a discrete
representation manifold to derive the best discrete architecture as output.
Oftentimes this projection step can cause significant performance drop between
the mixture architecture (found by gradient-based optimization in super-
network) and the obtained discrete architecture. It is shown in DA2S [14] that
even when the super-network achieves around $90\%$ accuracy when trained on
CIFAR-10 [15], sub-architecture without parameter re-training often reports
less than $20\%$ accuracy on the same dataset. DARTS adopts the Stochastic
Gradient Descent (SGD) approach to optimize the cost function and then derives
the desired architecture parameters. The SGD method makes DARTS mostly
converge to a sharp minimum which is sensitive to perturbation. Then, slight
perturbations will result insignificant cost increase and dramatically
decrease the test accuracy. Such “projection gap” prevents DARTS from
performing a full exploration of the architecture space in a stable way.

Mean-Shift based DARTS. This paper addresses this instability problem from two
viewpoints, i.e., robust minimization and optimization generalization. For
robust minimization, we search the optimal network architecture by minimizing
its worst-case cost. For optimization generalization, we smooth out sharp
minima and guide the convergence of SGD to wider and flatter minima by
proposing an improved DARTS approach based on the mean-shift sampling scheme
[16]. As shown in R-DARTS [11], the instability of DARTS converging into a
sharp minimum is caused by the searching process mostly along a peak surface
of the validation loss function. We study the stabilization of DARTS via
architecture perturbation, where the mean-shift serves as our perturbation and
smoothing policy that can effectively smooth out the landscape of the loss
function. We are the first to investigate the adaptation of mean-shift for
DARTS, and provide mathematical justification of this approach w.r.t. The
‘mean’ operation will smooth out sharp minima and guide the convergence of
DARTS to flatter minima which results in good generalization and accuracy
stability. Since the search surface becomes flatter, a large prediction can be
made via the ‘shift’ operation so that better efficiency in architecture
search is gained. The above reasons can explain why our proposed Mean-Shift
based DARTS (MS-DARTS) can stabilize and speed up the differentiable
architecture search. Figure.1 overviews our approach, where mean-shift governs
the projection of the continuous searched architecture into a discrete one in
a flat, stable, efficient, and accurate way.

Intuitively, our optimization of the architecture search is based on
perturbation and robust mean-shifting to generate flatter minima in
architecture configurations, and produce benefits in optimization generation.
For robust minimization, we search the optimal network architecture by
optimizing its worst-case cost which is the maximum loss among its local
weighted architecture candidates selected according to their performances
rather than relying on a single sample. MS-DARTS essentially adopts an
attention-based sampling scheme to smooth out sharp minima and thus can bypass
local fluctuations to generate flatter loss landscape. It uses a Gaussian
distribution to sample data points. This paper also investigates the behaviors
of its bandwidth parameter to smooth out the sharp minima. We will show later
that such smooth representation landscape can be quantified using the
eigenvalue of the Hessian matrix of the generated architectures. Better
stability can be observed by examining the accuracy differences before and
after discretizing the best continuous models. Experimental results in chapter
4 show that the MS-DARTS generated architectures achieve better stability and
efficiency, compared to SoTA DARTS methods without performance declining.

Contributions of this paper include:

  * •

Proving the mean-shift design can smooth out the sharp minima caused by DARTS-
based NAS to flatter minima from which significant performance drop can be
avoided.

  * •

Theoretical analysis is provided for the behaviors of mean-shift on affecting
the stability and accuracy of DARTS, together with the effects of bandwidth
selection.

  * •

Theoretical investigate for the effects of the bandwidth parameter of mean-
shift algorithm to smooth out the sharp minima is also provided.

  * •

Experimental evaluations comparing MS-DARTS against state-of-the-art (SoTA)
NAS models on various search spaces and image classification datasets
demonstrate improvements in both accuracy and computation cost.

##  2 Background

Network architecture search (NAS) is a computational approach that
automatically optimizes neural network architecture design. NAS is an
automated method that can use limited computing resources to design the best
network architecture with minimal human intervention.

###  2.1 NAS

Early NAS approaches [2, 17] train candidate architectures from scratch during
each search step, thus the computation cost is very high. ENAS [7] avoids
training all candidate architectures from scratch by weight sharing. Although
with speed up, this strategy may produce incorrect estimation of candidate
architectures [12]. It may be difficult for NAS to find a good network
architecture from a large number of candidate architectures, which further
reduces the effectiveness of the final searched network architecture.
Subsequently, DNA [18] converts the large NAS search space into blocks to
reduce parameter changes via weight sharing and can thus fully explore and
train candidate architectures. DAS [19] converts the discrete network
architecture search space into a continuously differentiable one, such that
gradient optimization can be applied for architecture search. The primary goal
of DAS is on finding hyperparameters of conv layers (filter size, number of
channels, and grouped convolutions). It is observed in Maskconnect [20] that
cell-based network structure typically follows a pre-determined pattern
between blocks, e.g., each block only connects to its first two blocks [21] or
all previous blocks [22]. Similarly, [23, 24] also search for network
architecture on continuous domains, where the aim is only on fine-tuning
specific structures.

###  2.2 DARTS

Differentiable ARchiTecture Search (DARTS) [3] methods address the above
challenges by adopting a differentiable framework for architecture parameter
search. Without searching over a discrete set of architectures, DARTS related
works search the optimal operations in a continuous and differentiable search
space, where a robust cell architecture can be efficiently determined with
gradient descent. An important issue of DARTS is that easy-to-optimize
operators (such as skip-connections and pooling operations) may dominate in
early stages, hence hinder the selection of more powerful operations (such as
convolutions of large kernels). This issue can be alleviated by freezing the
updates of the architecture parameter $\boldsymbol{A}$ in the early stages,
and allowing the weighting parameter W to better initialize the convolution
operations [8, 25, 26, 10]. In P-DARTS and DARTS+ [25, 27], a strong strategy
is enforced to control the number of skip connections within a cell to a pre-
determined value. The progressive search of P-DARTS [25] gradually increases
the depth of the network and reduces the candidate operations according to a
mixed operation weight. This approach alleviates the problem of excessive
calculations caused by the increasing of depth is alleviated and reduces
search instability.

###  2.3 Embedding of the evaluation procedure into the search procedure

Embedding of the evaluation procedure into the search procedure is another
issue of NAS optimization, which is not explicitly performed in previous
works. Various methods are designed to solve this problem (e.g., early
stopping [11, 27] and progressive optimization [25, 28]), to overcome the
issue of discretization gap [14]. It is observed in Fair-DARTS [29] that the
number of weak operators (such as skip connections) increases as the search
proceeds, which will cause unfair competitions among the operators. Fair-DARTS
[29] is proposed to address this issue via relaxing the probability of
operations, such that each operator has equal opportunity to develop the
architecture strength. Compared to our proposed MS-DARTS, Fair-DARTS is only
an indirect solution which cannot handle the discretization problem at the
end. SGAS [28] circumvents the discretization problem via a greedy strategy to
prevent the problematic skip connections or other weak operators to take
effect. However, potentially good operations might be pruned out as well due
to this greedy under-estimation. It is frequently observed in DARTS that the
resulting architecture is with good accuracy during the search stage, however
performing worse in the actual testing stage. The argues the collapse results
from the unfair advantage in an exclusive competitive environment, where skip
connections overly benefit from it, hence causing an aggregation. To suppress
such an advantage from overshooting, they convert the competition into
collaboration where each operation is independent of others. It is however an
indirect approach.

###  2.4 Stabilizing DARTS

Apart from the aforementioned issues, DARTS only optimizes a single point on
the simplex in each architecture search epoch. Such optimization may not
generalize well after the discretization in the evaluation stage. DARTS-based
algorithms prune operations on every edge except the one with the largest
architecture weight. Hence the stability and generalization of DARTS has been
widely challenged. Significant performance drop can occur in deriving the
discrete architecture from the continuous version based on projection. Several
approaches [11, 30] are proposed to investigate this stability and
generalizability issue of DARTS. Zela et al. [11] empirically point out that
the stability is highly correlated with the dominant eigenvalue
$\lambda_{\max}^{\boldsymbol{A}}$ of the Hessian matrix of the validation loss
function of an architecture $\boldsymbol{A}$ . They also present an early
stopping criterion to prevent $\lambda_{\max}^{\boldsymbol{A}}$ from
exploding. Other approaches, e.g., partial channel connection [8], scheduled
drop path [6], and regularization of architecture parameters are proposed to
address the stability of DARTS.

##  3 Method

Mean-Shift based Differentiable Architecture Search. Our goal is to construct
a novel differentiable architecture search algorithm which has a stabilizing
accuracy gap between search procedure and evaluation procedure. Meanwhile, we
could add less search cost as possible. We designed a perturb process with a
machine learning algorithm - Mean-Shift [16] into DARTS search procedure, then
we could balance the searching stability and search cost at the same time.

###  3.1 DARTS and Mean-Shift

DARTS [3] is a cell-based neural architecture search approach. It works on a
Directed Acyclic Graph (DAG) of nodes, where each node represents a set of
feature maps [31]. Specifically, each node $x^{(i)}$ is a latent
representation of feature map obtained from conv layers. Let ${{o}(.)}$
denotes an operation to be applied to a node $x^{(i)}$, e.g., convolution,
pooling, skip, etc.. Each directed edge $e(i,j)$ connecting nodes $x^{(i)}$
and $x^{(j)}$ is associated with an operation $o^{(i,j)}$ that transforms node
$x^{(i)}$ to node $x^{(j)}$ for $i<j$. Let ${O}$ denotes the set of all
possible candidate operations. Each intermediate node of the DAG is computed
depending on all of its predecessors according to
${x}^{(j)}=\sum\limits_{i\,<j}{o^{(i,j)}({x^{(i)}})}$. Figure.1 (left) shows
the cell structure of DARTS [3].As shown in Figure.2(a), to make the search
space continuous and differentiable, each operation ${o}^{(i,j)}$ transforming
node $x^{(i)}$ to node $x^{(j)}$ is replaced by a “continuous” operation
$\bar{o}^{(i,j)}$, which is obtained by mixing all possible candidate
operations with SoftMax: $\bar{o}^{(i,j)}(x)=\sum\limits_{o\in
O}{\frac{\exp(\alpha_{o}^{(i,j)})}{\sum\limits_{{o}^{\prime}\in
O}{\exp(\alpha_{{{o}^{\prime}}}^{(i,j)})}}}o({x}),$ where $\alpha_{o}^{(i,j)}$
is a weighting parameter for operation ${o(.)}$ from node $x^{(i)}$ to node
$x^{(j)}$. ${{\bar{o}}^{(i,j)}}(x)$ indicates a mixed result of weighted
summation of outputs of all operators from node ${x^{(}i)}$ to node
${x^{(}j)}$. Let $W$ denote the network weights, ${\mathcal{L}_{valid}}$ and
${\mathcal{L}_{train}}$ denote the outer and inner objectives in Eq.(1),
respectively. In DARTS, the searched architecture is mathematically
represented as a $d$-dimensional weighting vector
$\boldsymbol{A}=\\{\alpha_{o}^{(i,j)}\\}$. DARTS aims to learn the set of
continuous weighting variables $\boldsymbol{A}$ by solving the following bi-
level optimization:

$\displaystyle\begin{split}&\underset{\boldsymbol{A}}{\mathop{\min}}\,\,{\mathcal{L}_{valid}}(\boldsymbol{A},{{W}^{*}}(\boldsymbol{A}))\\\
s.t.&{\quad{W}^{*}}(\boldsymbol{A})=\arg\underset{W}{\mathop{\min}}\,{\mathcal{L}_{train}}(\boldsymbol{A},W).\\\
\end{split}$ (1)

At the end of the architecture search, a discrete architecture (Figure.2(c))
is obtained by replacing ${{\bar{o}}^{(i,j)}}$ with the most similar operation
(Figure.2(b)):

${o^{(i,j)}}=arg\max_{o\in O}{\ \alpha_{o}^{(i,j)}}.$ (2)

Despite the efficiency of DARTS, the method is not guaranteed to generalize
well for the evaluation test, as the optimization is performed for a single
point on the simplex in every search epoch. It is reported in [32] that DARTS-
based algorithm often yields deteriorating architectures, which can produce a
dramatic performance drop when deriving the actual discrete architecture from
the continuous mixture architecture using Eq.(2).

(a)  (b)  (c)

Figure 2:  DARTS [3] training search space from continuous stage to discrete
stage.

Mean-shift (MS) [33, 34, 16] is a well-studied non-parametric iterative
algorithm for estimating the modes of probability density function (PDF) via
kernel density estimation [35]. The principle of this algorithm is to update
the mode estimation iteratively via a weighted average around neighboring
points. Such weighted average estimation brings a smoothing effect to find a
stationary point more stably. Let
$\boldsymbol{D}\coloneqq\\{\boldsymbol{A}_{p}\in{{\mathbb{R}}^{d}}\\}_{p=1}^{N}$
denote the set of $N$ data points (or network architectures)
${\boldsymbol{A}_{p}}$, and
$\boldsymbol{\Omega}\coloneqq\\{{{\omega}_{p}}>0\\}_{p=1}^{N}$ denote the set
of weights $\omega_{p}$ to weight each ${\boldsymbol{A}_{p}}$. In addition,
let $K(\boldsymbol{A})$ be a multivariate normal kernel, i.e.,

$K(\boldsymbol{A})=(2\pi)^{-2d}\exp\left(-\frac{\|\boldsymbol{A}\|^{2}}{2}\right),$
(3)

where $\|\boldsymbol{A}\|^{2}$ is the norm of $\boldsymbol{A}$. The kernel
density estimate (KDE) with a kernel $K$ for $\boldsymbol{D}$ is given by:

$\hat{p}(\boldsymbol{A})=\sum\limits_{p=1}^{N}\omega_{p}\;K(\boldsymbol{A}-\boldsymbol{A}_{p}),$
(4)

where $\int\limits_{\boldsymbol{A}}{K(\boldsymbol{A})d\boldsymbol{A}=1}$ and
$\sum\limits_{p=1}^{N}{{{\omega}_{p}}=1}$. Let $k(z)$ be a profile, $i.e$.,
$k(z)=c\cdot\exp\left(-z/2\right)$, where $c$ is a constant and $z$ is a
scalar $\in(0,\infty)$. Gaussian kernel $G(z)=-{k}^{\prime}(z)$ can be
regarded as a special case of $K$. The weighted mean shift
$\delta_{\boldsymbol{A}}$ of the estimated density in the observation window
decided by $K$ can be represented by [16]:

${\delta}_{\boldsymbol{A}}=\frac{\sum\limits_{p=1}^{N}\boldsymbol{A}_{p}\;\omega_{p}\;G(\|\boldsymbol{A}-\boldsymbol{A}_{p}\|^{2})}{\sum\limits_{p=1}^{N}\;\omega_{p}\;G(\|\boldsymbol{A}-{{\boldsymbol{A}}_{p}}\|^{2})}-\boldsymbol{A}.$
(5)

Assume ${\boldsymbol{A}}^{t}$ is the solution estimated at the $t$-th
iteration. The MS algorithm updates its new estimation
${{\boldsymbol{A}}^{t+1}}$ as follows:

${\boldsymbol{A}}^{t+1}={\boldsymbol{A}}^{t}+\delta_{\boldsymbol{A}}^{t}.$ (6)

The iteration continues until
$\left\|{{\boldsymbol{A}}^{t+1}}-{\boldsymbol{A}}^{t}\right\|^{2}$ being
convergent. We adopt this MS algorithm to tune the architecture parameters for
searching a stable architecture with retained accuracy and less training time.

###  3.2 Motivation of Mean-Shift for DARTS

As aforementioned, DARTS-based methods are advantageous for their fast
gradient-based architecture search, however the conversion from the found
continuous architecture back to discrete one can cause significant performance
drop. Although DARTS-based methods can reduce the validation loss in the
training stage, it can still be unstable in terms of numerical blow-up in
computing the final architecture evaluation [11, 27]. Such instability is
caused by the selection of SGD to train over-parameterized deep neural
networks due to the problem of sharp minima [36]. Sharp minima will make the
found solution highly sensitive to perturbations (or noise) and have bad
generalization due to their high over-fitting to training data [37]. In DARTS
the problem of sharp minima of $\mathcal{L}_{valid}$ can explain why a small
perturbation $\delta$ on an architecture A will cause a significant reduction
of the validation accuracy (e.g., from over 90$\%$ to less than 10 $\%$ [32]).
It becomes an open question whether escaping sharp minima can improve the
generalization.

In this paper, the proposed MS-based DARTS is a solution to address the above
issue by smoothing out the landscape of ${\mathcal{L}_{valid}}$. We compare a
relevant work [32] using our formulation in the following. In [32], the
optimal weight $W^{*}(\boldsymbol{A})$ in Eq.(1) is re-defined to be
$\overline{W}(\boldsymbol{A})$ as:

$\overline{W}(\boldsymbol{A})=\operatorname*{arg\,min}_{W}\;\max_{\|\delta\|\leq\epsilon}\;\mathcal{L}_{train}(\boldsymbol{A}+\delta,W).$
(7)

The best architecture $\boldsymbol{A}^{*}$ can then be obtained by solving

$\boldsymbol{A}^{*}=\operatorname*{arg\,min}_{\boldsymbol{A}}\;\mathcal{L}_{val}\left(\boldsymbol{A},\overline{W}(\boldsymbol{A})\right).$
(8)

Eq.(7) solves the robust optimization problem [32] minimizing the worst-case
loss around a neighborhood of an architecture $\boldsymbol{A}$. In fact, at
each sampled iteration, $W$ is re-estimated by changing the original
architecture only once. However, this cannot provide enough a solution to flat
the loss landscape of $\mathcal{L}_{train}$, which is crucial for stability
control. In comparison, we propose MS-DARTS, the recursive update of the
architecture $\boldsymbol{A}$ with the mean-shift algorithm in Eq.(7) before
minimizing $\mathcal{L}_{train}$ can effectively smooth the landscape, such
that better accuracy and stability can be obtained. One additional advantage
of this recursive updating of $\boldsymbol{A}$ is efficiency, since less
iterations are needed to find $W$ by minimizing $\mathcal{L}_{train}$. In
addition, MS-DARTS also can address the flatness of minima. During training,
flatter minima [37], and thus better stability in architecture performance.

###  3.3 Robust Mean-shift based DARTS

The core challenge of designing robust DARTS architectures is to consider how
best to increase stability while reducing the optimization gap caused by the
sharp minima of $\mathcal{L}_{valid}$. In Eq.(7), the best weight
$\overline{W}$ is estimated by searching $\boldsymbol{A}$ with the worst case
around a neighborhood of the original architecture. However, this optimizer
cannot provide enough flatter (or smoothing) effect on the loss landscape,
which is crucial for stability control. Our work is inspired by recent
advances in understanding the loss surface of deep neural networks. In [38],
Izmailov et al. proposed a Stochastic Weight Averaging (SWA) scheme to smooth
the loss surface by averaging the weights at different checkpoints obtained
during training. In [39], a parameter averaging scheme was used to create
ensembles in natural language processing tasks. The idea of using averaging to
accelerate stochastic approximation algorithms can be traced back to the 1960s
[40]. ‘Averaging’ forms the core idea of this paper to generate flatter minima
for architecture searching by averaging multiple points along the trajectory
of SGD. One additional novelty of this paper is to look ahead the direction of
parameter searching during the training procedure to achieve faster
convergence via a shift operation.The proposed Mean-Shift DARTS can
effectively smooth the sharp landscape of $\mathcal{L}_{valid}$ by updating
the architecture $\boldsymbol{A}$ using Eq.(6) where $\delta_{A}$ is obtained
by a boosting or filtering method. Then, Eq.(7) can be reformulated as:

$\overline{W}({\boldsymbol{A}})=\operatorname*{arg\,min}_{W}\;\mathcal{L}_{train}({\boldsymbol{A}}+{\delta}_{\boldsymbol{A}},W).$
(9)

Based on $\overline{W}({\boldsymbol{A}})$, the best architecture
$\boldsymbol{A}^{*}$ is obtained by:

$\boldsymbol{A}^{*}=\operatorname*{arg\,min}_{{\boldsymbol{A}}}\;\mathcal{L}_{valid}\left(\boldsymbol{A},\overline{W}({\boldsymbol{A}})\right).$
(10)

We next explain how the proposed MS-DART based on mean-shift parameter tuning
can address the aforementioned instability of the found architecture
$\boldsymbol{A}$ in DARTS. Represent the architecture parameter
$\boldsymbol{\alpha}$ as a $d$-dimensional vector. Assume there are $N$ sample
points $\\{\boldsymbol{A}_{p}\\}$ obtained around the unknown $\boldsymbol{A}$
within a radius $\epsilon$, where each $\boldsymbol{A}_{p}\in\mathbb{R}^{d}$.
Different from the SWA scheme [38], we introduce a bandwidth parameter $h$ to
control the uniform distribution between the $\epsilon$ sampling of the
perturbation $\delta_{\boldsymbol{A}}$ in Eq.(7). Specifically, the sequence
of successive locations of the Gaussian kernel $G$ is denoted by
$\\{{\boldsymbol{A}}^{t}\\}_{t=0,1,...,T}$. Using Eq.(5), we estimate
${\delta}_{\boldsymbol{A}}^{t}$ as:

${\delta}_{\boldsymbol{A}}^{t}=\frac{\sum\limits_{p=1}^{N}\;\boldsymbol{A}^{t}_{p}\;\omega_{p}\;G\left(\|\frac{{\boldsymbol{A}}^{t}-\boldsymbol{A}^{t}_{p}}{h}\|^{2}\right)}{\sum\limits_{p=1}^{N}\;\omega_{p}\;G\left(\|\frac{{\boldsymbol{A}}^{t}-\boldsymbol{A}^{t}_{p}}{h}\|^{2}\right)}-{{\boldsymbol{A}}}^{t},\;t=1,2,3,...,T$
(11)

where $\omega_{p}$ is used to weight ${{\boldsymbol{A}}^{t}_{p}}$. Following
the previous optimization works, giving higher weights to worse cases result
in faster convergence and efficiently generate the adapted weights. We also
let the value of $\omega_{p}$ is made proportional to the validation loss of
${{\boldsymbol{A}}_{p}}$. This design can increase the impact of a sampling
architecture ${{\boldsymbol{A}}_{p}}$ producing worse accuracy over other
samples in our MS sampling process. This can enhance the desired landscape
flattening that can improve robust architecture search.

Given an initial architecture $\boldsymbol{A}^{0}$, mean-shift filtering is
applied to obtain a weight, smoothed architecture $\overline{\boldsymbol{A}}$
following a few iterative steps. Given the initial architecture
${\boldsymbol{A}}^{0}$ at $t=0$, we take $N$ samples
$\\{\boldsymbol{A}_{p}\\}_{p=1,...,N}$ around it within a radius $\epsilon$.
Eq.(11) is then used to compute ${\delta}_{\boldsymbol{A}}^{t=1}$ to obtain
${\boldsymbol{A}}^{1}$. These steps can go on iteratively until convergence.
Algorithm 1 describes detailed steps of this MS update scheme steps in
calculating ${\overline{\boldsymbol{A}}}$.

1 Input: Architecture ($\boldsymbol{A},W$), bandwidth ($h$), Sampling Number
($N$), Max Iterations($T$);

2 Output: New Architecture (${\overline{\boldsymbol{A}}}$)

3 Initialize $t=0$ and ${\boldsymbol{A}}^{0}=\boldsymbol{A}$;

4 do

5 for  _$i\in N$_ do

6 Sample and get ${\boldsymbol{A}}^{t}_{p}$ around ${\boldsymbol{A}}^{t}$
within a radius $\epsilon$;

7 Get $\omega_{p}$ based on the validation loss of ${\boldsymbol{A}}^{t}_{p}$;

8 Compute ${\delta}_{\boldsymbol{A}}^{t}$ based on Eq.(11);

9 ${\boldsymbol{A}}^{t+1}={\boldsymbol{A}}^{t}+{\delta}_{\boldsymbol{A}}^{t}$;

10 $t=t+1$;

11

12while  _ $t<T$ _;

return ${\overline{\boldsymbol{A}}}={\boldsymbol{A}}^{t+1}$

Algorithm 1 Mean-Shift Iteratively Updating Algorithm

1 Input: Architecture ($\boldsymbol{A},W$), bandwidth ($h$), Sampling Number
($N$), Max Iterations ($T$);

2 Output: New Architecture ($\boldsymbol{A}^{*}$)

3 while  _not converged_ do

4 Determine $\boldsymbol{A}^{*}$ by minimizing
${\mathcal{L}_{val}}(\boldsymbol{A},W)$ with Eq.(10).

5 Compute $\overline{\boldsymbol{A}}$ by Algorithm 1 with parameters (
$\boldsymbol{A}^{*},W$, $h$, $N$, $T$).

6 Update $\overline{W}({\boldsymbol{A}})$ by minimizing
${\mathcal{L}_{train}}(\overline{\boldsymbol{A}},W)$ with Eq.(9).

7 $\boldsymbol{A}=\boldsymbol{A}^{*}$; $W=\overline{W}(\boldsymbol{A})$.

8

9return Final Architecture $(\boldsymbol{A},W)$

Algorithm 2 MS-DARTS: Updating Architecture Before Mean-shift

MS-DARTS search and update scheme. DARTS first search for an initial
architecture $\boldsymbol{A}$. Mean-shift is then adopted to sample for
architectures within the bandwidth and find out poorly-performed
architectures. As discussed before, we weight more on pooly-performed
architectures in consideration to better avoid trapping into bad local
minimums. Eq.(9) is then applied to update weights. Algorithm 2 describes the
details of this iterative search and updating scheme for MS-DARTS.

###  3.4 MS-DARTS Bandwidth Effects Searching Stability

The bandwidth parameter $h$ in Eq.(11) controls the smoothing effect of the
architecture search in MS-DARTS. We next discuss the effects of $h$ w.r.t. the
stability of DARTS. We define the kernel function with a bandwidth parameter
$h$ as $K_{h}(\boldsymbol{A})=(2\pi
h)^{-2d}\exp{(-\frac{1}{2}\|\frac{\boldsymbol{A}}{h}\|^{2})},$ and represent
the DARTS loss function $\mathcal{L}(\boldsymbol{A})$ as:

$\mathcal{L}(\boldsymbol{A})=\frac{1}{N}\sum\limits_{i=1}^{N}{K_{h}(\boldsymbol{A}-\boldsymbol{A}_{p})}.$
(12)

To optimize $\mathcal{L}(\boldsymbol{A})$, let
$\tilde{\boldsymbol{A}}=\boldsymbol{A}-\boldsymbol{A}_{p}$, the Hessian
$K_{h}$ is calculated as:

$\nabla^{2}K_{h}(\tilde{\boldsymbol{A}})=\frac{1}{h}(\frac{1}{h}\tilde{\boldsymbol{A}}\tilde{\boldsymbol{A}}^{T}-\boldsymbol{I})K_{h}(\tilde{\boldsymbol{A}}),$
(13)

where $\boldsymbol{I}$ denotes identity matrix. The eigenvalues of
$\nabla^{2}K_{h}(\tilde{\boldsymbol{A}})$ are:

$(\frac{1}{h^{2}}\|{\tilde{\boldsymbol{A}}}\|^{2}-\frac{1}{h})K_{h}(\tilde{\boldsymbol{A}})\text{
and }\frac{-1}{h}K_{h}(\tilde{\boldsymbol{A}}).$ (14)

If $\|{\tilde{\boldsymbol{A}}}\|^{2}<2h$, the largest absolute eigenvalue is
$\frac{1}{h}K_{h}(\tilde{\boldsymbol{A}})$; otherwise, it is
$(\frac{1}{h^{2}}\|{\tilde{\boldsymbol{A}}}\|^{2}-\frac{1}{h})K_{h}(\tilde{\boldsymbol{A}})$.
Note that the largest eigenvalue of $\nabla^{2}\mathcal{L}({\boldsymbol{A}})$
is related to the sum of eigenvalues of
$\nabla^{2}K_{h}(\boldsymbol{A}-\boldsymbol{A}_{p})$. From Eq.(14), the
eigenvalues decrease according to the bandwidth $h$. The recent work [11]
shows that the performance of DARTS strongly depends on the largest absolute
eigenvalue $\nabla^{2}\mathcal{L}(\boldsymbol{A})$. During optimization, the
smaller this largest eigenvalue, the better the DARTS performance. Our
experimental results show that when a larger bandwidth $h$ is chosen, both the
accuracy and convergence rate are indeed improving. This shows that the
performance of MS-DARTS is highly influenced by the choice of the bandwidth
$h$. While it is obvious that a too small bandwidth leads to noisy results,
and too large bandwidth leads to over-smoothed landscapes, we empirically
determine a reasonably large bandwidth $h$ that produces the best desired
smoothed landscape. A good strategy is to track the bandwidth $h$ at each
sample along the optimization trajectory, in order to find the best
architecture. We thus make a reasonable claim that larger bandwidth $h$ leads
to better smoothed landscape. However, too large bandwidth $h$ can over-smooth
the landscape. On the other hand, a bandwidth that is too small will lead to
noisy results, where important data samples are not fully explored.

##  4 Experiments and Results

In this section, we first experiment our MS-DARTS across 3 search spaces on
NAS-bench-1shot1 [41] dataset to test the searching stability via each epoch
eigenvalue. Lower eigenvalue means the more stable in searching stage. Then we
evaluate MS-DARTS on the architecture parameters A value to prove that our
method has wider minima. Furthermore, we evaluate MS-DARTS on CIFAR-10 [15],
CIFAR-100 [42], and ImageNet [43] datasets for image classification tasks, and
compare its efficiency and accuracy against state-of-the-art (SoTA) NAS
models. Following existing works [11, 32] on experimental setup, seven
operators including skip connections were used to create 4 different search
spaces. We further transfer the cells found on CIFAR-10 to CIFAR-100 and
ImageNet for testing. Finally, we test several bandwidth ($h$) values to prove
our proposal in chapter 3.4. The following experiments show that the suitable
bandwidth ($h$) will efficiently effect the stability of MS-DARTS searching
stage.

###  4.1 Experiments on NAS-bench-1shot1

This dataset [41] consists of 3 search spaces based on CIFAR-10, with a
provided mapping between the continuous space of differentiable NAS to the
discrete space. Details are described in [11, 32]. Figure.3 shows the
comparison of our models against five SoTA methods, namely DARTS [3], PC-DARTS
[8], GDAS [44], and SDARTS-RS/ADV [32] on all 3 search spaces. We run every
NAS algorithm for 100 epochs to allow thorough and comprehensive analysis on
search stability and generalizability. All performance comparisons were
evaluated on a V100 GPU. PC-DARTS outperforms the original DARTS. GDAS,
SDARTS-RS, and SDARTS-ADV outperform PC-DARTS. However, GDAS suffers from a
pre-mature convergence to sub-optimal architectures. Our MS-DARTS outperform
all five comparison models. As shown in Figure.3. The final eigenvalue of MS-
DARTS is lower than all others, showing its capability in searching for DARTS
searching stability.

(a) NAS-bench-1shot1 search space 1

(b) NAS-bench-1shot1 search space 2

(c) NAS-bench-1shot1 search space 3

Figure 3:  Comparison of trajectories of the largest eigenvalues of Hessian
matrices for architecture search on NAS-bench-1shot1.

###  4.2 Experiments of Wider Minima

Let $\alpha_{t}$ denote the t-th architecture obtained at the t-th epoch and
$\alpha_{opt}$ be the final architecture after optimization. Then, we can
define the $\alpha$-distance for $\alpha_{t}$ as
$||\alpha_{t}-\alpha_{opt}||$. To prove our method can get wider minima, the
training-loss vs. the $\alpha$-distance on CIFAR-10 was plotted in Figure.4.
Clearly, the shape of train loss vs. $\alpha$–distance curve is considerably
wider for our MS-DARTS than for the original DARTS, suggesting that MS-DARTS
indeed converges to a wider solution. It also proves the stability of our MS-
DARTS is better than the original DARTS. Although the original DARTS gets
lower training errors than our MS-DARTS, its sharp minima lead to higher
evaluation testing errors than our MS-DARTS. It is noticed that the evaluation
testing error for MS-DARTS is $2.51$ which is lower than the one for the
original DARTS on CIFAR-10, i.e., $2.76$ (see Table 1). Furthermore, we find
that our MS-DARTS converges faster than previous DARTS works during
experiments in Figure.4 since wider minima works. We further reduce the
searching epochs from 50 to 40, this change gets lower training search time.
The validation accuracy curves also prove the same observation and conclusion.
The validation accuracy curve of MS-DARTS is flatter than the one of the
original DARTS. A flatter accuracy surface makes our MS-DARTS stable to get
higher test-accuracy although its validation accuracy is lower than DARTS.

Figure 4:  $\alpha$-distance during training search on Cifar-10.

###  4.3 Experiments on Cifar10 dataset

Training on CIFAR-10. We apply MS-DARTS to determine the convolutional cells
on CIFAR-10 [15] and then enlarges the network by stacking the learned cells.
Figure.5(a) shows the learned cell with $7$ ordered nodes ($2$ input nodes,
$4$ intermediate nodes, and $1$ output node); $8$ of such learned cells are
stacked to build the enlarged network. Each of these nodes is connected to the
previous nodes with forecasted operations. The learned cell is also connected
with the output from two previous cells. The continuous searching architecture
following in DARTS [3] is including the following operations:
max_pooling_3$\times$3, avg_pooling_3$\times$3, skip_connection,
sep_conv_$3\times$3, sep_conv_$5\times$5, dil_conv_$3\times$3,
dil_conv_5$\times$5. Different from [45], our results show that both the
normal and reduction cells consist of combination of operations that are
automatically selected from the search spaces. We set $40$ epochs with batch
size $64$ on the CIFAR-10 dataset, which was equally split into training and
validation set. We adopt the same weight optimization as in [3] using the SGD
optimizer with momentum $0.9$, weight decay $3\times{10}^{-4}$, and learning
rate annealed from $0.025$ to ${10}^{-3}$. We found that $T=2,\ 3$ performs
the best out of experiments of $T={2,\ 3,\ 4,\ 5}$, and $N={2,\ 3,\ 4}$ for
different search spaces and datasets. For search space s5, every epoch takes
nearly about 6 minutes on average on a single GPU. Architecture search
training takes about 11 hrs.

Evaluation on CIFAR-10. The parent architecture consists of $20$ learned cells
($18$ normal cells and $2$ reduction cells) and $36$ channels. Figure.5(b)
shows the evaluation architecture. Table 1 shows the evaluation of $(mean\pm
std)$ in $4$ independents runs with random seeds. The best architecture was
selected based on accuracy. Compared with SDARTS-ADV[32], MS-DARTS only takes
less than half of the training time with improved accuracy. Figure.6 shows the
normal and reduction cells of MS-DARTS on CIFAR-10 dataset, where test error
was $2.51\pm 0.02$.

(a)  (b)

Figure 5:  Training search and evaluation stacked architecture (a) The MS-
DARTS searched architecture on CIFAR-10. (b) MS-DARTS generated architecture
used for evaluation on CIFAR-10.

(a)

(b)

Figure 6:  MS-DARTS Normal and Reduce Cell. (a) MS-DARTS Normal Cell (b) MS-DARTS Reduction Cell.  Architecture |  Mean test error $(\%)$ |  Params (M) |  Search cost (GPU days) |  Search method
---|---|---|---|---
DenseNet-BC [22] | 3.46 | 25.6 | - | manual
NASNet-A [6] | 2.65 | 3.3 | 2000 | RL
ENAS [7] | 2.89 | 4.6 | 0.5 | RL
NAS [2] | 4.47 | 7.1 | 1800 | RL
AmoebaNet-A [4] | 3:34 $\pm 0.06$ | 3.2 | 3150 | evolution
AmoebaNet-B [4] | 2:55 $\pm 0.05$ | 2.8 | 3150 | evolution
Hierarchical Evolution [5] | 3.75 | 15.7 | 300 | evolution
PNAS [46] | 3:41 $\pm 0.09$ | 3.2 | 225 | SMBO
NAONet [47] | 3.53 | 3.1 | 0.4 | NAO
SNAS (mild) +cutout [9] | 2.98 | 2.9 | 1.5 | gradient-based
GDAS + cutout [44] | 2.93 | 3.4 | 0.2 | gradient-based
DARTS (2nd) [3] | 2.76 | 3.4 | 0.3 | gradient-based
PC-DARTS +cutout [8] | 2.57 | 3.6 | 0.1 | gradient-based
LA-DARTS +cutout [10] | 2.72 | 2.7 | 0.7 † | gradient-based
SDARTS-RS [32] | 2.67 $\pm 0.03$ | 3.4 | 0.4 † | gradient-based
SDARTS-ADV [32] | 2.61 $\pm 0.02$ | 3.3 | 1.3 † | gradient-based
MS-DARTS | 2.51 $\pm 0.02$ | 3.8 | 0.4 † | gradient-based
Table 1: Comparison with SoTA image classifiers on CIFAR-10. † Tested on a
Tesla-V100 GPU.

###  4.4 Experiments on parameter-free operations

As chapter 2.3 mentioned, excessive parameter-free operations such as noise,
skip-connection and null let the generated architecture’s performance crash,
though these operations in mixed-up continuous architecture could make
architecture converge faster in training search process. R-DARTS [11] proposed
4 simplified search spaces which only contain parameter-free and a portion of
candidate operations to test searching algorithm’s regularizations. We
implement our MS-DARTS and previous DARTS works across 2 datasets
(Cifar-10/Cifar-100) on these 4 search spaces (S1-S4). Training search on
CIFAR-10 / Cifar-100. As same setting in chapter 4.3, the generated normal
cell and reduction cell by our MS-DARTS searching algorithm will be
transferred to Figure.5(b) architecture. The results are shown in Table 2.

Evaluation on CIFAR-100. Contrary to Cifar-10, Cifar-100 has 100 classes and
more training/testing images. We take the generated architecture (learned
cells) created with CIFAR-100 and evaluate on CIFAR-100 [42]. To fairly
compare with other DARTS algorithms, we using the same training setting as
previous works which is different from chapter 4.2, the parent architecture
consists of $8$ learned cells and $16$ channels. Channel numbers are doubled
after each reduction cells. The parent architecture was trained by $600$
epochs using batch size $64$ and SGD optimizer with momentum $0.9$ and
learning rate cosine scheduled from $0.025$ to 0. For regularization, schedule
drop-path was used with $0$ to $0.2$ linear increase, auxiliary towers with
weight $0.4$, and cutout [48] data augmentation. In this experiment, all other
parameters and optimization functions are kept intact. The training and
validation tasks were performed using CIFAR-100 dataset with $4$ search spaces
(see R-DARTS [11]). Table 2 shows the comparisons of MS-DARTS methods against
other SoTA methods transferred from CIFAR-10 training to CIFAR-100 evaluation
in the $4$ search spaces. Clearly, SDARTS-ADV outperforms DARTS, R-DARTS,
DARTS-ES, and PC-DARTS. Our MS-DARTS also outperforms all SoTA DARTS methods
with lower or similar variations.

Dataset CIFAR-10 CIFAR-100 Method / Space S1 S2 S3 S4 S1 S2 S3 S4 DARTS 3.84
4.85 3.34 7.20 29.46 26.05 28.90 22.85 PC-DARTS 3.11 3.02 2.51 3.02 24.69
22.48 21.69 21.50 DARTS-ES 3.01 3.26 2.74 3.71 28.37 23.25 23.73 21.26 R-DARTS
(DP) 3.11 3.48 2.93 3.58 25.93 22.30 22.36 22.18 R-DARTS(L2) 2.78 3.31 2.51
3.56 24.25 22.44 23.99 21.94 SDATRS-RS 2.78 2.75 2.53 2.93 23.51 22.28 21.09
21.46 SDARTS-ADV 2.73 2.65 2.49 2.87 22.23 20.56 21.08 21.25 MS-DARTS 2.70
2.47 2.46 2.85 21.14 20.55 20.45 20.79

Table 2:  Comparison with SoTA DARTS-based methods on CIFAR-10 and CIFAR-100
datasets.

###  4.5 Experiments on ImageNet dataset

Evaluation on ImageNet. ImageNet Classification [43] is a high-resolution
image classification dataset proposed by Alex.et. This dataset consists of
1000 different classes and more 1.2 million images. Although ImageNet has been
released from 2012, it still be the most famous dataset in image
classification tasks. Not only in classification works, many object detection
and image segmentation tasks also use the dataset for training convolution
backbone. In view of this, we next compare MS-DARTS against SoTA methods on
ImageNet. Similar to previous works R-DARTS [11] and S-DARTS [32], the network
we constructed consists of $14$ cells and $48$ channels. Parameters were
trained for $300$ epochs using SGD optimizer with an annealing learning rate
initialized as $0.5$, momentum $0.9$, and weight decay $3\times{10}^{-5}$.
Table 3 compares 11 baseline methods on ImageNet evaluation. Both SDARTS-RS
and SDARTS-ADV perform better than other DARTS by a large margin. MS-DARTS
again outperforms other SoTA DARTS-based methods. Note that additional
regularization techniques including partial channel connection [8] can further
improve the accuracy [32]. For fair comparisons, those regularization
techniques [25] were not included here.

Architecture | Test Error $(\%)$
---|---
| top-1 | top-5
Inception-v1 [49] | 30.1 | 10.1
MobileNet [50] | 29.4 | 10.5
ShuffleNet-v2 [51] | 25.1 | 10.1
NASNet-A [6] | 26.0 | 8.4
PNAS [46] | 25.8 | 8.1
AtomNAS-A [4] | 25.4 | 7.9
MnasNet-92 [31] | 25.2 | 8.0
DARTS [3] | 26.7 | 8.7
SNAS [9] | 27.3 | 9.2
SDARTS-RS [32] | 25.6 | 8.5
SDARTS-ADV [32] | 25.2 | 7.8
MS-DARTS | 24.4 | 7.3
Table 3: Comparison with SoTA image classifiers on ImageNet.

###  4.6 Experiments of Bandwidth Effect w.r.t MS-DARTS Stability

To verify our suppose in chapter 3.4, we setting several numbers of the
hyperparameter – bandwidth (h). At the same setting of other hyperparameter,
likes sampling radius epsilon ($\epsilon$), mean-shift iteration ($T$), number
of sampling points ($N$) and training search epoch ($E$), we test the effect
of MS-DARTS stability with different bandwidth values on NAS-bench-1shot1
dataset. Table 4 shows the eigenvalue (mean$\pm$std) of each bandwidth value
($h$) during training search. Fitting our proposal in above-mentioned, a
reasonably large bandwidth h that produces the best desired smoothed landscape
and lower eigenvalue, but too large bandwidth h can over-smooth the landscape.

bandwidth | Epoch | Test
---|---|---
($h$) | 1$\sim$20 | 21$\sim$40 | 41$\sim$60 | 61$\sim$80 | error
0.2 | 0.59$\pm$0.25 | 0.33$\pm$0.12 | 0.22$\pm$0.10 | 0.18$\pm$0.08 | 0.067
0.4 | 0.58$\pm$0.19 | 0.31$\pm$0.11 | 0.23$\pm$0.09 | 0.19$\pm$0.08 | 0.067
0.6 | 0.59$\pm$0.20 | 0.30$\pm$0.12 | 0.24$\pm$0.07 | 0.16$\pm$0.07 | 0.067
0.8 | 0.49$\pm$0.13 | 0.32$\pm$0.12 | 0.22$\pm$0.05 |  0.15$\pm$0.06 | 0.064
1.0 | 0.58$\pm$0.28 | 0.30$\pm$0.12 | 0.23$\pm$0.09 | 0.15$\pm$0.04 | 0.061
1.2 | 0.52$\pm$0.18 | 0.30$\pm$0.09 | 0.24$\pm$0.08 | 0.17$\pm$0.09 | 0.064
1.4 | 0.53$\pm$0.19 | 0.36$\pm$0.11 | 0.23$\pm$0.08 | 0.17$\pm$0.06 | 0.067
Table 4:  Comparison eigenvalues with several bandwidth value on
NASbench-1shot1 s1.

##  5 Conclusion

We presented a mean-shift based DARTS approach to stabilize and improve DARTS
network architecture search. Experimental results also prove that the mean-
shift design can smooth out the sharp minima caused by DARTS-based NAS to
flatter minima from which significant performance drop can be avoided. We also
investigated the selection of the bandwidth hyper-parameter and the
convergence of the proposed mean-shift filtering. Extensive experiments on
CIFAR-10, CIFAR-100, and ImageNet show that our method outperforms various
state-of-the-art DART methods. The proposed DARTS stabilization is
advantageous and general. MS-DARTS is useful for practical applicability that
it can work on high-resolution images provided sufficient GPU computation.

Future work. MS-DARTS can be extended to explore additional network components
in other application domains, to generate models with better accuracy in
reduced time.

## References

  * [1] Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter.  Neural architecture search: A survey.  JMLR, pages 1–21, 2019.
  * [2] Barret Zoph and Quoc V Le.  Neural architecture search with reinforcement learning.  ICLR, 2017.
  * [3] Hanxiao Liu, Karen Simonyan, and Yiming Yang.  DARTS: Differentiable architecture search.  In ICLR, 2019.
  * [4] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le.  Regularized evolution for image classifier architecture search.  In AAAI, volume 33, pages 4780–4789, 2019.
  * [5] Hanxiao Liu, Karen Simonyan, Oriol Vinyals, Chrisantha Fernando, and Koray Kavukcuoglu.  Hierarchical representations for efficient architecture search.  In ICLR, 2018.
  * [6] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le.  Learning transferable architectures for scalable image recognition.  In CVPR, pages 8697–8710, 2018.
  * [7] Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean.  Efficient neural architecture search via parameters sharing.  In ICML, pages 4095–4104, 2018.
  * [8] Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi, Qi Tian, and Hongkai Xiong.  PC-DARTS: Partial channel connections for memory-efficient architecture search.  In ICLR, 2020.
  * [9] Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin.  SNAS: stochastic neural architecture search.  In ICLR, 2019.
  * [10] Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Bowen Shi, Qi Tian, and Hongkai Xiong.  Latency-aware differentiable neural architecture search.  arXiv:2001.06392, 2020.
  * [11] Arber Zela, Thomas Elsken, Tonmoy Saikia, Yassine Marrakchi, Thomas Brox, and Frank Hutter.  Understanding and robustifying differentiable architecture search.  In ICLR, 2020.
  * [12] Kaicheng Yu, Christian Sciuto, Martin Jaggi, Claudiu Musat, and Mathieu Salzmann.  Evaluating the search phase of neural architecture search.  ICLR, 2020.
  * [13] Yao Shu, Wei Wang, and Shaofeng Cai.  Understanding architectures learnt by cell-based neural architecture search.  In ICLR, 2019.
  * [14] Yunjie Tian, Chang Liu, Lingxi Xie, Jianbin Jiao, and Qixiang Ye.  Discretization-aware architecture search.  arXiv:2007.03154, 2020.
  * [15] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton.  CIFAR-10 dataset.  http://www.cs.toronto.edu/~kriz/cifar.html.
  * [16] Dorin Comaniciu and Peter Meer.  Mean shift: A robust approach toward feature space analysis.  IEEE Trans. PAMI, 24(5):603–619, 2002.
  * [17] Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar.  Designing neural network architectures using reinforcement learning.  ICLR, 2017.
  * [18] Changlin Li, Jiefeng Peng, Liuchun Yuan, Guangrun Wang, Xiaodan Liang, Liang Lin, and Xiaojun Chang.  Block-wisely supervised neural architecture search with knowledge distillation.  In CVPR, pages 1989–1998, 2020.
  * [19] Richard Shin, Charles Packer, and Dawn Song.  Differentiable neural network architecture search.  In CVPR, 2018.
  * [20] Karim Ahmed and Lorenzo Torresani.  Maskconnect: Connectivity learning by gradient descent.  In ECCV, pages 349–365, 2018.
  * [21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, pages 770–778, 2016.
  * [22] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger.  Densely connected convolutional networks.  In CVPR, pages 4700–4708, 2017.
  * [23] Shreyas Saxena and Jakob Verbeek.  Convolutional neural fabrics.  In NeurIPS, pages 4053–4061, 2016.
  * [24] Tom Veniat and Ludovic Denoyer.  Learning time/memory-efficient deep architectures with budgeted super networks.  In CVPR, pages 3492–3500, 2018.
  * [25] Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian.  Progressive differentiable architecture search: Bridging the depth gap between search and evaluation.  In ICCV, pages 1294–1303, 2019.
  * [26] Niv Nayman, Asaf Noy, Tal Ridnik, Itamar Friedman, Rong Jin, and Lihi Zelnik.  XNAS: Neural architecture search with expert advice.  In NeurIPS, pages 1977–1987, 2019.
  * [27] Hanwen Liang, Shifeng Zhang, Jiacheng Sun, Xingqiu He, Weiran Huang, Kechen Zhuang, and Zhenguo Li.  DARTS+: Improved differentiable architecture search with early stopping.  arXiv:1909.06035, 2019.
  * [28] Guohao Li, Guocheng Qian, Itzel C Delgadillo, Matthias Muller, Ali Thabet, and Bernard Ghanem.  SGAS: Sequential greedy architecture search.  In CVPR, pages 1620–1630, 2020.
  * [29] Xiangxiang Chu, Tianbao Zhou, Bo Zhang, and Jixiang Li.  Fair DARTS: Eliminating unfair advantages in differentiable architecture search.  In ECCV, 2020.
  * [30] Liam Li and Ameet Talwalkar.  Random search and reproducibility for neural architecture search.  In Uncertainty in Artificial Intelligence, pages 367–377. PMLR, 2020.
  * [31] Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V Le.  MnasNet: Platform-aware neural architecture search for mobile.  In CVPR, pages 2820–2828, 2019.
  * [32] Xiangning Chen and Cho-Jui Hsieh.  Stabilizing differentiable architecture search via perturbation-based regularization.  In ICML, 2020.
  * [33] Keinosuke Fukunaga and Larry Hostetler.  The estimation of the gradient of a density function, with applications in pattern recognition.  IEEE Transactions on Information Theory, 21(1):32–40, 1975.
  * [34] Yizong Cheng.  Mean shift, mode seeking, and clustering.  IEEE Trans. PAMI, 17(8):790–799, 1995.
  * [35] Bernard W Silverman.  Density estimation for statistics and data analysis, volume 26.  CRC press, 1986.
  * [36] Stanislaw Jastrzebski, Zachary Kenton, D. Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and A. Storkey.  Finding flatter minima with sgd.  In ICLR, 2018.
  * [37] Sepp Hochreiter and Jurgen Schmidhuber.  Flat minima.  Neural computation, 9:1–42, 02 1997.
  * [38] Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry P. Vetrov, and Andrew Gordon Wilson.  Averaging weights leads to wider optima and better generalization.  CoRR, abs/1803.05407, 2018.
  * [39] Sébastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio.  On using very large target vocabulary for neural machine translation.  CoRR, abs/1412.2007, 2014.
  * [40] Yakov Z. Tsypkin and S. J. Nikolic.  Adaptation and Learning in Automatic Systems.  Academic Press, Inc., USA, 1971.
  * [41] Arber Zela, Julien Siems, and Frank Hutter.  Nas-bench-1shot1: Benchmarking and dissecting one-shot neural architecture search.  In ICLR, 2020.
  * [42] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton.  Cifar-100 (canadian institute for advanced research).
  * [43] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei.  ImageNet: A large-scale hierarchical image database.  In CVPR, 2009.
  * [44] Xuanyi Dong and Yi Yang.  Searching for a robust neural architecture in four GPU hours.  In CVPR, pages 1761–1770, 2019.
  * [45] Quanming Yao, Ju Xu, Wei-Wei Tu, and Zhanxing Zhu.  Efficient neural architecture search via proximal iterations.  In AAAI, pages 6664–6671, 2020.
  * [46] Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy.  Progressive neural architecture search.  In ECCV, pages 19–34, 2018.
  * [47] Renqian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tie-Yan Liu.  Neural architecture optimization.  In NeurIPS, pages 7816–7827, 2018.
  * [48] Terrance DeVries and Graham W Taylor.  Improved regularization of convolutional neural networks with cutout.  arXiv:1708.04552, 2017.
  * [49] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich.  Going deeper with convolutions.  In CVPR, pages 1–9, 2015.
  * [50] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam.  MobileNets: Efficient convolutional neural networks for mobile vision applications.  arXiv:1704.04861, 2017.
  * [51] Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun.  ShuffleNet v2: Practical guidelines for efficient CNN architecture design.  In ECCV, 2018.