Date of publication xxxx 00, 0000, date of current version xxxx 00, 0000.
10.1109/ACCESS.2017.DOI

The research reported here was supported in part by the Defense Advanced
Research Projects Agency (DARPA) under contract number HR0011-17-2-0045. The
views and conclusions contained herein are those of the authors and should not
be interpreted as necessarily representing the official policies or
endorsements, either expressed or implied, of DARPA.

Corresponding author: Biswadeep Chakraborty (e-mail: biswadeep@gatech.edu). .

#  $\mu$DARTS: Model Uncertainty-Aware Differentiable Architecture Search

BISWADEEP CHAKRABORTY1 and  SAIBAL MUKHOPADHYAY1  Department of Electrical and
Computer Engineering, Georgia Institute of Technology, Atlanta, GA, 30332 USA

###### Abstract

We present a Model Uncertainty-aware Differentiable ARchiTecture Search
($\mu$DARTS) that optimizes neural networks to simultaneously achieve high
accuracy and low uncertainty. We introduce concrete dropout within DARTS cells
and include a Monte-Carlo regularizer within the training loss to optimize the
concrete dropout probabilities. A predictive variance term is introduced in
the validation loss to enable searching for architecture with minimal model
uncertainty. The experiments on CIFAR10, CIFAR100, SVHN, and ImageNet verify
the effectiveness of $\mu$DARTS in improving accuracy and reducing uncertainty
compared to existing DARTS methods. Moreover, the final architecture obtained
from $\mu$DARTS shows higher robustness to noise at the input image and model
parameters compared to the architecture obtained from existing DARTS methods.

###### Index Terms:

Uncertainty Estimation, Neural Architecture Search, Monte Carlo Dropout,
DARTS, ImageNet, Image classification.

=-15pt

##  I Introduction

Uncertainty estimation of neural networks is a critical challenge for many
practical application[1, 8, 9]. We can approximate the uncertainty of a neural
network by incorporating Monte-Carlo dropout [2]. Gal et al. used concrete
dropout as a continuous relaxation of dropout’s discrete masks to improve
accuracy and provide better uncertainty calibration [3]. For a given training
dataset, the model uncertainty of a network depends on its architecture.
However, no prior work aims to optimize network architectures to reduce model
uncertainty.

We approach this problem by exploring the neural architecture search (NAS)
[12, 11, 13, 4, 5]. Liu et al. have proposed a differentiable architecture
search (DARTS) [6] that uses continuous relaxation of the architecture
representation allowing an efficient architecture search using gradient
descent, thereby improving the efficiency of NAS. The DARTS perform a bi-level
optimization problem where an outer loop searches over architectures using a
validation loss ($\mathcal{L}_{\text{valid}}$) and the inner loop optimizes
parameters (weights) for each architecture using a training loss
($\mathcal{L}_{\text{train}}$).

We advance the baseline DARTS framework by introducing: (a) concrete dropout
[3] layers within DARTS cell to enable a well-calibrated uncertainty
estimation; (b) a Monte-Carlo dropout based regularizer [2] within
$\mathcal{L}_{\text{train}}$ to optimize dropout probabilities; and (c) a
predictive variance term in $\mathcal{L}_{\text{valid}}$ to search for
architecture with minimal model uncertainty. The proposed architecture search
is defined as Model Uncertainty-aware Differentiable ARchiTecture Search
($\mu$DARTS). This paper makes the following key contributions:

  * •

We develop the $\mu$DARTS framework and training process to improve accuracy
and simultaneously reduce model uncertainty of neural network.

  * •

We show that the architecture search process in $\mu$DARTS converges to a
flatter minima compared to the standard DARTS method.

  * •

We show that the final architecture obtained via $\mu$DARTS converges to a
flatter minima during model training compared to the model obtained using the
standard DARTS method.

  * •

We test the final DNN models obtained from architecture search methods on the
CIFAR10, CIFAR100, SVHN, and ImageNet datasets. We show that the $\mu$DARTS
method improves the accuracy and uncertainty of the final DNN model found
using the architecture search.

  * •

We also showed that the $\mu$DARTS method has better performance when
subjected to input noise and generalizes well when tested with parameter noise

This paper aims to find the architecture which not only maximizes the accuracy
but also minimizes the predictive uncertainty of the model. We do so using the
predictive variance as the regularizer of the bi-level objective function of
the DARTS architecture search method. This new architecture search method
finds architecture with lower model uncertainty and better generalizability as
it induces an implicit regularization on the Hessian of the loss function,
leading to more generalizable solutions for the architecture search process.

The rest of the paper is organized as follows: Section II discusses the
background and related works to this paper, Section III discusses the
theoretical description of the novel Model Uncertainty Aware DARTS methodology
proposed in this paper, while Section IV revolves around the implementation
details about the architecture search baselines for comparison with the
proposed $\mu$DARTS method. Section V deals with the experiments undertaken
and the results obtained thereby while finally, Section VI summarizes and
discusses the conclusions we arrived at from the experiments.

##  II Background and Related Works

###  II-A Uncertainty Estimation using Concrete Dropout.

Considering the likelihood function of the models to be a multivariate normal
distribution given as
$p\left(y^{*}|f^{\omega}\left(x^{*}\right)\right)=\mathcal{N}\left(y^{*};f^{\omega}\left(x^{*}\right),\Sigma\right)$,
we can get an approximate estimate of the expected value of the variational
predictive distribution by sampling $T$ sets of weights $\bar{\omega}_{t}$
$(t=1,\ldots,T)$ from the variational dropout distribution[2, 3]:

$\mathbf{E}_{q_{\theta}\left(y^{*}|x^{*}\right)}\left[y^{*}\right]\approx\frac{1}{T}\sum_{\iota}f^{\tilde{\omega}_{t}}\left(x^{*}\right)$
(1)

where $f^{\bar{\omega}_{t}}\left(x^{*}\right)$ denotes a forward pass through
the model with the weights which are sampled $\tilde{\omega}_{t},$ thus
effectively, performing $T$ forward passes through the network $f$ with
dropout and this process is known as Monte Carlo (MC) dropout.

For multiclass classification tasks with softmax likelihood
$p\left(y^{*}|x^{*},\omega\right)=\sigma\left(f^{\omega}\left(x^{*}\right)\right),$
we can approximated the variational predictive distribution as:

$q_{\theta}\left(y^{*}|x^{*}\right)\approx\frac{1}{T}\sum_{t}\sigma\left(f^{\tilde{\omega}_{t}}\left(x^{*}\right)\right)$
(2)

###  II-B Differentiable Architecture Search.

Liu et al. presented DARTS to perform a one-shot neural architectural search.
The DARTS method [6] is based on the principle of continuous relaxation of the
architecture representation, allowing efficient architectural space search
using a gradient descent approach. DARTS formulates the architecture search as
a differentiable problem, thus overcoming the scalability challenges faced by
other RL-based NAS methods. The DARTS optimization procedure is defined as a
bi-level optimization problem where $\mathcal{L}_{\text{val}}$ is the outer
objective and $\mathcal{L}_{\text{train}}$ is the inner objective as:

$\begin{array}[]{l}\min_{\alpha}\mathcal{L}_{val}\left(\alpha,w^{*}(\alpha)\right)\\\
\text{ s.t. }\quad
w^{*}(\alpha)={\arg\min}_{w}\mathcal{L}_{\text{train}}(\alpha,w)\end{array}$
(3)

where the validation loss function $\mathcal{L}_{\text{val}}$ determines the
architecture parameters $\alpha$(outer variables) and the training loss
$\mathcal{L}_{\text{train}}$ optimizes the network weights $w$ (inner
variables).

The computational graph is learned in one go in the DARTS-based architecture
search process. At the end of the search phase, the connections and their
associated operations are pruned, keeping the ones that have the highest
magnitude of their related architecture weights multipliers. In their paper,
Noy et al. [34] showed that the harsh pruning in DARTS, which occurs only once
at the end of the search phase, is sub-optimal and that a gradual pruning of
connections can improve both the search efficiency and accuracy. On the other
hand, Bi et al. [35] solve the problem of searching in a complex search space
by starting with a complete super-network and gradually pruning out weak
operators. Though these methods help in finding efficient architectures, these
methods neither estimate nor try to minimize the uncertainty of the model.

####  II-B1 Improving DARTS Search Space

Many different works have worked on improving the shortcomings of the DARTS
methodologies. For example, in the paper, [31], the authors pointed out that
despite the large gap between the architecture depths in search and evaluation
scenarios, the DARTS method report lower accuracy in evaluating the searched
architecture or when transferring to another architecture. The authors
gradually increased the depth of the searched architectures to address this
issue.

Alternatively, in the paper [32], the authors addressed the issue of large
memory and computing overheads in jointly training a super-network and
searching for an optimal architecture. The authors proposed an approach to
sample a small part of the super-network to reduce the redundancy in exploring
the network space, thereby performing a more efficient search without
comprising the performance. Again, Dong et al. [33] represented the search
space as a DAG to reduce the search time of the architecture search process.

This paper uses the DARTS search space and methodology as the baseline.
However, our method could very well be extended to perform for any such
methodologies mentioned above and search spaces when the aim is to find an
architecture in the search space which not only improves the accuracy but also
minimizes the uncertainty while making it more robust to noise.

####  II-B2 Improving Robustness and Generalizability of DARTS:

Arber et. al. [7] and Chen et.al [30] more stabilized neural architecture
search methods. SmoothDARTS (SDARTS) use a perturbation-based regularisation
to smooth the loss landscape and improve the generalizability. Empirical
results have shown that a generalization performance of the architecture found
by DARTS improves with a lower eigenvalue of the Hessian matrix of the
validation loss for the architectural parameters
($\nabla_{\alpha}^{2}\mathcal{L}_{\text{val}}^{\text{DARTS}}$). RobustDARTS
[7] has been proposed to improve robustness by (i) computing the Hessian and
stopping DARTS early to limit the eigenvalues (converge to a flatter minima)
and (ii) using an L2 regularization term in the training loss
($\mathcal{L}_{\text{train}}$).

###  II-C Neural Architecture Distribution Search:

Ardywibowo et al. showed that searching for a distribution of architectures
that performs well on a given task allows us to identify standard building
blocks among all uncertainty-aware architectures [29]. With this formulation,
the authors optimized a stochastic out-of-distribution detection (OoD)
objective and constructed an ensemble of models to perform OoD detection.
However, the work concentrates on the detection of out-of-distribution
uncertainty by optimizing the Widely Applicable Information Criterion (WAIC),
a penalized likelihood score used as the OoD detection criterion. However,
they do not discuss the model uncertainty that arises from the generalization
error and mainly focus on just the out-of-distribution uncertainty.

###  II-D Contribution of This Work.

The prior works on NAS and DARTS do not focus on minimizing uncertainty.
Therefore, the key contribution of this paper is an architecture search method
that can simultaneously maximize accuracy and reduce model uncertainty.

##  III Model Uncertainty Aware DARTS

We propose $\mu$DARTS, a neural architecture search method to find the optimal
architecture that simultaneously improves the accuracy and reduces the model
uncertainty while also giving a tighter estimate of the model uncertainty
using the concrete dropout layers. We formulate the $\mu$DARTS bi-level
optimization problem as:

$\displaystyle\min_{\alpha}\mathcal{L}^{\text{DARTS}}_{val}\left(\alpha,w^{*}(\alpha)\right)+\operatorname{Var}_{p(\mathbf{y}|\mathbf{x})}^{model}(\alpha,w^{*}(\alpha))$
$\displaystyle\text{ s.t. }\quad
w^{*}(\alpha)={\arg\min}_{w}\mathcal{L}^{\text{DARTS}}_{\text{train}}(\alpha,w)+\mathcal{L}_{\mathrm{MC}}(\theta)$
(4)

where, $\mathcal{L}^{\text{DARTS}}_{\mathrm{MC}}(\theta)$ is the Monte Carlo
dropout loss and
$\operatorname{Var}_{p(\mathbf{y}|\mathbf{x})}^{model}(\alpha,w^{*}(\alpha))$
is the predictive variance. As pointed out by Zela et al. [7], it can be seen
that increasing the inner objective regularization helps to control the
largest eigenvalue and more strongly allows it to find solutions with the
smaller Hessian spectrum and better generalization properties. Again, we
observe the implicit regularization effect on the outer objective, which
reduces the overfitting of the architectural parameters. This is further
discussed in Appendix A. Compared to the optimization problem for DARTS (see
3), we make the following key updates: We know that the validation loss,
$\mathcal{L}_{\text{val}}$, is used for the architecture search process, and
we want our neural architecture search method to search for the architecture
which also has the least predictive uncertainty along with high accuracy.
Hence, we add the predictive variance term to the validation loss term and the
new validation loss function is given as
$\mathcal{L}^{\text{DARTS}}_{val}\left(\alpha,w^{*}(\alpha)\right)+\operatorname{Var}_{p(\mathbf{y}|\mathbf{x})}^{model}(\alpha,w^{*}(\alpha))$,
where $w^{*}$ is the optimal set of weights found by optimizing the training
loss function in the other bi-level optimization problem. The predictive
variance is estimated as the variance of the $T$ Monte Carlo samples on the
network as:

$\operatorname{Var}_{p(\mathbf{y}|\mathbf{x})}^{model}(\mathbf{y})=\sigma_{model}=\sum_{D}\frac{1}{T}\sum_{t=1}^{T}\left(\mathbf{y}_{t}-\bar{\mathbf{y}}\right)^{2}$
(5)

where $\left\\{\mathbf{y}_{l}\right\\}_{t=1}^{T}$ is a set of $T$ sampled
outputs for weights instances $\omega^{l}\sim q(\omega;\Phi)$ and
$y=1/T\sum_{t}y_{t}$.

Again, we added a Monte-Carlo loss function in the training loss function
since the training loss determines the weights of a particular architecture.
To get well-calibrated uncertainty estimates, adapting the dropout probability
as a variational parameter to the data at hand is necessary. So, we use the
concrete dropout layers and add the Monte Carlo loss to calibrate the dropout
probabilities. As shown in [3], the optimization objective that follows from
the variational interpretation can be written as

$\displaystyle\mathcal{L}_{\mathrm{MC}}(\theta)$
$\displaystyle=\frac{1}{N}\mathrm{KL}\left(q_{\theta}(\omega)\|p(\omega)\right)$
$\displaystyle\Rightarrow\mathcal{L}^{\text{$\mu$DARTS}}_{\text{train}}(\alpha,w)$
$\displaystyle=-\frac{1}{M}\sum_{i\in S}\log
p\left(\mathbf{y}_{i}|\mathbf{f}^{\omega}\left(\mathbf{x}_{i}\right)\right)+$
$\displaystyle\frac{1}{N}\mathrm{KL}\left(q_{\theta}(\omega)\|p(\omega)\right)$
(6)

where $\theta$ is the parameters to optimize, $N$ is the number of data
points, $S$ is a random set of $M$ data points,
$\mathbf{f}^{\omega}\left(\mathbf{x}_{i}\right)$ is the neural network’s
output on input $\mathbf{x}_{i}$ when evaluated with weight matrices
realisation $\omega,$ and
$p\left(\mathbf{y}_{i}\mid\mathbf{f}^{\omega}\left(\mathbf{x}_{i}\right)\right)$
is the model’s likelihood, e.g. a Gaussian with mean
$\mathrm{f}^{\omega}\left(\mathrm{x}_{i}\right).$ The KL term
$\mathrm{KL}\left(q_{\theta}(\omega)\|p(\omega)\right)$ is a “regularization”
term which ensures that the approximate posterior $q_{\theta}(\omega)$ does
not deviate too far from the prior distribution $p(\omega).$

The total derivative of $\mathcal{L}_{\text{val}}$ w.r.t. $\alpha$ evaluated
on $\left(\alpha,w^{*}(\alpha)\right)$ would be:

$\displaystyle\frac{d}{d\alpha}(\mathcal{L}_{val}+\operatorname{Var}\left(\mathbf{y}^{*}\right))$
$\displaystyle=\nabla_{\alpha}(\mathcal{L}_{val}+\operatorname{Var}\left(\mathbf{y}^{*}\right))$
$\displaystyle-\nabla_{w}(\mathcal{L}_{val}+\operatorname{Var}\left(\mathbf{y}^{*}\right))$
$\displaystyle\left(\nabla_{w}^{2}\mathcal{L}_{train}+\nabla_{w}^{2}\mathcal{L}_{\mathrm{MC}}(\theta)\right)^{-1}$
$\displaystyle\nabla_{\alpha,w}^{2}(\mathcal{L}_{train}+\mathcal{L}_{\mathrm{MC}}(\theta))$
(7)

where
$\nabla_{\alpha}=\frac{\partial}{\partial\alpha},\nabla_{w}=\frac{\partial}{\partial
w}$ and $\nabla_{\alpha,w}^{2}=\frac{\partial^{2}}{\partial\alpha\partial w}.$
In general, computing the inverse of the Hessian is not possible considering
the high dimensionality of the model parameters $w$. Thus, we use gradient-
based iterative algorithms to find the optimal $w^{*}$. However, to avoid
repeated training of each architecture which is computationally very
expensive, we approximate $w^{*}(\alpha)$ by updating the current model
parameters $w$ using a single gradient descent similar to the approximation
step done in [6]:

$\displaystyle w^{*}(\alpha)\approx
w-\xi\nabla_{w}[\mathcal{L}_{train}(\alpha,w)+\mathcal{L}_{\mathrm{MC}}(\theta)]$
(8) $\displaystyle\Rightarrow\frac{\partial
w^{*}}{\partial\alpha}(\alpha)=-\xi\nabla_{\alpha,w}^{2}[\mathcal{L}_{train}(\alpha,w)+\mathcal{L}_{\mathrm{MC}}(\theta)]$

where $\xi$ is the learning rate for the virtual gradient step, DARTS takes
with respect to the model weights $w$. Therefore, we obtain:

$\displaystyle\frac{d}{d\alpha}\mathcal{(}{L}_{val}+\operatorname{Var}\left(\mathbf{y}^{*}\right))=\nabla_{\alpha}\mathcal{L}_{\text{val}}\left(\alpha,w^{*}\right)+\nabla_{\alpha}\operatorname{Var}\left(\mathbf{y}^{*}\right)$
$\displaystyle-\xi\nabla_{w}[\mathcal{L}_{\text{val}}\left(\alpha,w^{*}\right)+\operatorname{Var}\left(\mathbf{y}^{*}\right)]\nabla_{\alpha,w}^{2}(\mathcal{L}_{\text{train}}\left(\alpha,w^{*}\right)+\mathcal{L}_{\mathrm{MC}}(\theta))$
(9)

where the inverse Hessian $\nabla_{w}^{2}\mathcal{L}_{train}^{-1}$ is replaced
by the learning rate $\xi$.

The final output using the $\mu$DARTS method gives us the optimal
architecture, which has the maximum accuracy and the minimum uncertainty in
the architecture search space. However, there are some benefits to including
the predictive variance term in the validation loss and the Monte Carlo
dropout loss. We hypothesize a two-fold benefit from this modified loss
function structure.

Firstly, the predictive variance term acts as a regularizer to the validation
loss function. It makes the neural architecture search method more robust and
resilient to input or parameter noise. In this paper, we shall prove this
hypothesis empirically by computing the largest eigenvalue of the Hessian of
the validation loss to indicate the flatness of the loss minima of the
architecture search process. Also, analytical proof of this is given in
Appendix B, considering a simplified linear model.

Similarly, the Monte Carlo dropout loss added to the training loss function
acts as another regularizer. Hence, the final architecture obtained from the
neural architecture search method also gives better performance under noise
perturbation. Similarly, we also prove this empirically by computing the
largest eigenvalue of the Hessian of the training loss to indicate that the
architecture converges to a flat minima for the final architecture we got from
the neural architecture search process.

##  IV Implementation of Baselines and $\mu$DARTS

Fig. 1: Flowchart of the $\mu$DARTS method Fig. 2: Summary of Baselines and
$\mu$DARTS. $X$DARTS-CD refers to the baseline architecture search models
viz., P-DARTS-CD, PC-DARTS-CD, RobustDARTS-CD, GOLD-NAS-CD, ASAP-CD described
in Section IV

DARTS: We implemented the standard DARTS method [6] with the search space
constituting of the following operations: $\mathcal{O}:3\times 3,\quad 5\times
5$ separable convolutions (sep conv), $3\times 3,\quad 5\times 5$ dilated
separable convolutions (dil conv), $3\times 3$ max pooling, $3\times 3$
average pooling, identity, and zero (Fig. 1). All the operations in the method
are of unit stride length, wherever applicable, and paddings are added to the
convolved feature maps to preserve their spatial resolution. For the
convolution operations, we use the order ReLU/Conc Drop(ReLu)-Conv-BN as done
in [4, 5]. The output node of the convolutional cells is the depthwise
concatenation of all the intermediary nodes except the input nodes. The
network is then formed by stacking multiple cells following the same principle
used in [6]. We used the implementation of DARTS used in [6] for our results
and ran it for 50 epochs for each searching and training algorithm.

DARTS with Concrete Dropout (DARTS-CD). The final architecture obtained from
DARTS cannot be directly used for uncertainty estimation. We modify the
optimal architecture from the standard DARTS to enable uncertainty estimation.
We include a Concrete dropout [3] layer in all the layers of the final
architecture, and the DARTS-CD is obtained by adding a Monte Carlo dropout
loss to the training loss function $\mathcal{L}_{\text{train}}$ to calculate
the optimal dropout probabilities [3]. We use the re-trained final
architecture to estimate accuracy and uncertainty using multiple MC samples.

RobustDARTS with Concrete Dropout (RDARTS-CD). We implemented the RobustDARTS
(RDARTS) model using the L2 regularization in the training loss, and the early
stopping mechanism [7]. The RDARTS generates a final architecture. Like DARTS-
CD, concrete dropout layers are included in each of the layers in RDARTS for
uncertainty estimation. The modified architecture is re-trained by adding
Monte-Carlo dropout loss in the loss function. The modified final architecture
from RDARTS calculates the model uncertainty and accuracy.

Progressive Differentiable Architecture Search with Concrete Dropout (P-DARTS-
CD) We also implemented the Progressive Differentiable Architecture Search
(P-DARTS) [31]. P-DARTS is an efficient algorithm that gradually increases the
depth of the searched architectures while training. P-DARTS solves the
problems of heavier computational overheads and weaker search stability using
search space approximation and regularization, respectively. In this paper, we
added Concrete Dropout after each layer of the P-DARTS searched architecture
to get an uncertainty estimate using the Monte Carlo Dropout methodology. The
modified network is trained with added Monte Carlo dropout loss. The modified
final architecture from P-DARTS is then used to calculate the accuracy and
model uncertainty.

PC-DARTS-CD Partially-Connected DARTS (PC-DARTS)[32] performs a more efficient
search without comprising the performance by sampling a small part of the
super-network to reduce the redundancy in exploring the network space. Similar
to the previously mentioned architectures, we add Concrete Dropout layers to
the final architecture from the PC-DARTS method and, using the Monte Carlo
Dropout loss in the loss function, get an estimate of the uncertainty of the
model.

GOLD-NAS-CD Gradual One-Level Differentiable Neural Architecture Search (GOLD-
NAS) introduces a variable resource constraint to one-level optimization so
that the weak operators are gradually pruned out from the super-network.
Similar to the other methods, we add additional Concrete Dropout layers after
each layer in the final architecture to get the uncertainty estimate of the
model.

ASAP-CD Architecture Search, Anneal and Prune (ASAP) [34] uses a
differentiable search space that allows the annealing of architecture weights
while gradually pruning inferior operations. In this way, the search converges
to a single output network continuously. Similar to the other methods, we add
Concrete Dropout layers after each layer in the final architecture obtained in
the paper to get the uncertainty estimate of the model.

$\mathbf{\mu}$DARTS. Fig. 1 shows the overall architecture of the proposed
$\mu$DARTS method, including the internal details of a cell. We include the
following operations in $\mathcal{O}$: $3\times 3,5\times 5$ separable
convolutions (sep conv), $3\times 3,5\times 5$ dilated separable convolutions
(dil conv), $3\times 3$ max pooling, $3\times 3$ average pooling, identity,
and zero. The key difference between an $\mu$DARTS cell compared to the
standard DARTS cell is that each cell has an option to have a Concrete dropout
layer. The Concrete dropout layer, if included, enables computation of the
uncertainty values of the model. Wherever applicable, the operations are of
unit stride, and also we use padding in the convolved feature maps. $\mu$DARTS
also includes a Concrete dropout in the final softmax layer of the model.

The comprehensive details for all the methods described above are shown in
Fig. 2.

##  V Experimental Results

###  V-A Training Conditions

The experiments performed in this paper were performed on a single NVIDIA
GeForce GTX 1080 Ti Graphics Card. To get a fair comparison between the
models, the experiments performed in this paper keep the training parameters
constant for all the models. We trained all the methods for $50$ epochs for
the architecture search process with a batch size of $32$ and a learning rate
of $0.05$. After getting the final architecture, we train the final model for
$50$ epochs with a batch size of $64$ and a learning rate of $0.025$ to get
the best results of the model obtained. Table I summarizes the hyperparameters
used in the architecture search.

TABLE I: Table showing the hyperparameters of the architecture search process

Hyperparameters | Values
---|---
Learning Rate | 0.025
Weight Decay | 0.0243
Epochs | 50
Batch Size | 16
No. of cells | 14

The experimental results of the paper are divided into the following
subsections:

  * •

Comparative analysis of the architecture search methods, including ablation
studies, robustness, convergence, and run-time.

  * •

Comparative analysis of the final architecture, including flatness of the loss
surface and testing errors.

  * •

Comparative analysis of the final DNN models, including accuracy and
uncertainty comparisons and tolerance to input and parameter noise.

In the rest of this paper, we primarily train the models under the same
training conditions mentioned above for uniformity and evaluate them to get
the accuracy and uncertainty estimates. For reference, we used the vanilla
models (without adding concrete dropout layers), as implemented in the
original papers, and compared them with the accuracy and training condition of
$\mu$DARTS as implemented in this paper. The results are shown in Appendix C.

###  V-B Analysis of the Architecture Search Methods

The searched architecture using the $\mu$DARTS architecture search method is
shown in Fig. 3

Fig. 3: Figure showing the normal and reduction cell representations searched
using the $\mu$DARTS algorithm

####  V-B1 Ablation Studies

In this section, we perform an ablation study for the two different cases: 1.
removing the Monte Carlo loss function $\mathcal{L}_{\mathrm{MC}}(\theta)$
from the training loss $\mathcal{L}^{\text{DARTS}}_{val}$ 2\. removing the
prediction variance term
$\operatorname{Var}_{p(\mathbf{y}|\mathbf{x})}^{model}(\alpha,w^{*}(\alpha))$
from the validation loss term $\mathcal{L}^{\text{DARTS}}_{train}$.

Role of
$\operatorname{Var}_{p(\mathbf{y}|\mathbf{x})}^{model}(\alpha,w^{*}(\alpha))$:
Including $\mathcal{L}_{\mathrm{MC}}(\theta)$ results in calibrated
uncertainty values in the inner loop of the DARTS optimization loop. However,
if the predictive variance term is removed, the neural architecture search
method is not optimized for the uncertainty values. Removing the predictive
variance loss term but not the MC loss is equivalent to the DARTS-CD
architecture search method. We simulate this ablation study on the CIFAR 10
dataset and compare the performance with the case where we include the
predictive variance term in the training loss function. We plot the variation
of the predictive uncertainty with increasing epochs for $\mu$DARTS and DARTS-
CD in Fig. 4.

Fig. 4: Plot showing the mean and variance for the variation of model
uncertainty, measured using the predictive variance for different architecture
search methods with increasing epochs for CIFAR 10 dataset

In this paper, we search for an architecture that not only maximizes accuracy
but simultaneously minimizes the model uncertainty. We accomplish this by
using the predictive variance as a regularizer of the loss function as given
in Eq. 4. To verify that adding the predictive variance term helps in
searching for architectures with lower predictive uncertainty, we plotted the
evolution of the predictive uncertainty of different architecture search
results with respect to $\mu$DARTS. The results are shown in Fig. 5. We see
that the model uncertainty (measured using the predictive variance) remains
almost constant for models like DARTS-CD and RDARTS-CD, which steadily
decreases for the $\mu$DARTS architecture search method, which uses the
predictive variance as a regularizer of the Cross-Entropy loss function.

Fig. 5: Plot showing the variation of predictive variance for different
architecture search methods with increasing epochs for CIFAR 10 dataset (Note
that the y-axis is represented in the logarithmic scale)

Role of $\mathcal{L}_{\mathrm{MC}}(\theta)$: We see that on removing the Monte
Carlo dropout loss term from the training loss
$\mathcal{L}^{\text{DARTS}}_{val}$, the dropout probabilities in each of the
layers are not updated, and hence the uncertainty is not calibrated.
Therefore, we get a wrongful estimate of the uncertainty of the model. In Fig.
6 we show this by repeating the same experiment with different values of
dropout probabilities and show that each of the dropout probabilities gives
rise to a different uncertainty estimate as discussed in [3]. We also note
that if we remove the Monte Carlo loss function from the architecture search
process while keeping the predictive variance term, we will get an
architecture optimized for an uncalibrated noise. Hence, the optimal
architecture found using this method may not be the best architecture to
minimize the noise.

Fig. 6: Plot showing uncertainty estimates for different values of dropout
probability Fig. 7: Plot showing largest eigenvalue of the Hessian of the
validation loss function for increasing epochs for CIFAR 10 dataset

####  V-B2 Robustness.

We empirically evaluate the robustness of the architecture search methods by
estimating the largest eigenvalue of the Hessian of the validation loss
function ($\mathcal{L}_{\text{valid}}$) for each method. Table II shows that
the $\mu$DARTS method has a smaller value of the largest eigenvalue of the
Hessian of the validation loss compared to the other methods making it more
robust. We plot the evolution of the largest eigenvalue of the Hessian of the
validation loss in Fig. 7. We see that while for the standard DARTS method,
the largest eigenvalue keeps increasing for increasing epochs. However, the
largest eigenvalue for the Hessian of the validation loss function of the
$\mu$DARTS method increases very slowly and is much lower than the other
methods discussed. The RDARTS-CD can generate similar robustness but requires
the implementation of early stopping. The empirical analysis verifies that
including predictive variance
$\left(\operatorname{Var}_{p(\mathbf{y}|\mathbf{x})}^{model}(\alpha,w^{*}(\alpha))\right)$
in the validation loss $\left(\mathcal{L}_{\text{valid}}\right)$ improves
robustness of the architecture search method (an analytical proof of this
property is given in the Appendix assuming a linear model).

TABLE II: Robustness of Architecture Search Methods (Largest Eigenvalues of
the $\nabla_{w}^{2}\mathcal{L}_{\text{valid}}$)

Benchmark | DARTS |  | RDARTS-CD
---
| DARTS-CD
---
|  $\mu$DARTS
---
(This Work)
CIFAR10 | 0.396 | 0.110 | 0.142 | 0.097
CIFAR100 | 0.773 | 0.504 | 0.578 | 0.492
SVHN | 0.202 | 0.045 | 0.079 | 0.021
ImageNet | 0.202 | 0.045 | 0.079 | 0.021

####  V-B3 Convergence and Runtime Analysis

In this section, we discuss the run times and the convergence of the
architecture search method. Fig. 8 shows the validation error of the search
models for each epoch for the CIFAR 10, CIFAR 100, SVHN, and the ImageNet
datasets. $\mu$DARTS shows a faster convergence (or lower validation loss)
than other architectures. Table III shows that the run-time of $\mu$DARTS is
similar to DARTS, DARTS-CD, and RDARTS-CD.

TABLE III: Run-time in GPU hours for different architecture search methods

Dataset | CIFAR10 | CIFAR100 | SVHN | ImageNet
---|---|---|---|---
DARTS | 9.3 | 37.5 | 5.1 | 102.3
RDARTS-CD | 9.7 | 37.6 | 5.9 | 111.7
DARTS-CD | 9.4 | 37.5 | 5.2 | 104.9
$\mu$DARTS | 9.6 | 37.7 | 5.3 | 109.5

TABLE IV: Performance of Final Architectures

Benchmark | DARTS |  | RDARTS-CD
---
(Ours)
| DARTS-CD
---
(Ours)
|  $\mu$DARTS
---
(This Work)
Largest Eigenvalues of the $\nabla_{w}^{2}\mathcal{L}_{\text{training}}$
CIFAR10 | 0.217 | 0.093 | 0.102 | 0.089
CIFAR100 | 0.545 | 0.419 | 0.503 | 0.395
SVHN | 0.176 | 0.039 | 0.061 | 0.017
ImageNet | 0.176 | 0.039 | 0.061 | 0.017
Testing Errors (Mean $\pm$ Standard Deviation)
CIFAR10 | 5.68 $\pm$ 0.27 | 5.11 $\pm$ 0.21 | 5.71 $\pm$ 0.24 | 3.78 $\pm$ 0.20
CIFAR100 | 24.63 $\pm$ 0.81 | 22.67 $\pm$ 0.66 | 25.19 $\pm$ 0.74 | 20.02 $\pm$ 0.40
SVHN | 4.67 $\pm$ 0.19 | 3.93 $\pm$ 0.15 | 4.48 $\pm$ 0.17 | 2.12 $\pm$ 0.127
ImageNet | $29.37\pm 5.03$ | 27.14 $\pm$ 3.97 | 28.71 $\pm$ 4.98 | 25.36 $\pm$ 3.92

###  V-C Comparative Analysis of the Final Architecture

Fig. 8: Convergence of Architecture Search methods for CIFAR10 dataset

Performance of Final Architectures. We compare the performance of the final
architectures obtained from the architecture search methods by computing the
largest eigenvalue of the Hessian of the training loss function
$\mathcal{L}_{\text{train}}$. Table IV shows that the final architecture
obtained from $\mu$DARTS has a lower eigenvalue of the training loss function
indicating convergence to a flatter minima. We further evaluate robustness
considering the mean and standard deviation of errors by training the final
architecture multiple times and testing it on the test dataset. Table IV shows
that $\mu$DARTS has a lower mean and standard deviation of the testing error
compared to DARTS, DARTS-CD, and RDARTS-CD. Hence, we empirically verify that
using the Monte-Carlo dropout loss as a regularizer in the training loss in
$\mu$DARTS, instead of using an L2 regularizer as done in RobustDARTS [7],
improves the robustness of the final architecture. These results prove that
the $\mu$DARTS architecture search method converges to a flatter minima for
each iteration of the bi-level optimization.

Importance of MC Dropout Loss within Bi-level Optimization. Also, it is to be
noted that in the DARTS-CD method, we are manually adding the Concrete Dropout
layers after the architecture search process. So, the comparative analysis of
the DARTS-CD and the $\mu$DARTS shows us that just adding the concrete dropout
without solving the bi-level optimization problem does not provide us with the
optimal solution.

TABLE V: Comparison of Accuracy and Uncertainty of Final Models from Different
Architecture Search Processes (All models are trained for 50 epochs and batch
size of 64)

Model |  FLOPS(M) | CIFAR10 | CIFAR100 | SVHN | ImageNet
---|---|---|---|---|---
|  | Accuracy | Uncertainty | Accuracy | Uncertainty | Accuracy | Uncertainty | Accuracy | Uncertainty
MobileNetv2 [14] | 569 | 88.79 $\pm$ 0.38 | 0.915 | 68.54 $\pm 0.56$ | 1.266 | 93.67 $\pm 0.22$ | 0.247 | 68.83 $\pm$ 3.99 | 1.853
VGG16 [15] | 160 | 84.11 $\pm$ 0.41 | 1.034 | 71.19 $\pm$ 0.52 | 1.482 | 93.15 $\pm$ 0.21 | 0.344 | 62.46 $\pm$ 2.11 | 1.897
ResNet20 [16] | 320 | 86.47 $\pm$ 0.37 | 0.895 | 75.22 $\pm$ 0.46 | 1.246 | 93.74 $\pm$ 0.16 | 0.238 | 65.11 $\pm$ 2.02 | 1.816
EfficientNet-B0 [17] | 390 | 89.01 $\pm$ 0.47 | 1.059 | 76.02 $\pm$ 0.55 | 1.215 | 94.87 $\pm$ 0.24 | 0.291 | 70.83 $\pm$ 4.15 | 1.893
DARTS [6] | 595 | 94.32 $\pm$ 0.27 | N/A | 75.37 $\pm$ 0.81 | N/A | 95.33 $\pm$ 0.19 | N/A | 70.63 $\pm$ 5.03 | NA
P-DARTS-CD | 532 | 95.32 $\pm$ 0.78 | 0.496 | 78.94 $\pm$ 0.89 | 0.883 | 97.79 $\pm$ 0.33 | 0.266 | 72.13 $\pm$ 4.07 | 1.854
PC-DARTS-CD | 557 | 96.47 $\pm$ 0.81 | 0.463 | 80.01 $\pm$ 0.95 | 0.964 | 96.82 $\pm$ 0.36 | 0.279 | 72.81 $\pm$ 4.12 | 1.896
GOLD-NAS-A-CD | 278 | 95.06 $\pm$ 1.03 | 0.412 | 78.97 $\pm$ 1.15 | 0.992 | 96.13 $\pm$ 0.52 | 0.333 | 69.56 $\pm$ 4.95 | 2.038
GOLD-NAS-I-CD | 463 | 93.21 $\pm$ 0.91 | 0.519 | 79.15 $\pm$ 1.06 | 1.003 | 96.71 $\pm$ 0.47 | 0.303 | 72.21 $\pm$ 4.13 | 1.755
ASAP-CD | 573 | 95.72 $\pm$ 1.15 | 0.481 | 79.27 $\pm$ 1.11 | 0.992 | 98.01 $\pm$ 0.63 | 0.484 | 73.96 $\pm$ 4.28 | 1.812
DARTS-CD | 598 | 94.29 $\pm$ 0.24 | 0.474 | 74.81 $\pm$ 0.74 | 1.097 | 95.52 $\pm$ 0.17 | 0.195 | 71.29 $\pm$ 4.98 | 1.712
RDARTS-CD | 605 | 94.89 $\pm$ 0.21 | 0.396 | 77.34 $\pm$ 0.66 | 0.815 | 96.07 $\pm$ 0.15 | 0.189 | 72.86 $\pm$ 3.97 | 1.684
|  $\mu$DARTS
---
(This Work)
602 | 96.22 $\pm$ 0.20 | 0.162 | 79.98 $\pm$ 0.40 | 0.561 | 97.88 $\pm$ 0.11 | 0.127 | 74.64 $\pm$ 3.92 | 1.523

###  V-D Comparative Analysis of the Final Models

####  V-D1 Accuracy and Uncertainty

We compare the performance of $\mu$DARTS considering CIFAR10 , CIFAR100 [19],
SVHN [20] and ImageNet [21] datasets. All experiments are performed using
PyTorch [18] based models. For each dataset, we estimate the accuracy and
uncertainty of the final architectures obtained from $\mu$DARTS with the
following models: (1) architectures obtained from standard DARTS, (2)
architectures obtained from the other baseline architecture search methods,
and (3) existing deep networks for image classification with Concrete dropout.
We calculate the model uncertainty by the number of Monte Carlo passes of the
network and getting the predictive variance (see equation (5)).

As examples of image classification network, we consider MobileNetv2 [14] ,
VGG16 [15], ResNet20 [16], EfficientNet [17]. All networks were implemented
from the source code. We used the standard models for each of them and
implemented them in PyTorch. We also added a Concrete dropout-based ReLu layer
instead of the ReLu layer in the original implementation of each of the
models. An MC dropout loss is also added to the loss function to calculate the
optimal dropout probabilities to get a tighter bound on the uncertainty
estimate of the model. The modified models were re-trained, and uncertainty
estimations were performed considering multiple MC samples.

We obtained the confidence interval of the performances of the architectures
searched using the architecture search methods by calculating the mean and
standard deviation of the model accuracy. We did this by re-training and re-
evaluating the final searched model 10 times with different initializations.
The observed mean and variance of the accuracy of the models and the mean of
their uncertainty are shown in Table V. The table shows that the $\mu$DARTS
outperforms the standard differentiable architecture search methods like
DARTS-CD, RDARTS-CD, P-DARTS-CD, PC-DARTS-CD, GOLD-NAS-CD, and ASAP-CD. The
architecture obtained with $\mu$DARTS also outperforms standard non-NAS-based
architectures like MobileNetv2, VGG16, ResNet20, and EfficientNet-B0. It is to
be noted here that all of the baseline models shown in Table V are equipped
with concrete dropout to calculate the uncertainty of the model. We see that
under these training conditions, the CIFAR-10 results of $\mu$DARTS are much
higher (96.22%) than the standard DARTS (94.32%). A similar trend can be seen
for ImageNet (74.64% for $\mu$DARTS compared to 70.63% for DARTS). This shows
that the proposed $\mu$DARTS method can achieve good accuracy and uncertainty
scores with minimal training (50 epochs) compared to the other baselines. A
reason for the better performance of the model on the ImageNet dataset in
comparison to CIFAR10 might be attributed to the better regularization of the
loss functions obtained due to the addition of the predictive variance and the
Monte Carlo loss functions help in a more efficient search for architecture.
Also, adding the regularization terms helps stabilize the $\mu$DARTS search by
implicitly regularizing the Hessian of the loss functions. We see that the
implicit regularization leads to a smaller dominant eigenvalue of the Hessian,
which serves as a proxy for the sharpness and thus leads to a more
generalizable architecture.

TABLE VI: Accuracy and Uncertainty of Different DNN Models under Input Noise

Model — SNR | 40dB | 30dB | 20dB
---|---|---|---
| Accuracy | Uncertainty | Accuracy | Uncertainty | Accuracy | Uncertainty
Accuracy and Uncertainty of Different Network Architectures under Input Noise
for CIFAR 10
DARTS | 88.38 | N/A | 82.69 | N/A | 40.87 | NA
DARTS-CD | 89.16 | 0.574 | 82.17 | 0.913 | 44.11 | 1.502
RDARTS-CD | 90.88 | 0.488 | 82.75 | 0.899 | 44.29 | 1.479
|  $\mu$DARTS
---
92.04 | 0.198 | 84.16 | 0.397 | 49.34 | 1.206
Accuracy and Uncertainty of Different Network Architectures under Input Noise
for CIFAR 100
DARTS | 76.21 | N/A | 71.37 | N/A | 37.17 | NA
DARTS-CD | 76.89 | 0.632 | 71.04 | 1.174 | 38.41 | 2.047
RDARTS-CD | 77.39 | 0.599 | 72.58 | 1.007 | 39.16 | 1.992
|  $\mu$DARTS
---
78.54 | 0.483 | 75.82 | 0.397 | 46.22 | 1.632
Accuracy and Uncertainty of Different Network Architectures under Input Noise
for SVHN
DARTS | 94.77 | N/A | 89.19 | N/A | 70.22 | NA
DARTS-CD | 94.98 | 0.148 | 90.33 | 0.396 | 71.68 | 0.835
RDARTS-CD | 95.03 | 0.106 | 91.00 | 0.313 | 72.15 | 0.794
|  $\mu$DARTS
---
97.34 | 0.095 | 92.3 | 0.199 | 75.45 | 0.552

TABLE VII: Performance of Final DNN under Parameter noise

| CIFAR 10 | CIFAR 100 | SVHN
---|---|---|---
| Accuracy | Uncertainty | Accuracy | Uncertainty | Accuracy | Uncertainty
DARTS | 90.17 $\pm$ 0.298 | N/A | 74.13 $\pm$ 0.785 | N/A | 93.19 $\pm$ 0.211 | N/A
DARTS-CD | 91.86 $\pm$ 0.297 | 0.312 | 74.68 $\pm$ 0.587 | 0.638 | 93.91 $\pm$ 0.206 | 0.201
RDARTS-CD | 92.78 $\pm$ 0.29 | 0.217 | 75.76 $\pm$ 0.591 | 0.411 | 94.11 $\pm$ 0.210 | 0.109
$\mu$DARTS | 94.12 $\pm$ 0.25 | 0.134 | 79.89 $\pm$ 0.491 | 0.398 | 95.14 $\pm$ 0.17 | 0.074

####  V-D2 Input Noise Tolerance of Final DNN Models

We study the accuracy and uncertainty of the final DNN models obtained from
various architecture search methods under Gaussian noise to the input images.
We used the CIFAR10, CIFAR100, and SVHN datasets and compared varying Signal-
to-Noise Ratio (SNR). We do not re-train the models with noisy images.
Instead, noisy images are only applied during inference [Table VI]. The added
predictive variance term and the Monte-Carlo loss terms act as a regularizer,
improving the performance compared to the standard methods. We observe that
the $\mu$DARTS process has higher accuracy and lower uncertainty under noise
than DNN models from other architecture search methods.

####  V-D3 Tolerance of Final DNN Models to Noisy Parameters

We test the architecture’s performance when the model parameters are perturbed
by a small gaussian noise $\sim\mathcal{N}(0,1)$. We hypothesize that a more
stable neural architecture will lower the testing error and uncertainty
variance. Table VII shows that the DNN model generated by the $\mu$DARTS
method gives the least variance. Hence, we conclude that the $\mu$DARTS final
architecture is stable and can handle noise perturbations very well. Comparing
the results of Tables V and VII, we see that with the inclusion of the
parameter noise, though the accuracy for all the models fall and the
uncertainty increases, the change in the accuracy and uncertainty for the
$\mu$DARTS model [e.g. CIFAR10: $\Delta\text{Mean
Accuracy}=2.61,\Delta\text{Variance}=0.045$] is lower than the change in other
networks. Hence, we can conclude that the $\mu$DARTS model is more resilient
to parameter noise.

##  VI Conclusions

In this paper, we proposed a novel method, referred to as $\mu$DARTS, to
search for a neural architecture that simultaneously improves the accuracy and
the uncertainty of the final neural network architecture. $\mu$DARTS uses
concrete dropout layers within the cells of a neural architecture search
framework, and adds predictive variance and Monte-Carlo dropout losses as
regularizers in the validation and training losses respectively. We
experimentally demonstrate that $\mu$DARTS improves the performance of the
neural architecture search and the final architecture found using the
architecture search by showing that the optimization problem converges to a
flat minima. We also empirically show that the final architecture is stable
when perturbed with input and parameter noise.

## Appendix

###  VI-A Implicit Regularization of the Hessian

The addition of the regularizer to the loss function of the $\mu$DARTS
architecture search method induces implicit regularization on the Hessian
matrix. It has been empirically pointed out that the dominant eigenvalue of
$\nabla_{A}^{2}L_{\mathrm{val}}(w,A)$ (spectral norm of Hessian) is highly
correlated with the generalization quality of DARTS solutions [7]. In standard
DARTS training, the Hessian norm grows exponentially leading to bad test
performance. Chen et al. [30] plot this Hessian norm during the training
procedure and found that DARTS with perturbation-based regularization
consistently reduce the Hessian norms during the training procedure. They also
showed the spectral norm of Hessian is correlated with the solution quality,
and that perturbation-based regularization of the DARTS search strategy can
implicitly control the Hessian norm.

The update of $w$ in $\mu$DARTS can thus implicitly control the trace norm of
the Hessian of the loss function. If the matrix is close to positive semi-
definite, this is approximately regularizing the (positive) eigenvalues of
$\nabla_{\alpha}^{2}L^{\mu\text{DARTS}}_{\mathrm{val}}(w,\alpha)$. Therefore,
$\mu$DARTS reduces the Hessian norm through its training procedure.

###  VI-B Comparison of Stability using Largest Eigenvalues

In [7], the authors empirically showed the instability of the DARTS method is
related to the norm of Hessian
$\nabla_{\boldsymbol{\alpha}}^{2}\mathcal{L}_{\text{valid}}$. Chen et al. [30]
verified this by plotting the validation accuracy landscape of the DARTS and
showed it is extremely sharp and thus, even a small perturbation in
$\boldsymbol{\alpha}$ can drastically change the validation accuracy.

Here we prove that the eigenvalue of the Hessian of validation loss in
$\mu$DARTS
($\nabla_{\boldsymbol{\alpha}}^{2}\mathcal{L}_{\text{valid}}^{\text{$\mu$DARTS}}$)
is lower than the eigenvalue of the Hessian of validation loss in DARTS
($\nabla_{\boldsymbol{\alpha}}^{2}\mathcal{L}_{\text{valid}}^{\text{DARTS}}$).

###### Lemma VI.1

The largest eigenvalue of the Hessian of the validation loss of DARTS is given
by,

$\displaystyle\lambda_{\text{max}}(\nabla_{\alpha}^{2}\mathcal{L}_{\text{valid}}^{\text{DARTS}})=\frac{\max\sigma_{i}}{N}\lambda_{\max}(\sum_{i=1}^{N}\mathbf{x}_{i}\mathbf{x}_{i}^{T}),$
$\displaystyle\text{where,
}\sigma_{i}=\sigma\left(\mathbf{x}_{i}^{T}\boldsymbol{\alpha}\right)\left(1-\sigma\left(\mathbf{x}_{i}^{T}\boldsymbol{\alpha}\right)\right)$
(10)

Proof. We consider $\mathbf{x}_{i}$ is the input vector, $\alpha$ is the
weight matrix of architectural parameter, and $\mathbf{y}_{i}$ is the vector
denoting the labels of the classes. For simplicity, we consider a linear model
$\mathbf{x}_{i}^{T}\boldsymbol{\alpha}$ for this proof. Considering the cross
entropy loss as the validation loss ($\mathcal{L}_{\text{valid}}$) we obtain:

$\displaystyle\mathcal{L}_{\text{valid}}^{\text{DARTS}}=$
$\displaystyle-\frac{1}{N}\sum_{i=1}^{N}\mathbf{y}_{i}\log\left(\sigma\left(\mathbf{x}_{i}^{T}\boldsymbol{\alpha}\right)\right)+$
$\displaystyle\left(1-\mathbf{y}_{i}\right)\log\left(1-\sigma\left(\mathbf{x}_{i}^{T}\boldsymbol{\alpha}\right)\right)$
(11)
$\displaystyle\nabla_{\alpha}^{2}\mathcal{L}_{\text{valid}}^{\text{DARTS}}$
$\displaystyle=\frac{1}{N}\sum_{i=1}^{N}\sigma\left(\mathbf{x}_{i}^{T}\boldsymbol{\alpha}\right)\left(1-\sigma\left(\mathbf{x}_{i}^{T}\boldsymbol{\alpha}\right)\right)\mathbf{x}_{i}\mathbf{x}_{i}^{T}$
$\displaystyle=\frac{1}{N}\sum_{i=1}^{N}\sigma_{d}\mathbf{x}_{i}\mathbf{x}_{i}^{T}$
(12)

where, $\sigma(.)$ is the sigmoid function,
$\sigma_{d}=\sigma\left(\mathbf{x}_{i}^{T}\boldsymbol{\alpha}\right)\left(1-\sigma\left(\mathbf{x}_{i}^{T}\boldsymbol{\alpha}\right)\right)$
for each $i$. We know the Rayleigh quotient of
$\nabla_{\alpha}^{2}\mathcal{L}_{\text{valid}}^{\text{DARTS}}$ is given by:

$\displaystyle\mathcal{R}_{\text{DARTS}}(\mathbf{z})$
$\displaystyle=\mathbf{z}^{T}\nabla_{\alpha}^{2}\mathcal{L}_{\text{valid}}^{\text{DARTS}}\mathbf{z}$
$\displaystyle=\mathbf{z}^{T}\left(\frac{1}{N}\sum_{i=1}^{N}\sigma_{d}\mathbf{x}_{i}\mathbf{x}_{i}^{T}\right)\mathbf{z}$
$\displaystyle=\frac{1}{N}\left(\mathbf{z}^{T}\left(\sum_{i=1}^{N}\sigma_{d}\mathbf{x}_{i}\mathbf{x}_{i}^{T}\right)\mathbf{z}\right)$
(13)

where, $\mathbf{z}$ is any unit-length vector. Assuming maximum of
$\sigma_{d}$ is represented as $\sigma_{d}^{max}$, we obtain:

$\displaystyle\mathcal{R}_{\text{DARTS}}(\mathbf{z})$
$\displaystyle\leq\frac{\sigma_{d}^{max}}{N}\left(\mathbf{z}^{T}\left(\sum_{i=1}^{N}\mathbf{x}_{i}\mathbf{x}_{i}^{T}\right)\mathbf{z}\right)=\frac{\sigma_{d}^{max}}{N}\mathcal{R}_{M_{\mathbf{x}}}(\mathbf{z})$

where, $\mathcal{R}_{\text{M}}(\mathbf{z})$ is the Raleigh quotient of the
matrix $M_{\mathbf{x}}=\sum_{i=1}^{N}\mathbf{x}_{i}\mathbf{x}_{i}^{T}$ as
given by:

$\displaystyle\mathcal{R}_{\text{M}}(\mathbf{z})=\mathbf{z}^{T}\left(\sum_{i=1}^{N}\mathbf{x}_{i}\mathbf{x}_{i}^{T}\right)\mathbf{z}$
$\displaystyle=\sum_{i=1}^{N}\left(\mathbf{z}^{T}\mathbf{x}_{i}\right)\left(\mathbf{x}_{i}^{T}\mathbf{z}\right)$
$\displaystyle=\sum_{i=1}^{N}\left(\mathbf{z}^{T}\mathbf{x}_{i}\right)^{2}\geq
0$ (14)

Since, the maximum eigenvalue of a symmetric matrix ($A$) is equal to the
maximum value of its Raleigh quotient
$\left(\lambda_{\max}(A)=\mathcal{R}_{A}^{max})\right)$, we obtain:

$\displaystyle\lambda_{\text{max}}(\nabla_{\alpha}^{2}\mathcal{L}_{\text{valid}}^{\text{DARTS}})$
$\displaystyle=\mathcal{R}_{\text{DARTS}}^{\max}=\frac{\sigma_{d}^{\max}}{N}\mathcal{R}_{M}^{\max}$
$\displaystyle=\frac{\sigma_{d}^{\max}}{N}\lambda_{\max}\left(\sum_{i=1}^{N}\mathbf{x}_{i}\mathbf{x}_{i}^{T}\right)$
(15)

Corollary. The validation loss function of the standard DARTS
($\mathcal{L}_{\text{valid}}^{\text{DARTS}}$) method is convex in nature.

Proof. We know that, the smallest eigenvalue of any square symmetric matrix is
given as the minimum of the Rayleigh quotient of that matrix. Since
$\sigma_{d}\geq 0$, from equations 13 and 14, we obtain
$\mathcal{R}_{\text{DARTS}}(\mathbf{z})\geq 0$. Hence, the smallest possible
eigenvalue of $\nabla_{\alpha}^{2}\mathcal{L}_{\text{valid}}^{\text{DARTS}}$
must be zero or positive which implies that the validation loss function is
convex.

###### Lemma VI.2

The largest eigenvalue of the Hessian of the validation loss of $\mu$DARTS is
given by,

$\lambda_{\text{max}}(\nabla_{\alpha}^{2}\mathcal{L}_{\text{valid}}^{\text{$\mu$DARTS}})\leq\frac{\sigma_{ud}^{max}}{N}\lambda_{\text{max}}\left(\sum_{i=1}^{N}\mathbf{x}_{i}\mathbf{x}_{i}^{T}\right)$

$\displaystyle\text{ where, }\sigma_{ud}$
$\displaystyle=\sqrt{\sigma_{x^{2}}}\left(1-\sqrt{\sigma_{x^{2}}}\right)+4\alpha^{2}\sigma_{x^{2}}(1-\sigma_{x^{2}})$
$\displaystyle-8\alpha^{2}(\sigma_{x^{2}})^{2}(1-\sigma_{x^{2}})+2\sigma_{x^{2}}(1-\sigma_{x^{2}})$
$\displaystyle-2(\sqrt{\sigma_{x^{2}}}(1-\sqrt{\sigma_{x^{2}}})+\sqrt{\sigma_{x^{2}}})$
$\displaystyle\sigma_{x^{2}}$
$\displaystyle=\sigma(\boldsymbol{\alpha}\mathbf{x}_{i}\mathbf{x}_{i}^{T}\boldsymbol{\alpha}^{T})\text{
and }\sigma_{x}=\sigma(\mathbf{x}_{i}^{T}\boldsymbol{\alpha})$

Proof. To estimate
$\nabla_{\alpha}^{2}\mathcal{L}_{\text{valid}}^{\text{$\mu$DARTS}}$, we add
the predictive variance term to the DARTS validation loss.

$\displaystyle\operatorname{Var}_{p(\mathbf{y}|\mathbf{x})}^{model}(\boldsymbol{\alpha})$
$\displaystyle=\mathbb{E}((\mathbf{x}_{i}^{T}\alpha)^{2})-\mathbb{E}(\mathbf{x}^{T}_{i}\alpha)^{2}$
$\displaystyle=\frac{1}{T}\sum_{i=1}^{T}\sigma(\boldsymbol{\alpha}\mathbf{x}_{i}\mathbf{x}_{i}^{T}\boldsymbol{\alpha}^{T})-(\frac{1}{T}\sum_{i=1}^{T}\sigma(\mathbf{x}_{i}^{T}\boldsymbol{\alpha}))^{2}$
(16)
$\displaystyle\nabla_{\alpha}^{2}\mathcal{L}_{\text{valid}}^{\text{$\mu$DARTS}}$
$\displaystyle=\nabla_{\alpha}^{2}\mathcal{L}_{\text{valid}}^{\text{DARTS}}+\nabla_{\alpha}^{2}\operatorname{Var}_{p(\mathbf{y}|\mathbf{x})}^{model}(\boldsymbol{\alpha})$
$\displaystyle=\frac{1}{N}\sum_{i=1}^{N}\sigma_{x}\left(1-\sigma_{x}\right)\mathbf{x}_{i}\mathbf{x}_{i}^{T}$
$\displaystyle+\frac{1}{T}\sum_{i=1}^{T}\\{$
$\displaystyle[\sigma_{x^{2}}(1-\sigma_{x^{2}})-2\sigma_{x^{2}}^{2}(1-\sigma_{x^{2}})]4\alpha^{T}\alpha\mathbf{x}_{i}\mathbf{x}^{T}_{i}$
$\displaystyle\quad\quad\quad\quad\quad\quad+\sigma_{x^{2}}(1-\sigma_{x^{2}})2\mathbf{x}_{i}\mathbf{x}^{T}_{i}\\}$
$\displaystyle-\frac{2}{T^{2}}[\sum_{i=1}^{T}\sigma_{x}(1-\sigma_{x})\sum_{i=1}^{T}\sigma_{x}(1-\sigma_{x})]\mathbf{x}_{i}\mathbf{x}_{i}^{T}$
$\displaystyle-\frac{2}{T^{2}}[\sum_{i=1}^{T}\sigma_{x}$
$\displaystyle\sum_{i=1}^{T}\sigma_{x}(1-\sigma_{x})-2(\sigma_{x})^{2}(1-\sigma_{x})]\mathbf{x}_{i}\mathbf{x}_{i}^{T}$
(17)

Without loss of generality, we consider the case when $T=N$. Therefore, the
Raleigh quotients of
$\nabla_{\alpha}^{2}\mathcal{L}_{\text{valid}}^{\text{$\mu$DARTS}}$ can be
computed as:

$\displaystyle\mathcal{R}_{\text{$\mu$DARTS}}(\mathbf{z})=\mathbf{z}^{T}\nabla_{\alpha}^{2}\mathcal{L}_{\text{valid}}^{\text{$\mu$DARTS}}\mathbf{z}$
$\displaystyle=\mathbf{z}^{T}\Bigg{(}\frac{1}{N}\sum_{i=1}^{N}\bigg{[}\sigma_{x}\left(1-\sigma_{x}\right)+\Big{(}\sigma_{x^{2}}(1-\sigma_{x^{2}})$
$\displaystyle\quad\quad\quad-2\sigma_{x^{2}}^{2}(1-\sigma_{x^{2}})\Big{)}4\alpha^{T}\alpha+2\sigma_{x^{2}}(1-\sigma_{x^{2}})\bigg{]}\mathbf{x}_{i}\mathbf{x}_{i}^{T}$
$\displaystyle-\frac{2}{N^{2}}\bigg{[}\sum_{i=1}^{N}\sigma_{x}(1-\sigma_{x})\sum_{i=1}^{N}\sigma_{x}(1-\sigma_{x})$
$\displaystyle+\sum_{i=1}^{N}\sigma_{x}\sum_{i=1}^{N}\Big{(}\sigma_{x}(1-\sigma_{x})-2(\sigma_{x})^{2}(1-\sigma_{x})\Big{)}\bigg{]}\mathbf{x}_{i}\mathbf{x}_{i}^{T}\Bigg{)}\mathbf{z}$

Since $N$ is large, we note that $\sum_{i=1}^{N}\sigma_{x}(1-\sigma_{x})\geq
1;\quad\sum_{i=1}^{N}\big{[}\sigma_{x}(1-\sigma_{x})-2(\sigma_{x})^{2}(1-\sigma_{x})\big{]}\geq
1$. Thus, we can say that

$\displaystyle(i)\sum_{i=1}^{N}\sigma_{x}(1-\sigma_{x})\sum_{i=1}^{N}\sigma_{x}(1-\sigma_{x})\geq\sum_{i=1}^{N}\sigma_{x}(1-\sigma_{x})$
(18)
$\displaystyle(ii)\sum_{i=1}^{N}\sigma_{x}\sum_{i=1}^{N}\big{[}\sigma_{x}(1-\sigma_{x})-2(\sigma_{x})^{2}(1-\sigma_{x})\big{]}\geq\sum_{i=1}^{N}\sigma_{x}$
(19)

TABLE VIII: *

TABLE C.1: Comparison of Reported Test Errors of Models for Different
Architecture Search Processes (The test errors are the values reported in the
original papers)  Model CIFAR 10 ImageNet Params(M) Accuracy Epochs Batch Size
Params(M) Accuracy Epochs Batch Size P-DARTS [31] 10.5 2.25 250 1024 5.4 24.1
250 1024 PC-DARTS [32] 3.6 2.57 600 128 5.3 24.2 250 1024 SNAS [36] 2.8 2.85
600 96 4.3 27.3 250 96 GOLD-NAS-A [35] 1.58 2.93 200 96 - - - GOLD-NAS-Z [35]
- - - - 6.4 23.9 250 1024 ASAP [34] 6.0 1.68 1500 128 6.0 24.4 250 64 DARTS
[6] 3.3 2.76 600 96 4.7 26.7 250 128 RDARTS [7] 3.1 2.97 50 64 - - -
$\mu$DARTS (This Work) 4.1 3.78 50 32 5.3 25.36 50 64

We take
$\mathcal{R}_{\text{$\mu$DARTS}}(\mathbf{z})=\mathbf{z}^{T}\Bigg{(}\frac{1}{N}\sum_{i=1}^{N}\bigg{[}\sigma_{j}\bigg{]}\mathbf{x}_{i}\mathbf{x}_{i}^{T}\Bigg{)}\mathbf{z},$
where, $\sigma_{j}$ is the polynomial in $\sigma_{x}$, $\sigma_{x^{2}}$, and
$\alpha$. Also, since $\sigma_{x}$ is convex, using Jensen’s inequality
($\sigma_{x}^{2}\leq\sigma_{x^{2}}$) we get:

$\displaystyle\sigma_{j}$
$\displaystyle\leq\Big{[}\sqrt{\sigma_{x^{2}}}\left(1-\sqrt{\sigma_{x^{2}}}\right)+4\alpha^{T}\alpha\sigma_{x^{2}}(1-\sigma_{x^{2}})$
$\displaystyle-8\alpha^{T}\alpha(\sigma_{x^{2}})^{2}(1-\sigma_{x^{2}})+2\sigma_{x^{2}}(1-\sigma_{x^{2}})$
$\displaystyle-2\big{(}\sqrt{\sigma_{x^{2}}}(1-\sqrt{\sigma_{x^{2}}})+\sqrt{\sigma_{x^{2}}}\big{)}\Big{]}$
(20)

Assuming, maximum value of $\sigma_{ud}$ and $\sigma_{j}$ are given by
$\sigma_{ud}^{\max}$ and $\sigma_{j}^{\max}$, respectively, we obtain:

$\displaystyle\mathcal{R}_{\text{$\mu$DARTS}}(\mathbf{z})$
$\displaystyle\leq\frac{\sigma_{j}^{\max}}{N}\Bigg{(}\mathbf{z}^{T}\Bigg{(}\sum_{i=1}^{N}\mathbf{x}_{i}\mathbf{x}_{i}^{T}\Bigg{)}\mathbf{z}\Bigg{)}$
$\displaystyle\leq\frac{\sigma_{ud}^{\max}}{N}\Bigg{(}\mathbf{z}^{T}\Bigg{(}\sum_{i=1}^{N}\mathbf{x}_{i}\mathbf{x}_{i}^{T}\Bigg{)}\mathbf{z}\Bigg{)}$
(21)

Using the relation between maximum eigenvalue and Ralyeigh quotient, similar
to the Lemma VI.1, the largest eigenvalue of the $\mu$DARTS method is given
as:

$\displaystyle\lambda_{\text{max}}(\nabla_{\alpha}^{2}\mathcal{L}_{\text{valid}}^{\text{DARTS}})$
$\displaystyle=\mathcal{R}_{\text{$\mu$DARTS}}^{\max}$
$\displaystyle\leq\frac{\sigma_{ud}^{\max}}{N}\lambda_{\max}\left(\sum_{i=1}^{N}\mathbf{x}_{i}\mathbf{x}_{i}^{T}\right)$
(22)

###### Lemma VI.3

If
$\lambda_{\text{max}}(\nabla_{\alpha}^{2}\mathcal{L}_{\text{valid}}^{\text{DARTS}})$
and
$\lambda_{\text{max}}(\nabla_{\alpha}^{2}\mathcal{L}_{\text{valid}}^{\text{$\mu$DARTS}})$
be the maximum eigenvalues of the Hessian of the validation loss for DARTS and
the $\mu$DARTS, respectively, then

$\displaystyle\lambda_{\text{max}}(\nabla_{\alpha}^{2}\mathcal{L}_{\text{valid}}^{\text{$\mu$DARTS}})$
$\displaystyle\leq\lambda_{\text{max}}(\nabla_{\alpha}^{2}\mathcal{L}_{\text{valid}}^{\text{DARTS}})$

Proof. We represent $\sigma_{d}$ as: $\sigma_{d}=p(1-p)$, where
$p=\sigma\left(\mathbf{x}_{i}^{T}\boldsymbol{\alpha}\right)\in[0,1]$ due to
the properties of the sigmoid function. Moreover, since $\alpha$ is the
weights in the neural network we have $\alpha<1\implies\alpha^{T}\alpha<1$.
Hence, by maximizing $\sigma_{i}$ with respect to $p$ we
obtain:$\sigma_{d}^{\max}=0.25$

We now represent
$\sigma_{ud}=\sqrt{q}(1-\sqrt{q})+4\alpha^{T}\alpha[q(1-q)-2q^{2}(1-q)]+2q(1-q)-2(\sqrt{q}(1-\sqrt{q})+\sqrt{q}\\}$,
where
$q=\sigma_{{x}^{2}}=\sigma(\boldsymbol{\alpha}\mathbf{x}_{i}\mathbf{x}_{i}^{T}\boldsymbol{\alpha}^{T})\in[0,1]$.
We now maximize $\sigma_{j}$ with respect to $q$. Finally, noting
$\alpha^{T}\alpha<1$, we observe that $\sigma_{ud}\leq 0$. Thus the function
becomes: $\sigma_{ud}^{\max}<0<0.25=\sigma_{d}^{max}$. From equation (14) we
note that $\mathcal{R}_{\text{M}}^{max}\geq 0$. Therefore, using equations
(15), (22), and the bound on $\sigma_{ud}^{\max}$, we get

$\lambda_{\text{max}}(\nabla_{\alpha}^{2}\mathcal{L}_{\text{valid}}^{\text{$\mu$DARTS}})\leq\lambda_{\text{max}}(\nabla_{\alpha}^{2}\mathcal{L}_{\text{valid}}^{\text{DARTS}})$
(23)

###  VI-C Comparison of Accuracy of Unmodified Models

We demonstrate the comparative performance of the $\mu$DARTS method with
respect to models searched using other architecture search methods with the
training condition for each model as reported in the original paper. Note that
the original NAS and DARTS papers only reported accuracy of the final models
(not uncertainty). The results are shown in Table C.1. We see that the
$\mu$DARTS, trained with only 50 epochs gives comparable performance to other
networks trained for a much higher number of epochs and with a much higher
batch size. This result further showcases the robustness and easy trainability
of $\mu$DARTS.

## References

  * [1] Gal, Y. Uncertainty in deep learning. University Of Cambridge. 1 pp. 3 (2016)
  * [2] Gal, Y. & Ghahramani, Z. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. ArXiv Preprint ArXiv:1506.02142. (2015)
  * [3] Gal, Y., Hron, J. & Kendall, A. Concrete dropout. Advances In Neural Information Processing Systems. pp. 3581-3590 (2017)
  * [4] Zoph, B. & Le, Q. Neural architecture search with reinforcement learning. ArXiv Preprint ArXiv:1611.01578. (2016)
  * [5] Real, E., Aggarwal, A., Huang, Y. & Le, Q. Aging evolution for image classifier architecture search. AAAI Conference On Artificial Intelligence. (2019)
  * [6] Liu, H., Simonyan, K. & Yang, Y. Darts: Differentiable architecture search. ArXiv Preprint ArXiv:1806.09055. (2018)
  * [7] Arber Zela, T., Saikia, T., Marrakchi, Y., Brox, T. & Hutter, F. Understanding and robustifying differentiable architecture search. ArXiv Preprint ArXiv:1909.09656. 2, 9 (2019)
  * [8] Kendall, A. & Gal, Y. What uncertainties do we need in bayesian deep learning for computer vision?. Advances In Neural Information Processing Systems. pp. 5574-5584 (2017)
  * [9] Kendall, A., Badrinarayanan, V. & Cipolla, R. Bayesian segnet: Model uncertainty in deep convolutional encoder-decoder architectures for scene understanding. ArXiv Preprint ArXiv:1511.02680. (2015)
  * [10] Colson, B., Marcotte, P. & Savard, G. An overview of bilevel optimization. Annals Of Operations Research. 153, 235-256 (2007)
  * [11] Baker, B., Gupta, O., Naik, N. & Raskar, R. Designing neural network architectures using reinforcement learning. ArXiv Preprint ArXiv:1611.02167. (2016)
  * [12] Bender, G. Understanding and simplifying one-shot architecture search. (2019)
  * [13] Bergstra, J., Yamins, D. & Cox, D. Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. (Jmlr,2013)
  * [14] Sandler, M., Howard, A., Zhu, M., Zhmoginov, A. & Chen, L. Mobilenetv2: Inverted residuals and linear bottlenecks. Proceedings Of The IEEE Conference On Computer Vision And Pattern Recognition. pp. 4510-4520 (2018)
  * [15] Simonyan, K. & Zisserman, A. Very deep convolutional networks for large-scale image recognition. ArXiv Preprint ArXiv:1409.1556. (2014)
  * [16] He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. Proceedings Of The IEEE Conference On Computer Vision And Pattern Recognition. pp. 770-778 (2016)
  * [17] Tan, M. & Le, Q. Efficientnet: Rethinking model scaling for convolutional neural networks. ArXiv Preprint ArXiv:1905.11946. (2019)
  * [18] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L. & Others PyTorch: An imperative style, high-performance deep learning library. Advances In Neural Information Processing Systems. pp. 8024-8035 (2019)
  * [19] Krizhevsky, A., Hinton, G. & Others Learning multiple layers of features from tiny images. (Citeseer,2009)
  * [20] Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B. & Ng, A. Reading digits in natural images with unsupervised feature learning. (2011)
  * [21] Deng, J., Dong, W., Socher, R., Li, L., Li, K. & Fei-Fei, L. Imagenet: A large-scale hierarchical image database. 2009 IEEE Conference On Computer Vision And Pattern Recognition. pp. 248-255 (2009)
  * [22] Dinh, L., Pascanu, R., Bengio, S. & Bengio, Y. Sharp minima can generalize for deep nets. Proceedings Of The 34th International Conference On Machine Learning-Volume 70. pp. 1019-1028 (2017)
  * [23] Plappert, M., Houthooft, R., Dhariwal, P., Sidor, S., Chen, R., Chen, X., Asfour, T., Abbeel, P. & Andrychowicz, M. Parameter space noise for exploration. ArXiv Preprint ArXiv:1706.01905. (2017)
  * [24] Zhang, J., Pan, Y., Yang, H. & Fang, Y. Scalable deep multi-agent reinforcement learning via observation embedding and parameter noise. IEEE Access. 7 pp. 54615-54622 (2019)
  * [25] Salimans, T., Ho, J., Chen, X., Sidor, S. & Sutskever, I. Evolution strategies as a scalable alternative to reinforcement learning. ArXiv Preprint ArXiv:1703.03864. (2017)
  * [26] Ba, J., Kiros, J. & Hinton, G. Layer normalization. ArXiv Preprint ArXiv:1607.06450. (2016)
  * [27] An, G. The effects of adding noise during backpropagation training on a generalization performance. Neural Computation. 8, 643-674 (1996)
  * [28] Mulayoff, R. & Michaeli, T. Unique Properties of Flat Minima in Deep Networks.
  * [29] Ardywibowo, R., Boluki, S., Gong, X., Wang, Z. & Qian, X. NADS: Neural Architecture Distribution Search for Uncertainty Awareness. (2020)
  * [30] Chen, X. & Hsieh, C. Stabilizing Differentiable Architecture Search via Perturbation-based Regularization. ArXiv Preprint ArXiv:2002.05283. (2020)
  * [31] Chen, X., Xie, L., Wu, J. & Tian, Q. Progressive differentiable architecture search: Bridging the depth gap between search and evaluation. Proceedings Of The IEEE International Conference On Computer Vision. pp. 1294-1303 (2019)
  * [32] Xu, Y., Xie, L., Zhang, X., Chen, X., Qi, G., Tian, Q. & Xiong, H. Pc-darts: Partial channel connections for memory-efficient differentiable architecture search. ArXiv Preprint ArXiv:1907.05737. (2019)
  * [33] Dong, X. & Yang, Y. Searching for a robust neural architecture in four gpu hours. Proceedings Of The IEEE Conference On Computer Vision And Pattern Recognition. pp. 1761-1770 (2019)
  * [34] Noy, A., Nayman, N., Ridnik, T., Zamir, N., Doveh, S., Friedman, I., Giryes, R. & Zelnik, L. Asap: Architecture search, anneal and prune. International Conference On Artificial Intelligence And Statistics. pp. 493-503 (2020)
  * [35] Bi, K., Xie, L., Chen, X., Wei, L. & Tian, Q. Gold-nas: Gradual, one-level, differentiable. ArXiv Preprint ArXiv:2007.03331. (2020)
  * [36] Xie, S., Zheng, H., Liu, C. & Lin, L. SNAS: stochastic neural architecture search. ArXiv Preprint ArXiv:1812.09926. (2018)

|  Biswadeep Chakraborty (Graduate Student Member, IEEE) received the B.E.
degree in Electronics and Telecommunication engineering from Jadavpur
University, Kolkata, India, in 2019. He is currently pursuing the Ph.D. degree
in Electrical and Computer Engineering with the Georgia Institute of
Technology, Atlanta, GA, USA, under the supervision of Prof. S. Mukhopadhyay.
Before starting his Ph.D., he worked as a Research Assistant at the Singtel-
National University of Singapore Lab. His current research interests lies in
unsupervised learning methods using spiking neural networks for spatio-
temporal prediction and control.
---|---
|  Saibal Mukhopadhyay (Fellow, IEEE) received the B.E. degree in Electronics
and Telecommunication engineering from Jadavpur University, Kolkata, India, in
2000, and the Ph.D. degree in electrical and computer engineering from Purdue
University, West Lafayette, IN, USA, in 2006. He was a Research Staff Member
with the IBM Thomas J. Watson Research Center, Yorktown Heights, NY, USA, from
August 2007 to September 2007. He is currently a Joseph M. Pettit Professor
with the School of Electrical and Computer Engineering, Georgia Institute of
Technology, Atlanta, GA, USA. He has authored or coauthored over 200 articles
in refereed journals and conferences, and holds five U.S. patents. His
research interests include design of energy-efficient, intelligent, and secure
systems in nanometer technologies. He was a recipient of the Office of Naval
Research Young Investigator Award, in 2012, the National Science Foundation
CAREER Award, in 2011, the IBM Faculty Partnership Award, in 2009 and 2010,
the SRC Inventor Recognition Award in 2008, the SRC Technical Excellence
Award, in 2005, and the IBM Ph.D. Fellowship Award, for years 2004 to 2005.
---|---


##  0.1 Problem Formulation

The QRS-detection and R-peak localisation task was formulated as a
segmentation problem where each input sample receives a prediction through a
CNN model. By labeling five samples around an annotated R-location (approx.
0.05 seconds per side, forming a 0.1 second QRS region) as 1s and the rest
samples as 0s, similar to Jia et al. [jia_high_2019], a binary-mask was
created to train a CNN model to be able to predict a similar mask which later
used to localise R-peaks.

##  0.2 Application Framework

The proposed QRS detection algorithm fits within an application framework as
shown in Figure LABEL:fig:cnn_pp_config. ECG signal can be acquired using
sensors and sent to a destination using wireless protocols, including
Bluetooth low-energy (BLE), among others. A resource constraint embedded
device works as an end-point to receive raw ECG signal. A suitable QRS
detection method can be determined by leveraging the strengths of DL model and
required post-processing for a target resource-constraint environment,
including smart-phone, smart-watch, or an embedded router hardware, which may
connect multiple sensors Figure LABEL:fig:cnn_pp_config). The output
information (ECG signal and R-peak locations) can be processed locally or
could be transferred to the cloud server for further processing.

##  0.3 ECG Data

ECG datasets from PhysioNet data-bank [goldberger_physiobank_2000] were used
for this study (summarised in Table 1),

Table 1: Characteristics of PhysioNet datasets used in current study. _DB-Name_ | _Source Hz_ | _No. of Rec._ | _Used no. of Rec._ | _Length (minute/rec.)_ | _No. of Beats_
---|---|---|---|---|---
EDB | 250 | 90 | 90 | 120 | 791665
INCART | 257 | 75 | 75 | 30 | 175900
MIT-BIH-Arr | 360 | 48 | 46 | 30 | 102941
NSTDB | 360 | 12 | 12 | 30 | 25590
QT | 250 | 82 | 80 | 15 | 84883
SVDB | 128 | 78 | 78 | 30 | 184583
STDB | 360 | 28 | 28 | vary | 76175
TWADB | 500 | 100 | 100 | 2 | 18993

including the MIT-BIH-Arrhythmia [moody_impact_2001], INCART, QT
[laguna_database_1997], EDB (European ST-T Database) [taddei_european_1992],
STDB (MIT-BIH ST Change Database), TWADB (T-Wave Alternans Challenge
Database), NSTDB (MIT-BIH Noise Stress Test Database) [moody_noise_1984], and
SVDB (MIT-BIH Supraventricular Arrhythmia Database) [greenwald1990improved].
The first channel signal, among others, was used. The STDB contains variable
length records and on average, 28 minutes long.

##  0.4 Pre-processing

### Resampling

PhysioNet datasets are sampled at a range of frequencies, so they were
resampled at a minimal 100Hz frequency, considering the fact that average QRS-
complex may go up to 25Hz or beyond [ajdaraga_analysis_2017], so the Nyquest
sampling frequency [proakis_digital_2004] should be at least 50Hz.

### Segmentation

A three second segmentation was applied with two seconds overlapping, allowing
each QRS region to receive the model prediction multiple times, which may
increase the detection likelihood. A three second segmentation was applied
with two seconds overlapping, allowing more long-enough segments for the
training, without increasing the model’s complexity significantly (longer
input segment increases convolution operation computation). For test ECG
records, a non-overlapped 3s segmentation was used to generate predictions.

### Normalisation

The heterogeneous datasets were normalised using standard-scores and such
normalisation was performed at segment-level instead of the recording-level to
localise the effect of noise.

##  0.5 CNN Model

The focus of this study was to observe the performance of CNN models, with a
gradual increase of their complexity (by increasing their depths
[simonyan2014very], in the range of 2-64 layers) and leverage the strength of
post-processing (PP 1-3) to yield a set of configurations for a range of
resource-constrained environments. Shallow CNN models are often utilised in
QRS detection [chandra_robust_2019, Xiang2018, sarlija_convolutional_2017],
thus a simple convolution-only model (no sub-sampling layer, with output
dimension the same as the input, using a similar philosophy of Pelt et al.
[pelt_mixed-scale_2018]) was selected as a baseline model, which will be
referred as baseline-convnet in the text.

[b]0.29

Figure 1:  The schematic block-diagram of convolutional neural network used
for R-peak localisation.

The block-diagram of the network (Figure 1) consists of a low-convolution-
block, transforming raw-input into an intermediate representation, followed by
a feature-extraction-block, consisting of one or more convolution-layer(s),
and finally, a scoring-layer that projects into binary decision-planes to
decide sample-wise predictions. The convolution kernel hyper-parameter is
optimised by adopting a domain-specific heuristic that the average duration of
QRS-complex is 60 milliseconds and a 44 milliseconds equivalent (which is 5
samples for 100Hz signal) kernel should be long-enough to capture the most QRS
features, following a study of Guo et al. [guo_inter-patient_2019]. The CNN
model was implemented in Python 3.7 using PyTorch 1.8.1 API.

##  0.6 Model Training and Testing

The MIT-BIH-Arrhythmia dataset is comparatively a less noisy dataset and was
used for the model training to learn signal morphology, while the other
datasets for cross-database testing. The baseline-convnet was optimised by an
internal subject-wise five-fold cross-validation of the MIT-BIH-Arrhythmia
dataset, where subject-wise four-fold was used for training and the rest-fold
subjects were used to validate the model and stop the training when suitable.
The five-fold approach yields five such optimised models, which were then used
for cross-database testing, producing five validation scores per validation-
dataset and finally, the scores were averaged.

An early-stopping mechanism was used in model-training, instead of using a
fixed number of epochs, to early detect the convergence. At the end of each
epoch, the trained model was evaluated using one of the folds (internal five-
fold approach, as mentioned above) and if the validation-loss does not improve
for seven consecutive epochs (a.k.a. early-stopping patience), the training is
stopped, while the maximum number of epochs was set to 50. In most of the
cases, it was found that the training process was terminated well before
reaching the maximum epoch limit. An initial learning rate was set to 0.01 to
facilitate eager learning and a learning-rate-scheduler was used to decrease
the rate by a factor of 0.1 of patience value set to 5, meaning that if the
validation-loss does not decrease for five-consecutive epochs, the rate is
decreased. A cross-entropy loss was found suitable, possibly due to it’s usage
of natural logarithm that takes into account the difference between the actual
label and predicted outcomes in a more granular way. To back-propagate the
calculated loss, the Adam optimiser was adopted for faster convergence
[kingma2014adam].

##  0.7 Post-processing Algorithm

The post-processing tries to refine the CNN model prediction-stream just
before the R-peaks are localised. The post-processing is separated into three
levels - minimal (Algorithm 0.7), moderate (Algorithm 0.7), and advanced level
(Algorithm 0.7), based on the level of domain-specific knowledge required. [h]
Salt and Pepper filter (Minimal post-processing).

  * •

PRED_STREAM: Binary prediction stream (CNN model’s output.)

  * •

ONES_PATTERNS: sequence of 1s with isolated 0s (11011, 1101).

  * •

ZEROS_PATTERNS: sequence of 0s with isolated 1s (00100, 0010).

Refined binary prediction stream. KwMinimal: Salt and Pepper filter
Beginbeginend _Scan for ONES_PATTERNS:_ Scan binary stream to match ones
patterns in sequence and replace with all ones of corresponding pattern
length. _Scan for ZEROS_PATTERNS:_ Scan binary stream to match zeros patterns
in sequence and replace with all zeros of corresponding pattern length.
_Repeat above 2 steps:_ Repeat scan for ONES_PATTERNS and ZEROS_PATTERNS until
no match found. [h] Remove less confident QRS candidates (Moderate post-
processing).

  * •

PRED_STREAM: Binary prediction stream (CNN model’s output.)

  * •

DOMAIN_KNOWLEDGE: Valid QRS extent should be at least 64 milli-seconds. Note
that 64 milli-seconds of 100Hz signal yields 6 samples.

Refined binary prediction stream. KwModerate: Remove less confident QRS
candidates Beginbeginend _Do Salt and Pepper filtering (Algorithm 0.7)._
_Calculate confidence score:_ Scan all QRS candidates (occurrence of
consecutive ones) in the binary prediction stream, where each candidate’s
confidence is the number of consecutive ones. _Filter out less confidants:_
Scan all QRS candidates in the binary prediction stream and remove them with
confidence score less than 6. [h] Remove short-distant QRS neighbors (Advanced
post-processing).

  * •

PRED_STREAM: Binary prediction stream (CNN model’s output.)

  * •

DOMAIN_KNOWLEDGE: Two consecutive QRS nodes should be at least 200 milli-
seconds (20 samples for 100Hz) distant (minimum R-R distance).

Refined binary prediction stream. KwAdvanced: Remove short-distant QRS
neighbors Beginbeginend _Remove less confident QRS candidates (Algorithm
0.7)._ _Calculate confidence score:_ See Algorithm 0.7 _Calculate R-R
interval:_ Scan the QRS candidates and calculate the distance between two
neighboring nodes. _Filter candidates with small R-R interval:_ Scan the QRS
candidates and for each node with R-R interval less than 20, remove either the
candidate or its next neighbor whoever has comparatively a low confidence
score.

The minimal post-processing (PP) is a basic Salt and Pepper filter which
removes isolated ones (or zeros) recursively within a group of consecutive
zeros (or ones). Two patterns of consecutive ones (or zeros), with isolated
zeros (or ones) inside, $11011$ and $1101$ (or $00100$ and $0010$), were
searched in sequence for a match into the input binary stream, in order to
replace them with all ones (or zeros) of corresponding pattern lengths (i.e. 5
and 4 ones). Matching the longer pattern $11011$ (or $00100$) before the
shorter $1101$ (or $0010$) was found effective which seems maximised the
lengths of consecutive ones (or zeros). The moderate PP tries to filter out
nodes with the confidence-score less than a threshold of 64 milliseconds
equivalent number of samples [cai_qrs_2020] (approx. 6 samples for 100Hz
signal) to remove QRS-like short lived artefacts. The advanced PP takes into
account the R-R interval and filters out nodes whose R-R interval falls below
a minimum threshold of 200 milliseconds equivalent number of samples (i.e. 20
samples for 100Hz signal worked well for our case, but Cai et al.
[cai_qrs_2020] used 100 milliseconds).

##  0.8 Validation score

In this study, an F1-score was used as a validation score for proposed models.
F1-score is calculated as

$F1=2*\frac{PPV*Sensitivy}{PPV+Sensitivy}$ (1)