# A combined quantum-classical method applied to material design: optimization
and discovery of photochromic materials for photopharmacology applications

Qi Gao<EMAIL_ADDRESS>Mitsubishi Chemical Corporation, Science &
Innovation Center, 1000, Kamoshida-cho, Aoba-ku, Yokohama 227-8502, Japan
Quantum Computing Center, Keio University, Hiyoshi 3-14-1, Kohoku, Yokohama
223-8522, Japan  Michihiko Sugawara  Quantum Computing Center, Keio
University, Hiyoshi 3-14-1, Kohoku, Yokohama 223-8522, Japan  Paul D. Nation
IBM Quantum, Yorktown Heights, New York 10598, USA  Takao Kobayashi
Mitsubishi Chemical Corporation, Science & Innovation Center, 1000, Kamoshida-
cho, Aoba-ku, Yokohama 227-8502, Japan  Quantum Computing Center, Keio
University, Hiyoshi 3-14-1, Kohoku, Yokohama 223-8522, Japan  Yu-ya Ohnishi
Quantum Computing Center, Keio University, Hiyoshi 3-14-1, Kohoku, Yokohama
223-8522, Japan  Materials Informatics Initiative, RD technology and digital
transformation center, JSR Corporation, 3-103-9, Tonomachi, Kawasaki-ku,
Kawasaki, Kanagawa, 210-0821, Japan  Hiroyuki Tezuka  Quantum Computing
Center, Keio University, Hiyoshi 3-14-1, Kohoku, Yokohama 223-8522, Japan
Advanced Research Laboratory, Technology Infrastructure Center, Technology
Platform, Sony Group Corporation, 1-7-1 Konan, Minato-ku, Tokyo, 108-0075,
Japan  Naoki Yamamoto<EMAIL_ADDRESS>Quantum Computing Center, Keio
University, Hiyoshi 3-14-1, Kohoku, Yokohama 223-8522, Japan

###### Abstract

Integration of quantum chemistry simulations, machine learning techniques, and
optimization calculations is expected to accelerate material discovery by
making large chemical spaces amenable to computational study; a challenging
task for classical computers. In this work, we develop a combined quantum-
classical computing scheme involving the computational-basis Variational
Quantum Deflation (cVQD) method for calculating excited states of a general
classical Hamiltonian, such as Ising Hamiltonian. We apply this scheme to the
practical use case of generating photochromic diarylethene (DAE) derivatives
for photopharmacology applications. Using a data set of 384 DAE derivatives
quantum chemistry calculation results, we show that a factorization-machine-
based model can construct an Ising Hamiltonian to accurately predict the
wavelength of maximum absorbance of the derivatives, $\lambda_{\rm max}$, for
a larger set of 4096 DAE derivatives. A 12-qubit cVQD calculation for the
constructed Ising Hamiltonian provides the ground and first four excited
states corresponding to five DAE candidates possessing large $\lambda_{\rm
max}$. On a quantum simulator, results are found to be in excellent agreement
with those obtained by an exact eigensolver. Utilizing error suppression and
mitigation techniques, cVQD on a real quantum device produces results with
accuracy comparable to the ideal calculations on a simulator. Finally, we show
that quantum chemistry calculations for the five DAE candidates provides a
path to achieving large $\lambda_{\rm max}$ and oscillator strengths by
molecular engineering of DAE derivatives. These findings pave the way for
future work on applying hybrid quantum-classical approaches to large system
optimization and the discovery of novel materials.

††preprint: APS/123-QED

##  I Introduction

Unprecedented improvements in the cost-effectiveness ratio of computers,
together with improved computational techniques, enables quantum chemistry
calculations to be widely applied in material design for finding novel
molecules that have desirable properties [1, 2, 3]. On the other hand, since
the number of candidate molecules is numerous, even utilizing supercomputing
resources, it is time consuming to screen all molecules with $ab$ $initio$
calculations. Over the last 10 years, the use of machine learning and
optimization methods to search for optimal molecular candidates in large
chemical spaces has greatly accelerated [4] as the computational cost of these
methods is several orders of magnitude lower than corresponding $ab$ $initio$
methods.

However, although using machine learning and optimization methods for material
design shows great promise, there are fundamental problems associated with
these methods that will need to be overcome before mainstream usage. One issue
is related to the size and quality of data sets used for training machine
learning models derived from quantum chemistry calculations. Due to the
chemical diversity in standard data sets, the prediction of molecular
properties from a data set trained by a machine learning model derived from
differing data sources is challenging. Specifically, it has been shown that
some under represented functional groups in the golden standard quantum
chemistry-based QM9 data set [5] is the primary source of outliers in property
prediction [6]. Increasing the training data set size may improve the ability
of machine learning models to predict molecular properties, however, such
improvements come at the cost of requiring significantly more computational
resources.

Another issue relates to the difficulties inherent in searching a large
chemical space with discrete optimization methods. For example, while the
number of potential drug-like molecules is estimated to be between $10^{23}$
and $10^{60}$, the number of all the synthesized molecules is $\sim 10^{8}$
[7]. Since this small synthesized chemistry space is not a faithful
representation of the full molecular space, finding an optimal target from
this reduced data set is a difficult task. Moreover, since brute force search
is not tractable for large data sets, efficient heuristic algorithms, such as
simulated annealing [8] and Genetic algorithms [9], are needed to find the
solution in a reasonable amount of time. Work looking at converting a discrete
optimization to a gradient-based multidimensional continuous optimization also
seems promising [10].

Quantum computing offers one potential route to remarkably speed up
computational approaches for material design relative to classical computers.
For some problems with computational time that scales exponentially with
system size on classical computers, the corresponding time may increase only
polynomially using quantum algorithms [11, 12]. Some recent studies also show
that heuristic quantum optimization algorithms, such as the Variational
Quantum Eigensolver (VQE) [13] and Quantum Approximate Optimization Algorithm
(QAOA) [14], may have better polynomial-time computational cost than that of
the best classical heuristic methods [15, 16]. A number of benchmark studies
of applying quantum computing to the calculation of chemical reaction energy
profiles [17, 18], excited states energy spectra [19, 20] and searching lead
molecules [21] demonstrate the potential of using quantum computing in new
materials discovery.

In this work, we extend our previous quantum-classical computing scheme [21]
to search for optimal materials with a set of required properties. The scheme
uses quantum chemistry and machine learning on classical machinery followed by
optimization on a quantum device as part of a workflow for materials
discovery. Figure 1 shows the workflow typical for searching optimal
prothochromic molecules, illustrating the concept of the proposed molecular
design method. In detail, we first perform quantum chemistry calculations on a
classical computer to find the target properties for a set of candidate
molecules. We then use these results as a data set to construct an Ising
Hamiltonian via machine learning for predicting target properties of the
system. In particular, we perform a optimization computation on a quantum
device, together with quantum chemistry calculations on classical machinery,
to produce the ground and excited states of the constructed Ising Hamiltonian;
these states serve as a set of candidate molecules that may satisfy required
properties from which we can select the best molecule for the practical use.

Optimization methods like QAOA and VQE are only used to calculate the ground
state of a Hamiltonian, and thus these methods are limited to searching only
one solution from a pool of candidates molecules, and are difficult to provide
enough information to guide the design of novel molecules. For the purpose of
searching excited states, in addition to the ground state, we employ a
Variational Quantum Deflation (VQD) [22] technique. This method computes the
overlap between the ground state and the current state and takes advantage of
the fact that if the Hamiltonian is limited to a classical one that is
diagonal in the computational-basis (such as the Ising Hamiltonian), both the
ground and excited states are the computational-basis states. Accordingly, the
process of calculating the overlap is reduced to computing the ratio of
hitting the ground state on the computational-basis measurement result. This
can drastically reduce the technical noise induced into VQD. To emphasize this
advantage, we term this special type of VQD as the computational-basis VQD or
cVQD. Note that the SWAP test algorithm can be taken for the purpose of
diagonalization, but at the expense of an increase in the introduced noise.

We illustrate the ability of the combined quantum-classical method by taking a
first step toward a go-beyond-concept industrial relevant use case of
designing diarylethene (DAE) derivatives for photopharmacology application.
Photopharmacology proposed by Feringa, a 2016 Nobel laureate in chemistry
[23], is an emerging technology in medicine which activates and deactivates
photo-switchable molecules with light for target drug delivery to prevent side
effects [24, 25] and exposure to the environment of antibiotics [26, 27].
Among the prothochromic switch materials, DAE derivatives are particularly
attractive compounds for use in photopharmacology due to their outstanding
photochromic performance and thermal stability [28, 29]. The most common DAE
derivative shown in Fig. 2 has reversible transformation between open-ring and
closed-ring forms, where the open-ring and closed-ring forms have UV and
Visible absorption spectra, respectively. To decrease the damage to tissues
from light in photopharmacology applications, a key feature of designing DAE
derivative is to make the open-ring form have a long wavelength of absorption
light required to cause the photoisomerization.

The first step in designing DAE derivatives is generating a pool of 4096 DAE
derivative candidates. We show that, by using the quantum chemistry
calculation results of 384 DAE derivative as a training data set, the
factorization-machine (FM)-based model can construct an Ising Hamiltonian
which has 4096 states corresponding to 4096 DAE derivative; every state energy
is represented via the wavelength $\lambda_{\rm max}$ in the absorption
spectrum, such that the absorbance takes the maximum value of the
corresponding DAE derivatives. In this way, the machine learning will
accurately predict $\lambda_{\rm max}$ property in a much more rapid way,
compared with quantum chemistry calculations.

In the experiment, we performed VQE and QAOA methods to obtain the ground
state; then we apply VQD and cVQD methods to obtain the first four excited
state of the Ising Hamiltonian on a quantum simulator. We found that VQE
outperforms the QAOA method in terms of both calculation accuracy and circuit
depth. We also show that, in the absence of sampling error from quantum
simulator, cVQD is capable of yielding markedly more accurate excited states
than those from VQD. This is because the formalism in cVQD relies on the fact
that the excited state of the Ising Hamiltonian is one of the computational
basis, whereas VQD is originally developed for calculating the excited states
of a quantum chemistry Hamiltonian.

After showing the ability of cVQD to accurately calculate the excited states
of an Ising Hamiltonian, we then performed the same calculations on a quantum
processor. We show that, although the noise from device leads to a decrease of
calculation accuracy, cVQD can successfully produce accurate results for all
the excited states. We also show that a rather accurate, e.g., better than a
factor of 5, prediction of excited state can by achieved by using readout
error mitigation and reducing decoherence noise via dynamical-decoupling error
suppression.

Finally, we compute the oscillator strength (Osc) of 5 DAE derivatives
corresponding to the ground and first four excited states of the Ising
Hamiltonian with quantum chemistry methods on a classical computer. We also
analysed the molecular orbitals contribution to the excitation. These results
guide the design of new DAE deviates with large $\lambda_{\rm max}$ and Osc
which are prerequisites for photopharmacology applications. Broadly speaking,
these results show that the combination of quantum chemistry (on a classical
computer), machine learning, and quantum computation could provide chemists
with new tools in their arsenal for material discovery.

Figure 1:  Flowchart of the hybrid quantum-classical method for identifying
the optimal DAE derivatives with large $\lambda_{\rm max}$ and Osc. Figure 2:
(a) Reversible photoisomerization of the most common DAE derivative and the
numbering of 6 positions in thiophene rings used to generate various of DAE
derivatives by bonding with funtional groups of -H, -Me, -CN, and -F. (b)
Totally 4096 derivatives are classified as compounds containing 1 or 2, 3 and
4 kinds of functional groups.

##  II Methods

###  II.1 Computational Methodology for discovery of photochromic materials

Figure 1 shows the reversible photoisomerization of the most common DAE
derivative (i.e. a cyclic 1,2-ethenedienyl bridge connecting two thiophene
rings) between open-ring and closed-ring forms by the absorption of
electromagnetic radiation, where the open-ring and closed-ring forms have UV
and visible absorption spectra, respectively. Additional functional groups to
the 6 positions of two thiophene rings can shift the absorption band of the
open-ring form toward visible region, which is essential for the
photopharmacology application. In this study, we identify open-ring candidates
with long wavelength absorption and large Osc for the use as photo switch
material through combinatorial optimization from 4096 DAE derivatives which
are generated using -H, -Me, -CN and -F functional groups. By setting -H, -Me,
-CN and -F as bit values of 10, 11, 00 and 01 respectively, the combinatorial
optimization problems to find DAE derivatives which has required properties
can be viewed as a problem for searching the corresponding bitstrings
represented by 12 qubits on a quantum computer. Here, it is important to bear
in mind that, since these 4 groups have different donor and acceptor
strengths, the calculation results can also provide a path to tune the energy
and probability of absorption in DAE molecules.

We here again describe the entire procedure for identifying DAE candidates
with long wavelength absorption and large Osc, illustrated in Fig. 1. Once a
number of DAE derivatives are chosen, the wavelength of maximum absorbance of
the derivatives, $\lambda_{\rm max}$, is obtained from quantum chemistry
calculations on a classical computer. With the calculation results, two data
sets called training data set and test data set are prepared. The training
data set is used to train a machine learning model to predict $\lambda_{\rm
max}$. The test data set is used to validate the performance of the trained
model by comparing the predicted values with those in the test data set. If
the prediction accuracy of the trained model cannot achieve a desired
threshold, further quantum chemistry calculation results are added to the
training data set for training a better prediction model. The prediction model
is repeated until the desired accuracy is achieved. With the improved model,
an Ising Hamiltonian is constructed and is used to perform optimization on a
quantum computer to find the DAE candidate which has the largest $\lambda_{\rm
max}$. The Osc of the candidate are then obtained from quantum chemistry
calculation on a classical computer. If Osc is below a chosen threshold, an
”overlap” term is added to the cost function and additional optimization on a
quantum computer is performed to find the DAE candidate which has the 2nd
largest $\lambda_{\rm max}$. The procedure of the Osc calculation on classical
computer and optimization on quantum computer is repeated until a DAE
candidate with large $\lambda_{\rm max}$ and Osc is found. The following
sections describe the details of the calculation in each stage.

###  II.2 Construction of the Prediction Model

All the initial structures of 4096 DAE derivative are generated using
rdkit.Chem.rdChemReactions module in the RDkit Software [30]. The minimum
energy conformer of every DAE derivative is determined from the Universal
force field [31] in the same software. The determined conformer is used as the
initial structure to perform ground state (S0) geometry optimization at the
$\omega$B97X-D/def2-SVP [32] level with the density fitting (DF)
approximation. The $\lambda_{\rm max}$ (i.e. the first singlet excited state
(S1) absorption wavelength) and Osc (between S0 and S1 states) at the S0
optimized structures of 4096 DAE detivatives were calculated using the
TD-$\omega$B97X-D/def2-SVP with the DF approximation. The (TD-)DFT [33]
calculations were performed by Gaussian16 [34].

Setting functional groups of -H, -Me, -CN and -F as bit values of 10, 11, 00
and 01, every DAE derivative, which has these function groups on the positions
of R1 to R6 as shown in Fig. 1, is transformed into a binary feature vector
$\vec{x}^{(s)}$ comprised of 12 qubits. A data set is prepared with all the
$\vec{x}^{(s)}$ and its corresponding $\lambda_{\rm max}$ values of $y^{(s)}$
as shown in Table 1. We firstly chose all the DAE derivatives in which the
structure has only one kind of functional groups as the training data. We also
randomly select 60 derivatives from the structure containing two kinds of
functional groups and add them to the training data. The chosen training data
are used to train a prediction model described by quadratic unconstrained
binary optimization (QUBO) formulation [35]:

$\displaystyle E(q)$ $\displaystyle=$ $\displaystyle{\omega_{0}+}\sum_{1\leq
i\leq j\leq 12}Q_{ij}q_{i}q_{j}$ (1) $\displaystyle=$
$\displaystyle{\omega_{0}+}\sum_{i=1}^{12}Q_{ii}q_{i}+\sum_{i=1}^{11}\sum_{j=i+1}^{12}Q_{ij}q_{i}q_{j},$

where $\omega_{0}$ is a constant value, $q_{i}$ is the $i$th binary variable
of $\vec{x}$ which takes either 0 or 1, and the coefficients $Q_{ij}$ and
$Q_{ii}$ have real values. The second equality holds because
$q_{i}^{2}=q_{i}$. The quantities $Q_{ij}$ and $Q_{ii}$ are obtained through
the learning process; in particular, we use the FM predictor [36, 37, 38] with
the following model equation:

$\displaystyle\hat{y}(\vec{x})$ $\displaystyle=$
$\displaystyle{\omega_{0}+}\sum_{i=1}^{12}w_{i}q_{i}+\sum_{i=1}^{11}\sum_{j=i+1}^{12}(\vec{v}_{i}\cdot\vec{v}_{j})q_{i}q_{j},$
(3)
$\displaystyle(\vec{v}_{i}\cdot\vec{v}_{j})=\sum_{f=1}^{\kappa}v_{if}v_{jf},$

where $\kappa$, the dimension of the factorization, is set to 8 during the
learning process. The goal is to determine $(\omega_{0},w_{i},\vec{v}_{i})$
and accordingly $Q_{ii}=w_{i}$ and $Q_{ij}=\vec{v}_{i}\cdot\vec{v}_{j}$, by
minimizing the cost function $\sum_{s}(y^{(s)}-\hat{y}(\vec{x}^{(s)}))^{2}$.
Note that although the QUBO model can also be determined by the regression of
$Q_{ij}$ and $Q_{ii}$ in Eq. (1), the advantage of using the FM predictor is
that it can reliably fit the parameters $w_{i}$, $\vec{v}_{i}$ and
$\vec{v}_{j}$ with a small training data set and prevent the trained model
from over-fitting.

The performance of the QUBO model is evaluated by checking the correlation
(i.e. correlation coefficient denoted by $R$) between the $\lambda_{\rm max}$
values of the molecular structures in test data set predicted by QUBO model
and quantum chemistry calculations. If $R$ is less than a threshold value of
0.85, additional randomly selected 60 DAE derivatives, which have two kinds of
functional groups in the structure, are added to the training data to retrain
the QUBO model. The procedure repeats by adding every 60 DAE derivatives to
the training data until $R$ achieves the value larger than the threshold
value. After all the derivatives containing two different functional groups
(i.e. the derivatives in the light blue region of Fig. 4(b)) are added to the
training data set, the derivatives which have three and four kinds of the
functional groups (i.e. the derivatives in the blue and dark blue regions of
Fig. 4(b)) are also used as the training data.

Figure 3:  12-qubit circuits for (a) VQE with the $R_{y}$ Ansätze with depth 1 and (b) QAOA with depth $p=1$. Table 1: Dataset obtained from quantum chemistry calculations of DAE derivatives for training and validating the machine learning model. Every two bit values of 10, 11, 00 and 11 in the feature vector $\vec{x}^{(s)}$ correspond to the functional groups of -H, -Me, -CN and -F in the DAE derivatives.  feature vector $\vec{x}^{(s)}$ (functional groups of R1-R6) | target value $y^{(s)}$ ($\lambda_{\rm max}$)
---|---
$\vec{x}^{(1)}$ |  0 |  0 |  0 |  0 |  0 |  0 |  0 |  0 |  0 |  0 |  0 |  0 |  (CN-CN-CN-CN-CN-CN) |  $y^{(1)}$ |  274.99
$\vec{x}^{(2)}$ |  0 |  0 |  0 |  0 |  0 |  0 |  0 |  0 |  0 |  0 |  0 |  1 |  (CN-CN-CN-CN-CN-F) |  $y^{(2)}$ |  282.63
$\vec{x}^{(3)}$ |  0 |  0 |  0 |  0 |  0 |  0 |  0 |  0 |  0 |  0 |  1 |  0 |  (CN-CN-CN-CN-CN-H) |  $y^{(3)}$ |  283.59
$\vec{x}^{(4)}$ |  0 |  0 |  0 |  0 |  0 |  0 |  0 |  0 |  0 |  0 |  1 |  1 |  (CN-CN-CN-CN-CN-Me) |  $y^{(4)}$ |  282.42
$\vdots$ |  $\vdots$ |  $\vdots$ |  $\vdots$ |  $\vdots$ |  $\vdots$ |  $\vdots$ |  $\vdots$ |  $\vdots$ |  $\vdots$ |  $\vdots$ |  $\vdots$ |  $\vdots$ |  (R1-R2-R3-R4-R5-R6) |  $\vdots$ |  $\vdots$
$\vec{x}^{(4094)}$ |  1 |  1 |  1 |  1 |  1 |  1 |  1 |  1 |  1 |  1 |  0 |  1 |  (Me-Me-Me-Me-Me-F) |  $y^{(4094)}$ |  282.52
$\vec{x}^{(4095)}$ |  1 |  1 |  1 |  1 |  1 |  1 |  1 |  1 |  1 |  1 |  1 |  0 |  (Me-Me-Me-Me-Me-H) |  $y^{(4095)}$ |  284.31
$\vec{x}^{(4096)}$ |  1 |  1 |  1 |  1 |  1 |  1 |  1 |  1 |  1 |  1 |  1 |  1 |  (Me-Me-Me-Me-Me-Me) |  $y^{(4096)}$ |  282.58

###  II.3 Optimizations on the quantum device

Figure 4: $\lambda_{\rm max}$ of all the 4096 DAE derivatives from quantum
chemistry calculations

To perform the optimization task on a quantum computer, the QUBO model $E(q)$
is converted into a spin-based Ising Hamiltonian [39, 40, 41]:

$H=\sum_{i=1}^{12}h_{i}Z_{i}+\sum_{i=1}^{11}\sum_{j=i+1}^{12}J_{ij}Z_{i}Z_{j},$
(4)

where $Z_{i}$ is the single-qubit Pauli $Z$ operator acting on the $i$th
qubit. The optimization for calculating the ground states of Eq. (4) is
performed using VQE and QAOA methods.

Ideally, VQE gives us the ground state energy of $H$ by minimizing the mean
energy

$E(\vec{\theta})=\langle{\psi(\vec{\theta})|H|\psi(\vec{\theta})}\rangle,$ (5)

with respect to the vector of parameters $\vec{\theta}$ of Ansätze

$|\psi(\vec{\theta})\rangle=U(\vec{\theta})|\psi_{0}\rangle,$ (6)

where $U(\vec{\theta})$ is the unitary operator of a parameterized quantum
circuit and $|\psi_{0}\rangle$ is the initial state. In this work,
$U(\vec{\theta})$ is set to the $R_{y}$ heuristic Ansätze, which is
constructed of only fixed CNOT gates and $R_{y}$ gates with angle parameters
$\vec{\theta}$; the 12-qubit circuit featuring the $R_{y}$ Ansätze has 11
nearest-neighbor CNOT gates and 24 optimization parameters $\vec{\theta}$ as
shown in Fig. 3(a). The parameter $\vec{\theta}$ is updated so that the mean
energy $E(\vec{\theta})$ decreases in each iteration.

The parameterized Ansätze state for QAOA is constructed from the Hamiltonian
$H$ with single qubit Pauli $X$ operators as

$\displaystyle|\psi_{p}(\vec{\gamma},\vec{\beta)}\rangle$ $\displaystyle=$
$\displaystyle e^{-i\beta_{p}B}e^{-i\gamma_{p}H}\cdots
e^{-i\beta_{1}B}e^{-i\gamma_{1}H}|+\rangle^{\bigotimes 12},$ $\displaystyle B$
$\displaystyle=$
$\displaystyle\sum_{i=1}^{12}X_{i},~{}|+\rangle=\frac{1}{\sqrt{2}}(|0\rangle+|1\rangle),$
(7)

where $p$ is an integer parameter representing the number of repetitions of
the unitary operators. Like VQE, the parameters
$\\{(\gamma_{i},\beta_{i})\\}_{i=1,\ldots,p}$ are updated so that the mean
energy
$E(\vec{\gamma},\vec{\beta})=\langle{\psi_{p}(\vec{\gamma},\vec{\beta)}|H|\psi_{p}(\vec{\gamma},\vec{\beta)}}\rangle$
decreases in each iteration. An example of QAOA circuit with $p=1$ is shown in
Fig. 3(b). The advantage of using the QAOA Ansätze is that the adiabatic
dynamics can be used to effectively find the ground state of the Hamiltonian
to solve the combinatorical optimization problem.

All quantum optimization calculations were performed with the statevector
simulator contained in the Aer module of Qiskit 0.37 [42], as well as on the
27-qubit IBM Quantum $ibm\\_kawasaki$ quantum device. The statevector
simulator uses linear algebra operations to compute the Ansätze state and
expectation values exactly. The parameter values in the VQE algorithm with the
$R_{y}$ Ansätze and QAOA were calculated using the Sequential Least Squares
Programming (SLSQP) optimizer on the simulator. Since the computational cost
of QAOA is rather demanding, only VQE calculations were performed on the
quantum device with Simultaneous Perturbation Stochastic Approximation (SPSA)
optimization. Twelve linearly connected qubits were selected to avoid the need
for SWAP gates; see Appendix A.

###  II.4 Computational-basis variational quantum deflation for the
calculations of excited states of Ising Hamiltonian

QAOA and VQE are only used to calculate ground state of Ising Hamiltonian and
thus such methods are limited to search one solution from a pool of candidates
molecules and are difficult to provide enough information to guide the design
of novel molecules. For piratical use we need excited states for the Ising
Hamiltonian in order to construct a set of candidate molecules, which, in our
case, is a set of the DAE derivatives corresponding to different energy
levels. For this purpose, we employ VQD [22], which uses a variational
technique to compute the excited states for a general Hamiltonian. More
specifically, the first excited state is calculated by minimizing the
following modified cost function:

$E(\vec{\theta})=\langle{\psi(\vec{\theta})|H|\psi(\vec{\theta})}\rangle+\beta_{0}|\langle{\psi(\vec{\theta})|\psi_{0}}\rangle|^{2},$
(8)

where $|\psi_{0}\rangle$ is the reference ground state; that is, at first we
need to prepare the ground state for using VQD. That is, VQD decreases the
energy under the constraint that the current state
$|\psi(\vec{\theta})\rangle$ is orthogonal to the reference ground state
$|\psi_{0}\rangle$. The parameter ${\beta}_{0}$ represents the weight of the
constraint. Ideally, $|\psi(\vec{\theta})\rangle$ converges to the first
excited state.

cVQD is a special type of VQD, which focuses on a general classical
Hamiltonian that is diagonal in the computational basis $\\{|k\rangle\\}$. The
ground state is one of them, which we denote as $|k_{0}\rangle$. Now, let us
represent the current state in the computational-basis as follows:

$|\psi(\vec{\theta})\rangle=\sum_{k}\alpha_{k}(\vec{\theta})|k\rangle.$

By substituting this equation for Eq. (8), we have

$E(\vec{\theta})=\langle{\psi(\vec{\theta})|H|\psi(\vec{\theta})}\rangle+\beta_{0}|\alpha_{k_{0}}(\vec{\theta})|^{2}.$
(9)

Minimization of Eq. (9) with $\beta_{0}\gg 1$ will give a good approximated
state of the first excited state. A clear advantage of cVQD is that the
penalty term $|\alpha_{k_{0}}(\vec{\theta})|^{2}$ can be easily estimated on
classical computer by using the ratio of hitting the ground state on the
computational-basis measurement result. This is in stark contrast to VQD,
where the calculation accuracy of
$|\langle{\psi(\vec{\theta})|\psi_{0}}\rangle|^{2}$ is largely influenced by
the noise from the quantum device. Thus, cVQD is an efficient and robust
algorithm for specifically calculating excited states of a classical
Hamiltonian such as the Ising Hamiltonian. In Section III-B, we will show
that, on a noisy quantum simulator, cVQD provides much more accurate results
in searching the excited states than the standard VQD protocol that uses SWAP
test.

##  III Results and Discussion

###  III.1 Quantum Chemistry and Machine Learning

Figure 5:  Correlation values between $\lambda_{\rm max}$ obtained from the
quantum chemistry calculations and QUBO models, as a function of a training
data size, is shown in (a). Predicted $\lambda_{\rm max}$ for all the DAE
derivatives by the quantum chemistry calculations and QUBO models, trained by
the FM predictor with (b) 384, (c) 2916 and (d) 4096 DAE derivatives.

Figure 4 shows $\lambda_{\rm max}$ of all the 4096 DAE derivatives obtained
from the quantum chemistry calculations. We found that, by changing the
functional groups bonded to the thiophene rings, DAE derivatives exhibits
$\lambda_{\rm max}$ in the range from 242.83 nm to 348.75 nm. Such wide range
indicates that it may be sufficient to modify the absorption spectrum by
adding functional groups to the DAE molecule. The DAE derivatives that showed
the first and second largest $\lambda_{\rm max}$ among all the derivatives
were CN-Me-CN-Me-H-Me and Me-CN-Me-CN-Me-H. Functional groups of -Me which
have strong donor strengths are found in one thiophene ring of these two
derivatives. Similarly, functional groups of -CN which have strong acceptor
strengths are found in the another thiophene ring. These results mean that
$\lambda_{\rm max}$ may be sufficiently tunable by modifying the substituents
of the thiophene ring moieties to modulate their donor and acceptor strengths.

The ability of the machine learning model obtained from the FM approach with
different number of training data to accurately predict $\lambda_{\rm max}$
values is investigated by comparing the results predicted by the QUBO model
with quantum chemistry calculations, as shown in Fig. 5. With the smallest
training data set which has 64 DAE derivatives, the QUBO model fails to
accurately predict $\lambda_{\rm max}$ values, i.e., the correlation of $R$
between $\lambda_{\rm max}$ values from the QUBO model and quantum chemistry
calculation is 0.16. Increase of the training data set size from 64 to 436
derivatives significantly improves $R$ to 0.86, while further increase of the
data set size from 436 to 1036 derivatives has only a minor improvement of $R$
by 0.05. With the size bigger than 1036 derivatives, the accuracy of the the
QUBO model appears to level off. Comparison of $\lambda_{\rm max}$ predicted
by the QUBO model with the quantum chemistry calculation values also gives
evidence that the QUBO model slightly underestimates $\lambda_{\rm max}$ for
the derivatives which have relatively small and large $\lambda_{\rm max}$
values. The underestimation of $\lambda_{\rm max}$ is possibly attributed to
the imbalanced distribution of the training data set. In detail, $\lambda_{\rm
max}$ of a large proportion of DAE derivatives takes values in the range
270-300 nm. Thus, the training model spends most of its time on the large
proportion of the data set and cannot learn enough from the DAE derivatives of
which $\lambda_{\rm max}$ is smaller than 270 nm or larger than 300 nm.
Nevertheless, since the trend of $\lambda_{\rm max}$ predicted by the QUBO
model are in quantitative agreement with results of quantum chemistry
calculations and $\lambda_{\rm max}$ of the suggested DAE derivative will be
finally determined by quantum chemistry calculations, the performance of the
QUBO model is considered to be good enough to be used for the purpose of
finding the optimal DAE derivative.

Figures 5b-d show that the $\lambda_{\rm max}$ distribution obtained from the
quantum chemistry calculations is similar to those from the QUBO model trained
by using the dataset of DAE derivatives containing 1-2 kinds (Fig. 5b), 1-3
kinds (Fig. 5c) and 1-4 kinds (Fig. 5d) of functional groups. These results
suggest that the FM approach can learn $\lambda_{\rm max}$ values well, using
the 384 DAE derivatives which have 1-2 kinds of functional groups in the
structures. The good performance of the QUBO model trained by the small data
set of 384 DAE derivatives is presumably due to that the $\lambda_{\rm max}$
values of most of all the DAE derivatives can be mainly described as a
transition between two molecular orbitals localized on the two thiophene rings
of DAE derivatives. Since the number of the DAE derivatives grows on the order
of $n^{6}$ with $n$ the number of functional groups, even with $n=10$ it would
be a daunting task to explore an optimal DAE derivative with desired
$\lambda_{\rm max}$ and Osc using quantum chemistry approaches. Our results
show the possibility that, with machine leaning approach, the computational
cost of quantum chemistry calculations can be reduced to the order of
$2^{6}n^{2}$.

Figure 6: Energies of the ground state (GS) and 1-4 excited states (ES1-ES4)
of Ising Hamiltonian obtained from the exact diagonalization, VQD and cVQD
calculations with (a) statevector and (b) qasm simulators. The highest
probability bitstrings of the GS and ES1-ES4 obtained from VQD and cVQD
calculations with qasm simulator is shown in (c).

###  III.2 Simulations with the statevector and qasm simulator

Table 2: Ground state energies and corresponding bitstrings appearing with the highest probability, computed using various Ansätze with the statevector simulator. In addition, Ansätze properties comprising the number of CNOT gates and the number of optimization parameters (opt params) are shown.  Method | Ground state | Ansätze
---|---|---
| energy | bitstring (functional groups of R1-R6) | probability | CNOTs | opt params
exact eigensolver |  334.527 |  [110011001110] (Me-CN-Me-CN-Me-H) |  1.00 |  - |  -
VQE |  334.527 |  [110011001110] (Me-CN-Me-CN-Me-H) |  1.00 |  11 |  24
QAOA $(p=1)$ |  313.827 |  [110011001110] (Me-CN-Me-CN-Me-H) |  0.05 |  132 |  6
QAOA $(p=2)$ |  323.982 |  [110011001110] (Me-CN-Me-CN-Me-H) |  0.15 |  264 |  14
QAOA $(p=3)$ |  332.051 |  [110011001110] (Me-CN-Me-CN-Me-H) |  0.25 |  396 |  22

By converting the QUBO model to an Ising model (4), the problem of searching
an optimal DAE derivative can be mapped to a quantum optimization problem
whose ground state corresponds to the optimal DAE derivative. The ground state
energy and its corresponding bit-strings calculated by the exact eigensolver
(i.e., an exact diagonalization of the Ising model Hamiltonian), VQE and QAOA
algorithms on the statevector simulator for the Ising models, are shown in
Table 2. VQE with a heuristic $R_{y}$ Ansätze reproduces the ground state
obtained from the exact eigensolver. The bit-string of (110011001110) which
corresponds to the optimal DAE derivative has the probability of 1.00, meaning
that this correct solution is always obtained. That is, the quantum
optimization using VQE algorithm successfully found the exact ground state of
the Ising model Hamiltonian. On the other hand, unlike VQE calculation, all
three type of QAOAs with $p=1,2,3$ did not converge to the exact ground state;
still, in every case of $p$, the most probable bit-strings of the converged
state corresponds to the optimal DAE derivative. The converged energy of the
QAOA calculations increases and its corresponding highest probability of the
bit-string also increases, with increasing the value of $p$ in Eq. (7). The
improvement with the increase of the value $p$ is presumably due to that the
usage of a large value of $p$ gives an appreciably more accurate description
of the time-evolution operator [13]. On the other hand, even with $p=3$, a
relatively larger deviation of 2.476 nm between the QAOA calculation result
(331.051 nm) and the exact value (334.527 nm) is observed; moreover, the
highest provability value of hitting the correct solution is 0.25, which is
lower than that of VQE calculation by a factor of 4. Further increase of $p$
should improve the ability of the QAOA calculation to accurately converge to
the ground state; however, such improvements add significant computational
expense on both simulator and quantum devices. These results suggest that VQE
outperforms QAOA in terms of both calculation accuracy and circuit depth.

Based on the higher accuracy and lower computational cost of ground state
calculations using VQE than QAOA, we decide to use the heuristic $R_{y}$
Ansätze to calculate 1st to 4th excited states of the Ising Hamilitonian with
VQD and cVQD. The energies of the ground and the four excited states
calculated by the exact eigensolver, VQD and cVQD algorithms on the
statevector simulator are shown in Fig. 6(a). The same results are predicted
by VQD and cVQD for the energies of the ground state, due to the fact that
both algorithms use the same VQE to calculate the zeroth approximator for the
ground state. Although VQD and cVQD use different strategies to calculate
excited states, both predict very similar energies for all the four excited
states on statevector simulator. Moreover, both VQD and cVQD well reproduce
the excited energies predicted by the exact eigensolver. These results imply
that both VQD and cVQD can accurately predict excited state energies of the
Ising Hamiltonian on statevector simulator.

Using VQD and cVQD for calculating excited states of the Ising Hamiltonian on
statevector simulator shows great promise, but there is a fundamental problem
associated with performing VQD and cVQD on qasm and quantum device; namely,
the fact that sampling and device noise influence the overlap term, which is
necessary to search the excited states. Figure 6(b,c) compares all the
calculated excited state energies of the Ising Hamiltonian and their
corresponding highest probability bitstrings using VQD and cVQD on a qasm
simulator with 81920 shots. The excited state energies obtained from the
calculation using VQD is 0.085 nm higher than the exact values, in contrast to
that obtained by using cVQD which is only 0.038 nm higher. These results
indicate that the excited states computed from cVQD possesses more fidelity
with the exact excited state than that obtained by using VQD.

More importantly, VQD leads to a maximum error of 0.23 nm higher than the
energies obtained with the exact eigensolver, whereas cVQD results in a
maximum error that are only 0.06 nm larger than the exact values. These
results signify the effectiveness of cVQD to accurately calculate all the
overlap terms which are needed to reliably calculate excited states. Figure
6(b,c) also indicates that the accuracy of the cVQD calculations may be
invariant to changes in the level of excited state, because the exact states
are used as the reference states in cVQD. Accordingly, although the bitstrings
appearing with the highest probability for all the excited states are the same
between the VQD and cVQD calculations, VQD yields probabilities an average of
0.16 lower than cVQD.

The number of CNOT gates in the circuit used for VQE and QAOA is shown in
Table 2; this information shows which optimization algorithms are suitable for
use on the $ibm\\_kawasaki$ quantum device for the 12-qubit system
calculation. Quantum circuits for VQE with the $R_{y}$ Ansätze possess 11 CNOT
gates, whereas 132 CNOT gates are required for the QAOA calculation with
$p=1$. Furthermore, the number of CNOT gates linearly increases with the value
of $p$ with the proportional coefficient 132. Note that, since the qubits of
$R_{y}$ Ansätze are connected in a line shape, this Ansätze can be directly
implemented on $ibm\\_kawasaki$, without additional qubits for SWAP
operations. On the other hand, since QAOA does not have such linear structure
for connecting the qubits, overhead of the number of CNOT gates by a factor of
7-8 is needed for implementing this Ansätze on a quantum device. Because the
CNOT error rate largely influences on the computation result, VQE is expected
to provide higher accuracy than those provided by QAOA. Thus, we decided to
use $R_{y}$ Ansätze to implement VQE and cVQD to obtain the ground and excited
states, respectively, of Ising Hamiltonian on $ibm\\_kawasaki$.

Figure 7:  The energy minimization process of VQE and cVQD with
$ibm\\_kawasaki$ is shown in (a). Energies ((b) upper panel) and the highest
probability bitstring ((b) lower panel) of the ground states obtained from the
$ibm\\_kawasaki$, qasm and statevector simulations with $R_{y}$ Ansätze.
Optimized parameters for $R_{y}$ Ansätze were determined from VQE calculations
on $ibm\\_kawasaki$. Bitstrings versus their occurring probabilities, for the
states obtained from qasm and statevector simulators(c). The energies ((d)
upper panel) and the highest probability bitstrings ((d) lower panel) of the
GS and ES1-ES4 obtained from the unmitigated $ibm\\_kawasaki$,
$ibm\\_kawasaki$ with M3 and M3/DD methods. Figure 8:  (a) Quantum chemistry
calculation results of $\lambda_{\rm max}$ as a function of oscillator
strengths. (b) The HOMO, LUMO for the five DAE candidate derivatives suggested
by cVQD calculations on $ibm\\_kawasaki$.

###  III.3 Calculations on $ibm\\_kawasaki$

Simulations using the VQE and cVQD algorithms with an Ising Hamiltonian and
$R_{y}$ Ansätze in Fig. 3(a) were performed on the 27-qubit $ibm\\_kawasaki$
device 111Note that, although the controlled-z gates in Fig. 3(a) can be
executed in parallel, we choose to execute them in serial, with idle times
populated with dynamical decoupling sequences as it yields higher fidelity
results. This is likely due to the always-on-coupling of cross-resonance based
devices, where the run-in-parallel gate operation can have larger error rates
compared to isolated execution. with 8192 shots. Detailed system information
can be found in Appendix A. The energy values returned as a function of the
number of iterations for each method are shown in Fig. 7(a). The same
convergence trend is observed between energies calculated from VQE and cVQD on
$ibm\\_kawasaki$ toward the exact eigensolver solution, with VQE and cVQD
calculations predicting energies several nm higher than the exact values.
Specifically, the converged energies of the ground state and the first four
excited states are found to be 25.36, 25.24, 25.67, 25.62 and 25.71 nm higher,
respectively. We note that, even on the noisy present-day quantum device, the
cVQD method is capable of accurately computing the energy level of the nearly
degenerate 2nd and 3rd excited states. The results reveal that, through the
use of the overlap term in Eq. (9), cVQD is capable of faithfully computing
the excited states of the Ising Hamiltonian on a quantum device.

Extending the analysis to understand the influence of noise from the quantum
device on the accuracy, we compare the ground state energies and the highest
probability bit-string obtained by the use of exact eigensolver with those
obtained from $ibm\\_kawasaki$ with the readout error mitigation, qasm and
statevector simulations with the parameter optimized $R_{y}$ Ansätze. Note an
efficient readour error mitigation technique called M3 (i.e. Matrix-free
Measurement Mitigation) [44] is used to approximately cancel measurement
assignment error. Figure 7(b) shows that VQE calculations without the error
mitigation yield energies about $25.36$ nm higher than the exact value. This
difference is reduced to $\sim 10.45$ nm using the readout error mitigation.
Accordingly, the value of the highest probability of the bit-string is
improved from $0.16$ to $0.39$. We also found that the energies from qasm
simulations with optimized $R_{y}$ Ansätze obtained using VQE on
$ibm\\_kawasaki$ has an error of $2.29$ nm to the exact value, and the
corresponding highest probability of the bit-string is $0.78$. The error and
the probability is almost the same as those obtained from statevector
simulations. Moreover, we observe that the probability values for the
remaining bit-strings from qasm simulations are also similar to those from
statevector simulations, as shown in Fig. 7 (c). These results suggest that
the readout error is the dominant noise source (64% effect), with only 0.5%
due to finite-sampling limitations placed on measurements of single circuit
executions (i.e. shots). The effect to the remaining error is considered from
other known sources including depolarization, dephasing, and deviations of
quantum gates from their normal unitary operations (coherent errors).

Having identified the main noise sources on our device, we apply readout error
mitigation to the resulting noisy distribution. Here we use the M3 method that
corrects the probability with a matrix-free preconditioned iterative-solution.
Moreover, we apply Dynamical Decoupling (DD) [45] to reduce decoherence by
repetitively inserting CPMG sequences [46] into the idle periods between
gates. Figure 7(d) shows that energies of the ground state and the first four
excited states obtained from VQE and cVQD without error mitigation approaches
are on average 25.44 nm larger than the exact values. Using the readout error
mitigation alone, the difference is improved to $\sim 6.45$ nm. However,
applying both the error suppression (i.e., DD) and the error mitigation yields
the best results giving energies that are within $3$ nm of the exact values.
The corresponding improvement of the highest probability bit-strings for the
ground and first four excited states is improved from $0.14$ to $0.68$ using
DD and M3 together.

Overall, given the influence of the noise from quantum device to the
calculation accuracy, VQE and the proposed cVQD methods give the highest
probability bit-strings corresponding to the exact ground and excited states,
and thus can be applied to finding the optimal DAE derivatives. Moreover,
through the combined use of error suppression and readout error mitigation,
the accuracy of the energy differences to the exact values can be improved by
a factor of $\sim 8$ and the corresponding bit-string probabilities improved
by $5x$.

###  III.4 DAE Molecular design strategy for photopharmacology application

$\lambda_{\rm max}$ and Osc of five candidate DAE derivatives obtained from
VQE and cVQD on $ibm\\_kawasaki$ were calculated at the
TD-$\omega$B97X-D/def2-SVP level and are shown in Fig. 8(a). The DAE
derivatives that showed the first and second largest $\lambda_{\rm max}$ are
CN-Me-CN-Me-H-Me and Me-CN-Me-CN-Me-H, whose structures are almost identical
to each other since the unsubstituted DAE has $\it{C}_{2v}$-like symmetry.
Those showing the third and fourth largest $\lambda_{\rm max}$ are Me-CN-Me-
CN-Me-CN and CN-Me-CN-Me-CN-Me, whose structures are also almost identical to
each other. However, these four DAE derivatives, which correspond to the
ground state and 1-3 excited states of the Ising Hamiltonian, may not be the
suitable candidate for photopharmacology application due to their small values
of Osc. The good balance between $\lambda_{\rm max}$ and Osc of Me-CN-Me-CN-
H-H DAE derivative makes it a much better candidate as a photo switch molecule
for photopharmacology application. These results suggest that the properties
of $\lambda_{\rm max}$ and Osc can be tuned by various structural ”handles”.

As shown in Fig. 8(b), since the HOMO and LUMO of the five candidate DAE
derivatives are spatially fairly well separated, we can assume for simplicity
that the absorption arises mainly from the transition between HOMO and LUMO
configurations localized on the two thiophene rings, respectively. Thus, a
large $\lambda_{\rm max}$ can be achieved by modifying the functional groups
of the two thiophene rings moieties to increase their donor and acceptor
strengths. However, the HOMO and LUMO (i.e. the donor and acceptor
wavefunction) cannot be too well separated in space since Osc should remain
sufficiently high to have a large absorption rate. We conclude that the donor
and acceptor orbitals typically HOMO and LUMO should be somewhat, but not
completely, spatially separated to obtain an ideal DAE candidate with large
$\lambda_{\rm max}$ and Osc.

##  IV Conclusions

Integration of quantum chemistry simulations, machine learning techniques, and
optimization calculations has been shown to be capable of accelerating the
circle of novel materials discovery. However, even with these techniques, the
task of molecular discovery for large systems is still daunting. In this work,
we have introduced a combined quantum-classical method with the cVQD algorithm
to calculate the excited states of classical Hamiltonian for efficiently
yielding the targeted molecule candidates. This methodology is validated on
searches for optimal photochromic DAE derivatives which have large
$\lambda_{\rm max}$ and Osc for use as photo switches in photopharmacology
applications. We have demonstrated that an FM machine learning model can
accurately predict $\lambda_{\rm max}$ of 4096 DAE derivatives from a learning
data set of 384 DAE derivative quantum chemistry results.

With the Ising Hamiltonian constructed from the machine learning model, a
12-qubit cVQD calculation with a reference ground state obtained from VQE was
performed on the the statevector, qasm simulator and quantum device of
$ibm\\_kawasaki$. Calculation results from the statevector simulator show that
cVQD provides a robust approach to accurately calculate the excited states of
Ising Hamiltonian. Moreover, we found excited state energies obtained from
cVQD with limited shots on qasm simulator are within an average error of 0.038
to the exact values which outperforms conventional VQD by a factor of 2. The
good performance of cVQD relies on the fact that the penalty overlap term in
the cost function is almost accurately computed on classical computer.

Due to the influence of device noise, although the energies obtained from cVQD
on $ibm\\_kawasaki$ is on average 25 nm higher than the calculation results on
statevector simulator, the highest probability bitstings are the same. Thus,
the results from $ibm\\_kawasaki$ are also useful to find the optimal DAE
derivatives. We also show that the energy difference can be improved to 3 nm
by mitigating the readout error and the decoherence error. More importantly,
quantum chemistry calculations of five DAE derivatives candidates obtained
from cVQD show how structural variation in the thiophene rings of DAE
derivatives can tune the properties of $\lambda_{\rm max}$ and Osc and provide
a path to design an ideal DAE candidate.

While calculations on much larger systems will be required to show the
advantage of quantum computing, our work of searching optimal DAE candidates
from 4096 DAE derivatives goes beyond proof of concept and paves the way for
future work utilizing quantum computing to accelerate the discovery of novel
materials. A possible next step of our study is to undertake the cVQD
computations of large chemical space optimization on quantum devices with
improved error rates, and utilizing more advanced error mitigation approaches.

###### Acknowledgements.

Q.G., M.S., T.K., O.Y., H.T. and N.Y. acknowledge support from MEXT Quantum
Leap Flagship Program Grant Number JPMXS0118067285 and JPMXS0120319794. N.Y.
acknowledge support from JSPS KAKENHI Grant Number 20H05966. All the quantum
chemistry calculations were performed on the NAYUTA Grid Cluster in Science &
Innovation Center of Mitsubishi Chemical Corporation.

##  Appendix A IBM Quantum Kawasaki

The IBM Quantum Kawasaki device used in this work is a 27-qubit Falcon,
revision 5.11, system. The entangling gate topology, based on CNOT gates, and
error rates at the time of the experiments is shown in Fig. 9. In this work,
we used qubits comprising the linear chain $[0,1,2,3,5,8,11,14,16,19,22,25]$.

Figure 9:  Error rates of IBM Quantum Kawasaki on April 7th, 2023, as obtained
from the device calibration data. The mean values of $T_{1}$ and $T_{2}$ (not
shown) are $127~{}\rm{\mu s}$ and $132~{}\rm{\mu s}$, respectively.

## References

  * Kyzer-Knapp  _et al._ [2015] E. O. Kyzer-Knapp, C. Suh, R. Gomez-Bombarelli, J. Aguilera-Iparraguirre, and A. Aspuru-Guzik, Annual Review of Materials Research 45, 195 (2015).
  * Shoichet [2004] B. K. Shoichet, Nature 432, 862 (2004).
  * Hachmann  _et al._ [2014] J. Hachmann, R. Olivares-Amaya, A. Jinich, A. L. Appleton, M. A. Blood-Forsythe, L. R. Seress, C. Román-Salgado, K. Trepte, S. Atahan-Evrenk, S. Er, S. Shrestha, R. Mondal, A. Sokolov, Z. Bao, and A. Aspuru-Guzik, Energy & Environmental Science 7, 698 (2014).
  * Saal  _et al._ [2020] J. E. Saal, A. O. Oliynyk, and B. Meredig, Annual Review of Materials Research 50, 49 (2020).
  * Ramakrishnan  _et al._ [2014] R. Ramakrishnan, P. O. Dral, M. Rupp, and O. A. von Lilienfeld, Scientific Data 1, 140022 (2014).
  * Glavatskikh  _et al._ [2019] M. Glavatskikh, J. Leguy, G. Hunault, T. Cauchy, and B. Da Mota, Journal of Cheminformatics 11, 69 (2019).
  * Polishchuk  _et al._ [2013] P. G. Polishchuk, T. I. Madzhidov, and A. Varnek, Journal of Computer-Aided Molecular Design 27, 675 (2013).
  * Kirkpatrick  _et al._ [1983] S. Kirkpatrick, C. D. J. Gelatt, and M. P. Vecchi, Science 220, 671 (1983).
  * Mitchell [1998] M. Mitchell, The MIT Press (1998).
  * Gómez-Bombarelli†  _et al._ [2018] R. Gómez-Bombarelli†, J. N. Wei, D. Duvenaud, J. M. Hernandez-Lobato, B. Sanchez-Lengeling, D. Sheberla, J. Aguilera-Iparraguirre†, T. D. Hirzel, R. P. Adams, and A. Aspuru-Guzik, ACS Cent. Sci. 4, 268 (2018).
  * Feynman [1985] R. P. Feynman, Optics News 11, 11 (1985).
  * Aspuru-Guzik  _et al._ [2005] A. Aspuru-Guzik, A. D. Dutoi, P. J. Love, and M. Head-Gordon, Science 309, 1704 (2005).
  * Farhi  _et al._ [2014] E. Farhi, J. Goldstone, and S. Gutmann, arXiv:1411.4028 [quant-ph] (2014).
  * Peruzzo  _et al._ [2014] A. Peruzzo, J. McClean, P. Shadbolt, M.-H. Yung, X.-Q. Zhou, P. J. Love, A. Aspuru-Guzik, and J. L. O’Brien, Nat. Commun. 5, 4213 (2014).
  * Kempe  _et al._ [2010] J. Kempe, O. Regev, and B. Toner, SIAM Journal on Computing 39, 3207 (2010).
  * Egger  _et al._ [2021] D. J. Egger, J. Mareček, and S. Woerner, Quantum 5, 479 (2021).
  * Gao  _et al._ [2021a] Q. Gao, H. Nakamura, T. P. Gujarati, G. O. Jones, J. E. Rice, S. P. Wood, M. Pistoia, J. M. Garcia, and N. Yamamoto, The Journal of Physical Chemistry. A 125, 1827 (2021a).
  * Rice  _et al._ [2021] J. E. Rice, T. P. Gujarati, M. Motta, T. Y. Takeshita, E. Lee, J. A. Latone, and J. M. Garcia, The Journal of Chemical Physics 154, 134115 (2021).
  * Gao  _et al._ [2021b] Q. Gao, G. O. Jones, M. Motta, M. Sugawara, H. C. Watanabe, T. Kobayashi, E. Watanabe, Y.-y. Ohnishi, H. Nakamura, and N. Yamamoto, npj Computational Materials 7, 4213 (2021b).
  * Ibe  _et al._ [2021] Y. Ibe, Y. O. Nakagawa, N. Earnest, T. Yamamoto, K. Mitarai, Q. Gao, and T. Kobayashi, Physical Review Research 4, 013173 (2021).
  * Gao  _et al._ [2023] Q. Gao, G. O. Jones, M. Sugawara, T. Kobayashi, H. Yamashita, H. Kawaguchi, S. Tanaka, and N. Yamamoto, Intell. Comput. 2, 0037 (2023).
  * Higgott  _et al._ [2019] O. Higgott, D. Wang, and S. Brierley, Quantum 3, 156 (2019).
  * Velema  _et al._ [2014] W. A. Velema, W. Szymanski, and B. L. Feringa, JACS 136, 2178 (2014).
  * Edwards and Aronson [2000] I. R. Edwards and J. K. Aronson, Lancet 356, 1255 (2000).
  * Malhotra and Perry [2003] V. Malhotra and M. C. Perry, Cancer Biol. Ther. 2, S2 (2003).
  * Carlet  _et al._ [2011] J. Carlet, P. Collignon, D. Goldmann, H. Goossens, I. C. Gyssens, S. Harbarth, V. Jarlier, S. B. Levy, B. N’Doye, R. Pittet, D. Richtmann, W. H. Seto, J. W. M. van der Meer, and A. Voss, Lancet 378, 369 (2011).
  * Martinez [2008] J. L. Martinez, Science 321, 365 (2008).
  * Kudernac  _et al._ [2013] T. Kudernac, T. Kobayashi, A. Uyama, K. Uchida, S. Nakamura, and B. L. Feringa, The Journal of Physical Chemistry. A 117, 8222 (2013).
  * Irie  _et al._ [2014] M. Irie, T. Fukaminato, K. Matsuda, and S. Kobatake, Chemical Reviews 114, 12174 (2014).
  * Landrum [2010] G. Landrum, https://www.rdkit.org/ (2010).
  * Rappe  _et al._ [1992] A. K. Rappe, C. J. Casewit, K. S. Colwell, W. A. Goddard III, and W. M. Skiff, J. Am. Chem. Soc. 114, 10024 (1992).
  * Chai and Head-Gordon [2008] J.-D. Chai and M. Head-Gordon, Physical Chemistry Chemical Physics: PCCP 10, 6615 (2008).
  * Yanai  _et al._ [2004] T. Yanai, D. P. Tew, and N. C. Handy, Chem. Phys. Lett. 393, 51 (2004).
  * Frisch  _et al._ [2009] M. J. Frisch, G. W. Trucks, H. B. Schlegel, G. E. Scuseria, M. A. Robb, J. R. Cheeseman, G. Scalmani, V. Barone, H. Nakatsuji, X. Li, M. Caricato, A. Marenich, J. Bloino, B. G. Janesko, R. Gomperts, B. Mennucci, H. P. Hratchian, J. V. Ortiz, A. F. Izmaylov, J. L. Sonnenberg, D. Williams-Young, F. Ding, F. Lipparini, F. Egidi, J. Goings, B. Peng, A. Petrone, T. Henderson, D. Ranasinghe, V. G. Zakrzewski, J. Gao, N. Rega, G. Zheng, W. Liang, M. Hada, M. Ehara, K. Toyota, R. Fukuda, J. Hasegawa, M. Ishida, T. Nakajima, Y. Honda, O. Kitao, H. Nakai, T. Vreven, K. Throssell, J. A. M. Jr., J. E. Peralta, F. Ogliaro, M. Bearpark, J. J. Heyd, E. Brothers, K. N. Kudin, V. N. Staroverov, T. Keith, R. Kobayashi, J. Normand, K. Raghavachari, A. Rendell, J. C. Burant, S. S. Iyengar, J. Tomasi, M. Cossi, J. M. Millam, M. Klene, C. Adamo, R. Cammi, J. W. Ochterski, R. L. Martin, K. Morokuma, O. Farkas, J. B. Foresman, and D. J. Fox, Gaussian 09, Revision B.01 (2009).
  * Kochenberger  _et al._ [2014] G. Kochenberger, J.-K. Hao, F. Glover, M. Lewis, Z. Lü, H. Wang, and Y. Wang, J. Comb. Optim. 28, 58 (2014).
  * Rendle [2010] S. Rendle, in  _Proceedings of the 2010 IEEE International Conference on Data Mining_ , ICDM ’10 (IEEE Computer Society, USA, 2010) pp. 995–1000.
  * Rendle [2012] S. Rendle, ACM Trans. Intell. Syst. Technol. 3, 57:1 (2012).
  * Rendle and Schmidt-Thieme [2010] S. Rendle and L. Schmidt-Thieme, in  _Proceedings of the third ACM international conference on Web search and data mining - WSDM ’10_ (ACM Press, New York, New York, USA, 2010) pp. 81–90.
  * Ising [1925] E. Ising, Z. Physik 31, 253 (1925).
  * Tanaka  _et al._ [2017] S. Tanaka, R. Tamura, and B. K. Chakrabarti,  _Quantum Spin Glasses, Annealing and Computation_ (Cambridge University Press, 2017).
  * Tanahashi  _et al._ [2019] K. Tanahashi, S. Takayanagi, T. Motohashi, and S. Tanaka, Journal of the Physical Society of Japan 88, 061010 (2019), https://doi.org/10.7566/JPSJ.88.061010 .
  * Aleksandrowicz  _et al._ [2019] G. Aleksandrowicz, T. Alexander, P. Barkoutsos, L. Bello, Y. Ben-Haim, D. Bucher, F. J. Cabrera-Hernández, J. Carballo-Franquis, A. Chen, C.-F. Chen, J. M. Chow, A. D. Córcoles-Gonzales, A. J. Cross, A. Cross, J. Cruz-Benito, C. Culver, S. D. L. P. González, E. D. L. Torre, D. Ding, E. Dumitrescu, I. Duran, P. Eendebak, M. Everitt, I. F. Sertage, A. Frisch, A. Fuhrer, J. Gambetta, B. G. Gago, J. Gomez-Mosquera, D. Greenberg, I. Hamamura, V. Havlicek, J. Hellmers, Łukasz Herok, H. Horii, S. Hu, T. Imamichi, T. Itoko, A. Javadi-Abhari, N. Kanazawa, A. Karazeev, K. Krsulich, P. Liu, Y. Luh, Y. Maeng, M. Marques, F. J. Martín-Fernández, D. T. McClure, D. McKay, S. Meesala, A. Mezzacapo, N. Moll, D. M. Rodríguez, G. Nannicini, P. Nation, P. Ollitrault, L. J. O’Riordan, H. Paik, J. Pérez, A. Phan, M. Pistoia, V. Prutyanov, M. Reuter, J. Rice, A. R. Davila, R. H. P. Rudy, M. Ryu, N. Sathaye, C. Schnabel, E. Schoute, K. Setia, Y. Shi, A. Silva, Y. Siraichi, S. Sivarajah, J. A. Smolin, M. Soeken, H. Takahashi, I. Tavernelli, C. Taylor, P. Taylour, K. Trabing, M. Treinish, W. Turner, D. Vogt-Lee, C. Vuillot, J. A. Wildstrom, J. Wilson, E. Winston, C. Wood, S. Wood, S. Wörner, I. Y. Akhalwaya, and C. Zoufal, Qiskit: An Open-source Framework for Quantum Computing (2019).
  * Note [1] Note that, although the controlled-z gates in Fig. 3(a) can be executed in parallel, we choose to execute them in serial, with idle times populated with dynamical decoupling sequences as it yields higher fidelity results. This is likely due to the always-on-coupling of cross-resonance based devices, where the run-in-parallel gate operation can have larger error rates compared to isolated execution.
  * Nation  _et al._ [2021] P. D. Nation, H. Kang, N. Sundaresan, and J. M. Gambetta, PRX QUANTUM 2, 040326 (2021).
  * Souza  _et al._ [2012] A. M. Souza, G. A. Álvarez, and D. Suter, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 370, 4748 (2012).
  * Gill and Meiboom [1958] D. Gill and S. Meiboom, Review of scientific instruments 29, 688 (1958).


11institutetext: Indian Institute of Technology
11email<EMAIL_ADDRESS><EMAIL_ADDRESS>
# Search-time Efficient Device Constraints-Aware Neural Architecture Search

Anonymous Author  Oshin Dutta  Tanu Kanvar  Sumeet Agarwal

###### Abstract

Edge computing aims to enable edge devices, such as IoT devices, to process
data locally instead of relying on the cloud. However, deep learning
techniques like computer vision and natural language processing can be
computationally expensive and memory-intensive. Creating manual architectures
specialized for each device is infeasible due to their varying memory and
computational constraints. To address these concerns, we automate the
construction of task-specific deep learning architectures optimized for device
constraints through Neural Architecture Search (NAS). We present DCA-NAS, a
principled method of fast neural network architecture search that incorporates
edge-device constraints such as model size and floating-point operations. It
incorporates weight sharing and channel bottleneck techniques to speed up the
search time. Based on our experiments, we see that DCA-NAS outperforms manual
architectures for similar sized models and is comparable to popular mobile
architectures on various image classification datasets like CIFAR-10,
CIFAR-100, and Imagenet-1k. Experiments with search spaces—DARTS and NAS-
Bench-201 show the generalization capabilities of DCA-NAS. On further
evaluating our approach on Hardware-NAS-Bench, device-specific architectures
with low inference latency and state-of-the-art performance were discovered.

###### Keywords:

Neural Architecture Search DARTS Meta-Learning Edge Inference Constrained
Optimization

##  1 Introduction

In recent years, there has been significant progress in developing Deep Neural
Network (DNN) architectures [33, 47, 34] for edge and mobile devices.However,
designing DNN architectures for specific hardware constraints and tasks is a
time-consuming and computationally expensive process [3]. To address this,
Neural Architecture Search (NAS) [2, 32, 49] has become popular as it
discovers optimal architectures given a task and network operations. Despite
its success, traditional NAS techniques cannot guarantee optimal architecture
for specific devices with hardware constraints such as storage memory and
maximum supported FLOPs. To address this concern, researchers have developed
hardware-aware algorithms [36, 4] that find optimal device architectures with
low resource training overhead and search time. These methods often use
inference latency [4], FLOPs [36] or a combination of hardware metrics [36] as
constraints scaled by a tunable factor. However, the time to tune the scaling
factor is often not considered within the NAS search time and can be ten times
the reported search time. To address these issues, we propose the Device
Constraints-Aware NAS (DCA-NAS), a principled differentiable NAS method that
introduces total allowable model size or floating-point operations (FLOPs) as
constraints within the optimization problem, with minimal hyper-parameter
tuning. Unlike inference latency which is task dependent, FLOPs and memory are
specified with a given hardware and thus are appropriate for our generic
method. The approach is adaptable to other hardware metrics such as energy
consumption or inference latency using additional metric-measuring functions.
The paper make the following significant contributions:

  * •

It introduces a fast method that uses weight sharing among operations in the
search space and channel bottleneck, along with a differentiable resource
constraint, for continuous exploration of the search space.

  * •

A training pipeline that allows a user to input device memory or FLOPs and
search for optimal architecture with minimal hyper-parameter tuning.

  * •

Our extensive experimentation on vision datasets- CIFAR-10, CIFAR-100,
TinyImagenet, Imagenet-1k and inference-latency comparisons of trained models
on Hardware-NAS-bench demonstrate the efficiency of our method. The
generalization of our method to different search spaces is shown with
experiments on DARTS and NAS-Bench.

Figure 1: DCA-NAS framework:Weight sharing in the search space and Derived
cells lowers the search time from other DNAS. Target device constraint is used
to query search constraint from look-up graph for constrained optimization.

##  2 Related Work

Neural Architecture Search Popular approaches [12, 22, 1] designed
architectures for high performance on specific tasks or datasets with the
traditional deep learning perspective that bigger is better, resulting in
computationally and memory-intensive inference on edge devices. Network
pruning [13], channels removal [26, 34] and weights/activations quantization
[8, 50] can compress architectures, but require pre-training, hyperparameter
tuning, and often lack transferability.Neural Architecture Search (NAS)
methods such as Reinforcement Learning [30, 4], Evolutionary Learning [11, 21]
and Differentiable Neural Architecture Search (DNAS) [25, 43] can
automatically search for architectures without user intervention, and can
transfer across similar tasks. DNAS with surrogate metrics [42, 48] have also
been used to explore the architecture search space. However, architectures
found by DNAS methods are not optimized for deployment on edge devices and
smaller models obtained by reducing layers or channels are often sub-optimal.
Hardware-aware Neural Architecture search Certain NAS methods optimize [4, 40,
3, 19] for constraints such as latency, inference speed [41], FLOPS [36, 37],
memory usage [24]. Some use a separate DNN to predict constraint metrics and
evolutionary search to obtain hardware-aware optimal models [36, 3], while
others consider real-time latencies of edge devices or provide specific
architectures for specific devices [27, 7]. However, these methods require
significant search time and tuning of scaling factors controlling the trade-
off between the performance and the constraint, and do not always account for
optimal architectures. In contrast, we use a differentiable hardware-aware
objective function with generic hardware metrics, and do not require a tunable
scaling factor. Certain methods [3, 29, 9] train a supernet first and then
search for a smaller architecture, but this is only efficient when there are
more than fifteen different edge devices with different limitations or
deployment scenarios [3] as training the supernet takes huge resources-32
V100s taking about 1,200 GPU hours. Search stage followed by evaluation, as
done in our approach is more efficient when the different number of possible
edge devices is less than fifteen.

##  3 DCA-NAS: Device Constraints Aware Fast Neural Architecture Search

We present the preliminary gradient-based NAS objective function in section
3.1 and then formulate the problem of incorporating the hardware-awareness in
NAS as a constrained optimization problem in section 3.2 followed by
techniques to reduce the search time in section 3.3. The framework of our
approach is illustrated in Figure 1.

###  3.1 Gradient-based NAS Objective Function

Popular DNAS techniques [25, 46] have two stages, the search phase and the
evaluation phase. During the search phase, given a task or a dataset the
techniques search for a network of cells, which are directed acyclic graphs
with $N$ nodes. The edges of the graph are network layers, whose operations
are to be selected from a pre-defined set $\mathcal{O}$ containing operations
such as 3x3 separable convolution and identity operations with trainable
weights $w_{o}$. The search is made differentiable by making the choice of a
particular operation to be a softmax of architecture weights $\alpha$ of all
operations. Thus, the intermediate output $z_{j}$ at node $j$ is given by,

$\small
z_{j}=\sum_{o\in\mathcal{O}}\frac{\exp\left\\{\alpha_{o}^{i,j}\right\\}}{\sum_{o^{\prime}\in\mathcal{O}}\exp\left\\{\alpha_{o^{\prime}}^{i,j}\right\\}}\cdot
o\left(w_{o}^{i,j},\mathbf{z}_{i}\right)$ (1)

###  3.2 DCA-NAS formulation

Previous DNAS approaches [25, 45, 46] did not focus on searching architectures
specifically for inference on resource-constrained devices. In contrast, we
formulate the DNAS objective function as a constrained optimization problem by
incorporating device resource constraints (memory or FLOPs) in the search
objective function. The constrained bi-level optimization problem is written
as,

$\small\begin{array}[]{ll}\min_{\alpha}&\mathcal{L}_{\text{val
}}\left(w^{*}(\alpha),\alpha\right)\\\ \text{ s.t.
}&w^{*}(\alpha)=\operatorname{argmin}_{w}\mathcal{L}_{\text{train
}}(w,\alpha)\\\ \text{ s.t. }&k_{s}(\alpha)\leq K_{d}\end{array}$ (2)

where training dataset is split into $train$ and $val$ to optimize $w$ and
$\alpha$ simultaneously in each iteration subject to the constraint that the
architecture’s number of parameters or FLOPs $k_{s}$ must be less than or
equal to the device resource constraint $K_{d}$. The following equation
calculates the architecture’s number of parameters or FLOPs during search
given the number of cells$c_{n}$ . Our method can also be adapted to use other
metrics such as latency and energy consumption with additional metric
measuring functions.

$\vspace{-3mm}\small k_{s}(\alpha)=c_{n}\sum_{(i,j)\in
N}\sum_{o\in\mathcal{O}}\frac{\exp\\{{\alpha_{o}^{i,j}\\}}*b\left(o\right)}{\sum_{o^{\prime}\in\mathcal{O}}\exp\\{{\alpha_{o^{\prime}}^{i,j}}\\}}$
(3)

####  3.2.1 Tackling the difference in search and evaluation networks

The size of the architecture in the search phase $k_{s}$ is different from the
architecture size in evaluation phase due to the softmax weighting factor in
equation 3 (demonstration can be found in the appendix). To address this, we
introduce a tighter bound on the search constraint $K_{d^{\prime}}$, which is
less than the device resource constraint $K_{d}$. A lookup graph (LUG) needs
to be made for each dataset by varying $K_{d^{\prime}}$ within appropriate
bounds and running the algorithm until convergence each time to obtain the
corresponding device resource constraint $K_{d}$. The computation time of the
LUG can be reduced by running the searches in parallel. Thus, on incorporating
the tighter constraint by looking-up the graph for the given device resource
constraint $K_{d}$ along with the trainable Lagrange multiplier $\lambda$ in
Equation 2, the objective function is re-written as,

$\small\begin{array}[]{c}\widetilde{\mathcal{L}}=\mathcal{L}_{\text{val
}}\left(w^{*}(\alpha),\alpha\right)+\lambda(k_{s}(\alpha)-LUG(K_{d}))\\\
\text{ s.t. }w^{*}(\alpha)=\operatorname{argmin}_{w}\mathcal{L}_{\text{train
}}(w,\alpha)\end{array}$ (4)

Table 1: Performance comparison of architectures evaluated on visual datasets-
CIFAR-10 and TinyImagenet. ’(CIFAR-10)’ indicates search with CIFAR-10. ’X M’
in ’DCA-NAS-X M’ denotes the input memory constraint. RCAS- Resource
Constrained Architecture Search

Dataset | Search | Method | Accuracy | Parameters | GPU
---|---|---|---|---|---
| Strategy |  | (%) | (Million) | Hours
CIFAR-10 | manual | PyramidNet-110 (2017) [12] | 95.74 | 3.8 | -
| manual | VGG-16 pruned (2017) [16] | 93.4 | 5.4 | -
| evolution | Evolution + Cutout (2019) [39] | 96.43 | 5.8 | 12
| random | NAO Random-WS (2019) [31] | 96.08 | 3.9 | 7.2
| gradient | ENAS + micro + Cutout (2018) [30] | 96.46 | 4.6 | 12
| gradient | DARTS + Cutout (2nd) (2018) [25] | 97.24$\pm$0.09 | 3.3 | 24
| gradient | SNAS + Cutout (2018) [43] | 97.15 | 2.8 | 36
| gradient | PC-DARTS (2019) [45] | 97.43$\pm$ 0.07 | 3.6 | 2.4
| gradient | SGAS (2020) [23] | 97.34 | 3.7 | 6
| gradient | DrNAS (2020) [6] | 97.46 ± 0.03 | 4.0 | 9.6
| gradient | DARTS+PT (2021) [38] | 97.39 ± 0.08 | 3.0 | 19.2
| gradient | Shapley-NAS (2022) [42] | 97.53 ± 0.04 | 3.4 | 7.2
| RCAS | DCA-NAS- 3.5 M (CIFAR-10) | 97.2$\pm$0.09 | 3.4 | 1.37
Tiny ImageNet | manual | SqueezeNet (2016) [18] | 54.40 | - | -
| manual | PreActResNet18 (2020) [22] | 63.48 | - | -
| manual | ResNet18 (2016) [15] | 58.4 | 6.4 | -
| manual | DenseNet (2020) [1] | 62.73 | 11.8 | -
| gradient | DARTS+ Cutout (2018) [25] | 62.15$\pm$0.15 | 7.3 | 219
| RCAS | DCA-NAS- 3.5 M | 61.34$\pm$0.09 | 3.5 | 12.5
| RCAS | DCA-NAS- 3.5 M (CIFAR-10) | 61.4$\pm$0.15 | 3.4 | 1.37

Figure 2: Plots show that DCA-NAS method discovers models with fewer
parameters than other NAS methods and manual architectures without sacrificing
prediction performance to a large extent.

###  3.3 Techniques to reduce search time

Channel Bottleneck We use convolutional layers of 1x1 kernel to reduce the
depth of output channels of operations in the search space to save computation
time and memory overhead.
Derived Cell and Weight sharing. During architecture search, only one cell
with trainable $\alpha$ is used to optimize architecture parameters. The
target network for inference is built by stacking cells with architectures
derived from highly weighted operations. This can be done during search by
deriving the other cell architectures from the first at each iteration [46].
The arrangement of the cells for search is given in the appendix. This derived
cell saves computation and memory overhead. A weight sharing strategy [46]
among same operations with the same originating node $i$ to all nodes $i<j<N$
has been applied within a cell. This is motivated by the observation that non-
parametric operations operating on the representation of a node produce the
same feature map irrespective of the output node and thereby extended to
parametric operations. Thus, Equation 1 may be re-written to the following,

$\small
z_{j}=\sum_{o\in\mathcal{O}}\frac{\exp\left\\{\alpha_{o}^{i,j}\right\\}}{\sum_{o^{\prime}\in\mathcal{O}}\exp\left\\{\alpha_{o^{\prime}}^{i,j}\right\\}}\cdot
o\left(w_{o}^{i},\mathbf{z}_{i}\right)$ (5)

##  4 Experimental Results

Our approach is evaluated on two search spaces- DARTS and NAS-Bench with
vision datasets- CIFAR10, TinyImagenet, Imagenet-16-20 and Imagenet-1k. The
details of the search space and implementation is given in the appendix

###  4.1 Results on DARTS search space

Table 2: Performance and comparison of architectures evaluated on Imagenet-1k.
The label ”(Imagenet)” indicates that the architecture has been searched and
evaluated on Imagenet-1k.; else it is searched on CIFAR-10. ’X M’ in ’DCA-
NAS-X M’ denotes the input memory constraint

Method | Test Error (%) | Parameters | FLOPS | Search Cost | Search
---|---|---|---|---|---
| top-1 | top-5 | (Mil) | (Mil) | (GPU days) | Strategy
Inception-v1 (2015) [35] | 30.2 | 10.1 | 6.6 | 1448 | - | manual
MobileNet_V1 (2017) [17] | 29.4 | 10.5 | 4.2 | 569 | - | manual
MobileNet_V2 (2018) [33] | 72.0 | 91.0 | 3.4 | 300 | - | manual
ShuffleNet 2× (v2) (2018) [28] | 25.1 | - | 5 | 591 | - | manual
MnasNet-92 (2020) [14] | 25.2 | 8.0 | 4.4 | 388 | - | RL
AmoebaNet-C (2019) [31] | 24.3 | 7.6 | 6.4 | 570 | 3150 | evolution
DARTS+Cutout (2018) [25] | 26.7 | 8.7 | 4.7 | 574 | 1.0 | gradient
SNAS (2018) [43] | 27.3 | 9.2 | 4.3 | 522 | 1.5 | gradient
GDAS (2019) [10] | 26.0 | 8.5 | 5.3 | 545 | 0.3 | gradient
BayesNAS (2019) [49] | 26.5 | 8.9 | 3.9 | - | 0.2 | gradient
P-DARTS (2018) [30] | 24.4 | 7.4 | 4.9 | 557 | 0.3 | gradient
SGAS (Cri 1. best) (2020) [23] | 24.2 | 7.2 | 5.3 | 585 | 0.25 | gradient
SDARTS-ADV (2020) [5] | 25.2 | 7.8 | 6.1 | - | 0.4 | gradient
Shapley-NAS (2022) [42] | 24.3 | - | 5.1 | 566 | 0.3 | gradient
RC-DARTS (2019) [20] | 25.1 | 7.8 | 4.9 | 590 | 1 | RCAS
DCA-NAS | 25.1 | 8.1 | 5.1 | 578 | 0.06 | RCAS
ProxylessNAS (GPU) (2019) [4](Imagenet) | 24.9 | 7.5 | 7.1 | 465 | 8.3 | gradient
PC-DARTS (2019) [45] (Imagenet) | 24.2 | 7.3 | 5.3 | 597 | 3.8 | gradient
DrNAS (2020) [6] (Imagenet) | 24.2 | 7.3 | 5.2 | 644 | 3.9 | gradient
DARTS+PT (2021) [38] (Imagenet) | 25.5 | - | 4.7 | 538 | 3.4 | gradient
Shapley-NAS (2022) [42] (Imagenet) | 23.9 | - | 5.4 | 582 | 4.2 | gradient
RCNet-B (2019) [44] (ImageNet) | 25.3 | 8.0 | 4.7 | 471 | 9 | RCAS
DCA-NAS- 5.5 M(Imagenet) | 24.4 | 7.2 | 5.3 | 597 | 1.9 | RCAS

####  4.1.1 Transferability- learning of coarse features during search.

We transfer the architecture searched on CIFAR-10 to train and evaluate the
model weights on TinyImagenet in Table 1 and ImageNet-1k in Table 2. This
transferred model yields higher performance than manually designed
architectures [33, 28] for the target dataset. It is observed that performance
of the transferred model is comparable to the architecture searched on the
target dataset itself which can be attributed to the architecture learning
coarse features than objects during search.

####  4.1.2 Performance versus Device-Constraints trade-off

DCA-NAS discovers 2 to 4% better-performing architectures than manual designs
with a memory constraint of 3.5 million parameters on CIFAR-10 and similar
performance on TinyImagenet as in Table 1. On Imagenet-1k, DCA-NAS yields
models with similar performance to other NAS methods [42, 6, 45] with a
constraint of 5.5 million parameters (taken to yield similar sized models as
other NAS methods) as in Table 2. We vary the input device resource constraint
and plot the performance of the searched models against the number of
parameters in Figure 2. As observed, DCA-NAS searched models can yield 15x
lower sized models than manual architectures like PyramidNet-272 [12] with at
most 1% reduction in accuracy on CIFAR-10. On TinyImagenet, DCA-NAS yields
models similar in performance but 6x smaller in size than the manual Resnet
variant. In comparison to ProxylessNAS [4] for Imagenet-1k, DCA-NAS yields 32%
smaller model in terms of model parameters for similar accuracy. In comparison
to DNAS methods [25, 45] for each of the three datasets, we observe that the
performance of the DCA-NAS searched models is retained to a certain extent as
resources are further limited after which the model performance degrades. DCA-
NAS model of similar size has the advantage of better performance (by 1%) and
being automatically searched over MobileNet-v2 [33], a manually designed
network on Imagenet-1k.

####  4.1.3 Search time comparison

For evaluation on TinyImagenet in Table 1, the architecture searched on
CIFAR-10 with DCA-NAS yields model in the lowest search time which indicates
the search-time efficiency of the transferability property. Our method
requires about 4x lower search cost than SGAS [23] which performs the best
among the other transferred architectures and 16x lower search time than the
other resource-constrained approach [20] for similar performance as seen in
Table 2. Moreover, ProxylessNAS [4] takes about 4x more search time than DCA-
NAS whereas PC-DARTS takes about 2x more search time with no capability to
constraint model size.

Figure 3: Plots show DCA-NAS searched models with similar performance but
lower inference latency (on two devices- Pixel 3 and Raspberry Pi 4) to
previous SOTA NAS method- PC-DARTS when evaluated on NAS-Bench dataset.

###  4.2 Results on NAS-Bench-201 search space

####  4.2.1 Performance and Latency comparisons on different devices

Our method reports the mean by averaging over five runs with different random
seed. Figure 3 compares the performance of models searched with DCA-NAS and
PC-DARTS by varying the latency constraints. It shows that unlike PC-DARTS,
DCA-NAS can search for more efficient models which have lower inference
latency for similar test accuracy. Moreover, we observe that models with
similar performance have lower latency when tested on Pixel 3 than on
Raspberry Pi 4 due to a faster RAM in Pixel 3. DCA-NAS takes the lowest search
time among all the NAS methods due to the addition of search-time-efficient
techniques while being at-par in terms of performance across all datasets.

(a)

(b)

Figure 4: (a) Ablation study with CIFAR-10 dataset- Each component added to
DARTS leads to the reduction in the search cost of DCA-NAS while performance
is retained. WS- Weight Sharing, CB- Channel Bottleneck, DC- Derived Cell, RC-
Resource Constraint, BS- Batch Size (b) Shows stability of performance of DCA-
NAS searched models for runs with varying seeds on CIFAR-10 dataset.

##  5 Ablation Study

Effectiveness of various algorithmic augmentations for faster search: We
analyze the effectiveness of algorithmic augmentations mentioned preciously
3.3 to reduce search cost in our study. We sequentially add weight sharing,
channel bottleneck, and derived cells to the baseline DARTS [25] method and
measure search time and accuracy. Weight sharing, channel bottleneck, and
derived cells was observed to significantly reduce search memory overhead,
enabling us to use larger batch sizes and reducing overall search cost as seen
in Figure 4(a). Adding the resource-constraint in the final DCA-NAS method
negligibly increases search cost while maintaining performance.
Stability of the approach: We test stability by running the search algorithm
independently five times with different initial seeds and the same constraints
and hyperparameters. The architectures found during each run have similar
performance when re-trained and evaluated as shown in Fig. 4(b). Smaller
models have lower performance due to restrictions in model complexity compared
to larger models.

##  6 Conclusion

We present DCA-NAS, a device constraints-aware neural architecture search
framework which discovers architectures optimized to the memory and
computational constraints of an edge device in a time-efficient manner. It
does so by incorporating a constraint in terms of the number of parameters or
floating point operations (FLOPs) in the objective function with the help of a
Lagrange multiplier. DCA-NAS in essence searches for a Pareto optimal solution
given the edge device memory or FLOPs constraint. Moreover, it enables
architecture search with search cost 4 to 17 times lower than the previous
state-of-the-art Hardware-aware NAS approaches. DCA-NAS can discover models
with size about 10 to 15 times lower than manually designed architectures for
similar performance. In comparison to DARTS and its other NAS variants, DCA-
NAS can discover models upto 3x smaller in size with similar performance. This
hardware-aware approach can be generalized to any future updates to
differential neural architecture search and possibly to training-free methods
of NAS with some adaptation.

## Acknowledgement

We thank the anonymous reviewers; Profs. Surendra Prasad and Brejesh Lall of
IIT Delhi; and colleagues at Cadence India for their valuable feedback and
inputs. This research is supported by funding from Cadence India; the first
author is also supported by a fellowship from the Ministry of Education,
India.

## References

  * [1] Abai, Z., Rajmalwar, N.: Densenet models for tiny imagenet classification (2020)
  * [2] Baker, B., Gupta, O., Raskar, R., Naik, N.: Accelerating neural architecture search using performance prediction (2017)
  * [3] Cai, H., Gan, C., Wang, T., Zhang, Z., Han, S.: Once-for-All: Train One Network and Specialize it for Efficient Deployment (Apr 2020), http://arxiv.org/abs/1908.09791, arXiv:1908.09791 [cs, stat]
  * [4] Cai, H., Zhu, L., Han, S.: Proxylessnas: Direct neural architecture search on target task and hardware (2019)
  * [5] Chen, X., Hsieh, C.J.: Stabilizing differentiable architecture search via perturbation-based regularization. In: International conference on machine learning. pp. 1554–1565. PMLR (2020)
  * [6] Chen, X., Wang, R., Cheng, M., Tang, X., Hsieh, C.J.: Drnas: Dirichlet neural architecture search. arXiv preprint arXiv:2006.10355 (2020)
  * [7] Chu, G., Arikan, O., Bender, G., Wang, W., Brighton, A., Kindermans, P.J., Liu, H., Akin, B., Gupta, S., Howard, A.: Discovering multi-hardware mobile models via architecture search. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3022–3031 (2021)
  * [8] Courbariaux, M., Bengio, Y., David, J.P.: Binaryconnect: Training deep neural networks with binary weights during propagations (2016)
  * [9] Ding, Y., Wu, Y., Huang, C., Tang, S., Wu, F., Yang, Y., Zhu, W., Zhuang, Y.: Nap: Neural architecture search with pruning. Neurocomputing 477, 85–95 (2022)
  * [10] Dong, X., Yang, Y.: Searching for a robust neural architecture in four gpu hours. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1761–1770 (2019)
  * [11] Elsken, T., Metzen, J.H., Hutter, F.: Efficient multi-objective neural architecture search via lamarckian evolution. arXiv: Machine Learning (2019)
  * [12] Han, D., Kim, J., Kim, J.: Deep pyramidal residual networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 5927–5935 (2017)
  * [13] Han, S., Mao, H., Dally, W.J.: Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding (2016)
  * [14] He, C., Ye, H., Shen, L., Zhang, T.: Milenas: Efficient neural architecture search via mixed-level reformulation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11993–12002 (2020)
  * [15] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
  * [16] He, Y., Zhang, X., Sun, J.: Channel pruning for accelerating very deep neural networks. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 1389–1397 (2017)
  * [17] Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., Adam, H.: Mobilenets: Efficient convolutional neural networks for mobile vision applications (2017)
  * [18] Iandola, F.N., Han, S., Moskewicz, M.W., Ashraf, K., Dally, W.J., Keutzer, K.: Squeezenet: Alexnet-level accuracy with 50x fewer parameters and ¡0.5mb model size (2016)
  * [19] Jiang, Q., Zhang, X., Chen, D., Do, M.N., Yeh, R.A.: EH-DNAS: End-to-End Hardware-aware Differentiable Neural Architecture Search. arXiv:2111.12299 [cs] (Nov 2021), http://arxiv.org/abs/2111.12299, arXiv: 2111.12299
  * [20] Jin, X., Wang, J., Slocum, J., Yang, M.H., Dai, S., Yan, S., Feng, J.: Rc-darts: Resource constrained differentiable architecture search. arXiv preprint arXiv:1912.12814 (2019)
  * [21] Jozefowicz, R., Zaremba, W., Sutskever, I.: An empirical exploration of recurrent network architectures. In: Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37. p. 2342–2350. ICML’15, JMLR.org (2015)
  * [22] Kim, J.H., Choo, W., Song, H.O.: Puzzle mix: Exploiting saliency and local statistics for optimal mixup (2020)
  * [23] Li, G., Qian, G., Delgadillo, I.C., Müller, M., Thabet, A., Ghanem, B.: Sgas: Sequential greedy architecture search (2020)
  * [24] Lin, J., Chen, W.M., Lin, Y., Gan, C., Han, S., et al.: Mcunet: Tiny deep learning on iot devices. Advances in Neural Information Processing Systems 33, 11711–11722 (2020)
  * [25] Liu, H., Simonyan, K., Yang, Y.: Darts: Differentiable architecture search. arXiv preprint arXiv:1806.09055 (2018)
  * [26] Liu, Z., Li, J., Shen, Z., Huang, G., Yan, S., Zhang, C.: Learning efficient convolutional networks through network slimming (2017)
  * [27] Lyu, B., Yuan, H., Lu, L., Zhang, Y.: Resource-Constrained Neural Architecture Search on Edge Devices. IEEE Transactions on Network Science and Engineering 9(1), 134–142 (Jan 2022). https://doi.org/10.1109/TNSE.2021.3054583, conference Name: IEEE Transactions on Network Science and Engineering
  * [28] Ma, N., Zhang, X., Zheng, H.T., Sun, J.: Shufflenet v2: Practical guidelines for efficient cnn architecture design. In: Proceedings of the European conference on computer vision (ECCV). pp. 116–131 (2018)
  * [29] Muñoz, J.P., Lyalyushkin, N., Akhauri, Y., Senina, A., Kozlov, A., Jain, N.: Enabling NAS with Automated Super-Network Generation (Dec 2021), http://arxiv.org/abs/2112.10878, arXiv:2112.10878 [cs]
  * [30] Pham, H., Guan, M.Y., Zoph, B., Le, Q.V., Dean, J.: Efficient neural architecture search via parameter sharing. In: ICML (2018)
  * [31] Real, E., Aggarwal, A., Huang, Y., Le, Q.V.: Regularized evolution for image classifier architecture search (2019)
  * [32] Real, E., Moore, S., Selle, A., Saxena, S., Suematsu, Y.L., Tan, J., Le, Q.V., Kurakin, A.: Large-scale evolution of image classifiers. In: Proceedings of the 34th International Conference on Machine Learning - Volume 70. p. 2902–2911. ICML’17, JMLR.org (2017)
  * [33] Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.C.: Mobilenetv2: Inverted residuals and linear bottlenecks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4510–4520 (2018)
  * [34] Srivastava, A., Dutta, O., Gupta, J., Agarwal, S., AP, P.: A variational information bottleneck based method to compress sequential networks for human action recognition. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 2745–2754 (2021)
  * [35] Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1–9 (2015)
  * [36] Tan, M., Chen, B., Pang, R., Vasudevan, V., Sandler, M., Howard, A., Le QV, M.: platform-aware neural architecture search for mobile. 2019 ieee. In: CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 2815–2823 (2019)
  * [37] Tan, M., Le, Q.: Efficientnet: Rethinking model scaling for convolutional neural networks. In: International conference on machine learning. pp. 6105–6114. PMLR (2019)
  * [38] Wang, R., Cheng, M., Chen, X., Tang, X., Hsieh, C.J.: Rethinking architecture selection in differentiable nas. arXiv preprint arXiv:2108.04392 (2021)
  * [39] Wistuba, M.: Deep learning architecture search by neuro-cell-based evolution with function-preserving mutations. In: Berlingerio, M., Bonchi, F., Gärtner, T., Hurley, N., Ifrim, G. (eds.) Machine Learning and Knowledge Discovery in Databases. pp. 243–258. Springer International Publishing, Cham (2019)
  * [40] Wu, B., Dai, X., Zhang, P., Wang, Y., Sun, F., Wu, Y., Tian, Y., Vajda, P., Jia, Y., Keutzer, K.: Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10734–10742 (2019)
  * [41] Wu, Y., Gong, Y., Zhao, P., Li, Y., Zhan, Z., Niu, W., Tang, H., Qin, M., Ren, B., Wang, Y.: Compiler-Aware Neural Architecture Search for On-Mobile Real-time Super-Resolution (Jul 2022), http://arxiv.org/abs/2207.12577, arXiv:2207.12577 [cs, eess]
  * [42] Xiao, H., Wang, Z., Zhu, Z., Zhou, J., Lu, J.: Shapley-NAS: Discovering Operation Contribution for Neural Architecture Search (Jun 2022), http://arxiv.org/abs/2206.09811, arXiv:2206.09811 [cs]
  * [43] Xie, S., Zheng, H., Liu, C., Lin, L.: Snas: stochastic neural architecture search. In: International Conference on Learning Representations (2018)
  * [44] Xiong, Y., Mehta, R., Singh, V.: Resource constrained neural network architecture search: Will a submodularity assumption help? In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 1901–1910 (2019)
  * [45] Xu, Y., Xie, L., Zhang, X., Chen, X., Qi, G.J., Tian, Q., Xiong, H.: Pc-darts: Partial channel connections for memory-efficient architecture search. arXiv preprint arXiv:1907.05737 (2019)
  * [46] Yang, Y., You, S., Li, H., Wang, F., Qian, C., Lin, Z.: Towards improving the consistency, efficiency, and flexibility of differentiable neural architecture search. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6667–6676 (2021)
  * [47] Zhang, X., Zhou, X., Lin, M., Sun, J.: Shufflenet: An extremely efficient convolutional neural network for mobile devices. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 6848–6856 (2018)
  * [48] Zheng, X., Fei, X., Zhang, L., Wu, C., Chao, F., Liu, J., Zeng, W., Tian, Y., Ji, R.: Neural Architecture Search with Representation Mutual Information. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 11902–11911. IEEE, New Orleans, LA, USA (Jun 2022). https://doi.org/10.1109/CVPR52688.2022.01161, https://ieeexplore.ieee.org/document/9878903/
  * [49] Zhou, H., Yang, M., Wang, J., Pan, W.: Bayesnas: A bayesian approach for neural architecture search (2019)
  * [50] Zhu, C., Han, S., Mao, H., Dally, W.J.: Trained ternary quantization (2017)

Appendix

##  Appendix 0.A Deriving cell architectures

The searched cells are stacked to form the network whose weights are trained
and evaluated. The layers of this network during the evaluation phase is
varied from 4 to 20. It can be seen that the models searched with DARTS with
only 2-cells perform equally well as those of 8-cell search for target model
with layers more than 10. Hence, in our experiments, instead of training
architecture parameters for all 8 cells, we train only 2 cells- one normal and
the other reduction cell. The architecture of the other 6 cells stacked to
form the network during search are derived from either the normal or the
reduction cell as shown in Figure 5.

Figure 5: Top: shows the regular DARTS cell with nodes connected by weighted
operations and the derived cell made of top-weighted operations. Bottom: Shows
the network comprising the normal cell (bold border) and reduction cells
(dotted border) with trainable architecture parameters (red border) and the
derived cells (green border) without any architecture parameters.

##  Appendix 0.B Calculation of search-stage architecture size

The size of the architecture in the search phase $k_{s}$ is different from the
architecture size in evaluation phase due to the softmax weighting factor in
equation 3 (demonstrated in Figure 6). To address this, we introduce a tighter
bound on the search constraint $K_{d^{\prime}}$, which is less than the device
resource constraint $K_{d}$. A lookup graph (LUG) needs to be made for each
dataset by varying $K_{d^{\prime}}$ within appropriate bounds and running the
algorithm until convergence each time to obtain the corresponding device
resource constraint $K_{d}$. The computation time of the LUG can be reduced by
running the searches in parallel.

Figure 6: Demonstrates the calculation of memory size of a single cell in the
architecture during - Left: search phase. Right: evaluation phase

##  Appendix 0.C Algorithm

The practical implementation of our resource-constrained gradient descent-
based approach is illustrated in Algrorithm 1.

Algorithm 1 DCA-NAS - gradient descent based search method

Assign random weights to $\alpha^{i,j}$ on edges $i,j$ denoting weights of
operations in the mixed set

Input look-up graph $G$ and device memory constraint $K_{d}$

Look-up corresponding search memory constraint $K_{d^{\prime}}$ from $G$

Calculate total search time memory size $k_{s}(\alpha)$

while $not$ $converged$ do

Calculate
$\widetilde{\mathcal{L}}(w,\alpha,\lambda)=\mathcal{L}_{\text{val}}\left(w(\alpha),\alpha\right)+\lambda(k_{s}(\alpha)-K_{d^{\prime}})$

Update weights $w$ by descending
$\bigtriangledown_{w}\,\widetilde{\mathcal{L}}_{train}(w,\alpha,\lambda)$

Update $\alpha$ by descending
$\bigtriangledown_{\alpha}\,\widetilde{\mathcal{L}}_{val}(w^{*},\alpha,\lambda)$

Calculate total search time memory size $k_{s}(\alpha)$

Calculate loss as in equation 4

Update $\lambda$

end while

Derive the final architecture based on the learned $\alpha$ by connecting the
top weighted operations among the mixed set

##  Appendix 0.D Implementation Details

The experiments with the smaller vision datasets-MNIST, FashionMNIST,
CIFAR-10, Imagenet-16-120 and TinyImagenet were run on a single Tesla V100
GPU. Training and evaluation on Imagenet-1k was performed on a cluster
containing eight V100 GPUs.
The super-net used for search with smaller vision datasets except Imagenet-1k
consists of 8 cells, with 6 normal cells and 2 reduction cells, and an initial
number of channels set to 16. Each cell has 6 nodes, with the first 2 nodes in
cell k serving as input nodes. The super-net is trained for 50 epochs with a
batchsize of 512, and optimized using SGD with a momentum of 0.9 and weight
decay of $3e-4$. The learning rate is initially set to 0.2 and gradually
reduced to zero using a cosine scheduler. Architecture parameters $\alpha$ are
optimized using Adam optimizer, with a learning rate of $6e-4$, a momentum of
$(0.5,0.999)$, and a weight decay of $1e-3$. The search is run 5 times, and
the architecture with the highest validation accuracy is chosen. For
evaluation, the target-net has 20 cells, with 18 normal cells and 2 reduction
cells, and an initial number of channels set to 36. The target-net is trained
for 600 epochs with a batchsize of 96, optimized using SGD with a momentum of
0.9, weight decay of 3e-4, and gradient clipping of 5. The initial learning
rate is set to 0.025 and gradually reduced to zero using a cosine scheduler.
Additional settings include a cutout length of 16, dropout rate of 0.2, and
use of an auxiliary head. For Imagenet-1k, We reduce the input size from 224 ×
224 to 28 × 28 using three convolution layers with a stride of 2. The super-
net for search has 8 cells starting with 16 channels, and the target-net for
evaluation has 14 cells starting with 48 channels. Both search and evaluation
use a batch size of 1,024. In search, we train for 50 epochs with a learning
rate of 0.5 (annealed down to zero using a cosine scheduler), and a learning
rate of $6e-3$ for architecture parameters. In evaluation, we train for 250
epochs using the SGD optimizer with a momentum of 0.9 and a weight decay of
$3e-5$, and adopt an auxiliary head and the label smoothing technique.

##  Appendix 0.E Model performance by varying FLOPs constraint on CIFAR10,
TinyImagenet and Imagenet-1k

Instead of model parameters, we also experiment with FLOPs as the constraint
in our objective function. As shown in Figure 7, our method DCA-NAS retains
performance till a certain FLOPs constraint, after which it degrades. In
comparison to manual architectures, our NAS approach yields models which
require much smaller FLOPs and hence would have lower latency.

Figure 7:  Plots show that DCA-NAS method discovers models with fewer FLOPs
than other NAS methods and manual architectures without sacrificing prediction
performance.