# Deformation and breakup dynamics of droplets within a tapered channel

Andrea Montessori  Istituto per le Applicazioni del Calcolo CNR, via dei
Taurini 19, 00185, Rome, Italy  Michele La Rocca  Dipartimento di Ingegneria,
Università degli Studi Roma TRE, via Vito Volterra 62, Rome, 00146, Italy
Pietro Prestininzi  Dipartimento di Ingegneria, Università degli Studi Roma
TRE, via Vito Volterra 62, Rome, 00146, Italy  Adriano Tiribocchi  Istituto
per le Applicazioni del Calcolo CNR, via dei Taurini 19, 00185, Rome, Italy
Sauro Succi  Center for Life Nanoscience @ La Sapienza, Istituto Italiano di
Tecnologia, viale Regina Elena 295, 00161, Rome, Italy  Istituto per le
Applicazioni del Calcolo CNR, via dei Taurini 19, 00185, Rome, Italy Institute
for Applied Computational Science, Harvard John A. Paulson School of
Engineering and Applied Sciences, Cambridge, MA 02138, United States

###### Abstract

In this paper we numerically investigate the breakup dynamics of droplets in
an emulsion flowing in a tapered microchannel with a narrow constriction. The
mesoscale approach for multicomponent fluids with near contact interactions is
shown to capture the deformation and breakup dynamics of droplets interacting
within the constriction, in agreement with experimental evidences. In
addition, it permits to investigate in detail the hydrodynamic phenomena
occurring during the breakup stages. Finally, a suitable deformation parameter
is introduced and analyzed to characterize the state of deformation of the
system by inspecting pairs of interacting droplets flowing in the narrow
channel.

## Credit Line

The following article has been accepted by Physics of Fluids. After it is
published, it will be found at https://aip.scitation.org/journal/phf

##  I Introduction

In recent years droplet-based microfluidics, namely the art of manipulating,
controlling and fine-tuning the production of emulsions characterized by a
high degree of structural ordering and monodispersity [19, 28, 9], has
witnessed a giant leap forward, mostly thanks to the advancements in
miniaturization processes which, in turn, paved the way to a clever design of
micron and submicron-sized channels and devices [20, 33, 17]. These
technological boosts sustained a surge of experimental [5, 3] and theoretical
[22, 21, 32] activities aimed at shedding light on the underlying mechanisms
responsible for the emergence of complex dynamical behaviors in densely packed
systems evolving in strongly confined geometries.

Although both experimental and theoretical research have already driven many
improvements towards their understanding, the investigation of droplet-based
environments poses a steep challenge owing to i) the variety of spatial scales
in play, ii) the presence of highly non-linear behaviors, often out of reach
for analytical, perturbative approaches and iii) the presence of many local
field quantities, whose detailed knowledge is of decisive importance for a
clever design of novel microfluidic devices. From this standpoint, an accurate
numerical modeling of these complex, many-body, interacting systems stands as
a way to inspect in closer details a number of peculiar phenomena occurring at
different spacetime scales and to unveil a plethora of complex behaviors
otherwise inaccessible neither to theoretical nor to experimental approaches.
The present work, which falls precisely within this line of thinking, aims at
presenting ab-initio hydrodynamic, mesoscale simulations of deformation and
breakup phenomena occurring in dense emulsions, namely monodisperse ensembles
of interacting droplets immersed in a bulk fluid flowing within a tapered
microfluidic channel [2, 6, 8].

Indeed, a particularly relevant phenomenon usually compromising the accuracy
of the resulting emulsion is the breakup of droplets, especially in dense
suspensions where the volume fraction exceeds the close-packing limit [29, 6,
2]. Experimental results of a single droplet under an external flow support
the view that deformation and breakup are determined by a delicate balance
between viscous stress and interfacial tension, described by the Capillary
number ($Ca$). While the viscous stress tends to deform the drop, the
interfacial tension contrasts this effect and holds the drop in a spherical
shape. Once the capillary number overcomes a critical value, viscous stress
dominates and the breakup becomes inevitable. In concentrated emulsions, in
contrast, the volume fraction of the droplet phase significantly increases,
and experiments suggest that breakup arises from droplet-droplet and droplet-
wall interactions [6, 2, 11, 7] However many questions remain open.
At variance with the aforementioned experimental works, the goal of the
present work is to pinpoint the effect of viscous dissipation over surface
tension forces of individual droplets of a concentrated emulsion flowing
within a tapered channel. This target is achieved by varying the Capillary
number while keeping the Reynolds number constant, in order to decouple the
effects due to inertia, surface tension and viscous forces, a task generally
difficult (if not impossible) to realize experimentally. Besides confirming
the results described in Ref.[6], our model, being capable to deliver the full
hydrodynamic picture, offers a way to unveil highly non-trivial details of the
structure of the velocity field observed during the breakup stages. In
addition, by tracking the circularity of hundreds of interacting droplets, we
find that the maximum deformation, before a breakup event, is independent of
$Ca$ and the number of breakups occurring at the constriction linearly
increases with $Ca$.

##  II Method

In the following we briefly describe the extended mesoscale approach for
multicomponent flows with interacting interfaces developed in [23, 24]. The
multicomponent system, namely an emulsion formed by droplets immersed in a
bulk fluid, is modeled via a color-gradient [10, 30] regularized Lattice
Boltzmann method [25, 13, 4], which employs two sets of probability
distribution functions. Each set evolves via a sequence of streaming and
collision steps [12, 31, 1]

$f_{i}^{k}(\mathbf{x}+\mathbf{c}_{i}\Delta t,t+\Delta
t)=f_{i}^{k}(\mathbf{x},t)+\Omega_{i}^{k}(f_{i}^{k}(\mathbf{x},t))+F_{i}^{rep},$
(1)

where $f_{i}^{k}(\mathbf{x},t)$ is the $i^{th}$ discrete probability
distribution function for the $k^{th}$ component, giving the probability of
finding a fluid particle at position $\mathbf{x}$, time $t$ and with discrete
velocity $\mathbf{c}_{i}$. The index $k$ is such that $k=1,2$, while $i$
belongs to the range $0\leq i\leq N_{set}$, being $N_{set}$ the dimension of
the set of probability distribution functions. This is equal to $8$ for the
two-dimensional nine speeds lattice (D2Q9) adopted in this paper. The time
step $\Delta t$ is expressed in lattice units [12] and set to $1$, a usual
choice in the LBM [31]. Finally, $F_{i}^{rep}$ is a force aimed at upscaling
the repulsive near-contact interactions acting on scales much smaller than the
resolved ones [23].

Once the set of distribution is known at each lattice site, the macroscopic
fields of interest, namely the fluid density $\rho_{k}$, the linear momentum
$\rho_{k}\mathbf{u}_{k}$ and the pressure $p_{K}$, can be obtained by
computing the relevant statistical moments:

$\rho_{k}=\sum_{i=0}^{N_{set}}f_{i}^{k}(\mathbf{x},t),\\\
\rho_{k}\mathbf{u}_{k}=\sum_{i=0}^{N_{set}}\mathbf{c}_{i}f_{i}^{k}(\mathbf{x},t),\
p_{k}=\frac{\rho_{k}}{3}.$ (2)

The total fluid density $\rho$ and the total momentum $\rho\mathbf{u}$ of the
mixture can be obtained via the following relations:

$\rho=\sum_{k=1}^{2}\rho_{k},\\\
\rho\mathbf{u}=\sum_{k=1}^{2}\rho_{k}\mathbf{u}_{k}.$ (3)

It is worth recalling that the multicomponent approach employed in this work
is based on a variant of the color-gradient LB model. By employing the
standard formalism, the collision operator can be split into three parts [16,
15]:

$\Omega_{i}^{k}=(\Omega_{i}^{k})^{(3)}[(\Omega_{i}^{k})^{(1)}+(\Omega_{i}^{k})^{(2)}].$
(4)

The term $(\Omega_{i}^{k})^{(1)}$ is the usual single relaxation time
Bhatnagar–Gross–Krook collisional operator [31]

$(\Omega_{i}^{k})^{(1)}=\frac{f_{i,eq}^{k}(\mathbf{u},\mathbf{x},t)-f_{i}^{k}(\mathbf{x},t)}{\tau},$
(5)

where $\tau$ is the effective relaxation time, which depends on the viscosity
$\nu$ of the emulsion

$\nu=\left(\frac{\rho_{1}}{\rho_{1}+\rho_{2}}\frac{1}{\nu_{1}}+\frac{\rho_{2}}{\rho_{1}+\rho_{2}}\frac{1}{\nu_{2}}\right)^{-1},\
\tau=3\nu+\frac{1}{2}.$ (6)

Here $\nu_{1},\nu_{2},\rho_{1},\rho_{2}$ are the kinematic viscosities and the
densities of the two fluids in the bulk, respectively. Also,
$f_{i,eq}^{k}(\mathbf{x},t)$ is the $i^{th}$ equilibrium discrete probability
distribution function for the $k^{th}$ component, formally derived as a low
Mach, second-order expansion of a Maxwellian probability equilibrium
distribution [31]

$f_{i,eq}^{k}(\mathbf{u},\mathbf{x},t)=w_{i}\rho_{k}\Bigg{(}1+3\mathbf{c}_{i}\cdot\mathbf{u}+9\frac{(\mathbf{c}_{i}\cdot\mathbf{u})^{2}}{2}-3\frac{\mathbf{u}\cdot\mathbf{u}}{2}\Bigg{)},$
(7)

being $w_{i}$ the weights of the D2Q9 lattice adopted in this paper [31].
The term $(\Omega_{i}^{k})^{(2)}$ is the perturbation part, which accounts for
the interfacial tension, and reads as

$(\Omega_{i}^{k})^{(2)}=\frac{A_{k}}{2}|\nabla{\Theta}|\Bigg{(}w_{i}\bigg{(}\frac{\mathbf{c}_{i}\cdot\nabla{\Theta}}{|\nabla{\Theta}|}\bigg{)}^{2}-B_{i}\Bigg{)},$
(8)

where $A_{k}$ ($k=1,2)$ and $B_{i}$ $(i=0,N_{set})$ are suitable constants
defined in [23]. $\Theta$ is a scalar phase field, defined as

$\Theta=\frac{\rho_{1}-\rho_{2}}{\rho_{1}+\rho_{2}},$ (9)

assuming the value $1$ in the fluid component with density $\rho_{1}$ and the
value $-1$ in the fluid component with density $\rho_{2}$. It is worth
observing that the constants $A_{1},A_{2}$ are related to the surface tension
$\sigma$ by

$\sigma=\frac{2}{9}\tau(A_{1}+A_{2}).$ (10)

Finally the third term $(\Omega_{i}^{k})^{(3)}$ is the recolouring operator
which aims at minimizing the mutual diffusion between the fluid components,
thus favouring their separation [14]

$f_{i}^{1}(\mathbf{x},t)=\frac{\rho_{1}}{\rho}\sum_{k=1}^{2}f_{i}^{*k}(\mathbf{x},t)+\beta\frac{\rho_{1}\rho_{2}}{\rho^{2}}\Bigg{(}\frac{\nabla\Theta\cdot\mathbf{c}_{i}}{|\nabla\Theta|}\Bigg{)}\sum_{k=1}^{2}f_{i,eq}^{k}(0,\mathbf{x},t),$
(11)
$f_{i}^{2}(\mathbf{x},t)=\frac{\rho_{2}}{\rho}\sum_{k=1}^{2}f_{i}^{*k}(\mathbf{x},t)-\beta\frac{\rho_{1}\rho_{2}}{\rho^{2}}\Bigg{(}\frac{\nabla\Theta\cdot\mathbf{c}_{i}}{|\nabla\Theta|}\Bigg{)}\sum_{k=1}^{2}f_{i,eq}^{k}(0,\mathbf{x},t).$
(12)

Here $f_{i}^{*k}(\mathbf{x},t)\ (i=0,N_{set};\ k=1,2)$ are the post-collision
and post-perturbation probability distribution functions, while the
coefficient $\beta$ is a free parameter which tunes the width of the
interface.
By performing a multiscale Chapman–Enskog expansion, it can be shown that the
hydrodynamic limit of Eq.(1), in the low frequency-long wavelength and low
Mach number limit, is a set of conservation laws for mass and linear momentum

$\frac{\partial\rho}{\partial t}+\nabla\cdot(\rho\mathbf{u})=0$ (13)
$\frac{\partial(\rho\mathbf{u})}{\partial
t}+\nabla\cdot(\rho\mathbf{u}\mathbf{u})=-\nabla
p+\nabla\cdot(\rho\nu(\nabla(\mathbf{u}+\mathbf{u}^{T}))+(\nabla\sigma-\sigma(\nabla\cdot\mathbf{n})\mathbf{n}-A_{h}\mathbf{n})\delta_{I},$
(14)

where $p=\sum_{k=1}^{2}p_{k}$ is the total pressure. The last term at the
right hand side of Eq.(14) codes for the effects of the surface tension
$\sigma$ and for the additional, short-range, repulsive force at the interface
between the two fluid components. In Eq.(14), $\delta_{I}$ is an index
function, defined as $\delta_{I}=|\nabla\Theta|/2$, which localizes the force
on the interface, while $A_{h}$ is a parameter controlling the strength of the
repulsive force oriented as the local normal $\mathbf{n}$ to the interface
[23].

##  III Results

In the following, after a brief description of the numerical setup employed,
we discuss in detail the deformation and breakup phenomena by tracking the
evolution of the droplets flowing within the narrow channel.

###  III.1 Numerical setup

The geometrical setup, conveniently sketched in fig.1(a), consists of a
tapered geometry with an outlet narrow channel positioned at its end. The
height of the domain is $L_{y}=416lu$ (lattice units), its length is
$L_{x}=750lu$, while the height of the constriction is $h\sim 0.75d=45lu$,
being $d$ the diameter of the droplet. Finally, the angle of the tapered
geometry is set to $30^{\circ}$ as in [6, 2, 8, 29].

Figure 1: Panel (a): Sketch of the microchannel geometry employed in this
work. Panel (b): Snapshot of the droplets’ field flowing within the tapered
channel. The droplets are continuously injected at the inlet via an internal
periodic boundary condition, as proposed in [26, 27].

The emulsion, flowing within the microchannel and continuously injected via an
internal periodic boundary condition as in [27, 26], is made of a set of
monodisperse droplets of radius $d/2$, as shown fig.1(b). The relevant
parameters employed in the simulations are the following: kinematic viscosity
$\nu=0.023$, droplets’ reference velocity $U=0.018$, droplet’s diameter
$d=60$, while the surface tension ranges between $\sigma=0.006\div 0.04$.
It is worth noting that the viscosity $\nu$ (taken equal in both the dispersed
and the bulk fluid) and the inlet carrier velocity $U$ were set to constant
values in all simulations, while the surface tension was varied within a range
of an order of magnitude.  This particular choice permits to vary the
Capillary number ($Ca=\rho U\nu/\sigma=0.011\div 0.072$) independently of the
Reynolds number ($Re=Uh/\nu$), which instead was kept fixed in all
simulations. By doing so, we isolated the effects of the variations of the
elongational viscous forces versus the surface tension forces, regardless of
the inertia.

###  III.2 Deformation dynamics of interacting droplets within a constriction

Firstly, we start with a brief description of the salient dynamical features
emerging during the interaction of a pair of flowing droplets. With reference
to fig.2 (showing two sequences of droplets interacting within the
microchannel), once the droplets approach the constriction, they start to
deform under the effect of the confinement, which is caused by i) the walls of
the constriction and ii) the presence of the neighboring droplet. As evidenced
in fig.2, two possible outcomes may occur once two droplets concur to enter
within the narrow channel. In the first one (panel (a)), the leading edge of a
droplet undergoes a fast expansion which favors its escape towards the outlet
of the constriction. In this case the droplets align horizontally within the
narrow channel and no breakup occurs. In the second one (panel (b)), two
droplets stretch under the effect of the viscous forces acting at the
interface level, until one of them attains a critical deformation beyond which
surface tension forces are not able to withstand the elongational forces, thus
causing the split of the droplet in two.

Figure 2: Droplets’ deformation and breakup in the narrow constriction at
$Ca=0.043$. Left columns report experimental results [6], while the right
columns show the simulations. Two possible outcomes may occur once the
droplets concur to enter the narrow channel. In the first one (panel(a)), the
droplet undergoes a fast expansion which favors its escape towards the outlet
of the constriction. In the second one (panel (b)), the two droplets stretch
under the effect of the viscous forces acting at the interface level until one
breaks. Bottom panels. Comparison between experiments[6] (solid line) and
simulations (open circles) of the deformation history.

It is interesting to note that, although the volume fraction of the droplets
in our simulation is evidently lower than that of the experiments reported in
[6] (left rows in panel (a) and (b)), the overall deformation and breakup
dynamics are still well captured by the mesoscale approach employed herewith.
To gain a more quantitative insight on the deformation and breakup of the
droplets, we compared the deformation history, between experiments and
simulations (bottom panels of fig.2), for the cases reported in figure.
Interestingly, the agreement between the two is remarkable, since our model
captures the time evolution of the deformation parameter (defined as
$\frac{P}{2\sqrt{\pi A}}$ following Ref.[6], where $P$ and $A$ are the
perimeter and the area of the droplet) and correctly predicts the value of its
maximum deformation.

Figure 3:  (a-d) Sequence of the velocity field during a breakup event at
$Ca=0.043$. The color map represents the magnitude of the velocity field
divided by its maximum value in the narrow channel, while the arrows denote
the local direction of the flow field. Panel (e) shows a zoom of the area
highlighted in panel (c).

###  III.3 Structure of the velocity field

Since the LB approach grants the access to local hydrodynamic quantities not
accessible to experiments, it is worthwhile inspecting the characteristics of
the fluid flow developed during the breakup stages, to shed light on the
complex hydrodynamic phenomena emerging in the rupture process.

Figure 4:  (a-e) Flow field during the pinching and breakup stages, computed
on a frame of reference moving with the droplets. During the pinching stage, a
recirculation forms in the upstream region, opposes to the fluid flux crossing
the neck of the droplet and augments in size as the droplet moves forward and
deforms. The color map denotes the magnitude of the velocity (made
dimensionless since divided to the maximum value of the velocity $U_{M}$),
while the arrows indicate the direction of the velocity field. The arrow in
the pinching region indicates the direction of the fast flow through the
droplet’s neck

To this aim, in fig.3 (a-d) we show the velocity field during a breakup
sequence at $Ca=0.043$, where the color map represents the magnitude of the
velocity and the arrows denote the local direction of the flow field. As one
can see, in an early stage (panel(a)), the leading edge of the lower droplet
moves faster than its neighbor. This causes the head of the lower droplet to
be quickly squeezed into the constriction by the upper one (see panel (b)).
Consequently, the leading edge of the upper droplet enters the narrow channel
and, due to the effect of the confinement, starts to accelerate, as evidenced
by the magnitude of the flow field (see panel (c)). Afterwards, a high speed
area develops at the droplet’s neck (see panel (e) which shows a close-up view
of the region enclosed by the rectangle in panel(c)), determining an increase
of the shear between the interfaces in close contact, thus leading to the
pinch-off of the neck of the lower droplet (panel (d)).

To further elucidate the complex fluid dynamics involved in the breakup
process, we inspected the flow field during the pinching and breakup stages in
a relative frame of reference moving at the average speed on each section. As
shown in fig.4, during the pinching stage a recirculation forms in the
upstream region, opposes to the fluid flux crossing the neck of the droplet
and increases in size as the droplet moves forward and deforms (panels b-b1
and c-c1). This means that a “fluid particle” trapped in the upstream vortex
would take a time, in order to advance towards the exit of the channel, longer
than the one needed by a particle flowing through the pinch. Note also that
the size of the fluid recirculation increases as the droplet squeezes during
its passage through the constriction, becoming comparable with the length of
the upstream part of the droplet when it breaks. It is precisely this
“momentum unbalance” mechanism (recurring at each breakup event) that produces
the stretching of the droplet followed by its rupture.

To conclude this part, we would like to point out that, in each simulation,
the droplets’ diameter were set to $\sim 60$ lattice units, which means having
a Cahn number of the order of $\sim 0.05$, a typical value in simulating the
physics of resolved, complex interfaces (see, for instance, [18]). Moreover,
compressibility effects are negligible since the maximum values of the
velocity reached in our simulations are of order $10^{-2}$ (lattice units over
step), delivering a Mach number of the same order. These values ensure that
our approach is adequate to capture, indeed with remarkable accuracy, the
correct physics observed in droplet microfluidics and that the provided fluid
dynamics picture is not affected by spurious numerical artifacts.

At this point, several questions naturally arise: What is the effect of the
Capillary number, the main parameter governing the processes detailed above,
on the breakup statistics? Is there a maximum deformation over which the
droplet inevitably breaks? And, if the latter exists, does it depend on the
flow characteristics? In the next sections we try to address these issues by
introducing suitable observables which help to put the phenomenon under
scrutiny on a more quantitative ground.

###  III.4 Breakup statistics: Circularity, Breakup rates and Critical
Deformation

As introduced at the end of section IIIC, it is now of interest to investigate
the behavior of the flowing emulsion from a statistical perspective, to
pinpoint the effect of the governing parameters on shape deformation and
breakup.

To do so, we introduce the circularity $\chi$, a dimensionless parameter
assessing the deformation of the droplets flowing within the constriction. It
is defined as

$\chi=\frac{4\pi A}{P^{2}}$ (15)

and it ranges between $1$ (circular shape) and $0$ (needle-like geometry);
thus monitoring its evolution allows to evaluate the departure of the droplet
from its circular shape. The observations are carried out by tracking the
deformation and breakup events occurring within an Eulerian volume of control,
coinciding with the narrow channel. Also, to gain significant statistics over
the number of such events occurring in the channel, each simulation was run
for $10^{6}$ steps.

In fig.5 we report the plot of circularity vs time for each doublet flowing
within the constriction and for each value of the Capillary number inspected.
In each plot, the red points denote the minimum value of circularity (i.e. the
maximum droplet’s deformation) reached just before a breakup event.

Figure 5: Circularity versus time for different Capillary numbers. (a)
$Ca=0.011$. (b) $Ca=0.022$. (c) $Ca=0.029$. (d) $Ca=0.043$. (e) $Ca=0.054$.
(f) $Ca=0.072$. Red spots indicate the minimum value of $\chi$, achieved
before a breakup event occurs.

An interesting feature is that the ratio between the number of droplets which
break and the total number of considered events steadily increases with $Ca$,
in agreement with previous experimental data [6]. This fact is clearly
supported by the increasing number of red dots, from panel $(a)$ to box $(f)$.
Moreover, as the Capillary number increases, the distribution of points in the
plots of fig.5 appears more uniform, thus suggesting the two following
observations:

i) the probability that two droplets simultaneously arrive to the constriction
and compete to enter it, depends on $Ca$.

ii) when two droplets concur to enter the constriction, for low values of $Ca$
the surface tension forces prevail over the viscous (extensional) ones, thus
favoring faster expansions of the droplets and preventing their breakup.

A second aspect is that the minimum circularity reached by the droplets before
the breakup is approximately independent of the Capillary number and roughly
equal to $\chi_{min}\sim 0.22$. This observation is further confirmed in
fig.6(a), which shows the plot of the $pdf(\chi)$ obtained by collecting the
values of $\chi_{min}$ for each breakup event. As clearly visible, the pdf
follows a Gaussian trend, centered on $\overline{\chi}_{min}\sim 0.22$ with a
rather small standard deviation, $std\sim 0.026$.

Figure 6: Panel(a): probability density function of $\chi_{min}$. Panel(b):
frequency of breakup events (i.e. number of breakup events versus the total
simulation time) versus $Ca$. The colored regions denote the Capillary bounded
($Ca$-bounded) and the droplets’ volume fraction (($\phi$-bounded)) bounded
regions.

From the above, it seems reasonable to conclude that, at least for the setup
under investigation and in the limit of laminar flow, the maximum deformation
attained by the droplets before their breakup is independent of the flow
details, being a universal characteristics of the phenomenon in play.

On the contrary the breakup rates steadily increase as $Ca$ augments. This
effect is described in fig.6(b), which reports the frequency of the breakup
events $N^{*}$, defined as the ratio of the number of breakups over the number
of time steps, versus $Ca$. The breakup rate scales roughly proportional with
$Ca$ in the range $0.011\leq Ca\leq 0.054$, while it saturates for $Ca>0.054$.
This fact can be explained as follows. In the first region of the plot
($Ca$-bounded), the number of breakup events is limited by the Capillary
number which, as shown before, governs the deformation dynamics of the
droplets concurring to enter the constriction. For low values of $Ca$, two
droplets approaching the inlet of the narrow channel have a smaller
probability to deform and break, since the surface tension forces dominate
over the extensional actions of the viscous ones. As $Ca$ increases, the
effect of the viscous forces grows accordingly and the droplets are more prone
to break. By contrast, for $Ca$ larger than a critical value ($\phi$-bounded
region) the number of breakup events is limited by the number of doublets
competing to enter the narrow channel. In other words, for $Ca\geq Ca_{sat}\
(Ca_{sat}\sim 0.05)$, the number of breakup events becomes independent of the
Capillary number, attaining a saturation value which depends only on the
volume fraction of the emulsion. This observation is consistent with the fact
that the droplet breakup in a dense emulsion is due to droplet-droplet
interactions, whose number is expected to considerably decrease when the
volume fraction diminishes, as discussed in Ref.[6] for a dilute emulsion. It
is thus reasonable to speculate that the critical value of the Capillary
number depends on the droplet volume fraction of the emulsion.

To conclude, it is interesting to observe that the linear relationship between
the number of breakup events and the Capillary number (within the Capillary
bounded region) can be deduced by calling on simple energetic arguments.
Firstly, the work per unit time done by the viscous stresses within the
droplet $\dot{L}_{v}$ can be written as

$\dot{L}_{v}=2\rho\nu\int_{A}\mathcal{D}:\mathcal{D}dA\approx
2\rho\nu\bigg{(}\pi\frac{U}{P}\bigg{)}^{2}A,$ (16)

where $\mathcal{D}$ is the symmetric part of the velocity gradient
($\mathcal{D}=\frac{\nabla\mathbf{u}+\nabla\mathbf{u}^{T}}{2}$), whose order
of magnitude has been estimated as $O(\mathcal{D})\approx\pi U/P$ with $P$ and
$A$ perimeter and area of the droplet, respectively. Moreover, it is
straightforward to express $\dot{L}_{v}$ in terms of the circularity

$\dot{L}_{v}\approx\frac{1}{2}\rho\nu U^{2}\chi.$ (17)

On the other hand, the work per unit time done by the surface tension on the
droplet’s surface is proportional to $\dot{L}_{st}\approx\sigma U$.

By taking the ratio between $\dot{L}_{v},\dot{L}_{st}$ we can introduce a non-
dimensional group $\mathcal{N}_{vst}$

$\mathcal{N}_{vst}=\frac{1}{2}\frac{\rho\nu U}{\sigma}\chi=\frac{1}{2}Ca\chi,$
(18)

which, interestingly, depends linearly on both $Ca$ and $\chi$. As observed
before, when a breakup occurs, the minimum circularity is independent of the
Capillary number and can be considered constant while $\mathcal{N}_{vst}$ (the
ratio between extensional viscous and surface tension works per unit time)
scales linearly with the Capillary number.

##  IV Conclusions

In this work we have numerically investigated, using a lattice Boltzmann
approach for multicomponent fluids augmented with disjoining near contact
interactions, the deformation and breakup dynamics of droplets flowing within
a tapered channel with a constriction. The built-in hydrodynamic features of
such mesoscale approach allow us to study in details the fluid dynamics
emerging at the interface level during the breakup phenomena. In particular,
our simulations confirm the experimental observation reported in [6], by
reproducing the peculiar behaviors observed during the deformation and breakup
stages of doublets interacting within the narrow constriction. Moreover, the
introduction of a suitable parameter allows for a quantitative assessment of
the deformation state of the system. We find that i) the maximum deformation
reached by the droplets before breakup is independent of $Ca$ and ii) the
number of breakup events linearly depends on $Ca$ until a critical value is
attained. This one is likely to have a strong dependence on the packing
fraction of the emulsion.

As a perspective, the present model, properly extended, could be potentially
employed in a number of biological applications, one for all hemodynamics. In
this respect, an all-mesoscale approach capable of modelling the coupled
evolution of plasma and red blood cells could pave the way to large scale
hemodynamic simulations with unprecedented resolution and computational
efficiency.

## Acknowledgements

A.M., A.T.and S.S. acknowledge funding from the European Research Council
under the European Union’s Horizon 2020 Framework Programme (No. FP/2014-2020)
ERC Grant Agreement No.739964 (COPMAT).
A.M. acknowledges the CINECA Computational Grant ISCRA-C IsC83 - “SDROMOL”,
id. HP10CZXK6R under the ISCRA initiative, for the availability of high
performance computing resources and support.
M.L.R and P.P. acknowledge funding from the Italian Ministry of Education,
University and Research (MIUR), in the frame of the Departments of Excellence
Initiative 2018-2022, attributed to the Department of Engineering of Roma Tre
University.

## Data Availability

The data that supports the findings of this study are available within the
article

## References

  * [1] R. Benzi, S. Succi, and M. Vergassola.  The lattice boltzmann equation: theory and applications.  Physics Reports, 222(3):145–197, 1992.
  * [2] A. D. Bick and S. K. Tang.  Effect of volume fraction on droplet break-up in an emulsion flowing through a microfluidic constriction.  Applied Physics Letters, 115(9):093702, 2019.
  * [3] L. Chu, A. S. Utada, R. K. Shah, J. W. Kim, and D. A. Weitz.  Controllable monodisperse multiple emulsions.  Angew. Chem. Int. Ed. Engl., 46:8970, 2007.
  * [4] C. Coreixas, B. Chopard, and J. Latt.  Comprehensive comparison of collision models in the lattice boltzmann framework: Theoretical investigations.  Physical Review E, 100(3):033305, 2019.
  * [5] M. Costantini, C. Colosi, J. Jaroszewicz, A. Tosato, W. Swieszkowski, M. Dentini, P. Garstecki, and A. Barbetta.  Microfluidic foaming: A powerful tool for tailoring the morphological and permeability properties of sponge-like biopolymeric scaffolds.  ACS Appl. Mater. Interfaces, 7:23660–23671, 2015.
  * [6] Y. Gai, J. W. Khor, and S. K. Tang.  Confinement and viscosity ratio effect on droplet break-up in a concentrated emulsion flowing through a narrow constriction.  Lab on a Chip, 16(16):3058–3064, 2016.
  * [7] Y. Gai, M. Kim, M. Pan, and S. K. Tang.  Amphiphilic nanoparticles suppress droplet break-up in a concentrated emulsion flowing through a narrow constriction.  Biomicrofluidics, 11(3):034117, 2017.
  * [8] Y. Gai, C. M. Leong, W. Cai, and S. K. Tang.  Spatiotemporal periodicity of dislocation dynamics in a two-dimensional microfluidic crystal flowing in a tapered channel.  Proceedings of the National Academy of Sciences, 113(43):12082–12087, 2016.
  * [9] P. Garstecki and G. M. Whitesides.  Flowing crystals: nonequilibrium structure of foam.  Physical review letters, 97(2):024503, 2006.
  * [10] A. K. Gunstensen, D. H. Rothman, S. Zaleski, and G. Zanetti.  Lattice boltzmann model of immiscible fluids.  Phys. Rev. A, 43:4320, 1991.
  * [11] J. W. Khor, M. Kim, S. S. Schütz, T. M. Schneider, and S. K. Tang.  Time-varying droplet configuration determines break-up probability of drops within a concentrated emulsion.  Applied Physics Letters, 111(12):124102, 2017.
  * [12] T. Krüger, H. Kusumaatmaja, A. Kuzmin, O. Shardt, G. Silva, and E. M. Viggen.  The lattice boltzmann method.  Springer International Publishing, 10(978-3):4–15, 2017.
  * [13] J. Latt and B. Chopard.  Lattice boltzmann method with regularized pre-collision distribution functions.  Mathematics and Computers in Simulation, 72(2-6):165–168, 2006\.
  * [14] M. Latva-Kokko and D. H. Rothman.  Diffusion properties of gradient-based lattice boltzmann models of immiscible fluids.  Physical Review E, 71(5):056702, 2005.
  * [15] S. Leclaire, A. Parmigiani, O. Malaspinas, B. Chopard, and J. Latt.  Generalized three-dimensional lattice boltzmann color-gradient method for immiscible two-phase pore-scale imbibition and drainage in porous media.  Phys. Rev. E, 95:033306, 2017.
  * [16] S. Leclaire, M. Reggio, and J.-Y. Trépanier.  Numerical evaluation of two recoloring operators for an immiscible two-phase flow lattice boltzmann model.  Applied Mathematical Modelling, 36(5):2237–2252, 2012.
  * [17] C. M. Leong, Y. Gai, and S. K. Tang.  Internal flow inside droplets within a concentrated emulsion during droplet rearrangement.  Physics of Fluids, 30(3):032002, 2018.
  * [18] F. Magaletti, F. Picano, M. Chinappi, L. Marino, and C. M. Casciola.  The sharp-interface limit of the cahn–hilliard/navier–stokes model for binary fluids.  Journal of Fluid Mechanics, 714:95–126, 2013.
  * [19] P. Marmottant and J.-P. Raven.  Microfluidics with foams.  Soft Matter, 5(18):3385–3388, 2009.
  * [20] S. Mehraji and M. Saadatmand.  Flow regime mapping for a two-phase system of aqueous alginate and water droplets in t-junction geometry.  Physics of Fluids, 33(7):072009, 2021.
  * [21] A. Montessori, M. Lauricella, E. Stolovicki, D. A. Weitz, and S. Succi.  Jetting to dripping transition: Critical aspect ratio in step emulsifiers.  Physics of Fluids, 31(2):021703, 2019.
  * [22] A. Montessori, M. Lauricella, S. Succi, E. Stolovicki, and D. Weitz.  Elucidating the mechanism of step emulsification.  Phys. Rev. F., 3:072202, 2018.
  * [23] A. Montessori, M. Lauricella, N. Tirelli, and S. Succi.  Mesoscale modelling of near-contact interactions for complex flowing interfaces.  Journal of Fluid Mechanics, 872:327–347, 2019.
  * [24] A. Montessori, M. Lauricella, A. Tiribocchi, and S. Succi.  Modeling pattern formation in soft flowing crystals.  Physical Review Fluids, 4(7):072201, 2019.
  * [25] A. Montessori, P. Prestininzi, M. La Rocca, and S. Succi.  Lattice boltzmann approach for complex nonequilibrium flows.  Physical Review E, 92(4):043308, 2015.
  * [26] A. Montessori, A. Tiribocchi, M. Lauricella, F. Bonaccorso, and S. Succi.  Mesoscale modelling of droplets’ self-assembly in microfluidic channels.  Soft Matter, 17(9):2374–2383, 2021.
  * [27] A. Montessori, A. Tiribocchi, M. Lauricella, F. Bonaccorso, and S. Succi.  Wet to dry self-transitions in dense emulsions: From order to disorder and back.  Physical Review Fluids, 6(2):023606, 2021.
  * [28] J. P. Raven and P. Marmottant.  Microfluidic crystals: dynamic interplay between rearrangement waves and flow.  Phys. Rev. Lett., 102:084501, 2009.
  * [29] L. Rosenfeld, L. Fan, Y. Chen, R. Swoboda, and S. K. Tang.  Break-up of droplets in a concentrated emulsion flowing through a narrow constriction.  Soft Matter, 10(3):421–430, 2014.
  * [30] D. H. Rothman and J. M. Keller.  Immiscible cellular-automaton fluids.  Journal of Statistical Physics, 52(3-4):1119–1127, 1988.
  * [31] S. Succi.  The lattice Boltzmann equation: for complex states of flowing matter.  Oxford University Press, 2018.
  * [32] A. Tiribocchi, A. Montessori, F. Bonaccorso, M. Lauricella, and S. Succi.  Shear dynamics of polydisperse double emulsions.  Physics of Fluids, 33(4):047105, 2021.
  * [33] S. A. Vagner, S. A. Patlazhan, C. A. Serra, and D. Funfschilling.  Vortex flow evolution in a growing microdroplet during co-flow in coaxial capillaries.  Physics of Fluids, 33(7):072010, 2021.


# Inter-layer Transition in Neural Architecture Search

Benteng Ma1,2 Jing Zhang2 Yong Xia1 Dacheng Tao2 1Northwestern Polytechnical
University  China
2The University of Sydney  Australia

###### Abstract

Differential Neural Architecture Search (NAS) methods represent the network
architecture as a repetitive proxy directed acyclic graph (DAG) and optimize
the network weights and architecture weights alternatively in a differential
manner. However, existing methods model the architecture weights on each edge
(i.e., a layer in the network) as statistically independent variables,
ignoring the dependency between edges in DAG induced by their directed
topological connections. In this paper, we make the first attempt to
investigate such dependency by proposing a novel Inter-layer Transition NAS
method. It casts the architecture optimization into a sequential decision
process where the dependency between the architecture weights of connected
edges is explicitly modeled. Specifically, edges are divided into inner and
outer groups according to whether or not their predecessor edges are in the
same cell. While the architecture weights of outer edges are optimized
independently, those of inner edges are derived sequentially based on the
architecture weights of their predecessor edges and the learnable transition
matrices in an attentive probability transition manner. Experiments on five
benchmarks confirm the value of modeling inter-layer dependency and
demonstrate the proposed method outperforms state-of-the-art methods. Code
will be available at https://github.com/btma48/ITNAS.

Figure 1: (a) An exemplar directed acyclic graph of a cell. The blue and green
nodes are outputs of previous cells. The orange node is the output of the
current cell. The white nodes are features learned by layers in the current
cell (, the directed edges), where $\bm{\alpha^{(i,j)}}$ represents the
categorical probability of operations on edge $(i,j)$. (b) Existing methods
optimize $\bm{\alpha^{(i,j)}}$ independently. (c) Our method learns the inter-
layer transition to model the dependency between topologically connected edges
in an attentive probability transition manner.

##  1 Introduction

Recently, the progress in computer vision research has been advanced by deep
learning dramatically. The essential ingredient of deep learning is the design
and optimization of the target deep neural networks. While the early stages of
deep learning placed great emphasis on feature engineering [12, 14, 35], the
focus has now migrated to architecture surgery. However, manually designing
neural network architectures entails both specialized expertise and tremendous
computational resources. Neural Architecture Search (NAS) presents a promising
alternative approach to alleviating this affliction by automatically searching
architectures superior to those by manual design [8]. NAS has achieved
remarkable success in image recognition[9], semantic segmentation [20, 27],
and object detection [37, 10, 34].

The linchpin to NAS methodology is to build a massive network architecture
space, devise an efficient algorithm to explore this space, and uncover the
optimal architecture with a portfolio of training data and constraints [8].
Ordinarily, the cell to be searched is defined as a directed acyclic graph, as
illustrated in Figure 1(a). And NAS methods formulate the search problem as a
bi-level optimization problem, , model training as the lower-level
optimization problem and architectural search as the upper-level one [23].
Such an optimization problem has been shown to be computationally difficult in
previous studies due to the fact that the bi-level optimization solutions
dynamically interact with and affect each other. Extensive research has been
conducted to address the bi-level optimization problem, , reinforcement
learning (RL)-based methods [43, 44, 1, 41], evolutionary algorithms (EA) [31,
32], and gradient-based methods [23, 11, 24, 36].

In both RL-based and EA-based methodologies, the validation precision of a
multitude of network architecture candidates is expected in their search
strategies, which is computationally expensive and extremely time-consuming.
For example, RL-based methods are generally established as a Markov decision
process and have often used validation precision as a reward to optimize
architecture generators in each iteration. EA-based methods decide whether to
select or remove an architecture candidate from the population until the
architecture is chosen and evaluated. In these methods, thousands of GPU days
are devoted to finding the optimal architecture on the CIFAR-10 dataset, which
is inefficient and overwhelming. Recent works, epitomized by DARTS [23], have
erected a new lineage of methodologies which relax the discrete search space
to be continuous and differentiable. These methods employ gradient-based
strategies to optimize neural architectures, resulting in orders of magnitude
computations reductions in search costs, , from thousands of GPU days to a few
GPU hours. It opens a computationally feasible route for architecture search.
In DARTS [23], the entire computation graph composed of repeating computation
cells is learned together. At the end of the search phase, a pruning process
is applied to the edge connections and their associated operations within the
cells by keeping those with the highest architecture weights, which represent
the strength of edge connections.

While DARTS achieves impressive performance, these methods model the
architecture weights on each edge as statistically independent variables and
ignore the dependency among edges in DAG induced by their directed topological
connections, as shown in Figure 1(b). In addition, the hard pruning process
after search is heuristic and sub-optimal. To address these issues, we make
the first attempt to investigate the dependency by proposing a novel Inter-
layer Transition NAS (ITNAS) method. It adopts an architecture transition
learning paradigm that can explicitly model the dependency between the
architecture weights of predecessor and posterior edges. As shown in Figure
1(c), we divide the edges in each cell into two groups, , inner (denoted in
color) and outer group, according to whether or not their predecessor edges
are in the same cell. While the architecture weights of outer edges are
optimized independently, those of inner edges are derived sequentially based
on the weights of their predecessor edges and associated learnable transition
matrices in an attentive probability transition manner. Moreover, a novel
transition-induced iterative edge pruning method is proposed to prune edges
and generate the final architecture.

The contribution of this paper can be summarized as follows. First, we propose
a novel NAS method to model the inter-layer dependency for the first time.
With learnable transition matrices and attention vectors, the weights of inner
edges can be derived from the weights of their predecessor edges sequentially
rather than being learned independently. Second, we propose a new transition-
induced iterative edge pruning strategy that can progressively prune edges in
the context of the architecture transition paradigm. Third, extensive
experiments on five benchmarks demonstrate that the proposed ITNAS can search
better architectures than state-of-the-art methods, validating that modeling
the inter-layer dependency matters in NAS.

##  2 Related Work

Recently, the automatic design of neural architectures, a.k.a. Neural
Architecture Search (NAS), has attracted increasing attention. NAS methods are
commonly classified from three perspectives, , the search space, search
strategy, and performance estimation strategy [8]. The majority of earlier NAS
works typically choose the macro search space, where the entire network is
discovered [43]. For example, the pioneering NAS method relies on the
evolutionary algorithm, which simultaneously optimizes architectures and
network weights using vast computational resources [31, 32]. Later,
reinforcement learning-based approaches have been proposed for NAS [43, 44, 1,
41]. They formulate the design process of a neural network as a sequence of
actions and regard the model accuracy as a reward. These methods usually use a
recurrent network as a controller [44] to generate the model representation of
a sampled neural network designed for a given task.

In order to search in continuous domains, DARTS [23] proposes a continuous
relaxation of the architecture representation then leverages the efficient
gradient back-propagation to optimize the architecture weights and network
weights alternatively. As a result, DARTS is able to find good convolutional
architectures at a much less computational cost, making NAS broadly
accessible. Owed to the success of DARTS, several extensions have been
proposed. SNAS [36] optimizes the parameters of a joint distribution for the
search space in a cell. The authors propose a search gradient method that
optimizes the same objective as RL-based NAS but leads to more efficient
structural decisions. P-DARTS [2] attempts to overcome the depth gap issue
between the search phase and evaluation phase, which is accomplished by
increasing the depth of searched architectures gradually during the training
phase. PC-DARTS [38] leverages the redundancy in the network space and only
samples a subset of channels in super-net during search to reduce computation
cost. MileNAS [11] incorporates both bi-level and single-level optimization
schemes into a mixed-level framework. SGAS [18] chooses and prunes candidate
operations in a greedy fashion by dividing the search procedure into
subproblems. All these differential NAS methods model the architecture weights
of edges in a cell as independent variables and optimize them simultaneously
under a framework. However, some edges (, inner edges) within the cell are
actually not independent from others due to the directed topological
connections between them and their predecessor edges. In contrast to the
methods mentioned above, we make the first attempt to investigate the
dependency between edges in DAG and propose a novel NAS method by modeling the
inter-layer transition.

Most differential NAS methods [23, 38] adopt a hard pruning strategy to derive
the final architecture from the searched network. First, only the operation
with the largest architecture weight for each edge is retained. Then, for each
intermediate node, only the two edges with the largest edge importance are
retained. Recently, ASAP [29] adopts a progressive pruning strategy to prune
the operations whose architecture weights are below a threshold that is
updated gradually. However, these heuristic pruning strategies prune all weak
edges without considering the dependency between them and their descendent
edges induced by the directed topological connections, resulting in a sub-
optimal solution. By contrast, we propose a new transition-induced iterative
edge pruning strategy method that can progressively prune edges while updating
the edge importance of their descendent edges dynamically by leveraging the
inter-layer transition matrices.

##  3 Method

###  3.1 Preliminary

Architecture search in DARTS focuses on searching optimal repeating patterns
of structure in the network, which are called cells. This formulation is
inspired by the observation that the advanced manually designed deep neural
networks always consist of a few computational blocks that are stacked
sequentially to form the final structure of the network. No matter how many
blocks the network contains, the structures within each block are repetitive.
Thus, to learn one or two structures, which are represented as unit cells, is
sufficient to design an entire network.

Given an architecture space, we can represent the architecture of a cell as a
directed acyclic graph (DAG), , $\bm{\alpha=(I,E)}$, where
$\mathbf{I}=\\{\mathbf{I}^{(i)}\\}$ is a set of nodes that represents the
feature maps in the neural network and $E$ is an edge set. The directed edge
$(i,j)\in E$ denotes a layer that transforms the feature map from the node
$\mathbf{I}^{(i)}$ to $\mathbf{I}^{(j)}$. Since the DAG is composed of an
ordered sequence of nodes, where each edge can only point from a low-indexed
node to a high-indexed node, thereby we have $i<j$ for each edge. The output
of the intermediate node is computed as:

$\mathbf{I}^{(j)}=\sum_{i<j}\hat{o}^{(i,j)}(\mathbf{I}^{(i)}),$ (1)

where $\hat{o}^{(i,j)}\in\mathcal{O}$ is the selected operation (, convolution
or pooling) at edge $(i,j)$ from a operation set $\mathcal{O}$ containing $K$
operations. An architecture weight vector $\bm{\alpha^{(i,j)}}$ is introduced
to represent the probability distribution of the operation categorical choice
on the edge $(i,j)$. Thus, the differential NAS methods target to optimize the
$\bm{\alpha^{(i,j)}}$ to search the optimal operation $\hat{o}^{(i,j)}$. In
DARTS [23] and its variants [38], the optimal architecture is searched from
the discrete search space by relaxing the selection of the operations to a
continuous problem. By sampling all the possible operations $o$ for edge
$(i,j)$, Eq. (1) can be re-written as:

$\mathbf{I}^{(j)}=\sum_{i<j}\left(\sum_{o\in\mathcal{O}}\frac{\exp({\alpha_{o}^{(i,j)}})}{\sum_{o^{\prime}\in\mathcal{O}}\exp({\alpha_{o^{\prime}}^{(i,j)}})}\cdot
o(\mathbf{I}^{(i)})\right).$ (2)

Note that we apply the softmax to all architecture weights
$\alpha_{o}^{(i,j)}$ of operation $o$ to obtain the probability distribution.
The output of a node is the weighted sum of the outputs from all candidate
operations. Then, searching the optimal architectural parameters $\bm{\alpha}$
can be formulated as a bi-level optimization problem:

$\displaystyle\mathop{\text{min}}\limits_{\bm{\alpha}}\ \ \
\mathcal{L}_{val}(\bm{\omega^{*}(\alpha)},\bm{\alpha})$ s.t.
$\displaystyle\bm{\omega^{*}(\alpha)}=\mathop{\text{argmin}}\limits_{\bm{\omega}}\mathcal{L}_{train}(\bm{\omega},\bm{\alpha}),$
(3)

where $\bm{\omega}$ denotes the learnable weights of the supernet,
$\mathcal{L}_{train}$ and $\mathcal{L}_{val}$ denote the training loss and
validation loss, respectively. The search is carried out by optimizing the
supernet under the continuous operation relaxation.

###  3.2 Anneal Gumbel Softmax

As shown in Figure 1, the first two nodes $\mathbf{I}^{(0)}$ and
$\mathbf{I}^{(1)}$ are outputs from previous two cells, the intermediate nodes
from $\mathbf{I}^{(2)}$ to $\mathbf{I}^{(5)}$ are calculated according to Eq.
(2), and the node $\mathbf{I}^{(6)}=\bigcup_{i=2}^{5}\mathbf{I}^{(i)}$ is the
output of the current cell, which is the channel-wise concatenation of the
four intermediate nodes. The operation $o^{(i,j)}$ of predecessors edge is
selected according to the architecture weight $\bm{\alpha^{(i,j)}}$ based on a
specific rule. The most straightforward way is argmax that selects the one
with the largest architecture weight. After the edge operation selection, the
architecture weight vector $\bm{\alpha^{(i,j)}}$ can be represented by an one-
hot vector $\hat{\mathbf{Z}}^{(i,j)}$ and Eq. (2) can be re-written as:

$\displaystyle\mathbf{I}^{(j)}=\sum_{i<j}\sum_{k=0}^{K-1}\hat{\mathbf{Z}}^{(i,j)}_{k}\cdot
o_{k}(\mathbf{I}^{(i)}),$ s.t.
$\displaystyle\hat{\mathbf{Z}}^{(i,j)}=\text{one-
hot}\left(\underset{k}{\text{argmax}}\\{\alpha_{k}^{(i,j)}\\}\right).$ (4)

The typical way to represent the probability distribution of a discrete random
variable is to employ a categorical distribution. However, it will be
extremely challenging to back-propagate errors since the discrete
representation $\hat{\mathbf{Z}}^{(i,j)}$ is not differentiable. Thereby, the
reparameterization trick is used to relax the discrete architecture
distribution by using the concrete distribution [26], ,

$Z^{(i,j)}_{k}=\frac{\exp\left(\left(\log\alpha^{(i,j)}_{k}+G^{(i,j)}_{k}\right)/\tau\right)}{\sum_{k^{\prime}=0}^{K-1}\exp\left(\left(\log\alpha^{(i,j)}_{k^{\prime}}+G^{(i,j)}_{k^{\prime}}\right)/\tau\right)},$
(5)

where $Z^{(i,j)}_{k}$ is the $k$-th element of $\mathbf{Z}^{(i,j)}$, , the
softened one-hot vector for operation selection at edge $(i,j)$,
$G^{(i,j)}_{k}=-\log(-\log(U^{(i,j)}_{k}))$ is the $k$-th Gumbel random
variable, $U^{(i,j)}_{k}$ is a uniform random variable, and $\tau$ is the
temperature of the softmax, which is steadily annealed to be closed to zero
during the architecture searching phase. In this way, the non-differentiable
$\hat{\mathbf{Z}}^{(i,j)}$ in Eq. (4) can be refactored into a differentiable
$\mathbf{Z}^{(i,j)}$ in Eq. (5). And we can get the following equation as been
proven in [26]:

$p\left(\mathop{\text{lim}}\limits_{\tau\to
0}Z^{(i,j)}_{k}=1\right)=\frac{\alpha^{(i,j)}_{k}}{\sum_{k^{\prime}=0}^{K-1}\alpha^{(i,j)}_{k^{\prime}}},$
(6)

, the above relaxation is asymptotically unbiased. When
$\tau\rightarrow\infty$, each element in $\mathbf{Z}^{(i,j)}$ will be the
same. When $\tau\rightarrow 0$, $\mathbf{Z}^{(i,j)}$ will be an one-hot
vector.

The optimization of an architecture is typically based on its validation loss
in the bi-level optimization. To evaluate the performance of a sampled
architecture $\mathbf{Z}=\\{\mathbf{Z}^{(i,j)}\\}$, we build a target
supernetwork $\bm{\hat{y}}=f(\bm{x};\bm{\omega},\mathbf{Z})$ by stacking
several copies of the cells sequentially, where $\bm{x}$ is the network input
and $\bm{\hat{y}}$ is the network output. The loss of the target supernetwork
for classification tasks is measured by the commonly used negative log
likelihood:

$\mathcal{L}\left(\bm{D};\bm{\omega},\mathbf{Z}\right)=\mathbf{E}_{(\bm{x},\bm{y})\sim\bm{D}}\left[-\log\left(f\left(\bm{x};\bm{\omega},\mathbf{Z}\right)\right)\right],$
(7)

where $\bm{D}$ is the corresponding training, validation or testing dataset
depend on the context. With the validation loss
$\mathcal{L}(\bm{D}_{val};\bm{\omega},\mathbf{Z})$, the architecture weights
can be jointly optimized with the target network weights by solving the bi-
level optimization problem in Eq. (3). From Eq. (4) and Eq. (5), the gradient
of the loss $\mathcal{L}$ in Eq. (7) w.r.t. $\alpha^{(i,j)}_{k}$ is:

$\displaystyle\frac{\partial\mathcal{L}}{\partial\alpha^{(i,j)}_{k}}=\frac{\mathcal{L}}{\partial\mathbf{I}^{(j)}}o^{(i,j)}(\mathbf{I}^{(i)})(\delta(k-k_{0})-\mathbf{Z}^{(i,j)})Z^{(i,j)}_{k}\frac{1}{\tau\alpha^{(i,j)}_{k}}.$
(8)

We apply the Anneal Gumbel Softmax on the operation search of the outer edges
$E_{out}={\\{\bigcup_{i\geq 2}(0,i),\bigcup_{j\geq 2}(1,j)\\}}$ whose root
node is $\mathbf{I}^{(0)}$ or $\mathbf{I}^{(1)}$ generated by the previous
cells. These eight outer edges connect and transform the feature map from the
previous cells to the intermediate nodes in the current cell. Since they do
not contain any predecessor edge in the current cell, we treat their
architecture weights as independent variables. By contrast, we introduce a new
inter-layer transition learning paradigm for the inner edges
$E_{in}=\bigcup_{2\leq i\leq 4}(i,j)$ to derive their architecture weights
explicitly, since there is a clear dependency between an inner edge and its
predecessor edges.

###  3.3 Inter-layer Transition Learning

Figure 2: Illustration of the information flow among nodes (left) and edges
(right). $\mathbf{I}^{(3)}$ is generated from the mixed operations on its
predecessor nodes $\mathbf{I}^{(0)}$, $\mathbf{I}^{(1)}$, and
$\mathbf{I}^{(2)}$. The architecture weights of these edges are
$\mathbf{Z}^{(0,3)}$, $\mathbf{Z}^{(1,3)}$, and $\mathbf{Z}^{(2,3)}$. We model
the dependency between $\mathbf{Z}^{(3,4)}$ and its predecessor edges, ,
$\mathbf{Z}^{(0,3)}$, by a learnable transition matrix $\mathbf{P}^{(0,3,4)}$.

DARTS and its variants consider that the architecture weights of edges are
independent learnable variables. For example, as shown in Figure 2, edges
$(0,3),(1,3),(2,3)$ are the predecessor of the edge $(3,4)$. DARTS and its
variants optimize $\mathbf{Z}^{(3,4)}$ independently w.r.t.
$\mathbf{Z}^{(0,3)}$, $\mathbf{Z}^{(1,3)}$, and $\mathbf{Z}^{(2,3)}$. However,
the architecture weight $\mathbf{Z}^{(3,4)}$ is indeed influenced by the
architecture weights of its predecessor edges $\mathbf{Z}^{(0,3)}$,
$\mathbf{Z}^{(1,3)}$, and $\mathbf{Z}^{(2,3)}$, since the information flow is
explicitly defined by the directed topological connections between these edges
and influenced by their architecture weights. Thereby, the operation choice of
an edge matters for the operation choices of its descendent edges.

To account for this dependency, we introduce a novel inter-layer transition
learning paradigm for those inner edges. As shown in the right part of Figure
2, we define a learnable transition matrix to denote how the architecture
weight of a predecessor edge (, the “state” of operation choice) transits to
the architecture weight of its descendent edge. For example, the architecture
weight $\mathbf{Z}^{(i,j)}$ of edge $(i,j)$ can be deduced from the
architecture weight $\mathbf{Z}^{(m,i)}$ of its predecessor edge $(m,i)$ as
follows:

$\mathbf{Z}^{(i,j)}=\mathbf{P}^{(m,i,j)}\mathbf{Z}^{(m,i)}.$ (9)

Here, $\mathbf{P}^{(m,i,j)}$ is a learnable transition matrix subjected to:

$\displaystyle\sum_{t=0}^{K-1}p_{s,t}$ $\displaystyle=1,\forall s\in[0,K-1],$
$\displaystyle\text{s.t.}\ \ \ 0$ $\displaystyle\leq p_{s,t}\leq 1,$ (10)

where $p_{s,t}$ is the element in $\mathbf{P}^{(m,i,j)}$, denoting the
transition probability from the $s$-th operation on the edge $(m,i)$ to $t$-th
operation on the edge $(i,j)$.

Figure 3: Illustration of the Inter-layer Transition. The architecture weight
of an inner edge is deduced from those of its predecessor edges via the
learnable transition matrices and attention vector.

For an inner edge $(i,j)$, it may have multiple predecessor edges, which may
have different impact on $(i,j)$. Thereby, we adopt an attention mechanism to
adaptively weigh the importance of different transition matrices. As shown in
Figure 3, the architecture weight on an inner edge is deduced by a weighted
sum of those of its predecessor edges via the learnable transition matrices
and attention vectors. Mathematically, this process can be formulated as:

$\displaystyle\mathbf{Z}^{(i,j)}=\sum_{i<j}\sum_{m<i}\beta^{(m,i,j)}\mathbf{P}^{(m,i,j)}\mathbf{Z}^{(m,i)}.$
(11)

Here, $\beta^{(m,i,j)}$ denotes the attention score of the transition matrix
$\mathbf{P}^{(m,i,j)}$, which is subjected to:

$\displaystyle\sum_{m}\beta^{(m,i,j)}$ $\displaystyle=1,$
$\displaystyle\text{s.t.}\ \ \ \ 0\leq\beta^{(m,i,j)}$ $\displaystyle\leq 1.$
(12)

Note that both the transition matrix $\mathbf{P}^{(m,i,j)}$ and attention
score $\beta^{(m,i,j)}$ are learnable variables. In addition, since
$\mathbf{Z}^{(i,j)}$ in Eq. (11) is differentiable w.r.t.
$\mathbf{P}^{(m,i,j)}$, $\beta^{(m,i,j)}$, and $\mathbf{Z}^{(m,i)}$, thereby
$\mathbf{P}^{(m,i,j)}$ and $\beta^{(m,i,j)}$ can be optimized together with
the architecture weights of outer edges by minimizing the validation loss in
the bi-level optimization framework. In this way, the dependency between each
inner edge and its predecessor edges can be learned dynamically.

###  3.4 Transition-induced Iterative Edge Pruning

Algorithm 1 Transition-induced Iterative Edge Pruning.

1: Input: architecture weight $\mathbf{Z}^{(m,i)}$ of outer edges, transition
matrix $\mathbf{P}^{(m,i,j)}$, attention $\beta^{(m,i,j)}$

2: Output: the retained edges and operations after pruning $\bigcup_{2\leq
j\leq 5}(M_{j},Z_{j})$

3: Topology Initialization:

4: for edge $(i,j)\in E_{in}$ do

5:
$\mathbf{Z}^{(i,j)}=\sum_{i<j}\sum_{m<i}\beta^{(m,i,j)}\mathbf{P}^{(m,i,j)}\mathbf{Z}^{(m,i)}$

6: end for

7: Iterative Edge Pruning:

8: Initialize $M_{2}=\\{(0,2),(1,2)\\}$, $Z_{2}=\\{Z^{(0,2)},Z^{(1,2)}\\}$

9: for node $j\in$ intermediate nodes $\\{3,4,5\\}$ do

10: for $i=2,i<j,i++$ do

11: if $(i,j)\in E_{in}$ then

12: for $m=0,m<i,m++$ do

13: if $(m,i)\in M_{i}$ then

14: $\mathbf{Z}^{(m,i)}\leftarrow\text{one-
hot}(\text{argmax}_{k}\mathbf{Z}^{(m,i)}_{k}))$

15: end if

16: end for

17:
$\mathbf{Z}^{(i,j)}=\sum_{m,i}\beta^{(m,i,j)}\mathbf{P}^{(m,i,j)}\mathbf{Z}^{(m,i)}$

18: end if

19: end for

20: $\\{(i_{0},j),(i_{1},j)\\}\leftarrow$ edges with top-2
$\text{max}(\mathbf{Z}^{(i,j)})$

21: $M_{j}=\\{(i_{0},j),(i_{1},j)\\}$,
$Z_{j}=\\{Z^{(i_{0},j)},Z^{(i_{1},j)}\\}$

22: end for

After the search phase, we will get an over-parametered supernet. To generate
the final architecture, the redundant operation candidates (, edges) should be
eliminated, aside from the crucial ones in the supernet. The importance of the
edge $(i,j)$ in the searched cell is indicated by the largest architecture
operation weight of $\mathbf{Z}^{(i,j)}$. The commonly used edge pruning
strategy in DARTS and its variants is to retain two edges with the largest
edge importance between each pair of nodes. However, this heuristic pruning
strategy prunes all weak edges at once without considering the dependency
between them and their descendent edges induced by the directed topological
connections. To address this issue, we propose a new method named transition-
induced iterative edge pruning (TIEP).

At the initialization stage for edge pruning, we can calculate the
architecture weights of all outer edges $\mathbf{Z}^{(i,j)}$ according to Eq.
(11). Then, we prune the redundant edge iteratively and retain two edges for
each intermediate node in a cell. As shown in Figure 1, the node
$\mathbf{I}^{(2)}$ is the first intermediate node and only two edges ($(0,2)$
and $(1,2)$) point to it. Thereby, these two edges will be retained.
Accordingly, we select the operations on them and update the architecture
weights by selecting the maximum operation probability:

$\displaystyle\mathbf{Z}^{(i,j)}$ $\displaystyle\leftarrow\text{one-
hot}\left(k^{*}=\text{argmax}_{k}\mathbf{Z}^{(i,j)}_{k}\right),$
$\displaystyle o^{(i,j)}$ $\displaystyle\leftarrow o_{k^{*}},i<j,j=2.$ (13)

For node $\mathbf{I}^{(3)}$, there are three edges pointing to it, including
$(0,3),(1,3)$, and $(2,3)$. The edges $(0,3)$ and $(1,3)$ are outer edges
while the edge $(2,3)$ is an inner edge, whose architecture weight is
calculated according to Eq. (11), ,

$\mathbf{Z}^{(2,3)}=\beta^{(0,2,3)}\mathbf{P}^{(0,2,3)}\mathbf{Z}^{(0,2)}+\beta^{(1,2,3)}\mathbf{P}^{(1,2,3)}\mathbf{Z}^{(1,2)}.$
(14)

Note that $\mathbf{Z}^{(0,2)}$ and $\mathbf{Z}^{(1,2)}$ have been updated as
the one-hot vectors according to Eq.(13) in the previous pruning phase. The
attention score $\beta^{(0,2,3)}$ and $\beta^{(1,2,3)}$ and the transition
matrices $\mathbf{P}^{(0,2,3)}$ and $\mathbf{P}^{(1,2,3)}$ have been learned
during the architecture search phase. Then we prune the edge $(r,3)$ subject
to $r={\text{argmin}}_{i}({\text{max}}({Z}^{(i,3)}))$. Since the pruned edge
may be the predecessor edge of other inner edges, their architecture weights
should be updated to account for the absence of this predecessor edge
according to Eq. (11). For example, the transition matrix
$\mathbf{P}^{(r,3,j)}$ and attention score $\beta^{(r,3,j)}$ will become zero
due to the removal of edge $(r,3)$, if edge $(3,j)$ is a descendent inner edge
of the edge $(r,3)$. Then, we reweight the attention score from edge $(m,3)$
to edge $(3,j)$ by softmax and calculate the architecture weight of the edge
$(3,j)$ as follows:

$\displaystyle\mathbf{Z}^{(3,j)}$ $\displaystyle=\sum_{3<j}\sum_{m\neq
r}\beta^{(m,3,j)}\mathbf{P}^{(m,3,j)}\mathbf{Z}^{(m,3)},$
$\displaystyle\text{s.t.}\ \sum_{m\neq r}\beta^{(m,3,j)}=1.$ (15)

Since there are only two edges left for the node $\mathbf{I}^{(3)}$ after
pruning the edge $(r,3)$, they will be retained and their architecture weights
should be updated to the one-hot vectors similar to Eq. (13). In this way, we
can iteratively prune those redundant and unimportant edges while updating the
architecture weight $\mathbf{Z}$ at the same time. The complete process of the
transition-induced iterative edge pruning method is illustrated in Algorithm
1.

##  4 Experiments

We evaluated the performance of architectures searched by ITNAS on the
CIFAR-10 [17] and we further evaluated their generalization ability on other
popular image classification benchmarks, including the widely used ImageNet
[5], CIFAR-100 [17], CINIC-10 [4], and SVHN [28].

###  4.1 Results on CIFAR-10

####  4.1.1 Architecture Search and Evaluation

Following the common protocol in previous work, we used the CIFAR-10 dataset
for architecture search and evaluation. CIFAR-10 contains $50,000$ training
images together with $10,000$ testing images from $10$ classes. During the
search phase, we shuffled the training set and divide it into two parts of
equal size for training network weights and architecture weights,
respectively.

We set two types of cells in the search space, , normal cell and reduction
cell. The operations in the normal cells preserve the spatial dimensions,
while the operations on the edges connected to the input nodes in the
reduction cells have a stride of two, and the spatial solution of the output
features is halved. We set the number of candidate operations to $K=7$ to
include four convolutional operations (, $3\times 3$ separable convolution,
$5\times 5$ separable convolution, $3\times 3$ dilated separable convolution,
and $5\times 5$ dilated separable convolution), two pooling operations (,
$3\times 3$ average pooling and $3\times 3$ max pooling), and a special
$identity$ operation denoting the skip connection. The target supernet to be
searched is constructed by stacking six normal cells and two reduction cells.
The $3rd$ and $6th$ cells are reduction cells, while others are normal cells.
The initial number of feature channels is $16$ and doubled after each
reduction cell.

We trained the supernet on CIFAR-10 for $50$ epochs. To optimize the network
weight $\bm{\omega}$, we used the SGD optimizer. The initial learning rate is
$0.025$ , which is gradually decreased to $1e-3$ using cosine annealing. The
Softmax temperature $\tau$ in Gumbel-Softmax is initialized as $5$ and
linearly reduced to $0.5$. The architecture weights $\bm{\alpha}$ of outer
edges, the transition matrix $\mathbf{P}$, and attention vector $\bm{\beta}$
are jointly optimized using the Adam optimizer with a learning rate of $3e-4$
and a weight decay of $1e-3$.

For a fair comparison, every architecture shares the same hyper-parameters
adopted from DARTS for training. During the evaluation, $20$ searched cells
are stacked to form a larger network, which is then trained from scratch for
$600$ epochs with batch size of $96$. The learning rate is initialized as
$0.025$ and decreased to $0$ using cosine annealing. The probability of path
drop is $0.2$, and the weight of the auxiliary classifier is $0.4$. We used
the common pre-possessing and data augmentation techniques, , randomly
cropping, horizontally flipping, normalization, and cutout [6].

Architecture | Error (%) | Params (M) | Cost
---|---|---|---
DenseNet-BC [15] | 3.46 | 25.6 | -
NASNet-A [44] | 2.65 | 3.3 | 1800
AmoebaNet-A [31] | 3.34 | 3.2 | 3150
AmoebaNet-B [31] | 2.55 | 2.8 | 3150
Hier-Evolution [22] | 3.75 | 15.7 | 300
PNAS [21] | 3.41 | 3.2 | 225
ENAS [30] | 2.89 | 4.6 | 0.50
NAONet-WS [24] | 3.53 | 3.1 | 0.40
DARTS (1st) [23] | 3.00 | 3.3 | 0.4
DARTS (2nd) [23] | 2.76 | 3.3 | 1.0
SNAS [36] | 2.98 | 2.9 | 1.5
GDAS [7] | 2.93 | 3.4 | 0.21
BayesNAS [42] | 2.81 | 3.4 | 0.20
PVLL-NAS [19] | 2.70 | 3.3 | 0.20
MiLeNAS [11] | 2.51 | 3.9 | 0.30
SGAS (Cri.1) [18] | 2.66 | 3.7 | 0.25
SGAS (Cri.2) [18] | 2.52 | 4.1 | 0.25
ITNAS | 2.45 | 4.0 | 0.30
Table 1: Comparison between ITNAS and state-of-the-art methods on CIFAR-10.
The searching cost is measured as GPU days.

We compared the network searched by our ITNAS with both hand-crafted ones and
those searched by state-of-the-art (SOTA) NAS methods in Table 1. ITNAS
outperforms all previous methods with comparable amount of parameters. It
discovers an architecture with $4.0$M parameters in $0.3$ GPU day, which
achieves $2.45\%$ classification error on CIFAR-10. The number of architecture
parameters optimized in ITNAS is slightly more than DARTS (, only increasing
$1376$ parameters) by introducing the transition matrices and attention
vectors. The dependency of architecture weights between inner edges and their
predecessor edges are modeled effectively by these parameters. Intuitively,
the proposed inter-layer transition learning paradigm can regularize the
search space of inner edges and explicitly influence the optimization of the
operation choices on outer edges via error back-propagation according to Eq.
(11). Thereby, better architectures can be discovered.

Gumbel | Transition | TIEP | Error (%)
---|---|---|---
✔ | ✗ | ✗ | 2.72
✔ | ✔ | ✗ | 2.53
✔ | ✔ | ✔ | 2.45
Table 2: Ablation study of ITNAS on CIFAR-10, , Gumbel: Annealing Gumbel
Softmax, Transition: Inter-layer Transition Learning, TIEP: Transition-induced
Iterative Edge Pruning.

####  4.1.2 Ablation Study

. When we remove all these three modules, , Annealing Gumbel Softmax, Inter-
layer Transition Learning, and TIEP, our ITNAS will degenerate to DARTS, which
searches an architecture with $2.76\%$ classification error. As shown in Table
2, each of these modules contributes to the gains of ITNAS over DARTS, , the
classification error is reduced to $2.72\%$, $2.53\%$, and $2.45\%$, after
incorporating them one-by-one. The inter-layer transition learning brings more
gains than the other two modules. Nevertheless, the proposed pruning method
TIEP is also effective since it reduces the classification error further
considering that the score has been saturated on CIFAR-10. It is noteworthy
that this new pruning method is also induced by the inter-layer transition
learning paradigm. Thereby, these results confirm the value of modeling the
inter-layer dependency for architecture search.

###  4.2 Result on ImageNet

Figure 4: Illustration of the searched normal (the upper one) and reduction (the lower one) cells. Architecture | Top-1 (%) | Top-5 (%) | Params (M) | FLOPs (M) | Cost | Method
---|---|---|---|---|---|---
Inception-v1 [33] | 30.2 | 10.1 | 6.6 | 1448 | - | manual
MobileNet [13] | 29.4 | 10.5 | 4.2 | 569 | - | manual
ShuffleNet 2x (v1) [40] | 26.4 | 10.2 | 5 | 524 | - | manual
ShuffleNet 2x (v2) [25] | 25.1 | - | 5 | 591 | - | manual
NASNet-A [44] | 26.0 | 8.4 | 5.3 | 564 | 1800 | RL
NASNet-B [44] | 27.2 | 8.7 | 5.3 | 488 | 1800 | RL
NASNet-C [44] | 27.5 | 8.9 | 4.9 | 558 | 1800 | RL
AmoebaNet-A [31] | 25.5 | 8 | 5.1 | 555 | 3150 | EA
AmoebaNet-B [31] | 26.0 | 8.5 | 5.3 | 555 | 3150 | EA
AmoebaNet-C [31] | 24.3 | 7.6 | 6.4 | 570 | 3150 | EA
PNAS [21] | 25.8 | 8.1 | 5.1 | 588 | 225 | SMBO
DARTS(2nd) [23] | 26.7 | 8.7 | 4.7 | 574 | 4 | gradient
PDARTS [2] | 24.4 | 7.4 | 4.9 | 557 | 0.3 | gradient
SNAS [36] | 27.3 | 9.2 | 4.3 | 522 | 1.5 | gradient
ProxylessNAS [2] | 24.9 | 7.5 | 7.1 | 465 | 8.3 | gradient
GDAS [7] | 26.0 | 8.5 | 5.3 | 581 | 0.21 | gradient
PVLL-NAS [19] | 25.60 | 8.10 | 5.0 | 556 | 0.20 | gradient
CARS-I [39] | 24.80 | 7.50 | 5.1 | 591 | 0.40 | evolution
MiLeNAS [11] | 24.70 | 7.60 | 5.3 | 584 | 0.30 | gradient
SGAS (Cri.1) [18] | 24.41 | 7.29 | 5.3 | 579 | 0.25 | gradient
SGAS (Cri.2) [18] | 24.44 | 7.41 | 5.7 | 621 | 0.25 | gradient
ITNAS | 24.04 | 7.29 | 5.5 | 627 | 0.30 | gradient
Table 3: Comparison between ITNAS and state-of-the-art methods on ImageNet.

The normal and reduction cells obtained by ITNAS are illustrated in Figure 4.
We evaluated their transferability on the large-scale ImageNet dataset [5].
ImageNet contains 1.3 million training images and $50,000$ testing images from
$1,000$ categories. We followed the common single-crop evaluation protocol,
where the input size is $224\times 224$. Training hyper-parameters are adopted
from PC-DARTS [38]. The final network on ImageNet has $14$ layers, and the
initial number of feature channels is $48$. It is trained for $250$ epochs
with a batch size of $1024$. The SGD optimizer is used with an initial
learning rate of $0.5$, which is decreased to 0 linearly. The momentum and
weight decay are $0.9$ and $3e-5$. Learning rate warm-up is used in the first
$5$ epochs.

In Table 3, we compared ITNAS with SOTA NAS methods and manually designed
networks on ImageNet. It reveals that the networks searched by NAS are
superior to those manually designed. In addition, the differential NAS methods
significantly reduce the search cost compared with RL-based and EA-based ones.
Our model achieves a top-1 error of $24.04\%$, outperforming all the others.
It has comparable amount of parameters and computation complexity as other
SOTA NAS methods, , $5.5$M parameters and 627M FLOPs. It is searched by ITNAS
in $0.3$ GPU day, retaining the efficiency of gradient-based NAS methods. Note
that SGAS [18] chooses and prunes candidate operations in a greedy fashion and
obtains good results. Nevertheless, it also ignores the dependency between
edges. By contrast, our ITNAS benefits from modeling such dependency by
learning inter-layer transition and outperforms SGAS.

Architecture | CINIC-10 | CIFAR-100 | SVHN
---|---|---|---
Known SOTA | 8.6 [4] | 8.7 [16] | 1.02 [3]
NASNet [44] | 6.93 | 15.8 | 1.96
AmoebaNet-A [31] | 7.18 | 15.9 | 1.93
PNAS [21] | 7.03 | 15.9 | 1.83
SNAS [36] | 7.13 | 16.5 | 1.98
DARTS(1st) [23] | 7.05 | 15.8 | 1.94
DARTS(2nd) [23] | 6.88 | 15.7 | 1.85
ASAP [29] | 6.83 | 15.6 | 1.81
ITNAS | 6.72 | 15.5 | 1.78
Table 4: Transferability of the searched cells by ITNAS and SOTA NAS methods
on CNIC-10 [4], CIFAR-100 [17] and SVHN [28].

###  4.3 Results on other three benchmark datasets

We further evaluated the transferability of the searched cells by ITNAS on
other three benchmarks, , CNIC-10 [4], CIFAR-100 [17] and SVHN [28]. CINIC-10
is an extension of CIFAR-10 via the addition of downsampled ImageNet images.
It contains $270,000$ images of $10$ classes. CIFAR-100 contains $60,000$
images from $100$ classes, including $50,000$ images for training and $10,000$
images for test. The image size is $32\times 32$. SVHN contains $600,000$
$32\times 32$ real-world images of $10$ classes of digits and numbers from
natural scenes. For a fair comparison, we adopted the same network
configuration and hyper-parameters as ASAP [29], except from the searched
cells. The results of ITNAS and SOTA methods are summarized in Table 4,
showing that the cells searched by ITNAS can generalize well on other
classification tasks and outperforms other cells searched by SOTA methods. In
addition, ITNAS sets a new SOTA on CINIC-10, , $6.72\%$ classification error.

##  5 Conclusion

In this paper, we demonstrate that modeling the dependency between edges in
the DAG matters for NAS. We propose a novel inter-layer transition NAS method
(ITNAS) to explicitly model such dependency in an attentive probability
transition manner. In addition, a new iterative edge pruning method is derived
naturally from the inter-layer transition paradigm, which can iteratively
prune redundant searched edges and dynamically updating architecture weights.
ITNAS outperforms state-of-the-art NAS methods on five benchmarks. This study
makes the first attempt to model inter-layer dependency and shows promising
results, though more efforts are expected to advance NAS in this direction.
For example, other forms of dependency formulation can be attempted, ,
learning a nonlinear mapping by a neural module. Besides, pruning unimportant
edges guided by the dependency information during search is also worth
studying to improve the searching efficiency.

## References

  * [1] Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar.  Designing neural network architectures using reinforcement learning.  International Conference on Learning Representations, (ICLR), 2017\.
  * [2] Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian.  Progressive differentiable architecture search: Bridging the depth gap between search and evaluation.  In IEEE International Conference on Computer Vision, (ICCV), 2019\.
  * [3] Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le.  Autoaugment: Learning augmentation strategies from data.  In IEEE Conference on Computer Vision and Pattern Recognition, (CVPR), 2019.
  * [4] Luke N Darlow, Elliot J Crowley, Antreas Antoniou, and Amos J Storkey.  Cinic-10 is not imagenet or cifar-10.  arXiv preprint arXiv:1810.03505, 2018.
  * [5] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In IEEE Conference on Computer Vision and Pattern Recognition, (CVPR), 2009.
  * [6] Terrance DeVries and Graham W Taylor.  Improved regularization of convolutional neural networks with cutout.  In arXiv preprint arXiv:1708.04552, 2017.
  * [7] Xuanyi Dong and Yi Yang.  Searching for a robust neural architecture in four gpu hours.  In IEEE Conference on Computer Vision and Pattern Recognition, (CVPR), 2019.
  * [8] Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter.  Neural architecture search: A survey.  Journal of Machine Learning Research, (JMLR), 20:1–21, 2019.
  * [9] Jiemin Fang, Yuzhu Sun, Qian Zhang, Yuan Li, Wenyu Liu, and Xinggang Wang.  Densely connected search space for more flexible neural architecture search.  In IEEE Conference on Computer Vision and Pattern Recognition, (CVPR), 2020.
  * [10] Jianyuan Guo, Kai Han, Yunhe Wang, Chao Zhang, Zhaohui Yang, Han Wu, Xinghao Chen, and Chang Xu.  Hit-detector: Hierarchical trinity architecture search for object detection.  In IEEE Conference on Computer Vision and Pattern Recognition, (CVPR), 2020.
  * [11] Chaoyang He, Haishan Ye, Li Shen, and Tong Zhang.  Milenas: Efficient neural architecture search via mixed-level reformulation.  In IEEE Conference on Computer Vision and Pattern Recognition, (CVPR), 2020.
  * [12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In IEEE Conference on Computer Vision and Pattern Recognition, (CVPR), 2016.
  * [13] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam.  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  arXiv preprint arXiv:1704.04861, 2017.
  * [14] Jie Hu, Li Shen, and Gang Sun.  Squeeze-and-excitation networks.  In IEEE Conference on Computer Vision and Pattern Recognition, (CVPR), 2018.
  * [15] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger.  Densely connected convolutional networks.  In IEEE Conference on Computer Vision and Pattern Recognition, (CVPR), 2017.
  * [16] Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al.  Gpipe: Efficient training of giant neural networks using pipeline parallelism.  In Neural Information Processing Systems, (NIPS), 2019.
  * [17] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  Citeseer, Tech. Rep, 2009.
  * [18] Guohao Li, Guocheng Qian, Itzel C Delgadillo, Matthias Muller, Ali Thabet, and Bernard Ghanem.  Sgas: Sequential greedy architecture search.  In IEEE Conference on Computer Vision and Pattern Recognition, (CVPR), 2020.
  * [19] Yanxi Li, Minjing Dong, Yunhe Wang, and Chang Xu.  Neural architecture search in a proxy validation loss landscape.  In International Conference on Machine Learning, (ICML), 2020.
  * [20] Chenxi Liu, Liang-Chieh Chen, Florian Schroff, Hartwig Adam, Wei Hua, Alan L Yuille, and Li Fei-Fei.  Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation.  In IEEE Conference on Computer Vision and Pattern Recognition, (CVPR), 2019.
  * [21] Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy.  Progressive neural architecture search.  In European Conference on Computer Vision, (ECCV), 2018.
  * [22] Hanxiao Liu, Karen Simonyan, Oriol Vinyals, Chrisantha Fernando, and Koray Kavukcuoglu.  Hierarchical representations for efficient architecture search.  In International Conference on Learning Representations, (ICLR), 2018.
  * [23] Hanxiao Liu, Karen Simonyan, and Yiming Yang.  Darts: Differentiable architecture search.  In International Conference on Learning Representations, (ICLR), 2018.
  * [24] Renqian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tie-Yan Liu.  Neural architecture optimization.  In Neural Information Processing Systems, (NIPS), 2018.
  * [25] Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun.  Shufflenet v2: Practical guidelines for efficient cnn architecture design.  In European conference on computer vision, (ECCV), 2018.
  * [26] Chris J Maddison, Andriy Mnih, and Yee Whye Teh.  The concrete distribution: A continuous relaxation of discrete random variables.  International Conference on Learning Representations, (ICLR), 2017\.
  * [27] Vladimir Nekrasov, Hao Chen, Chunhua Shen, and Ian Reid.  Fast neural architecture search of compact semantic segmentation models via auxiliary cells.  In IEEE Conference on Computer Vision and Pattern Recognition, (CVPR), 2019.
  * [28] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng.  Reading digits in natural images with unsupervised feature learning.  In Neural Information Processing Systems Workshop, (NIPS Workshop), 2011.
  * [29] Asaf Noy, Niv Nayman, Tal Ridnik, Nadav Zamir, Sivan Doveh, Itamar Friedman, Raja Giryes, and Lihi Zelnik.  Asap: Architecture search, anneal and prune.  In International Conference on Artificial Intelligence and Statistics, (AISTATS), 2020.
  * [30] Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean.  Efficient neural architecture search via parameters sharing.  In International Conference on Machine Learning, (ICML), 2018.
  * [31] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le.  Regularized evolution for image classifier architecture search.  In AAAI Conference on Artificial Intelligence, (AAAI), 2019.
  * [32] Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc V Le, and Alexey Kurakin.  Large-scale evolution of image classifiers.  In International Conference on Machine Learning, (ICML), 2017.
  * [33] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich.  Going deeper with convolutions.  In IEEE Conference on Computer Vision and Pattern Recognition,(CVPR), 2015.
  * [34] Mingxing Tan, Ruoming Pang, and Quoc V Le.  Efficientdet: Scalable and efficient object detection.  In IEEE Conference on Computer Vision and Pattern Recognition, (CVPR), 2020.
  * [35] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He.  Aggregated residual transformations for deep neural networks.  In IEEE Conference on Computer Vision and Pattern Recognition, (CVPR), 2017.
  * [36] Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin.  Snas: stochastic neural architecture search.  In International Conference on Learning Representations, (ICLR), 2018.
  * [37] Hang Xu, Lewei Yao, Wei Zhang, Xiaodan Liang, and Zhenguo Li.  Auto-fpn: Automatic network architecture adaptation for object detection beyond classification.  In IEEE Conference on Computer Vision and Pattern Recognition, (CVPR), 2019.
  * [38] Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi, Qi Tian, and Hongkai Xiong.  Pc-darts: Partial channel connections for memory-efficient architecture search.  In International Conference on Learning Representations, (ICLR), 2019.
  * [39] Zhaohui Yang, Yunhe Wang, Xinghao Chen, Boxin Shi, Chao Xu, Chunjing Xu, Qi Tian, and Chang Xu.  Cars: Continuous evolution for efficient neural architecture search.  In IEEE Conference on Computer Vision and Pattern Recognition, (CVPR), 2020.
  * [40] Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun.  Shufflenet: An extremely efficient convolutional neural network for mobile devices.  In IEEE Conference on Computer Vision and Pattern Recognition, (CVPR), 2018.
  * [41] Zhao Zhong, Junjie Yan, Wei Wu, Jing Shao, and Cheng-Lin Liu.  Practical block-wise neural network architecture generation.  In IEEE Conference on Computer Vision and Pattern Recognition, (CVPR), 2018.
  * [42] H Zhou, M Yang, J Wang, and W Pan.  Bayesnas: A bayesian approach for neural architecture search.  In International Conference on Machine Learning, (ICML), 2019.
  * [43] Barret Zoph and Quoc V Le.  Neural architecture search with reinforcement learning.  International Conference on Learning Representations, (ICLR), 2017\.
  * [44] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le.  Learning transferable architectures for scalable image recognition.  In IEEE Conference on Computer Vision and Pattern Recognition, (CVPR), 2018.