# Quantum Phase Estimation and the Aharonov-Bohm effect

K. Splittorff  NNF Quantum Computing Programme, Niels Bohr Institute,
University of Copenhagen, Denmark, Blegdamsvej 17, DK-2100, Copenhagen Ø,
Denmark

###### Abstract

We consider the time evolution of a particle on a ring with a long solenoid
through and show that due to the Aharonov-Bohm effect this system naturally
makes up a physical implementation of the quantum phase estimation algorithm
for a $U(1)$ unitary operator. The implementation of the full quantum phase
estimation algorithm with a $U(N)$ unitary operator is realised through the
non-abelian Aharonov-Bohm effect. The implementation allows for a more
physically intuitive understanding of the algorithm. As an example we use the
path integral formulation of the implemented quantum phase estimation
algorithm to analyse the classical limit $\hbar\to 0$.

##  I Introduction

The arguably most important quantum algorithm is quantum phase estimation
Kitaev . It enters as the central building block in quantum algorithms, such
as for example Shors algorithm Shor , which provide an exponential speed-up as
compared to classical algorithms. The implementation of the quantum phase
estimation algorithm on a universal quantum computer, is made using a number
of qubits and operations thereon, see eg. the book NC  or the lecture notes SA
.

The elegant nature of the quantum phase estimation algorithm suggest an
alternative: Could there be a single quantum system with many quantum states
which naturally collaborate to implement the quantum phase estimation
algorithm? In other words a dedicated quantum computer device that can do
quantum phase estimation, but is not a universal quantum computer build out of
qubits. Here we point out that there is indeed a physical system which does
just that, namely a particle on a ring with a long solenoid through. We show
that due to the Aharonov-Bohm effect AB  the states of this system naturally
undergoes a time evolution which makes up a physical implementation of the
quantum phase estimation algorithm. The continuum of position states around
the ring make up the first register in the quantum phase estimation
algorithm111Note that we here use a single continuous variable rather than a
register of qubits. Universal quantum computing over continuous variables have
been studied in LB  and Grovers search algorithm Grover  has been implemented
using continuous variables in PBL . For a review of quantum information with
continuous variables, see BvanL . and the information about the phase of the
eigenvalue of the unitary operator from the second register is induced by the
flux through the solenoid. Initially we will restrict ourselves to the case
where there is an electromagnetic flux through the ring. This system will
implement the quantum phase estimation algorithm for a $U(1)$ unitary
operator. To implement the full quantum phase estimation algorithm with a
$U(N)$ unitary operator we then extend to the non-abelian Aharanov-Bohm effect
WuYang .

Having a physical system which implements a quantum algorithm can help us
understand the algorithm better222One example of this is Lloyd  where a
physical system which realises Grovers algorithm helped clarify the role of
entanglement in obtaining quantum advantage. An other example is CEMM  where a
Mach–Zehnder interferometer which solves the Deutsch’s problem Deutsch  helps
highlight the importance of quantum interference.. Here, we use the
implementation by means of the Aharanov-Bohm effect to analyse the classical
limit, $\hbar\to 0$, of the quantum phase estimation algorithm. To this end we
consider the path integral formulation for the implemented quantum phase
estimation algorithm and show that, the time it takes to complete the
algorithm diverges and the path which minimise the action does not dominate in
the classical limit $\hbar\to 0$. The physical implementation of the quantum
phase estimation algorithm by means of the Aharonov-Bohm effect hence does not
have a standard classical limit. While perhaps surprising at first, in
retrospect this is quite natural, as otherwise the classical system
corresponding to the classical limit of the implemented quantum algorithm
could potentially 1) solve the same task as the quantum algorithm and 2) be
simulated efficiently on a classical computer.

The paper is organised as follows. First we will recall the steps of the
quantum phase estimation algorithm as well as the Aharonov-Bohm effect for a
particle on a ring. Then we consider the time evolution of an initially
localised particle on the ring and establish the connection to the quantum
phase estimation algorithm. The full implementation of the quantum phase
estimation algorithm through the non-abelian Aharanov-Bohm effect is then
presented. Finally we discuss the Aharonov-Bohm effect in the path integral
formalism and the classical limit of the implemented quantum phase estimation
algorithm.

##  II Quantum Phase Estimation

To set the notation we start by briefly recalling the quantum phase estimation
(QPE) algorithm Kitaev . Given a unitary operator $U$ and an eigenstate
$|u\rangle$ of $U$ where333Note that $\phi_{u}\in[-\pi,\pi]$ whereas often in
the literature a factor of $2\pi$ is factorised such that $\phi_{u}$ is a
number between 0 and 1. Our convention is more natural when we turn to the
particle on the ring.

$\displaystyle U|u\rangle=e^{i\phi_{u}}|u\rangle\ ,$ (1)

the aim of the algorithm is to provide an estimate of the phase $\phi_{u}$.
The steps of the quantum phase estimation algorithm are listed in FIG. 1. Note
that the exponent of the phase factor, $e^{i\phi_{u}j}$, after the controlled
unitary operations have been applied, is linear in $j$ and in $\phi_{u}$.

Operation | State
---|---
Initial state | $|0\rangle|u\rangle$
QF-transform | $\frac{1}{\sqrt{2^{t}}}\sum_{j=0}^{2^{t}-1}|j\rangle|u\rangle$
Controlled U’s | $\frac{1}{\sqrt{2^{t}}}\sum_{j=0}^{2^{t}-1}e^{i\phi_{u}j}|j\rangle|u\rangle$
QF-transform-1 | $\frac{1}{2^{t}}\sum_{j,k=0}^{2^{t}-1}e^{i(\phi_{u}-2\pi\frac{k}{2^{t}})j}|k\rangle|u\rangle$
Measurement | $|k^{\prime}\rangle|u\rangle$
Figure 1:  The steps of the quantum phase estimation algorithm Kitaev . Note
the term in the exponent in the third line which is linear in $\phi_{u}$ and
$j$. The final measurement gives the estimate $2\pi k^{\prime}/2^{t}$ for the
phase.

##  III Intuition

Our aim is to find a single quantum system which naturally implements the
quantum phase estimation algorithm. Before we get into the details, we here
give a short (and hopefully) intuitive picture of why we are naturally lead to
consider a particle on a ring with a long solenoid through.

We seek a system in which there is a variable which when measured gives us the
phase we seek to estimate. This leads us to consider a particle on a ring,
where the position along the ring is described by a single angular variable
$\phi$. In this setting the quantum Fourier transforms employed in the QPE
algorithm transforms between the states with definite value of the angle
$\phi$ and the basis of states with definite angular momentum. Now as pointed
out above the phases induced by the controlled unitary operations in the QPE
algorithm must be linear in $j$ and $\phi_{u}$. Our aim is to implement these
phases through a time evolution of the particle on the ring. Hence we must
find a way to induce a Hamiltonian which is linear in the angular momentum
quantum number. Now the standard Hamiltonian for the particle on the ring is
quadratic in the angular momentum, due to the kinetic term. However, if we
introduce a minimally coupled gauge field it will naturally provide the
desired linear term in the angular momentum. To get such a minimal coupling to
a gauge field into play we introduce a long solenoid through the ring, such
that the vector potential associated with the magnetic flux through the
solenoid becomes the minimally coupled gauge field. Moreover the new term
which is linear in angular momentum will also automatically be linear in the
magnetic flux through the solenoid. Hence the information about the phase of
the eigenvalue of the unitary operator can be induced through the magnetic
flux. However, in the Hamilton operator we are still left with the original
term quadratic in the angular momenta, and there are no analogous terms in the
QPE algorithm. Fortunately, it is possible to nullify the effect of this
quadratic term by picking the evolution time for the system carefully. The
Aharanov-Bohm effect will in this way implement the QPE algorithm for a $U(1)$
unitary operator. To implement the QPE algorithm with a $U(N)$ unitary
operator we extend to the non-abelian Aharanov-Bohm effect where the particle
is minimally coupled to a $U(N)$ gauge field.

Hopefully, this intuitive picture can serve as a guide through the subsequent
sections.

##  IV Particle on a ring

For a start let us coinsider a particle on a ring of radius $r$, described by
the Hamiltonian

$\displaystyle
H^{0}=\frac{L_{z}^{2}}{2m_{q}r^{2}}=-\frac{\hbar^{2}}{2m_{q}r^{2}}\frac{\partial^{2}}{\partial\phi^{2}}\
.$ (2)

Here $\phi\in[-\pi,\pi]$ is the angular coordinate of the particle and

$\displaystyle L_{z}=-i\hbar\frac{\partial}{\partial\phi}$ (3)

is the $z$-component of the angular momentum of the particle about the center
of the ring. (Our coordinate system is such that the ring is in the
$(x,y)$-plane with the center at the origin.) We use $m_{q}$ to denote the
mass of the particle.

The normalised eigenfunctions of $H^{0}$ are

$\displaystyle\psi_{m}(\phi)=\frac{1}{\sqrt{2\pi}}e^{im\phi}$ (4)

with corresponding eigenenergies

$\displaystyle E^{0}_{m}=\frac{\hbar^{2}}{2m_{q}r^{2}}m^{2}\ ,$ (5)

where $m=\ldots,-1,0,1,\ldots$.

###  IV.1 Time evolution and quantum mechanical return time

Next let’s consider the time evolution of the state of the particle on the
ring. If at time $t=0$ the particle is in the state

$\displaystyle\Psi(\phi,t=0)=\sum_{m=-\infty}^{\infty}c_{m}\psi_{m}(\phi)$ (6)

then the state at time $t$ is obtained by tagging on the wiggle-factors,
$e^{-iE^{0}_{m}t/\hbar}$, that is

$\displaystyle\Psi(\phi,t)=\sum_{m=-\infty}^{\infty}c_{m}e^{-iE^{0}_{m}t/\hbar}\psi_{m}(\phi)\
.$ (7)

Note that since $E^{0}_{m}=\frac{\hbar^{2}}{2m_{q}r^{2}}m^{2}$ and $m$ is an
integer, all the wiggle-factors will equal 1 at the quantum mechanical return
time

$\displaystyle t_{R}=\frac{4\pi m_{q}r^{2}}{\hbar}\ .$ (8)

The state of the particle will therefore return to its initial state at
$t=t_{R}$ for any initial state of the particle on the ring.

###  IV.2 Time evolution of an initially localised particle.

For example, if the particle is fully localised at $\phi=0$ for time $t=0$

$\displaystyle\Psi(\phi,t=0)=\delta(\phi)\ ,$ (9)

then with time the state will become delocalised. However, at time $t=t_{R}$
the state of the system will again be localised at $\phi=0$, that is
$\Psi(\phi,t_{R})=\Psi(\phi,t=0)=\delta(\phi)$.

In order to see this explicitly, we represent the $\delta$-function as

$\displaystyle\Psi(\phi,t=0)=\delta(\phi)=\lim_{l\to\infty}\frac{1}{\sqrt{2\pi(2l+1)}}\sum_{m=-l}^{l}e^{im\phi}\
,$ (10)

and note that this can be re-written as

$\displaystyle\Psi(\phi,t=0)=\delta(\phi)=\lim_{l\to\infty}\frac{1}{\sqrt{2l+1}}\sum_{m=-l}^{l}\psi_{m}(\phi)\
.$ (11)

The initial state of the system is now expressed in terms of the stationary
states of $H$ and the time evolution of the state is obtained by tagging on
the wiggle-factors, which for $t=t_{R}$ all equal 1. So at $t=t_{R}$ the
particle initially localised at $\phi=0$ will again be localised at $\phi=0$.

##  V The Aharonov-Bohm effect for a particle on a ring

We now place a solenoid through the ring such that its symmetry axis is
identical to the $z$-axis and the radius of the solenoid is smaller than that
of the ring, see FIG. 2. In the limit where the length of the solenoid is much
greater than the radius of the ring the magnetic field at the ring is zero,
but newer the less the state of the particle on the ring will affected by the
solenoid, through the Aharanov-Bohm effect AB . The Aharanov-Bohm effect is
mediated by the vector potential $A$ which will be non-zero outside the
solenoid.

For a static flux $\Phi$ through the solenoid the Hamiltonian for the particle
on the ring becomes

$\displaystyle
H=\frac{1}{2m_{q}}\left(-i\frac{\hbar}{r}\frac{\partial}{\partial\phi}-q\frac{\Phi}{2\pi
r}\right)^{2}\ ,$ (12)

where $q$ is the charge of the particle. Note that the shift of the derivative
is realised through the minimal coupling of the vector potential of magnitude
$\Phi/(2\pi r)$, see e.g. GS .

Because the rotational symmetry around the $z$-axis remains, the
eigenfunctions (4) of $H$ are unaffected by the presence of the solenoid.
However, the corresponding eigenenergies now become

$\displaystyle
E_{m}=\frac{\hbar^{2}}{2m_{q}r^{2}}\left(m-\frac{q\Phi}{2\pi\hbar}\right)^{2}\
.$ (13)

$\Phi$$\phi=0$$t=0$$\phi=\phi_{\Phi}$$t=t_{R}$

Figure 2:  A particle on a ring with a long solenoid through. At time $t=0$
the particle is localised at $\phi=0$ and at $t=t_{R}$ it is localised at
$\phi_{\Phi}=-2\frac{q\Phi}{\hbar}$, where $\Phi$ is the flux through the
solenoid. The localised wave function is illustrated in red, see FIG. 4 for a
plot.

###  V.1 Time evolution and re-appearance at shifted angle

As a first step towards establishing the connection to the quantum phase
estimation algorithm let us now consider the time evolution of the state of
the particle on the ring in the presence of a flux through the solenoid.

Because the wiggle-factors are determined by the energies and these depend on
$\Phi$ cf. (13) the solenoid affects the time evolution of the state of the
particle on the ring. At non-zero $\Phi$ the eigenenergies, $E_{m}$, no longer
have the simple relationship of some number times an integer which depends on
$m$, hence the initial state of the system will no longer automatically
reappear at a later time. However, if we evolve the system for the quantum
mechanical return time (8) obtained for $\Phi=0$, the state of the system
returns to the exact initial form except that it is shifted by an angle
$\phi_{\Phi}$ determined by the flux $\Phi$.

To determine the shift $\phi_{\Phi}$ we again start the particle in an
arbitrary normalised state (6) and tag on the wiggle-factors,
$e^{-iE_{m}t/\hbar}$,

$\displaystyle\Psi_{\Phi}(\phi,t)$ $\displaystyle=$ $\displaystyle
e^{-i\frac{\hbar}{2m_{q}r^{2}}(\frac{q\Phi}{2\pi\hbar})^{2}t}\sum_{m=-\infty}^{\infty}c_{m}e^{-i\frac{\hbar}{2m_{q}r^{2}}m^{2}t}e^{i\frac{\hbar}{2m_{q}r^{2}}\frac{q\Phi}{\pi\hbar}mt}\psi_{m}(\phi)\
.$ (14)

Note in particular the exponent which is proportional to the flux $\Phi$ and
$m$. This is exactly the desired term which will allow us to implement the
quantum phase estimation algorithm by means of the Aharanov-Bohm effect.

Inserting the eigenstates $\psi_{m}(\phi)$ of (4) we get

$\displaystyle\Psi_{\Phi}(\phi,t)$ $\displaystyle=$
$\displaystyle\frac{1}{\sqrt{2\pi}}e^{-i\frac{\hbar}{2m_{q}r^{2}}(\frac{q\Phi}{2\pi\hbar})^{2}t}\sum_{m=-\infty}^{\infty}c_{m}e^{-i\frac{\hbar}{2m_{q}r^{2}}m^{2}t}e^{im(\phi+\frac{\hbar}{2m_{q}r^{2}}\frac{q\Phi}{\pi\hbar}t)}$
$\displaystyle=$ $\displaystyle
e^{-i\frac{\hbar}{2m_{q}r^{2}}(\frac{q\Phi}{2\pi\hbar})^{2}t}\Psi_{\Phi=0}(\phi+\frac{\hbar}{2m_{q}r^{2}}\frac{q\Phi}{\pi\hbar}t,t)\
.$

So up to an overall phase factor, which does not affect the probability
distribution, we have

$\displaystyle\Psi_{\Phi}(\phi,t)$ $\displaystyle=$
$\displaystyle\Psi_{\Phi=0}(\phi+\omega_{\Phi}t,t)\ ,$

where we introduced the angular velocity

$\displaystyle\omega_{\Phi}\equiv\frac{\hbar}{2m_{q}r^{2}}\frac{q\Phi}{\pi\hbar}\
.$ (16)

In words, the wave function of the particle on the ring evolves exactly the
same way as it did for zero flux through the solenoid, except that it now in
addition rotates with an angular velocity which depends linearly on the flux.
So just as for zero flux the initial state of the system will reappear at time
$t=t_{R}$ only now it is shifted around the ring by

$\displaystyle\phi_{\Phi}\equiv-\omega_{\Phi}t_{R}=-2\frac{q\Phi}{\hbar}.$
(17)

Operation | State | Physical picture
---|---|---
Initial state | $\delta(\phi)|u\rangle$ | localised at $\phi=0$
Express in $L_{z}$-basis | $\lim_{l\to\infty}\frac{1}{\sqrt{2l+1}}\sum_{m=-l}^{l}e^{im\phi}|u\rangle$ | delocalised in $L_{z}$-basis
Evolve for time $t_{R}$ | $\lim_{l\to\infty}\frac{1}{\sqrt{2l+1}}\sum_{m=-l}^{l}e^{i(\phi-\phi_{\Phi})m}|u\rangle$ | delocalised in $L_{z}$ but with phase input
Express in $\phi$ basis | $\delta(\phi-\phi_{\Phi})|u\rangle$ | localised at $\phi=\phi_{\Phi}$
Measurement of $\phi$ | $\delta(\phi-\phi_{\Phi})|u\rangle$ | result is $\phi_{\Phi}$
Figure 3:  The steps in the evolution of an initially localised particle on a
ring between time $t=0$ and $t=t_{R}$ due to the Aharonov-Bohm effect. Note
the exponent in the third line which is linear in $\phi_{\Phi}$ and $m$.
Compare to the steps of the QPE algorithm, FIG. 1.

##  VI The Aharonov-Bohm effect and the QPE algorithm

We are now in a position to make the connection between the Aharonov-Bohm
effect and the quantum phase estimation algorithm, all we need is to consider
an initially localised particle on the ring, $\Psi(\phi,t=0)=\delta(\phi)$. In
this case we have at time $t_{R}$ that the particle is fully localised at
$\phi_{\Phi}=-2\frac{q\Phi}{\hbar}$

$\displaystyle\Psi(\phi,t=t_{R})=\delta(\phi+2\frac{q\Phi}{\hbar})\ .$ (18)

A measurement of the particles position (as given by the angular coordinate)
at time $t=t_{R}$ hence will give $\phi=-2\frac{q\Phi}{\hbar}$.

The steps are summarised in FIG. 3. Note that they match those of the QPE
algorithm, compare FIG. 1.

The corresponding transcription table for the phase is

$\displaystyle\phi_{u}=\phi_{\Phi}=-2\frac{q\Phi}{\hbar}\ .$ (19)

For an illustration of the time development of the wave function of the
particle on the ring see Fig. 4.

(a) t=0

(b) $t=t_{R}/\sqrt{2}$

(c) $t=t_{R}$

Figure 4: The time evolution of the wave function of a particle on the ring
with a long solenoid through. (a) Initially the particle is localised at
$\phi=0$. (b) It then delocalises and finally (c) at $t=t_{R}$ it is localised
at $\phi=-2\frac{q\Phi}{\hbar}$, where $\Phi$ is the flux through the
solenoid. Red: real part of the wave function. Blue: imaginary part. For the
illustration we have chosen $l=100$ in (10).

To summarise this far: Because the Aharonov-Bohm effect induce a term linear
in respectively $m$ and $\phi_{\Phi}$, and because the effect of the
$m^{2}$-term is nullified at $t=t_{R}$, the state of the particle on the ring
at $t=t_{R}$ matches that obtained by the QPE algorithm, compare FIG. 1 and
FIG. 3. In the section below we to extend to a non-abelian flux. This will
allow us to fully implement the second register and the unitary operator $U$
of (1). Before we turn to this let’s first make one cross check of the
arguments this far which in addition will be very useful when we extend to the
non-abelian Aharanov-Bohm effect.

###  VI.1 Aharanov-Bohm phase directly

As stressed in WuYang  the Aharanov-Bohm effect is due to the phase factor

$\exp\left(i\frac{q}{\hbar}\oint A\cdot dx)\right)\ .$ (20)

Here $A$ is the vector potential and the integral is around a closed loop, in
our case around the ring. We will now show that this is completely consistent
with our analysis this far.

First recall that after the quantum Fourier transform the state of the
initially localised particle is expressed in terms of states,
$\psi_{m}(\phi)$, with well defined angular momentum $\hbar m$, see (10).
Second, if we equate $\hbar m=m_{q}r^{2}\omega$ we have $\omega t_{R}=2\pi
2m$, ie. the state $\psi_{m}(\phi)$ winds around the ring $2m$ times in time
$t_{R}$. Now, as the vector potential, $A$, at the ring has a constant
magnitude, $\Phi/2\pi r$, and points along the ring, the integral in (20)
gives $2\Phi m$ and we get the Aharanov-Bohm phase factor for the state
$\psi_{m}(\phi)$

$\exp\left(i2\frac{q}{\hbar}\Phi m\right)\ .$ (21)

This phase factor is fully consistent with (V.1) at time $t_{R}$ and thus we
reproduce the results of section VI.

We will use the formulation in terms of the Aharanov-Bohm phase factor as we
now extend to the non-abelian Aharanov-Bohm effect and the full QPE algorithm.

##  VII The non-abelian Aharanov-Bohm effect

This far we have considered the flux through the solenoid as an
electromagnetic source for the Aharonov-Bohm effect. As we now show the full
quantum phase estimation algorithm is implemented when we generalise to a non-
abelian flux corresponding to a $U(N)$ gauge field, that is if we consider the
non-abelian Aharonov-Bohm effect WuYang  for the particle on the ring444Note,
that our analysis this far corresponds to the case $U(1)$ and that the
particle on the ring in this setting acts as a quantum sensor of the magnetic
flux which works by means the quantum phase estimation algorithm..

In the non-abelian Aharonov-Bohm effect WuYang  the vector potential, $A$, is
replaced by a vector field, $A^{k}X_{k}$, from a non-abelian gauge group which
in our case is taken to be $U(N)$. Here $A^{k}$ for $k=1,\ldots,N^{2}$
parameterise the gauge fields, $X_{k}$ are the generators of $U(N)$ and
summation on $k$ is implied. The corresponding non-abelian Aharanov-Bohm phase
factor is WuYang

$U_{AB}=\exp\left(i\frac{1}{\hbar}\oint A^{k}X_{k}\cdot dx\right)\ .$ (22)

As the generators $X_{k}$ are Hermitian matrices the non-abelian Aharanov-Bohm
phase factor it self is a unitary $N$ by $N$ matrix and the state,
$\psi(\phi)$, of the particle on the ring now carries a corresponding
index555For the gauge group $SU(3)$ we can consider the strong interactions
ie. quantum chromodynamics, where $A^{k}$ can be thought of as the gluons and
the particle on the ring as a quark which carries a color index. (we can write
eg. $X^{ab}\psi^{b}(\phi)$ where summation over $b$ is implied). The degree of
freedom indicated by the matrix index will make up the second register in the
QPE algorithm.

Recall that in the QPE algorithm the state $|u\rangle$ of the second register
is picked as an eigenstate of $U$ such that the controlled application of $U$
to the state

$\displaystyle|j\rangle|u\rangle$ $\displaystyle cU\atop\longrightarrow$
$\displaystyle e^{i\phi_{u}j}|j\rangle|u\rangle\ .$ (23)

Likewise we pick $\psi_{m}^{a}(\phi)$ in the space indicated by the index $a$,
such that the non-abelian Aharanov-Bohm phase factor $U_{AB}$ for
$\psi_{m}^{a}(\phi)$ is diagonal

$\displaystyle\psi_{m}^{a}(\phi)$ $\displaystyle U_{AB}\atop\longrightarrow$
$\displaystyle e^{i\phi_{AB}m}\psi_{m}^{a}(\phi)\ .$ (24)

Note that the crucial factor of $m$ in the exponent comes automatically:
$\psi_{m}^{a}(\phi)$ completes $m$ times as many windings around the ring in
time $t_{R}$ as the component with $m=1$ and therefore the loop integral in
the non-abelian Aharanov-Bohm phase factor is completed a factor $m$ times
more. As we have seen above for the electromagnetic $U(1)$ case the phase
factor in (24) is exactly what we need in order to transfer the knowledge
about the eigenvalue from the second register to the first. This implies that
the non-abelian Aharanov-Bohm effect for the particle on the ring will make up
a physical implementation of the QPE algorithm for the unitary operator
$U_{AB}$ corresponding to the non-abelian Aharanov-Bohm phase factor for
$\psi^{a}_{m=1}$, with a flux determined by the gauge fields $A^{k}$. As we
are free to pick the gauge fields, we conclude that the non-abelian Aharonov-
Bohm effect for the particle on the ring makes up a physical implementation of
the full QPE algorithm.

The implementation of the QPE algorithm by a physical system gives a more
physically intuitive framework in which we may analyse the QPE algorithm666The
technological realisation of the quantum phase estimation algorithm by means
of the non-abelian Aharonov-Bohm effect could of course be extremely
challenging. Even if manage to set up a non-abelian flux and even if schemes
to perform error correction for continuous quantum variables do exist
Braunstein ; LS ; DKP , the resolution of the detector PBL  and the resources
needed to perform the final measurement of the phase must be considered Lloyd
.. As a first example we now consider the classical limit of the implemented
QPE algorithm.

##  VIII The classical limit of implemented QPE algorithm

Our aim is to examine the classical limit where $\hbar\to 0$ of the
implemented QPE algorithm, and hence it is natural to consider the system in
the path integral formalism777For discussions of quantum computing,
computational paths and path integrals, see e.g. BV ; DHHMNO ; PKS . feynman .
In this description one identifies the path for which the action is minimal
(extreme) and the standard scenario is that this path dominates the path
integral in the classical limit. The path which minimises the action, in this
case, becomes the classical path. Suppose that this standard scenario holds
true for our single quantum system which implements the QPE algorithm. Then
perhaps one could maintain the quantum advantage of the QPE algorithm by using
only the classical path and get a remarkable breakthrough in classical
computing. Alternatively something (almost) equally exiting must happen in our
single quantum system, namely that the path which minimises the action does
not dominate the path integral in the classical limit. As we show below it is
this latter option which is realised.

###  VIII.1 Path integral approach

Path integrals offer a way to analyse and compute the quantum mechanical
transition amplitude between the state of the system at two different times
feynman . To be more precise, if we know that the particle is in a state with
a certain well defined position at time $t=0$, the path integral allow us to
compute the quantum mechanical amplitude to detect the particle at a given
position at a later time $t$. The fundamental approach is to subdivide the
time interval $[0,t]$ into many smaller intervals and in each of these to
insert complete sets of respectively position and momentum states into the
transition amplitude. In each of these tiny time intervals the time evolution
operator acts for a short duration. This leads to the path integral in phase
space. Note that this is a path integral over both space and momenta and that
these variables are not linked by the equations of motion. Only if the action
is quadratic in momentum then the momentum space integral can be performed by
a Gaussian integration, resulting in the Feynman path integral representation
in configuration space, see eg. the lecture notes AP .

###  VIII.2 The path integral for the Aharonov-Bohm implemented QPE algorithm

To derive the path integral formulation for the QPE algorithm as implemented
by the particle on the ring, we follow the general approach outlined above and
split the interval $[0,t_{R}]$ into $N$ time steps of length $\delta
t=t_{R}/N$. Next we insert a unity in the form of projections on all of
angular momentum space, $1=\sum|m_{j}\rangle\langle m_{j}|$, before the time
interval, and angular space, $1=\int
d\phi_{l}\,|\phi_{l}\rangle\langle\phi_{l}|$, after the time interval888As the
second register is unaltered throughout the QPE algorithm we suppress the
state $|u\rangle$ below.

$\displaystyle\langle\phi|e^{-iHt_{R}/\hbar}|0\rangle$ $\displaystyle=$
$\displaystyle\sum_{m_{1}\ldots m_{N}}\int d\phi_{1}d\phi_{2}\ldots d\phi_{N}$
$\displaystyle\times\langle\phi|\ldots|\phi_{2}\rangle\langle\phi_{2}|e^{-iH\delta
t/\hbar}|m_{2}\rangle\langle m_{2}|\phi_{1}\rangle\langle\phi_{1}|e^{-iH\delta
t/\hbar}|m_{1}\rangle\langle m_{1}|0\rangle$

where $t_{R}=N\cdot\delta t$. The states $|m\rangle$ are eigenstates of $H$
with eigenenergies given by (13) and $\langle
m|\phi\rangle=\frac{1}{\sqrt{2\pi}}e^{-i\phi m}$. Inserting this we get

$\displaystyle\langle\phi|e^{-iHt_{R}/\hbar}|0\rangle$ $\displaystyle=$
$\displaystyle\sum_{m_{1}\ldots m_{N}}\int d\phi_{1}d\phi_{2}\ldots d\phi_{N}$
$\displaystyle\times\ldots
e^{i\phi_{2}\,m_{2}}e^{-i\frac{\hbar}{2m_{q}r^{2}}\left(m_{2}-\frac{q\Phi}{2\pi\hbar}\right)^{2}\delta
t}e^{-i\phi_{1}\,m_{2}}e^{i\phi_{1}m_{1}}e^{-i\frac{\hbar}{2m_{q}r^{2}}\left(m_{1}-\frac{q\Phi}{2\pi\hbar}\right)^{2}\delta
t}e^{-i0m_{1}}\ .$

This is the path integral in phase space for the QPE algorithm as implemented
by the particle on the ring in the presence of the solenoid. To get the path
integral in configuration space we now do the sum over $m_{j}$ by Poisson
summation (for simplicity we do not keep track of the $\phi_{j}$-independent
pre-exponential factors)

$\displaystyle\sum_{m_{j}=-\infty}^{\infty}\
e^{-i\frac{\hbar}{2m_{q}r^{2}}\left(m_{j}-\frac{q\Phi}{2\pi\hbar}\right)^{2}\delta
t}e^{i(\phi_{j}-\phi_{j-1})\,m_{j}}$ (27)
$\displaystyle\simeq\sum_{n=-\infty}^{\infty}e^{i\frac{1}{2}\frac{m_{q}r^{2}}{\hbar}\left(\phi_{j}-\phi_{j-1}+2\pi
n\right)^{2}\frac{1}{\delta t}}e^{i(\phi_{j}-\phi_{j-1}+2\pi
n)\frac{q\Phi}{2\pi\hbar}}\ .$

In the limit $\delta t\to 0$ the $n=0$ term dominates the sum. The path
integral in configuration space therefore becomes

$\displaystyle\langle\phi|e^{-iHt_{R}/\hbar}|0\rangle$ $\displaystyle\simeq$
$\displaystyle\int d\phi_{1}d\phi_{2}\ldots\
\prod_{j=1}^{N}e^{i\frac{1}{2}\frac{m_{q}r^{2}}{\hbar}\left(\phi_{j}-\phi_{j-1}\right)^{2}\frac{1}{\delta
t}}e^{i(\phi_{j}-\phi_{j-1})\frac{q\Phi}{2\pi\hbar}}$ $\displaystyle=$
$\displaystyle\int d\phi_{1}d\phi_{2}\ldots\
e^{i\sum_{j=1}^{N}\frac{1}{2}\frac{m_{q}r^{2}}{\hbar}\left(\phi_{j}-\phi_{j-1}\right)^{2}\frac{1}{\delta
t}+(\phi_{j}-\phi_{j-1})\frac{q\Phi}{2\pi\hbar}}$ $\displaystyle=$
$\displaystyle\int d\phi_{1}d\phi_{2}\ldots\ e^{iA(\\{\phi_{j}\\})/\hbar}\ ,$

where $\phi_{0}=0$ and $\phi_{N}=\phi$ and we introduced the action in
configuration space

$\displaystyle
A(\\{\phi_{j}\\})=\hbar\sum_{j=1}^{N}\frac{1}{2}\frac{m_{q}r^{2}}{\hbar}\left(\phi_{j}-\phi_{j-1}\right)^{2}\frac{1}{\delta
t}+(\phi_{j}-\phi_{j-1})\frac{q\Phi}{2\pi\hbar}\ .$ (29)

The path which minimise the action has a constant step size, ie.
$\phi_{j}-\phi_{j-1}$ is independent of $j$,

$\displaystyle\phi_{j}-\phi_{j-1}=-\frac{q\Phi}{2\pi\hbar}\frac{\hbar}{m_{q}r^{2}}\delta
t\ .$ (30)

Note that the slope

$\displaystyle\frac{\phi_{j}-\phi_{j-1}}{\delta
t}=-\frac{q\Phi}{2\pi}\frac{1}{m_{q}r^{2}}\ ,$ (31)

can be thought of as an angular velocity and matches the result of (16).

To convince ourselves that the path given by (30) indeed minimises the action
let us introduce a deviation from this path

$\displaystyle\phi_{j}-\phi_{j-1}=-\frac{q\Phi}{2\pi\hbar}\frac{\hbar}{m_{q}r^{2}}\delta
t+\epsilon_{j}\ .$ (32)

Note that $\epsilon_{j}$ are increments away from the path which minimise the
action. At step $j^{\prime}$ the deviation from this minimising path hence
will be $\sum_{j=1}^{j^{\prime}}\epsilon_{j}$ and to ensure that the path ends
in $\phi_{N}=\phi$ we have $\sum_{j=1}^{N}\epsilon_{j}=0$.

As a check that the linear terms in $\epsilon$ of the action vanish we insert
(32) in (29)

$\displaystyle A(\\{k_{j}\\})$ $\displaystyle=$
$\displaystyle\hbar\sum_{j=1}^{N}\frac{1}{2}\frac{m_{q}r^{2}}{\hbar}\left(-\frac{q\Phi}{2\pi\hbar}\frac{\hbar}{m_{q}r^{2}}\delta
t+\epsilon_{j}\right)^{2}\frac{1}{\delta
t}-\frac{q\Phi}{2\pi\hbar}\frac{\hbar}{m_{q}r^{2}}\delta
t+\epsilon_{j}\frac{q\Phi}{2\pi\hbar}$ $\displaystyle=$
$\displaystyle\hbar\sum_{j=1}^{N}\frac{1}{2}\left(-\frac{q\Phi}{2\pi\hbar}\right)^{2}\frac{\hbar}{m_{q}r^{2}}\delta
t-\frac{q\Phi}{2\pi\hbar}\frac{\hbar}{m_{q}r^{2}}\delta
t+\frac{1}{2}\frac{m_{q}r^{2}}{\hbar}\frac{1}{\delta t}\epsilon_{j}^{2}\ .$

We see that the linear terms in $\epsilon_{j}$ indeed vanish, demonstrating
that we have identified the path which minimises the action. Moreover, since
the action is quadratic note that this expression gives the total action.

Now let’s use that the time interval $[0,t_{R}]$ has been divided in to $N$
steps of length $\delta t$ and that $t_{R}=\frac{4\pi m_{q}r^{2}}{\hbar}$
(note that $t_{R}$ goes as $1/\hbar$), to rewrite this action as

$\displaystyle A(\\{k_{j}\\})$ $\displaystyle=$
$\displaystyle\hbar\sum_{j=1}^{N}\frac{1}{2}\left(-\frac{q\Phi}{2\pi\hbar}\right)^{2}\frac{4\pi}{t_{R}}\delta
t-\frac{q\Phi}{2\pi\hbar}\frac{4\pi}{t_{R}}\delta
t+\frac{1}{2}\frac{t_{R}}{4\pi}\frac{1}{\delta t}\epsilon_{j}^{2}$
$\displaystyle=$
$\displaystyle\hbar\frac{1}{2}\left(-\frac{q\Phi}{2\pi\hbar}\right)^{2}4\pi-\hbar\frac{q\Phi}{2\pi\hbar}4\pi+\hbar\frac{1}{8\pi}N\sum_{j=1}^{N}\epsilon_{j}^{2}\
.$

Our aim is examine, in the classical limit $\hbar\to 0$, how much a path can
deviate from the path which minimise the action before its contribution to the
amplitude is wiped out by destructive interference. The destructive
interference in the path integral sets in when the deviation introduced by the
$\epsilon_{j}$’s causes a change of $A/\hbar$ which becomes much greater than
1. Therefore as long as the sum over $\epsilon_{j}^{2}$ in the second line of
(VIII.2) is of order $1/N$, the corresponding path contributes to the
constructive interference in the path integral. If for instance
$\epsilon_{j}=\epsilon$ for $1\leq j\leq N/2$ and $\epsilon_{j}=-\epsilon$ for
$N/2+1\leq j\leq N$, then

$\displaystyle N\sum_{j=1}^{N}\epsilon_{j}^{2}=N^{2}\epsilon^{2}\ .$

For this to be of order $1$ all we need $\epsilon\sim 1/N$. This implies that
a path can get a distance of order 1 away from the path which minimises the
action, while the path still contributes constructively to the amplitude:
First take $N/2$ increments of size $\epsilon$ and subsequently take $N/2$
increments of size $-\epsilon$ back to the path which minimise the action.
This path deviates of order $\epsilon N/2$ from the path which minimise the
action and since $\epsilon$ can be of order $1/N$ this deviation is of order
1. As the maximal magnitude of $\phi$ is $\pi$ this implies that paths which
goes through all values of $\phi$ continue to contribute to the path integral
in configuration space even in the classical limit $\hbar\to 0$.

To summarise: One might have thought that since there is a physical quantum
system which realises the QPE algorithm, then it should also have a classical
limit where the path which minimise the action dominates the path integral. We
see however that in the classical limit $\hbar\to 0$ the return time
$t_{R}\to\infty$ and that paths with all values of $\phi$ contributes.

##  IX Summary and outlook

As stressed in EHKM  computers (classical or quantum) are physical systems and
computation is a physical process. Here we have identified a physical quantum
system, the particle on a ring with a flux through, for which the physical
process of time evolution, preforms the quantum computation of the quantum
phase estimation algorithm. This system is not a universal quantum computer,
but rather can be thought of as a dedicated quantum computer device which can
perform quantum phase estimation. The first register is made up by the
position (or after the quantum Fourier transform the angular momentum) states
of the particle on the ring. To realise the quantum phase estimation algorithm
for a $U(1)$ unitary operator it is sufficient to consider a standard
electromagnetic flux. The full quantum phase estimation algorithm is realised
with non-abelian flux: The unitary operator is encoded in non-abelian
Aharanov-Bohm phase factor, and the controlled unitary operations are realised
by the windings around the ring of the components of the particle with
definite angular momentum.

Having a physical system which realises the quantum phase estimation algorithm
gives a more intuitive way to think of the algorithm and allows us to ask
questions such as what happens in the classical limit $\hbar\to 0$. To address
this we have derived the path integral in configuration space for the quantum
phase estimation algorithm as implemented by the Aharonov-Bohm effect for a
particle on a ring. For this specific physical implementation, we have found
that the time it takes to complete the algorithm diverges as $1/\hbar$ and
that the path which minimises the action does not dominate in the classical
limit. Thus there is no simplification to a single classical path in the
classical limit.

The novel link between the quantum phase estimation algorithm and the non-
abelian Aharanov-Bohm effect established here potentially opens for a number
of new ways to analyse the quantum phase estimation algorithm. We hope that
further studies along these lines can help cast more light on the quantum
phase estimation algorithm and help advance the field of quantum computation.

###### Acknowledgements.

We thank Klaus Mølmer, Peter K. Jeppesen, Jan W. Thomsen, Svend K. Møller,
Frederik R. Klausen and Anna L.P. Bjerregaard for discussions. This work is
supported by the Novo Nordisk Foundation, Grant number NNF22SA0081175, NNF
Quantum Computing Programme.

## References

  * (1) A.Y. Kitaev, “Quantum measurements and the Abelian stabilizer problem,” Electr. Coll. Comput. Complex. 3, article no. 3 (1996), [arXiv:quant-ph/9511026 [quant-ph]].
  * (2) P.W. Shor ”Algorithms for quantum computation: Discrete logarithms and factoring”. Proceedings 35th Annual Symposium on Foundations of Computer Science. IEEE Comput. Soc. Press. pp. 124–134, (1994). P.W. Shor ”Polynomial-Time Algorithms for Prime Factorization and Discrete Logarithms on a Quantum Computer”. SIAM Journal on Computing. 26 (5): 1484–1509 (1997), arXiv:quant-ph/9508027.
  * (3) M.A. Nielsen and I.L. Chuang, ”Quantum Computation and Quantum Information”: 10th Anniversary Edition. Cambridge University Press (2010). doi:10.1017/CBO9780511976667.
  * (4) S. Aaronson, ”Introduction to Quantum Information Science” (2016) and ”Introduction to Quantum Information Science II” (2022), https://www.scottaaronson.com/.
  * (5) Y. Aharonov and D. Bohm, ”Significance of Electromagnetic Potentials in the Quantum Theory”, PhysRev.115.485, (1959).
  * (6) S. Lloyd and S.L. Braunstein, ”Quantum Computation over Continuous Variables”, Phys. Rev. Lett. 82, 1784 (1999).
  * (7) L.K. Grover ”A fast quantum mechanical algorithm for database search”. Proceedings of the twenty-eighth annual ACM symposium on Theory of computing - STOC ’96. Philadelphia, Pennsylvania, USA: Association for Computing Machinery. pp. 212–219. arXiv:quant-ph/9605043.
  * (8) A.K. Pati, S.L. Braunstein, S. Lloyd, ”Quantum searching with continuous variables”, arXiv:quant-ph/0002082.
  * (9) S.L. Braunstein and P. van Loock, ”Quantum information with continuous variables”, Rev. Mod. Phys. 77, 513 (2005).
  * (10) T.T. Wu and C.N. Yang, ”Concept of nonintegrable phase factors and global formulation of gauge fields ”, Phys. Rev. D 12, 3845 (1975).
  * (11) S. Lloyd, ”Quantum search without entanglement”, Phys. Rev. A 61, 010301(R), (1999).
  * (12) R. Cleve, A. Ekert, C. Macchiavello and M. Mosca ”Quantum algorithms revisited”, Proc. R. Soc. Lond. A.454: 339–354 (1998).
  * (13) D. Deutsch, ”Quantum theory, the Church–Turing principle and the universal quantum computer”, Proc. R. Soc. London A 400, 97 (1985).
  * (14) D.J. Griffiths and D.F. Schroeter (2018). ”Introduction to Quantum Mechanics” (3rd ed.). Cambridge: Cambridge University Press.
  * (15) S.L. Braunstein, ”Error Correction for Continuous Quantum Variables”, Phys. Rev. Lett. 80, 4084 (1998).
  * (16) S. Lloyd and J-J.E. Slotine, ”Analog Quantum Error Correction”, Phys. Rev. Lett. 80, 4088 (1998).
  * (17) D. Gottesman, A. Kitaev, and J. Preskill, ”Encoding a qubit in an oscillator” Phys. Rev. A 64, 012310 (2001).
  * (18) E. Bernstein and U. Vazirani, ”Quantum Complexity Theory”, SIAM J. Comput. 26, 1411 (1997).
  * (19) C.M. Dawson, H.L. Haselgrove, A.P. Hines, D. Mortimer, M.A. Nielsen, and T.J. Osborne, ”Quantum computing and polynomial equations over the finite field $Z_{2}$”, Quantum Inf. Comput. 5, 102 (2004), arXiv:quant-ph/0408129.
  * (20) M.D. Penney, D.E. Koh, and R.W. Spekkens, ”Quantum circuit dynamics via path integrals: Is there a classical action for discrete-time paths?”, New J. Phys. 19, 073006 (2017).
  * (21) R.P. Feynman and L.M. Brown (ed.) ”Feynman’s Thesis — A New Approach to Quantum Theory. World Scientific.” (2005) [1942/1948].
  * (22) J. Ambjørn and J.L. Petersen, ”Quantum Field Theory”, University of Copenhagen, The Niels Bohr Institute, 1998, https://nbi.ku.dk/bibliotek/noter-og-undervisningsmateriale-i-fysik/quantum-field-theory-by-jan-ambjoern-and-jens-lyng-petersen/
  * (23) A. Ekert, T. Hosgood, A. Kay, C. Macchiavello, ”Introduction to Quantum Information Science”, https://qubit.guide/.


* Jin et al. (2012) Guoliang Jin, Linhai Song, Xiaoming Shi, Joel Scherpelz, and Shan Lu. 2012.  Understanding and detecting real-world performance bugs.  _ACM SIGPLAN Notices_ 47, 6 (2012), 77–88.
  * Jorda et al. (2019) Marc Jorda, Pedro Valero-Lara, and Antonio J Pena. 2019\.  Performance evaluation of cudnn convolution algorithms on nvidia volta gpus.  _IEEE Access_ 7 (2019), 70461–70473.
  * Just et al. (2014) René Just, Darioush Jalali, and Michael D Ernst. 2014\.  Defects4J: A database of existing faults to enable controlled testing studies for Java programs. In _Proceedings of the 2014 international symposium on software testing and analysis_. 437–440.
  * Kang et al. (2022) Hong Jin Kang, Pattarakrit Rattanukul, Stefanus Agus Haryono, Truong Giang Nguyen, Chaiyong Ragkhitwetsagul, Corina Pasareanu, and David Lo. 2022.  SkipFuzz: Active Learning-based Input Selection for Fuzzing Deep Learning Libraries.  _arXiv preprint arXiv:2212.04038_ (2022).
  * Ken ([n. d.]) K Ken. [n. d.].  _Exclusive: surveillance footage of tesla crash on sf’s bay bridge hours after elon musk announces “self-driving” feature_.  https://theintercept.com/2023/01/10/tesla-crash-footage-autopilot/
  * Ketkar and Ketkar (2017) Nikhil Ketkar and Nikhil Ketkar. 2017.  Introduction to keras.  _Deep learning with python: a hands-on introduction_ (2017), 97–111.
  * Kim et al. (2021) Misoo Kim, Youngkyoung Kim, and Eunseok Lee. 2021.  Denchmark: A bug benchmark of deep learning-related software. In _2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR)_. IEEE, 540–544.
  * Kundu et al. (2021) Shamik Kundu, Suvadeep Banerjee, Arnab Raha, Suriyaprakash Natarajan, and Kanad Basu. 2021\.  Toward functional safety of systolic array-based deep learning hardware accelerators.  _IEEE Transactions on Very Large Scale Integration (VLSI) Systems_ 29, 3 (2021), 485–498.
  * Kundu et al. (2023) Shamik Kundu, Suvadeep Banerjee, Arnab Raha, Fei Su, Suriyaprakash Natarajan, and Kanad Basu. 2023.  Trouble-shooting at GAN Point: Improving Functional Safety in Deep Learning Accelerators.  _IEEE Trans. Comput._ (2023).
  * Kwon et al. (2023) Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023.  Efficient Memory Management for Large Language Model Serving with PagedAttention. In _Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles_.
  * Levental and Orlova (2020) Maksim Levental and Elena Orlova. 2020.  Comparing the costs of abstraction for dl frameworks.  _arXiv preprint arXiv:2012.07163_ (2020).
  * Li (2018) Hang Li. 2018.  Deep learning for natural language processing: advantages and challenges.  _National Science Review_ 5, 1 (2018), 24–26.
  * Li et al. (2022a) Junqiang Li, Senyi Li, Jiawei Wu, Long Luo, Yang Bai, and Hongfang Yu. 2022a.  MMOS: Multi-Staged Mutation Operator Scheduling for Deep Learning Library Testing. In _GLOBECOM 2022-2022 IEEE Global Communications Conference_. IEEE, 6103–6108.
  * Li et al. (2023b) Meiziniu Li, Jialun Cao, Yongqiang Tian, Tsz On Li, Ming Wen, and Shing-Chi Cheung. 2023b.  Comet: Coverage-guided model generation for deep learning library testing.  _ACM Transactions on Software Engineering and Methodology_ 32, 5 (2023), 1–34.
  * Li et al. (2020) Mingzhen Li, Yi Liu, Xiaoyan Liu, Qingxiao Sun, Xin You, Hailong Yang, Zhongzhi Luan, Lin Gan, Guangwen Yang, and Depei Qian. 2020\.  The deep learning compiler: A comprehensive survey.  _IEEE Transactions on Parallel and Distributed Systems_ 32, 3 (2020), 708–727.
  * Li et al. (2023a) Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. 2023a.  StarCoder: may the source be with you!  _arXiv preprint arXiv:2305.06161_ (2023).
  * Li et al. (2022b) Xiaoting Li, Xiao Liu, Lingwei Chen, Rupesh Prajapati, and Dinghao Wu. 2022b.  ALPHAPROG: reinforcement generation of valid programs for compiler fuzzing. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 36. 12559–12565.
  * Li et al. (2021) Yi Li, Shaohua Wang, and Tien Nguyen. 2021.  Fault localization with code coverage representation learning. In _2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)_. IEEE, 661–673.
  * Liang et al. (2018) Jie Liang, Mingzhe Wang, Yuanliang Chen, Yu Jiang, and Renwei Zhang. 2018.  Fuzz testing in practice: Obstacles and solutions. In _2018 IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER)_. IEEE, 562–566.
  * Lidbury et al. (2015) Christopher Lidbury, Andrei Lascu, Nathan Chong, and Alastair F Donaldson. 2015.  Many-core compiler fuzzing.  _ACM SIGPLAN Notices_ 50, 6 (2015), 65–76.
  * Liu et al. (2012) Bingchang Liu, Liang Shi, Zhuhua Cai, and Min Li. 2012\.  Software vulnerability discovery techniques: A survey. In _2012 fourth international conference on multimedia information networking and security_. IEEE, 152–156.
  * Liu et al. (2022a) Haiyi Liu, Shaoying Liu, Chenglong Wen, and W Eric Wong. 2022a.  TBEM: Testing-Based GPU-Memory Consumption Estimation for Deep Learning.  _IEEE Access_ 10 (2022), 39674–39680.
  * Liu et al. (2023a) Jiawei Liu, Jinkun Lin, Fabian Ruffy, Cheng Tan, Jinyang Li, Aurojit Panda, and Lingming Zhang. 2023a.  Nnsmith: Generating diverse and valid test cases for deep learning compilers. In _Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2_. 530–543.
  * Liu et al. (2023b) Jiawei Liu, Jinjun Peng, Yuyao Wang, and Lingming Zhang. 2023b.  Neuri: Diversifying dnn generation via inductive rule inference.  _arXiv preprint arXiv:2302.02261_ (2023).
  * Liu et al. (2022b) Jiawei Liu, Yuxiang Wei, Sen Yang, Yinlin Deng, and Lingming Zhang. 2022b.  Coverage-guided tensor compiler fuzzing with joint ir-pass mutation.  _Proceedings of the ACM on Programming Languages_ 6, OOPSLA1 (2022), 1–26.
  * Liu et al. (2018) Ling Liu, Yanzhao Wu, Wenqi Wei, Wenqi Cao, Semih Sahin, and Qi Zhang. 2018\.  Benchmarking deep learning frameworks: Design considerations, metrics and beyond. In _2018 IEEE 38th International Conference on Distributed Computing Systems (ICDCS)_. IEEE, 1258–1269.
  * Liu et al. (2022c) Zhihao Liu, Yang Zheng, Xiaoting Du, Zheng Hu, Wenjie Ding, Yanming Miao, and Zheng Zheng. 2022c.  Taxonomy of Aging-related Bugs in Deep Learning Libraries. In _2022 IEEE 33rd International Symposium on Software Reliability Engineering (ISSRE)_. IEEE, 423–434.
  * Ma et al. (2023b) Haoyang Ma, Qingchao Shen, Yongqiang Tian, Junjie Chen, and Shing-Chi Cheung. 2023b.  Fuzzing Deep Learning Compilers with HirGen. In _Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis_. 248–260.
  * Ma et al. (2023a) Xiangyue Ma, Xiaoting Du, Qing Cai, Yang Zheng, Jing Hu, and Zheng Zheng. 2023a.  A Survey on Testing of Deep Learning Frameworks (in Chinese).  _Journal of Software_ (2023).
  * Manès et al. (2019) Valentin JM Manès, HyungSeok Han, Choongwoo Han, Sang Kil Cha, Manuel Egele, Edward J Schwartz, and Maverick Woo. 2019\.  The art, science, and engineering of fuzzing: A survey.  _IEEE Transactions on Software Engineering_ 47, 11 (2019), 2312–2331.
  * Martínez-Fernández et al. (2022) Silverio Martínez-Fernández, Justus Bogner, Xavier Franch, Marc Oriol, Julien Siebert, Adam Trendowicz, Anna Maria Vollmer, and Stefan Wagner. 2022.  Software engineering for AI-based systems: a survey.  _ACM Transactions on Software Engineering and Methodology (TOSEM)_ 31, 2 (2022), 1–59.
  * McKeeman (1998) William M McKeeman. 1998\.  Differential testing for software.  _Digital Technical Journal_ 10, 1 (1998), 100–107.
  * Meng et al. (2022) Xiangxin Meng, Xu Wang, Hongyu Zhang, Hailong Sun, and Xudong Liu. 2022.  Improving fault localization and program repair with deep semantic features and transferred knowledge. In _Proceedings of the 44th International Conference on Software Engineering_. 1169–1180.
  * Microsoft (2023) Microsoft. 2023\.  _ONNX Github repository_.  https://github.com/onnx/onnx
  * Murphy et al. (2009) Christian Murphy, Kuang Shen, and Gail Kaiser. 2009.  Using JML runtime assertion checking to automate metamorphic testing in applications without test oracles. In _2009 International Conference on Software Testing Verification and Validation_. IEEE, 436–445.
  * Nazir et al. (2023) Zhumakhan Nazir, Vladislav Yarovenko, and Jurn-Gyu Park. 2023\.  Interpretable ML enhanced CNN Performance Analysis of cuBLAS, cuDNN and TensorRT. In _Proceedings of the 38th ACM/SIGAPP Symposium on Applied Computing_. 1260–1265.
  * Nijkamp et al. (2022) Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022\.  Codegen: An open large language model for code with multi-turn program synthesis.  _arXiv preprint arXiv:2203.13474_ (2022).
  * Nistor et al. (2015) Adrian Nistor, Po-Chun Chang, Cosmin Radoi, and Shan Lu. 2015\.  Caramel: Detecting and fixing performance problems that have non-intrusive fixes. In _2015 IEEE/ACM 37th IEEE International Conference on Software Engineering_ , Vol. 1. IEEE, 902–912.
  * Oehlert (2005) Peter Oehlert. 2005\.  Violating assumptions with fuzzing.  _IEEE Security & Privacy_ 3, 2 (2005), 58–62.
  * of Science and Policy ([n. d.]) Office of Science and Technology Policy. [n. d.].  _National Artificial Intelligence Initiative Act of 2020_.  https://www.ai.gov/wp-content/uploads/2023/04/National-Artificial-Intelligence-Initiative-Act-of-2020.pdf
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019\.  Pytorch: An imperative style, high-performance deep learning library.  _Advances in neural information processing systems_ 32 (2019).
  * Patterson et al. (2021) David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. 2021.  Carbon emissions and large neural network training.  _arXiv preprint arXiv:2104.10350_ (2021).
  * Pham et al. (2019) Hung Viet Pham, Thibaud Lutellier, Weizhen Qi, and Lin Tan. 2019\.  CRADLE: cross-backend validation to detect and localize bugs in deep learning libraries. In _2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE)_. IEEE, 1027–1038.
  * Prochnow and Yang (2022) Alexander Prochnow and Jinqiu Yang. 2022.  DiffWatch: watch out for the evolving differential testing in deep learning libraries. In _Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings_. 46–50.
  * Quan et al. (2022) Lili Quan, Qianyu Guo, Xiaofei Xie, Sen Chen, Xiaohong Li, and Yang Liu. 2022\.  Towards understanding the faults of javascript-based deep learning systems. In _Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering_. 1–13.
  * Rasley et al. (2020) Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020.  Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 3505–3506.
  * Ren et al. (2023) Luyao Ren, ZiHeng Wang, Yingfei Xiong, Li Zhang, Guoyue Jiang, and Tao Xie. 2023\.  Effective Random Test Generation for Deep Learning Compilers.  _arXiv preprint arXiv:2302.00842_ (2023).
  * Rotem et al. (2018) Nadav Rotem, Jordan Fix, Saleem Abdulrasool, Garret Catron, Summer Deng, Roman Dzhabarov, Nick Gibson, James Hegeman, Meghan Lele, Roman Levenstein, et al. 2018\.  Glow: Graph lowering compiler techniques for neural networks.  _arXiv preprint arXiv:1805.00907_ (2018).
  * Schumi and Sun (2022) Richard Schumi and Jun Sun. 2022.  ExAIS: executable AI semantics. In _Proceedings of the 44th International Conference on Software Engineering_. 859–870.
  * Segura et al. (2016) Sergio Segura, Gordon Fraser, Ana B Sanchez, and Antonio Ruiz-Cortés. 2016.  A survey on metamorphic testing.  _IEEE Transactions on software engineering_ 42, 9 (2016), 805–824.
  * Shen et al. (2021) Qingchao Shen, Haoyang Ma, Junjie Chen, Yongqiang Tian, Shing-Chi Cheung, and Xiang Chen. 2021\.  A comprehensive study of deep learning compiler bugs. In _Proceedings of the 29th ACM Joint meeting on european software engineering conference and symposium on the foundations of software engineering_. 968–980.
  * Shi et al. (2023) Jingyi Shi, Yang Xiao, Yuekang Li, Yeting Li, Dongsong Yu, Chendong Yu, Hui Su, Yufeng Chen, and Wei Huo. 2023.  ACETest: Automated Constraint Extraction for Testing Deep Learning Operators.  _arXiv preprint arXiv:2305.17914_ (2023).
  * staff ([n. d.]) ABC7.com staff. [n. d.].  _Uber gives up testing of self-driving cars in California in wake of fatal Arizona crash_.  https://abc7.com/self-driving-uber-crash-video-pedestrian-hit-by-car-autonomous-vehicles/3269690/
  * Sun et al. (2022) Bing Sun, Jun Sun, Long H Pham, and Jie Shi. 2022\.  Causality-based neural network repair. In _Proceedings of the 44th International Conference on Software Engineering_. 338–349.
  * Sun (2020) Y Sun. 2020.  _Tesla and PyTorch: PyTorch Developer Conference Highlights_.  https://medium.com/data-science-bootcamp/tesla-and-pytorch-pytorch-developer-conference-highlights-part-3ed36f2c9d5e
  * Tao et al. (2010) Qiuming Tao, Wei Wu, Chen Zhao, and Wuwei Shen. 2010\.  An automatic testing approach for compiler based on metamorphic testing technique. In _2010 Asia Pacific Software Engineering Conference_. IEEE, 270–279.
  * TensorFlow (2020) TensorFlow. 2020\.  _Learn how TensorFlow solves real, everyday machine learning problems_.  https://www.tensorflow.org/about/case-studies
  * Udeshi et al. (2018) Sakshi Udeshi, Pryanshu Arora, and Sudipta Chattopadhyay. 2018\.  Automated directed fairness testing. In _Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering_. 98–108.
  * Uezono et al. (2022) Takumi Uezono, Yi He, and Yanjing Li. 2022.  Achieving automotive safety requirements through functional in-field self-test for deep learning accelerators. In _2022 IEEE International Test Conference (ITC)_. IEEE, 465–473.
  * Wang et al. (2020a) Chaojin Wang, Jian Shen, Chunrong Fang, Xiangsheng Guan, Kaitao Wu, and Jiang Wang. 2020a.  Accuracy measurement of deep neural network accelerator via metamorphic testing. In _2020 IEEE International Conference On Artificial Intelligence Testing (AITest)_. IEEE, 55–61.
  * Wang et al. (2022) Jiannan Wang, Thibaud Lutellier, Shangshu Qian, Hung Viet Pham, and Lin Tan. 2022.  EAGLE: creating equivalent graphs to test deep learning libraries. In _Proceedings of the 44th International Conference on Software Engineering_. 798–810.
  * Wang et al. (2021) Jiyuan Wang, Qian Zhang, Guoqing Harry Xu, and Miryung Kim. 2021\.  Qdiff: Differential testing of quantum software stacks. In _2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE)_. IEEE, 692–704.
  * Wang et al. (2020b) Zan Wang, Ming Yan, Junjie Chen, Shuang Liu, and Dongdi Zhang. 2020b.  Deep learning library testing via effective model generation. In _Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_. 788–799.
  * Wei et al. (2022) Anjiang Wei, Yinlin Deng, Chenyuan Yang, and Lingming Zhang. 2022\.  Free lunch for testing: Fuzzing deep-learning libraries from open source. In _Proceedings of the 44th International Conference on Software Engineering_. 995–1007.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020\.  Transformers: State-of-the-Art Natural Language Processing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_. Association for Computational Linguistics, Online, 38–45.  https://www.aclweb.org/anthology/2020.emnlp-demos.6
  * Wu et al. (2023) Mingyuan Wu, Minghai Lu, Heming Cui, Junjie Chen, Yuqun Zhang, and Lingming Zhang. 2023\.  Jitfuzz: Coverage-guided fuzzing for jvm just-in-time compilers. In _2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)_. IEEE, 56–68.
  * Xiao et al. (2022) Dongwei Xiao, Zhibo Liu, Yuanyuan Yuan, Qi Pang, and Shuai Wang. 2022.  Metamorphic testing of deep learning compilers.  _Proceedings of the ACM on Measurement and Analysis of Computing Systems_ 6, 1 (2022), 1–28.
  * Xie et al. (2022) Danning Xie, Yitong Li, Mijung Kim, Hung Viet Pham, Lin Tan, Xiangyu Zhang, and Michael W Godfrey. 2022.  DocTer: documentation-guided fuzzing for testing deep learning API functions. In _Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis_. 176–188.
  * Xie et al. (2009) Xiaoyuan Xie, Joshua Ho, Christian Murphy, Gail Kaiser, Baowen Xu, and Tsong Yueh Chen. 2009\.  Application of metamorphic testing to supervised classifiers. In _2009 Ninth International Conference on Quality Software_. IEEE, 135–144.
  * Xie et al. (2011) Xiaoyuan Xie, Joshua WK Ho, Christian Murphy, Gail Kaiser, Baowen Xu, and Tsong Yueh Chen. 2011\.  Testing and validating machine learning classifiers by metamorphic testing.  _Journal of Systems and Software_ 84, 4 (2011), 544–558.
  * Yang et al. (2023) Chenyuan Yang, Yinlin Deng, Jiayi Yao, Yuxing Tu, Hanchi Li, and Lingming Zhang. 2023\.  Fuzzing automatic differentiation in deep-learning libraries.  _arXiv preprint arXiv:2302.04351_ (2023).
  * Yang et al. (2011) Xuejun Yang, Yang Chen, Eric Eide, and John Regehr. 2011\.  Finding and understanding bugs in C compilers. In _Proceedings of the 32nd ACM SIGPLAN conference on Programming language design and implementation_. 283–294.
  * Yang et al. (2022) Yilin Yang, Tianxing He, Zhilong Xia, and Yang Feng. 2022\.  A comprehensive empirical study on bug characteristics of deep learning frameworks.  _Information and Software Technology_ 151 (2022), 107004.
  * Zhang et al. (2020a) Jie M Zhang, Mark Harman, Lei Ma, and Yang Liu. 2020a.  Machine learning testing: Survey, landscapes and horizons.  _IEEE Transactions on Software Engineering_ 48, 1 (2020), 1–36.
  * Zhang et al. (2020b) Ru Zhang, Wencong Xiao, Hongyu Zhang, Yu Liu, Haoxiang Lin, and Mao Yang. 2020b.  An empirical study on program failures of deep learning jobs. In _Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering_. 1159–1170.
  * Zhang et al. (2021a) Xufan Zhang, Jiawei Liu, Ning Sun, Chunrong Fang, Jia Liu, Jiang Wang, Dong Chai, and Zhenyu Chen. 2021a.  Duo: Differential fuzzing for deep learning operators.  _IEEE Transactions on Reliability_ 70, 4 (2021), 1671–1685.
  * Zhang et al. (2022) Xiaoyu Zhang, Chao Shen, Chenhao Lin, Qian Li, Qian Wang, Qi Li, and Xiaohong Guan. 2022.  The Testing and Repairing Methods for Machine Learning Model Security.  _ACTA ELECTONICA SINICA_ 50, 12 (2022), 2884\.
  * Zhang et al. (2021b) Xufan Zhang, Ning Sun, Chunrong Fang, Jiawei Liu, Jia Liu, Dong Chai, Jiang Wang, and Zhenyu Chen. 2021b.  Predoo: precision testing of deep learning operators. In _Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis_. 400–412.
  * Zhang et al. (2019) Xufan Zhang, Yilin Yang, Yang Feng, and Zhenyu Chen. 2019\.  Software engineering practice in the development of deep learning applications.  _arXiv preprint arXiv:1910.03156_ (2019).
  * Zhang et al. (2021d) Xiaoyu Zhang, Juan Zhai, Shiqing Ma, and Chao Shen. 2021d.  Autotrainer: An automatic dnn training problem detection and repair system. In _2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)_. IEEE, 359–371.
  * Zhang et al. (2018) Yuhao Zhang, Yifan Chen, Shing-Chi Cheung, Yingfei Xiong, and Lu Zhang. 2018.  An empirical study on TensorFlow program bugs. In _Proceedings of the 27th ACM SIGSOFT international symposium on software testing and analysis_. 129–140.
  * Zhang et al. (2021c) Zhiyi Zhang, Pu Wang, Hongjing Guo, Ziyuan Wang, Yuqian Zhou, and Zhiqiu Huang. 2021c.  Deepbackground: Metamorphic testing for deep-learning-driven image recognition systems accompanied by background-relevance.  _Information and Software Technology_ 140 (2021), 106701.
  * Zhong (2022) Hao Zhong. 2022\.  Enriching compiler testing with real program from bug report. In _Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering_. 1–12.
  * Zou et al. (2023) Yinglong Zou, Haofeng Sun, Chunrong Fang, Jiawei Liu, and Zhenping Zhang. 2023.  Deep learning framework testing via hierarchical and heuristic model generation.  _Journal of Systems and Software_ 201 (2023), 111681.


11institutetext:  Sichuan University - Pittsburgh Institute, Chengdu, China
11email<EMAIL_ADDRESS><EMAIL_ADDRESS>zengyunpu<EMAIL_ADDRESS><EMAIL_ADDRESS>
# A CT-Based Airway Segmentation Using U2-net Trained by the Dice Loss
Function

Kunpeng Wang  Yuexi Dong  Yunpu Zeng  Zhichun Ye  Yangzhe Wang

###### Abstract

Airway segmentation from chest computed tomography scans has played an
essential role in the pulmonary disease diagnosis. The computer-assisted
airway segmentation based on the U-net architecture is more efficient and
accurate compared to the manual segmentation. In this paper we employ the
U2-net trained by the Dice loss function to model the airway tree from the
multi-site CT scans based on 299 training CT scans provided by the ATM’22. The
derived saliency probability map from the training is applied to the
validation data to extract the corresponding airway trees. The observation
shows that the majority of the segmented airway trees behave well from the
perspective of accuracy and connectivity. Refinements such as non-airway
regions labeling and removing are applied to certain obtained airway tree
models to display the largest component of the binary results.

###### Keywords:

U2-net Dice loss function Airway segmentation Multi-sites CT scans Saliency
probability map Non-airway region labeling

##  1 Introduction

The airway tree is one of the crucial structures of the human respiratory
system. The diagnosis and assessment of many pulmonary diseases, such as
asthma, bronchiectasis, and emphysema, depend on the careful examination on
the structure of the airway tree. The airway tree segmentation on CT scans
belongs to the fine-grained image segmentation. Manual segmentation is usually
time-consuming, inaccurate, and depends on the clinical experience of
physicians, due to which, computer aids are mostly used in clinical practice
to help complete the pulmonary airway segmentation .

The development of the convolutional neural networks permits the effective
automatic segmentation of airway trees. The U-net architecture as well as
different variants based on the original U-net have become the most up-to-date
and widely applied convolutional neural network in the filed of airway
segmentation [4, 5, 8].

The main challenge of the segmentation of the airway tree is the celebrated
input imbalance problem [3, 5] since the airway tree normally occupies less
pixel/voxel values in a CT scan. In order to further improve the segmentation
performance in terms of the extraction of not only the most obvious features
at the main trachea but also the small features at the bronchi, we employ the
U2-net, which is a two-level nested U-architecture proposed by Qin _et al._
[1], to segment the complex structure of the airway tree. Note in the original
architecture the U2-net is proposed to supervise by the binary cross entropy
function. However, due to the imbalance problem in the extraction, we train
the U2-net with the Dice loss function [9, 10], which has proven to outperform
at the imbalance problems. With the 299 annotated CT scans provided by ATM’22
Grand Challenge [2, 3, 7], we employ the U2-net supervised by the Dice loss
function to obtain the most accurate possible airway tree modeling based on
the optimization of the Dice Similarity Coefficient (DSC).

##  2 The Methodology

In this section, we first introduce the preprocessing for the provided
training data. Next, we discuss the details about the employed U2-net and the
corresponding loss function. The final part of the section revolves around the
generation of the binary masks for the validation data with the corresponding
labeling and post-treatment. The airway segmentation treatment is illustrated
in Fig. 1.

Figure 1: Flowchart of the employed U2-net segmentation.

###  2.1 Data Preprocessing

The ATM’22 provides 499 three dimensional CT scans from multi-sites, of which
299 are given for training, 50 for validation and the rest 150 will be used by
the challenge organizer for testing the submitted algorithms. The axial slices
of the provided CT scans are of the size $512\times 512$ in length and width
but with varying depth sizes.

Given a chest training CT image, we first normalize the voxel values to
prepare for faster computations in the following network training. The
original distribution of the CT voxel intensity is around $[-1024,3071]$. We
employ a CT-dependent normalization, that is, the intensity value is
normalized according to the mean and standard deviation values of each slice.
The normalized voxel intensity is rescaled to around $[0,2.5]$. Note that the
irrelevant dark parts with low intensities in a CT image, which normally enjoy
the constant standard deviation value, are further reset to a different value,
for example, $-9$ to distinguish from the rescaled array value $0$.

In our training, we adopt the original size $512\times 512$ of each slice with
the GPU memory permitting. However, the depth sizes are doubled, that is, each
input slice image is of $512\times 512\times 2$ to improve the performance
accuracy of the employed U2-net in the feature extraction. Although the
treatment may demand more GPU memory, we have shown, with the comparison to
the input images in the original depth size, that the double size treatment
indeed significantly increases the training accuracy.

###  2.2 Network Architecture

In the training process we employ the Un-net architecture [1] to obtain the
saliency probability function for the airway tree. The Un-net is a n-level
nested U-structure, where the exponent $n$ denotes the the level of the nested
U-structures. We adopt $n=2$ in the application from the practical perspective
by following [1]. The U2-net is built by two levels of U-structures. In
details, the top level U-structure consists of a six-stage encoder, a five-
stage decoder, and a saliency map fusion module and each stage among the
encoders and decoders contains the newly proposed Residual U-block
(abbreviated as RSU), which is the nested bottom level downsampling upsampling
encoder-decoder U-net. Each level of the downsampling/upsampling paths in the
corresponding RSU consists of a $3\times 3$ convolutional layer, a batch
normalization layer, a rectified linear unit (ReLu) activation function, and a
$2\times 2$ max-pooling/upsampling layer. The nested RSU allows each block to
use the multiscale features as residuals instead of the original features,
which makes the whole architecture maintain high resolution feature with a
reasonable requirement of the GPU memory and computing ability.

While the input data are processed by each RSU, the corresponding encoder and
decoder with a $3\times 3$ convolution layer, followed by a sigmoid function
in each layer of the upper level U-net will generate six side output saliency
probability maps
$S_{side}^{(6)},S_{side}^{(5)},S_{side}^{(4)},S_{side}^{(3)},S_{side}^{(2)},S_{side}^{(1)}$,
by applying a $1\times 1$ convolution layer and a sigmoid function to the
concatenation of which leads to a final saliency probability map $S_{fuse}$.
In addition to the economical cost of GPU memory and computing capacity, the
training of the U2-net does not rely on any backbones due to the
aforementioned nested design.

As mentioned in Subsection 2.1, the input is $512\times 512\times 2$
duplicated slices, hence, we change the channel number of the side probability
output at each level to 3 to improve the performance of the architecture in
extracting information and features at different levels and facilitate the
recognition of the trachea as well as the small peripheral bronchi.

###  2.3 The Loss Function

The training of the U2-net follows an end-to-end supervised manner, whose
training loss is defined as the weighted sum of the loss of the side and final
output saliency probability maps

$L=\sum_{n=1}^{N}w_{side}^{(n)}l_{side}^{(n)}+w_{fuse}l_{fuse},$ (1)

where $l_{side}^{(n)}$ ($n=6$, since there are 6 side output in total) is the
loss of the side output probability map, $l_{fuse}$ is the loss of the final
output, and $w_{side}$ as well as ${w_{fuse}}$ are the associated weights of
each loss. More specifically, we utilize the Dice loss function for each $l$
term (including both $l_{side}$ and $l_{fuse}$),

$l=1-\frac{2\sum\limits_{i=1}^{M}P_{G(i)}P_{S(i)}+\epsilon}{\sum\limits_{i=1}^{M}P_{G(i)}^{2}+\sum\limits_{i=1}^{M}P_{S(i)}^{2}+\epsilon},$
(2)

where $M$ denotes the total voxel index in a ground truth, $P_{G(i)}$ is the
numerical array value of the ground truth and $P_{S(i)}$ represents the
corresponding predicted value of the saliency probability map, $\epsilon$ is a
coefficient used to avoid division by zero and we take $\epsilon=1$. The
network is then trained via optimizing the overall loss $L$ by evaluating the
network output with the given ground truth.

###  2.4 Post-processing

We implement a threshold of 0.5 on the output map $S_{fuse}$ to obtain the
predicted mask for the 50 CT for validation. However, the predicted mask may
contain spurious non-airway regions. Notably, the airway and non-airway
regions can be generally topologically viewed in a Hausdorff space (see Fig. 2
for example). We then refine the segmented results by removing the
distinguishable large non-airway parts. Due to the separability, we assign
different labels to each separate region, for example, $1$ for the airway
region and $2$ for the non-airway region as shown in Fig. 3. Similar post-
processing methods can be found in [6]. Extracting the mask numerically
labelled with $1$ and applying to the validation CT scans, we finally obtained
the refined airway tree segmentation results.

Figure 2: Unrefined airway tree segmentation for the validation case
ATM_292_0000. Figure 3: Labeled airway tree segmentation for the validation
case ATM_292_0000.

##  3 Experiments and Results

The U2-net is implemented in the Pytorch framework and trained on NVIDIA
GeForce GTX 1080TI with 11GB memory as well as RTX 3090 with 24GB memory. The
ADAM (Adaptive Moment Estimation) and SGD (Stochastic Gradient Descent) are
chosen as the respective global and local optimizers. We employ the learning
rate of $0.001$ with momentum of $0.9$ and set the batch size to 2. Also, we
also choose $25\%$ of the input data of 299 CT scans as the test during the
training of the network. We examine the training loss and validation loss over
epochs. The training of one epoch takes about 10 hours. The training loss
stabilizes at $0.0525$ while the validation loss converges to $0.0555$ both
after around $12.5$ epochs as shown in Fig. 4. We terminate the training of
the network after 20 epochs and employ the thus-obtained parameters.

Figure 4: Training loss and validation loss over epochs.

Fig. 2 shows the unrefined result of the predicted mask applying to the
validation case ATM_292_0000. Following the post-processing method in
Subsection 2.4, the refined airway segmentation of the case ATM_292_0000 is
presented in Fig. 5.

Figure 5: Final airway tree segmentation for the validation case ATM_292_0000.

The saliency function obtained from the U2-net generally performs well on the
validation CT scans. The observation shows that though the end part of the
bronchial reflects certain weakness in connectivity, the main segmented airway
tree including the tracheal and main sub-branches behaves reasonably well in
terms of connectivity and accuracy.

##  4 Conclusion

We have obtained the airway tree segmentation of the given validation CT scans
using the U2-net. The experiment has indeed demonstrated that the U2-net can
generate a rather goodness of fit salicency probability function without any
_a priori_ backbones requiring a reasonable GPU memory and computational cost.
Our refined airway tree modeling shows good connectivity and accuracy by pure
observation. Yet, we still need the ground truth to numerically examine the
derived airway tree model. Also, the post-processing treatment can also
improve, for example, using the connectivity analysis given the ground truth
of the validation data.

## References

  * [1] Qin, X., Zhang, Z., Huang, C., Dehghan, M., Zaiane, O. R.,: U2-Net: Going deeper with nested U-structure for salient object detection. Pattern Recognition 106, 107404 (2020)
  * [2] Lo, P., Van Ginneken, B., Reinhardt, J.M., et al.: Extraction of airways from CT (EXACT’09). IEEE Transactions on Medical Imaging 31(11), 2093–2107 (2012)
  * [3] Zheng, H., Qin, Y., Gu, Y., et al.: Alleviating class-wise gradient imbalance for pulmonary airway segmentation. IEEE Transactions on Medical Imaging 40(9), 2452–2462 (2021)
  * [4] Zhang, M., Yu, X., Zhang, H.,et al.: FDA: Feature Decomposition and Aggregation for Robust Airway Segmentation//Domain Adaption and Representation Transfer, and Affordable Healthcare and AI for Resource Diverse Global Health, pp. 25–34. Springer, Cham (2021)
  * [5] Garcia-Uceda, A., Selvan, R., Saghir, Z. et al.: Automatic airway segmentation from computed tomography using robust and efficient 3-D convolutional neural networks. Sci Rep 11, 16001 (2021)
  * [6] Cheng, G., Wu, X., Xiang, W., et al.:Segmentation of the Airway Tree from Chest CT Using Tiny Atrous Concolutional Network. IEEE Access.9, 33583–33594 (2021)
  * [7] Yu, W., Zheng, H., Zhang, M.,et al.: BREAK: Bronchi Reconstruction by gEodesic transformation And sKeleton embedding. In: 2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI), pp. 1–5. IEEE (2022)
  * [8] Qin, Y., Chen, M., Zheng H., et al.: Airwaynet: A voxel-connectivity aware approach for accurate airway segmentation using convolutional neural networks. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 212–220. Springer, Cham (2019)
  * [9] Milletari, F., Navab, N., Ahmadi, S.: V-net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation. In: 2016 Fourth International Conference on 3D Vision (SDV), pp. 565–571. IEEE Computer Society, Los Alamitos (2016). 10.1109/3DV.2016.79
  * [10] Jadon, S.: A survey of loss functions for semantic segmentation. In: 2020 IEEE Conference on Computational Intelligence in Bioinformatics and Computational Biology (CIBCB). IEEE, (2020)


# Parameter-Free Average Attention Improves Convolutional Neural Network
Performance (Almost) Free of Charge

Nils Koerber
Center for Artificial Intelligence in Public Health Research
Robert Koch Institute
Berlin
Germany
<EMAIL_ADDRESS>

###### Abstract

Visual perception is driven by the focus on relevant aspects in the
surrounding world. To transfer this observation to the digital information
processing of computers, attention mechanisms have been introduced to
highlight salient image regions. Here, we introduce a parameter-free attention
mechanism called PfAAM, that is a simple yet effective module. It can be
plugged into various convolutional neural network architectures with a little
computational overhead and without affecting model size. PfAAM was tested on
multiple architectures for classification and segmentic segmentation leading
to improved model performance for all tested cases. This demonstrates its wide
applicability as a general easy-to-use module for computer vision tasks. The
implementation of PfAAM can be found on https://github.com/nkoerb/pfaam.

##  1 Introduction

Convolutional neural networks have demonstrated an impressive ability to solve
a broad range of computer vision tasks [1, 2, 3, 4]. Typically, a
convolutional neural network is built modular and the local receptive field is
increasing step-wise with network depth. According to that architecture, the
network captures hierarchical patterns based on input image representations
within the network. Increasing the representational power of neural networks
is of ongoing research interest to emphasize the most important features for a
given task. Previous work has shown improvements based on adaptations
regarding the inner connectivity [5, 6] or in utilizing an attention mechanism
that globally highlight relevant features [7, 8, 9].

However, existing attention mechanisms rely on trainable parameters, only
regard spatial or channel-wise attention, or introduce additional tunable
hyperparameters. Here, we introduce Parameter-free Average Attention Module
(PfAAM) which improves performance solely by basic mathematical operations and
is based on averaging input feature maps. PfAAM can be introduced into network
architectures of arbitrary form and do not add trainable parameters or non-
trainable hyperparameters and therefore do not change the overall size or
complexity of the network. Furthermore, we show that the network performance
of different architectures is enhanced by using PfAAM for both classification
and semantic segmentation tasks. While most previous work has focused on hand-
crafted modules with additional parameters, we present PfAAM as a lightweight
plug-and-play module that is compatible with most neural network
architectures, enhancing their performance and can be used for various
computer vision tasks.

##  2 Related Work

Figure 1: Parameter-free Average Attention Module (PfAAM).

In this section, we briefly review related model architectures and attention
mechanisms.

##### Network Architecture.

With the release of AlexNet [1] in 2012 for ImageNet LSVRC-2012 competition
[10], outperforming all other submissions by a large margin, convolutional
neural networks became state-of-the-art for computer vision tasks, having only
recently been challenged by vision transformer models [11], which, however,
require expensive pre-training on huge amounts of data. Several improvements
to the neural network architecture have been proposed to improve their
discriminative abilities. The application of deeper architectures [12, 13,
14], wider networks [15], increased connectivity [5], grouped convolutions
[16], depthwise convolution [16] or reduced computational requirements [17,
18] have lead to a plethora of potential architectures. Attempts have been
made to automatically search for the best network architecture reducing the
manual design of neural network architectures [19, 20]. However, building
blocks that boost network performance at low computational costs and can be
incorporated in any network architecture without the need for manual
adjustment or hyperparameter search remain of high value and motivated the
design of PfAAM.

##### Attention Modules.

Human perception is highly selective and filters information based on its
relevance for decision making. According to this, so called attention
mechnisms have been proposed for computer vision models. See [21] for a
comprehensive overview of attention mechanisms. Generally, attention
mechanisms can be split into methods related to channel attention [7, 9]
focusing on ’what is important’ in the image and spatial attention [22]
highlighting ’where is the important information’ in the image or a
combination of both [8, 23]. However, most of these attention modules are
implemented by adding learnable model parameters [7, 9, 22, 8] during training
increasing computational cost and model size, relate only on spatial or
channel attention, or depend on tunable hyperparameters [23]. As an extension
to existing modules, PfAAM captures channel and spatial attention without
adding parameters or hyperparameters and is simple by design, promoting a
self-reinforcing effect by averaging activations.

##  3 Parameter-free Average Attention Module

The general structure and computation of PfAAM is shown in figure 1. Consider
a feature map $F\in\mathbb{R}^{H\times W\times C}$ as intermediate input, the
PfAAM separates the input in a spatial attention part
$A_{sp}\in\mathbb{R}^{H\times W\times 1}$ by averaging the input along its
channels and a channel attention part $A_{ch}\in\mathbb{R}^{1\times 1\times
C}$ by calculating the average of each channel. The resulting attention maps
are then expanded along their reduced dimensions and recombined to depict the
attention to the most important parts of the feature input map. The final
recombined attention map uses a sigmoidal gating mechanism to enhance the
representational power of the input.

The overall process can be summarized as follows:

$F^{\prime}=\sigma(A_{sp}\otimes A_{ch})\otimes F,$ (1)

with $\otimes$ denoting the element-wise multiplication, $\sigma$ the sigmoid
function $\sigma(x)=\frac{1}{1+e^{-x}}$ and $F^{\prime}$ the output of PfAAM.
By element-wise multiplication of $A_{sp}$ and $A_{ch}$ the values are
broadcasted (copied) along the axes by which they were reduced during
averaging to regain their input sizes.

In contrast to attention modules that emphasize features by learned parameters
PfAAM is parameter-free and is solely highlighting features via averaging
spatially and along channels.

###  3.1 Spatial Attention Component

To emphasize the spatial attention in a feature map we produce a spatial
attention map. This is performed by averaging the spatial features along their
channels. As a result, the attention is focused on parts in the feature map
where a feature is detected. The average of each spatial element $x_{H\times
W}\in\mathbb{R}^{C}$ can be calculated as follows:

$A_{sp}(x_{H\times W})=\frac{1}{C}\sum\limits_{i=1}^{C}x_{H\times W}(i).$ (2)

By averaging along the channels, the dimension is reduced and produces a
spatial map where each element represents the average across channels. As a
consequence the spatial areas with high activations are emphasized while areas
with low activations are suppressed, thus highlighting positions with detected
features.

###  3.2 Channel Attention Component

In accordance to spatial attention the channel attention is calculated by
averaging along the spatial dimensions of the feature map. Formally, for each
channel $y_{C}\in\mathbb{R}^{H\times W}$ the average along its spatial
dimensions can be calculated as:

$A_{ch}(y_{C})=\frac{1}{H\times
W}\sum\limits_{i=1}^{H}\sum\limits_{j=1}^{W}y_{C}(i,j).$ (3)

By averaging along the spatial dimensions, channels are emphasized in which a
feature is detected and reducing the influence of channels with low
activations for their corresponding features.

###  3.3 Model Integration

The three-dimensional input to PfAAM is processed into a matrix with the same
dimensions, which can be used as an element-wise multiplier to amplify the
activations within the input. Because of its simplicity, the PfAAM block can
be easily integrated into different network architectures and positions,
allowing it to be used as a general building block for convolutional neural
networks. In the following section, the position of PfAAM within the residual
blocks and different pooling operations were analyzed, showing that averaging
is slightly preferred over maximization. Finally, PfAAM was successfully
tested in different network architectures for classification and segmentation,
showing an increase in performance.

##  4 Experiments

In this section we tested the optimal PfAAM setup and network integration in
an ablation study and performed experiments for classification and semantic
segmentation with different network architectures.

###  4.1 Ablation Study

To maximize the effect of the PfAAM block we tested different implementation
options. First, we tested averaging versus maximizing as channel and spatial
pooling operations within PfAAM to analyze their effect on the overall
performance. Max pooling enhance the effect of individual strong activation
whereas averaging increases areas with overall strong activation. The
performance of a baseline ResNet-164 [24] with PfAAM blocks added to each
residual block of the network were compared based on classification error.
Furthermore, we tested the influence of an additional Batch Normalization [25]
before each PfAAM. The accuracy of the network was tested using the CIFAR-10
dataset [26], that consists of 50k training and 10k test images with a size of
32 x 32 RGB pixels belonging to 10 different classes. See table 1 for
classification errors of the PfAAM implementations. In total, there is no
large difference in the resulting classification error and all PfAAM
implementations improve the performance compared to the baseline model, from
which we concluded that each of the implementations performs reasonably well.
Averaging without Batch Normalization showed the best performance overall,
which is why we continued further experiments with this configuration, unless
stated otherwise.

Table 1: Comparison of different PfAAM implementations CIFAR-10 using averaging or maximizing and an additional Batch Normalization (BN). The lowest classification error is shown in bold. | error (%)
---|---
ResNet-164 [24] | 5.46
ResNet-164+PfAAM(max) | 4.79
ResNet-164+PfAAM(avg) | 4.76
ResNet-164+BN+PfAAM(max) | 4.94
ResNet-164+BN+PfAAM(avg) | 4.86

###  4.2 Experiments

To analyze the effect of PfAAM on neural network performance we used baseline
architectures for classification and semantic segmentation and compared the
performance of the regular architecture to the same architecture but with
additional PfAAM blocks incorporated.

####  4.2.1 Image Classification

To investigate the effect of PfAAM in a classification task, we conducted
experiments using CIFAR-10 and CIFAR-100 as benchmark. Both data sets have the
same size, but are divided into 10 and 100 classes, respectively. As model
architectures Residual Networks [24] and Wide Residual Networks [15] with
varying depth and width were used to cover basic architectures from shallow to
deep and thin to wide. Results in table 2 show an reduction of the
classification error for all tested architectures with integrated PfAAM. For
deeper architectures the effect of PfAAM is larger showing a reduction of the
error rate of over 12% for ResNet-110 and ResNet-164 on CIFAR-10, thus
ResNet-110+PfAAM almost matches the performance of the regular ResNet-164
which has 40% more trainable parameters. For wider but shallower architectures
with fewer residual blocks, the effect of PfAAM is smaller (1.4% reduction for
WRN-16-8 on CIFAR-10), suggesting that the effect scales with the number of
PfAAM units per network. Since PfAAM does not introduce additional learnable
parameters, it generally increases the performance of the network in image
classification by improving the utilization of the existing parameters.

Table 2: Classification error (%) on CIFAR-10 and CIFAR-100. The lowest error
per model architecture and data set is shown in bold. CIFAR-10
---
| # params | original | +PfAAM
ResNet-110 [24] | 1.2M | 6.37 | 5.57
ResNet-164 [24] | 1.7M | 5.46 | 4.76
WRN-28-2 [15] | 1.5M | 5.73 | 5.29
WRN-16-8 [15] | 11M | 4.27 | 4.21
CIFAR-100
| # params | original | +PfAAM
ResNet-110 [24] | 1.2M | 26.88 | 24.22
ResNet-164 [24] | 1.7M | 24.33 | 23.05
WRN-28-2 [15] | 1.5M | 26.69 | 25.38
WRN-16-8 [15] | 11M | 20.43 | 20.33

####  4.2.2 Semantic Segmentation

To test PfAAM for semantic segmentation, we used the PASCAL VOC 2012
segmentation dataset [27] consisting of 1464 training and 1449 validation
images of 20 categories and an additional background class. Following previous
work [28, 29, 2], we used the extended dataset with annotations from [30]
resulting in 10582 training images. We trained a U-Net [31] and a Feature
Pyramid Network (FPN) [32] on the training images and compared the results to
the same architectures with added PfAAM. Each model used a ResNet-50 [24] as
encoder-backbone, which was pre-trained on the ImageNet dataset [10]. The
results in table 3 show the mean intersection over union (mIoU) on the
validation images. Both models show increased performance when trained with
PfAAM increasing the mIoU by 7.7% for U-Net and 5.3% for FPN, respectively.
The averaged validation mIoU for U-Net with and without PfAAM during training
are depicted in figure 3 showing a clear improvement for the PfAAM-model.
Similar to classification, the introduction of the PfAAM in the model
architecture improves the performance, underlining its general applicability
as neural network building block enhancing model performance.

Table 3: Segmentation results (mIoU, %) on PASCAL VOC 2012 validation set. Best results per model architecture are shown in bold. | original | +PfAAM
---|---|---
U-Net | 55.7 | 60.3
FPN | 56.5 | 59.7
Figure 2: Validation mIoU during training for a regular U-Net and the same
architecture extended by PfAAM, shown is $mean\pm std$.

####  4.2.3 Implementation Details

For CIFAR training we followed the established standard training procedure
used by the original publications [24, 15]. Each 32x32 image or its its
horizontally mirrored version was padded by 4 pixels and randomly cropped back
to 32x32 pixel. The neural networks were trained for 200 epochs by optimizing
the cross-entropy loss using SGD (stochastic gradient decent) with a momentum
of 0.9, a weight decay of 0.0005 a mini-batch size of 128 and an initial
learning rate of 0.1. The learning rate was step-wise decreased after 60
epochs, 120 epochs and 160 by a factor of 0.2.

For semantic segmentation using the PASCAL VOC dataset, the training images
were randomly horizontally flipped and scaled by a factor of 0.5 to 2 for each
axis, from which random 224x224 patches were cut and fed into the neural
network. Optimization was performed using SGD with a momentum of 0.9 and a
constant learning rate of 0.0001 for 200 epochs, optimizing the cross-entropy
loss function excluding pixels labeled as void.

Unless stated otherwise, all results are reported as the median over 5 runs.

##  5 Conclusion

In this work, we present a novel attention mechanism PfAAM based on
highlighting areas of high activation. When PfAAM is used in different network
architectures for classification and semantic segmentation, the performance
increases for all tested architectures, while the network size remained
unchanged and the computational cost is low. Even though PfAAM does not add
additional trainable parameters to the network and does not rely on other
theoretical considerations, its positive effect is surprisingly robust,
suggesting that it leads to a self-focusing effect on relevant features. In
summary, PfAAM provides a simple novel building block that might be considered
for future neural network design in computer vision tasks.

## References

  * [1] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In Advances in neural information processing systems, pages 1097–1105, 2012.
  * [2] Jonathan Long, Evan Shelhamer, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431–3440, 2015.
  * [3] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi.  You only look once: Unified, real-time object detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 779–788, 2016.
  * [4] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017.
  * [5] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger.  Densely connected convolutional networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700–4708, 2017.
  * [6] François Chollet.  Xception: Deep learning with depthwise separable convolutions.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1251–1258, 2017.
  * [7] Jie Hu, Li Shen, and Gang Sun.  Squeeze-and-excitation networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132–7141, 2018.
  * [8] Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In So Kweon.  Cbam: Convolutional block attention module.  In Proceedings of the European conference on computer vision (ECCV), pages 3–19, 2018.
  * [9] Wang Qilong, Wu Banggu, Zhu Pengfei, Li Peihua, Zuo Wangmeng, and Hu Qinghua.  Eca-net: Efficient channel attention for deep convolutional neural networks.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [10] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.  Imagenet large scale visual recognition challenge.  International journal of computer vision, 115(3):211–252, 2015\.
  * [11] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [12] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [13] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich.  Going deeper with convolutions.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1–9, 2015.
  * [14] Fei Wang, Mengqing Jiang, Chen Qian, Shuo Yang, Cheng Li, Honggang Zhang, Xiaogang Wang, and Xiaoou Tang.  Residual attention network for image classification.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3156–3164, 2017.
  * [15] Sergey Zagoruyko and Nikos Komodakis.  Wide residual networks.  arXiv preprint arXiv:1605.07146, 2016.
  * [16] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He.  Aggregated residual transformations for deep neural networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1492–1500, 2017.
  * [17] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam.  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  arXiv preprint arXiv:1704.04861, 2017.
  * [18] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen.  Mobilenetv2: Inverted residuals and linear bottlenecks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510–4520, 2018.
  * [19] Mingxing Tan and Quoc Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  In International conference on machine learning, pages 6105–6114. PMLR, 2019.
  * [20] Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc V Le, and Alexey Kurakin.  Large-scale evolution of image classifiers.  In International Conference on Machine Learning, pages 2902–2911. PMLR, 2017.
  * [21] Meng-Hao Guo, Tian-Xing Xu, Jiang-Jiang Liu, Zheng-Ning Liu, Peng-Tao Jiang, Tai-Jiang Mu, Song-Hai Zhang, Ralph R Martin, Ming-Ming Cheng, and Shi-Min Hu.  Attention mechanisms in computer vision: A survey.  Computational Visual Media, pages 1–38, 2022.
  * [22] Ozan Oktay, Jo Schlemper, Loic Le Folgoc, Matthew Lee, Mattias Heinrich, Kazunari Misawa, Kensaku Mori, Steven McDonagh, Nils Y Hammerla, Bernhard Kainz, et al.  Attention u-net: Learning where to look for the pancreas.  arXiv preprint arXiv:1804.03999, 2018.
  * [23] Lingxiao Yang, Ru-Yuan Zhang, Lida Li, and Xiaohua Xie.  Simam: A simple, parameter-free attention module for convolutional neural networks.  In International conference on machine learning, pages 11863–11874. PMLR, 2021.
  * [24] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [25] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In International conference on machine learning, pages 448–456. PMLR, 2015.
  * [26] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * [27] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes (voc) challenge.  International journal of computer vision, 88(2):303–338, 2010.
  * [28] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia.  Pyramid scene parsing network.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2881–2890, 2017.
  * [29] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille.  Semantic image segmentation with deep convolutional nets and fully connected crfs.  arXiv preprint arXiv:1412.7062, 2014.
  * [30] Bharath Hariharan, Pablo Arbeláez, Lubomir Bourdev, Subhransu Maji, and Jitendra Malik.  Semantic contours from inverse detectors.  In 2011 international conference on computer vision, pages 991–998. IEEE, 2011.
  * [31] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In International Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer, 2015.
  * [32] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117–2125, 2017.