# Energy level shift of quantum systems via the scalar electric Aharonov-Bohm
effect

RY Chiao<EMAIL_ADDRESS>University of California, Merced, School of
Natural Sciences, P.O. Box 2039, Merced, CA 95344, USA  H Hart
<EMAIL_ADDRESS>University of California, Merced, School of Natural
Sciences, P.O. Box 2039, Merced, CA 95344, USA  NA Inan<EMAIL_ADDRESS>Clovis Community College, 10309 N. Willow, Fresno, CA 93730 USA  University of
California, Merced, School of Natural Sciences, P.O. Box 2039, Merced, CA
95344, USA  Department of Physics, California State University Fresno, Fresno,
CA 93740-8031, USA  M Scheibner<EMAIL_ADDRESS>University of
California, Merced, School of Natural Sciences, P.O. Box 2039, Merced, CA
95344, USA  J Sharping<EMAIL_ADDRESS>University of California,
Merced, School of Natural Sciences, P.O. Box 2039, Merced, CA 95344, USA  DA
Singleton<EMAIL_ADDRESS>Department of Physics, California State
University Fresno, Fresno, CA 93740-8031, USA  ME Tobar
<EMAIL_ADDRESS>Quantum Technologies and Dark Matter Labs, Department
of Physics, University of Western Australia, Crawley, WA 6009, Australia.

###### Abstract

A novel version of the electric Aharonov-Bohm effect is proposed where the
quantum system which picks up the Aharonov-Bohm phase is confined to a Faraday
cage with a time varying, spatially uniform scalar potential. The electric and
magnetic fields in this region are effectively zero for the entire period of
the experiment. The observable consequence of this version of the electric
Aharonov-Bohmn effect is to shift the energy levels of the quantum system
rather than shift the fringes of the 2-slit interference pattern. We show a
strong mathematical connection between this version of the scalar electric AB
effect and the AC Stark effect.

##  I The Aharonov-Bohm effect

The Aharonov-Bohm effect [1] (AB effect hereafter) shows that in the quantum
mechanical context the scalar and vector potentials have a greater reality
than is implied by classical E&M, where the potentials can be eliminated in
favor of the fields. The two forms of the AB effect are known as the magnetic
or vector AB effect (since this involves the magnetic field coming from the
vector potential) and electric or scalar AB effect (since this involves the
electric field coming from the scalar potential). The term scalar AB effect is
also applied to cases when the system develops an AB phase connected with the
scalar interaction ${\bf\mu\cdot B}$ [2, 3, 4]. In this work the scalar
interaction we focus on is $qV$, i.e. the scalar coupling between the charge
and the electric scalar potential. The original experimental setup of the
scalar electric AB effect, which is the focus of this work, is shown in Figure
1.

The vector AB effect - which occurs by placing an infinite solenoid carrying a
constant magnetic flux behind the slits of a 2-slit interference experiment -
was first confirmed experimentally by Chambers [7] a year after the original
theoretical work by Aharonov and Bohm. This initial experimental demonstration
of the vector AB effect had some experimental loop-holes, chief among these
being that the electrons did not move in a purely ${\bf B}$-field free region.
These loop-holes where later plugged in a tour-de-force experiment [8] by
Tonomura et al., which replaced the unrealistic, infinite solenoid with a
micro-scale torus.

Next, the scalar AB effect connected with the interaction ${\bf\mu\cdot B}$,
while not as extensively tested as the magnetic or vector AB effect, has
nevertheless received a substantial amount of experimental confirmation - in
addition to references [2, 3, 4] mentioned above we note the works [5, 6].
Reference [2] in particular very clearly lays out the distinction between the
scalar AB effect coming from ${\bf\mu\cdot B}$ versus that coming from $qV$.
Additionally we will use the term “scalar electric AB effect” in the present
work, since there is a dual formulation of electromagnetism where the electric
field can arise from the curl of a dual vector potential, which then leads to
a dual AB effect [9, 10, 11]. This leads to a dual “vector electric AB
effect”.

Figure 1: The original proposed set-up for the electric AB effect.

Finally, in contrast with the above two versions of the AB effect, the scalar
electric AB effect has received much less experimental attention. A prominent
experimental verification of the scalar electric AB effect was given in [12].
However this experiment measured the effects of both the scalar electric and
vector AB effect, rather than the effect of only the scalar electric AB
effect. Also at some point in the experiment the electrons did move in a
region where the ${\bf E}$-field was non-zero.

In this work we propose a new version of the scalar electric AB effect, that
improves on the set-up in Figure 1. The basic set-up, for our new proposal to
test the scalar electric AB effect is given in Figure 2. It consists of a
Faraday shell with a time varying voltage on its surface. Outside the Faraday
shell there will be both a time varying ${\bf E}(t)$-field and time varying
scalar potential $V(t)$. However, inside the Faraday shell the ${\bf E}$-field
will be zero; there will be only a time-varying, spatially uniform scalar
potential $V(t)$. The quantum mechanical system that we use to register the
effect of this $V(t)$ inside the Faraday shell is a gas of hydrogen-like
atoms. We propose using rubidium gas since rubidium can more easily be
obtained in atomic form, whereas hydrogen generally comes as a molecule,
$H_{2}$. In the next section we discuss some of the basic theoretical aspects
of this new version of the scalar electric AB effect. We find is that the time
varying, spatial uniform potential, $V(t)$, will split the energy levels into
a series of energy levels. In terms of the mathematical analysis this almost
identical to the AC Stark effect [13, 14] where a time varying electric field
will split the energy levels of the atom. The difference here is that the
energy level splitting occurs in atoms placed in a region with a time varying
scalar potential, $V(t)$, but zero electric field ${\bf E}=-\nabla V(t)=0$.
This shift in energy levels as a means to probe the AB effect is different
from the usual signature of the AB effect, which involves a shift in
interference fringes.

##  II Analysis for new proposed scalar electric AB effect

Here we give the analysis for the new proposed scalar electric AB effect. The
quantum system used to probe the scalar electric AB effect is assumed to have
a Hamiltonian, $H_{0}$, for which the solutions to the time-independent
Schrödinger equation are known i.e. $H_{0}\Psi_{i}({\bf x})=E_{i}\Psi_{i}({\bf
x})$. To observe the scalar electric AB effect we place the quantum system
inside a Faraday shell connected to a time varying voltage as in Figure 2. The
new Hamiltonian is

$H=H_{0}+eV(t)$ (1)

We take the scalar potential $V(t)$ to be sinusoidal and of the form

$\displaystyle V(t)$ $\displaystyle=$ $\displaystyle 0~{}~{}~{}~{}{\rm
for}~{}t<0$ $\displaystyle V(t)$ $\displaystyle=$ $\displaystyle
V_{0}\cos\Omega t~{}~{}~{}~{}{\rm for}~{}t\geq 0$ (2)

where $\frac{\Omega}{2\pi}$ is the frequency and $V_{0}$ is the amplitude. For
$t<0$ where $V(t)=0$ the Hamiltonian is just $H_{0}$ with wave function
solutions $\Psi_{i}({\bf x})$, and energy eigenvalues $E_{i}$. The time-
dependent Schrödinger equation for this new system is given by

$i\hbar\frac{\partial\psi}{\partial t}=H\psi=\left(H_{0}+eV(t)\right)\psi$ (3)

Note that while $V(t)$ does depend on time $t$, it does not depend upon
$\mathbf{x}$, the position of the electron. In contrast, the unperturbed
Hamiltonian $H_{0}$ will in general depend upon $\mathbf{x}$, but it does not
depend on time $t$. Hamiltonians of the form (3) with a piecewise, continuous
periodic potential can be solved using Floquet’s theorem. We now present a
brief summary.

First we apply the separation-of-variables ansatz

$\psi(\mathbf{x},t)=X(\mathbf{x})T(t)~{}.$ (4)

Substituting this ansatz into (3) we find

$i\hbar\frac{\partial\psi}{\partial t}=i\hbar
X\frac{dT}{dt}=\left(H_{0}+eV\right)XT=TH_{0}X+X\left(eV\right)T$ (5)

Dividing this equation by $XT$ and moving the $eV(t)$ term to the left hand
side gives

$-eV+i\hbar\frac{1}{T}\frac{dT}{dt}=\frac{1}{X}H_{0}X$ (6)

This equation has the form $f(t)=g\left(\mathbf{x}\right)$ where $f(t)$ is
$only$ a function of $t$, and $g\left(\mathbf{x}\right)$ is $only$ a function
of $\mathbf{x}$. The only way that this can be true is if each function is
equal to a constant, $E$ i.e. $f(t)=g\left(\mathbf{x}\right)=E$. This gives
the separated equations

$\displaystyle-eV+i\hbar\frac{d\ln
T}{dt}=E~{}~{}~{}~{}{\text{a}nd}~{}~{}~{}~{}H_{0}X=EX$ (7)

Setting $X=\Psi_{i}({\bf x})$ and $E=E_{i}$, gives the time-independent
Schrödinger equation

$H_{0}\Psi_{i}({\bf x})=E_{i}\Psi_{i}({\bf x})~{},$ (8)

which is the known eigenvalue problem for the unperturbed hydrogen-like atom.
Integrating the first, temporal equation in (7) over $t$ gives,

$-e\int V(t)dt+i\hbar\int\frac{d\ln T(t)}{dt}dt=\int E_{i}dt$ (9)

Carrying out the integrations in (9) and solving for $T(t)$, gives

$\displaystyle T(t)$ $\displaystyle=$
$\displaystyle\exp\left(-\frac{i}{\hbar}E_{i}t\right)\exp\left(-\frac{i}{\hbar}e\int
Vdt\right)$ (10) $\displaystyle=$
$\displaystyle\exp\left(-\frac{i}{\hbar}E_{i}t-i\alpha\sin\Omega
t\right)=\exp\left(-\frac{i}{\hbar}E_{i}t-i\varphi(t)\right)~{},$

where we have defined $\alpha$, the FM depth of modulation parameter

$\alpha=\frac{eV_{0}}{\hbar\Omega}$ (11)

Figure 2: The basic set-up of our proposed test of the scalar electric AB
effect

Multiplying $X({\bf x})=\Psi_{i}({\bf x})$ and $T(t)$ from (10) gives the wave
function, $\psi_{i}({\bf r},t)$, for the full Hamiltonian, $H_{0}+eV(t)$ from
(1) gives

$\psi_{i}(\mathbf{r},t)=\Psi_{i}(\mathbf{r})\exp\left(-\frac{iE_{i}t}{\hbar}-i\varphi(t)\right)~{}.$
(12)

Note that this new wave function is the original wave function with an added
AB phase factor $\exp\left(-i\varphi(t)\right)$. From (10) $\varphi(t)$ is
given by

$\varphi(t)=\frac{e}{\hbar}\int V(t)dt=\alpha\sin\Omega t~{},$ (13)

The expression $\varphi(t)=\frac{e}{\hbar}\int V(t)dt$ above shows that
$\varphi(t)$ is the usual scalar electric AB phase. Exponentiating the scalar
electric AB phase and using the Jacobi-Anger expansion gives

$\exp\left(-i\varphi(t)\right)=\exp\left(-i\alpha\sin\Omega
t\right)=\sum_{n=-\infty}^{\infty}(-1)^{n}J_{n}(\alpha)\exp\left(in\Omega
t\right)~{}.$ (14)

Inserting the result from (14) back into (12), the wave function reads

$\displaystyle\psi_{i}(\mathbf{r},t)$ $\displaystyle=$
$\displaystyle\Psi_{i}(\mathbf{r})\sum_{n=-\infty}^{\infty}(-1)^{n}J_{n}(\alpha)\exp\left(in\Omega
t\right)\exp\left(-\frac{iE_{i}t}{\hbar}\right)$ (15) $\displaystyle=$
$\displaystyle\Psi_{i}(\mathbf{r})\sum_{n=-\infty}^{\infty}(-1)^{n}J_{n}(\alpha)\exp\left(-\frac{i\left(E_{i}-n\hbar\Omega\right)t}{\hbar}\right)$

Thus each energy level $E_{i}$ will be split into a multiplet $E_{i}^{(n)}$
with

$E_{i}^{(n)}=E_{i}\pm n\hbar\Omega,\text{ with}\ n,\text{an integer}$ (16)

where $E_{i}^{(n)}$ are evenly spaced energy levels, with an energy step
$\hbar\Omega$. This new energy spectrum, $E_{i}^{(n)}$, is of the form of the
quasi-energies discussed in [15]. If one takes the results of equations (15)
and (16) at face value this would seem to imply a new spectrum with an
infinite number of new states labeled by the sideband index $n$. However, from
(15) one finds that the different contributions are weighed by the Bessel
functions $J_{n}(\alpha)$. In Figure 3 we plot $J_{n}(\alpha)$ for a fixed
$\alpha$ as a function of $n$ and find a maximum index given by

$n_{max}\approx\alpha~{}.$ (17)

From Figure 3, where we take $\alpha=1000$, one can see that the Bessel
function weighting exponentially drops to zero beyond $n_{max}$. Thus for
$n>n_{max}$ the contributions in the sum in (15) will be suppressed as will
the energies $E_{i}^{(n)}$ with $n>n_{max}$. Further from Figure 3 one sees
that the states and energies which contribute the most and have the largest
weighting are those with $n\approx\pm n_{max}$. Taking these $n=n_{max}$
states to dominate one finds that the energy level $E_{i}$ has been split into
two levels: $E_{i}\pm n_{max}\hbar\Omega$. Recalling that
$n_{max}\approx\alpha=\frac{eV_{0}}{\hbar\Omega}$ then gives new energies
levels of $E_{i}^{(n_{max})}=E_{i}\pm eV_{0}$.

Figure 3: A plot of $J_{n}(\alpha)$ vs. $n$ for the case when $\alpha=1000$.
There are sidebands in the energy $E^{(n)}_{i}$ which occurs up to some
maximum index $n$ given by $n_{max}\approx\alpha$. From the plot one can see
that the weighting, $J_{n}(\alpha)$, is largest when $n=n_{max}\approx\alpha$
and it is this state which contributes the most.

This result of the splitting of the energy levels - our version of the scalar
electric AB effect - looks almost mathematically identical to the Autler-
Townes effect or AC Stark effect [13]. Further in reference [16] Townes and
Schawlow point out that the radio-frequency methodology of frequency
modulation (FM) can be applied to quantum mechanical wave functions, whenever
their phases undergo sinusoidal frequency modulation due to external
perturbations, which mirrors the influence of the external, time varying
potential, $V(t)$, on the wave function $\psi_{i}$ as shown in (15). The key
difference of course is that for the AC Stark effect the quantum system is in
a region of non-zero electric field, while here the quantum system is in a
region where the electric field is zero but the scalar potential is non-zero.

We now show in detail the mathematical similarity between our version of the
scalar electric AB effect and the AC Stark effect. To lay out the similarities
we compare our results to those of the AC Stark effect as given in the review
article [14]. Our modified wave function (15) matches with the wave function
from the review article (equation (13) of [14]). The weighting factor in [14],
instead of being $(-1)^{n}J_{n}(\alpha)$, takes the more complex form

$C_{n}=\sum_{S=-\infty}^{+\infty}(-1)^{n}J_{S}\left(\frac{\beta
E_{0}^{2}}{8\Omega}\right)J_{n+2S}\left(\frac{dE_{0}}{\Omega}\right)~{},$ (18)

where $E_{0}$ is the magnitude of the electric field that oscillates with
frequency $\Omega$, $d$ is a dipole moment, and $\beta$ is a polarizability.
In (18) we have changed some of the notation relative to reference [14] to
avoid overlap with our notation. In [14] there were interactions terms between
the electric field and quantum system - one linear and one quadratic in
$E_{0}$. In our case we have only linear terms in the $E_{0}$ or $V_{0}$ i.e.
for us $\beta=0$ so that the middle term in the sum in (18) is $J_{S}(0)$. Now
$J_{S}(0)=0$ except for $S=0$ ($J_{0}(0)=1$). Thus the sum in (18) reduces to
only the $S=0$ term namely $C_{n}=(-1)^{n}J_{n}(dE_{0}/\Omega)$. This exactly
matches our weighting $(-1)^{n}J_{n}(eV_{0}/\hbar\Omega)$. Taking the dipole
as $d=ex$ (charge times distance) we find that $dE_{0}\sim exE_{0}$, noting
that a distance times electric field has units of a potential ($xE_{0}\sim
V_{0}$), and finally recalling that reference [14] took $\hbar=1$ one can see
that $\frac{dE_{0}}{\Omega}\sim\frac{eV_{0}}{\hbar\Omega}$. The $C_{n}$
weighting in (18) exactly matches our $(-1)^{n}J_{n}(\alpha)$ weighting.

The reason that [14] had both linear and quadratic electric field terms,
whereas we only have linear terms, comes about due to the difference in
minimal coupling of the scalar versus vector potential in the time-independent
Schrödinger equation $i\hbar\frac{\partial}{\partial t}\Psi=\frac{{\hat{\bf
p}^{2}}}{2m}\Psi$. For the vector potential case with a sinusoidal potential
${\bf A}(t)=\frac{E_{0}}{\Omega}\cos(\Omega t){\hat{\bf z}}$ minimal coupling
is ${\hat{\bf p}}^{2}\to({\hat{\bf p}}-e{\bf A})^{2}$ which then gives both
linear (${\bf A}\sim E_{0}$) and quadratic (${\bf A}^{2}\sim E_{0}^{2}$)
terms. In contrast the minimal coupling of the scalar potential arises via the
minimal coupling $i\hbar\frac{\partial}{\partial t}\to
i\hbar\frac{\partial}{\partial t}-eV(t)$ with $V(t)=V_{0}\cos(\Omega t)$, and
this only gives a term linear in the scalar potential. More physically the
quadratic term in [14] is connected with polarization of the quantum system by
the electric field, but for our scalar electric AB setup there is no electric
field and hence no polarization term.

This connection between the AC Stark effect and the scalar electric AB effect
also supports the earlier statement that only a few of the many energy levels
from (16) are populated and thus play a role. In the AC Stark effect, both
theory and experiment [14] indicate that the $n=\pm n_{max}$ states are
dominant, in the limit of low frequency and strong field. It is in this limit
that the AC Stark effect and scalar electric AB effect are corrected due to
the absence of the quadratic term for the scalar electric AB effect. Thus this
close mathematical connection between the AC Stark effect and the scalar
electric AB effect indicate that, as for the AC Stark effect, the $n=\pm
n_{max}$ states are dominant for the scalar electric AB effect.

After drawing this exact, mathematical identity between the AC Stark effect
and our scalar electric AB effect proposal, we now state that physically these
two are very different. In the case of the AC Stark effect the quantum system
is in a region where the electric field is non-zero, so qualitatively (but not
quantitatively) all the effects can be explained classically since in
classical electromagnetism electric fields can change the energy of the
system. In our proposal the scalar potential $V(t)$ is uniform within the
entire interior of the Faraday shell, so that the ${\bf E}$-field is zero
inside the Faraday shell and there are _no electrical forces_ exerted on the
atom. Classically one expects no effect in a Faraday cage i.e. no energy
change for the quantum system. However the connection between the AC Stark
effect and our proposal for the scalar electric AB effect, can be compared to
Feynman’s analysis of the vector AB effect in Volume II of the Feynman Lecture
series. In [18] Feynman first derives the standard vector AB effect to obtain
the shift in the interference pattern when there is an infinite solenoid
placed between the slits. He then takes the magnetic field and smears it out
into a continuous strip in the region behind the slits and re-analyzes what
happens. In this second case the charges going through the slits are subjected
to a ${\bf v}\times{\bf B}$ force which changes their momentum and thus shifts
the interference pattern. Feynman shows that the shift in the interference
pattern is the same in both cases as long as the flux enclosed by the
particles path is the same. In the usual AB set-up the shift can only be
interpreted as arising from the flux enclosed by the path of the charges,
while in the case where the magnetic field is smeared out in a strip behind
the slits one can view the shift as coming either from the AB effect and the
enclosed flux or from the shift in the momentum of the charged particles due
to a ${\bf v}\times{\bf B}$ force. In an analogous way for the AC Stark effect
one can interpret the change in energy levels as coming from either the direct
effect of the electric field or from the influence of the time varying vector
potential, ${\bf A}(t)$, giving rise to the electric field. But for our
version of the scalar electric AB effect the explanation in terms of the
scalar potential, $V(t)$, is the only available explanation for the shift in
energy levels.

We conclude with observations about this version of the scalar electric AB
effect:

  * •

The scalar electric AB phase, $\varphi(t)$, creates energy level sidebands,
equation (16), which can be probed via absorption spectroscopy. The dominant
energy sidebands occur for $n=n_{max}\approx\alpha$. These energy sidebands
are essentially the quasi-energy levels of Zeldovich [15] and studied later in
more detail by Sambe [17].

  * •

The set up here and the analysis leading to wave function (15) and split
energy levels (16) is mathematically identical to the AC Stark/Autler-Townes
effect. In the above we have shown how our proposed set up for the scalar
electric AB effect matches the AC stark effect, except for the absence of the
quadratic term in the electric field.

  * •

One can question this setup for the scalar electric AB effect since it is
possible to gauge away the scalar potential, $V(t)=V_{0}\cos(\Omega t)$. A
general gauge transformation of the scalar and vector potential is

$V^{\prime}=V-\partial_{t}\lambda~{}~{}~{}{\rm and}~{}~{}~{}{\bf
A}^{\prime}={\bf A}+\nabla\lambda~{},$ (19)

with the gauge function $\lambda({\bf r},t)$. By choosing
$\lambda(t)=\frac{V_{0}}{\Omega}\sin(\Omega t)$ one can cancel out
$V(t)=V_{0}\cos(\Omega t)$ so that $V^{\prime}=0$. Also since
$\nabla\lambda=0$ the new vector potential will remain zero, ${\bf
A}^{\prime}=0$. Thus one has gauge transformed away all electromagnetic
potentials so how can there be any effect? The resolution to this is that (19)
is only half of the gauge transformation. One must also transform the wave
function as

$\displaystyle\psi^{\prime}_{i}({\bf r},t)$ $\displaystyle=$
$\displaystyle\exp\left(-i\frac{e}{\hbar}\lambda\right)\psi_{i}({\bf
r},t)=\exp\left(-i\frac{eV_{0}}{\hbar\Omega}\sin(\Omega t)\right)\Psi_{i}({\bf
r})\exp\left(-i\frac{E_{i}t}{\hbar}\right)$ (20) $\displaystyle=$
$\displaystyle\Psi_{i}({\bf
r})\exp\left(-i\frac{E_{i}t}{\hbar}-i\varphi(t)\right)~{}.$

In arriving at the results in (20) we have used results from (12) and (13).
The wave function $\psi^{\prime}$ from (20), which is in the gauge where all
electromagnetic potentials are zero, matches the wave function $\psi$ from
(12) which has the non-zero, sinusoidal scalar potential. The results leading
to the energy side bands are gauge invariant.

  * •

This version of the scalar electric AB effect is cleaner than the original
proposal shown in Figure 1. We do not have to time the turn-on and turn-off of
the potential difference with the charge entering or exiting the metal tubes.
Also for the setup in Figure 1 there are always fringing fields, while for the
Faraday sphere setup the electric field is zero inside the shell, modulo very
small electric and magnetic fields that arise whenever there is a time
variation in charges/fields.

##  III Experiment for testing the scalar electric AB effect

In the preceding section we have shown that our setup for the scalar electric
AB effect is mathematically equivalent to the AC Stark effect minus the
quadratic term. Thus many experimental tests for the AC Stark effect should
apply to our setup for the scalar AB effect, but the quantum systems that act
as probes of the effect would be in a region completely free of electric
field, in contrast to the AC Stark effect.

The simplest test for the scalar electric AB effect in our setup is to look
for the splitting of the initial energy levels, $E_{i}$, into two levels
$E_{i}\pm n_{max}\hbar\Omega=E_{i}\pm eV_{0}$. This splitting into these two
dominant sidebands rather than into a large number of sidebands requires
$eV_{0}\gg\hbar\Omega$. For example, one could place hydrogen-like atoms
inside the Faraday sphere of Figure 2 with $\Omega=10^{8}~{}s^{-1}$ and
$V_{0}=0.5~{}m$V so that the two dominant sidebands appear shifted by $\sim
10^{12}~{}s^{-1}$ in the absorption spectrum. This set of parameters would
give $\alpha=\frac{eV_{0}}{\hbar\Omega}\sim 10^{4}$ for which
$eV_{0}\gg\hbar\Omega$ is satisfied. For this type of proposed probe of the
scalar AB effect, a vapor of rubidium atoms would be optimal since the
expected tunable AB effect feature can be compared with the well-known fine
and hyperfine lines.

Another possible test of our setup of the scalar AB effect is
electromagnetically induced transparency (EIT). Observing EIT involves two
lasers - a probe laser and coupling laser - which are tuned to interact with
three quantum states. The probe laser is tuned near resonance between two of
the states and measures the absorption spectrum of the transition. A stronger,
coupling laser is tuned near resonance at a different transition. By properly
selecting the states, the presence of the coupling laser will create a
spectral window of transparency which can be detected by the probe laser.
Figure 4 is a sketch of the “Lambda” version of EIT. This is a three-level
scheme to observe EIT. Again we have in mind using rubidium with the three
energy levels being the ground state, $5S_{1/2}$, the first excited state,
$5P_{1/2}$ and the upper sideband of the ground state, $E_{1}^{(+)}$. Note
that for EIT one needs quantum interference between two paths in energy space.
The original proposal for both the vector and scalar AB effect from [1] also
worked via interference, but interference in coordinate space, rather than
energy space. Therefore it is possible to use a neutral atom such as a
rubidium atom, in order to observe the scalar AB effect, since one could use
the two distinct electronic pathways, as depicted in Figure 4.

Figure 4: Electromagnetic induced transparency as a demonstration of the
scalar AB effect.

Two of the three energy levels are the ground state $E_{5S_{1/2}}$ (black) of
a two-level atom (rubidium), and its Jacobi-Anger image state $E_{1}^{(+)}$
(red), and the other is the unperturbed excited state $E_{5P_{1/2}}$ (black)
of this atom. In absence of any perturbations, the probe laser (orange) would
detect a Lorentzian absorption lineshape in a strong resonant absorption-line
transition from $E_{5S_{1/2}}$ to $E_{5P_{1/2}}$.

However, in the presence of the external sine-wave generator driving the
outside of the Faraday cage in Figure 2 with an AC voltage
$V(t)=V_{0}\cos\Omega t$, there arises the third energy level, $E_{1}^{(+)}$
(red), which is a Jacobi-Anger image sideband state separated by an energy
$+eV_{0}$ from the ground state $E_{5S_{1/2}}$. This image sideband has the
same S-wave character as that of the ground state $E_{5S_{1/2}}$. As a result
of the action of the coupling laser (deep red), an image of this S-wave state
of the ground state is created at the center of the Lorentzian line profile
(whose linewidth arises from the $A$ coefficient of spontaneous emission) of
the excited P-wave state $E_{5P_{1/2}}$. A selection rule forbids S-wave to
S-wave transitions, and this will lead to a central hole of transparency in
the middle of the absorption line profile arising from the $E_{5S_{1/2}}$ to
$E_{5P_{1/2}}$ transition that is detected by the probe laser, as it is
scanned across the strong Lorentzian resonance absorption lineshape of this
transition. One can understand this central hole of transparency as arising
from the destructive interference between path 1 and path 2 in the energy-
level diagram of Figure 4. The interference of the two paths in this energy-
level space is analogous to the interference between the two paths in
coordinate space in the usual vector AB effect.

We note that the experimental setup required to observe our version of the
scalar electric AB effect is very similar to the setup shown in Figure 1 of
[19]. There is a crucial difference between the setup in reference [19] and
our generic setup from Figure 2. In Figure 2 we have a Faraday shell which
screens the quantum system from the electric field, while in reference [19]
the Faraday shell is replaced by a resonating cavity where the quantum system
is embedded in a time varying electric field.

In this section we have focused on using rubidium atoms to probe our version
of the scalar electric AB effect. However one could replace rubidium atoms by
quantum dots as in [20] where charged quantum dots were used to probe the AC
Stark effect.

##  IV Summary and Conclusions

We have presented a new setup for the scalar electric AB effect which avoids
some of the pitfalls of the original set up given in Figure 1 (i.e. no
fringing electric fields, no need to time the turn-on/turn-off of the
potential difference with the motion of the charge). In contrast to the vector
AB effect where a quantum system develops an AB phase by moving through a
time-independent but spatially varying vector potential, ${\bf A}({\bf r})$,
here the quantum system develops an AB phase by sitting at rest in a spatially
uniform but time varying scalar potential, $V(t)$. In the vector AB effect
with the magnetic flux contained inside an infinite solenoid, the physical
consequence of the AB phase is to shift the interference pattern by an amount
that depends on the magnetic flux. This shift occurs despite the charged
particle moving in a region that is free of magnetic field.

Similar, to the Feynman analysis of the vector AB effect [18] we first have
the scalar electric AB effect setup in Figure 2 where the quantum system is in
a region with a time varying, but spatially uniform scalar potential, $V(t)$,
with no electric field. For this setup we find that the wave function is
shifted by an AB phase (see equation (15)) and the energy spectrum develops
sidebands (see equation (16)). These features of the wave function and energy
spectrum are mathematically identical to what is found in the AC Stark or
Townes-Autler effect [13] where the quantum system is placed in a region with
a non-zero and time varying electric field. This closes the connection with
the Feynman analysis of the vector AB effect since for the AC Stark setup one
can use either the electric field or the scalar potential to obtain the
effect, whereas for in the setup in Figure 2 one can only see these effects as
coming from the scalar electric AB effect, since there is no electric field.
There are previous works which have drawn connections between the AC Stark
effect and the AB effect [21, 22, 23, 24]. However these prior works usually
involved the scalar AB effect via the interaction ${\bf\mu\cdot B}$, rather
than the scalar electric AB effect via the interaction $qV$, and the
experimental signature was still, mostly, a phase shift rather than an energy
level shift.

This opens up the possibility to perform a clean test of the scalar electric
AB effect, which looks very different from the original setup of reference [1]
illustrated in Figure 1.

MET is funded by the ARC Centre of Excellence for Engineered Quantum Systems,
Grant No. CE170100009 and the ARC Centre of Excellence for Dark Matter
Particle Physics, Grant No. CE200100008.

## References

  * [1] Y. Aharonov and D. Bohm, Phys. Rev. 115, 484 (1959).
  * [2] B. E. Allman, et al., Phys. Rev. A 48, 1799 (1993).
  * [3] B. E. Allman, W.-T. Lee, O. I. Motrunich, and S. A. Werner,Phys. Rev. A 60, 4272 (1999).
  * [4] W.-T. Lee, O. Motrunich, B. E. Allman, and S. A. Werner, Phys. Rev. Lett. 80, 3165 (1998).
  * [5] K. Shinohara, T. Aoki, and A. Morinaga, Phys. Rev. A, 66, 042106 (2002).
  * [6] G. Badurek, et al., Phys. Rev. Lett. 71, 307 (1993).
  * [7] R.G. Chambers, Phys. Rev. Lett. 5, 3 (1960).
  * [8] A. Tonomura et al., Phys. Rev. Lett. 56, 792 (1986).
  * [9] H. Wei, R. Han, and X. Wei, Phys. Rev. Letts. 75, 2071, (1995)
  * [10] J. P. Dowling, C. P. Williams, and J. D. Franson, Phys. Rev. Letts. 83, 2486 (1999).
  * [11] M. E. Tobar, R. Y. Chiao, and M. Goryachev, Sensors 22, 7029 (2022).
  * [12] A. van Oudenaarden, M.H. Devoret, V. Yu. Nazarov, and J.E. Mooij, Nature, 391, 768 (1998)
  * [13] S.H. Autler and C. H. Townes, Phys. Rev., 100, 703 (1955).
  * [14] N.B. Delone and V.P. Krainov, Physics-Uspekhi, 42, 669 (1999).
  * [15] Y. Zel’dovich, Sov. Phys.-JETP 24, 1492 (1967).
  * [16] C.H. Townes and A.L. Schawlow, _Microwave Spectroscopy_ (Dover, 1975).
  * [17] H. Sambe, Phys. Rev. A 7, 2203 (1973).
  * [18] R. P. Feynman, The Feynman Lectures on Physics Volume II, section 15.5 (Basic Books 2011).
  * [19] F. Levi et al., Phys, Rev. A 93, 023433 (2016).
  * [20] T. A. Wilkinson, et al., Appl. Phys. Lett. 114, 133104 (2019).
  * [21] T. Akatsuka, et al., Phys. Rev. A 86, 023418 (2012).
  * [22] K. Sengstock, et al., Opts. Comm., 103, 73 (1993).
  * [23] H. Imai, Y. Otsubo, and A. Morinaga, Phys. Rev. A 76, 012116 (2007).
  * [24] K. Numazaki, H. Imai, and A. Morinaga, Phys. Rev. A 81, 032124 (2010).


# MSTAR: Multi-Scale Backbone Architecture Search for Timeseries
Classification

Tue M. Cao  Hanoi Universiry of Science and Technology  VinUni-Illinois Smart
Health Center, VinUniversity  Nhat H. Tran  Hanoi Universiry of Science and
Technology  VinUni-Illinois Smart Health Center, VinUniversity  Hieu H. Pham
Corresponding author. VinUni-Illinois Smart Health Center, VinUniversity
Thanh Hung Nguyen  Hanoi Universiry of Science and Technology  Phi Le Nguyen
Hanoi Universiry of Science and Technology

###### Abstract

Most of the previous approaches to Time Series Classification (TSC) highlight
the significance of receptive fields and frequencies while overlooking the
time resolution. Hence, unavoidably suffered from scalability issues as they
integrated an extensive range of receptive fields into classification models.
Other methods, while having a better adaptation for large datasets, require
manual design and yet not being able to reach the optimal architecture due to
the uniqueness of each dataset. We overcome these challenges by proposing a
novel multi-scale search space and a framework for Neural architecture search
(NAS), which addresses both the problem of frequency and time resolution,
discovering the suitable scale for a specific dataset. We further show that
our model can serve as a backbone to employ a powerful Transformer module with
both untrained and pre-trained weights. Our search space reaches the state-of-
the-art performance on four datasets on four different domains while
introducing more than ten highly fine-tuned models for each data.

##  1 Introduction

Time Series data (TS) is a sequence of data points recorded or collected at
successive points in time. A time series, by nature, can be decomposed into an
infinite number of frequencies, each having its own coefficient by novel
transformation methods such as Fourier transform, Wavelet transform, etc.
Within a particular dataset, each frequency can be identified as either an
informative attribute or mere noise based on its significance to the outcomes
of a classification model. Consequently, in Time Series Classification (TSC),
most methods have developed many flexible ways to extract and differentiate
between useful and noisy frequencies. However, due to a wide range of
approaches and settings to extract real-world data, time series appears in
many different forms, containing various types of signals on various time
scales [8, 40, 32]. Furthermore, each dataset prioritizes a distinct set of
informative frequencies while disregarding others as noise. Therefore, the
primary objective of TSC is to effectively isolate and leverage crucial
signals across various time scales while minimizing the impact of
insignificant ones.

Several strategies have been formulated to tackle this issue. One primary
method is to concentrate on receptive fields [40]. Omni-Scale CNN aims to
resolve the dilemma of the optimal receptive fields by introducing a unique
architecture that integrates diverse prime-kernel-size convolutions. This
design can capture an extensive range of receptive fields, ensuring
comprehensive coverage of frequencies across all time scales. However, this
approach faces a challenging problem of scaling with larger datasets because
of the unique architecture, which comprises only three convolutional layers,
limiting the complexity of the feature space that can be explored.
Additionally, the use of multiple convolutions within a single layer
($O(r)/log(r)$ where $r$ is the largest receptive field) poses difficulties in
scaling, as adding more layers would significantly increase the number of
parameters. Expanding the receptive field size to accommodate longer signals
may also result in overfitting [40]. Another popular approach is the ensemble
of 1-layer convolutions with various kernel sizes and dilations [11, 9, 10,
38] to extract frequencies in a rapid and efficient manner. Despite its
innovative nature, this method also suffers from ineffectiveness when it comes
to large data compared to deep learning models [15] as their models are mainly
tested on UCR and UEA archive [8, 3] that mostly consist of relatively small
or tiny datasets. Other approaches such as [32, 33] leverage different novel
transformations that represent time series in more comprehensible ways, and
then use a machine learning classifier to label each data point. The state-of-
the-art approach, [26], ensembles others’ classifier, which intuitively
creates a robust classifier that is proven to be the best method on [8]. All
of these techniques also revolve around extracting signals in time series at
different scales, but most importantly, they all need to overcome the
scalability challenge of a machine learning based approach.

On the other hand, while deep learning models such as Inception Time [20],
1D-ResNet [17], and xResNet [18] exhibit greater adaptability to larger
datasets, their ability to capture every size of receptive field is limited by
their fixed structures and parameter constraints. Furthermore, although having
to manually design the structure of deep learning models, the most critical
frequencies for a dataset may not be obvious to researchers and can vary with
the type of data. This makes the design more time-consuming, and yet, does not
guarantee the optimal architecture for specific datasets. We summarize the
shortcomings of previous approaches in Table 1.

Task | Multi-scale | Scalability | Auto design
---|---|---|---
Deep learning [30, 18, 20] | $X$ | $\surd$ | $X$
OmniScaleCNN [40] | $\surd$ | $X$ | $X$
Others [26, 32, 33, 9, 10, 38, 11] | $\surd$ | $X$ | $\triangle$
MSTAR (our) | $\surd$ | $\surd$ | $\surd$
Table 1: The overview of previous methods in Time series classification.
$\triangle$ means not relevant.

Lastly, although the main challenge addressed by previous methods to reach
state-of-the-art results in TSC is through filtering frequencies [32, 20, 40,
9], we argue that the resolution in the time axis of the extracted frequency
significantly influences the predictive accuracy of classification models.
Which, to get a better view, we show the relation between receptive field (as
well as frequency) and time frame in the state-of-the-art model in section 6.

These mentioned challenges urge us to come up with MSTAR (Multi-Scale Time
series ARchitecture search), a deep learning backbone that leverages the
multi-scale convolution, drawing inspiration from the Inception-liked models
[37, 20] and neural architecture search (NAS) [56]. Our approach of NAS for
time series classification suggests a novel way to tackle the formidable
problem of architecture manual construction. By employing NAS, our approach
aims to search for the optimal range of receptive fields tailored to a
specific dataset, which lies at the heart of the successes of other TSC
models, and further bring out the architecture that best preserves the time
resolution for each critical frequency. We summarize our contributions as
below:

  * •

We introduce MSTAR as a multi-scale convolutional backbone search space, of
which the primary goal is to search for the optimal architecture that is
capable of extracting the essential frequencies needed for time series
classification. MSTAR found 4 completely different architectures that reached
state-of-the-art performance at 4 datasets ranging from relatively small (10K)
and up to large (1M) at distinct domains compared to many previous TSC
approaches.

  * •

We provide a novel framework along with an effective autoencoder that aids our
search space to find novel architectures in any dataset.

  * •

We do analytic experiments to show that the time resolution is also a critical
factor to be taken into consideration when building TSC models.

  * •

We find that our CNN-backbone can be paired effectively with pre-trained
Vision Tranformer (VIT) [13] compared with other 1D-CNNs. Allowing our
backbone to harness the power of global contextual understanding, as well as
leveraging sophisticated pattern recognition offered by large vision models.

##  2 Related works

###  2.1 Time series classification

In the exploration of Time Series Classification (TSC), methods such as [9,
10, 38, 11], have marked significant strides. They utilize convolution without
training the weights as the main strategies for frequency extraction in an
efficient but robust manner. Other methods such as [32] use symbolic
representation to minimize the noise while aiming to enhance signal
representation. HIVE-COTE v2 [26] stands out as the state-of-the-art,
integrating multiple classifiers for robust performance. However, those
mentioned methods predominantly undergo evaluation on smaller datasets from
the UCR and UEA archives, highlighting a scalability challenge. Deep learning
approaches like InceptioneTime [20], TCN [4] and others [30, 18] have better
adaptability to larger datasets, offering flexibility and overcoming some
limitations of earlier models. Yet they too face challenges in manual design
complexity and the architectures may not be suitable for some specific types
of datasets.

To alleviate the challenges for deep learning models, Omni-Scale CNN [40]
offers an innovative architecture to capture a broad spectrum of frequencies
by integrating diverse prime-kernel-size convolutions. This method is adept at
ensuring comprehensive coverage across all time scales. However, its
scalability is hampered by the architecture’s simplicity, which includes only
three convolutional layers. This limitation not only restricts the complexity
of the feature space that can be explored but also the model scaling due to
the exponential increase in parameters with added layers. Moreover, the risk
of overfitting escalates with the expansion of the receptive field to
encompass longer signals.

###  2.2 Neural architecture search

Neural architecture search (NAS) is a technique that automates the process of
designing and optimizing the architecture of neural networks instead of
manually specifying the network’s structure [56]. Prior surveys [14, 48] have
pointed out three main components of NAS: search space, search strategy, and
performance evaluation strategy.

When it comes to NAS search spaces, they are divided into cell-based [25],
chain-based [39], and hierarchical search spaces [7], with cell-based search
spaces perhaps being the most popular ones. Cell-based search spaces are
inspired by the fact that the most successful CNNs designed with repeated
patterns stacked on top of each other, therefore narrowing down the search to
only relatively small cells, as opposed to searching the entire network from
scratch [56]. This type of search space has significantly lower complexity
compared to the other two, making it our choice of design in this paper [48].
In the field of image classification, there have been many search spaces of
high quality, such as NAS-Bench-101 [53] and NAS-Bench-201 [12], and inspired
by the famous manually designed ResNet [17]. However, to the best of our
knowledge, there has been a lack of effective search spaces to encapsulate
effective 1D-CNN architectures for the task of time-series data
classification. Our novel cell-based search space is built upon the basis of
the Inception-liked structure [37, 20], a powerful convolution layer that
incorporates multi-scale convolution filters and inception-like blocks to
capture diverse temporal patterns.

On the other hand, search strategies have become the main focus of NAS
research in recent years. Search strategies can be generally divided into two
categories, namely: black box optimization [56, 2, 47] and one-shot technique
[25, 29, 5], each of which has its own unique characteristics and advantages
over the other. One-shot, which has been prominent in the field of NAS due to
its efficiency, has every model in the search space evaluated using the
weights from one large model called supernet [25] or hypernet [5]. However,
the trustworthiness of not training a model directly has been a debatable
topic [48]. Meanwhile, the black box mechanism picks several models by either
training from scratch [56] or by using predictors [47]. Algorithms for the
construction of architecture batches range from evolutionary and genetic
algorithms [2], reinforcement learning [56], Bayesian Optimization [16, 47],
etc. Although the black box method usually involves a great requirement of
computational resources (due to individual training of many architectures for
evaluation), their conceptual simplicity makes them a more accessible option
in most applications and this work is no exception. Our approach is inspired
by the framework of BANANAS [47] and Bayesian Optimization [16], however, with
great modifications to suit our applications.

##  3 Methodology

In this section, we first discuss in more depth the impact of the time
resolution of non-trivial frequencies and the reason for the design of our
search space in subsection 3.1. Afterward, we will elaborate on the specific
detail of the search space and components in the framework in the remaining
subsections.

###  3.1 Motivation

Continuous Wavelet Transform (CWT) has been applied widely in signal
processing for AI [45, 23, 49] due to its effectiveness in understanding how
the frequency content of the signal evolves over time. In addition, CWT for
discrete signals bears great similarities to the convolutional layer in CNNs:

$CWT_{a,b}\,=\,\frac{1}{\sqrt{a}}\sum_{n=0}^{N-1}x[n]\,.\,\psi^{*}\left({\scriptstyle\frac{n-b}{a}}\right)$
(1)

Where $x[n]$ is the discrete signal at time $n$, $\psi^{*}$ is the complex
conjugate of the mother wavelet function, $a$ and $b$ are the scale and the
translation of the wavelet function respectively. Parameter $a$ controls the
scaling (or frequency) of the mother wavelet while $b$ handles the translation
(or shifting) where it slides the center of the wavelet along the input
signal. The combination of both parameters allows capture features at various
frequencies and time locations, analyzing non-stationary signals by
compromising between time and frequency resolution. Similarly, convolutional
layers operate on the principle of local receptive fields, scanning the input
signal by shifting the kernel weights along the time axis where these kernel
can have different sizes or scales. Previous works have pointed out that
convolution kernels often include filters for frequencies [22, 54, 55] where
sliding convolution kernel helps remove noise while enhancing useful
frequencies. We can view the convolution as a general case of CWT where the
kernel’s weights (or wavelet filter) are not determined by mathematical
properties but rather constructed from training with the data. Consequently,
applying convolutional weights on signals resembles the mechanism of
downgrading the time resolution in exchange for extracting frequencies.

Therefore, we contend that TSC should put more weight on the time resolution
of the extracted features. This proposition is intuitive, as in certain time
series types, the range of beneficial frequencies may vary across different
time locations [41]. By knowing the exact time frame of the processing
frequencies, classification models are provided with critical predictive
information that is often overlooked by previous methods [40, 9, 10, 11].
Focusing only on the frequency problem, one naive approach would employ very
large kernel size convolutions which contain an aggregation of various
frequency filters. Based on Fourier analysis that a signal can be decomposed
into multiple distinct frequencies, scaling from fine-grained (high frequency)
up to the length of the receptive field (low frequency) Figure 1, each acts as
a filter to process the input signal [34]. In theory, these convolution can
capture all of the necessary features, thus, satisfying the frequency
extraction requirement. However, this method would not work as the time
uncertainty is high for shorter signals (higher frequency) due to the
extensive kernels that blur out the precise time locations [34].

Figure 1: A large kernel, or a complex signal, can be dissected into multiple
frequency components. Each component functions as a distinct filter, isolating
a specific frequency from the input signals.

We conduct an experiment to verify the claim Table 2. We create five models
denoted as $M_{i}$ where $i$ is the length of the extensive convolution
kernel, and an additional model $M_{all}$ which contains a set of distinct
kernel sizes scaling from 3 to 20. Other hyperparameters are the same between
the models which ensures that they have roughly the same magnitude of the
number of trainable weights. The dataset for testing is PTB-XL [44], a popular
dataset in electrocardiography (ECG) fields. Further details regarding the
experiment are elaborated in Appendix B. As indicated in Table 2, the multi-
scale model demonstrates superior performance in classification tasks. In
contrast, the other models’ performances gradually decrease as the kernel size
gets broader. This implies that the downgrade of time resolution extracted by
large convolution leads to the loss of crucial information in TSC. It is
important to note that, [40] witnessed the same phenomenon, however, they only
discuss about the impact of the receptive fields. We will discuss about the
relation between time and receptive fields further in the section 6.

Task | $M_{all}$ | $M_{20}$ | $M_{30}$ | $M_{40}$ | $M_{50}$ | $M_{60}$
---|---|---|---|---|---|---
Super | 0.9325 | 0.9308 | 0.9301 | 0.9275 | 0.9231 | 0.9222
Table 2: The comparison of multi-scale model and large-kernel-size models with
the "super" task in the PTB-XL dataset. The reported metric is AUC, the best
performance is highlighted in bold.

To address the problem, one solution comes naturally. We employ NAS [56] as a
novel way to search for an architecture that could be highly finetuned for a
specific dataset, where it is chosen to have a unique set of convolutions to
capture all beneficial frequencies with the highest time resolutions. In
addition, by only selecting necessary frequencies, our search space can avoid
having excessive convolutions, thus, having better scalablity than previous
multi-scale approach [40] shown in section 5. Our design for the search space
is inspired by Inception-liked architectures [37, 20] which favor a "wide"
convolutional layer over "deep" like ResNet [17]. Due to the great flexibility
of Inception, our search space can construct various models that, by stacking
convolution form a wide range of receptive fields [37], allowing extraction of
frequencies across multiple time scales.

###  3.2 Search Space

We constructed a cell-based search space that can be represented with a graph
as shown in Figure 2. The graph includes 13 nodes
$\\{N_{i}\>|\>i\subset{0,...,12}\\}$ partitioned into 3 layers, the first node
and the last node (which have the indexes of 0 and 12) are the input node and
output node respectively. The first layer contains node 1 ($N_{1}$), the
second contains 5 nodes from node 2 ($N_{2}$) to node 6 ($N_{6}$), the last
layer consists of node 7 ($N_{7}$) to node 11 ($N_{11}$). Each node has an
attribute of the number of channels, denoted as $C_{i}$, which indicates the
number of output channels of that node, the initial channel size of each node
is provided in Figure 2. As shown, there are two types of channels: 128 and
32, the former is the standard channel size of a node while the latter serves
as a bottleneck channel size (based on InceptionTime) which minimizes the
computational complexity of the model.

Figure 2: search space’s cell visualization: The default channel sizes along
with the preprocessed channel sizes are provided. Edges that connect 2 nodes
are denoted as operation-edge (orange) and contain an operation, other edges
are used purely for connection (green).

The edge that connects a pair of nodes denoted as $E_{ij}$ symbolizes one of
the predefined operations namely: convolution, max pooling, average pooling,
and identity. The output of nodes with smaller indexes will be forwarded to
the respective operation and transformed into the output of nodes with larger
indexes. For example, if $N_{i}$ and $N_{j}$ are connected and $i\,<\,j$, the
output of $N_{i}$, which has $C_{i}$ channels, will be passed through an
operation stored in edge $E_{ij}$ to become the output of node $N_{j}$ having
the channel of $C_{j}$. Furthermore, the channel of each node will be
determined by the minimum channel of all the previous nodes (nodes with
smaller indexes) that are connected to it, and nodes within the same layer
cannot be wired together. Lastly, nodes that are not connected to any upper
layers’ node will be transformed to have the default of 128 channels (by
concatenating with other bottleneck nodes and passing through depth-wise
convolution) and then added to the output of the last node to be forwarded to
the next cell. An example is provided in Figure 2.

In the search space, kernel sizes are selectable for different operations.
Specifically, for convolution, the default sizes are 1, 3, 5, 9, 19, and 39,
while for max and average pooling, the options are 3, 5, and 9. This choice of
kernel sizes, which is inspired from previous TSC model InceptionTime [20],
allows our search space to construct multiple receptive fields by stacking
convolutions [37], providing flexibility in extracting necessary frequencies
while maintaining reasonable time resolution. Ultimately, the model is formed
by assembling multiple identical cells and can either be attached with a
classifier for classification or serves as a backbone for time series
processing.

###  3.3 Search space encoding

A model can be represented by its unique cell, we encoded our search space
with a $4\times 13\times 13$ matrix denoted as $A\in\textbf{A}$ which is
concatenation of four $13\times 13$ adjacency matrices for each operation,
with each element in an adjacency matrix indicates the connection between a
pair of nodes. Specifically, if $N_{i}$ and $N_{j}$ are wired together then
$A[i,j]$ contains the kernel size of the operation ($A[i,j]$ = 1 for the
Identity), if $N_{i}$ and $N_{j}$ are not connected then $A[i,j]=0$. An
illustration can be found in Figure 3.

Figure 3: search space’s encoding visualization: The search space can be
represented as a $4\times 13\times 13$ Adjacency matrix ($A$). This matrix
then will be preprocessed to meet the criteria of the search space.

###  3.4 Framework

We will discuss in details about the framework in this section. The overall
framework is inspired by the BO neural predictor of BANANAS [47], we use
Bayesian Optimization (BO) [16] to search for superior models at each
iteration. The acquisition function is defined as the Expected Improvement
(EI) where the mean and variance are derived from calculating the mean and
variance of the outputs of a group of predictors. These predictors are trained
to predict the performance of a given model.

We first pre-train an autoencoder on the search space so that it could capture
and project the adjacency matrix to the vector space V. Furthermore, we
initialize a population by training a set of $N$ models that were generated
randomly $P_{init}=\\{label_{i}:A_{i}\,|\,i=1,2,...,N_{init}\\}$. A set of
predictors, denoted as $Preds=\\{p_{i},|,i=1,2,...,N_{preds}\\}$, is trained
on the initial population over $E_{p}$ epochs. This training involves
processing adjacency matrices via the autoencoder’s encoder to derive
representations within the V space, serving as inputs for the predictors,
$Preds:V\mapsto label$. At each iteration, the predictors are periodically
retrained with the current population for a limited number of epochs to
prevent overfiting. Subsequently, a provisional population,
$P_{temp}=\\{label^{t}_{i}:A^{t}_{i},|,i=1,2,...,N_{temp}\\}$, is assembled by
selecting $K$ top-performing architectures from the current population and
applying mutation operations at a predetermined probability. To maintain a
balance between exploration and exploitation, this process is complemented by
the introduction of newly generated models into $P_{temp}$. Each model within
this temporary population is then processed through the encoder,
$Encoder:\textbf{A}\mapsto\textbf{V}$, and evaluated by the predictors. The
top $Q$ models are then trained from scratch to determine their true
performance labels. These models are subsequently integrated into the main
population pool. The pseudo code of the framework is shown in 1. It is
important to note that throughout this search process, the pre-trained
autoencoder remains fixed, and only the encoder component is utilized.

Algorithm 1 Pseudo code of the framework

1:pre-trained autoencoder $CAE$ with the search space

2:$P_{init}=\\{label_{i}:A_{i}\,|\,i=1,2,...,N_{init}\\}$

3:Train predictor $Preds=\\{p_{i}\,|\,i=1,2,...,N_{preds}\\}$ with $P_{init}$
for $E_{p}$ epochs

4:Load and freeze the weights of $CAE$

5:$Population=P_{init}$

6:while epoch $<E_{s}$ do

7: Train $Preds$ for $e_{p}$ epochs on $Population$

8: Draw out $K$ models from $Population$

9: Sample $P_{temp}$ by randomly generate or mutate from $K$ best models in
$Population$

10: Feeding $A_{i}\in P_{temp}$ to the encoder to get the vector
$v_{i}\in\textbf{V}$

11: Pass $v_{i}$ through $Preds$ and calculate the acquisition function

12: Train from $Q$ highest score models and add to $Population$

13:end while

####  3.4.1 Autoencoder and predictors

Our search space is represented in the form of a $4\times 13\times 13$ matrix
which is the same format as an image, hence, we propose an autoencoder denoted
as $CAE$ with convolutional layer Equation 2 and global average pooling (GAP)
so that it converts the matrix to a vector space $v_{i}\in\textbf{V}$ that
contains information about the model Equation 3. The role of the decoder is to
help the encoder create a vector space, projecting from discrete to continuous
space, that retains enough information so that the original adjacency matrix
can be reconstructed by the decoder Equation 4. In addition to the
autoencoder, we use a simple linear layer for the implementation of the
predictors. The predictors take the output of the encoder (vector in the
vector space V) and feed through a linear layer with sigmoid as the activation
function to predict the labels of the processing models Figure 4. Using both
autoencoder and predictors prove to have a significant impact on the
performance of the predictors compared to only using predictors, which we
analyze in detail in section 6.

$O=\textrm{{ReLU}}(\textrm{{BatchNorm}}(\textrm{{Conv}}(A)))$ (2)
$v_{i}=\textrm{{GAP}}(O)$ (3)
$\hat{A}=\textrm{{Decoder}}(O)=\textrm{{ReLU}}(\textrm{{Conv}}(OO^{T}))$ (4)

Figure 4: The architecture of our convolutional autoencoder $CAE$ and
predictors.

##  4 Experiments

###  4.1 Datasets

We conduct our experiments on four datasets at four distinct domains, namely:
electrocardiography (ECG), electroencephalography (EEG), human activity
recognition (HAR), and satellite image time series (SITS).

####  4.1.1 PTB-XL

The PTB-XL dataset [44] is a large-scale electrocardiography (ECG) dataset
that contains 21,837 clinical 12-lead ECGs from 18,885 patients, recorded
between October 1989 and June 1996. These records include extensive
annotations by cardiologists, conforming to the SCP-ECG standard, and cover
diagnostic, form, and rhythm statements for a comprehensive array of cardiac
conditions. The dataset is sampled at both 100 Hz and 500 Hz with a length of
10 seconds, however, we follow [35, 6] and only process the former.

The dataset was split into ten folds, with the tenth fold recommended as the
test set and the remaining nine folds as training and validation sets.
Following this, we used the first eight folds for training, the ninth fold for
validation, and the tenth fold for testing. Moreover, we also use the bandpass
filter to remove outlier frequencies, allowing the frequencies in the range of
3-45Hz only, following [6]. The evaluating task is "super-diagnosis" (super)
as it is recorded with the least fluctuation in the performances of the models
[6]. The main metric for this dataset is AUCROC.

####  4.1.2 EEGEyeNet

The EEGEyeNet [21] dataset provides a unique combination of high-density
electroencephalography (EEG) and eye-tracking (ET) data from 356 healthy
adults aged between 18 and 80 years. The dataset was recorded using a
128-channel EEG system and an infrared video-based ET system, both at a
sampling rate of 500 Hz. It includes data from three experimental paradigms
designed to explore the intersection of brain activities and eye movements.
Annotations include the start and end times of saccades, fixations, and
blinks, as well as the position of saccades and average position of fixations.

We use the task "absolute position" similar to [52], which is the hardest task
according to [21] and contains about 35000 data points. The data is split into
three parts with the proportion of 70-15-15 for training, validating, and
testing respectively, which is recommended by the benchmark [21]. The metric
used for this task is RMSE.

####  4.1.3 Smartphone HAR

The smartphone HAR [31] is recorded from 30 volunteers aged 19 to 48, who
performed six different activities: walking, walking upstairs, walking
downstairs, sitting, standing, and lying. These activities were recorded while
each participant wore a Samsung Galaxy S II smartphone, sampling at a rate of
50Hz. The raw sensor signals underwent preprocessing to remove noise and were
segmented using sliding windows of 2.56 seconds with a 50% overlap, resulting
in 128 readings per window.

The dataset contains 10300 data points. We implement the split of 5-2-3 for
our experiments with the test data as the default provided by [31]. We follow
[50] which adopts $F_{1-micro}$ as the main metric.

####  4.1.4 Satellite

The dataset [28] is derived from images taken by the Formosat-2 satellite
gathered at Toulouse area in France. These images are multi-channel, allowing
for the observation of land cover changes across several years. Each pixel
within the images corresponds to a specific geographic area. The primary
objective of using this satellite imagery is to create land cover maps that
aid in geographical surveying. This involves analyzing sequences of pixels
within a particular color channel to classify them into one of 24 different
land cover categories.

We utilize the preprocessed data from [42], which contains 1M data points and
splited into three part with the percentages of 80%, 10%, and 10% for
training, validating, and testing respectively. The main reported metric is
accuracy.

###  4.2 Setups

The following hyperparameters are constant throughout the experiments. The
autoencoder is trained with 800k randomly generated models, with the split of
792k for training and 8k for testing and the learning rate of $1\times e^{-2}$
for 100 epochs. The more models used to pre-train the better, however, we
chose 800k as the autoencoder showing no further noticeable improvement when
we increased the number of models. $L_{2}$ loss is utilized for the pre-train
of autoencoder which evaluates the distance between real and reconstructed
adjacency matrices. We initializes $N_{init}=400$ models then trains the
$N_{preds}=10$ predictors on $P_{init}$ for $E_{p}=200$ epochs with $L_{2}$
loss and learning rate of $1\times e^{-3}$. The framework runs for $E_{s}=50$
iterations. In each iteration, the predictors are trained for $e_{p}=10$
epochs on the current population. The probability of mutation is 0.8 which
modifies 15% of the original adjacency matrix. Other hyperparameters are set
as follow: $K=4$, $N_{temp}=24$, and $Q=4$. More details about training models
for each dataset can be found in the Appendix A.

##  5 Main results

After each run, our population consists of 600 different models and we draw
out the top 10 with the most promising score for evaluation. The best
architecture is chosen by averaging throughout five runs and then selecting
the highest performance. Our results are presented in the following
subsections.

###  5.1 PTB-XL

We conducted an experiment on the "super-diagnosis" task within the PTB-XL
dataset, as detailed in Table 3. Our backbone architecture incorporates both
average and max pooling layers alongside a standard linear classifier, in line
with the structure proposed by [20]. To ensure a balanced evaluation, we
scaled the number of parameters in various renowned Time Series Classification
(TSC) models, including [40, 4, 30], to match our model’s scale. This
normalization allows for a more equitable comparison. The presented results
encapsulate the average outcomes of five distinct runs.

It is observed that our model surpasses the state-of-the-art performance by a
noticeable margin. Other Inception-liked models’ performances vary
significantly. Specifically, IncepSE [6] was the previous best model reaching
0.9343, while XceptionTime performs poorly in the PTB-XL. This shows the
flexibility of Inception models which could be exploited to create models that
are highly tuned to a specific task. Furthermore, our model exhibits a greater
adaptability for larger dataset compared to other receptive-field-based
approaches [40, 4], illustrating the need for precise time location rather
than focusing on only frequency.

Models |  AUC |  Parameters
---|---|---
XceptionTime [30] |  0.8949 |  1.5M
TCN [4] |  0.9309 |  1.7M
OmniScaleCNN [40] |  0.9191 |  1.7M
xResNet [18] |  0.928 |  -
ResNet1D-Wang [46] |  0.930 |  -
InceptionTime [20] |  0.921 |  -
ST-CNN-GAP-5 [1] |  0.9318 |  -
IncepSE [6] |  0.9343 |  1.1M
MSTAR (Our) |  0.9355 |  2.3M
Table 3: The AUCROC in super-diagnosis task on PTB-XL.

###  5.2 Satellite

For this dataset, we also evaluate popular TSC models with the number of
parameters scaled Table 4. The reported results are the mean after five runs.
We chose a linear classifier without any pooling for our backbone CNN as it
has a superior outcome compared to other classifiers. Our experiment reveals
that, for larger datasets, our model consistently has better performance.
MSTAR found a model that exceeds receptive-field methods significantly,
outperforming OmniScaleCNN [40] and TCN [4] by 7,7% and 3% respectively. The
same trend happens when other Inception-liked models’ adaptability greatly
fluctuates, however, this time, XceptionTime [30] exhibits a more dominant
performance. Once again, this implies the potential for a highly tuned
Inception model for a specific time series dataset that has yet to be
discovered.

Models |  Accuracy |  Parameters
---|---|---
XceptionTime [30] |  88,7 |  9.6M
TCN [4] |  86,0 |  10.7M
OmniScaleCNN [40] |  85,9 |  12.0M
xResNet34-deeper [18] |  88,0 |  9.1M
ResNet1D [17] |  86.6 |  -
InceptionTime [20] |  84,8 |  -
MSTAR (Our) |  89.0 |  12.0M
Table 4: The accuracy of TSC models on Satellite.

###  5.3 Smartphone HAR

Following [50], we report the top-1 $F_{1-micro}$ out of 10 runs and include
their results. In alignment with our previous approach, we meticulously
calibrated the number of parameters in renowned TSC models to a uniform scale,
ensuring a fair and balanced comparison. Table 5 demonstrates the adaptation
of our search space on a relatively small dataset. Our discovered model
surpasses the previous state-of-the-art [50] in the human activity recognition
task. This accomplishment underscores the robustness of our framework,
particularly in scenarios characterized by limited data availability.

Models |  $F_{1-micro}$ |  Parameters
---|---|---
XceptionTime [30] |  0.934 |  3.5M
TCN [4] |  0.876 |  2.2M
OmniScaleCNN [40] |  0.932 |  1.7M
xResNet19-deep [18] |  0.931 |  5.0M
DeepConvLSTM [27] |  0.910 |  -
InnoHAR [17] |  0.945 |  -
InceptionTime [20] |  0.944 |  1.8M
MSTAR (Our) |  0.953 |  3.7M
Table 5: The $F_{1-micro}$ on Smartphone HAR.

###  5.4 EEGEyeNet

Finally, we show that our model can be used as a backbone for transformer
[43]. We follow previous works on the dataset [21, 52] where the means of RMSE
metric are reported after 5 runs. [52] point out that, Vision transformer
(VIT) [13] can be paired effectively with CNN. Furthermore, the pre-trained
weight of VIT provides an essential pattern recognition mechanism, which
substantially enhances the ability of the hybrid model (VIT2EEG [52]) to
predict the eye gaze position. We also use VIT and pre-trained VIT for this
experiment while using MSTAR and InceptionTime [20] as the backbone.
InceptionTime is chosen for its similarity to our model in terms of the
overall macro structure as an Inception-liked architecture, suitable for a
backbone. Moreover, we conduct two separate NAS runs for pre-trained and base
VIT respectively and the best models in both categories are reported. The
hyperparameters for the pre-trained VIT are the same between each backbone in
this experiment. Note that our contribution is not to provide the best result
as we could utilize more advanced VIT-based models or use tuned
hyperparameters for VIT. Instead, our intention is to demonstrate the ability
of our backbone CNN to perform well on tasks other than classification and the
flexibility to work with transformers. The results in Table 6 show that both
models, MSTAR-VIT-base and MSTAR-VIT-pre-trained, surpass the previous state-
of-the-art by substantial margins. Notably, our base model (without pre-
trained weights) exceeds the pre-trained EEGVIT, further underscoring the
potential of using our backbone with transformers.

We also evaluate the performance of InceptionTime as a backbone CNN. Although
the impact of VIT on the base model is limited compared to InceptionTime, the
weights of VIT trained on image data boost the performance of InceptionTime-
VIT-pre-trained significantly. This could be that the InceptionTime is not
suitable to be a backbone but the pre-trained weights, with crucial additional
information, support a strong performance.

Models |  RMSE
---|---
Naive |  123,3
XceptionTime [30] |  78,7
PyramidalCNN [21] |  73,9
InceptionTime [20] |  70,7
InceptionTime-VIT-base |  69,5
InceptionTime-VIT-pre-trained |  54,2
VIT-base [13] |  61,5
VIT-pre-trained [13] |  58,1
EEGVIT [52] |  61,7
EEGVIT-pre-trained [52] |  55,4
MSTAR-VIT-base (Our) |  53,3
MSTAR-VIT-pre-trained (Our) |  51,3
Table 6: The results on the EEGEyeNet dataset. The numbers of parameters are
not presented because most of the results are concluded from [52, 21].

##  6 Ablation study

We believe that two problems should be discussed further in this section. The
first problem is the impact of receptive fields on the frequency and time
precision. Moreover, we provide proof of the importance of accurately
identifying time to the performance of our TSC model. Secondly, we focus on
the effect of using CAE compared to the prominent Variational Autoencoder
(VAE) and the accuracy of our predictor.

###  6.1 Receptive fields and time resolution

We first calculate the receptive fields of each node in our state-of-the-art
model on the dataset PTB-XL Table 7. Afterward, we visualize the time and
frequency of the feature maps in each node via the spectrograms generated by
CWT. The results are presented in Figure 5 where we show the spectrogram at
layers 1, 2, and 6 of the last 6 nodes excluding the output node. These nodes
were chosen because each of them contains a distinct range of receptive
fields. At first glance, the plot contains multiple figures that all have
vertical dark color stripes, which is expected as ECG is periodical and the
stripes mean no heart pulse. At the first layer, it is observed that the
broader the receptive fields span, the less rigorous the stripes are. In
contrast, larger receptive fields mean stronger frequency intensities, this is
indicated by the brighter color. Specifically, $N_{6}$ only has the receptive
field of 1, which means its feature maps are the most similar to the ECG
signal with distinctive stripes and high time resolution. While $N_{7}$
contains more and broader receptive fields, hence the range of frequencies
captured is wider and stronger but the precision regarding time is degraded.
This aligns with our motivation subsection 3.1. Furthermore, the same pattern
happens with the next layers. As we dive deeper, the receptive fields get much
larger and more complex, which results in the downgrade of time resolution and
the features are more abstract as the stripes have been blurred out. It is
worth noting that, the nodes with modest receptive fields in the lower layer
still have small receptive fields compared to other nodes at the higher layer.
Thus, the rank, in terms of brightness of the feature maps (indicating the
magnitude of the frequencies) within the set of nodes, is still the same for
every layer.

Node |  Receptive fields
---|---
1 |  Not connected
2 |  1
3 |  1
4 |  9
5 |  9
6 |  1
7 |  1, 3, 17, 27, 39
8 |  1, 3, 9
9 |  3, 9
10 |  1, 3, 5, 9, 11
11 |  1, 9, 17
Table 7: The receptive fields of every node at the first layer of our MSTAR
model on PTB-XL. Figure 5: Spectrograms of the feature maps created by
Continuous Wavelet Transform (CWT) from MSTAR’s model on PTB-XL [44]. The
larger the receptive field, the stronger the frequency intensities. However,
this is the trade-off for time resolution.

We show the importance of time resolution in the first layer in Figure 6.
Layer 1 is chosen because of its clear distinction regarding time while still
having a variety of ranges of receptive fields. We evaluate the contributions
by taking the mean of $L_{1}$ norm of the scores for the output of each node
given by Integrated Gradient [36], all while zeroing out other nodes. The
higher the scores, the greater the contributions to the overall
classification. The results in Figure 6 show that the first five nodes have
significant gradient quality, which is expected as they also serve as the
skip-connection. But most importantly, the last five nodes ($N_{7}$ to
$N_{11}$), although having different contributions among the classes, none of
the contributions can be disregarded. This implies that both time and
frequency resolution are essential to our model, providing critical
information for classification.

Figure 6: The scores of the nodes in layer 1 from MSTAR’s model on PTB-XL with
Integrated Gradient [36]. None of the node has a neglectable contribution.

Lastly, with the prior knowledge of the rank in terms of time resolution still
remaining the same at each layer, we remove nodes from the adjacency matrix to
see their impacts on the performance of our model. We zero out several nodes
that deliver the most accurate time view, which includes $N_{8}$, $N_{9}$, and
$N_{10}$. We do not select $N_{6}$ as it serves as the skip connection to the
model and would already have an important role anyway. In contrast, $N_{7}$
and $N_{11}$ are included in our experiment, as they are among the most
critical nodes in terms of frequency resolution, for the sake of comparison.
Furthermore, discarding these nodes would not change the structure of another
node because they are at the last layer in the cell besides the output node,
hence, would not affect the performance if it is not important. Table 8 shows
that without the information of precise time location, our model performs
noticeably worse, commensurable with the loss of AUC of removing $N_{7}$ and
$N_{11}$. This further implies the essential of time resolution in TSC.

Removing node | $N_{7}$ | $N_{8}$ | $N_{9}$ | $N_{10}$ |  $N_{9}$ and $N_{10}$ | $N_{11}$
---|---|---|---|---|---|---
AUC | 0.9326 | 0.9338 | 0.9352 | 0.9350 | 0.9332 | 0.9345
Table 8: The performances of our model when removing $N_{8}$, $N_{9}$, and
$N_{10}$ respectively when comparing with $N_{7}$ and $N_{11}$. Our model is
trained from scratch, the mean AUC is recorded after 5 runs.

###  6.2 Impact of autoencoder

The utilization of autoencoder has been proofed to have a significant impact
on various frameworks [51]. However, different search spaces, that have unique
representations, require specific types of autoencoder. Consequently, we show
that our Convolutional Autoencoder (CAE) is more suitable than the popular
counterpart Variational Autoencoder (VAE). We start by constructing a VAE, of
which architecture is demonstrated in Appendix B. The standard $L_{2}$ with
$KL$ divergence are adopted as the loss function of the VAE. Furthermore, both
of the autoencoders are trained with the setups refered in subsection 4.2
which ultimately evaluate the performance by the $L_{2}$ metric. The results
are presented in Table 9.

Autoencoder | CAE | VAE
---|---|---
MSE Loss | 5.389 | 14.202
Table 9: The $L_{2}$ loss of both autoencoders on 8K randomly initialized
adjacency matrices. The CAE is better at reconstructing our search space
representation.

We also show the performance of our predictors in measuring the performances
of models based on the adjacency matrices. We conduct an experiment with our
standard CAE and 10 predictors that compare with modified predictors. These
predictors are integrated with a Convolutional block with a pooling layer at
the stem so that they can effectively process the matrices. We also make
modifications so that the new predictors would reach their best potential as
shown in Appendix B. The comparison is shown in Table 10. The Spearman
correlation coefficient is adopted as the main metric. We randomly initialize
800 models and train them from scratch across all 4 datasets. We then split
with the proportion of 9-1 for training, and validation respectively. The
results demonstrate that the pre-trained CAE boosted the accuracy of the
predictors substantially, essential for our framework to search on different
time series datasets.

Method | CAE and linear predictors | Convolutional predictors
---|---|---
smartphone HAR | 0.730 | 0.382
PTB-XL | 0.528 | 0.246
Satellite | 0.321 | 0.156
EEGEyeNet | 0.242 | 0.238
Table 10: The effectiveness of our autoencoder on the precision of the neural
predictors measured in Spearman Correlation. The baseline is random guessing
which has the correlation of zero.

##  7 Conclusion

###  7.1 Future works

Our research can be reproduced for follow-up works. We encourage improvements
in the framework to improve the efficiency and accuracy of the search.
Moreover, another promising direction is to incorporate more advanced
operations such as convolution with dilation and groups or the squeeze and
excitation module [19]. Lastly, our approach to integrating transformer is
still naive, we believe this topic could spark many renovating ideas, and open
more possibilities in TSC.

###  7.2 Conclusion

In this study, we propose a multi-scale search space that comes with a novel
framework to search for the optimal architecture as well as receptive field
range for time series datasets. Our proposed MSTAR backbone overcomes the
formidable scalability challenges, leveraging automated searching without
human intervention and accomplishing optimistic results. Our experiments
reveal that while frequency extraction is crucial, time resolution also plays
an important role, providing valuable information in the TSC field. We believe
that these findings could provide a broader view of time series processing,
incorporating larger models to tackle more extended datasets.

##  8 Acknowledgements

This research is supported by the VinUni-Illinois Smart Health Center,
VinUniversity.

## References

  * [1] Atul Anand, Tushar Kadian, Manu Kumar Shetty, and Anubha Gupta.  Explainable ai decision model for ecg data of cardiac disorders.  Biomedical Signal Processing and Control, 75:103584, 2022.
  * [2] Noor Awad, Neeratyoy Mallik, and Frank Hutter.  Differential evolution for neural architecture search, 2021.
  * [3] Anthony J. Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Eamonn J. Keogh.  The UEA multivariate time series classification archive, 2018.  CoRR, abs/1811.00075, 2018.
  * [4] Shaojie Bai, J. Zico Kolter, and Vladlen Koltun.  An empirical evaluation of generic convolutional and recurrent networks for sequence modeling, 2018.
  * [5] Andrew Brock, Theodore Lim, J. M. Ritchie, and Nick Weston.  Smash: One-shot model architecture search through hypernetworks, 2017.
  * [6] Tue Cao, Nhat Tran, Le Nguyen, Hung Nguyen, and Hieu Pham.  Incepse: Leveraging inceptiontime’s performance with squeeze and excitaion mechanism in ecg analysis.  In Proceedings of the 12th International Symposium on Information and Communication Technology, SOICT ’23, page 578–584, New York, NY, USA, 2023. Association for Computing Machinery.
  * [7] Aristeidis Christoforidis, George Kyriakides, and Konstantinos Margaritis.  A novel evolutionary algorithm for hierarchical neural architecture search, 2023.
  * [8] Hoang Anh Dau, Anthony J. Bagnall, Kaveh Kamgar, Chin-Chia Michael Yeh, Yan Zhu, Shaghayegh Gharghabi, Chotirat Ann Ratanamahatana, and Eamonn J. Keogh.  The UCR time series archive.  CoRR, abs/1810.07758, 2018.
  * [9] Angus Dempster, François Petitjean, and Geoffrey I. Webb.  Rocket: exceptionally fast and accurate time series classification using random convolutional kernels.  Data Mining and Knowledge Discovery, 34(5):1454–1495, July 2020.
  * [10] Angus Dempster, Daniel F. Schmidt, and Geoffrey I. Webb.  Minirocket: A very fast (almost) deterministic transform for time series classification.  In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, KDD ’21, page 248–257, New York, NY, USA, 2021. Association for Computing Machinery.
  * [11] Angus Dempster, Daniel F. Schmidt, and Geoffrey I. Webb.  Hydra: Competing convolutional kernels for fast and accurate time series classification.  Data Min. Knowl. Discov., 37(5):1779–1805, may 2023.
  * [12] Xuanyi Dong and Yi Yang.  Nas-bench-201: Extending the scope of reproducible neural architecture search, 2020.
  * [13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [14] Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter.  Neural architecture search: a survey.  J. Mach. Learn. Res., 20(1):1997–2017, jan 2019.
  * [15] Navid Mohammadi Foumani, Lynn Miller, Chang Wei Tan, Geoffrey I. Webb, Germain Forestier, and Mahsa Salehi.  Deep learning for time series classification and extrinsic regression: A current survey, 2023.
  * [16] Peter I. Frazier.  A tutorial on bayesian optimization, 2018.
  * [17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  CoRR, abs/1512.03385, 2015.
  * [18] Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li.  Bag of tricks for image classification with convolutional neural networks.  CoRR, abs/1812.01187, 2018.
  * [19] Jie Hu, Li Shen, Samuel Albanie, Gang Sun, and Enhua Wu.  Squeeze-and-excitation networks, 2019.
  * [20] Hassan Ismail Fawaz, Benjamin Lucas, Germain Forestier, Charlotte Pelletier, Daniel F. Schmidt, Jonathan Weber, Geoffrey I. Webb, Lhassane Idoumghar, Pierre-Alain Muller, and François Petitjean.  Inceptiontime: Finding alexnet for time series classification.  Data Min. Knowl. Discov., 34(6):1936–1962, nov 2020.
  * [21] Ard Kastrati, Martyna Beata Płomecka, Damián Pascual, Lukas Wolf, Victor Gillioz, Roger Wattenhofer, and Nicolas Langer.  Eegeyenet: a simultaneous electroencephalography and eye-tracking dataset and benchmark for eye movement prediction, 2021.
  * [22] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton.  Imagenet classification with deep convolutional neural networks.  Commun. ACM, 60(6):84–90, may 2017.
  * [23] Hyeon Kyu Lee and Young-Seok Choi.  Application of continuous wavelet transform and convolutional neural network in decoding motor imagery brain-computer interface.  Entropy, 21(12):1199, 2019.
  * [24] Lightning-AI.  Pytorch lightning, 2024.
  * [25] Hanxiao Liu, Karen Simonyan, and Yiming Yang.  Darts: Differentiable architecture search, 2019.
  * [26] Matthew Middlehurst, James Large, Michael Flynn, Jason Lines, Aaron Bostrom, and Anthony Bagnall.  Hive-cote 2.0: A new meta ensemble for time series classification.  Mach. Learn., 110(11–12):3211–3243, dec 2021.
  * [27] Francisco Javier Ordóñez and Daniel Roggen.  Deep convolutional and lstm recurrent neural networks for multimodal wearable activity recognition.  Sensors, 16(1), 2016.
  * [28] François Petitjean, Jordi Inglada, and Pierre Gancarski.  Satellite image time series analysis under time warping.  IEEE Transactions on Geoscience and Remote Sensing, 50(8):3081–3095, 2012.
  * [29] Hieu Pham, Melody Y. Guan, Barret Zoph, Quoc V. Le, and Jeff Dean.  Efficient neural architecture search via parameter sharing, 2018.
  * [30] Elahe Rahimian, Soheil Zabihi, Seyed Farokh Atashzar, Amir Asif, and Arash Mohammadi.  Xceptiontime: Independent time-window xceptiontime architecture for hand gesture classification.  In ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1304–1308, 2020.
  * [31] Reyes-Ortiz, Jorge, Anguita, Davide, Ghio, Alessandro, Oneto Luca, Parra, and Xavier.  Human activity recognition using smartphones.  UCI Machine Learning Repository, 2012.
  * [32] Patrick Schäfer.  The boss is concerned with time series classification in the presence of noise.  Data Min. Knowl. Discov., 29(6):1505–1530, nov 2015.
  * [33] Ahmed Shifaz, Charlotte Pelletier, François Petitjean, and Geoffrey I. Webb.  TS-CHIEF: A scalable and accurate forest algorithm for time series classification.  CoRR, abs/1906.10329, 2019.
  * [34] Steven W. Smith.  The scientist and engineer’s guide to digital signal processing.  California Technical Publishing, USA, 1997.
  * [35] Nils Strodthoff, Patrick Wagner, Tobias Schaeffter, and Wojciech Samek.  Deep learning for ecg analysis: Benchmarks and insights from ptb-xl.  IEEE Journal of Biomedical and Health Informatics, 25(5):1519–1528, 2021.
  * [36] Mukund Sundararajan, Ankur Taly, and Qiqi Yan.  Axiomatic attribution for deep networks, 2017.
  * [37] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna.  Rethinking the inception architecture for computer vision.  CoRR, abs/1512.00567, 2015.
  * [38] Chang Wei Tan, Angus Dempster, Christoph Bergmeir, and Geoffrey I. Webb.  Multirocket: Multiple pooling operators and transformations for fast and effective time series classification.  Data Min. Knowl. Discov., 36(5):1623–1646, sep 2022.
  * [39] Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V. Le.  Mnasnet: Platform-aware neural architecture search for mobile, 2019.
  * [40] Wensi Tang, Guodong Long, Lu Liu, Tianyi Zhou, Michael Blumenstein, and Jing Jiang.  Omni-scale CNNs: a simple and effective kernel size configuration for time series classification.  In International Conference on Learning Representations, 2022.
  * [41] Larisa G. Tereshchenko and Mark E. Josephson.  Frequency content and characteristics of ventricular conduction.  Journal of Electrocardiology, 48(6):933–937, 2015.
  * [42] Renbo Tu, Nicholas Roberts, Mikhail Khodak, Junhong Shen, Frederic Sala, and Ameet Talwalkar.  NAS-bench-360: Benchmarking neural architecture search on diverse tasks.  In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
  * [43] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need, 2023.
  * [44] Patrick Wagner, Nils Strodthoff, Ralf-Dieter Bousseljot, Dieter Kreiseler, Fatima I Lunze, Wojciech Samek, and Tobias Schaeffter.  Ptb-xl, a large publicly available electrocardiography dataset.  Scientific data, 7(1):154, 2020.
  * [45] Tao Wang, Changhua Lu, Yining Sun, Mei Yang, Chun Liu, and Chunsheng Ou.  Automatic ecg classification using continuous wavelet transform and convolutional neural network.  Entropy, 23(1):119, 2021.
  * [46] Zhiguang Wang, Weizhong Yan, and Tim Oates.  Time series classification from scratch with deep neural networks: A strong baseline, 2016.
  * [47] Colin White, Willie Neiswanger, and Yash Savani.  BANANAS: bayesian optimization with neural architectures for neural architecture search.  CoRR, abs/1910.11858, 2019.
  * [48] Colin White, Mahmoud Safari, Rhea Sukthanker, Binxin Ru, Thomas Elsken, Arber Zela, Debadeepta Dey, and Frank Hutter.  Neural architecture search: Insights from 1000 papers, 2023.
  * [49] Baoguo Xu, Linlin Zhang, Aiguo Song, Changcheng Wu, Wenlong Li, Dalin Zhang, Guozheng Xu, Huijun Li, and Hong Zeng.  Wavelet transform time-frequency image and convolutional network-based motor imagery eeg classification.  Ieee Access, 7:6084–6093, 2018.
  * [50] Cheng Xu, Duo Chai, Jie He, Xiaotong Zhang, and Shihong Duan.  Innohar: A deep neural network for complex human activity recognition.  IEEE Access, 7:9893–9902, 2019.
  * [51] Shen Yan, Yu Zheng, Wei Ao, Xiao Zeng, and Mi Zhang.  Does unsupervised architecture representation learning help neural architecture search?, 2020.
  * [52] Ruiqi Yang and Eric Modesitt.  Vit2eeg: Leveraging hybrid pretrained vision transformers for eeg data, 2023.
  * [53] Chris Ying, Aaron Klein, Esteban Real, Eric Christiansen, Kevin Murphy, and Frank Hutter.  Nas-bench-101: Towards reproducible neural architecture search, 2019.
  * [54] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson.  How transferable are features in deep neural networks?  In Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2, NIPS’14, page 3320–3328, Cambridge, MA, USA, 2014. MIT Press.
  * [55] Matthew D. Zeiler and Rob Fergus.  Visualizing and understanding convolutional networks.  In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision – ECCV 2014, pages 818–833, Cham, 2014. Springer International Publishing.
  * [56] Barret Zoph and Quoc V. Le.  Neural architecture search with reinforcement learning, 2017.

##  Appendix A Training setups

Our detailed training setups are presented as follows Table 11. Note that for
smartphone HAR, the $F_{1-micro}$ fluctuated widely and the size of the
dataset is limited, hence, we evaluate the architectures for NAS by randomly
segmenting the data into 5 folds and implementing 5-fold testing. Moreover, we
use Pytorch Lighting API trainer [24] with the learning rate of $1e^{-3}$ and
implement stochastic weight averaging with $1e^{-4}$ learning rate while
running NAS. The setups to evaluate every model after NAS (including our
model) in Table 5 are presented in Table 11. For other datasets, we employ
training and validating on the default training and validating set, the test
set is utilized to evaluate after we run the architecture search for a fair
comparison. The training for the remaining datasets is the same throughout the
search and benchmarking.

Setups | lr | Optimizer | Scheduler | Population - | Loss - | Others
---|---|---|---|---|---|---
|  |  |  | Datasets | Criterion | parameters
Autoencoder | $1e^{-2}$ | Adam | OneCycle: | 792K - 8K |  $L_{2}$ loss | Epochs:100
pre-trained |  |  |  | (models) |  | Batchsize: 256
Predictors x 10 | $1e^{-3}$ | Adam | None | $P_{init}$ |  $L_{2}$ loss | Epochs: 200
pre-trained |  |  |  | (400 models) |  | Batchsize: 16
Predictors x 10 | $1e^{-3}$ | Adam | None | $Population$ |  $L_{2}$ loss | Epochs: 10
each Iteration |  |  |  |  |  | Batchsize: 16
PTB-XL | $1e^{-2}$ | AdamW | OneCycle | PTB-XL | Binary cross | Epochs: 18
training |  |  |  |  | entropy | Batchsize: 128
EEGEyeNet | $1e^{-3}$ | Adam | OneCycle | EEGEyeNet |  $L_{2}$ loss | Epochs: 30
training |  |  |  |  |  | Batchsize: 128
Smartphone | $1e^{-4}$ | AdamW | OneCycle | Smartphone | Cross | Epochs: 30
HAR training |  |  |  | HAR | entropy | Batchsize: 128
Satellite | $2e^{-3}$ | AdamW | OneCycle | Satellite | Cross | Epochs: 12
training |  |  |  |  | entropy | Batchsize: 1024
Table 11: The training setups

##  Appendix B Hyperparameters

###  B.1 Multi-scale experiment

We create six models with multiple scales as described in subsection 3.1. The
hyperparameters of all the architectures are the same and are inspired by
InceptionTime [20], which includes five layers with 128 channels, each layer
contains the standard convolution, batch normalization, and ReLU as the
activation. We use both global average and global max pooling and a linear
layer as the classification.

###  B.2 VAE and convolutional predictor

The architecture of the VAE that we build for subsection 6.2 is shown in this
section. Equation 5 and Equation 6 show the information flow of our encoder,
where $A_{j}$ is the adjacency matrix of the $j$th operation. The hidden
representation at layer $i$ is denoted as $H_{i}$, it can be further split
into four parts $\\{H_{i,\,j}\,|\,j\in\\{1,..\,4\\}\\}$ at the last dimension,
carry global information of other operations when multiplying with $A_{j}$.
The MLP consists of linear, batch normalization, and a leaky ReLU activation
function. We chose the leaky ReLU and also the overall architecture because
they are the best setups in our empirical experiments. The representation in V
is derived from $V\sim\mathcal{N}(\mu,\sigma^{2})$ where $\mu$ and $\sigma$
are acquired from feeding the last $H$ through two linear layers respectively.
On the other hand, the decoder is described in Equation 7 and Equation 8 where
$\hat{A}$ is the reconstructed matrix.

$H_{0}=\sum_{j=0}^{3}A_{j}\,W_{encoder}$ (5)
$H_{i+1}=\textrm{{MLP}}(\textrm{{Concat}}_{j}((1+\epsilon)H_{i}+A_{j}\,H_{i,\,j}))$
(6) $Z_{j}=V\,W_{decoder}$ (7)
$\hat{A}=\textrm{{ReLU}}(\textrm{{Concat}}_{j}(Z_{j}\,Z_{j}^{T}))$ (8)

The convolutional predictor is constructed by using a simple block, which
includes convolution, batch normalization, global average pooling, and several
feed-forward layers with $\tanh$ activation. This architecture resembles the
combination of our CAE and linear predictor. The hyperparameters of both VAE
and predictor are tuned so that they leverage the best performances. For the
variational autoencoder, we implement five layers with a hidden dimension of
128 and the MLP contains two linear layers. The predictors have 2300 channels
for convolution (same as the CAE) and contain four feed-forward layers with
the number of neurons decreasing by half each layer, starting with 2300.