#  INR-TH-2023-004 Light-shining-through-wall cavity setups for probing ALPs

Dmitry Salnikov1,2, Petr Satunin1,2, D. V. Kirpichnikov1, Maxim Fitkevich1,3
1 Institute for Nuclear Research of the Russian Academy of Sciences,
60th October Anniversary Prospect, 7a, 117312 Moscow, Russia
2 Lomonosov Moscow State University,
Leninskiye Gory, 119991 Moscow, Russia
3 Moscow Institute of Physics and Technology,
Institutskiy per., 9, 141701 Dolgoprudny, Moscow Region, Russia e-mail:
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

We discuss the aspects of axion-like-particles (ALPs) searches with Light-
Shining-through-Wall (LSW) experimental setups consisted of two radio-
frequency cavities. We compare the efficiencies of three setups which involve
the cavity pump modes and external magnetic fields. Additionally, we discuss
the sensitivity dependence both on the relative position of two cylindrical
cavities and on their radius-to-length ratio.

##  1 Introduction

Light feebly-interacting pseudoscalar particles appear in modern particle
physics in various ways. Originally, a pseudoscalar particle called an axion
was proposed in late 1970s to explain the strong CP problem in quantum
chromodynamics [1, 2]. More general axion-like-particles (ALPs) are motivated
by the string theory and appear in its low-energy phenomenological description
[3, 4]. In addition to the motivation for the particle physics models, axions
and ALPs are of a great interest in cosmology because they could make up a
significant fraction of the dark matter in the Universe [5, 6, 7].

The Lagrangian for interacting ALPs and photons can be written as follows

$\mathcal{L}=-\frac{1}{4}F_{\mu\nu}F^{\mu\nu}+\frac{1}{2}\,\partial_{\mu}a\,\partial^{\mu}a-\frac{1}{2}\,m_{a}^{2}a^{2}-\frac{1}{4}\,g_{a\gamma\gamma}\,a\,F_{\mu\nu}\tilde{F}^{\mu\nu}\;,$
(1)

where $F_{\mu\nu}$ is the electromagnetic tensor and
$\tilde{F}^{\mu\nu}=\frac{1}{2}\epsilon^{\mu\nu\alpha\beta}F_{\alpha\beta}$ is
its dual, $a$ is the ALP field of mass $m_{a}$ with dimensionful photon-axion
coupling $g_{a\gamma\gamma}$. The natural system of units $\hbar=c=k_{B}=1$ is
used. Generally, $m_{a}$ and $g_{a\gamma\gamma}$ are treated as independent
parameters.

A popular strategy of ALP searches is related to the cosmological (dark
matter) and astrophysical probing. These ALPs can be detected by ground-based
haloscopes (detection of dark matter ALPs) and helioscopes (ALPs can be
produced hypothetically in the Sun) [8] (see e.g. [9] for a recent review).

Another approach to probing ALPs implies both their production and detection
in a laboratory, and usually called Light-Shining-through-Wall (LSW)
experiments [10, 11, 12, 13]. The LSW setups consist of two cavities separated
by a non-transparent wall. ALPs are produced in the first cavity by
interaction of electromagnetic field components. Generated ALPs can pass
through the wall and convert back to photons in the detection cavity. High
intensity of initial electromagnetic field and the resonant amplification for
the signal inside the cavities are required because of the extremely small
coupling $g_{a\gamma\gamma}$. Two wavelength ranges of EM fields are
applicable to LSW: the optical range setup including high intensity lasers and
the radio range setup consisting of radio frequency cavities with high quality
factors. Both ideas were realised in the experiments, ALPS (optical) [14] and
CROWS (radio) [15]. These experiments set the bound $g_{a\gamma\gamma}\simeq
10^{-7}\,\mbox{GeV}^{-1}$ for a wide range of ALP masses. However, this bound
is three orders of magnitude weaker than the CAST helioscope limit
$g_{a\gamma\gamma}\lesssim 6\times 10^{-11}\,\mbox{GeV}^{-1}$, [8]. For the
moment, the ALPS-II laser experiment [16] is under construction and its
projected sensitivity exceeds CAST level.

In addition, the LSW radio experiments aimed at the ALP searches are of a
great interest [17]. Recently, several proposals with LSW radio cavities
appeared in the literature including superconducting radio frequency (SRF)
cavities [18, 19, 20]. In this letter we compare different LSW cavity setups
including modification of the CROWS [13, 15]. Specifically, we study three
setups: (i) an electromagnetic pump mode plus static magnetic field in the
emitter cavity, static magnetic field in the receiver cavity [13]; (ii) two
electromagnetic pump modes in the emitter cavity; an electromagnetic pump mode
in the receiver cavity [18]; (iii) two electromagnetic pump modes in the
emitter cavity, static magnetic field in receiver cavity [19].

Figure 1:  Two types of the experimental setup consisting of two cylindrical
cavities with (left panel) coaxial or (right panel) parallel orientation and
screened by axion-penetrable wall. Wavy and straight lines represent
electromagnetic field (cavity mode or magnetic field) and ALPs respectively.

Another aspect of our analysis is geometry of the setup which can be adjusted
in order to achieve higher sensitivity to ALPs parameters. We study transfer
of ALPs from the emitter to the receiver for all aforementioned designs
(i-iii) and discuss their optimal configuration, either coaxial or parallel
(see e. g. Fig. 1 for detail). Further, we investigate $g_{a\gamma\gamma}$
sensitivity dependence on the radius-to-length ratio of production cylindrical
cavity.

The paper is organized as follows. The section 2 is dedicated to the axion
electrodynamics involving both the emitter and receiver cavities. In section 3
we derive the expected reach of the considered setups. We summarize our
results in section 4.

##  2 Axion electrodynamics.

We briefly review the axion electrodynamics with the Lagrangian (1). The
Euler-Lagrange equation for the ALP field reads,

$(\partial_{\mu}\partial^{\mu}+m_{a}^{2})\,a=-\frac{1}{4}\,g_{a\gamma\gamma}\,F_{\mu\nu}\tilde{F}^{\mu\nu}\;,$
(2)

while the Maxwell’s equations with an ALP-induced current read,

$\partial_{\mu}F^{\mu\nu}=-g_{a\gamma\gamma}\,\partial_{\mu}a\,\tilde{F}^{\mu\nu}\;.$
(3)

One can rewrite Eqs. (2) and (3) in terms of the electric and magnetic fields,

$(\partial_{\mu}\partial^{\mu}+m_{a}^{2})\,a=g_{a\gamma\gamma}(\vec{E}\cdot\vec{B})\;,$
(4)
$(\vec{\nabla}\cdot\vec{E})=\rho_{a}\;,\quad\quad[\vec{\nabla}\times\vec{B}]=\dot{\vec{E}}+\vec{j}_{a}\;,$
(5)

where the density of charge $\rho_{a}$ and current $\vec{j}_{a}$ are
respectively given by

$\rho_{a}=-g_{a\gamma\gamma}(\vec{\nabla}a\cdot\vec{B})\;,\quad\vec{j}_{a}=g_{a\gamma\gamma}([\vec{\nabla}a\times\vec{E}]+\dot{a}\vec{B})\;.$
(6)

Aforementioned equations describe a way to produce ALPs by the electromagnetic
field and the approach to detection of the ALP field in presence of background
electromagnetic field. We further elaborate on this idea to compare the
sensitivities of three types of the LSW setup for probing ALPs.

###  2.1 The emitter cavity

In this section, we consider the ALPs production. It is worth noting that Eq.
(4) implies the invariant
$F_{\mu\nu}\tilde{F}^{\mu\nu}=-4(\vec{E}\cdot\vec{B})$ should be non-zero in
order to produce ALPs by the electromagnetic field. Following this
requirement, we consider two options for the production of ALPs using RF
cavities:

  * •

a normally conducting RF cavity with a single pump mode with frequency
$\omega_{0}$ immersed in a strong static magnetic field $\vec{B}_{\rm ext}$.
We use the notation MF emitter (i. e. pump mode (M) $+$ magnetic field (F))
for this case throughout the paper;

  * •

a superconducting RF cavity with two pump modes at frequencies $\omega_{1,2}$.
We use notation MM emitter (pump mode (M) $+$ pump mode (M)) for this setup.

It is worth mentioning, that in the steady regime both EM-source (r.h.s of Eq.
4) and the induced axion field have the same frequency $\omega_{a}$. For the
MF emitter case, the source function in the Eq. (4) contains a single
component oscillating at the frequency $\omega_{a}=\omega_{0}$. However, for
the MM emitter case, there are two components at frequencies
$\omega_{a}=\omega_{\pm}=\omega_{2}\pm\omega_{1}$ ($\omega_{2}>\omega_{1}$).
As a result, each particular combination of the field for both MF emitter and
MM emitter can be written in the general form,

$f(t,\vec{x})=g_{a\gamma\gamma}E^{\rm em}_{0}B^{\rm em}_{0}\,{\rm
Re}\,\left[(\vec{\cal E}\cdot\vec{\cal B})(\vec{x})\cdot
e^{-i\omega_{a}t}\right]\;,$ (7)

where $E^{\rm em}_{0},B^{\rm em}_{0}$ are typical values of the emitter EM
fields, $(\vec{\cal E}\cdot\vec{\cal B})(\vec{x})$ is a dimensionless function
determined by the production approach.

For the RF cavity that emits the ALPs (MF emitter), $E_{0}^{\rm em}$ is a
typical amplitude of pump mode taken on the cavity wall111The peak amplitude
is greater by the factor of ${\cal O}(1)$ and depends on the particular mode
and cavity geometry., $B_{0}^{\rm em}$ is a magnitude of the static magnetic
field, and the typical combination of the normalized fields in Eq. (7) can be
written as follows

$(\vec{\cal E}\cdot\vec{\cal B})(\vec{x})=(\vec{\cal
E}_{0}(\vec{x})\cdot\vec{\cal B}_{\rm ext})\;,$ (8)

where $\vec{\cal E}_{0}(\vec{x})$ is a dimensionless electric field of the
pump mode, and $\vec{\cal B}_{\rm ext}$ is a unit vector that is collinear to
the magnetic field direction.

For the MM emitter case, $E_{0}^{\rm em}$ and $B_{0}^{\rm em}$ represent the
surface electric and magnetic fields amplitudes of both pump modes and the
dimensionless functions in Eq. (7) can be written for $\omega_{a}=\omega_{+}$
and $\omega_{a}=\omega_{-}$ respectively as follows

$\displaystyle(\vec{\cal E}\\!\cdot\\!\vec{\cal
B})_{+}(\vec{x})=\\!\dfrac{1}{2}\\!\left[(\vec{\cal
E}_{1}(\vec{x})\\!\cdot\\!\vec{\cal B}_{2}(\vec{x}))\\!+\\!(\vec{\cal
E}_{2}(\vec{x})\\!\cdot\\!\vec{\cal B}_{1}(\vec{x}))\right],$ (9)
$\displaystyle(\vec{\cal E}\\!\cdot\\!\vec{\cal
B})_{-}(\vec{x})=\\!\dfrac{1}{2}\\!\left[(\vec{\cal
E}^{*}_{1}(\vec{x})\\!\cdot\\!\vec{\cal B}_{2}(\vec{x}))\\!+\\!(\vec{\cal
E}_{2}(\vec{x})\\!\cdot\\!\vec{\cal B}^{*}_{1}(\vec{x}))\right],$ (10)

where $\vec{\cal E}_{i}(\vec{x})$, $\vec{\cal B}_{i}(\vec{x})$, $i=1,2$, are
dimensionless electric and magnetic fields of pump modes.

Eq. (4) implies the following solution

$\displaystyle a(t,\vec{x})=g_{a\gamma\gamma}\,E^{\rm em}_{0}B^{\rm
em}_{0}\int_{V_{\mathrm{em}}}d^{3}x^{\prime}\;\mathrm{Re}\Bigl{[}(\vec{\cal
E}\cdot\vec{\cal
B})(\vec{x}^{\prime})\dfrac{e^{ik_{a}|\vec{x}-\vec{x}^{\prime}|-i\omega_{a}t}}{4\pi|\vec{x}-\vec{x}^{\prime}|}\Bigr{]}\equiv\mathrm{Re}\left[a(\vec{x})\,e^{-i\omega_{a}t}\right]\;,$
(11)

where $k_{a}=\sqrt{\omega_{a}^{2}-m_{a}^{2}}$ are typical momenta of the
produced ALPs, integration is performed over the emitter volume $V_{\rm em}$.
For MF emitter, the dimensionless factor in Eq. (11) is defined by Eq. (8).
However, for MM emitter the solution of Eq. (4) splits into two frequency
components $\omega_{\pm}$. Note that the produced ALPs of frequency component
$\omega_{-}$ is at least an order of magnitude smaller than $\omega_{+}$
component [19], so further we deal with $\omega_{+}$ only. One can replace
formally $ik_{a}$ with $-\kappa_{a}=-\sqrt{m_{a}^{2}-\omega_{a}^{2}}$ in Eq.
(11) for the relatively heavy ALP mass limit $m_{a}\gtrsim\omega_{a}$. Then
the ALP field amplitude decreases exponentially as
$a(\vec{x})\propto\exp{\left(-\kappa_{a}|\vec{x}|\right)}/|\vec{x}|$ outside
the production cavity. Therefore, the detecting cavity signal is suppressed
for $m_{a}\gtrsim\omega_{a}$ mass range.

###  2.2 The receiver cavity

In this section, we investigate the issue of the ALP signal detection in the
receiver cavity. A resonant generation of electromagnetic modes in the
detecting cavity caused by the axion-induced current,
$j^{\nu}_{a}=(\rho_{a},\vec{j}_{a})$ where the density of the effective charge
$\rho_{a}(\vec{x},t)$ and the effective current $\vec{j}_{a}(\vec{x},t)$ are
given by Eqs. (6). Two options are assumed for detection:

  * •

the receiver cavity is a normally conducting one, and it is immersed into
external constant magnetic field $\vec{B}_{\rm ext}$. We use the notation M∗F
receiver (induced signal mode (M∗) + magnetic field (F) of the receiver) for
that case (the label M∗ denotes the mode that we expect to detect throughout
the paper);

  * •

the receiver cavity is superconducting, and it is pumped by the detecting
mode. We use the notation M∗M receiver (induced signal mode (M∗) + pump mode
(M) of the receiver) for this setup of the cavity.

One can show (see e. g. Ref. [21] and references therein for detail) that
generating field is a combination of solenoidal and potential modes, however
only the solenoidal modes can be resonantly enhanced. The typical magnitude of
the signal can be characterized by the expression [18, 20]

$G=-\dfrac{Q_{\rm rec}}{\omega_{s}}\cdot\dfrac{1}{V_{\rm rec}}\int_{V_{\rm
rec}}d^{3}x\,(\vec{\cal E}^{*}_{s}\cdot\vec{j}_{a})\;,$ (12)

where $Q_{\rm rec}$ is a quality factor for the receiver eigenmode that
depends on the electric field near the cavity walls and corresponding power
losses due to non-linearities (see e. g. Ref. [22] and references therein for
details), $V_{\rm rec}$ is the volume of the receiver cavity, $\omega_{s}$ is
a frequency of the receiver signal eigenmode, and $\vec{\cal E}_{s}(\vec{x})$
is a dimensionless signal eigenmode that is normalized as follows [20]

$\int\limits_{V_{\rm rec}}d^{3}x\,|\vec{\cal E}_{s}(\vec{x})|^{2}=V_{\rm
rec}\;.$ (13)

The specific form of the current $\vec{j}_{a}$ in the Eq. (12) depends on the
way of ALP detection. It is remarkable that the general expression of the
overlapping integral in Eq. (12) for both M∗F and M∗M receivers can be written
in the following explicit form

$\int\limits_{V_{\rm rec}}\\!\\!\\!d^{3}x\,(\vec{\cal
E}^{*}_{s}\\!\cdot\\!\vec{j}_{a})=-i\omega_{s}g_{a\gamma\gamma}B^{\rm
rec}_{0}\\!\\!\\!\int\limits_{V_{\rm rec}}\\!\\!\\!d^{3}x\,(\vec{\cal
E}\\!\cdot\\!\vec{\cal B})^{*}\\!(\vec{x})a(\vec{x})\;,$ (14)

where $B^{\rm rec}_{0}$ is a characteristic magnetic field of the detection
cavity and $(\vec{\cal E}\cdot\vec{\cal B})^{*}(\vec{x})$ is a dimensionless
complex-conjugated function that is associated with a specific way of ALP
detection.

More specifically, for the M∗F receiver, $B_{0}^{\rm rec}$ is the value of the
external magnetic field, and dimensionless function has the following form

$(\vec{\cal E}\cdot\vec{\cal B})^{*}(\vec{x})=(\vec{\cal
E}_{s}(\vec{x})\cdot\vec{\cal B}_{\rm ext})^{*}\;,$ (15)

where $\vec{\cal B}_{\rm ext}$ is a unit vector co-directed with a magnetic
field inside the receiver cavity.

For the M∗M receiver, $B_{0}^{\rm rec}$ is the magnetic field amplitude of the
pump mode and the combinations of dimensionless functions are given by

$\displaystyle(\vec{\cal E}\\!\cdot\\!\vec{\cal B})^{*}_{+}(\vec{x})$
$\displaystyle\\!=\\!\dfrac{1}{2}\\!\left[(\vec{\cal
E}_{s}(\vec{x})\\!\cdot\\!\vec{\cal B}_{d}(\vec{x}))+(\vec{\cal
E}_{d}(\vec{x})\\!\cdot\\!\vec{\cal B}_{s}(\vec{x}))\right]^{*},$ (16)
$\displaystyle(\vec{\cal E}\\!\cdot\\!\vec{\cal B})^{*}_{-}(\vec{x})$
$\displaystyle\\!=\\!\dfrac{1}{2}\\!\left[(\vec{\cal
E}_{s}(\vec{x})\\!\cdot\\!\vec{\cal B}^{*}_{d}(\vec{x}))\\!+\\!(\vec{\cal
E}^{*}_{d}(\vec{x})\\!\cdot\\!\vec{\cal B}_{s}(\vec{x}))\right]^{*}\;,$ (17)

for $\omega_{a}=\omega_{s}+\omega_{d}$ and $\omega_{a}=\omega_{s}-\omega_{d}\
(\omega_{s}>\omega_{d})$ respectively, here $\omega_{d}$ is a receiver pump
mode frequency, $\vec{\cal E}_{s(d)}(\vec{x})$ and $\vec{\cal
B}_{s(d)}(\vec{x})$ are dimensionless electric and magnetic fields
respectively for the signal mode (detection pump mode in Ref. [18]).

###  2.3 Signal power

Here we discuss the signal induced by the axion field for the cavity
experimental setups. To be more concrete, by using Eqs. (11) and (14) we can
rewrite the amplitude in Eq. (12) in general form

$G=iQ_{\rm rec}g^{2}_{a\gamma\gamma}E^{\rm em}_{0}B^{\rm em}_{0}B^{\rm
rec}_{0}\cdot\dfrac{V_{\rm em}V_{\rm rec}{\cal G}}{\Delta}\;,$ (18)

where $\Delta$ is typical distance between cavities, and the dimensionless
factor ${\cal G}$ is given by the following expression

${\cal G}\,=\int\limits_{V_{\rm rec}}\dfrac{d^{3}x}{V_{\rm
rec}}\int\limits_{V_{\rm em}}\dfrac{d^{3}x^{\prime}}{V_{\rm em}}\,(\vec{\cal
E}\cdot\vec{\cal B})^{*}(\vec{x})\,(\vec{\cal E}\cdot\vec{\cal
B})(\vec{x}^{\prime})\,\dfrac{e^{ik_{a}|\vec{x}-\vec{x}^{\prime}|}}{4\pi}\dfrac{\Delta}{|\vec{x}-\vec{x}^{\prime}|}\;.$
(19)

In the steady regime the averaged signal power can be expressed in the
following form

$P_{\rm signal}=\dfrac{\omega_{s}}{Q_{\rm rec}}\int\limits_{V_{\rm
rec}}d^{3}x\,\langle|\vec{E}_{s}^{2}(\vec{x},t)|\rangle_{t}=\dfrac{\omega_{s}}{Q_{\rm
rec}}\dfrac{1}{2}|G|^{2}V_{\rm rec}\;,$ (20)

where $\vec{E}_{s}(\vec{x},t)$ is a signal solenoidal electric field that is
resonantly enhanced by the ALP in the receiver. It is important to note that
Eq. (20) implies
$\langle|\vec{E}(\vec{x},t)|^{2}\rangle_{t}=\langle|\vec{B}(\vec{x},t)|^{2}\rangle_{t}$
(see e. g. Ref. [21] and references therein for detail). We imply that time
averaging of the squared magnitude results in the replacement
$\langle...\rangle_{t}\to 1/2$ [20]. We note that this approach provides the
same result as the power spectral density calculation [18, 22] for the narrow
signal bandwidth limit.

We estimate sensitivity numerically as maximum output in the receiver cavity
that is given by the Dicke radiometer equation,

$\mathrm{SNR}=\frac{P_{\mathrm{signal}}}{P_{\mathrm{noise}}}\cdot\sqrt{t\Delta\nu}\;,$
(21)

where $t$ is an integration time for a signal, $\Delta\nu$ is its bandwidth
and $P_{\rm noise}$ is a power of thermal noise which can be estimated as
$P_{\rm noise}\simeq T\Delta\nu$ in the limit $\omega_{s}\ll T$, where $T$ is
the temperature of the receiver. We consider two options for $\Delta\nu$: the
bandwidth of a cavity mode itself (i.e. $\Delta\nu\simeq\nu_{s}/Q_{\rm rec}$,
where $\nu_{s}=\omega_{s}/(2\pi)$) and the narrowest possible bandwidth of a
pump generator, which can be as small as $\Delta\nu\simeq 1/t$ (see e. g.
Refs. [20, 18] and references therein).

Finally, by using Eqs. (20) and (21) one can obtain the general formula for
the expected sensitivity,

$g_{a\gamma\gamma}=\left[\dfrac{2\Delta^{2}T\,{\rm SNR}}{\omega_{s}Q_{\rm
rec}E^{2}_{0,{\rm em}}\\!B^{2}_{0,{\rm em}}B^{2}_{0,{\rm rec}}V^{2}_{\rm
em}V_{\rm rec}{|\cal
G}|^{2}}\right]^{\frac{1}{4}}\left(\dfrac{\Delta\nu}{t}\right)^{\frac{1}{8}}\;,$
(22)

which is valid for estimation of the ALP expected limit (${\rm SNR}\simeq 5$)
for all benchmark setups considered in the next section.

##  3 The expected reach

Now we compare the efficiencies of three different experimental setups for
probing ALPs with LSW methods: (i) MF (RF) emitter + M∗F (RF) receiver; (ii)
MM (SRF) emitter + M∗M (SRF) receiver; (iii) MM (SRF) emitter + M∗F (RF)
receiver. In addition, we study in detail the sensitivity dependence on the
spatial geometry (relative position of emitter and receiver cylindrical
cavities) and radius to length ratio $R/L$ of the ALP emitter for these
benchmark experimental proposals.

###  3.1 MF emitter + M∗F receiver

At first let us consider the typical LSW setup consisting of two RF cavities
which are placed both into a strong static magnetic field [13]. The
experimental realization of that idea was carried out by the CROWS experiment
[15]. We show the sensitivity of this type of experiment for the
characteristic volume of the emitter and receiver cavities $V_{\rm rec}=V_{\rm
em}\simeq 1\,\mbox{m}^{3}$, their quality factor $Q\simeq 10^{5}$, and the
typical value of the static magnetic field $B_{0}^{\rm em}=B_{0}^{\rm
rec}=3\,{\rm T}$ in Fig. 2. We consider the benchmark emitter power $P_{\rm
em}\simeq 100\ {\rm kW}$ which leads to the magnitude of the pump mode $E^{\rm
em}_{0}=3\,{\rm MV/m}\ (B^{\rm em}_{0}=0.01\,\rm{T})$ for the corresponding
volume and quality factor. The distance between receiver and emitter walls is
$\Delta=0.5$ m. The pump mode of the emitter and the signal mode of the
receiver are TM010.

Figure 2: The sensitivity of MF emitter + M∗F receiver setup for both coaxial
and parallel cavity locations and the TM010 emitter and receiver modes. Left
panel: the dependence on the emitter cavity radius-to-length ratio $R/L$ for
the typical volume $V_{\rm em}=1\,{\rm m}^{3}$. Right panel: expected reach as
a function of ALPs mass at optimal $R/L$ for coaxial ($R/L\simeq 1.67$,
$R\simeq 0.81\,{\rm m}$, $L\simeq 0.49\,{\rm m}$) and parallel ($R/L\simeq
0.37$, $R\simeq 0.49\,{\rm m}$, $L\simeq 1.32\,{\rm m}$) geometries. The
distance between cavity walls is $\Delta=0.5$ m, the cavity volumes are
$V_{\rm rec}=V_{\rm em}=1$ m3. The integration time is $t=10^{6}\ {\rm s}$.

In Fig. 2 (left panel) we show the expected sensitivity of the setup as a
function of $R/L$ for both parallel and coaxial designs of the cavities (see
e. g. Fig. 1 for detail), we also set the ALP benchmark masses to be $m_{a}=0$
and $m_{a}\simeq\omega_{a}$. We take into account that the $Q_{\rm rec}$
depends on the chosen mode and cavity geometry, $Q_{\rm
rec}={\omega_{s}}/R_{s}\,\cdot\,V_{\rm rec}/S_{\rm
rec}\propto\sqrt{\omega_{s}}\,V_{\rm rec}/S_{\rm rec}$, where $R_{s}$ is the
surface resistance and $S_{\rm rec}$ is the surface area of the detector
cavity [23], here we also set the value $Q_{\rm rec}=10^{5}$ for $R/L=1$. We
consider sensitivity for $m_{a}=0$ as the most important setup characteristic
compared to the resonant bound at $m_{a}=\omega_{s}$ regime throughout the
paper. It implies that the typical bounds at $m_{a}=0$ cover the larger
logarithmic mass scale range ($m_{a}\lesssim\omega_{a}/2$) in
$(g_{a\gamma\gamma},m_{a})$ plane. It turns out that coaxial design for
$R/L\gtrsim 1$ is more preferable. It is remarkable that in this case the
typical expected reaches for both masses $m_{a}=0$ and $m_{a}\simeq\omega_{a}$
coincide by the order of the magnitude at the level of
$g_{a\gamma\gamma}\simeq 3\times 10^{-11}\;\mbox{GeV}^{-1}$. However, there is
a notable difference between the expected reaches at $m_{a}=0$ and
$m_{a}\simeq\omega_{s}$ for the parallel design. Note that optimal radius to
length ratio (that implies better sensitivity on $g_{a\gamma\gamma}$ in case
of $m_{a}=0$) is $R/L\simeq 1.67$ for coaxial design and $R/L\simeq 0.37$ for
parallel design.

In Fig. 2 (right panel) we show the expected reach as a function of the ALP
mass $m_{a}$ for both coaxial and parallel locations of the cavities at the
optimal ratios $R/L$ assuming two options of the signal bandwidth
$\Delta\nu\simeq\nu/Q_{\rm rec}$ and $\Delta\nu\simeq 1/t$, where $t\simeq
10^{6}\,\mbox{s}$ is the typical time of measurement. The conservative cavity
bandwidth $\Delta\nu\simeq\nu/Q_{\rm rec}$ yields the expected limit
$g_{a\gamma\gamma}\lesssim 5\times 10^{-10}\,\mbox{GeV}^{-1}$ that is weaker
than the CAST constraint [8]. However, the optimistic bandwidth
$\Delta\nu\simeq 1/t$ can provide the expected reach
$g_{a\gamma\gamma}\lesssim 3\times 10^{-11}\,\mbox{GeV}^{-1}$ for
$m_{a}\lesssim\omega_{a}/2$.

It is worth noting that corresponding heat production makes it challenging to
keep the emitter temperature at a required level. On the other hand,
relatively small temperatures $T\lesssim 1\,\mbox{K}$ are required for the
receiver cavity in order to avoid the problem of thermal noise. So that
keeping two cavities at different temperatures would require a development of
sophisticated RF methods for the regarding setup.

###  3.2 MM emitter + M∗M receiver

Figure 3:  The sensitivity of MM emitter + M∗M receiver cavity setup. This
facility implies combination of TM010 \+ TE011 production pump modes. The pump
mode of a receiver and its signal mode are chosen to be TM010 and TE011
respectively. The case of ALPs frequency $\omega_{a}=\omega_{+}$ is
considered. Left panel: the expected limit $g_{a\gamma\gamma}$ as a function
of production cavity radius-to-length ratio $R/L$ (we set the emitter volume
at $V_{\rm em}=1\,{\rm m}^{3}$). Right panel: Sensitivity as a function of
ALPs mass at optimal $R/L$ for coaxial ($R/L\simeq 1.60$, $R\simeq 0.80\,{\rm
m}$, $L\simeq 0.50\,{\rm m}$) and parallel ($R/L\simeq 0.35$, $R\simeq
0.48\,{\rm m}$, $L\simeq 1.37\,{\rm m}$) designs. The distance between cavity
walls is $\Delta=0.5\,\mbox{m}$, the volume of each cavity is $V_{\rm
em}=V_{\rm rec}=1$ m3. Integration time is $t=10^{6}\,{\rm s}.$

.

The second setup of our interest consists of two equal SRF cavities [18]. In
the emitter cavity, ALPs are generated by an interaction of two cavity modes.
In the detection cavity, produced ALPs interact with a single pump mode (which
coincides with one of the production cavity pump modes), producing the
resonantly enhanced signal mode in the receiver cavity. The magnitude of the
surface amplitude of pump modes for an SRF cavity to be as small as $B^{\rm
em,rec}_{0}\lesssim 0.1\,{\rm T}\ (E_{0}^{\rm em,rec}\lesssim 30\,{\rm MV/m})$
to avoid the superconductivity state destruction. The volume of the emitter
and receiver cavities $V_{\rm rec}=V_{\rm em}\simeq 1\,\mbox{m}^{3}$, their
quality factor $Q\simeq 10^{10}$. The expected power of the emitter cavity is
$P_{\rm em}\simeq 0.1\,{\rm kW}$. In Fig. 3 the typical sensitivities for the
regarding LSW setup are presented.

In Fig. 3 (left panel) the expected reach as function of emitter radius-to-
length ratio $R/L$ is shown. As in previous case, we take into account the
dependence of the quality factor $Q$ on cavity geometry which reads $Q_{\rm
rec}=\omega_{s}/R_{s}\cdot V_{\rm rec}/S_{\rm
rec}\propto{\omega_{s}}^{-1}\,V_{\rm rec}/S_{\rm rec}$ for SRF cavity [24],
and fix the value as $Q_{\rm rec}=10^{10}$ for $R/L=1$. It turns out that the
optimal magnitude of $R/L$ for the coaxial cavity location and for the ALP
mass limit $m_{a}=0$ is $R/L\simeq 1.6$. The regarding expected sensitivity is
$g_{a\gamma\gamma}\lesssim 5\times 10^{-11}\,\mbox{GeV}^{-1}$ that is
comparable with the CAST bound $g_{a\gamma\gamma}\lesssim 6\times
10^{-11}\,\mbox{GeV}^{-1}$. For parallel location of the cavities, the optimal
radius-to-length ratio is $R/L\simeq 0.35$ implying $m_{a}=0$. We note that
zero axion mass bounds $g_{a\gamma\gamma}\lesssim 6\times
10^{-10}\,\mbox{GeV}^{-1}$ are ruled out by the CAST. The signal cavity
bandwidth is chosen to be at the level $\Delta\nu\simeq 1/t$, where $t\simeq
10^{6}\,\mbox{s}$ is a typical time of the measurements.

In Fig. 3 (right panel) we show the expected limit $g_{a\gamma\gamma}$ of this
setup as a function of the ALP mass $m_{a}$. It turns out that the sensitivity
has a sharp peak at the resonance $m_{a}\simeq\omega_{a}$ for both coaxial and
parallel designs. For the optimistic signal bandwidth $\Delta\nu\simeq 1/t$
regarding expected limit is estimated at the level of
$g_{a\gamma\gamma}\lesssim 5\times 10^{-11}\,\mbox{GeV}^{-1}$ for
$m_{a}\lesssim\omega_{a}/2$.

The detection of a relatively small signal in a cavity with high intensity
pump mode may lead to challenging technical issues, mainly related to
filtering of the tiny signal mode from the very intensive pump mode. To
resolve this issue, one may consider a “bottle-shape” cavity geometry similar
to that discussed in [20].

###  3.3  MM emitter + M∗F receiver

Figure 4: The sensitivity of the MM emitter + M∗F receiver setup for TM010 \+
TE011 emitter pump modes and TM010 detection signal mode. The case of ALPs
frequency $\omega_{a}=\omega_{+}$ is considered. Left panel: the sensitivity
dependence on emitter cavity radius-to-length ratio $R/L$ (fixed volume of
$V_{\rm em}=1\,{\rm m}^{3}$ and fixed length of $L_{\rm rec}=0.5\,\mbox{\rm
m}$). Right panel: sensitivity as a function of ALPs mass at optimal $R/L$ for
coaxial ($R/L\simeq 1.44$, $R\simeq 0.77\,{\rm m}$, $L\simeq 0.54\,{\rm m}$,
$R_{\rm rec}\simeq 0.22\,{\rm m}$, $L_{\rm rec}\simeq 0.5\,{\rm m}$) and
parallel ($R/L\simeq 0.33$, $R\simeq 0.47\,{\rm m}$, $L\simeq 1.43\,{\rm m}$,
$R_{\rm rec}\simeq 0.18\,{\rm m}$, $L_{\rm rec}\simeq 0.5\,{\rm m}$) design.
The distance between cavity walls is $\Delta=0.5$ m, the volume of the emitter
cavity is $V_{\rm em}=1$ m3. Integration time is $t=10^{6}\,{\rm s}.$

.

The final setup that we consider in our study consists of a production SRF
cavity with two pump modes and a detection RF cavity immersed into static
magnetic field [19, 25].

In Fig. 4 we show the sensitivity of this type of experiment for the
characteristic volume of the emitter cavity $V_{\rm em}\simeq 1\,\mbox{m}^{3}$
and its quality factor $Q_{\rm em}\simeq 10^{10}$. Amplitudes of the emitter
pump modes are $B_{0}^{\rm em}=0.1\,{\rm T}\ (E_{0}^{\rm em}=30\,{\rm MV/m})$
to avoid destruction of the superconducting state. The expected power of the
emitter cavity is $P_{\rm em}\simeq 0.1\,{\rm kW}$. The distance between
receiver and emitter walls is $\Delta=0.5$ m. The pump modes of the emitter
are TM010 and TE011, and the signal mode of the receiver is TM010. Note that
the receiver must be smaller than the emitter for the frequency equality
$\omega_{1}+\omega_{2}=\omega_{s}$ to be fulfilled. The receiver quality
factor is $Q_{\rm rec}\simeq 10^{5}$ and the typical value of the static
magnetic field $B_{0}^{\rm rec}=3\,{\rm T}$.

In Fig. 4 (left panel) we show the typical expected reach for this setup as a
function of $R/L$ for the emitter cavity. We emphasize that the regarding
bounds are ruled out by the CAST facility at $g_{a\gamma\gamma}\lesssim
3.0\times 10^{-10}\,\mbox{GeV}^{-1}$. This can be also justified from the
right panel of Fig. 4 where the typical bounds are shown in the
$(g_{a\gamma\gamma},m_{a})$ plane.

##  4 Results and discussion

We compared three types of the LSW radio setups for ALP searches and
determined the best design for them. We summarize our study presenting
important parameters for each setup in Table 1.

Type of the setup | $B^{\rm em,(1)}_{0}$ | $B^{\rm em,(2)}_{0}$ | $B^{\rm rec}_{0}$ | $Q_{\rm rec}$ | $P_{\rm em}$ | $|\cal G|$ | $g_{a\gamma\gamma}$
---|---|---|---|---|---|---|---
MF em. + M∗F rec. | $0.01\,\rm T$ | $3\,\rm T$ | $3\,\rm T$ | $10^{5}$ | $100\,\rm kW$ | $10^{-2}$ | $3\times 10^{-11}\,{\rm GeV}^{-1}$
MM em. + M∗M rec. | $0.1\,\rm T$ | $0.1\,\rm T$ | $0.1\,\rm T$ | $10^{10}$ | $0.1\,\rm kW$ | $10^{-3}$ | $5\times 10^{-11}\,{\rm GeV}^{-1}$
MM em. + M∗F rec. | $0.1\,\rm T$ | $0.1\,\rm T$ | $3\,\rm T$ | $10^{5}$ | $0.1\,\rm kW$ | $10^{-3}$ | $3\times 10^{-10}\,{\rm GeV}^{-1}$
Table 1: Comparison of the characteristics for various experimental setups.
The geometrical formfactor $|{\cal G}|$ and the setup sensitivity
$g_{a\gamma\gamma}$ are presented for the best ratio of $R/L$ of coaxial
location and the mass of ALPs $m_{a}\lesssim\omega_{a}/2$.

We concluded that the MF emitter + M∗F receiver and the MM emitter + M∗M
receiver setups can achieve the similar top sensitivity
$g_{a\gamma\gamma}\lesssim(3-5)\times 10^{-11}\,\mbox{GeV}^{-1}$ at
$m_{a}\lesssim\omega_{a}/2$. In particular, it turns out that the larger
electromagnetic field combination and the geometrical formfactor of RF
cavities compensate its smaller quality factor. Moreover, we find that the
best relative location of the cavities is coaxial with the ratio of $R/L\simeq
1.6$.

The MF emitter + M∗F receiver setup is a modification of the CROWS experiment
[15] that implies larger volume of the cavities $V_{\rm em}\simeq V_{\rm
rec}\simeq 1\,\mbox{m}^{3}$, lower temperature, and narrower bandwidth of the
signal, $\Delta\nu\simeq 1/t$. However, there is a disadvantage of this setup
that implies the relatively large emitter power $P_{\rm em}\simeq 100\,{\rm
kW}$.

The advantage of the MM emitter + M∗M receiver setup is that its emitter power
is 4 orders of magnitude smaller than the previous one. However, in this case
the main technical challenge would be related to the signal mode filtering
from the pump mode.

Given the benchmark parameters, the last setup, MM emitter + M∗F receiver, has
the weakest sensitivity. The typical bounds $g_{a\gamma\gamma}\lesssim 3\times
10^{-10}\,\mbox{GeV}^{-1}$ would be ruled out by the CAST. Also, there is a
disadvantage of this setup. In particular, the condition
$\omega_{1}+\omega_{2}=\omega_{s}$ implies the specific type of the emitter
modes, the latter is linked to the sizes of the cavity. Moreover, the
modification of the pump modes would require the changing of the receiver
geometry.

#### Acknowledgements

The authors thank Sergey Troitsky, Yury Senichev, and Yonatan Kahn for
valuable discussions and helpful suggestions. The work on optimal geometry of
the setups is supported by RSF grant 21-72-10151. The work of DS on design
comparison of the setups is supported by the BASIS Foundation, grant no.
22-2-1-17-1.

## References

  * [1] Peccei, R. D. and Quinn, H. R.  Phys. Rev. Lett. 38, 1440–1443 (1977).
  * [2] Peccei, R. D. and Quinn, H. R.  Phys. Rev. D 16, 1791–1797 (1977).
  * [3] Svrcek, P. and Witten, E.  JHEP 06, 051 (2006).
  * [4] Arvanitaki, A., Dimopoulos, S., Dubovsky, S., Kaloper, N., and March-Russell, J.  Phys. Rev. D 81, 123530 (2010).
  * [5] Preskill, J., Wise, M. B., and Wilczek, F.  Phys. Lett. B 120, 127–132 (1983).
  * [6] Abbott, L. F. and Sikivie, P.  Phys. Lett. B 120, 133–136 (1983).
  * [7] Dine, M. and Fischler, W.  Phys. Lett. B 120, 137–141 (1983).
  * [8] Anastassopoulos, V. et al.  Nature Phys. 13, 584–590 (2017).
  * [9] Irastorza, I. G. and Redondo, J.  Prog. Part. Nucl. Phys. 102, 89–159 (2018).
  * [10] Sikivie, P.  Phys. Rev. Lett. 51, 1415–1417 (1983).  [Erratum: Phys.Rev.Lett. 52, 695 (1984)].
  * [11] Anselm, A. A.  Yad. Fiz. 42, 1480–1483 (1985).
  * [12] Van Bibber, K., Dagdeviren, N. R., Koonin, S. E., Kerman, A., and Nelson, H. N.  Phys. Rev. Lett. 59, 759–762 (1987).
  * [13] Hoogeveen, F.  Phys. Lett. B 288, 195–200 (1992).
  * [14] Ehret, K. et al.  Phys. Lett. B 689, 149–155 (2010).
  * [15] Betz, M., Caspers, F., Gasior, M., Thumm, M., and Rieger, S. W.  Phys. Rev. D 88(7), 075014 (2013).
  * [16] Ortiz, M. D. et al.  Phys. Dark Univ. 35, 100968 (2022).
  * [17] Berlin, A. et al.  3 (2022).
  * [18] Gao, C. and Harnik, R.  JHEP 07, 053 (2021).
  * [19] Salnikov, D., Satunin, P., Kirpichnikov, D. V., and Fitkevich, M.  JHEP 03, 143 (2021).
  * [20] Bogorad, Z., Hook, A., Kahn, Y., and Soreq, Y.  Phys. Rev. Lett. 123(2), 021801 (2019).
  * [21] Berlin, A., Blas, D., Tito D’Agnolo, R., Ellis, S. A. R., Harnik, R., Kahn, Y., and Schütte-Engel, J.  Phys. Rev. D 105(11), 116011 (2022).
  * [22] Berlin, A., D’Agnolo, R. T., Ellis, S. A. R., Nantista, C., Neilson, J., Schuster, P., Tantawi, S., Toro, N., and Zhou, K.  JHEP 07(07), 088 (2020).
  * [23] Hill, D.  Antennas and Propagation Magazine, IEEE 56, 306–306 02 (2014).
  * [24] Padamsee, H.  IEEE Transactions on Magnetics 19(3), 1322–1325 (1983).
  * [25] Salnikov, D., Satunin, P., Kirpichnikov, D. V., and Fitkevich, M.  Moscow Univ. Phys. Bull. 77(2), 145–147 (2022).


A configurable accelerator for manycores:
the Explicitly Many-Processor Approach

János Végh
J. Végh is with Faculty of Mechanical Engineering and Informatics, University of Miskolc, Hungary

E-mail<EMAIL_ADDRESS>
Manuscript received July 10, 2016; revised August 26, 2016.

The Explicitly Many-Processor Approach, Vol. 14, No. 8, August 2016
V : the Explicitly Many-Processor Approach

A new approach to designing processor accelerators is presented.
A new computing model and a special kind of accelerator with dynamic
(end-user programmable) architecture
is suggested. The new model considers a processor, in which a newly introduced
supervisor layer coordinates the job of the cores.
The cores have the ability
(based on the parallelization information provided by the compiler, and
using the help of the supervisor) to outsource part of the job they received
to some neighbouring core. The introduced changes essentially and advantageously
modify the architecture and operation of the computing systems.
The computing throughput drastically increases,
the efficiency of the technological implementation (computing performance per logic gates) increases, the non-payload activity for using operating system services decreases,
the real-time behavior changes advantageously, and connecting accelerators
to the processor greatly simplifies.
Here only some details of the architecture and operation
of the processor are discussed, the rest is described elsewhere.

computer architecture, processor accelerator, manycore processor, many-processor approach

language=[x86masm]Assembler, basicstyle=

§ INTRODUCTION

About a decade ago, "growth
in single-processor performance has stalled – or at best
is being increased only marginally over time" <cit.>.
Since the computing industry heavily influences
virtually all industrial segments, the stalling crashed the long-term forecasts in the industry, economy, etc.
The need for growing performance triggered research in many directions,
from "rebooting computing" <cit.> through
using a "cross-layer approach" <cit.> to using accelerators
combining different technologies, like reconfigurable GPGPU <cit.>.
Especially the popular and powerful many-core processors provide huge computing capacity
and sever problems, like
"multicore and manycore
vendors and runtime systems cannot possibly support
the virtually unlimited number of processor configurations" <cit.>.
It is not yet checked, however, what hidden reserves for acceleration can be disclosed
in the operation of the conventional processors itself.

In addition to the HW issues, also the SW makes its contribution:
"parallel programs ... are notoriously difficult to write,
test, analyze, debug, and verify, much more so than
the sequential versions" <cit.>,
and doubts like
"Chip multi-processors have emerged as one of the most effective uses of the huge
number of transistors available today and in the future,
but questions remain as to the best way to leverage CMPs
to accelerate single threaded applications" <cit.>.
This is why "cross-layer approach spanning from hardware to user-
facing software is necessary to successfully address this
problem" <cit.>.

In this paper a "better way" of accelerating  single threaded applications is searched,
first scrutinizing the hidden reserves  in the operation of the stored program processor based computing systems.
During this, it is shown that the final reason of the present stalling
(in addition to the already known reason: the finite speed of the light)
is the 70-years old single-processor approach, which dominates both computer
construction and programming. The introduced EMPA viewpoint solves
some old problems of computing. It increases the computing troughput, depending on
the context up to several dozens times higher, allows to build more deterministic real-time systems, etc. At the same time, it allows to simplify the internal architecture,
to use less transistors for the chips.

§ THE SINGLE-PROCESSOR APPROACH

According to the state of the art of his time, Neumann considered only one processor
and formulated the principles of operating a processor considering only the execution
of a single machine instruction. Considering his paradigms carefully,
neither of them contradicts to the present many-processor approach
(see section <ref>).

§.§ Lack of time dependence

The roots of the event-oriented processing
were implicitly present in Neumann's approach: the next instruction can only be executed when the
processor is not any more busy with executing the current instruction.
The time, however, in the paradigms is considered as an implementation detail.
Although later the lack of considering the time explicitly
was identified as one of the fundamental issues in computing
<cit.>, handling time (as well as synchronizing)
became the task of operating systems, which they can solve in
a quite resource-wasting way <cit.>.

The availability of the processing unit,
the instruction and the data are critical factors, and all they have
a timely behavior. The most successful
approaches to improving performance of processors modified the conditions of processor availability (methods for  reducing the
instruction and data memory access time are not touched here).
The pipelining separated the signal into "ready to accept data" and
"result is available" signals, the hyper-threading connected the ready-to-run thread to the processing unit.
Critics like "HT generally improves processor resource utilization efficiency,
but does not necessarily translate into overall application performance gain" <cit.>, call  to scrutinize
the processor availability condition and cross-layer functionality, if one wants to improve single-processor performance
using several processing units.

§.§ Atomic processing unit

Hyper-threading separates
the hardware accelerators (like registers and core-level cache)
from the raw processing power, which is a clear recognition of the
fact that from programming point of view, the single machine instruction
(a HW unit) is too small (cannot make reasonable functionality),
while a complete process (a SW unit)
is too big (wastes time with waiting),
so some intermediate unit: HW-supported QT
should be introduced. This suggests to check, what the optimum
size of the right unit is, and how it should be supported from HW and SW.
Also it shows that the effective problem solution can only be
reached through HW/SW codesign, i.e through crossing HW/SW
layers <cit.>.

§.§ Multiple processing units

From again another side, from the simple out-of-order processing through
multiple arithmetic units to speculative evaluation,
several kinds of hidden processing units have been introduced
and they successfully accelerated the operation of the processor.
There is no question, that more processing units are needed
to make parallel operations (and in this way  apparently the processing) quicker.
However, those solutions forget about one of the most important principles
of Neumann: computer should be simple.

In order to achive higher performance, "computers have thus far achieved this goal
at the expense of tremendous hardware complexity –
a complexity that has grown so large as to challenge
the industry's ability to deliver ever-higher performance" …and
…"the ever- increasing complexity of superscalar processors would have a
negative impact upon their clock rate, eventually leading to a leveling off of the rate
of increase in microprocessor performance" <cit.>.
This suggests to check what the optimum way of using
multiple processing units is, without wasting computing and electric power?

§.§ Multitasking issues

Different reasons directed the designers to share the computing
capacity and other resources between different tasks, starting with the
age of single-processor systems. The methods, however, have
been prolonged to  the era, when several processing units could be
used. Even today, the external peripherals interrupt the control flow
(although some other core could do interrupt servicing), in
the many-processor systems all PUs are central, the
operating systems providing services and scheduling the operation
of the running tasks take the processor time from the payload jobs,
changing context between user and kernel modes causes a considerable
non-payload activity, the hardware scheduling makes the software
operation non predictable, etc.

A related special issue for accelerators is that an accelerator
is always outside the processor, and it is efficient because
it works differently from the conventional, programmable processors.
Several problems must be solved in order to connect
the stored program (von Neumann) processors with the rest of the world.
The processor only offers I/O bus for connecting an accelerator
to the processor. However, in the today's environment an OS must
provide protection (virtualization) for the I/O operations.
It is only possible in protected mode, and the context change from and back to user mode is
extremely expensive: it is in the range of dozens of thousands clock periods
for the modern HW architectures and OSs <cit.>.
This fact restricts the utilization of general-purpose accelerators
to accelerate only activities long enough to be not disproportional with that
offset time.

§ THE MANY-PROCESSOR APPROACH

The principle of operation of stored program processors
(see Fig <ref>(a)) and its engineering implementation
(see Fig <ref>(b)) became
quite different during the time, due to the efforts to enhance
processing speed of the computer. Some of the enhancements are none-essential:
although without cache memories the operation of the processor
would be painfully slow, the processor would work.
Also, the highly successful accelerators, the internal registers,
are not strictly needed for the operation.
The processors could work with a strongly limited number of registers <cit.>, or as the example of the very first
EDVACs prove <cit.>, even without registers. Also recall, that it is advantageous to separate registers (as "glue" material, together with internal state and cache) from the processing unit, see hyper-threading or shadow register set at some interrupt servicing.

[Principle of the computation in stored program computers ]

[ball color=magenta,shading=ball, font=

(4,1.5) ellipse (1.55 and .47);
[color = white] at (4.1,1.5) Processing;

[rotate=90,right color=green!0,left color=green!70](2.95,-2.9) ellipse (1.1 and .3);
[rotate=90] at (2.9,2.6) Data;

[rotate=90,right color=green!70,left color=green!0]	(2.95,-5.1) ellipse (1.1 and .3);
[rotate=90] at (5.1,2.8) Instruction;

[bottom color=blue!00,top color=blue!70] (4,3.83) ellipse (1.3 and .4);

at (4.0,3.83) Memory;

[bottom color=blue!00,top color=blue!70]
(4,.6) ellipse (1.2 and .5);
at (4.0,0.6) Processing unit;

[rotate=90,bottom color=yellow!0,top color=yellow!70]
(-0.5,-3.3) ellipse (0.7 and .3);
[rotate=90] at (3.3,-0.3) Registers;

[Engineering implementation of stored program computers ]

[bottom color=yellow!70,top color=yellow!0]
(4,2.1) ellipse (1.1 and .4);
[font=] at (4.0,2.1) Cache;

[ball color=magenta,shading=ball, font=]
(4,1.5) ellipse (1.55 and .47);
[color = white] at (4.1,1.5) Processing;

[rotate=90,right color=green!0,left color=green!70](2.95,-2.9) ellipse (1.1 and .3);
[rotate=90] at (2.9,2.6) Data;

[rotate=90,right color=green!70,left color=green!0]	(2.95,-5.1) ellipse (1.1 and .3);
[rotate=90] at (5.1,2.8) Instruction;

[bottom color=blue!00,top color=blue!70] (4,3.83) ellipse (1.3 and .4);

at (4.0,3.83) Memory;

[bottom color=blue!00,top color=blue!70]
(4,.6) ellipse (1.2 and .5);
at (4.0,0.6) Processing unit;

[rotate=90,bottom color=yellow!0,top color=yellow!70]
(-0.2,-3.3) ellipse (0.7 and .3);
[rotate=90] at (3.3,-0.2) Registers1;

[rotate=90,bottom color=yellow!0,top color=yellow!70]
(-0.3,-3.9) ellipse (0.7 and .3);
[rotate=90] at (3.9,-0.0) Cache;

[rotate=90,bottom color=yellow!0,top color=yellow!70]
(-0.2,-4.5) ellipse (0.7 and .3);
[rotate=90] at (4.5,-0.2) Registers2;

[rotate=0,ball color=red,shading=ball, font=] (2.4,1.8) ellipse (1.0 and .3);
[color=white]  at (2.45,1.8) Data avail;

[rotate=0,ball color=red,shading=ball, font=] (5.6,1.8) ellipse (1.0 and .3);
[color=white] at (5.6,1.8) Instr avail;

[rotate=0,ball color=red,shading=ball, font=] (4,1.05) ellipse (1.1 and .3);
[color=white] at (4.0,1.05) ALU available;

Comparing the theoretical and engineering operation of computers

The many-core processors provide an ideal field to implement
an EMPA environment.
The idea here is to provide the cores with the ability to outsource
part of the job they received. I.e. they remain responsible for doing the job,
but not necessarily "personally".
The approach mathematically covered by the theory of "communicating serial processes" <cit.>
and some additional aspects are discussed in <cit.>.

It requires of course a lot of changes:
some external logic must concert the work of cores (data transfer, dependency
and synchronization), the information about the possible outsourcing must be
prepared at compile time rather than at runtime, the code must be cut to optimally
sized, partly independent QTs, the processor must be notified about
the pre-calculated parallelization possibilities, etc.
The QTs can be embedded into each other, so the core receiving
some outsourced QT can also attempt to find helper cores[Since in any given moment there is a one-to-one correspondance
between an allocated core and the QT running on it,
these two terms are used interchangably: they emphasize the HW and the
SW side of the same thing, similarly as the processor and the
process do in single-processor approach.] to make further outsourcing.

§.§ Availability of processing unit

In the Neumann paradigms,
the task of the control unit is only to provide a "proper sequencing".
In the single-processor approach, the control unit allows sequential stepping,
jumping, calling and returning, and even interrupt servicing.

In many-processor approach, it is also possible to report the
availability of several processing units (this time explicitly,
unlike hidden processing units in the single-processor approach).
For compatibility,
a new control layer is introduced. The SV is a second
control layer in the many-core processor above the cores, which (among other functionalities)
partly takes care of providing this availability signal. Since it knows
about all cores in the processor, it can provide 'ready' signal
as long as at least one core is ready to process instructions.
In this way the processor is able to receive new instructions as long as
at least one of the cores is ready to work.

§.§ Atomic units

Unfortunately, only absolutely independent machine instructions
can be distributed between PUs in this way: the "glue" carries valuable state
information between individual machine instructions.
For allowing  this internal state transfer, the code must be cut into QTs
in a reasonable way.
However, to select the proper granularity is hard.

As mentioned, the machine instructions are (in general) too small to form a QT.
The ideal way would be to use the PUs as stateless automatons, which would result in too coarse granularity,
like threads. To allow for a more effective QT size, a quasi-stateless
automaton model is used. Upon beginning to execute a QT,
a full-fledged "glue" set, comprising register file, flags (clone) and cache (shared),
is provided for the newly hired PU,
and a limited amount of "glue" can be returned
in a (by the supervisor) synchronized way when a QT is finished.
Since QTs are running on the cores in a separated way,
for providing atomic units,
using QTs is ideal: when the core returns 'ready' signal, the action
protected by the QT is finished: both 'owner' and 'others' must wait it.

§.§ Mapping QTs to cores

As the QTs can create other (child) QTs, they form a kind of processing graph.
A virtually infinite number of graph nodes must be mapped to a finite
number of physical cores, so sometimes the new QTs must wait for computing resource.
The supervisor's core allocation algorithm prefers reaching
the leaves of the graph rather than opening new forks, and as
an emergency mechanism for the case when SV runs out of available cores, the cores can suspend processing
their own QTs, borrowing their own resources to their
child-QTs while they are executed.

Another crucial question is how the at compile time labelled QTs
can refer to the QTs distributed at run time to cores, having "random" availability.
The QTs wanting to communicate with other QTs refer to the compile-time
address of the QT and the SV translates this address to a runtime physical
core number. Because of this, QTs have data structures similar to the processes
running under OSs.

§.§ Synchronizing cores

The supervisor maintains configurable[
This feature is between programmable, since it takes the configuration information from the object file and reconfigurable, because it changes
the type of connection between the components inside the processor]
parent-child relationships between the cores,
so the cores have information whether a QT is running, either in their
own scope, or in scope of their parent. Without explicit syncronization, the cores work in parallel on different QTs
of the originally single-threaded code. This kind of synchronization
is done at processor level, in one clock cycle; rather then the
conventional methods, which need awfully long times <cit.>.
Even, since waiting is handled by the SV based on signals handled by the SV,
no time is used when there is no need to wait.

§.§ Linking cores

As contented by Amdahl <cit.>, to increase further
the performance, some cooperation between cores is necessary. In EMPA it is
implemented in such a way that the cores pass the control signals and data to the SV,
which – like a telephone switching center the calls – forwards them
to the right core.
The SV plays a key role here: it handles all resources
of the processor. It connects and synchronizes
the cores  and delivers the limited amount of information,
and even, it provides partners when one of the cores signals that
it could advantageously use more processing capacity.
Also, it concerts collective processing and takes over some functionalities from the OS,
which functionalities are traditionally missing from processors.

When creating a new child QT, the new core
must be able to continue the processing from the point where its parent is,
i.e. the "glue" of the parent must be cloned (using dedicated wiring) to the child.
Also, upon termination,
the child might send back the content of the link register (clone back, using similar method).
The cores are working in parallel, so a special syncronization mechanism
must be used. The data intended to be sent to the other party, are latched by the sender (see also Figs. <ref> and <ref>).
Later some triggering signal transfers the data to its destination,
where they are latched again, and the cores need explicitly use the latch register
to access its content. In this way all partners send data to and receive data from their partner
when they need it, providing a proper syncronization.
The cores see those transfers as using pseudo-registers (see section <ref>).

§.§ Subroutines, interrupts, traps

A main task related to subroutines is to remember the return address.
In the EMPA way, starting a QT means providing also a new core for the
QT, and the processing in the parent code continues immediately,
at the address next to the QT, i.e. at the return address.
That means, that the "return address"
shall only be stored for the time of creating a QT (one clock period)
and it is automatically remembered by the SV.
So, a QT behaves very similarly to a subroutine call,
but the QT itself is embedded in the "calling" code flow.
Implementing a special metainstruction for subroutine call just allows
to place the body of the subroutine outside the main code flow.

In the single-processor approach, when another process is to be run,
the processor must be stolen
from the running process, which needs a lot of saving and restoring,
as well as context changing in advanced systems; i.e. rather much
non-payload activity. In EMPA approach, a core can be reserved
for interrupt servicing. It can be prepared (even in kernel mode)
and waiting for the interrupt. When the interrupt arrives
to the core waiting in power economy mode, it immediately starts its servicing,
without any duty to save and restore, saving processing time and memory cycles.
Note that in EMPA approach no context change is needed,
resulting in several hundreds of performance gain relative to the conventional

Similarly, cores can be prepared to provide kernel services.
Some system services, for example semaphore handling, do not really need
all the facilities of the OS, they can be implemented in some alternative way.
As our former measurements on soft system <cit.> proved,
such alternative implementation resulted in performance gain about 30,
although in that case no context changing was needed.
Similar gain can be expected when implementing OS services with EMPA.
The gain factor will surely be increased because of the eliminated
context change, but the concrete gain will depend on the functionality of the service.
In addition to making context changing
(in both directions) obsolete, the kernel and user codes can run even partly parallel.

§.§ Mass processing

The physical vicinity also allows to implement certain kinds of
cooperation between the cores.
Here the source of performance gain can be to eliminate
control-type instructions (like loop counter advancing, checking, jumping)
as well as to eliminate unnecessary stages of instruction execution,
read and writeback when only the final result is interesting.

A typical example is summing up elements of a vector (see Listing <ref>): the read and write back of the partial sum is needed only when considering the machine instructions
as atomic unit. In mass operating mode,
the parent can sum up summands provided by its children,
in frame of a machine instruction.

§.§ Linking special accelerator

As it follows from the description above, for the SV a core is represented
as a source and destination of signals and data.
i.e. an extremely simple interface is provided for linking QTs.
Since the QT can be as large as a SW thread,
and SV only "sees" the signals and data, but no HW at all,
EMPA provides an
extremely simple interface for linking any kind of external accelerator
(without any non-payload activity) or even HW-implemented SW processes
like <cit.>.

The communication signals and data between parent and child cores in EMPA

§ ARCHITECTURE OF AN EMPA PROCESSOR

Many of the experts of the field expect some kind of solution
from the RC technology (see for example <cit.> and its cited references).
However, since the computing density <cit.> cannot be increased using this
technology, the RC elements are mostly used to control the
conventional components of the processor (for example <cit.>)
or to accelerate some OS services <cit.>, or one can find even complete
RT OSs (for example <cit.>) implemented in HW.
At the other
end of the scale one finds also operating systems
with SW threads implemented in HW <cit.>, too.
When thinking about the EMPA architecture, one should have in mind also
RC technology combined with the conventional processor technology.
An EMPA processor is akin an end-user
configured dynamic (processor) architecture using dedicated wiring and
special LUs (cores).

§.§ Using multiple processing units, explicitly

Of course, the introduced new operating principle requires modifications also
to the internals of the processor. Since our goal (in addition to
boost single-thread performance) is to preserve as much compatibility
as possible, with the conventional computing,
some components are greatly similar to the conventional ones,
and also the newly introduced components have functionalities,
which can be formulated using terms similar to the conventional computing.

§.§.§ The processor

For the outside world, the processor is nearly unchanged.
It receives a stream of instruction codes, which is enriched
with metainstructions, describing the suggestions of the compiler
about increasing performance. Internally, however, it works

§.§.§ The cores

The cores in an EMPA processor are mostly similar to the present single-core
processor, with some extra functionality.
The cores provide and receive extra signals (see also see also Fig. <ref>) for/from the SV

* Availability a core is available when it is not executing a code chunk, not preallocated for a future task,
and not disabled for some reason (like overheating)

* Enabling when a core is rented, it gets enabled and disabled again
when the code chunk terminates

* Waiting a core can be blocked when waiting for
the termination of some QT running on another core

* Metainstruction signals that during pre-fetch a metainstruction found

Also store the cores extra information (see also Fig. <ref> and section <ref>):

* Identity The cores are identified by a (hard) "one-hot" bitmask

* Parent The (configurable) identifying bit mask of the parent of the core

* Children The (configurable) ORed value of the bitmasks of cores with QT created by the QT running on this core

* Preallocated The (configurable) ORed value of the bitmasks  of cores preallocated for this core

* Offset The (configurable) memory address of the
QT the core runs

§.§.§ The supervisor

This new unit is a second, end-user configurable control layer,
which, among others, handles the computing resources
and reports "ALU avail" signal.
It can be considered as a kind of second-level control unit,
implemented as a control layer above and between the PUs (cores).
It transfers signals to and from cores; as such, all data and control transfers can be implemented between SV and PU, so
a 'star' topology shall be implemented, and there is no need
to wire the cores with all other cores.

The  SV is responsible for providing a new processing unit
(to rent a core) when needed, to  handle properly the termination signals from the
rented cores (including handling the mentioned bitmasks), to provide synchronization signals for a core running
a QT, provide (limited) synchronized data connection between the cores, etc.
It  comprises several simple,
easy to implement and quick to execute  functionalities.

The SV is responsible for all resources,
so  it can only be used in a sequential way, one operation at a time.
It could be a bottleneck for the performable operation of the processor,
so its proper handling  requires special attention when designing
cooperation between the  PUs, working in parallel.

The conclusion that "the ever-increasing complexity of superscalar processors would have a
negative impact upon their clock rate, eventually leading to a leveling off of the rate
of increase in microprocessor performance" <cit.> also means, that when introducing
a new control layer (SV) on top of cores in the processor, its simple combinational logic
can be operated at a frequency, allowing  high-speed coordination of the cores' work;
much higher than the clock frequency needed for the cores for
making sophisticated computations.

§.§.§ The memory

The more PUs obviously need broader memory access bandwidth,
but the burden of memory bandwidth is not as bad as one might think for the first look.
In the SPA systems
one processor is linked trough one bus to one memory decoder.
However, in the hyper-threaded architectures several outstanding memory
access requests can coexist.

One must remember, the producer/consumer model:
the "memory wall" is still active.
Even with  EMPA,  the many-core processors
cannot receive more memory contents, than the memory subsystem can produce.
EMPA can make, however, good use of multiple memory access devices.
This ability might need to change the memory access architecture:
the many-core processors might need more than one independent memory buses,
the buses can be (time or space) multiplexed, and the memories might need
multiple decoders to the same memory address space.
To broaden the memory-access bandwidth, independent multi-port memories are needed, like <cit.>.

In many-processor systems a lot of efforts are needed to provide
coherent operation of the CPUs. In EMPA, the PUs
have coordinated operation, so the accidental simultaneous access
can be eliminated by the compiler/SV. For examples see
the sample programs below; the interrupt or OS service operation
or direct memory access: the logic of the (cooperative) operation
excludes the simultanous access, so using the (relatively slow and energy wasting)
shared memories (like in <cit.>) are not necessary here.

§.§ The parent-child relationship

The cores are uniform and independent, but
using the mentioned bitmasks, they can be in parent-child relationship,
to arbitrary depth. A core can have only one parent,
but an arbitrary number of children. This relationship allows for
several generations, unlike the master-slave relationship,
used  in some other architectures like the "Desktop Supercomputer" <cit.>
and allows for dynamic behaviour similar to that of the "Invasive Computing" <cit.>.

Fig. <ref> attempts to summarize the signal and data
traffic of the cores. At the top of figure the core is in role parent and at the bottom in role child.
These roles are of course context-dependent, in another context
a child can be parent of another core, or a core cannot have a child at all. The shown storages and signals are typical for that role.
Akin in FPGAs, some well-defined, mostly fixed functionality blocks are
placed in close vicinity to each other, and the end-user has the possibility
to connect them, changing some configuration parameters, which selects
one of the predefined functionalities.

§.§ The dynamic architecture

The individual cores take the responsibility for executing a dedicated QT.
The QTs can be nested, i.e. a core
may face the task to delegate part of its job to another PU.
It can be solved (with the active help of SV)
using the parent-child architecture.
If there is at least one available core, the  SV rents it
from the pool of cores for the requesting core, and administers it as a child of the requesting core, in both cores in the configurable bitmasks.
This means, that the "processing graph" will be mapped to the available cores.
parent-child relationship
mapping!QTs to cores

The child core gets enabled and begins its  independent (and parallel!) life. A child core might find a termination metainstruction (in contrast with the conventional processor, where only 'halt' is possible), which leads to notifying the SV (see also Figs. <ref> and <ref>).
The SV administers the termination of the parent-child relationship,
and puts back the (former child) core into the pool.
From that moment that core might be rented for another task.

The parent is responsible for performing the complete task of the QT
it received, even if it delegated part of the job (in form of child QTs)
to its child cores. This means, that a parent QT must wait the
termination of all of its child QTs, in order to be sure the work completed. To do so, the SV will
block the termination of a parent QT until its children mask
gets cleared.

§.§ Data passing

A crucial question is passing data between cooperating cores during processing.
Splitting the code into QTs in a reasonable way, allows to
make a bargain between loosing performance because of transferring data
and gaining performance because of using more PUs.
Anyhow, some data passing in inevitable.

When a piece of the code delegated originally to the parent core
is delegated to the child core, the child core must inherit also the
internal status of the parent, including the contents of the registers.

It needs dedicated wiring between the cores and the SV,
and (depending on the
physical location of the cores) can take somewhat longer time than the
other SV operations. In this case the synchronization is not a problem: the child core commences its life after it received the needed data.

Upon termination of QTs, however, synchronization of the eventual returned data might be an issue: the children cannot know
when to copy back data into registers of the parent.
To solve this problem, the SV latches the data  returned by the child, and when the
parent is about to terminate or explicitly waits for the
termination of the child, transfers it to the parent core.
This type of information transfer requires dedicated wiring
between the core and the SV, and should be implemented
as a two-stage transfer.
In this case the SV acts as a switching center, so making
dedicated wiring between a core and the rest of cores can be avoided.

[scale=1,every node/.style=minimum size=1cm,on grid]

every node/.append style=yslant=,xslant=,

[black, dashed, thin] (0,0) rectangle (5,6);

(1,5) circle (.2)
(2,4) circle (.2)
(3,3) circle (.2)
(3,5) circle (.2)
(4,3) circle (.2)

(1.2,5) to[out=0,in=90] (2.1,4.2);
(1.8,4) to[out=180,in=-90] (1.0,4.8);
(2.2,4) to[out=0,in=90] (3.0,3.2);
(2.8,3) to[out=180,in=-90] (2.0,3.8);
(3.2,3.0) to[out=45,in=-45] (3.2,5.0);
(3.0,4.8) to[out=-90,in=135] (2.8,3.0);
(3.0,3.2) to[out=90,in=90] (4.0,3.2);
(4.0,2.8) to[out=-90,in=-90] (3.0,2.8);

(1.5,5.4) node[left,scale=.7]
(2.5,4.4) node[left,scale=.7]
(1.9,4.9) node [scale=.6, rotate=-40]
(1.1,4.1) node [scale=.6, rotate=-40]
(2.9,3.9) node [scale=.6, rotate=-40]
(2.1,3.1) node [scale=.6, rotate=-40]
(3.4,4.6) node [scale=.6, rotate=-40]
(4.6,2.4) node[left,scale=.7]
(4.2,3.4) node [scale=.6, rotate=-40]
(3.4,2.3) node[left,scale=.7]
(0.5,6.5) node[right, scale=.7]

[ultra thin](-2,4) to (-2,-0.25);
[ultra thin](-0.4,3.6) to (-0.4,-0.45);
[ultra thin](1.2,3.5) to (1.2,-0.65);
[ultra thin](0.0,5) to (0.0,0.65);
(-0.75,2.1) node [scale=.6, rotate=90] cloning, cache sharing
(-0.55,2.1) node [scale=.6, rotate=90] ForChild,FromParent
(-0.25,2.1) node [scale=.6, rotate=-90] FromChild,ForParent
(-0.05,2.1) node [scale=.6, rotate=-90] link register

every node/.append style=yslant=,xslant=,

[white,fill opacity=.75] (0,0) rectangle (5,6);
[black, dashed, thin] (0,0) rectangle (5,6);

(1,5) circle (.2)
(2,4) circle (.2)
(3,3) circle (.2)
(3,5) circle (.2)
[-latex, thin]
(2.8,5) to[out=180,in=120] (1.8,4);
[-latex, thin]
(2.2,4) to[out=0,in=-60] (3.2,5);
[-latex, thin]
(2.8,5) to[out=-120,in=120] (2.8,3);
[-latex, thin]
(3.2,3) to[out=60,in=-30] (3.2,5);
[-latex, thin]
(3,5.2) to[out=120,in=60] (1,5.2);
[-latex, thin]
(1,4.8) to[out=-60,in=-120] (3,4.8);

(0.5,6.5) node[right, scale=.7]
(1.5,5.4) node[left,scale=.7]
(2.4,3.5) node[left,scale=.7]
(3.5,2.6) node[left,scale=.7]
(3.5,5.4) node[left,scale=.7]
(2.1,5.0) node [scale=.6, rotate=-40] QxCreate
(2.7,3.4) node [scale=.6, rotate=-40] QxWait
(2.8,4.0) node [scale=.6, rotate=-40] QTerm

How EMPA processor operates at supervisor and core levels

§.§ Two-level operation

Fig. <ref>is somewhat similar to a state diagram,
and might help two understand the operation of EMPA.
Actually, the actions take place at two different levels.

At the beginning, the SV "creates" the cores, i.e. it
initializes its internal data structures and places them
in a pool of sharable PUs. When in the pool, the operation
of the cores is of course not enabled. One of the cores gets
allocated and will be enabled.

In that state, the core will work as a traditional processor,
with the exception that the SV can disable its operation,
and during the pre-fetch stage it decides whether the next instruction
is a normal executable instruction or a meta-instruction.
In the former case, the core executes the executable instruction
as the conventional processors do.
In the latter case, using its 'Meta' signal, the core notifies
SV. In response, the SV takes over the execution
of the metainstruction: advances the PC of the core
to the next instruction at the core level, and 'executes'
the meta-instruction at the supervisor level.

If the metainstruction is to create a QT, it means that
new core(s) must be rented from the pool (the HW is provided),
and equipped with proper internals (the SW is provided).
Then the new core begins its independent life. When it reaches
a metainstruction, it notifies the SV.

Waiting can occur only as an effect of a metainstruction.
When there are no more cores in the pool at the moment, or a parent is about
to terminate without its children being terminated or an explicit
waiting requested, the SV simply disables the core,
until the condition fulfilled.

Parameter passing also happens under SV control.
The latched registers (to/from child/parent) are filled in the partners correspondingly, when creating or terminating a QT
(executing |QxCreate|
or |QTerm|) triggers the action. As shown,
SV is fully responsible for syncronized data transfer between latched registers,
but its operation is based on how the programmer configures
SV through issuing the corresponding metainstructions.

§.§ Pseudo registers

Another crucial question is how such unusual data passing can be
used ("programmed") in a way, close to the conventional one.
The conventional cores use registers for quick manipulation of data,
so a useful idea is  to use pseudo-registers for transferring
data between a child and its parent.
In this way both the parent and the child see a "register", which is
a well-known term for the cores. However, it is not an item in the
register file: it has a register address, but it also has a
context-dependent functionality.
In this way the parent and the child can share some data
in a way, which appears as handling registers.

This pseudo-register migh have a bit longer access time
(depending on the physical position of the cores and the additional electronic
functionality, hidden under the facades of the register and depending on the operating mode),
but surely shorter than reaching any memory or using any kind of internal network.
Different operating modes for collective work can be defined,
and through those (pseudo)registers  the quickest possible
data transfer can be reached.

In normal mode a pseudoregister behaves (nearly) as a traditional register,
except that it is mapped to a latch register of the core.
In mass processing mode  a pseudo register behaves in a quite special way. For example, what the parent writes in its own register,
children can read from their own register, and what children write in their own register,
the parent can read from their own register. In order to avoid syncronization issues,
the SV latches the sent data when the sender is ready
to send it, and allows the receiver to read the data from the
latch when the receiver is ready to accept it.

As shown in Figs. <ref> and <ref>, (pseudo)register transfer
might happen in several situations, and in all cases under the control
of the SV.
The first data transfer occurs when renting a new core.
First of all, the SV handles the corresponding bitmasks
'Parent' and 'Children', and clones the complete internal state
(including the register file and the PC)
of the parent to the new child. When ready, the SV
enables the child, which begins its independent (and parallel!)
life with executing the delegated code chunk.
After this, the parent skips that (logically already executed) code fragment
and continues execution at that new address.
When the child is about to terminate, it notifies its SV
with the 'Meta' signal. In response, the SV latches
the content of the link register of the child core for the parent,
changes the contents of register 'Children' and 'Parent' correspondingly. The latched register content is available
for the destination parent core through issuing an explicit or implicit 'Wait' (recall that  termination implies a 'Wait'),
and will be written from the latch into the corresponding register
only when the parent requests so. In this way no syncronization
issues might happen.

When cores are allocated for mass processing, registers 'ForChild' and 'FromChild'
will be initialized in the parent core. The parent can write 'ForChild'
through writing its own pseudoregister and read 'FromChild' through
reading its own pseudoregister. (Remember, the PC of the parent
might stall at the address whare mass processing begins; and also
that SV can read/write any of the registers, independently
of the operation of the  cores.)

In mass processing mode, the 'Mode' code and 'ForChild' are latched by the child
when preallocatin cores.
If a child writes to its own pseudoregister,
the value will be latched in 'FromChild' in the parent, and the next (repeated)
QT creation might consider that value.
Depending on the 'Mode', the parent might watch and read
through reading its own pseudoregister
register!latched for another core
the latched value written by the child to its own pseudoregister.
Since for reading and writing pseudoregister as 'parent' or as 'child' there are two different directions
(two different latched registers), some special rules determine its context-dependent utilization, see <cit.>.
To forward data (i.e. to transmit, data received on its input,
to its output) the core needs to use an explicit copying from the
input pseudoregister to the output pseudoregister instruction.
For the programming implementation see <cit.>.

[language=[y86]Assembler, wide, options=numbers=left,number=-1,

] Qasum0_4.Eyo
The sum-up routine in traditional codinglst:Qasum04

§ UTILIZING THE ARCHITECTURE

After having an architecture presented above, one shall find utilization
possibilities, and implement them. As an example of the possibilities,
a program summing up elements of a vector is presented.
The example in Listing <ref> has been adapted from the 'asumup' program <cit.>.
The program is written in Y86 assembly language, extended with EMPA metainstructions.
The toolchain for  EMPA
(including the assembler and simulator, as well as some programming methodologies)
is described in a separate paper <cit.>.
Since Y86 is a simplified for education version of the widely known Intel x86 processor,
the coding can be followed easily. Here only a qualitative description
can be given. More programming details are described in <cit.>,
the mentioned sample programs, toolchain, their user guides
are available from <cit.>.

First the number of arguments and vector address (lines 3-4)

are loaded into registers.
Then the program clears the sum (line 6),

and  verifies the number of items (line 8).
The kernel of the calculation is in lines 9-15.
First the actual vector element is loaded

into |
then  it is added

(lines 9-10)
to the partial sum stored in |
After this, the address of the actual element is advanced properly (lines 11-12),
then the loop counter advanced (lines 13-14).

Finally, depending on the actual conditions (line 15)

the loop
returns back to repeat the procedure for the next vector element.

§.§ Eliminating obsolete instructions'(FOR)

As it can be seen from Listing <ref>, this implementation is
rather ineffective:  the payload work is done in lines 9-10,
while lines 11-15 only serve loop organization. If the SV
could take over loop organization, we could reach a performance
gain about 3. This needs thinking in terms of EMPA.

Actually, the program fragment needs the address of the current summand
and a register to store the partial sum. I.e. we can separate lines 9-10
in a QT and organize the job around this approach.
So, lines 9-10 will be executed by the child, on the request from the
parent. As mentioned, the parent clones its "glue" to the child,
so the address of the array is accessible for the child in |
the old partial sum is delivered in |
partial sum is cloned back to the parent also in |
The child terminates after executing lines 9-10, allowing the parent
to know that one element has been added to the partial sum.
So, the job for providing the right address and counting the iterations
remains for the parent. Since the parent is only waiting while the
child terminates, its arithmetic facilities can be used for this task.

The parent pre-allocates a child for the work. The pre-allocation
is needed because the other cores might allocate cores in parallel,
so through the preallocation is guaranteed that always will be available
core for the iterations. That child core will be allocated by the parent as many times as needed,
and so the loop will be jointly executed by the child and the parent.
The parent starts executing an iteration (as many times as needed),
and waits until the child terminates. The SV also participates
in the game: calculates the address of the vector element for the next iteration
and delivers data and signals between parent and child.

The parent writes the current address to its latched 'ForChild' register
(see Fig. <ref>)
and it will be latched by the child upon creation into its own the latched 'FromParent'
register. The number of the remaining iterations are stored in the latched
'FromChild' in the parent. Before instructing the parent to start a new iteration,
SV checks if the content of 'FromChild' is cleared. If not,
it instructs the parent to start the next iteration and decrements
the latched 'FromChild' in the parent. Since the child can write its latched
'ForParent' register, the content of which is transferred upon termination
to the latched 'FromChild' in the parent, the child can |break|
the loop.

§.§ Eliminating obsolete stages (SUMUP)

Summing up elements in a vector is a simple and ideal example of processes
which cannot be parallelized: in the addition one of the summands is the
previous partial sum, so the next summing cannot be started until the
previous one is terminated.

The closer look shows that it is because in the frame of executing
a single instruction the processors must read the content of a register and write it back,
updating it with the new partial sum. We can also see that the partial
sum is never used, we are only interested in the final sum.
This means, that if we can find a way in the architecture, which allows a
cooperative execution of adding, furthermore a separated readout or
the final sum, we can parallelize this strictly sequential process.

To implement this, needs a bit more functionality from the SV.
Suppose we have enough cores which can be allocated, and preallocate them.
In this special operating mode which allows opening execution stages
for a child, an adder is prepared in the parent, which on one input
receives the latched 'FromChild', and on the other input receives
its own output (the previous partial sum).
The child receives the current address as described in the previous section,
and in this special mode executing |addl| to a special
pseudo register means writing to 'ForParent' in the child, which also
triggers transferring to 'FromChild' in the parent, i.e. the content
read out by the child will be added to the partial sum stored in the parent.

§.§ System services and other uses

Some system services, for example semaphore handling, do not really need
all the facilities of the OS, so they can be implemented in alternative way.
As our former measurements on soft system <cit.> proved,
such alternative implementation resulted in performance gain about 30,
although in that case no context changing was needed.
Similar gain can be expected when implementing OS services with EMPA.
The gain factor will surely be increased because of the eliminated
context change, but the concrete gain will depend on the functionality of the service.

§ PERFORMANCE OF THE ACCELERATOR

As discussed above, using EMPA architecture can distribute
the code between PUs,
can eliminate
obsolete instructions, obsolete execution stages, obsolete context changes, etc.
The machine instruction execution remains the same, so to measure performance
is not simple at all. Practically, the speedup (the ratio of the execution times)
remains the only measurable quantitity. Recently, a figure of merit characterizing
the effective parallelization <cit.> has been developed,
so below that merit will be used to describe the performance gain due to utilizing EMPA,
and it will be compared to the traditionally used merit: the speedup
divided by the number of cores ($\frac{S}{k}$).
The effective parallelization can be derived from the number
of PUs $k$ and the measured speedup $S$ as

\begin{equation}
\alpha_{eff} = \frac{k}{k-1}\frac{S-1}{S} \label{equ:alphaeff}
\end{equation}

The conventional methods of parallelization
suffer from inefficiency in using computing power of multiple PUs:
because of the presence of the sequential-only part,
the more cores are used, the lower is the value of $\frac{S}{k}$,
while $\alpha_{eff}$ really describes correctly how effectively
the cores are utilized.  The 'sumup' program has been tested in three versions:
using the conventional programming (i.e. NO EMPA acceleration),
replacing "control" instructions with SV activity in the FOR mode,
and using SUMUP mode, where (in addition to eliminating control instructions) the cooperation eliminates
the unneeded  read/write back cycles within a machine instruction.

The results (measured using the simulator <cit.>) are compared in Table <ref>.
for different vector length and different number of cores.
The simulator uses arbitrary, but reasonable
execution times, expressed in units of the control clock driving
the SV. The actual values might change when an electronic
version (RC implementation) allows to provide more accurate data.

tableheadcolorVector    Mode of     Time	        No of          Speedup      $\frac{S}{k}$    $\alpha_{eff}$

tableheadcolor length     mass proc     (clocks)	    cores (k)        (s)

1     NO        52        1       1          1        1

1     FOR      31         2      1.68       0.84      0.81

1     SUMUP    33         2      1.58       0.79      0.73

2     NO        82        1       1          1        1

2     FOR       42         2      1.95       0.98      0.97

2     SUMUP     34         3      2.41       0.80      0.87

4     NO        142        1       1          1        1

4     FOR      64         2      2.22       1.11      1.10

4     SUMUP    36         5      3.94       0.79      0.93

6     NO        202        1       1          1        1

6     FOR       86         2      2.34       1.17      1.15

6     SUMUP     38         7      5.31       0.76      0.95

Effective parallelization in EMPA processor, in different operating modestab:effectivealphaEMPA

/pgf/number format/.cd,
use comma,
1000 sep=,
legend style=
legend pos=north west,
xmin=0, xmax=7,
ymin=1, ymax=5.5,
xlabel=Vector length,
ylabel=Measurable speedup,
[ very thick, color=red,mark=*] plot coordinates
FOR performance gain
[ very thick, color=blue,mark=*] plot coordinates
SUMUP performance gain

Diagram showing data from Table
the measurable speedup for two different mass processing methods, in function of the vector length.

§.§ Analyzing speedup results

As Table <ref> shows, both the conventional
and EMPA execution times increase linearly with the length
of the vector. Their intersect and slope values, however,
are very different.

The FOR mode of EMPA is nearly
3 times quicker than the conventional method,
as some computed control statements are replaced by the
much more effective SV functionality.
This requires only 2 PUs.

In the SUMUP method, in addition to omitting the
computed control machine instructions, even the obsolete
fetch, decode, writeback, etc. stages of one instruction execution
in the loop kernel
are replaced by  SV functionality.

For the SUMUP
mode an extra element increases the execution time by one clock cycle,
at the price of utilizing one more PU.

This behavior is especially valuable, because using conventional
methods of parallelization the algorithm cannot be parallelized at all.

The measured speedup values are derived from a mixture of different types of instructions: both conventional and EMPA code contains both sequential and parallel parts, so despite of the linear
increase, the measured speedup will not linearly depend on the
vector length, see Fig. <ref>. The two speedup values
will saturate for high vector lengths at values $\frac{30}{11}$
and $30$, respectively.

§.§ Analyzing parallelization efficiency

When eliminating
in this very simple loop the control instructions in FOR mode,
the $\frac{S}{k}$ values can even be above unity.
This means not a higher PU performance,
it is due to the more clever organization of cycles.

/pgf/number format/.cd,
use comma,
1000 sep=,
legend style=
legend pos=north west,
xmin=0, xmax=7,
ymin=.7, ymax=1.25,
xlabel=Vector length,
ylabel=Core utilization efficiency,
[ very thick, color=red,mark=*] plot coordinates
FOR $\frac{s}{k}$
[ very thick, color=red,mark=x] plot coordinates
FOR $\alpha*{eff}$
[ very thick, color=blue,mark=*] plot coordinates
SUMUP $\frac{s}{k}$
[ very thick, color=blue,mark=x] plot coordinates
SUMUP $\alpha*{eff}$

Diagram showing data from Table
the core utilization efficiency for two different mass processing methods, in function of the vector length.

In the SUMUP mode, the helper cores are utilized only for a short
period of time, so the utilization efficiency is low for short vectors.

Note that since the PUs
are put back in the pool, for very long vectors much lower number of cores might be needed.
If the compiler can find out the length of processing in that mode
(in our example it is 30 clock cycles), it should not allocate
more that that number of cores: when the parent needs the 31st core,
the 1st core is available again, so the summing can be continued
for arbitrary vector length.
In calculating the effective utilization of cores using Equ. <ref>,
$k$ should be replaced with
$\max\limits_{k\leq  k_{eff} \leq 30}$.

/pgf/number format/.cd,
use comma,
1000 sep=,
legend style=
legend pos=north west,
xmin=0, xmax=600,
ymin=.45, ymax=1.15,
xlabel=Vector length,
ylabel=Core utilization efficiency,
[ very thick, color=blue,mark=*] plot coordinates
SUMUP $\frac{s}{k}$
[ very thick, color=blue,mark=x] plot coordinates
SUMUP $\alpha*{eff}$

Efficiency $\frac{S}{k}$ and $\alpha_{eff}$ for EMPA processor in the SUMUP mode, in function of the vector length.

The two different points of view of the two merits is best reflected in Fig. <ref>,
where $\frac{S}{k}$ and $\alpha_{eff}$ are shown in function of the vector length,
using an EMPA processor in SUMUP mode.
Because of the effect of sequential code fragments, both curves
start increasing with increasing the number of the cores.
As mentioned,
in this mass processing mode max. 31 cores (1 parent plus 30 child cores) can be used.
There is no sense to use more then 30 child cores: they would need to wait for sending
their summand for the parent.
Because of this, both the
number of threads and the speedup keep raising, while the number of the cores
saturates at 31. For short vectors, $\alpha_{eff}$  is relatively low,
because the helper cores are used only in a fragment of time.
As all the 30 helper cores have "full time job", the $\alpha_{eff}$
dependence saturates at value $1$. In contrast, $\frac{S}{k}$ starts
to decrease with increasing the number of the cores,
and after reaching 30 cores, the speedup continues, but $k$ remains constant, so the dependence turns back and saturates also at value $1$, but approaches it much more slowly.

§ CONCLUSIONS

The idea of introducing EMPA in processor technology opens
a series of new possibilities. As main accelerator, it allows to turn
a many-core processor to an extremely high-power single-core processor.
To make an old single-thread program many-core aware, it is enough to
recompile the program using an EMPA-aware compiler and run it
on an EMPA architecture processor.
EMPA uses no hidden PUs: the same cores can be "rented" for normal
code execution and out-of-order or speculative evaluation.
This means that the superfluous logic <cit.> concerting and hiding the extra PUs
can be omitted, simplifying the internal architecture, reducing the number of
transistors and reducing also the power consumption.

For calculational applications, several hundred times higher
parallelization can be reached: the compile-time discovery of parallelization
possibilities and mixing thread and instruction level parallelism
tends to reach the ideal case of "infinite resources" <cit.>.
In addition to that theoretically checked possibility, the SV can
more efficiently perform some control functions from loop organization
to opening the closed von Neumann execution frames for the helper cores,
raising at least one more order of magnitude in the performance.

Using OSs are getting more simple and more effective with
using EMPA: no context changing is needed, and the user mode and
kernel mode programs can run in (at least partly) parallel.
Since QTs are by their nature atomic processing units, a big part of
operating systems dealing with semaphors for shared resource usage,
scheduling, etc. becomes obsolete (greatly reducing the amount of codes,
both writing and testing),
and also the built-in synchronization of EMPA can replace those
services offered by the OSs.

The real-time characteristics of processors also benefit from EMPA.
To service an interrupt, no state saving and restoring is needed,
saving memory cycles and code. The program execution will be predictable:
the processor need not be stolen from the running main process.
The atomic nature of executing QTs will prevent issues like
priority inversion, eliminating the need for special protection protocols.

From the point of view of accelerators, an EMPA processor
provides a natural interface for linking special accelerators to the
processor. Any circuit, being able to handle data and signals shown in
Fig. <ref> can be linked to an EMPA processor with easy.

[]VÉGH, János
received his PhD in Physics in 1991, ScD in 2006.
He is working in computing since the early 80's, and since 2008
is a full professor in Informatics, currently with University of Miskolc, Hungary.
In research, he is looking for ways to prepare more performant and
more predictable computing, especially using mainly multi-core processors.
He is also dealing with soft processors and hardware-assisted operating
systems, reconfigurable and many-processor computing.