# Topological interface states – a possible path towards a Landau-level laser
in the THz regime

Mark O. Goerbig<EMAIL_ADDRESS>Laboratoire de
Physique des Solides, Université Paris Saclay, CNRS UMR 8502, F-91405 Orsay
Cedex, France

###### Abstract

Volkov-Pankratov surface bands arise in smooth topological interfaces, i.e.
interfaces between a topological and a trivial insulator, in addition to the
chiral surface state imposed by the bulk-surface correspondence of topological
materials. These two-dimensional bands become Landau-quantized if a magnetic
field is applied perpendicular to the interface. I show that the energy
scales, which are typically in the $10-100$ meV range, can be controlled both
by the perpendicular magnetic field and the interface width. The latter can
still be varied with the help of a magnetic-field component in the interface.
The Landau levels of the different Volkov-Pankratov bands are optically
coupled, and their arrangement may allow one to obtain population inversion by
resonant optical pumping. This could serve as the elementary brick of a multi-
level laser based on Landau levels. Moreover, the photons are absorbed and
emitted either parallel or perpendicular to the magnetic field, respectively
in the Voigt and Faraday geometry, depending on the Volkov-Pankratov bands and
Landau levels involved in the optical transitions.

##  I Introduction

Landau levels (LLs), which arise due to the quantization of the electrons’
energy in a strong magnetic field, have been regularly proposed to be a
promising system for a frequency-tunable laser in the THz regime Tager and
Gladun (1959); Wolff (1964); Aoki (1986); Morimoto  _et al._ (2008); Gornik
_et al._ (2021). Indeed, upon a putative population inversion between the LLs
$n$ and $n+1$ in parabolic bands, one may expect cyclotron emission with a
typical frequency $\Omega_{n+1,n}=\omega_{c}$ given by the cyclotron frequency
$\omega_{c}=eB/m_{B}$, which is directly controlled by the strength of the
magnetic field $B$ and the band mass $m_{B}$. In spite of this conceptually
appealing proposal, the path to the realization of a working LL laser is
barred by strong obstacles that are mainly concerned with population
inversion. The latter requires rather long-lived electrons in the excited LL,
but their lifetime is strongly reduced by non-radiative recombinations, namely
Auger processes that are prominent due to the equidistant LL separation
Potemski  _et al._ (1991) (for a detailed discussion of these processes, see
Ref. [But  _et al._ , 2019]). In such processes, an electron in the excited LL
$n+1$ can be promoted due to electron-electron interactions to the LL $n+2$
while the required energy is provided by a simultaneous desexcitation of
another electron from $n+1$ to $n$. Instead of using one excited electron to
emit a photon of frequency $\omega_{c}$, two electrons in the LL $n+1$ are
thus lost without emission of any photon. Another obstacle equally related to
equisitant LLs is reabsorption of cyclotron light due to the transition
$(n+1)\rightarrow(n+2)$, which is resonant with the $(n+1)\rightarrow n$
transition used in the emission of light Lax (1960); But  _et al._ (2019);
Gebert  _et al._ (2023).

Soon after the isolation of graphene, physicists explored this material in
cyclotron-emission experiments in the perspective of realizing a LL laser
Morimoto  _et al._ (2008); Wendler and Malic (2015); Gornik  _et al._ (2021).
Due to the linearly dispersing bands of graphene electrons in the vicinity of
charge neutrality, the LL spectrum is given by
$E_{n}=\pm\hbar(v/l_{B})\sqrt{2n}$, in terms of the Fermi velocity $v\simeq
10^{6}$ m/s and the magnetic length $l_{B}=\sqrt{\hbar/eB}\simeq
26\,\text{nm}/\sqrt{B\text{[T]}}$, i.e. the levels are no longer equidistant.
While the orders of magnitude with a fundamental gap of $\hbar\Omega_{1,0}\sim
100$ meV for magnetic fields $B\sim 10$ T are promising for possible THz
applications, Auger processes remain a relevant source of non-radiative
recombination processes also in these relativistic systems Plochocka  _et al._
(2009). For example, while the $1\rightarrow 0$ transition is no longer in
resonance with the neighboring $2\rightarrow 1$ transition, it is in resonance
with the transition $4\rightarrow 1$ due to the square-root dependence of the
LL on the level index $n$ But  _et al._ (2019); Goerbig (2011). Furthermore,
it has been shown that the optical phonon responsible for the G band in
graphene (at $\sim 200$ meV) also enhances decay processes that are
detrimental to population inversion Kazimierczuk  _et al._ (2019). The
drawback of resonant transitions and enhanced Auger processes can to some
extent be healed, e.g. in gapless HgTe/CdTe quantum wells, where the low-
energy electrons are described in terms of so-called Kane fermions. While
their zero-field spectrum is similar to that of massless Dirac fermions, LLs
with even indices are absent in the spectrum so that some transitions are
absent, such as the above-mentioned transition $4\rightarrow 1$.

An extremely interesting route towards the realization of a LL laser is the
use of Dirac materials with a (mass) gap $\Delta$ that is on the same order of
magnitude as the typical LL spacing, i.e. in the 100 meV range, for systems
with a characteristic velocity parameter of $v\simeq 10^{6}$ m/s. In this
case, the LL spectrum is given by

$E_{\lambda,n}=\lambda\sqrt{\Delta^{2}+2\hbar^{2}v^{2}n/l_{B}^{2}},$ (1)

where $\lambda=\pm$ is the band index. Indeed, if $\Delta\sim\hbar v/l_{B}$,
the LL spectrum is neither (approximately) linear in $n$ and $B$ as it would
be in the limit $\Delta\gg\hbar v/l_{B}$ nor does it follow the square-root
dependence of graphene in the opposite limit $\Delta\ll\hbar v/l_{B}$. In this
case, the absence of simultaneous resonant transitions suppress both
reabsorption and non-radiative Auger scattering. First encouraging results in
this direction have been obtained in gapped HgTe/CdTe quantum wells Gebert
_et al._ (2023). Another system in which massive Dirac fermions occur is the
interface of a topological and a trivial insulator, in the form of Volkov-
Pankratov (VP) states Volkov and Pankratov (1985); Pankratov  _et al._ (1987);
Tchoumakov  _et al._ (2017a). The bulk-surface correspondence for topological
materials enforces indeed the occurence of a massless chiral state at such an
interface, but it has been shown that the interface spectrum is much richer in
systems with smooth interfaces, e.g. when the gap changes over a certain
distance $\ell$ that characterizes the interface width and that is larger than
an intrinsic length $\lambdabar_{C}=\hbar v/\Delta$. In smooth interfaces
between a topological and a trivial insulator, one finds a whole family of
surface states the spectrum of which is indeed given by Tchoumakov  _et al._
(2017a)

$\epsilon_{m}({\bf q})\simeq\lambda\hbar v\sqrt{{\bf q}^{2}+2m/l_{S}^{2}}.$
(2)

Here, ${\bf q}=(q_{x},q_{y})$ is the two-dimensional (2D) wave vector in the
interface, $m$ denotes the index of the surface band, and
$l_{S}=\sqrt{\ell\lambdabar_{C}}$ is a characteristic length determining the
extension of the interface states in the $z$ direction perpendicular to the
interface. Equation (2) is indeed valid as long as the energy of the surface
bands at ${\bf q}=0$ is smaller than the bulk gap, $\sqrt{2m}\hbar
v/l_{S}\leq\Delta$. The latter condition is equivalent to requiring that the
interface width $\ell$ be larger than $m$ times the intrinsic length
$\lambdabar_{C}$ Tchoumakov  _et al._ (2017a). The $n=0$ surface state is
precisely the chiral state that survives in the abrupt limit, $\ell\rightarrow
0$, while the VP states (for $m\neq 0$) disappear in the continuum of bulk
states as soon as $\ell<\lambdabar_{C}$. Notice that the formation of VP
states is a universal property of topological materials that has been studied
not only in topological insulators Tchoumakov  _et al._ (2017a); Inhofer  _et
al._ (2017); Mahler  _et al._ (2019); Lu and Goerbig (2020); van den Berg  _et
al._ (2020a); Adak  _et al._ (2022), but also in Weyl semimetals Tchoumakov
_et al._ (2017b); Mukherjee  _et al._ (2019), graphene van den Berg  _et al._
(2020b), and topological superconductors Alspaugh  _et al._ (2020).

Very recently, inter-VP transitions have been measured within magneto-optical
spectroscopy in Pb1-xSnxSe crystals Bermejo-Ortiz  _et al._ (2023) in which
the Sn concentration determines whether the system is a trivial or a
topological (crystalline) insulator Martinez (1973); Wojek  _et al._ (2014);
Krizman  _et al._ (2018). Moreover, the concentration determines the size of
the bulk gap so that smooth interfaces may be obtained by molecular-beam
epitaxy (MBE) in which the Sn concentration is smoothly varied during the
growth process, and where the absolute band gap in the topological regime can
be designed such as to be identical to that in the trivial insulator Bermejo-
Ortiz  _et al._ (2023). This allows for a strong versatility in the
fabrication of interfaces of various widths and thus of systems with specially
designed fundamental gaps

$\Delta_{\text{VP}}=\sqrt{2}\hbar
v/l_{S}=\sqrt{2}\Delta\sqrt{\frac{\lambdabar_{C}}{\ell}}=\sqrt{\frac{2\hbar
v\Delta}{\ell}}$ (3)

between the $m=1$ VP and the chiral ($m=0$) surface states.

In the present paper, I argue that smooth topological interfaces, such as in
the above-mentioned Pb1-xSnxSe crystals, may be extremely promising systems
for the realization of long-lived population inversion if a magnetic field is
applied perpendicular to the interface that quantizes the 2D electronic motion
in the interface into LLs. The main reason for this expectation is the fact
that VP bands provide us with several families of LLs that can to some extent
be brought into close energetic proximity with LLs of the chiral surface band.
This would allow for devices similar to three- or four-level lasers in which
population inversion could be more easily achieved than in the usual LL setup.
Furthermore, optical pumping and radiative desexcitation can be chosen to
happen in different directions via an intelligent choice of the involved
transitions. Indeed, while the optical selection rules in the Faraday geometry
impose that the emitted or absorbed photons propagate in the direction of the
magnetic field for a transition coupling the LLs $n$ and $n\pm 1$, it has been
shown previously Lu and Goerbig (2019) that such transitions must obey an
optical selection rule $m\rightarrow m$ for the VP states. The selection rules
are inverted in the Voigt geometry, where the emitted or absorbed photon
propagates in a direction perpendicular to the magnetic field. The underlying
reason for these selection rules and their geometry dependence is an
intriguing analogy between the spatially changing gap parameter and LL
quantization. Indeed, the spatially varying gap parameter can be viewed as a
fake magnetic field that is oriented in the plane of the interface, and the
characteristic length $l_{S}$ plays the role of an effective magnetic length.
Via an intelligent choice of the geometry of a cavity hosting the topological
material and the involved transitions, one may therefore expected to obtain a
strong cyclotron emission in the direction of the interface while pumping the
system with photons propagating perpendicular to the interface, or vice versa.

Figure 1: Landau levels for VP surface states. The red levels correspond to
the chiral surface state with $m=0$, while the blue levels correspond to the
$m=\pm 1$ VP states. The sign indicates the band index $\lambda$ here for
notational convenience. As a consequence of the parity anomaly, the $n=0$ LL
is found only in the upper VP band ($m=+1$). The inset, adapted from Ref.
[Bermejo-Ortiz  _et al._ , 2023], shows the setup of a smooth interface
between a trivial insulator (PbSe) at $z<-\ell$ and a topological insulator
(Pb0.76Sn0.24Se) at $z>\ell$. The magnetic field is oriented in the $[001]$
direction perpendicular to the interface.

##  II Volkov-Pankratov states and optical selections in a magnetic field

Let us first review some basic features of VP states and their coupling to
light in the presence of a magnetic field along the lines of Ref. [Lu and
Goerbig, 2019]. We consider an interface between a trivial insulator in the
lower part of the device ($z<-\ell$) and a topological one in the upper part
($z>\ell$) (see inset of Fig. 1). Due to the magnetic field, the VP bands (2)
get quantized into LLs whose spectrum reads (for $n\neq 0$)

$E_{\lambda,m,n\neq 0}=\lambda\hbar
v\sqrt{\frac{2|m|}{l_{S}^{2}}+\frac{2|n|}{l_{B}^{2}}},$ (4)

where we have considered the magnetic field to be oriented perpendicular to
the interface (in the $z$-direction). For notational simplicity, we merge the
band index $\lambda$ from now on with the VP and LL indices so that $(-m,-n)$
corresponds to the $n$-th LL in the $m$-th VP band of negative energy
($\lambda=-$), whence the modulus of the indices in the spectrum (4) to avoid
confusion. Due to the parity anomaly, the above spectrum is only valid for LLs
with an index $n\neq 0$, while the $n=0$ LLs of the VP bands stick either to
the bottom of the positive-energy bands ($\xi=+$) or to the top of the
negative-energy VP state ($\xi=-$)

$E_{m,n=0}=\xi\hbar v\sqrt{\frac{2|m|}{l_{S}^{2}}},$ (5)

depending on the chirality index $\xi$. The latter can be changed if we change
the order between the topological and the trivial insulator (interface between
a topological insulator in the lower part and a trivial one in the upper
part), and it can also be altered easily by changing the orientation of the
magnetic field. The LL spectrum for the $m=0$ and the $m=1$ VP states [both
for the conduction ($m=+1$) and the valence band ($m=-1$)] are shown in Fig.
1. Notice that the surface-state-width parameter $l_{S}$ can be decreased
effectively with the help of an inplane magnetic field $B_{\parallel}$,
$l_{S}(B_{\parallel}=0)^{-4}=1/\ell^{2}\lambdabar_{C}^{2}\rightarrow
l_{S}^{-4}=l_{S}(B_{\parallel}=0)^{-4}+(eB_{\parallel}/\hbar)^{2}$ so that the
effective energy separation between the VP states, given by Eq. (3) is
increased to Lu and Goerbig (2019)

$\Delta_{\text{VP}}=\sqrt{\frac{2\hbar
v\Delta}{\ell}}\left(1+\frac{e^{2}v^{2}B_{\parallel}^{2}\ell^{2}}{\Delta^{2}}\right)^{1/4}.$
(6)

The writing of Eq. (4) unveils the reminiscence of LLs and VP states. Indeed,
if we linearize the gap inversion over the smooth interface by a linear
function connecting a gap parameter of $+\Delta$ in the trivial insulator (at
$z<-\ell$) and $-\Delta$ in the topological insulator (at $z>\ell$), i.e.
$-\Delta z/\ell$, the system may be mapped to the LL problem of massive Dirac
fermions Tchoumakov  _et al._ (2017a). Within this analogy, the variation of
the gap parameter in the $z$-direction may be viewed as a vector potential
that stems from a “fake” magnetic field oriented in the interface, while the
physical magnetic field is oriented in the $z$-direction. Notice furthermore
that the above description can easily be generalized to a situation where the
gap in the topological insulator is not of the same size as that in the
trivial one Tchoumakov  _et al._ (2017a), in which case the effective
interface width $l_{S}$ is determined by an average between the two gaps. The
analogy between interface width and magnetic field finally yields a physical
understanding of the optical selection rules between the levels $(m,n)$ and
$(m^{\prime},n^{\prime})$, where henceforth the first index indicates the VP
band and the second one the physical LL. In the Faraday geometry, in which the
absorbed or emitted photon propagates in the direction of the magnetic field,
angular-momentum conservation imposes that the only optically active
transitions involve adjacent LL indices, $n\rightarrow n^{\prime}=\pm(n\pm
1)$, regardless of the band index $\lambda$. This needs to be contrasted to
the Voigt geometry, where the photon propagates in the plane perpendicular to
the magnetic field and where the LL index remains unchanged $n\rightarrow
n^{\prime}=\pm n$. Since the fake magnetic field that yields the VP bands is
oriented in the interface, Voigt and Faraday geometry are inverted, and a
photon propagating perpendicular to the interface couples VP bands with the
same index ($m\rightarrow m^{\prime}=\pm m$) while a photon with a wave vector
in the interface couples adjacent VP bands [$m\rightarrow m^{\prime}=\pm(m\pm
1)$]. As in the LL problem, the selection rules, which are summarized in the
table above, do not depend on the band index. In both cases, VP states and
LLs, it is the circular polarization of the photon determines which of the
adjacent levels or bands are optically coupled.

geometry | VP states | Landau levels
---|---|---
Faraday | $m\rightarrow\pm m$ | $\pm n\rightarrow n\pm 1$
Voigt | $m\rightarrow\pm m\pm 1$ | $n\rightarrow\pm n$
Table 1: Optical selection rules in the Faraday (photon propagation
perpendicular to the interface) and the Voigt geometry (photon propagation in
the interface).

##  III Three-level scheme

Figure 2: Sketch of a three-level scheme for stimulated cyclotron emission.
Ideally, the Fermi level is considered to be situated above the $(m=0,n=0$)
level so that the zero-energy level is filled. (a) Pumping in the Faraday
geometry for $\sqrt{2}\hbar v/l_{B}>\Delta_{\text{VP}}$. The $(m=0,n=1)$ level
is populated via pumping (dashed blue arrow) from the zero-energy $(m=0,n=0)$
level, and emission can take place in the Voigt geometry in the transition
$(m=1,n=0)\rightarrow(m=0,n=0)$ (dashed red arrow). The level $(m=1,n=0)$ is
rapidly populated by the $(n=1,m=0)$ level if the latter is almost resonant
with the former, via a rapid non-radiative decay (dashed black arrow). (b)
Similar process with pumping in the Voigt geometry for $\sqrt{2}\hbar
v/l_{B}<\Delta_{\text{VP}}$. The pumping transition is
$(m=0,n=0)\rightarrow(m=1,n=0)$ (dashed blue arrow) while emission takes place
in the $(m=0,n=1)$ transition (dashed red arrow) that is populated via rapid
non-radiative decay processes (dashed black arrow) from the $(m=1,n=0)$ level,
which is slightly higher in energy. The insets represent the device
geometries, with the direction of propagation of the absorbed (dashed blue
arrow) and emitted (dashed red arrow) photon, for the two configurations,
respectively.

Let us first illustrate schematically the different emission processes in
terms of resonant (optical) pumping within a three-level picture to fix some
basic ideas. In a first step, we consider the situation depicted in Fig. 2(a)
where the LL energy scale $\sqrt{2}\hbar v/l_{B}$ is slightly larger than the
VP gap $\Delta_{\text{VP}}$ given in Eq. (3), i.e. the magnetic length is
slightly smaller than the effective surface width $l_{S}$. We show below in
Sec. V that this situation can be easily achieved e.g. in MBE-grown
$\text{Pb}_{1-x}\text{Sn}_{x}\text{Se}$ crystals. In this case, the $n=1$ LL
of the chiral $m=0$ surface state is slightly above the $n=0$ level of the
upper VP band with an index $m=1$. In Fig. 2(a), we consider optical pumping
in the Faraday geometry, where the light frequency is resonant with the
$(m=0,n=0)\rightarrow(m=0,n=1)$ transition. If the target level is only
slightly above the lowest LL of the $m=1$ VP band, $(m=1,n=0)$, one may expect
rapid non-radiative decay of the excited electrons to the latter level. These
electrons may then decay to the zero-energy level $(m=0,n=0)$ by emitting
light of the frequency $\omega=\sqrt{2}v/l_{S}$ in the Voigt geometry, i.e.
absorbed and emitted photons, even if they may be almost resonant, propagate
in perpendicular directions. While the magnetic field does then not allow one
to control the frequency of the transition, which is determined by the
interface width $\ell$, it allows us to bring the levels $(m=1,n=0)$ and
$(m=0,n=1)$ into close energetic vicinity and thus to increase the transition
rate between the two levels, which is proportional to

$\Gamma\sim\frac{1/\tau}{1/\tau^{2}+2v^{2}(1/l_{B}-1/l_{S})^{2}},$ (7)

if we consider Lorentzian level broadening due to a dephasing time $\tau$ Wang
_et al._ (2015). For a typical value of $\tau\sim 100$ fs, the level
broadening is then on the order of some meV. Notice, however, that the
frequency of the emitted light may to some extent be varied with the help of
an inplane magnetic field, according to Eq. (6).

Similarly, one may use the Voigt goemetry for pumping the transition
$(m=0,n=0)\rightarrow(m=1,n=0)$. If the latter is now slightly above the
$(m=0,n=1)$ level [see Fig. 2(b)], i.e. for smaller magnetic fields with
$\sqrt{2}\hbar v/l_{B}<\Delta_{\text{VP}}$, the $n=1$ LL of the chiral surface
band may be populated by non-radiative decay processes, and one may expect a
population inversion between the $(m=0,n=0)$ and $(m=0,n=1)$ levels, with
cyclotron emission at the fundamental frequency $\omega_{C}=\sqrt{2}v/l_{B}$.
As before, the emitted photon propagates then in a direction perpendicular to
that of the absorbed photon, but Faraday and Voigt geometries are inverted.

##  IV Four-level scheme

Figure 3: Sketch of a four-level scheme for stimulated cyclotron emission.
Ideally, the Fermi level is now situated below the $(m=0,n=0$) level so that
the zero-energy level is empty. We consider pumping in the Voigt geometry for
$\sqrt{2}\hbar v/l_{B}<\Delta_{\text{VP}}$ and a situation, where the $n=0$ LL
of the $m=1$ VP state is in the negative-energy branch. The $(m=1,n=1)$ level
is populated via pumping (dashed blue arrow) from the $(m=0,n=-1)$ level, and
the pumped electrons can decay radiatively to the $(m=0,n=1)$ level by
emitting light also in the Voigt geometry. Electrons in the $(m=0,n=1)$ LL can
then decay to the zero-energy level $(m=0,n=0)$ emitting cyclotron light in
the direction perpendicular to the interface (in the Faraday geometry, dashed
red arrows). Furthermore, this transition is resonant with that
$(m=0,n=0)\rightarrow(m=0,n=-1)$, and a second photon with the cyclotron
frequency can thus be emitted. Auger processes, shown in green now enhance the
depopulation of the zero-energy LL $(m=0,n=0)$ and are therefore helpful for
cyclotron emission.

We now investigate a possible four-level scheme for population inversion, as
shown in Fig. 3. For the sake of the argument, we consider the $n=0$ LLs of
the VP bands now to be situated in the negative-energy branch. As already
mentioned, this can easily be achieved by switching the orientation of the
magnetic field. Let us choose optical pumping by light in the Voigt geometry
that is resonant with the transition $(m=0,n=-1)\rightarrow(m=1,n=1)$. In
contrast to the three-level scheme discussed in the previous section, the
target level is no longer in close vicinity with the level below that is
$(m=0,n=1)$. However, both are optically coupled, and an electron can transit
from $(m=1,n=1)$ to $(m=0,n=1)$ by emitting a photon in the Voigt geometry
again. While this photon is sacrified in the present scheme, its emission
allows for an enhanced population of the $n=1$ LL in the chiral surface band.
This is particularly interesting since the transition
$(m=0,n=1)\rightarrow(m=0,n=0)$ to the central zero-energy level, which we
consider to be non or only sparsely populated, is resonant with the
$(m=0,n=0)\rightarrow(m=0,n=-1)$ transition to the original level served in
the pumping process. Under strong pumping and thus a strong depletion of the
$(m=0,n=-1)$ level, it is therefore possible to emit two photons at the
cyclotron frequency.

It is noteworthy that the above-mentioned resonant cyclotron transitions are
also involved in non-radiative Auger processes. Such Auger processes have been
shown to be detrimental to population inversion in GaAs and graphene. Here,
however, this is not the case. Indeed, one of the two electrons that take part
in the Auger process, where both electrons originally reside in the
$(m=0,n=0)$ LL, is kicked back into the $(m=0,n=1)$ LL thus maintaining the
fertile population inversion. While the electron that transits simultaneously
to the level $(m=0,n=-1)$ is energetically lost, i.e. it does not emit a
photon, the first electron emits another photon at the cyclotron frequency
before it can take part in another Auger process or radiatively transit to
$(m=0,n=-1)$.

##  V Possible realization in $\text{Pb}_{1-x}\text{Sn}_{x}\text{Se}$ crystals

While the above arguments are not restricted to a particular topological
insulator, it is useful to discuss the orders of magnitude of the probably
best controlled system in which VP states occur that is MBE-grown
$\text{Pb}_{1-x}\text{Sn}_{x}\text{Se}$ crystals Martinez (1973); Wojek  _et
al._ (2014); Krizman  _et al._ (2018). As mentioned in the introduction, the
MBE growth allows one to obtain interfaces with a well-controlled interface
width in which the VP states obey to great accuracy the dispersion (2)
Bermejo-Ortiz  _et al._ (2023). Most saliently, the Sn concentration $x$ in
the Pb substitution allows one to trigger the electronic nature of the
material. While, for $x=0$, the system is a trivial band insulator it becomes
a crystalline topological insulator above a critical concentration on the
order of ($x_{c}\simeq 0.12$). Moreover, the Sn concentration determines the
size of the gap, which is on the order of $90$ meV in the trivial insulator at
$x=0$ Wojek  _et al._ (2014); Krizman  _et al._ (2018). The choice $x=0.24$
allows one to obtain the same magnitude for the gap in the topological
insulator ($2\Delta\sim 90$ meV) Bermejo-Ortiz  _et al._ (2023), but even
larger gaps on the order of $2\Delta\sim 200$ meV may be obtained upon
variation of temperature and strain on the crystals Martinez (1973); Krizman
_et al._ (2018). Magneto-optical experiments indicate that the fundamental VP
gap (3) scales as Bermejo-Ortiz  _et al._ (2023)

$\Delta_{\text{VP}}\simeq 45\,\text{meV}/\sqrt{\ell/100\,\text{nm}},$ (8)

and samples with interface widths between $\ell=50$ and 200 nm have been
obtained, while the intrinsic length has been estimated to be
$\lambdabar_{C}\simeq 6$ nm so that the effective surface width varies between
$l_{S}\sim 17$ nm and $l_{S}\sim 35$ nm. In order for the magnetic length to
be on the same order of magnitude as $l_{S}$ – situation considered in the
present paper – one would require magnetic fields in the range $0.5...3$ T
that are easily accessible experimentally.

Finally, the Fermi velocity is roughly half of that in graphene so that
$v/c\sim 1/600$, in terms of the speed of light $c$. If we consider the
fundamental cyclotron resonance associated with the transition
$(m=0,n=1)\rightarrow(m=0,n=0)$, the energy of the transition is thus roughly

$\sqrt{2}\frac{\hbar v}{l_{B}}\simeq 20\,\text{meV}\times\sqrt{B[\text{T}]}.$
(9)

This implies a the transition rate Morimoto  _et al._ (2008)

$\displaystyle\Gamma_{(m=0,n=1)\rightarrow(m=0,n=0)}$ $\displaystyle=$
$\displaystyle 2\alpha\left(\frac{v}{c}\right)^{2}\omega$
$\displaystyle\simeq$ $\displaystyle 2.4\times
10^{6}\,\text{s}^{-1}\times\sqrt{B[\text{T}]},$

in terms of the fine-structure constant $\alpha=1/137$, if we consider dipolar
light coupling. This is roughly a factor of four smaller than in graphene due
to the reduced Fermi velocity $v$. Notice that interaction-induced decay
processes take place at much shorter time scales, typically in the fs range.
In the case of almost resonant levels, as discussed in the previous section
[see e.g. the levels $(m=1,n=0)$ and $(m=0,n=1)$ in Fig. 2(a) and (b)], the
decay rate from the higher to the lower level is on the order of Wang  _et
al._ (2015)

$\Gamma\sim\frac{2\pi}{\hbar}\left(\frac{e^{2}}{\epsilon
l_{B}}\right)^{2}\left(\frac{\tau}{\hbar}\right)\simeq\epsilon^{-1}\times
10^{16}\,\text{s}^{-1}\times B[\text{T}],$ (11)

where $\epsilon$ is the dielectric constant of the host material.

Finally, one should notice that the $(m=1,n=1)$ level, used e.g. in the above
four-level scheme, can also be brought into close energetic vicinity of the
bottom of the bulk conduction band. While the schemes discussed above in terms
of resonant pumping in a specific geometry require themselves a THz source,
one might then alternatively use the conduction band as a target of pulsed or
continuous pumping at higher energies and rely on rapid decay processes
towards the band bottom, which then serves as a reservoir for the $(m=1,n=1)$
level. However, to test this possibility, one would need to rely on a decay
towards this target level at the interface that is more rapid than the bulk
recombination. The quantitative study of these decay processes is beyond the
scope of the present paper.

##  VI Conclusions

In conclusion, I have argued that the particular surface-state spectrum that
is formed in smooth interfaces between a trivial and a topological insulator
is a promising path towards the LL laser. In addition to the chiral surface
state, which may be described in terms of a massless Dirac fermion, VP states
are formed if the gap parameter varies over a width $\ell$ that must be larger
than the intrinsic length scale $\lambdabar_{C}=\hbar v/\Delta$, in terms of
the bulk gap $\Delta$. These surface bands have the form of a massive 2D Dirac
fermion, and each of the bands gives rise to LLs if a magnetic field is
applied perpendicular to the interface. One is thus confronted with families
of LLs the energy of which can to great extent be controlled, by the magnetic
field for the LL separation and by the interface width for the energy
separation between the VP bands. While the latter is given by the sample
growth, it can still be varied in situ with the help of an inplane magnetic
field that effectively reduces the interface width and thus increases the gap
between the VP bands. The magnetic field does not only allow one to change the
cyclotron frequency, at which light is emitted in certain setups, but also to
bring LLs associated with different VP bands into close energetic vicinity.
When the gap between the VP bands is on the same order of magnitude as the
typical LL separation – this situation can be easily achieved experimentally,
e.g. in Pb${}_{(}1-x)$SnxSe crystals – the LL spectra are neither equidistant
nor follow a square-root law so that both Auger and reabsorption processes are
maximally suppressed.

Another highly unusual and, for devices, potentially extremely fertile aspect
of light emission in VP LLs is the direction of propagation of the absorbed
and emitted photons. Indeed, photons with a wave vector perpendicular to the
interface (Faraday geometry) are absorbed and emitted in transitions involving
adjacent LL indices $n$ and $n\pm 1$ but the same VP band index $m$,
regardless of whether the LLs are formed in the positive- or negative energy
branch of the VP bands. On the other hand, photons propagate inside the
interface (Voigt geometry) for transitions $(m,n)\rightarrow(m\pm 1,n)$. This
would allow for a smart design of the Fabry-Pérot cavities such that the
extension in the $z$\- and $x/y$-directions match the photon wavelength of the
respective transitions, especially if pumping and emission are associated with
the two different geometries (Faraday and Voigt). Finally, I have argued that
the often detrimental Auger processes may be less efficient in the proposed
setup so that they do not hinder the population inversion required for a LL
laser, in contrast to most proposals for LL lasers.

###### Acknowledgements.

I would like to thank Gauthier Krizman, Louis-Anne de Vaulcher, and Milan
Orlita for fruitful discussions.

## References

  * Tager and Gladun (1959) A. S. Tager and A. D. Gladun, Sov. Phys. JETP 8, 560 (1959).
  * Wolff (1964) P. A. Wolff, Physics Physique Fizika 1, 147 (1964).
  * Aoki (1986) H. Aoki, Applied Physics Letters 48, 559 (1986).
  * Morimoto  _et al._ (2008) T. Morimoto, Y. Hatsugai, and H. Aoki, Phys. Rev. B 78, 073406 (2008).
  * Gornik  _et al._ (2021) E. Gornik, G. Strasser, and K. Unterrainer, Nature Photonics 15, 875 (2021).
  * Potemski  _et al._ (1991) M. Potemski, R. Stepniewski, J. C. Maan, G. Martinez, P. Wyder, and B. Etienne, Phys. Rev. Lett. 66, 2239 (1991).
  * But  _et al._ (2019) D. B. But, M. Mittendorff, C. Consejo, F. Teppe, N. N. Mikhailov, S. A. Dvoretskii, C. Faugeras, S. Winnerl, M. Helm, W. Knap, M. Potemski, and M. Orlita, Nature Photonics 13, 783 (2019).
  * Lax (1960) B. Lax,  _Cyclotron resonance and impurity levels in semiconductors. In Townes, C.H. (ed.) Proceedings of the International Symposium of Quantum Electronics_ (Columbia University Press, New York, 1960).
  * Gebert  _et al._ (2023) S. Gebert, C. Consejo, S. S. Krishtopenko, S. Ruffenach, M. Szola, J. Torres, C. Bray, B. Jouault, M. Orlita, X. Baudry, P. Ballet, S. V. Morozov, V. I. Gavrilenko, N. N. Mikhailov, S. A. Dvoretskii, and F. Teppe, Nature Photonics 17, 244 (2023).
  * Wendler and Malic (2015) F. Wendler and E. Malic, Scientific Reports 5, 12646 (2015).
  * Plochocka  _et al._ (2009) P. Plochocka, P. Kossacki, A. Golnik, T. Kazimierczuk, C. Berger, W. A. de Heer, and M. Potemski, Phys. Rev. B 80, 245415 (2009).
  * Goerbig (2011) M. O. Goerbig, Rev. Mod. Phys. 83, 1193 (2011).
  * Kazimierczuk  _et al._ (2019) T. Kazimierczuk, A. Bogucki, T. Smoleński, M. Goryca, C. Faugeras, P. Machnikowski, M. Potemski, and P. Kossacki, Phys. Rev. B 100, 075401 (2019).
  * Volkov and Pankratov (1985) B. Volkov and O. Pankratov, JETP Lett 42, 178 (1985).
  * Pankratov  _et al._ (1987) O. Pankratov, S. Pakhomov, and B. Volkov, Solid state communications 61, 93 (1987).
  * Tchoumakov  _et al._ (2017a) S. Tchoumakov, V. Jouffrey, A. Inhofer, E. Bocquillon, B. Plaçais, D. Carpentier, and M. O. Goerbig, Phys. Rev. B 96, 201302 (2017a).
  * Inhofer  _et al._ (2017) A. Inhofer, S. Tchoumakov, B. A. Assaf, G. Fève, J. M. Berroir, V. Jouffrey, D. Carpentier, M. O. Goerbig, B. Plaçais, K. Bendias, D. M. Mahler, E. Bocquillon, R. Schlereth, C. Brüne, H. Buhmann, and L. W. Molenkamp, Phys. Rev. B 96, 195104 (2017).
  * Mahler  _et al._ (2019) D. M. Mahler, J.-B. Mayer, P. Leubner, L. Lunczer, D. Di Sante, G. Sangiovanni, R. Thomale, E. M. Hankiewicz, H. Buhmann, C. Gould, and L. W. Molenkamp, Phys. Rev. X 9, 031034 (2019).
  * Lu and Goerbig (2020) X. Lu and M. O. Goerbig, Phys. Rev. B 102, 155311 (2020).
  * van den Berg  _et al._ (2020a) T. L. van den Berg, M. R. Calvo, and D. Bercioux, Phys. Rev. Res. 2, 013171 (2020a).
  * Adak  _et al._ (2022) V. Adak, S. Chakraborty, K. Roychowdhury, and S. Das, “Volkov-pankratov states on the edge of a quantum spin hall system,” (2022), arXiv:2208.13799 [cond-mat.mes-hall] .
  * Tchoumakov  _et al._ (2017b) S. Tchoumakov, M. Civelli, and M. O. Goerbig, Phys. Rev. B 95, 125306 (2017b).
  * Mukherjee  _et al._ (2019) D. K. Mukherjee, D. Carpentier, and M. O. Goerbig, Physical Review B 100, 195412 (2019).
  * van den Berg  _et al._ (2020b) T. L. van den Berg, A. De Martino, M. R. Calvo, and D. Bercioux, Phys. Rev. Research 2, 023373 (2020b).
  * Alspaugh  _et al._ (2020) D. J. Alspaugh, D. E. Sheehy, M. O. Goerbig, and P. Simon, Phys. Rev. Research 2, 023146 (2020).
  * Bermejo-Ortiz  _et al._ (2023) J. Bermejo-Ortiz, G. Krizman, R. Jakiela, Z. Khosravizadeh, M. Hajlaoui, G. Bauer, G. Springholz, L.-A. de Vaulchier, and Y. Guldner, Phys. Rev. B 107, 075129 (2023).
  * Martinez (1973) G. Martinez, Phys. Rev. B 8, 4686 (1973).
  * Wojek  _et al._ (2014) B. M. Wojek, P. Dziawa, B. J. Kowalski, A. Szczerbakow, A. M. Black-Schaffer, M. H. Berntsen, T. Balasubramanian, T. Story, and O. Tjernberg, Phys. Rev. B 90, 161202 (2014).
  * Krizman  _et al._ (2018) G. Krizman, B. A. Assaf, T. Phuphachong, G. Bauer, G. Springholz, L. A. de Vaulchier, and Y. Guldner, Phys. Rev. B 98, 245202 (2018).
  * Lu and Goerbig (2019) X. Lu and M. O. Goerbig, EPL (Europhysics Letters) 126, 67004 (2019).
  * Wang  _et al._ (2015) Y. Wang, M. Tokman, and A. Belyanin, Phys. Rev. A 91, 033821 (2015).


# Characterizing Speed Performance of
Multi-Agent Reinforcement Learning ††thanks: This work is supported by the
U.S. National Science Foundation (NSF) under grant CNS-2009057 and the Army
Research Lab (ARL) under grant W911NF2220159.

Samuel Wiggins1 , Yuan Meng1 , Rajgopal Kannan2 and Viktor Prasanna1
1Ming Hsieh Department of Electrical and Computer Engineering, University of
Southern California, Los Angeles, USA
2DEVCOM Army Research Lab, Los Angeles, USA
{wigginss, ymeng643<EMAIL_ADDRESS><EMAIL_ADDRESS>
https://orcid.org/0000-0002-0069-8213 https://orcid.org/0000-0001-6468-8623
https://orcid.org/0000-0001-8736-3012 https://orcid.org/0000-0002-1609-8589

###### Abstract

Multi-Agent Reinforcement Learning (MARL) has achieved significant success in
large-scale AI systems and big-data applications such as smart grids,
surveillance, etc. Existing advancements in MARL algorithms focus on improving
the rewards obtained by introducing various mechanisms for inter-agent
cooperation. However, these optimizations are usually compute- and memory-
intensive, thus leading to suboptimal speed performance in end-to-end training
time. In this work, we analyze the speed performance (i.e., latency-bounded
throughput) as the key metric in MARL implementations. Specifically, we first
introduce a taxonomy of MARL algorithms from an acceleration perspective
categorized by (1) training scheme and (2) communication method. Using our
taxonomy, we identify three state-of-the-art MARL algorithms - Multi-Agent
Deep Deterministic Policy Gradient (MADDPG), Target-oriented Multi-agent
Communication and Cooperation (ToM2C), and Networked Multi-agent RL (NeurComm)
- as target benchmark algorithms, and provide a systematic analysis of their
performance bottlenecks on a homogeneous multi-core CPU platform. We justify
the need for MARL latency-bounded throughput to be a key performance metric in
future literature while also addressing opportunities for parallelization and
acceleration.

##  1 INTRODUCTION

Reinforcement Learning (RL) is a crucial technique for model development and
algorithmic innovation in data science. Multi-agent Reinforcement Learning
(MARL), an extension of single-agent RL, has shown advancement in many big-
data application domains such as e-commerce [Choi et al., 2022] and
surveillance in large-scale systems [Ivan and Ivan, 2020]. Compared to single-
agent RL, MARL introduces agent-to-agent interactions using algorithm-specific
communication protocols. MARL also poses challenges in data processing since
each agent generates a large amount of data, which is high-dimensional and
correlated with the data from other agents. The speed performance of
processing data in MARL is critical as MARL training is extremely time-
consuming. In Deep MARL, using Deep Neural Network (DNN) as a policy model,
multiple autonomous agents interact with a shared environment to achieve a
joint goal. The majority of state-of-the-art MARL algorithms utilize DNNs, so
we use MARL and Deep MARL interchangeably in this paper.

Figure 1 shows a high-level diagram of Deep MARL in a real-world scenario. A
typical workflow consists of two major stages: Training in Simulation and
Deployment. In this paper, we focus on Training using a simulation, which is
the critical process before deploying a MARL system into its actual physical
environment. This Training in Simulation process is the most time-consuming
part of MARL system development, often taking days to months to train an
acceptable model [Kaelbling et al., 1996]. There has been extensive work
accelerating this stage for single-agent RL. However, Training in Simulation
for MARL systems brings non-trivial computational challenges compared to
single-agent scenarios due to the requirement of facilitating inter-agent
communications. There is a lack of literature on parallelization and
acceleration in a multi-agent setting stemming from the omission of MARL
system execution time as a performance metric. In addition to focusing on
maximizing cumulative reward through novel algorithmic optimizations, we argue
that optimizing the MARL system execution time through parallelization and
acceleration should be considered.

Figure 1: MARL application workflow

In practice, the efficiency of developing and deploying high-speed MARL
systems is dependent upon several factors such as the dependency requirement
within RL execution loops, the various Computation intensities of MARL
primitives, and the suitability of the memory hierarchy. However, widely-used
homogeneous platforms (i.e., CPUs) cannot satisfy all of the above factors
simultaneously, leading to various time overheads that prevent the MARL system
from achieving theoretical peak throughput.

In this paper, we provide a systematic empirical analysis of the performance
of three state-of-the-art MARL algorithms - MADDPG, ToM2C, and NeurComm on
CPUs. To the best of our knowledge, this is the first work that analyzes MARL
algorithms from a speed performance and acceleration point of view. The main
contributions of the paper are:

  * •

We provide a taxonomy of MARL algorithms from an acceleration point of view,
summarize their parallelization parameters and highlight the computation
characteristics of each category.

  * •

We compare the timing breakdown and performance scalability with respect to
the parallelization parameters of state-of-the-art MARL implementations on a
multi-core CPU platform.

  * •

We provide reasoning into why MARL system execution time should be considered
a key performance metric, and show new acceleration challenges and
opportunities that emerge.

##  2 BACKGROUND

###  2.1 Multi-Agent Reinforcement Learning

We formulate the decision-making problem in MARL as an $n-$agent Markov game
[Shapley, 1953], which can be defined as a tuple
$\left(\mathcal{S},\mathcal{A}^{1},\ldots,\mathcal{A}^{n},R^{1},\ldots,R^{n},\mathcal{T},\gamma\right)$.
In this tuple, ${1...n}$ is the set of agents, $\mathcal{S}$ is the state
space, $\mathcal{A}^{i}$ is the action space of agent $i$,
$R^{i}:\mathcal{S}\times\mathcal{A}\mapsto\mathbb{R}$ is the reward function
of agent $i$,
$\mathcal{T}:\mathcal{S}\times\mathcal{A}\mapsto\Delta(\mathcal{S})$ denotes
the transition probability of each state-action pair to another state,
$\gamma\in[0,1)$ is a discount factor for future rewards. For agent $i$, we
denote its policy as a probability distribution over its action space
$\pi^{i}:\mathcal{S}\rightarrow\Delta\left(\mathcal{A}^{i}\right)$, where
$\pi^{i}\left(a_{t}\mid s_{t}\right)$ is the probability of taking action
$a_{t}$ upon state $s_{t}$ at a certain time step $t$. By denoting the other
agents’ actions as $a_{t}^{-i}=\left\\{a_{t}^{j}\right\\}_{j\neq i}$, we
formulate the other agents’ joint policy as $\pi^{-i}\left(a_{t}^{-i}\mid
s_{t}\right)=\Pi_{j\in\\{-i\\}}\pi^{j}\left(a_{t}^{j}\mid s_{t}\right)$. At
each time step, actions are taken simultaneously. Each agent $i$ aims at
finding its optimal policy to maximize the expected return (cumulative
reward), defined as

$\mathbb{E}_{\left(s_{t},a_{t}^{i},a_{t}^{-i}\right)\sim\mathcal{T},\pi^{i},\pi^{-i}}\left[\sum_{t=1}^{\infty}\gamma^{t}R^{i}\left(s_{t},a_{t}^{i},a_{t}^{-i}\right)\right]$
(1)

From Equation 1, the optimal policy of agent $i$ depends not only on its own
policy but also on the behaviors of other agents. Depending on the assumptions
of how agents communicate, MARL policy optimization can be categorized into
several scenarios which will be further discussed in Section 3.

As shown in Figure 2, each iteration of the Training-in-Simulation process can
be divided into the Sample Generation and Model Update phases.

Figure 2: Training in Simulation

In the Sample Generation phase, the actor of each agent takes the environment
state as input to the DNN policy and inference an action. The joint action of
all the agents is simultaneously executed to transit to a new state. This
process is repeated until a preset maximum trajectory length value is achieved
or a terminal state is reached. At the end of the Sample Generation phase, the
agents produce a batch of experiences
$\\{s_{t},\overrightarrow{a_{t}},r_{t},s_{t+1}\\}$. Then, in the Model Update
phase, the learner of each agent samples (mini-)batches of experiences to
train the DNN policy and propagate the updated policy back to the actor of the
same agent.

###  2.2 Related Work

####  2.2.1 Reinforcement Learning Acceleration

There is a plural of work for parallelization and acceleration of single-agent
RL. On general-purpose platforms (i.e., CPU and GPU), existing work adopts
coarse-grained parallelism by deploying and distributing multiple actors and
learners for the same agent [Zhang et al., 2021, Liang et al., 2018], as well
as fine-grained parallelism within each actor or learner for simulations,
batched inference, and training [Rocki and Suda, 2011]. Additionally,
specialized hardware accelerators targeting specific single-agent RL
algorithms are developed [Cho et al., 2019, Meng et al., 2021, Meng et al.,
2020]. However, not all of these acceleration techniques can be trivially
adapted to MARL. This is because MARL poses different challenges. For example,
the latency bottleneck of MARL may become the overhead from coordination
between agents, which is nonexistent in single-agent RL.

####  2.2.2 MARL Implementation: Summary and Challenges

Compared to single-agent RL, MARL leads to additional problems in terms of
increased state-action complexity, partial observability from each agent, and
coordination requirement in cooperative settings. Many MARL algorithms have
been proposed and implemented to address these problems, where their major
contributions are targeted toward learning to communicate effectively between
RL agents. [Lowe et al., 2017, Jiang et al., 2018] allow agents to perform in
the environment in a decentralized way with copies of a shared neural network
and enable instantaneous communication between all the agents for training the
policies. [Chu et al., 2020, Wang et al., 2021] propose dedicated compute
modules to learn using a shared graph for agents to decide whether and with
whom to communicate. These works focus on increasing the sample efficiency and
communication efficiency to improve the convergence rate to a high joint
reward. They are implemented on multi-core CPUs, and computationally-intensive
workloads such as neural network training are offloaded to a GPU. However,
these implementations are usually highly time-consuming. A unique challenge
that impedes the speed performance of MARL is the overheads from
communication. The compute primitives and memory modules for learning
communication, including message generation (e.g., Recurrent neural networks
[Kim et al., 2021]) and message propagation (e.g., Sparse linear algebra based
on Graph Neural Networks [Jiang et al., 2018]), are tightly coupled with agent
policy training even though they have highly dissimilar compute patterns and
memory operations compared to policy training. This leads to various
computation overheads that lower the effective hardware utilization on CPU,
which in turn impedes the policy execution and training speed. In this paper,
we aim to characterize these overheads and identify acceleration opportunities
in MARL training.

##  3 A TAXONOMY OF MARL ALGORITHMS

We illustrate a structural way of categorizing MARL algorithms based on their
computation characteristics and major acceleration requirements. This is shown
in two dimensions: training scheme (centralized vs. decentralized) and
communication method (pre-defined vs. online learnt). Figure 3 shows example
algorithms in each category.

Figure 3: MARL Taxonomy

###  3.1 Training Scheme

Based on the requirement of MARL applications, each agent can learn in a
decentralized way utilizing respective local experiences, or in a centralized
manner where all the agents share a global pool of model parameters and joint
experiences.

Centralized Training - Decentralized Execution (CTDE): In the Model Update
phase under Centralized Training - Decentralized Execution (CTDE), a
centralized controller or learner is responsible for coordinating the policy
training of all the agents. The centralized controller has access to global
observations and rewards. Once the training is complete, however, each agent
executes its actions independently, based on its own local observations,
without any further communication with the centralized controller.
Communication is required during Model Update and is usually implemented using
a shared memory architecture on modern CPUs and GPUs [Wang et al., 2021, Lowe
et al., 2017].

Decentralized Training: In Decentralized Training, each agent in the system
learns its own policy without direct access to the experiences of other agents
in the system. From the perspective of a single agent, the environment becomes
non-stationary due to the co-adaption of the other agents, such that the
policy learned can have a mismatched expectation about the other agents’
policies. Without a centralized controller, the problems of partial
observability and non-stationarity become more complex. Therefore,
communication that allows agents to exchange information becomes a critical
factor in the coordination of their behaviors and overcoming the partial
observability and non-stationarity problems [Wang et al., 2022].

###  3.2 Communication Method

Communication Policy defines how to make the decisions of ‘whom’ and ‘when’ to
communicate with and enable message transferring across agents. A
communication policy can be either pre-defined or learnt.

Pre-Defined Communication: Pre-defined communication specifies a fixed
communication protocol and message format before the training process begins.
Agents may perform all-to-all communication [Lowe et al., 2017], or capture
the communication paths using a pre-defined graph that is associated with the
given application or environment [Chu et al., 2020, Jiang et al., 2018].

Learnt Communication: Learnt communication offers generalization abilities to
more scenarios and has become increasingly popular due to its flexibility. One
popular method is to adaptively learn the evolving node and edge features of
the multi-agent system represented as a dynamic graph [Wang et al., 2021].

##  4 LATENCY-BOUNDED THROUGHPUT OF MARL ALGORITHMS: AN ANALYSIS

###  4.1 Metrics and Parallel Parameters

####  4.1.1 Latency-Bounded Throughput

Given a particular algorithm and benchmark that assumes a fixed sample
complexity (i.e., the number of iterations needed to reach a certain reward),
the primary metric for measuring MARL training speed is the throughput in
terms of the number of Iterations executed Per Second ($IPS$). As the
iterations in Training-in-Simulation are sequential by nature, it is hard to
overlap consecutive iterations for faster convergence [Cho et al., 2019, Meng
et al., 2021], so minimizing the latency of each iteration is critical for
high $IPS$ throughput. Based on the Sample Generation and Model Update phases
in each iteration (Figure 2), we formulate the throughput $IPS$ as:

$IPS=\frac{1}{T_{\text{iteration
}}}=\frac{1}{\circledast\left(T_{SG},T_{\text{MU}}\right)}\text{, }$ (2)

where $T_{SG}$ and $T_{\text{MU}}$ are the execution times of Sample
Generation and Model Update phases, respectively. The operator
$\circledast(x,y)$ can be $x+y$ or $max(x,y)$ based on whether the algorithm
is on-policy (where Model Update is strictly dependent on Sample Generation in
each iteration) or off-policy (where Sample Generation and Model Update phases
can be overlapped).

In the following subsections, we provide a detailed analysis of the main
computation primitive latency and throughput using representative algorithms
in different categories of MARL. We will show the time breakdown and
performance scalability with respect to a few key parallel parameters.

####  4.1.2 Key Parallel Parameters

A MARL algorithm is defined using a plural of hyper-parameters, where a subset
of these hyper-parameters affects the parallelism, data processing speed, and
latency-bounded throughput of the system. We define the following terms
relevant to the key parallel parameters that are common in all categories of
MARL:

  * •

Agent(s): Agents execute by interacting with the environment and improving
their policies.

  * •

Actor(s): An actor is a process used by an agent to perform the Sample
Generation phase. An actor infers on the agent’s policy to perform an action
in a simulation environment. Each agent can spawn multiple actors implemented
on multiple rollout thread(s) defined below.

  * •

Rollout thread(s): A rollout thread is a CPU thread used to execute the actor
process. A rollout thread has a local copy of the environment and executes
independently of the other actor/rollout thread(s). Typically, each agent
deploys multiple actors, and each actor is mapped to a rollout thread.

  * •

Learner(s): A learner is the process used by an agent to train its policy in
the Model Update phase. In CTDE, the learner is typically shared by all the
agents, so the number of learners in the MARL system is 1. In Decentralized
training, the number of learners is equal to the number of agents.

###  4.2 Centralized Training with Knowledge Concatenation

Multi-Agent Deep Deterministic Policy Gradient (MADDPG) falls under the
Centralized Training Decentralized Execution category with pre-defined all-to-
all communication using knowledge concatenation [Lowe et al., 2017]. In
MADDPG, each agent is composed of four DNN models: (1) The Policy network
(i.e., decentralized actor) executes on the environment using only local
information independent of other agents. (2) The Value network (i.e.,
centralized critic) helps train the actor’s policy. Although it is considered
centralized, each agent still trains its own value network. The training is
centralized in the sense that the inputs to each value network depend on the
action and observations (i.e., transition information) of all agents. (3,4)
The Target Policy and Target Value network are used for training stability.

In the MADDPG implementation, multiple rollout threads are deployed to sample
the environment using the policies from the decentralized actors, where the
transition information is stored in a global replay buffer. Multiple training
threads can then use this information to train the various DNN networks
described above.

During Sample Generation, each agent samples an action using its policy and
executes it in the environment. The latency breakdown among Policy Inference,
Communication, and Environmental Steps in one iteration is displayed in Figure
4. During Model Update, training occurs in an actor-critic fashion similar to
the single-agent variant of MADDPG: Deep Deterministic Policy Gradient (DDPG)
[Lillicrap et al., 2015]. Overall, we observe that communication in the sample
generation and model update phase is not the dominating factor in terms of
latency during Training in Simulation, which is attributed to the
straightforward insertion and concatenation of transition information.

Figure 4: MADDPG Training in Simulation Breakdown (8 rollout threads, 1
training thread)

We observe that the actor/critic policy and value updates are the dominating
factor in terms of latency when varying the number of rollout threads as seen
in Figure 5. In the Sample Generation phase, the fixed amount of data
generation takes a shorter amount of time since multiple parallel actors
sample copies of the environment simultaneously while adding to the replay
buffer. During the Model Update phase, latency for each gradient update step
remains unchanged with respect to the number of rollout threads. This is
expected given that training for each agent happens using a single thread
sequentially regardless of the number of rollout threads. Increasing the
number of rollout threads does increase the number of gradient steps needed to
complete the model update phase in MADDPG. These additional steps cause the
overall system throughput ($IPS$) to remain consistent.

Figure 5: MADDPG Training Throughput Scalability varying rollout threads

###  4.3 Centralized Training with Learnt Communication

ToM2C (Target-oriented Multi-agent Communication and Cooperation) is a recent
work under the CTDE training scheme [Wang et al., 2021]. In ToM2C, each agent
learns its actions based on its local observation space along with inference
of the mental states of others and learns the communication paths for
enhancing cooperation. Specifically, it comprises four functional DNN models:
(1) The Observation Encoder is a fully-connected DNN layer that encodes the
agent’s local observation into a single feature using a weighted sum
mechanism. (2) The Theory of Mind network (ToM Net) is a 2-layer Multi-Layer
Perceptron on a Gate Recurrent Unit that estimates the joint intention of all
the other agents. (3) The Message Sender is a Graph Neural Network (GNN) that
uses the ToM Net output to decide the communication graph structure. (4) The
Decision Maker (i.e., policy model) learns the actions once the agent receives
all the messages based the communication graph returned by the Message Sender
of all other agents.

In the ToM2C implementation, multiple actors on different rollout threads are
used to generate datasets and store them in a shared buffer; a single training
thread then uses the data from the shared buffer to update the policy, ToM
Net, and GNN described above.

During Sample Generation, each actor sequentially performs inference through
the four DNN models. The time breakdown among Policy Inference, Communication,
and Environmental Interaction in one iteration is shown in Figure 6. During
Model Update, the training is performed in an actor-critic manner [Mnih et
al., 2016], where a centralized critic (i.e., Value network) shared by all the
agents obtain global feature of the states to compute their values and
facilitate the training of the policy network (i.e., the Decision Maker).

Overall, different from MADDPG which also follows the CTDE scheme, in ToM2C we
observe a non-trivial overhead from learning of communication in the training
pipeline. This is due to the added computation complexity from inferring other
agents’ intentions and deciding the communication paths using a graph
representation of the global states of all the agents. On the other hand, the
overheads from communication during agents execution (Sample Generation) are
smaller compared to that in the Model Update, since the decentralized
execution only requires each actor to perform one-pass inference on its own
models.

Figure 6: ToM2C Training in Simulation Breakdown (6 rollout threads, 1
training thread) Figure 7: ToM2C Training Throughput Scalability varying the
number of rollout threads Figure 8: ToM2C Training Throughput Scalability
varying the number of agents

In Sample Generation, adding rollout threads increases the data generation
throughput (number of samples generated in unit time) without increasing the
total latency needed for each actor to collect data, as shown in Figure 7. The
actors and the learner use separate threads and only synchronize through a
shared buffer; the Sample Generation and Model Update phases can run
concurrently. Therefore, the latency of actors is completely hidden by the
training process, such that increasing the number of rollout threads does not
lead to higher system throughput ($IPS$).

Figure 8 shows the execution time and $IPS$ scalability with increasing the
number of agents cooperating in the same environment, with a fixed number of
rollout threads serving each agent. We observe a faster rate of latency
scaling (at approximately $8\times$ increase in gradient update time as the
number of agents doubles). This factor is contributed by both the increased
state size (processed state size doubles as the number of agents doubles) and
the increased communication (worst-time communication complexity is the square
of the number of agents). This means both algorithmic optimization and
hardware acceleration for communication reduction are needed to increase the
scalability of communication-learning MARL systems to a large number of
agents.

###  4.4 Decentralized Training with Pre-Defined Graph Communication

Figure 9: NeurComm Training in Simulation Breakdown (8 agents) Figure 10:
NeurComm Training Throughput Scalability varying the number of agents Table 1:
Comparisons of target MARL algorithms.

Algorithm | MADDPG | ToM2C | NeurComm
---|---|---|---
Training Scheme | Centralized | Centralized | Decentralized
Communication Method |  | Pre-Defined,
---
All-to-all
Learnt |  | Pre-Defined,
---
Graph-based
$T_{Comm}\%$: Execution | 5.89% | 25.8% | 72.2%
$T_{Comm}\%$: Training | 21.1% | 47.0% | 42.4%
Parallel Rollout Support | Yes | Yes | No
Parallel Training Support | Yes | No | No
SG/MU Overlapping | Yes | Yes | No
Cumulative Rewards |  | CN1 : -2.75+-0.61
---
ATSC1 : Not Supported
| CN: -0.79+-0.39
---
ATSC: Not Supported
| CN: Not Supported
---
ATSC: -136.1

  * 1

Test Benchmarks: CN denotes Cooperative Navigation, ATSC denotes Adaptive
Traffic Signal Control.

  * 2

$T_{Comm}\%$ stands for Communication Time Percentage.

NeurComm-enabled MARL follows the Decentralized Training with Pre-Defined
Communication paradigm aimed at targetting networked Multi-agent Reinforcement
Learning (NMARL) scenarios, including traffic light and wireless networks
systems [Chu et al., 2020]. This is formulated using a spatiotemporal Markov
Decision Process, where each agent’s actions are learned based on the state
and policy of neighboring agents. The main contributions of this paper is a
differentiable neural communication protocol called NeurComm. NeurComm
introduces a message primitive called an agent’s “belief,” which is propagated
to the neighboring agents and optimizes their performance iteratively. The
additional communication overhead lies in the belief propagation function.
Each decentralized agent uses the belief, local states, and action
probabilities of neighboring agents in order to make a decision.

In this Decentralized NMARL using NeurComm communication implementation, A2C
agents are used, which follow an on-policy actor-critic method [Mnih et al.,
2016]. Figure 9 shows the Training in Simulation breakdown of NeurComm during
one iteration of Sample Generation and Model Update. During Sample Generation,
messages are received and sent while control is performed. During Model
Update, each agent’s belief is updated along with the gradients of the actor,
critic, and neural communication network. Overall, we observe that
communication takes up a large percentage of the total execution time during
an iteration of Sample Generation and Model Update.

NeurComm lacks a parallel setup and instead utilizes a single thread for both
the Sample Generation and Model Update phases. From Figure 10, we observe a
$2\times$ increase in all latency measurements when the number of agents
doubles. When increasing the number of agents, various data structures,
including belief, agent states, number of policies, etc., also increase in
size proportionally. $IPS$ decreases proportionally with the number of agents,
which is expected given the single-threaded nature of this implementation and
the increasing communication and state size. While the addition of the belief
primitive leads to optimized control performance, each agent needs to compute
its own belief using the belief of other neighboring agents, leading to a
large amount of communication and computation overhead.

###  4.5 Comparison of MARL Algorithms

Based on the analyses in Sections 4.2, 4.3 and 4.4, we summarize some
tradeoffs between these different categories of algorithms and their parallel
implementations in Table 1.

Comparison in terms of Communication Methods: In CTDE (columns MADDPG and
ToM2C of Table 1, row “Cumulative Rewards”), it has been verified that
learning to communicate through predicting other agents’ future rollouts
outperforms simple all-to-all concatenation of current experiences. However,
learnt communication also leads to higher communication cost in both
decentralized execution (Sample generation) and centralized training (Model
Update), as shown in Table 1, rows “Communication Time Percentage,” where the
time spent on generating the message and data transfer for communication in
ToM2C accounts for 3 times larger percentage than that in MADDPG. This calls
for fine-grained acceleration of training to alleviate the training bottleneck
in MARL frameworks supporting CTDE with learnt communication.

Comparison in terms of Training Schemes: Note that Centralized training vs.
Decentralized training implies different assumptions in terms of the
accessibility of global information on the joint policy. Decentralized
training can better support applications requiring autonomous acting using
local information (e.g., traffic network, stock market), but each agent needs
to cope with a non-stationary environment due to the instability of other
changing and adapting agents. In contrast, centralized training relies on a
controlling authority of all agents’ policies and faces a stationary
environment. However, the large (exponentially increasing) joint policy space
could be too difficult to search. We observe that in order to learn a stable
policy, decentralized training relies more on communication due to the lack of
global information. For instance, in NeurComm (Table 1, row “Communication
Time Percentage: Agent Execution”), communication accounts for as much as 70%
of the total training time. The additional overhead involves DNN layer
computation for the encoding and extraction of an agent’s belief. This
additional communication overhead justifies the need for fine-grain
acceleration of the various added kernels.

##  5 DISCUSSION & CONCLUSION

In this work, we provided an extensive analysis of various MARL algorithms
based on a taxonomy. This is also the first work in the field of MARL that
characterizes key algorithms from a parallelization and acceleration
perspective. We proposed the need for latency-bounded throughput to be
considered a key optimization metric in future literature. Based on our
observation, the need for communication brings a non-trivial overhead that
needs fine-grained optimization and acceleration depending on the category of
the algorithm described in our taxonomy. There is a plethora of future work
that can be conducted on MARL in terms of acceleration:

  * •

Specialized accelerator design for reducing communication overheads:
Specialized acceleration platforms such as Field Programmable Gate Arrays
(FPGA) offer pipeline parallelism along with large distributed on-chip memory
that features single-cycle data access. To take full advantage of low-latency
memory, specialized data layout and partition for the communicated message
pool need to be exploited.

  * •

Fine-grained Task Mapping using heterogeneous platforms: We have seen the
success of bringing single-agent RL algorithm to heterogeneous platforms
composed of CPU, GPU and FPGA [Meng et al., 2021, Zhang et al., 2023] and plan
to extend this to MARL.

## REFERENCES

  * Cho et al., 2019 Cho, H., Oh, P., Park, J., Jung, W., and Lee, J. (2019).  Fa3c: Fpga-accelerated deep reinforcement learning.  In Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems, pages 499–513.
  * Choi et al., 2022 Choi, H.-B., Kim, J.-B., Han, Y.-H., Oh, S.-W., and Kim, K. (2022).  Marl-based cooperative multi-agv control in warehouse systems.  IEEE Access, 10:100478–100488.
  * Chu et al., 2020 Chu, T., Chinchali, S., and Katti, S. (2020).  Multi-agent reinforcement learning for networked system control.  arXiv preprint arXiv:2004.01339.
  * Ivan and Ivan, 2020 Ivan, M.-C. and Ivan, G. (2020).  Methods of exercising the surveillance of criminal prosecution.  Rev. Stiinte Juridice, page 160.
  * Jiang et al., 2018 Jiang, J., Dun, C., Huang, T., and Lu, Z. (2018).  Graph convolutional reinforcement learning.  arXiv preprint arXiv:1810.09202.
  * Kaelbling et al., 1996 Kaelbling, L. P., Littman, M. L., and Moore, A. W. (1996).  Reinforcement learning: A survey.  CoRR, cs.AI/9605103.
  * Kim et al., 2021 Kim, W., Park, J., and Sung, Y. (2021).  Communication in multi-agent reinforcement learning: Intention sharing.  In International Conference on Learning Representations.
  * Liang et al., 2018 Liang, E., Liaw, R., Nishihara, R., Moritz, P., Fox, R., Goldberg, K., Gonzalez, J., Jordan, M., and Stoica, I. (2018).  Rllib: Abstractions for distributed reinforcement learning.  In International Conference on Machine Learning, pages 3053–3062. PMLR.
  * Lillicrap et al., 2015 Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (2015).  Continuous control with deep reinforcement learning.
  * Lowe et al., 2017 Lowe, R., Wu, Y. I., Tamar, A., Harb, J., Pieter Abbeel, O., and Mordatch, I. (2017).  Multi-agent actor-critic for mixed cooperative-competitive environments.  Advances in neural information processing systems, 30.
  * Meng et al., 2021 Meng, Y., Kuppannagari, S., Kannan, R., and Prasanna, V. (2021).  Ppoaccel: A high-throughput acceleration framework for proximal policy optimization.  IEEE Transactions on Parallel and Distributed Systems, 33(9):2066–2078.
  * Meng et al., 2020 Meng, Y., Yang, Y., Kuppannagari, S., Kannan, R., and Prasanna, V. (2020).  How to efficiently train your ai agent? characterizing and evaluating deep reinforcement learning on heterogeneous platforms.  In 2020 IEEE High Performance Extreme Computing Conference (HPEC), pages 1–7. IEEE.
  * Mnih et al., 2016 Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., and Kavukcuoglu, K. (2016).  Asynchronous methods for deep reinforcement learning.  In International conference on machine learning, pages 1928–1937. PMLR.
  * Rocki and Suda, 2011 Rocki, K. and Suda, R. (2011).  Parallel monte carlo tree search on gpu.  In Eleventh Scandinavian Conference on Artificial Intelligence, pages 80–89. IOS Press.
  * Shapley, 1953 Shapley, L. S. (1953).  Stochastic games.  Proceedings of the national academy of sciences, 39(10):1095–1100.
  * Wang et al., 2022 Wang, X., Zhang, Z., and Zhang, W. (2022).  Model-based multi-agent reinforcement learning: Recent progress and prospects.  arXiv preprint arXiv:2203.10603.
  * Wang et al., 2021 Wang, Y., Zhong, F., Xu, J., and Wang, Y. (2021).  Tom2c: Target-oriented multi-agent communication and cooperation with theory of mind.  arXiv preprint arXiv:2111.09189.
  * Zhang et al., 2021 Zhang, C., Kuppannagari, S. R., and Prasanna, V. K. (2021).  Parallel actors and learners: A framework for generating scalable rl implementations.  In 2021 IEEE 28th International Conference on High Performance Computing, Data, and Analytics (HiPC), pages 1–10. IEEE.
  * Zhang et al., 2023 Zhang, C., Meng, Y., and Prasanna, V. (2023).  A framework for mapping drl algorithms with prioritized replay buffer onto heterogeneous platforms.  IEEE Transactions on Parallel and Distributed Systems.