#  Spectra of a Gapped Quantum Spin Liquid with a Strong Chiral Excitation on
the Triangular Lattice

Ta Tang  Department of Applied Physics, Stanford University, California 94305,
USA.  Brian Moritz  Stanford Institute for Materials and Energy Sciences, SLAC
National Accelerator Laboratory, 2575 Sand Hill Road, Menlo Park, California
94025, USA.  Thomas P. Devereaux  Stanford Institute for Materials and Energy
Sciences, SLAC National Accelerator Laboratory, 2575 Sand Hill Road, Menlo
Park, California 94025, USA.  Department of Materials Science and Engineering,
Stanford University, Stanford CA 94305.  Geballe Laboratory for Advanced
Materials, Stanford University, Stanford, CA 94305, USA.

###### Abstract

While a quantum spin liquid (QSL) phase has been identified in the
$J_{1}$-$J_{2}$ Heisenberg model on a triangular lattice via numerical
calculations, debate persists about whether or not such a QSL is gapped or
gapless, with contradictory conclusions from different techniques. Moreover,
information about excitations and dynamics is crucial for the experimental
detection of such a phase. In this work, we use exact diagonalization to
characterize signatures of a QSL phase on the triangular lattice through the
dynamical spin structure factor $\mathcal{S}(q,\omega)$ and Raman
susceptibility $\mathcal{\chi}(\omega)$. We find that spectra for the QSL
phase show distinct features compared to those of neighboring phases; and both
the Raman spectra and spin structure factor show gapped behaviour in the QSL
phase. Interestingly, there is a prominent excitation mode in the Raman
$A_{2}$ channel, indicating a strong subleading tendency toward a chiral spin
liquid phase.

###### pacs:

Valid PACS appear here

Quantum spin liquids (QSLs), characterized by the lack of magnetic order
approaching zero temperature, were first considered by Anderson [1] as an
alternative ground state to the antiferromagnetic Néel phase. Later, QSLs were
suggested as a possible route to high temperature superconductivity [2, 3],
where preexisting singlet pairs may become superconducting upon doping. In
addition to possible links with superconductivity, QSLs are massively
entangled and can support exotic excitations, which can be utilized for
topological quantum computation [4].

Among various lattices that have been suggested to host QSL phases, the
triangular lattice plays an important role, as it was originally proposed by
Anderson and many QSL candidates have this underlying lattice structure [5,
6]. Although the Heisenberg model with a nearest neighbor interaction $J_{1}$
on the triangular lattice has been found to have long range antiferromagnetic
order [7, 8, 9, 10], adding longer-range interactions may increase frustration
and help realize a QSL state. Numerical studies have reached a consensus that
there is indeed a QSL phase on the triangular lattice with a next-nearest
neighbor interaction $0.08\lesssim J_{2}/J_{1}\lesssim 0.16$ [11, 12, 13, 14,
15, 16]. However, the nature of this QSL phase remains under active
investigation, as some density matrix renormalization group (DMRG)
calculations suggest that the QSL phase on the triangular lattice is a gapped
spin liquid [11, 12, 13], while variational quantum Monte Carlo (VMC)
simulations [14, 15] and a DMRG simulation with flux insertion [17] suggest
that the phase is a $U(1)$ gapless spin liquid .

While theoretical debates persist, tremendous progress has been made in the
experimental identification of QSLs. Promising QSL candidates include
triangular lattice systems such as $\kappa\text{-}\mathrm{(ET)_{2}X}$ [5, 6]
and $\mathrm{EtMe_{3}SB[Pd(dmit)_{2}]_{2}}$ [18, 19]. The lack of magnetic
order down to the lowest accessible temperatures in these materials is a
strong indication for the presence of a QSL ground state. However, critical
questions remain about how to identify/distinguish experimentally between QSL
phases, and how to link experimental measurements to theoretical models. While
numerical methods like DMRG and VMC are powerful tools for studying ground
state properties for large system simulations, it can be more difficult to
study the dynamical properties of the system, and therefore difficult to
provide results that can be compared directly to certain experimental
measurements, such as the dynamical spin structure factor, as measured in
neutron scattering, or the Raman spectra [20, 21, 22].

Here we study the $J_{1}$-$J_{2}$ Heisenberg model using exact diagonalization
(ED) [23], which exactly captures low lying eigenstates and can provide
information about excitations and dynamics, albeit for small system sizes.
Specifically, we obtain the dynamical spin structure factor
$\mathcal{S}(q,\omega)$, which shows distinct features while tuning the ratio
$J_{2}/J_{1}$, indicating the presence of distinct phases. We also extract the
value of the spin excitation gap from finite-size scaling of
$\mathcal{S}(q,\omega)$, which extrapolates to a finite value. In addition to
$\mathcal{S}(q,\omega)$, the Raman spectrum also serves as an important
experimental probe for QSLs [20]. Here, we derive the lowest order Raman
scattering operators for different symmetry channels and calculate the Raman
susceptibility $\mathcal{\chi}(\omega)$ to characterize different phases. The
QSL phase possesses distinct spectral features when compared to the nearby
phases, tuning through the $J_{2}/J_{1}$ phase diagram.

The $J_{1}$-$J_{2}$ Heisenberg Hamiltonian is defined as

$H=\sum_{\left<ij\right>}J_{1}\bm{S}_{i}\cdot\bm{S}_{j}+\sum_{\left<\left<ij\right>\right>}J_{2}\bm{S}_{i}\cdot\bm{S}_{j},$
(1)

where $\bm{S}_{i}=(S_{i}^{x},S_{i}^{y},S_{i}^{z})$ denotes the spin vector on
site i; $J_{1}$ is the nearest neighbor spin-exchange interaction and is set
to 1; $J_{2}$ is the next nearest neighbor spin-exchange interaction;
$\left<ij\right>$ denotes nearest-neighbor sites and
$\left<\left<ij\right>\right>$ denotes next-nearest-neighbor sites.

Previous numerical studies [11, 12, 14, 15, 16] have established that for
small $J_{2}/J_{1}$, the system is in a $120\degree$ antiferromagnetically
ordered state (hereafter $120\degree$ AF). Increasing $J_{2}~{}(\gtrsim
0.08J_{1})$, the system transitions into a QSL phase characterized by
exponentially vanishing spin-spin correlations. For larger $J_{2}~{}(\gtrsim
0.16J_{1})$, the system is in a two-sublattice striped phase. In Fig. 1, we
show the spin-spin correlations and the static spin structure factor obtained
using ED on a 36-site cluster for three different values of $J_{2}$,
representing the three phases. The spin-spin correlations are defined as

$\mathcal{S}_{r}=\frac{1}{N}\sum_{i=0}^{N-1}\left<\bm{S}_{r_{i}}\cdot\bm{S}_{r_{i}+r}\right>,$
(2)

and the static spin structure factor is obtained by Fourier transforming
$\mathcal{S}_{r}$

$\mathcal{S}_{q}=\frac{1}{N}\sum_{i=0}^{N-1}\mathcal{S}_{r_{i}}\mathrm{exp}(i\bm{q}\cdot\bm{r}_{i}),$
(3)

where N is the number of sites. For $J_{2}/J_{1}=0$, there are prominent peaks
at the Brillouin zone (BZ) corners ($K$ points), indicating the $120\degree$
AF order; large $J_{2}/J_{1}$ yields $\mathcal{S}_{q}$ peaks at the BZ edges
($M$ points), characteristic of stripe order. In the intermediate region
$0.08\lesssim J_{2}/J_{1}\lesssim 0.16$, a ring of peaks around the BZ
boundary form, where the intensity at the $K$ and $M$ points is comparable,
yet largely suppressed in comparison to the ordered states. The rapid decay of
the real-space spin-spin correlations in the intermediate phase also serves as
an indication of the QSL phase in this parameter regime. These results are
consistent with previous ED [8, 16] and DMRG [11, 12, 13] studies.

Figure 1:  Spin-spin correlation plotted in real space $\mathcal{S}_{r}$
(upper panel) and momentum space $\mathcal{S}_{q}$ (lower panel). The left
column is for $J_{2}/J_{1}=0.0$, featuring peaks at BZ corners ($K$ points)
for the $120\degree$ AF order. The right column with $J_{2}/J_{1}=0.24$ shows
stripe order with peaks at the centers of the BZ edges ($M$ points); in the
spin liquid phase parameter regime ($J_{2}/J_{1}=0.15$, middle column), spin-
spin correlations decay very fast and the peaks at $K$ and $M$ points are
largely suppressed and have similar intensity.

Next, signatures of a gap in the excited state spectra of the various phases
are investigated via the dynamical spin structure factor and the polarization-
dependent Raman susceptibilities.

Figure 2:  Dynamical spin structure factor at representative momentum points
in three different phases: (a) $J_{2}/J_{1}=0$ in the $120^{o}$ AF phase,
(b)(c) $J_{2}/J_{1}=0.12,0.15$ in the spin liquid phase, and (d)
$J_{2}/J_{1}=0.24$ in the stripe phase. The inset shows the path in momentum
space starting at $\Gamma$ point for drawing $\mathcal{S}^{zz}(q,\omega)$.
$\Gamma$ is the BZ center, $M$ is the BZ edge center, and $K$ is the BZ
corner. Momentum points near the Brillouin zone boundary are colored in red.

The dynamical spin structure factor is defined as

$\mathcal{S}^{zz}(q,\omega)=-\frac{1}{\pi}\mathrm{Im}\left<G\right|S^{z}_{-q}\frac{1}{\omega+E_{0}-H+i\eta}S^{z}_{q}\left|G\right>,$
(4)

where
$S^{z}_{q}\\!=\\!\frac{1}{\sqrt{N}}\sum_{i=0}^{N-1}S^{z}_{r_{i}}\mathrm{exp}(i\bm{q}\cdot\bm{r}_{i})$,
$\left|G\right>$ is the ground state and $E_{0}$ is the ground state energy.

$S^{zz}(q,\omega)$ for different values of $J_{2}$ representing the three
different phases are shown Fig. 2. $S^{zz}(q,\omega)$ for different momentum
points in each plot from bottom to top correspond to those labeled by the path
starting from the $\Gamma$ point in the inset of Fig. 2(a). All other momentum
points in the BZ are related to these points through rotation or reflection
symmetries. In the $120\degree$ AF phase, the lowest excitation is at the
ordering wave vector $K$. Increasing $J_{2}$, the excitation gap at $K$
increases while the excitation gap at $M$ and other momentum points near the
BZ boundary shrink. In the spin liquid phase, the excitation gaps become
comparable at $K,M$, and other points near the BZ boundary [colored in red in
the inset of Fig. 2(a)]. The gap at $M$ becomes the smallest once the system
enters the stripe phase when further increasing $J_{2}$.

These three phases exhibit distinct spin excitation spectral features on
finite clusters using ED. We note that our result qualitatively agrees well
with that obtained by a dynamical variational Monte Carlo approach [24]. In
the thermodynamic limit, the two ordered phases are expected to become
gapless, having gapless excitations emanating from the ordering wave vectors
in $S^{zz}(q,\omega)$. However, whether the QSL phase is gapped or gapless
remains unclear.

Spin excitation gaps extracted from $S^{zz}(q,\omega)$ may be compared for
different simulation cluster sizes to extrapolate to a thermodynamic limit.
Unfortunately, only two clusters (12-site and 36-site) are available with
$D_{6}$ point group symmetry and $S_{z}^{tot}=0$ for reasonable computational
cost. The next larger system size would be 48-sites, which is near the
computational limit for ED. Therefore, it is hard to truly extrapolate to the
2-dimensional thermodynamic limit using ED.

In order to perform finite-size scaling, we instead use a set of clusters with
size $4\times L_{x}$, where $L_{x}=4,5,6,7,8$. This set of clusters all have
the $M$ point but lack the $K$ point. Therefore, they are not appropriate to
capture the $120\degree$ AF phase and the spin gap extracted on these clusters
for the $120\degree$ AF phase will have severe finite-size effects.
Nevertheless, we can focus our attention on the behavior of the gap extracted
in either the QSL phase or the stripe phase, depending on the Heisenberg
exchange parameters. We note that although the other spatial direction only
has 4 sites, finite size effects may be not severe as the spin-spin
correlations decay quickly in the QSL phase, as observed in DMRG results with
more sites in the other spatial direction [11, 12].

In Fig. 3 (lower panel), the spin gap denoted as $\Delta E$ is extracted for
$4\times L_{x}$ clusters with $J_{2}/J_{1}=0.12$, deep in the spin liquid
phase, and $J_{2}/J_{1}=0.24$, well into the striped phase. Fitting $\Delta
E\approx c\cdot k_{x}+\delta$ (a gapless linear spin wave), we see that the
gap remains finite $\Delta E\approx 0.1J_{1}$ in the QSL phase, while the gap
tends to zero for the striped phase. Fitting with quadratic dispersion
($\Delta E\approx c\cdot k_{x}^{2}+\delta$) gives an even larger gap $\Delta
E\approx 0.28J_{1}$ in the QSL phase as $k_{x}\rightarrow 0$. The extrapolated
spin gap using the quadratic fit is consistent with the result from DMRG [11,
12] on cylindrical clusters, which favour a gapped QSL.

Figure 3:  (a) and (b) are the spin-spin correlations for the $4\times 8$
cluster in real space. (c) and (d) are the finite size scaling of the spin
excitation gaps extracted from $S(q,\omega)$ on $4\times L_{x}$ clusters. In
the stripe phase (right column), the spin-spin correlation shows stripe order
along the y direction, and the spin gap approaches 0 when $L_{x}\to\infty$
using a linear fit $\Delta E\sim 1/L_{x}$. In the spin liquid phase parameter
regime (left column), the spin-spin correlation decays very fast along the y
direction, and the system remains gapped with $\Delta E\approx 0.28J_{1}$ when
$L_{x}\to\infty$ using a quadratic fit $\Delta E\sim 1/L_{x}^{2}$. It is also
gapped with $\Delta E\approx 0.1J_{1}$ if using a linear fit.

The dynamical spin structure factor shows distinct features for the QSL phase
and its neighboring phases. However, neutron scattering experiments usually
require large samples or accumulating measurements from many samples to obtain
a sizable signal [20]. This may hinder use of the dynamical spin structure
factor to diagnosis a QSL experimentally. In contrast, inelastic light
scattering can often yield larger scattering intensity from smaller samples
and can be much easier to obtain experimentally. In the next section, we
explore Raman scattering for the QSL and its neighboring phases.

In addition to neutron scattering, Raman spectroscopy serves as an important
experimental probe [25, 26, 27, 28, 29, 30, 20]. While $\mathcal{S}(q,\omega)$
probes excitations induced by flipping one spin ($\Delta S_{z}=\pm 1$), Raman
scattering processes involve even numbers of spin flips ($\Delta S_{z}=0,\,\pm
2$). Furthermore, the scattering geometry (in-coming and out-going
polarization discrimination) may be used to probe different symmetry channels.
Thus Raman spectroscopy is capable of providing extra information about
excitations and the interplay between lattice symmetry and underlying order.
First, we derive the lowest order Raman scattering operators in different
symmetry channels. Using these, we compute and compare the Raman
susceptibility on the 36-site cluster for the distinct phases outlined in the
previous analysis.

We can obtain the effective scattering operator in the spin basis by first
considering Raman scattering (a photon-in/photon-out process) using the
underlying light-matter interaction Hubbard Hamiltonian, and then project out
double occupancies [25, 26].

Since the Raman scattering operator contains dot products with the incoming
and outgoing light polarizations(see supplementary material), it can be
written in the general form

$\hat{M}=\sum_{\alpha\beta}M_{\alpha\beta}e_{f}^{\alpha}e_{i}^{\beta},$ (5)

where $\bm{e}_{i}$ and $\bm{e}_{j}$ are the incoming and outgoing light
polarization, and $\alpha$, $\beta$ denote spatial basis.

We focus on the 36-site cluster with $D_{6}$ point group symmetry. The Raman
scattering operator can be decomposed according to the irreducible
representations of the $D_{6}$ symmetry group

$\displaystyle\hat{M}$ $\displaystyle=$
$\displaystyle\hat{O}_{A_{1}}(e_{f}^{x}e_{i}^{x}+e_{f}^{y}e_{i}^{y})+\hat{O}_{A_{2}}(e_{f}^{x}e_{i}^{y}-e_{f}^{y}e_{i}^{x})$
(6) $\displaystyle+$
$\displaystyle\hat{O}_{E_{2}^{(1)}}(e_{f}^{x}e_{i}^{x}-e_{f}^{y}e_{i}^{y})+\hat{O}_{E_{2}^{(2)}}(e_{f}^{x}e_{i}^{y}+e_{f}^{y}e_{i}^{x}),$

where $A_{1}$, $A_{2}$ and $E_{2}^{(1/2)}$ denote different symmetry channels.

To lowest order $O(t_{1}^{2}/U+t_{2}^{2}/U)$, where $t_{1}$ and $t_{2}$
denotes nearest neighbor and next nearest neighbor hopping integrals
respectively, we obtain the Elliot-Fleury-Loudon scattering operator [31, 32]
which consists of terms $S_{i}\cdot S_{j}$

$\hat{M}_{EFL}\propto\sum_{\bm{r},\bm{r^{\prime}}}\frac{2t_{\bm{r}\bm{r^{\prime}}}^{2}}{U-\omega_{i}}(\bm{e}_{i}\cdot\bm{\delta})(\bm{e}_{f}\cdot\bm{\delta})\bm{S_{r}}\cdot\bm{S_{r}^{\prime}},$
(7)

where $t_{\bm{r}\bm{r^{\prime}}}$ is the hopping between site $\bm{r}$ and
site $\bm{r^{\prime}}$, $\bm{\delta}=\bm{r^{\prime}}-\bm{r}$, $U$ is the
onsite repulsion and $\omega_{i}$ is the incident photon energy. We note that
the derivation of $\hat{M}_{EFL}$ involves the same two-step virtual hoppings
in the derivation of the Heisenberg Hamiltonian from the Hubbard Hamiltonian,
and we have $J_{1}\sim 4t_{1}^{2}/U$ and $J_{2}\sim 4t_{2}^{2}/U$. Elliot-
Fleury-Loudon scattering operator gives the lowest order expressions for
$\hat{O}_{A_{1}}$, $\hat{O}_{E_{2}^{(1)}}$ and $\hat{O}_{E_{2}^{(2)}}$.
However, we note that the $A_{2}$ channel changes sign under reflection, but
the Elliot-Fleury-Loudon term is invariant under reflection. Thus,
$\hat{O}_{A_{2}}$ vanishes at this order. The lowest order non-zero
$\hat{O}_{A_{2}}$ would be found at $O(t_{1}^{3}t_{2}/U^{3})$, and consists of
a sum of chiral terms
$\bm{S_{r_{1};r_{2};r_{3}}}=\bm{S_{r_{1}}}\cdot(\bm{S_{r_{2}}}\times\bm{S_{r_{3}}})$.
The derivation and specific forms for these scattering operators are given in
the supplementary material.

It is helpful for the understanding of the Raman susceptibility to first look
at the ground state point group symmetry across the three phases when tuning
the ratio $J_{2}/J_{1}$. From the eigenvalue spectrum (see the supplementary
material or Ref. [16]), we know that the transition from the QSL phase to the
striped phase is related to a level crossing in the ground state. In the QSL
phase, the ground state belongs to the symmetry subgroup $\Gamma.A_{1}$, where
$\Gamma$ denotes momentum $0$ and $A_{1}$ denotes the trivial representation
of the $D_{6}$ point group. The first excited state belongs to $\Gamma.E_{2}$
and is doubly degenerate, since $E_{2}$ is the two dimensional representation
of the $D_{6}$ point group. Increasing $J_{2}$, the energy of the
$\Gamma.E_{2}$ doublet goes down and eventually crosses $\Gamma.A_{1}$ at the
phase transition between the QSL phase and the striped phase. Beyond the level
crossing, the $\Gamma.A_{1}$ state energy remains slightly above that of
$\Gamma.E_{2}$ in the striped phase, and as we will see, contributes to the
very low frequency peak in the two $E_{2}$-channel Raman susceptibilities. In
contrast, the ground state in both the $120\degree$ AF phase and the QSL phase
belongs to the $\Gamma.A_{1}$ sector and there is no level crossing.

Figure 4:  Raman susceptibility for different symmetry channels as a function
of $J_{2}/J_{1}$. We do not include the $(t_{1}/U)^{4}$ factor in the $A_{2}$
channel spectra. All spectra are normalized to the maximum value of $A_{2}$
spectra. We enhanced the $A_{1}$ and $E_{2}$ spectra to make them visible. The
red and blue dashed line, corresponding to $J_{2}=0.08$ and $J_{2}=0.16$, mark
the boundary between the $120\degree$ AF phase, the QSL phase and the stripe
phase. In the stripe phase, the low excitation peaks in the $E_{2}$ channel
originates from an excited state nearly degenerate to the ground state.

The Raman spectra are obtained using

$R^{\alpha}(\omega)=-\frac{1}{\pi}\text{Im}\left<G\left|\hat{O}_{\alpha}^{\dagger}\frac{1}{\omega+E_{0}+i\epsilon-\hat{H}}\hat{O}_{\alpha}\right|G\right>,$
(8)

where $\hat{O}_{\alpha}$ denotes a Raman scattering operator in channel
$\alpha$. The Raman scattering susceptibility is defined as

$\chi^{\alpha}(\omega)=R^{\alpha}(\omega)-R^{\alpha}(-\omega),$ (9)

which removes the elastic peak in $R^{\alpha}(\omega)$.

In Fig. 4, we plot $\chi^{\alpha}(\omega)$ as a function of $J_{2}/J_{1}$.
$\chi^{\alpha}(\omega)$ changes dramatically when transitioning from the QSL
phase to the striped phase as expected. As mentioned before, there is a very
low energy peak in the striped phase $E_{2}$ channel susceptibility coming
from the $\Gamma.A_{1}$ state. Because only the $E_{2}$ channel Raman
scattering operators connect the $\Gamma.E_{2}$ ground state with the
$\Gamma.A_{1}$ excited state, this low energy peak is visible only in the
$E_{2}$ Raman scattering channel. In addition to this low frequency peak,
compared to the QSL phase there are two strong peaks close to each other below
$0.5\,J_{1}$ in the striped phase $A_{1}$ channel; they also are visible in
$E_{2}$, but weaker. In the $A_{2}$ channel, there is a strong peak around
$0.5\,J_{1}$ in the QSL phase, while there is a much weaker peak below
$0.5\,J_{1}$ in the striped phase.

In contrast to the discontinuity observed in $\chi^{\alpha}(\omega)$ caused by
the first order phase transition from the QSL phase to the striped phase,
$\chi^{\alpha}(\omega)$ changes smoothly from the $120\degree$ AF phase into
the QSL phase. However, we see that there is a level crossing for the lowest
energy excitation in the $E_{2}$ channels, which occurs with the phase
transition from the $120\degree$ AF phase into the QSL phase. This signals a
second order phase transition. Simultaneously, we see that a low frequency
peak below $0.5\,J_{1}$ develops in the $A_{1}$ channel after entering the QSL
phase. These distinct features in the Raman spectrum for different phases may
be utilized to identify them experimentally.

We note that since the $A_{2}$ channel scattering operator is derived from
higher order terms, it has a $(t_{1}/U)^{2}$ prefactor compared to scattering
operators from other channels. Consequently, the spectral intensity will scale
relatively as $(t_{1}/U)^{4}$ compared to other channels. In Fig. 4, we do not
include this prefactor in the $A_{2}$ channel spectra, so one can compare the
relative intensity of excitations induced by scattering operators coming from
different symmetry as if they are treated on the same order. This is helpful
to truly identify the dominant excitations. As we can see, the $A_{2}$ channel
excitation is dominant and is especially strong in the QSL phase. The
intensity of features in the $A_{1}$ and $E_{2}$ channels of Fig. 4 is
multiplied by $2000$ and $200$, respectively, so as to make them visible in
comparison to the intensity of features in the $A_{2}$ channel. This reveals
that the dominant excitation in the QSL phase is a chiral mode and the system
may have a strong subleading tendency toward chiral order. Tendency towards
chirality in the $J_{1}$-$J_{2}$ model was analyzed in an early work by
Baskaran [33]. However, recent works have suggested that a $J_{2}$ term alone
is not enough to break time reversal symmetry in the ground state. Wietek et
al. [16] suggested that in the QSL phase parameter regime, a chiral spin
liquid (CSL) phase can be realized by adding a small chiral term via a
magnetic field. Alternatively, an additional four-spin ring exchange
interaction originating from the underlying Hubbard Hamiltonian is also
suggested to realize a CSL [34, 35].

In summary, we obtained the dynamical spin structure factor and Raman spectra
for the QSL phase, and its neighboring phases, on the triangular lattice
$J_{1}$-$J_{2}$ Heisenberg model. For the $120\degree$ AF phase and the
striped phase, the lowest excitation in $\mathcal{S}(q,\omega)$ occurs at the
corresponding wave vector and should become gapless in the thermodynamic
limit. In contrast, the spin gap in the QSL phase extrapolates to a finite
value using the $4\times L_{x}$ clusters, in agreement with DMRG results [11,
12, 13]. We also find that the lowest spin excitations in the QSL phase spread
accross the entire Brillouin zone boundary, with gap sizes that are comparable
on the 36-site cluster. In the Raman spectra, we see a level crossing in the
$E_{2}$ channel and a low energy peak below $0.5\,J_{1}$ that gradually
develops in the $A_{1}$ channel as the system transitions from the
$120\degree$ AF phase to the QSL phase. There are abrupt changes in the
spectra that occur in all channels transitioning from the QSL phase to the
striped phase.

We note that there is no sign of a gappless continuum [20] in any Raman
channel in the QSL phase. Combining results from the dynamical spin structure
factor and Raman susceptibility (Elliot-Fleury-Loudon terms), the QSL phase is
gapped in the $\Delta S_{z}=0,\,\pm 1,\,\pm 2$ spin excitation channels,
stronly suggestive of a gapped QSL phase. Interestingly, the Raman $A_{2}$
channel scattering operator consists of chiral terms and its spectra show a
very prominent, but gapped mode across the QSL phase, suggesting a strong
subleading tendency toward chiral order. Because the excitation in the $A_{2}$
channel is so strong in the QSL phase, it still can have comparable strength
relative to other Raman channels, even accounting for all prefactors, making
it easier to access experimentally.

The distinct spectral features for different phases can serve as a fingerprint
for identifying QSL signals in experiments on triangular lattice materials.
One also can apply isotropic pressure or strain; and since $J_{2}$ falls-off
faster as we increase lattice spacing, the $J_{2}/J_{1}$ ratio can be tuned in
this fashion. The ratio also will be different in different materials; and one
may tune $J_{2}/J_{1}$ through synthesis to observe the evolution of spectral
features.

The authors would like to thank Yifan Jiang, Hongchen Jiang and Johannes
Motruk for helpful discussions and suggestions. This work was supported by the
U.S. Department of Energy, Office of Basic Energy Sciences, Division of
Materials Sciences and Engineering, under Contract No. DE-AC02-76SF00515. The
computational results utilized the resources of the National Energy Research
Scientific Computing Center (NERSC) supported by the U.S. Department of
Energy, Office of Science, under Contract No. DE-AC02-05CH11231.

## supplementary Materials

### Eigenvalue Spectrum

Due to translation symmetry, the Hilbert space can be decomposed into
subspaces labeled by different momenta and the Hamiltonian is block diagonal
in these subspaces. In Fig. 5, we plot eigenvalues for the $S_{z}=0$ and
$S_{z}=1$ sectors in different momentum subspaces as a function of
$J_{2}/J_{1}$. The ground state is always in the $\Gamma$ (total momentum 0)
subspace. We can further decompose the total momentum 0 subspace using the
$D_{6}$ point group symmetry. There is a level crossing around
$J_{2}/J_{1}\approx 0.17$, before which the ground state is in the
$\Gamma.A_{1}$ sector, and after which the ground state is doubly degenerate
and comes from the $\Gamma.E_{2}$ sector. Here, $A_{1}$ is the trivial
representation of the $D_{6}$ point group; $E_{2}$ is the two dimensional
representation of the $D_{6}$ point group and has two copies $E_{2}^{(1)}$ and
$E_{2}^{(2)}$.

Figure 5:  Eigenvalues of the 36-site cluster as functions of $J_{2}/J_{1}$.
Different colors represent different momentum subspaces. Here $\Gamma$ denotes
the BZ center, $K$ dentotes the BZ corner, $M$ denotes the BZ edge center, and
$A-E$ denotes moment points with increasing distance from the $\Gamma$ point.
Downward triangles represent $S_{z}=0$ states and upward triangles represent
$S_{z}=1$ states.

### Raman Scattering Operators

The light-mater interaction Hamiltonian is

$H=H_{el}+H_{\gamma}+H_{int},$ (10)

where $H_{el}$ is the electronic part modeled by a Hubbard Hamiltonian with
on-site Coulomb repulsion, nearest neighbor and next-nearest neighbor hopping,
given by

$\displaystyle H_{el}$ $\displaystyle=$ $\displaystyle H_{t}+H_{U}$ (11)
$\displaystyle=$
$\displaystyle\sum_{\left<ij\right>,\sigma}\\!t_{1}c_{i\sigma}^{\dagger}c_{j\sigma}+\\!\\!\\!\sum_{\left<\left<ij\right>\right>,\sigma}\\!\\!t_{2}c_{i\sigma}^{\dagger}c_{j\sigma}+U\sum_{i}n_{i\uparrow}n_{i\downarrow},$

and $H_{\gamma}=\omega_{\gamma}n_{\gamma}$ is the free photon Hamiltonian. The
light-matter interaction is obtained by using Peierls substitution
$c_{i\sigma}^{\dagger}c_{j\sigma}\rightarrow
c_{i\sigma}^{\dagger}c_{j\sigma}\text{exp}(\frac{ie}{\hbar
c}\int_{j}^{i}\bm{A}\cdot\text{d}\bm{r})$ and expanding to second order in the
vector potential

$\displaystyle H_{int}$ $\displaystyle=$ $\displaystyle
H_{int}^{(1)}+H_{int}^{(2)}$ (12) $\displaystyle=$
$\displaystyle\sum_{ij\sigma}t_{ij}c_{i\sigma}^{\dagger}c_{j\sigma}\bigg{\\{}\frac{ie}{\hbar
c}\bm{A}(\frac{\bm{x}_{i}+\bm{x}_{j}}{2})\cdot(\bm{x}_{i}-\bm{x}_{j})$
$\displaystyle-\frac{e^{2}}{2\hbar^{2}c^{2}}\left[\bm{A}(\frac{\bm{x}_{i}+\bm{x}_{j}}{2})\cdot(\bm{x}_{i}-\bm{x}_{j})\right]^{2}\bigg{\\}},$

where $H_{int}^{(1/2)}$ represent terms which are first/second order in
$\bm{A}$. We can obtain the resonant scattering operator via a perturbative
expansion

$\hat{M}_{R}=H_{int}^{(1)}W\sum_{n=0}^{\infty}\left(H_{t}W\right)^{n}H_{int}^{(1)},$
(13)

where $W=1/(\xi_{i}-(H_{U}+H_{\gamma})+i\eta)$, and $\xi_{i}$ is the initial
state energy. We only consider intermediate states with one hole and one
double occupancy, thus $W\approx 1/(\omega_{i}-U)$ in our calculation.

We will use the following spin operator identities valid for singly occupied
states

$c^{\dagger}_{\sigma}c_{\sigma^{\prime}}=\tilde{\chi}_{\sigma^{\prime}\sigma}=\frac{1}{2}\delta_{\sigma^{\prime}\sigma}+\bm{S}\cdot\bm{\tau}_{\sigma^{\prime}\sigma},$
(14)
$c_{\sigma}c^{\dagger}_{\sigma^{\prime}}=\chi_{\sigma\sigma^{\prime}}=\frac{1}{2}\delta_{\sigma\sigma^{\prime}}-\bm{S}\cdot\bm{\tau}_{\sigma\sigma^{\prime}},$
(15)
$(\bm{a}\cdot\bm{\tau})(\bm{b}\cdot\bm{\tau})=(\bm{a}\cdot\bm{b})I+i(\bm{a}\times\bm{b})\cdot\bm{\tau}$
(16)

where $\bm{\tau}$ is pauli matrices and
$\bm{S}=\frac{1}{2}c^{\dagger}_{\sigma}\bm{\tau}_{\sigma\sigma^{\prime}}c_{\sigma^{\prime}}$
is the spin operator.

For convenience, we define the following vectors using triangular lattice
basis vectors $\bm{a}_{1}=(1,0)$ and $\bm{a}_{2}=(1/2,\sqrt{3}/2)$

$\bm{a}_{3}=\bm{a}_{2}-\bm{a}_{1},\bm{R}_{1}=\bm{a}_{1}-\bm{a}_{3},\bm{R}_{2}=\bm{a}_{1}+\bm{a}_{2},\bm{R}_{3}=\bm{a}_{2}+\bm{a}_{3}$
(17)

####  .0.1 Zeroth Order

For the lowest order, we have two pathways for each bond

$\displaystyle T_{0,a}$ $\displaystyle=$
$\displaystyle(\bm{e}_{f}\cdot\bm{\delta}_{1,2})(\bm{e}_{i}\cdot\bm{\delta}_{2,1})\frac{it_{2,1}\cdot
it_{1,2}}{(\omega_{i}-U)}(c^{\dagger}_{1}c_{2})(c^{\dagger}_{2}c_{1})$
$\displaystyle=$ $\displaystyle C_{0}\mathrm{tr}(\chi_{2}\tilde{\chi}_{1})$
$\displaystyle=$ $\displaystyle
C_{0}\mathrm{tr}(\frac{1}{4}I-(\bm{S}_{2}-\bm{S}_{1})\cdot\bm{\tau}-(\bm{S}_{1}\cdot\bm{\tau})(\bm{S}_{2}\cdot\bm{\tau}))$
$\displaystyle=$ $\displaystyle
2C_{0}(\frac{1}{4}-\bm{S}_{1}\cdot\bm{S}_{2}),$

where $t_{i,j}$ denotes the hopping between vertex $v_{i}$ and $v_{j}$,
$\bm{\delta}_{i,j}$ denotes the vector pointing from $v_{i}$ to $v_{j}$,
$C_{0}$ equals to
$(\bm{e}_{f}\cdot\bm{\delta}_{1,2})(\bm{e}_{i}\cdot\bm{\delta}_{1,2}){t^{2}_{1,2}}/{(\omega_{i}-U)}$,
and $(c^{\dagger}_{i}c_{j})$ denotes summation over spin
$\sum_{\sigma}c^{\dagger}_{i\sigma}c_{j\sigma}$. The inverse path (just
exchange the index 1 and 2) gives the same answer. Thus we reproduces the
Elliot-Fleury-Loudon scattering operator

$\hat{M}_{EFL}\propto\sum_{\bm{r},\bm{r}^{\prime}}\frac{4t_{\bm{r}\bm{r}^{\prime}}^{2}}{\omega_{i}-U}(\bm{e}_{i}\cdot\bm{\delta})(\bm{e}_{f}\cdot\bm{\delta})\bm{(}\frac{1}{4}-{\bm{S}_{r}}\cdot\bm{S}_{r^{\prime}}),$
(18)

where $\bm{\delta}=\bm{r}^{\prime}-\bm{r}$. We can decompose it into $A_{1}$
and $E_{2}$ channels (ignoring the constant terms in the $A_{1}$ channel)

$\hat{O}_{A_{1}}=\frac{t_{1}^{2}}{U-\omega_{i}}\sum_{nn}\bm{S}_{i}\cdot\bm{S}_{j}+\frac{t_{2}^{2}}{U-\omega_{i}}\sum_{nnn}3\bm{S}_{i}\cdot\bm{S}_{j}.$
(19)

$\displaystyle\hat{O}_{E_{2}^{(1)}}$ $\displaystyle=$
$\displaystyle\frac{t_{1}^{2}}{U-\omega_{i}}\sum_{r}(\bm{S_{r}}\cdot\bm{S_{r+a_{1}}}-\frac{1}{2}\bm{S_{r}}\cdot\bm{S_{r+a_{2}}}$
$\displaystyle-\frac{1}{2}\bm{S_{r}}\cdot\bm{S_{r+a_{3}}})+\frac{t_{2}^{2}}{U-\omega_{i}}\sum_{r}(\frac{3}{2}\bm{S_{r}}\cdot\bm{S_{r+R_{1}}}$
$\displaystyle+\frac{3}{2}\bm{S_{r}}\cdot\bm{S_{r+R_{2}}}-3\bm{S_{r}}\cdot\bm{S_{r+R_{3}}}).$

$\displaystyle\hat{O}_{E_{2}^{(2)}}$ $\displaystyle=$
$\displaystyle\frac{t_{1}^{2}}{U-\omega_{i}}\sum_{r}\frac{\sqrt{3}}{2}(\bm{S_{r}}\cdot\bm{S_{r+a_{2}}}-\bm{S_{r}}\cdot\bm{S_{r+a_{3}}})$
$\displaystyle+$
$\displaystyle\frac{t_{2}^{2}}{U-\omega_{i}}\sum_{r}\frac{3\sqrt{3}}{2}(\bm{S_{r}}\cdot\bm{S_{r+R_{2}}}-\bm{S_{r}}\cdot\bm{S_{r+R_{1}}}).$

We now try to find the lowest order terms for the $A_{2}$ channel.

####  .0.2 First Order

The first order terms are produced by paths forming a closed triangle, and it
was shown that paths cancel each other in pairs in the $A_{2}$ channel [26].

####  .0.3 Second Order

If we only consider nearest neighbor hopping, the pathways involved in a
parallelogram cancel out in the $A_{2}$ channel [26]. Thus we consider the
next lowest order terms which involve three nearest neighbor hopping $t_{1}$
and one next-nearest neighbor hopping $t_{2}$. The relevant paths form a
closed triangle by 4 vertexes $v_{1}\to v_{2}\to v_{3}\to v_{4}\to v_{1}$,
involving three nearest neighbor edges and one next-nearest neighbor edge. For
convenience, we label the edge connecting vertex $v_{i}$ and $v_{i+1}$ as
$\bm{\delta}_{i}$ where $4+1$ is identified as 1. Without loss of generality,
we can assume the first three edges are nearest neighbor edges and
$\bm{\delta}_{4}$ is a next nearest neighbor edge. We first consider paths
which originate from vertex $v_{1}$ along edge $\bm{\delta}_{1}$: $v_{1}\to
v_{2}$. There are in total 4 such pathways, and the first one is

$\displaystyle T_{2,a}$ $\displaystyle=$ $\displaystyle
C_{2}(\bm{\delta}_{1},\bm{\delta}_{4})(c^{\dagger}_{1}c_{4})(c^{\dagger}_{4}c_{3})(c^{\dagger}_{3}c_{2})(c^{\dagger}_{2}c_{1})$
$\displaystyle=$ $\displaystyle
C_{2}(\bm{\delta}_{1},\bm{\delta}_{4})\mathrm{tr}\\{\chi_{4}\chi_{3}\chi_{2}\tilde{\chi}_{1}\\}$
$\displaystyle\to$ $\displaystyle
iC_{2}(\bm{\delta}_{1},\bm{\delta}_{4})(S_{321}+S_{421}+S_{431}-S_{432}),$

where
$C_{2}(\bm{\delta}_{1},\bm{\delta}_{2})=-(\bm{e}_{i}\cdot\bm{\delta}_{1})(\bm{e}_{f}\cdot\bm{\delta}_{2})t_{1}^{3}t_{2}/(\omega_{i}-U)^{3}$,
$S_{ijk}=(\bm{S}_{i}\times\bm{S}_{j})\cdot\bm{S}_{k}$, and ”$\to$” in the last
line means we only keep chiral terms. The other three pathways’ contributions
are

$\displaystyle T_{2,b}$ $\displaystyle=$ $\displaystyle
C_{2}(\bm{\delta}_{1},\bm{\delta}_{3})(c^{\dagger}_{4}c_{3})(c^{\dagger}_{1}c_{4})(c^{\dagger}_{3}c_{2})(c^{\dagger}_{2}c_{1})$
$\displaystyle=$ $\displaystyle-
C_{2}(\bm{\delta}_{1},\bm{\delta}_{3})\mathrm{tr}\\{\tilde{\chi}_{4}\chi_{3}\chi_{2}\tilde{\chi}_{1}\\}$
$\displaystyle\to$ $\displaystyle-
iC_{2}(\bm{\delta}_{1},\bm{\delta}_{3})(S_{321}-S_{421}-S_{431}+S_{432}),$

$\displaystyle T_{2,c}$ $\displaystyle=$ $\displaystyle
C_{2}(\bm{\delta}_{1},\bm{\delta}_{3})(c^{\dagger}_{4}c_{3})(c^{\dagger}_{3}c_{2})(c^{\dagger}_{1}c_{4})(c^{\dagger}_{2}c_{1})$
$\displaystyle=$ $\displaystyle-
C_{2}(\bm{\delta}_{1},\bm{\delta}_{3})\mathrm{tr}\\{\tilde{\chi}_{4}\chi_{3}\chi_{2}\tilde{\chi}_{1}\\}$
$\displaystyle\to$ $\displaystyle-
iC_{2}(\bm{\delta}_{1},\bm{\delta}_{3})(S_{321}-S_{421}-S_{431}+S_{432}),$

$\displaystyle T_{2,d}$ $\displaystyle=$ $\displaystyle
C_{2}(\bm{\delta}_{1},\bm{\delta}_{2})(c^{\dagger}_{3}c_{2})(c^{\dagger}_{4}c_{3})(c^{\dagger}_{1}c_{4})(c^{\dagger}_{2}c_{1})$
$\displaystyle=$ $\displaystyle
C_{2}(\bm{\delta}_{1},\bm{\delta}_{2})\mathrm{tr}\\{\tilde{\chi_{4}}\tilde{\chi_{3}}\chi_{2}\tilde{\chi}_{1}\\}$
$\displaystyle\to$ $\displaystyle
iC_{2}(\bm{\delta}_{1},\bm{\delta}_{2})(-S_{321}-S_{421}+S_{431}-S_{432}),$

Other pathways on this triangle can be generated by cyclic permutation or
inverse of the vertex indexes: $(1\to 2\to 3\to 4)\to(2\to 3\to 4\to 1),(1\to
2\to 3\to 4)\to(4\to 3\to 2\to 1)\dots$ There are 8 such permutations and each
one has 4 pathways as listed above. After summing them up, and considering all
the possible triangles, we have the lowest order $A_{2}$ channel scattering
operator

$\displaystyle\hat{O}_{A_{2}}$ $\displaystyle=$
$\displaystyle\frac{t_{1}^{3}t_{2}}{(U-\omega_{i})^{3}}\sum_{r}4\sqrt{3}i(\bm{S_{r;r+R_{2};r+a_{1}}}+\bm{S_{r;r+R_{2};r+2a_{1}}}$
$\displaystyle+$
$\displaystyle\bm{S_{r+a_{1};r+R_{2};r+2a_{1}}}+\text{Rotations}-\text{Reflections}),$

where
$\bm{S_{r_{1};r_{2};r_{3}}}=\bm{S_{r_{1}}}\cdot(\bm{S_{r_{2}}}\times\bm{S_{r_{3}}})$,
“Rotations” means terms obtained by rotating previous terms by
$\pi/6,\dots,5\pi/6$, and “Reflections” means terms obtained by reflecting
previous terms along an axis.

## References

  * Anderson [1973] P. W. Anderson, Materials Research Bulletin 8, 153 (1973).
  * Anderson [1987] P. W. Anderson, Science 235, 1196 (1987).
  * Baskaran  _et al._ [1987] G. Baskaran, Z. Zou, and P. W. Anderson, Solid State Communications 63, 973 (1987).
  * Kitaev [2003] A. Y. Kitaev, Annals of Physics 303, 2 (2003).
  * Shimizu  _et al._ [2003] Y. Shimizu, K. Miyagawa, K. Kanoda, M. Maesato, and G. Saito, Physical Review Letters 91, 107001 (2003).
  * Kurosaki  _et al._ [2005] Y. Kurosaki, Y. Shimizu, K. Miyagawa, K. Kanoda, and G. Saito, Physical Review Letters 95, 177001 (2005).
  * Jolicoeur  _et al._ [1990] T. Jolicoeur, E. Dagotto, E. Gagliano, and S. Bacci, Physical Review B 42, 4800 (1990).
  * Bernu  _et al._ [1992] B. Bernu, C. Lhuillier, and L. Pierre, Physical Review Letters 69, 2590 (1992).
  * Bernu  _et al._ [1994] B. Bernu, P. Lecheminant, C. Lhuillier, and L. Pierre, Physical Review B 50, 10048 (1994).
  * Capriotti  _et al._ [1999] L. Capriotti, A. E. Trumper, and S. Sorella, Physical Review Letters 82, 3899 (1999).
  * Zhu and White [2015] Z. Zhu and S. R. White, Physical Review B 92, 041105 (2015).
  * Hu  _et al._ [2015] W.-J. Hu, S.-S. Gong, W. Zhu, and D. N. Sheng, Phys. Rev. B 92, 140403 (2015).
  * Jiang and Jiang [2022] Y.-F. Jiang and H.-C. Jiang, arXiv:2203.10216 [cond-mat] (2022), arXiv:2203.10216 [cond-mat] .
  * Kaneko  _et al._ [2014] R. Kaneko, S. Morita, and M. Imada, Journal of the Physical Society of Japan 83, 093707 (2014).
  * Iqbal  _et al._ [2016] Y. Iqbal, W.-J. Hu, R. Thomale, D. Poilblanc, and F. Becca, Physical Review B 93, 144411 (2016).
  * Wietek and Läuchli [2017] A. Wietek and A. M. Läuchli, Physical Review B 95, 035141 (2017).
  * Hu  _et al._ [2019] S. Hu, W. Zhu, S. Eggert, and Y.-C. He, Physical Review Letters 123, 207203 (2019).
  * Itou  _et al._ [2008] T. Itou, A. Oyamada, S. Maegawa, M. Tamura, and R. Kato, Physical Review B 77, 104413 (2008).
  * Yamashita  _et al._ [2010] M. Yamashita, N. Nakata, Y. Senshu, M. Nagata, H. M. Yamamoto, R. Kato, T. Shibauchi, and Y. Matsuda, Science 328, 1246 (2010).
  * Wulferding  _et al._ [2019] D. Wulferding, Y. Choi, W. Lee, and K.-Y. Choi,  32, 043001 (2019).
  * Broholm  _et al._ [2020] C. Broholm, R. J. Cava, S. A. Kivelson, D. G. Nocera, M. R. Norman, and T. Senthil, Science 367, eaay0668 (2020).
  * Shaginyan  _et al._ [2020] V. R. Shaginyan, V. A. Stephanovich, A. Z. Msezane, G. S. Japaridze, J. W. Clark, M. Y. Amusia, and E. V. Kirichenko, Journal of Materials Science 55, 2257 (2020).
  * Dagotto [1994] E. Dagotto, Reviews of Modern Physics 66, 763 (1994).
  * Ferrari and Becca [2019] F. Ferrari and F. Becca, Physical Review X 9, 031026 (2019).
  * Devereaux and Hackl [2007] T. P. Devereaux and R. Hackl, Reviews of Modern Physics 79, 175 (2007).
  * Ko  _et al._ [2010] W.-H. Ko, Z.-X. Liu, T.-K. Ng, and P. A. Lee, Physical Review B 81, 024414 (2010).
  * Vernay  _et al._ [2007] F. Vernay, T. P. Devereaux, and M. J. P. Gingras, Journal of Physics: Condensed Matter 19, 145243 (2007).
  * Perkins and Brenig [2008] N. Perkins and W. Brenig, Physical Review B 77, 174412 (2008).
  * Knolle  _et al._ [2014] J. Knolle, G.-W. Chern, D. L. Kovrizhin, R. Moessner, and N. B. Perkins, Physical Review Letters 113, 187201 (2014).
  * Hassan  _et al._ [2018] N. Hassan, S. Cunningham, E. I. Zhilyaeva, S. A. Torunova, R. N. Lyubovskaya, J. A. Schlueter, and N. Drichko, Crystals 8 (2018), 10.3390/cryst8060233.
  * Elliott and Loudon [1963] R. J. Elliott and R. Loudon, Physics Letters 3, 189 (1963).
  * Fleury and Loudon [1968] P. A. Fleury and R. Loudon, Physical Review 166, 514 (1968).
  * Baskaran [1989] G. Baskaran, Physical Review Letters 63, 2524 (1989).
  * Cookmeyer  _et al._ [2021] T. Cookmeyer, J. Motruk, and J. E. Moore, Physical Review Letters 127, 087201 (2021).
  * Szasz  _et al._ [2020] A. Szasz, J. Motruk, M. P. Zaletel, and J. E. Moore, Physical Review X 10, 021042 (2020).


# Godot Reinforcement Learning Agents

Edward Beeching,1 Jilles Debangoye, 1 Olivier Simonin 1 Christian Wolf2

###### Abstract

We present Godot Reinforcement Learning (RL) Agents, an open-source interface
for developing environments and agents in the Godot Game Engine. The Godot RL
Agents interface allows the design, creation and learning of agent behaviors
in challenging 2D and 3D environments with various on-policy and off-policy
Deep RL algorithms. We provide a standard Gym interface, with wrappers for
learning in the Ray RLlib and Stable Baselines RL frameworks. This allows
users access to over 20 state of the art on-policy, off-policy and multi-agent
RL algorithms. The framework is a versatile tool that allows researchers and
game designers the ability to create environments with discrete, continuous
and mixed action spaces. The interface is relatively performant, with 12k
interactions per second on a high end laptop computer, when parallized on 4
CPU cores. An overview video is available here: https://youtu.be/g1MlZSFqIj4.

## Introduction

Over the next decade advances in AI algorithms, notably in the fields of Deep
Learning (Lecun, Bengio, and Hinton 2015) and Deep Reinforcement Learning, are
primed to revolutionize the Video Games industry. Customizable enemies, worlds
and story telling will lead to diverse game-play experiences and new genres of
games. Currently the field is dominated by large organizations and pay to use
engines that have the resources to create such AI enhanced agents. The
objective of the Godot RL Agents package is to lower the bar of accessibility;
so that game developers, researchers and hobbyists can take their idea from
creation to publication end-to-end with an open-source and free package. The
Godot RL Agent provides wrappers for two well known open-source Deep
Reinforcement Learning libraries: Ray Rllib (Liang et al. 2018) and Stable
Baselines (Hill et al. 2018). In addition the interface is compatible with any
RL framework that can interact with a gym wrapper. Although the Godot RL
Agents interface was predominantly designed for the creation and prototyping
of RL approaches in video games, there is nothing prohibiting users from the
implementation of tasks involving mobile robotics, grasping or designing
complex memory-based scenarios to test the limitations of current Deep RL
architectures and algorithms. The contributions the Godot RL Agents packages
are as follows:

  * •

A free and open source tool for Deep RL research and game development.

  * •

By providing an interface between the Godot Game Engine and Deep RL
algorithms, we enable game creators to imbue their non-player characters with
unique behaviors, learned through interaction.

  * •

The framework enables automated game-play testing with an RL agent.

  * •

The Godot RL Agents framework enables researchers and game designers the
flexibility to design, create and perform rapid iteration on new ideas and
scenarios.

The library is open-source with an MIT licence, code is available on our
github page:
https://github.com/edbeeching/godot_rl_agents

Figure 1: An example Godot RL Agents environment Fly By, where an agent learns
to perform way-point based aerial navigation in a 3D space. Figure 2: The
training architecture used for the Godot RL Agents interface. Parallel actor
processes collect trajectories from several environment executables. Each
environment contains multiple parallel agents, each with their own
instantiation of the environment.

## Related work

The last decade has seen rapid advancement in the field of artificial
intelligence, this has been driven by the application of Deep Learning
approaches and in particular Convolutional Neural Networks (Fukushima and
Miyake 1982; Lecun et al. 1998) . Supervised Deep Learning in its current
state often considered to be narrow AI, that performs particularly well at
performing predictions on static images but does not learn from interaction.

### Deep Reinforcement Learning

Reinforcement Learning approaches provide the ability to learn in sequential
decision making problems, where the objective is to maximize accumulated
reward. As we encounter large state spaces, continuous actions, partial
observability (Åström 1965) and the desire to generalize to unseen environment
configurations; Deep Reinforcement Learning (RL) provides a general framework
for solving these problems. In recent years the field of Deep RL has gained
attention with successes on board games (Silver et al. 2016) and Atari Games
(Mnih et al. 2015). One key component was the application of deep neural
networks to frames from the environment or game board states. Recent works
that have applied Deep RL for the control of an agent in 3D environments such
as maze navigation are (Mirowski et al. 2017) and (Jaderberg et al. 2017)
which explored the use of auxiliary tasks such as depth prediction, loop
detection and reward prediction to accelerate learning. Meta RL approaches for
3D navigation have been applied by (Wang et al. 2016) and (Lample and Chaplot
2017) also accelerated the learning process in 3D environments by prediction
of tailored game features. There has also been recent work in the use of
street-view scenes in order train an agent to navigate in city environments
(Kayalibay et al. 2018). In order to infer long term dependencies and store
pertinent information about the environment; network architectures typically
incorporate recurrent memory such as Gated Recurrent Units (Chung et al. 2015)
or Long Short-Term Memory (Hochreiter and Schmidhuber 1997).
The scaling of Deep RL has produced some impressive results, such as in the
IMPALA (Espeholt et al. 2018) architecture which successfully trained an agent
that can achieve human level performance in all 30 of the 3D, partially
observable, DeepMind Lab (Beattie et al. 2016) tasks; accelerated methods such
as in (Stooke and Abbeel 2018) which solve many Atari environments in tens of
minutes. The achievements in long term planing and strategy in DOTA (OpenAI
2018) and StartCraft 2 (Vinyals et al. 2019), has demonstrated the Deep RL can
learn policies with complex reasoning and long term decision making. There has
been a recent push towards photo-realistic simulation, with the release of the
Gibson (Xia et al. 2018) dataset and the Habitat simulator (Manolis Savva et
al. 2019; Szot et al. 2021). Many research teams focus on Structured Deep
Reinforcement Learning, where priors are incorporated in the agent
architecture, leading to advancements in GPS guided point-to-point navigation
(Chaplot et al. 2020a, d), memory-based tasks (Wayne et al. 2018; Beeching et
al. 2020a, b), semantic prediction (Chaplot et al. 2020b, c). Other approaches
aim to solve these challenging problems with scale, using distributed
computing to training larger networks for billions of environment interactions
(Wijmans et al. 2019; Espeholt et al. 2018). The video games industry has
identified the advancements in Deep Reinforcement Learning as an opportunity
to enrich video game experiences. With short term objectives being that of
automated testing (Gordillo et al. 2021) and content generation (Gisslén et
al. 2021), and medium to longer term being player facing bots (Alonso et al.
2021), controlled by policies learned through interaction with the game
environment.

Figure 3: 4 of the example environments available in the Godot RL Agents
framework. Top left: Fly By, where the agent must learn to fly a place between
way-points in 3D space. Top right: Jumper, where the objective is the
navigated a bipedal robot between platforms, jumping where necessary. Bottom
left: Space Shooter, a 2D multi agent scenario, where one team must shoot and
and destroy the agents from the other team. Bottom-right: Ball Chase, where
the agent must learn to collect pink fruits, while avoiding walls and
obstacles.

### Deep RL environments

There has been a large expansion to the number of Deep RL environments
available in the last five years, the most well known being the gym framework
(Brockman et al. 2016) which provides an interface to fully observable games
such as the Atari-57 benchmark. Mujuco (Todorov, Erez, and Tassa 2012) has
remained the de facto standard for continuous control tasks, its recent open-
source release can only benefit the larger RL community. OpenSpiel (Lanctot et
al. 2019) provides a framework for RL in games such as Chess, Poker, Go,
TicTacToe and Habani, to name a few. There are a variety of partially
observable 3D simulators available, with more game-like environments such as
the ViZDoom simulator (Wydmuch, Kempka, and Jaśkowski 2018), DeepMind-Lab
(Beattie et al. 2016) and Malmo (Johnson et al. 2016). A number of photo-
realistic simulators have also been released such as Gibson (Xia et al. 2018),
AI2Thor (Kolve et al. 2017), and the highly efficient Habitat-Lab simulator
(Manolis Savva et al. 2019; Szot et al. 2021). The aforementioned environments
were designed to train and evaluate RL agents in relatively small environments
and typically are somewhat inflexible to addition of new tasks, abilities and
action spaces. The Unity ML agents framework (Juliani et al. 2018), provides
flexibility in terms of design of environments and tasks, but with the
downside of being a closed-source and pay to use (if your organization has a
revenue of more than $100k per year). In addition its closed-source
implementation is inflexible if you wish to, for example, make changes to the
protocol used to exchange observations between the environment and the RL
algorithm. For these reasons the platform is platform somewhat prohibitive to
the wider RL community.

### The Godot Game Engine

The Godot Game Engine (Linietsky, Manzur, and Verschelde 2014) is an open-
source tool for developing 2D and 3D games. Currently Godot is the most
popular open-source Game Engine available and can export to MacOS, Linux,
Windows, Android, iOS, HTML and web assembly. Godot supports multiple
programming languages including Python, C#, C++ and GDScript (a Python-like
scripting language). The Game Engine provides an interactive editor for the
design and creation of video game environments, characters, animations and
menus, which enables fast iteration for prototyping new ideas. Godot has all
the features of a modern Game Engine such physics simulation, custom
animations, plugins and physically-based rendering.

### Deep RL Frameworks

When it comes to applying a state of the art Deep RL algorithm, there are many
quality open-source implementations available such as RLlib (Liang et al.
2018), ACME (Hoffman et al. 2020), SampleFactory (Petrenko et al. 2020), Seed
RL (Espeholt et al. 2019) and Stable Baselines (Hill et al. 2018) to name a
few. Due to the availability of such high-quality, featured and well tested
Deep RL frameworks; in this work we chose to design and implement an interface
between the Godot Game Engine and two well known RL frameworks: RLlib (Liang
et al. 2018) and Stable Baselines (Hill et al. 2018). Our objective is to
focus on the interface and quality examples of 2D and 3D environments. This is
contrary to the Unity ML Agents framework, which also implemented a limited
set of RL algorithms.

Figure 4: Training curves available in Godot RL Agents for the Ball Chase
environment. Shown are a subset of available statistics output during
training, Mean return, Entropy loss, Policy loss and Value loss

## Godot RL Agents

The Godot RL Agents framework is an interface between the Godot Game Engine
and Deep RL frameworks, allowing the creation of 2D, 3D and text-based
environments, custom observations and reward functions. It is a flexible, fast
and robust interface between RL training code running in Python and the Godot
environment running interactively or as a compiled executable. We implement
two forms of parallelization in the Godot RL Agents interface: in-game
environment duplication and multiprocessing with several parallel executables.
The interface has been designed to support continuous, discrete and mixed
action spaces. Action repetition is implemented on the environment side to
avoid unnecessary inter-process communication. Training can be performed
interactively, while the environment is running in the Godot game editor:
enabling mid-training pausing, analysis and debugging of variables. Once the
user is confident with their environment configuration though interactive
testing, the environment can be exported and run as an executable. Environment
export enables accelerated, faster than real-time physics and headless
training. The connection between the Game Engine and the Python training code
is implemented with a TCP socket client-server relationship; this allows the
environment to be run locally or distributed across many machines.

We provide a standard Gym wrapper and wrappers to Ray RLlib and Stable
Baselines, which support the following algorithms.
On-policy algorithms:

  * •

A2C/A3C (Mnih et al. 2016)

  * •

PPO/APPO (Schulman et al. 2017)

  * •

IMPALA (Espeholt et al. 2018)

  * •

DD-PPO (Wijmans et al. 2019)

Off-policy algorithms:

  * •

DQN (Mnih et al. 2013)

  * •

Rainbow (Hessel et al. 2018)

  * •

CQL (Kumar et al. 2020)

  * •

DDPG / TD3 (Lillicrap et al. 2015)

  * •

APEX (Horgan et al. 2018)

  * •

R2D2 (Kapturowski et al. 2018)

  * •

SAC continuous (Haarnoja et al. 2018) and discrete (Christodoulou 2019).

Other algorithms are also available such as Behavior Cloning (Wang et al.
2018), Intrinsic curiosity (Pathak et al. 2017) and Evolutionary Strategies
(Salimans et al. 2017). These pre-existing implementations enable researchers
and game developers to rapidly evaluate different baselines architectures and
algorithms, with robust implementations of state of the art RL algorithms. The
detail the distributed architecture used in Godot RL Agents in Figure 2.

### Sensors

Godot RL Agents allows users to add custom sensors to their agents to observe
the environment. While simulated monocular cameras are possible, rendering is
an expensive operation and rendered images do no always provide the more
pertinent information. We provide implementations of circular and spherical
”Raycast” sensors in both 2D and 3D that perform depth measurements,
essentially a virtual LIDAR. Raycasts are a popular technique in video games
and Game Engines such as Godot offer a highly optimized raycast
implementation, making ray-casts a cheap option to augment an agent’s
observation. Other custom information, such as position, can be included in
the agents observation, if the environment designer believes it is applicable
to the task.

### Example environments

In version 0.1.0 of Godot RL Agents, we have implemented 4 example
environments as a reference for researchers and game creators who wish to use
our tool to build new worlds. We have created two 3D environments and two 2D
environments. The source-code and further details (such as reward functions,
reset conditions, action spaces) of these reference implementations, is
available in our open-source repository
https://github.com/edbeeching/godot˙rl˙agents.

### Jumper

Jumper is a 3D single agent environment where the agent must learn to jump
from one randomly placed platform to the next. The action space is continuous
for turning and movement, and discrete for the jump action. The episode is
terminate if the agent falls off the platform. Observations are a cone of 3D
ray-casts in front of the agent and vector pointing towards the next platform.

### Ball Chase

Ball Chase is a 2D single agent environment where the agent must navigate
around room and collect pink balls. The episode is terminated if the agent
hits a wall. Observations are a circle of 2D ray-casts around the agent and
vector pointing towards the next fruit. The action space is a continuous 2D
vector indicating the move direction in x and y.

### Fly By

Fly By is a 3D single environment where the agent must learn to fly from one
way-point to the next. The episode is terminated if the agent hits a leave the
game are. Observations are two vectors to the next two subsequent way-points.

### Space Shooter

Space Shooter is a 2D multi-agent environment with two teams of 8 agents,
where the objective is to shoot and destroy the enemy team. Currently
observations are an array of relative positions of the enemy and friend teams.
Results, observations and training methodology in the environment are still
work in progress and will change in the next version of Godot RL Agents.

## Experiments and Benchmarks

As part of our reference implementation, we provide a PPO example with
reasonable default parameters in order to provide a good starting point for
training RL agents. We believe this algorithm is a suitable starting point, as
it allows for both continuous and discrete action spaces, is robust to hyper-
parameter configurations, and is relatively sample efficient. In Figure 4 we
show example training curves for the Ball Chase environment. We provide pre-
trained models for all example environments in our repository, refer to our
overview video for examples of these behaviors:

https://youtu.be/g1MlZSFqIj4.

We benchmark the interaction rate of the Jumper environment and achieve 12k
environment interactions per second with up to 4 parallel processes, results
shown in Figure 5. Benchmarking was performed on a relatively high-end laptop
computer 111Dell XPS - i7-10750H (6 cores), GTX 1650 Ti, 32 GB RAM, future
work on Godot RL Agents will benchmark on dedicated hardware.

Figure 5: Benchmark of interactions per second when performing multi-process
training in the Jumper environment with an action repeat of 4. We achieve 12k
interactions per second with 4 parallel processes. Ideally scaling would be
linear, future versions of Godot RL Agents will aim to profile and optimize
the scaling of environment interactions.

## Conclusions and future work

We have introduced the Godot RL Agents framework, a tool for building 2D and
3D environments in the Godot Game Engine and learning agent behaviors with
Deep Reinforcement Learning. We have created 4 example environments to
demonstrate the capabilities of this versatile tool. By interfacing with well
known open-source Deep RL implementations, we provide access to over 20 RL
algorithms out of the box. We have implemented sensors, tools for parallel
interaction with environments, and support for continuous, discrete and mixed
action spaces.

Future work on the framework aims to include: variable size observations
coupled with attention mechanisms, text-based environments, environments with
discrete state spaces, virtual monocular cameras for vision-based agent
control, multi-modal observations (1D, 2D, text & audio) and model export for
in-engine inference of trained behaviors. We plan to extend our focus to
multi-agent environments, preliminary work in the Space Shooter environment
shows this area is a challenging area of research, with applications in both
video games and collaborative robotics.

External datasets can also be loaded in to the Godot Game Engine. In Figure 6
we load photo-realistic scans from the Gibson (Xia et al. 2018) dataset, to
demonstrate that is tool can be used to build realistic 3D scenes for learning
robotic control and sim2real transfer. We believe that there are near endless
possibilities for this tool and are interesting to discuss with the research
community the future direction of this work.

Figure 6: Future work, an example of a virtual monocular camera, rendering
part of a scene from the Gibson dataset, loaded as an environment in the Godot
RL Agents framework.

## References

  * Alonso et al. (2021) Alonso, E.; Peter, M.; Goumard, D.; and Romoff, J. 2021.  Deep Reinforcement Learning for Navigation in AAA Video Games.  In Zhou, Z.-H., ed., _Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21_ , 2133–2139. International Joint Conferences on Artificial Intelligence Organization.  Main Track.
  * Åström (1965) Åström, K. J. 1965.  Optimal control of Markov processes with incomplete state information I.  _Journal of Mathematical Analysis and Applications_ , 10: 174–205.
  * Beattie et al. (2016) Beattie, C.; Leibo, J. Z.; Teplyashin, D.; Ward, T.; Wainwright, M.; Küttler, H.; Lefrancq, A.; Green, S.; Valdés, V.; Sadik, A.; Schrittwieser, J.; Anderson, K.; York, S.; Cant, M.; Cain, A.; Bolton, A.; Gaffney, S.; King, H.; Hassabis, D.; Legg, S.; and Petersen, S. 2016.  DeepMind Lab.  _arxiv pre-print 1612.03801_.
  * Beeching et al. (2020a) Beeching, E.; Dibangoye, J.; Simonin, O.; and Wolf, C. 2020a.  EgoMap: Projective mapping and structured egocentric memory for Deep RL.  In _ECMLPKDD_.
  * Beeching et al. (2020b) Beeching, E.; Dibangoye, J.; Simonin, O.; and Wolf, C. 2020b.  Learning to plan with uncertain topological maps.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16_ , 473–490. Springer.
  * Brockman et al. (2016) Brockman, G.; Cheung, V.; Pettersson, L.; Schneider, J.; Schulman, J.; Tang, J.; and Zaremba, W. 2016.  OpenAI Gym.  _arxiv pre-print 1606.01540_.
  * Chaplot et al. (2020a) Chaplot, D. S.; Gandhi, D.; Gupta, S.; Gupta, A.; and Salakhutdinov, R. 2020a.  Learning to explore using active neural slam.  _arXiv preprint arXiv:2004.05155_.
  * Chaplot et al. (2020b) Chaplot, D. S.; Gandhi, D. P.; Gupta, A.; and Salakhutdinov, R. R. 2020b.  Object goal navigation using goal-oriented semantic exploration.  _Advances in Neural Information Processing Systems_ , 33.
  * Chaplot et al. (2020c) Chaplot, D. S.; Jiang, H.; Gupta, S.; and Gupta, A. 2020c.  Semantic curiosity for active visual learning.  In _European Conference on Computer Vision_ , 309–326. Springer.
  * Chaplot et al. (2020d) Chaplot, D. S.; Salakhutdinov, R.; Gupta, A.; and Gupta, S. 2020d.  Neural topological slam for visual navigation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 12875–12884.
  * Christodoulou (2019) Christodoulou, P. 2019.  Soft actor-critic for discrete action settings.  _arXiv preprint arXiv:1910.07207_.
  * Chung et al. (2015) Chung, J.; Gulcehre, C.; Cho, K.; and Bengio, Y. 2015.  Gated Feedback Recurrent Neural Networks.  In _ICML_.
  * Espeholt et al. (2019) Espeholt, L.; Marinier, R.; Stanczyk, P.; Wang, K.; and Michalski, M. 2019.  SEED RL: Scalable and Efficient Deep-RL with Accelerated Central Inference.
  * Espeholt et al. (2018) Espeholt, L.; Soyer, H.; Munos, R.; Simonyan, K.; Mnih, V.; Ward, T.; Doron, Y.; Firoiu, V.; Harley, T.; Dunning, I.; et al. 2018.  IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures.  In _Proceedings of the International Conference on Machine Learning (ICML)_.
  * Fukushima and Miyake (1982) Fukushima, K.; and Miyake, S. 1982.  Neocognitron: A self-organizing neural network model for a mechanism of visual pattern recognition.  In _Competition and cooperation in neural nets_ , 267–285. Springer.
  * Gisslén et al. (2021) Gisslén, L.; Eakins, A.; Gordillo, C.; Bergdahl, J.; and Tollmar, K. 2021.  Adversarial reinforcement learning for procedural content generation.  _arXiv preprint arXiv:2103.04847_.
  * Gordillo et al. (2021) Gordillo, C.; Bergdahl, J.; Tollmar, K.; and Gisslén, L. 2021.  Improving Playtesting Coverage via Curiosity Driven Reinforcement Learning Agents.  arXiv:2103.13798.
  * Haarnoja et al. (2018) Haarnoja, T.; Zhou, A.; Abbeel, P.; and Levine, S. 2018.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.  _arXiv preprint arXiv:1801.01290_.
  * Hessel et al. (2018) Hessel, M.; Modayil, J.; van Hasselt, H. P.; Schaul, T.; Ostrovski, G.; Dabney, W.; Horgan, D.; Piot, B.; Azar, M.; and Silver, D. 2018.  Rainbow: Combining improvements in deep reinforcement learning.  In _Thirty-Second AAAI Conference on Artificial Intelligence_.
  * Hill et al. (2018) Hill, A.; Raffin, A.; Ernestus, M.; Gleave, A.; Kanervisto, A.; Traore, R.; Dhariwal, P.; Hesse, C.; Klimov, O.; Nichol, A.; Plappert, M.; Radford, A.; Schulman, J.; Sidor, S.; and Wu, Y. 2018.  Stable Baselines.  https://github.com/hill-a/stable-baselines.
  * Hochreiter and Schmidhuber (1997) Hochreiter, S.; and Schmidhuber, J. 1997.  Long Short-Term Memory.  _Neural Computation_ , 9(8): 1735–1780.
  * Hoffman et al. (2020) Hoffman, M.; Shahriari, B.; Aslanides, J.; Barth-Maron, G.; Behbahani, F.; Norman, T.; Abdolmaleki, A.; Cassirer, A.; Yang, F.; Baumli, K.; Henderson, S.; Novikov, A.; Colmenarejo, S. G.; Cabi, S.; Gulcehre, C.; Paine, T. L.; Cowie, A.; Wang, Z.; Piot, B.; and de Freitas, N. 2020.  Acme: A Research Framework for Distributed Reinforcement Learning.  _arXiv preprint arXiv:2006.00979_.
  * Horgan et al. (2018) Horgan, D.; Quan, J.; Budden, D.; Barth-Maron, G.; Hessel, M.; Van Hasselt, H.; and Silver, D. 2018.  Distributed prioritized experience replay.  _arXiv preprint arXiv:1803.00933_.
  * Jaderberg et al. (2017) Jaderberg, M.; Mnih, V.; Czarnecki, W. M.; Schaul, T.; Leibo, J. Z.; Silver, D.; and Kavukcuoglu, K. 2017.  Reinforcement Learning with Unsupervised Auxiliary Tasks.  In _ICLR_.
  * Johnson et al. (2016) Johnson, M.; Hofmann, K.; Hutton, T.; and Microsoft, D. B. 2016.  The Malmo Platform for Artificial Intelligence Experimentation.  In _IJCAI_.
  * Juliani et al. (2018) Juliani, A.; Berges, V.-P.; Vckay, E.; Gao, Y.; Henry, H.; Mattar, M.; and Lange, D. 2018.  Unity: A general platform for intelligent agents.  _arXiv preprint arXiv:1809.02627_.
  * Kapturowski et al. (2018) Kapturowski, S.; Ostrovski, G.; Quan, J.; Munos, R.; and Dabney, W. 2018.  Recurrent experience replay in distributed reinforcement learning.  In _International conference on learning representations_.
  * Kayalibay et al. (2018) Kayalibay, B.; Mirchev, A.; Soelch, M.; Van Der Smagt, P.; and Bayer, J. 2018.  Navigation and planning in latent maps.  In _FAIM workshop “Prediction and Generative Modeling in Reinforcement Learning”_.
  * Kolve et al. (2017) Kolve, E.; Mottaghi, R.; Gordon, D.; Zhu, Y.; Gupta, A.; and Farhadi, A. 2017.  AI2-THOR: An Interactive 3D Environment for Visual AI.  _arxiv pre-print 1712.05474v1_.
  * Kumar et al. (2020) Kumar, A.; Zhou, A.; Tucker, G.; and Levine, S. 2020.  Conservative q-learning for offline reinforcement learning.  _arXiv preprint arXiv:2006.04779_.
  * Lample and Chaplot (2017) Lample, G.; and Chaplot, D. S. 2017.  Playing FPS Games with Deep Reinforcement Learning.  In _AAAI_.
  * Lanctot et al. (2019) Lanctot, M.; Lockhart, E.; Lespiau, J.-B.; Zambaldi, V.; Upadhyay, S.; Pérolat, J.; Srinivasan, S.; Timbers, F.; Tuyls, K.; Omidshafiei, S.; Hennes, D.; Morrill, D.; Muller, P.; Ewalds, T.; Faulkner, R.; Kramár, J.; Vylder, B. D.; Saeta, B.; Bradbury, J.; Ding, D.; Borgeaud, S.; Lai, M.; Schrittwieser, J.; Anthony, T.; Hughes, E.; Danihelka, I.; and Ryan-Davis, J. 2019\.  OpenSpiel: A Framework for Reinforcement Learning in Games.  _CoRR_ , abs/1908.09453.
  * Lecun, Bengio, and Hinton (2015) Lecun, Y.; Bengio, Y.; and Hinton, G. 2015.  Deep learning.  _nature_ , 521(7553): 436–444.
  * Lecun et al. (1998) Lecun, Y.; Eon Bottou, L.; Bengio, Y.; and Haaner, P. 1998.  Gradient-Based Learning Applied to Document Recognition.  _IEEE_ , 86(11): 2278–2324.
  * Liang et al. (2018) Liang, E.; Liaw, R.; Nishihara, R.; Moritz, P.; Fox, R.; Goldberg, K.; Gonzalez, J.; Jordan, M.; and Stoica, I. 2018.  RLlib: Abstractions for distributed reinforcement learning.  In _International Conference on Machine Learning_ , 3053–3062. PMLR.
  * Lillicrap et al. (2015) Lillicrap, T. P.; Hunt, J. J.; Pritzel, A.; Heess, N.; Erez, T.; Tassa, Y.; Silver, D.; and Wierstra, D. 2015.  Continuous control with deep reinforcement learning.  _arXiv preprint arXiv:1509.02971_.
  * Linietsky, Manzur, and Verschelde (2014) Linietsky, J.; Manzur, A.; and Verschelde, R. 2014.  Godot.  https://github.com/godotengine/godot.
  * Manolis Savva et al. (2019) Manolis Savva; Abhishek Kadian; Oleksandr Maksymets; Zhao, Y.; Wijmans, E.; Jain, B.; Straub, J.; Liu, J.; Koltun, V.; Malik, J.; Parikh, D.; and Batra, D. 2019.  Habitat: A Platform for Embodied AI Research.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_.
  * Mirowski et al. (2017) Mirowski, P.; Pascanu, R.; Viola, F.; Soyer, H.; Ballard, A. J.; Banino, A.; Denil, M.; Goroshin, R.; Sifre, L.; Kavukcuoglu, K.; Kumaran, D.; and Hadsell, R. 2017.  Learning to Navigate in Complex Environments.  In _ICLR_.
  * Mnih et al. (2016) Mnih, V.; Badia, A. P.; Mirza, M.; Graves, A.; Lillicrap, T.; Harley, T.; Silver, D.; and Kavukcuoglu, K. 2016.  Asynchronous methods for deep reinforcement learning.  In _Proceedings of the Thirty-Third International Conference on Machine Learning_ , 1928–1937. PMLR.
  * Mnih et al. (2013) Mnih, V.; Kavukcuoglu, K.; Silver, D.; Graves, A.; Antonoglou, I.; Wierstra, D.; and Riedmiller, M. 2013.  Playing atari with deep reinforcement learning.  _arXiv preprint arXiv:1312.5602_.
  * Mnih et al. (2015) Mnih, V.; Kavukcuoglu, K.; Silver, D.; Rusu, A. A.; Veness, J.; Bellemare, M. G.; Graves, A.; Riedmiller, M.; Fidjeland, A. K.; Ostrovski, G.; Petersen, S.; Beattie, C.; Sadik, A.; Antonoglou, I.; King, H.; Kumaran, D.; Wierstra, D.; Legg, S.; and Hassabis, D. 2015.  Human-level control through deep reinforcement learning.  _Nature_.
  * OpenAI (2018) OpenAI. 2018.  OpenAI Five.  https://blog.openai.com/openai-five/.
  * Pathak et al. (2017) Pathak, D.; Agrawal, P.; Efros, A. A.; and Darrell, T. 2017.  Curiosity-driven exploration by self-supervised prediction.  In _International conference on machine learning_ , 2778–2787. PMLR.
  * Petrenko et al. (2020) Petrenko, A.; Huang, Z.; Kumar, T.; Sukhatme, G.; and Koltun, V. 2020.  Sample Factory: Egocentric 3D Control from Pixels at 100000 FPS with Asynchronous Reinforcement Learning.  In _ICML_.
  * Salimans et al. (2017) Salimans, T.; Ho, J.; Chen, X.; Sidor, S.; and Sutskever, I. 2017.  Evolution strategies as a scalable alternative to reinforcement learning.  _arXiv preprint arXiv:1703.03864_.
  * Schulman et al. (2017) Schulman, J.; Wolski, F.; Dhariwal, P.; Radford, A.; and Klimov, O. 2017.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_.
  * Silver et al. (2016) Silver, D.; Huang, A.; Maddison, C. J.; Guez, A.; Sifre, L.; van den Driessche, G.; Schrittwieser, J.; Antonoglou, I.; Panneershelvam, V.; Lanctot, M.; Dieleman, S.; Grewe, D.; Nham, J.; Kalchbrenner, N.; Sutskever, I.; Lillicrap, T.; Leach, M.; Kavukcuoglu, K.; Graepel, T.; and Hassabis, D. 2016\.  Mastering the Game of Go with Deep Neural Networks and Tree Search.  _Nature_ , 529(7587): 484–489.
  * Stooke and Abbeel (2018) Stooke, A.; and Abbeel, P. 2018.  Accelerated Methods for Deep Reinforcement Learning.  _arxiv pre-print 1803.02811_.
  * Szot et al. (2021) Szot, A.; Clegg, A.; Undersander, E.; Wijmans, E.; Zhao, Y.; Turner, J.; Maestre, N.; Mukadam, M.; Chaplot, D.; Maksymets, O.; Gokaslan, A.; Vondrus, V.; Dharur, S.; Meier, F.; Galuba, W.; Chang, A.; Kira, Z.; Koltun, V.; Malik, J.; Savva, M.; and Batra, D. 2021.  Habitat 2.0: Training Home Assistants to Rearrange their Habitat.  _arXiv preprint arXiv:2106.14405_.
  * Todorov, Erez, and Tassa (2012) Todorov, E.; Erez, T.; and Tassa, Y. 2012.  Mujoco: A physics engine for model-based control.  In _Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on_ , 5026–5033. IEEE.
  * Vinyals et al. (2019) Vinyals, O.; Babuschkin, I.; Czarnecki, W.; Mathieu, M.; Dudzik, A.; Chung, J.; Choi, D.; Powell, R.; Ewalds, T.; Georgiev, P.; Oh, J.; Horgan, D.; Kroiss, M.; Danihelka, I.; Huang, A.; Sifre, L.; Cai, T.; Agapiou, J.; Jaderberg, M.; and Silver, D. 2019.  Grandmaster level in StarCraft II using multi-agent reinforcement learning.  _Nature_ , 575(7782): 350–354.
  * Wang et al. (2016) Wang, J. X.; Kurth-Nelson, Z.; Tirumala, D.; Soyer, H.; Leibo, J. Z.; Munos, R.; Blundell, C.; Kumaran, D.; and Botvinick, M. 2016.  Learning to reinforcement learn.  _arxiv pre-print 1611.05763_.
  * Wang et al. (2018) Wang, Q.; Xiong, J.; Han, L.; sun, p.; Liu, H.; and Zhang, T. 2018.  Exponentially Weighted Imitation Learning for Batched Historical Data.  In Bengio, S.; Wallach, H.; Larochelle, H.; Grauman, K.; Cesa-Bianchi, N.; and Garnett, R., eds., _Advances in Neural Information Processing Systems_ , volume 31. Curran Associates, Inc.
  * Wayne et al. (2018) Wayne, G.; Hung, C.-C.; Amos, D.; Mirza, M.; Ahuja, A.; Grabska-Barwinska, A.; Rae, J.; Mirowski, P.; Leibo, J. Z.; Santoro, A.; et al. 2018.  Unsupervised predictive memory in a goal-directed agent.  _arXiv preprint arXiv:1803.10760_.
  * Wijmans et al. (2019) Wijmans, E.; Kadian, A.; Morcos, A.; Lee, S.; Essa, I.; Parikh, D.; Savva, M.; and Batra, D. 2019.  DD-PPO: Learning near-perfect pointgoal navigators from 2.5 billion frames.  _arXiv preprint arXiv:1911.00357_.
  * Wydmuch, Kempka, and Jaśkowski (2018) Wydmuch, M.; Kempka, M.; and Jaśkowski, W. 2018.  ViZDoom Competitions: Playing Doom from Pixels.  _arXiv preprint arXiv:1809.03470_.
  * Xia et al. (2018) Xia, F.; R. Zamir, A.; He, Z.-Y.; Sax, A.; Malik, J.; and Savarese, S. 2018.  Gibson env: real-world perception for embodied agents.  In _Computer Vision and Pattern Recognition (CVPR), 2018 IEEE Conference on_. IEEE.