# The long-range interacting Fermi polaron

Krzysztof Myśliwy and Krzysztof Jachymski  Faculty of Physics, University of
Warsaw, Pasteura 5, 02-093 Warsaw, Poland

###### Abstract

We construct the simplest density functional for the problem of a single
impurity interacting with a Fermi gas via a long–ranged potential using the
Thomas–Fermi approach. We find that the Fermi polaron is fully bosonized in
two dimensions, as the model results in a suitable Landau–Pekar functional
known from the Bose polaron problem which describes a self–interacting
impurity. In other dimensions, the impurity self–interacts with an infinite
number of its own images, and no bosonization occurs. We discuss applications
of our theory for the $2d$ exciton–polaron and the ionic polaron problem and
compute the effective mass for these cases, finding a self–trapping transition
with order depending on the dimensionality.

Introduction. Polarons are quasiparticles that emerge from dressing of
impurities with the excitations of the quantum many-body medium that they are
embedded in. Their introduction was motivated mainly by modeling the motion of
electrons in solids Pekar (1946); Landau and Pekar (1948); Lee  _et al._
(1953); Fröhlich (1954). However, the idea of dressed states with modified
energy and effective mass is general and can be applied to other systems
regardless of their nature Gross (1962). Polarons are nowadays among the
central concepts in understanding quantum transport Devreese and Alexandrov
(2009). Experiments with ultracold atomic gases enabled the study of polarons
in both bosonic and fermionic media, and provided the possibility of tuning
the interaction strength via Feshbach resonances, as well as new detection
schemes which greatly advanced the field Massignan  _et al._ (2014).

One of the most intriguing aspects of the impurity problem is the calculation
of effective mass, an important quantity characterizing the polaron along with
its energy and quasiparticle residue. In the perturbative weak interaction
limit the correction to bare impurity mass is expected to be small. However,
for strong interactions, as well as for strongly correlated media, the
situation becomes drastically different and the mass renormalization can be
exponentially large and diverge with the coupling constant Rosch (1999).
Furthermore, a self–localization effect occurs in some approaches where the
effective mass becomes infinite at finite interaction strength Gross (1959);
Cucchietti and Timmermans (2006); Yakaboylu  _et al._ (2018); Seiringer
(2021). It is still under debate whether such transition can occur in
realistic polaron models.

The simplest variational Ansatz for the polaron wave function would consist of
the free particle dressed with a single medium excitation. However, the
interactions need not be perturbative Schmidt  _et al._ (2018). Full
description of the system has to take into account the possibility of forming
few- and many-body bound states and losing the quasiparticle picture, making
fully numerical treatment such as Monte Carlo methods necessary Ardila (2022).
However, field theoretical and variational approaches still bring valuable
insights into the nature of the system. One example is the impact of Efimov
trimers and larger clusters arising if one considers multiple Bogoliubov
excitations of the Bose-Einstein condensate interacting with the impurity
Levinsen  _et al._ (2015); Christianen  _et al._ (2022, 2023). Another
interesting case is when the impurity-medium interaction is attractive and
long-ranged, allowing for large density increase around the impurity. This can
be realized using ions embedded in cold gases Tomza  _et al._ (2019);
Christensen  _et al._ (2022) and in the bosonic case results in formation of
many-body bound states Astrakharchik  _et al._ (2021). Similar phenomena could
be expected for excitons in a Fermi sea of electrons, as the electron-exciton
interaction has the same nature as the ion-atom one and exciton polarons have
already been observed Verzelen  _et al._ (2002); Sidler  _et al._ (2017); Wang
_et al._ (2018); Fey  _et al._ (2020); Efimkin  _et al._ (2021).

While the polaron problem is clearly very challenging, there also exist
remarkably simple theories of Fermi gases with large predictive power. In
particular, Thomas-Fermi and density functional theories (DFT) Lieb and Simon
(1973); Hohenberg and Kohn (1964) rely on effective single-particle
description of the inhomogeneous medium and have proven to be extremely
successful. Inspired by this approach, in this work we study the Fermi polaron
problem in the regime where a DFT-like theory can be constructed, assuming the
interaction potential is slowly varying. We demonstrate that large Fermi
energy enables the series expansion of the energy functional, resulting in
effective theory with a clear and intuitive structure. Remarkably, in two
dimensions the bosonization occurs and one arrives at an analog of the seminal
bosonic Pekar functional.

The model. We consider a system of $N$ free fermions of mass $m_{F}$ and spin
$s$ and one impurity particle of mass $m_{I}$ in $d$ dimensions enclosed in a
large box of volume $L^{d}$. The impurity interacts with the fermions via a
two-body potential $V$ which we assume to be bounded and to vanish
sufficiently fast at infinity, but at the same time to be slowly varying, such
that the system can be described using the Thomas-Fermi approximation with the
functional

$\begin{split}\mathcal{H}(\psi,\rho)&=\frac{\hbar^{2}}{2m_{I}}\int|\nabla\psi(x)|^{2}\mathrm{d}x+\frac{d\epsilon_{\mathrm{F}}\rho^{-\frac{2}{d}}}{d+2}\int\rho(x)^{1+\frac{2}{d}}\mathrm{d}x\\\
&+\iint\rho(x)V(x-y)|\psi(y)|^{2}\mathrm{d}x\mathrm{d}y\end{split}$ (1)

to be minimized with the normalization constraints $\int\rho(x)\mathrm{d}x=N$
and $\qquad\int|\psi(x)|^{2}\mathrm{d}x=1$. In the above, $\rho=NL^{-d}$ is
the density of the uniform gas, and
$\epsilon_{\mathrm{F}}=\left(\frac{\Gamma(\frac{d+2}{2})}{2s+1}\right)^{\frac{2}{d}}\frac{2\pi\hbar^{2}}{m_{F}}\rho^{2/d}$
is the corresponding Fermi level, while $\psi$ is the wave function of the
impurity, and $\rho(x)$ is the local density of fermions at point $x$. This
approach appears natural in the context of long range forces: in fact, the
system described by eq. (1) is essentially a collection of infinitesimally
small boxes, each filled with the free Fermi gas with local density $\rho(x)$
having internal energy $\sim\rho(x)^{1+\frac{2}{d}}\mathrm{d}x$ due to the
Fermi pressure, and subject to an external potential with a source whose
position is smeared out due to the quantum nature of the impurity,
$\int|\psi(y)|^{2}V(x-y)\mathrm{d}y$.

We first optimize the functional in eq. (1) over $\rho$ at fixed $\psi$. This
immediately results in

$\rho_{\psi}(x)=\rho\left(\frac{\mu-
V_{\psi}(x)}{\epsilon_{\mathrm{F}}}\right)^{\frac{d}{2}}\Theta(\mu-
V_{\psi}(x))$ (2)

where

$V_{\psi}(x)=(|\psi|^{2}\ast V)(x)=\int V(x-y)|\psi(y)|^{2}\mathrm{d}y,$ (3)

$\Theta$ denotes the Heaviside theta function, and $\mu$ is the chemical
potential, satisfying

$\rho\int\left(\frac{\mu-
V_{\psi}(x)}{\epsilon_{\mathrm{F}}}\right)^{\frac{d}{2}}\Theta(\mu-
V_{\psi}(x))dx=N.$ (4)

The theta functions can be removed, provided that the number of fermions, and
hence the chemical potential, are large enough Sup . The optimized
$\rho_{\psi}$ can now be plugged back into (1), which yields a non–linear
functional of $\psi$ alone. In what follows, we shall perform this procedure
in different dimensions separately.

Bosonization in two dimensions. The resulting equations are particularly
simple for $d=2$. In fact, for fixed $N$, we can easily determine $\mu$ from
(2)

$\displaystyle\mu=\epsilon_{\mathrm{F}}+\frac{1}{L^{2}}\int
V_{\psi}(x)\mathrm{d}x=\epsilon_{\mathrm{F}}+\langle V\rangle\,,$ (5)

where $\langle V\rangle$ denotes the mean value of the potential, which is
independent of $\psi$ if the box is suitably large compared to the range of
the potential, such that the integration might be extended to infinity.

Using the resulting density profile as well as the expression for the Fermi
level $\epsilon_{\mathrm{F}}$ in two dimensions, the total energy can be
written as

$\mathcal{H}(\psi,\rho_{\psi})=E_{\mathrm{F}}\left(1+\frac{\langle
V\rangle}{\epsilon_{\mathrm{F}}}\right)^{2}+\mathcal{E}^{\rm{Pek}}(\psi)$ (6)

where $E_{\mathrm{F}}=\frac{1}{2}N\epsilon_{\mathrm{F}}$ is the ground state
energy of the uniform Fermi gas, and the _Pekar functional_
$\mathcal{E}^{\rm{Pek}}(\psi)$ reads

$\begin{split}\mathcal{E}^{\rm{Pek}}(\psi)=&\frac{\hbar^{2}}{2m}\int|\nabla\psi(x)|^{2}\mathrm{d}x\\\
&-\frac{g_{s}m_{F}}{4\pi\hbar^{2}}\iint|\psi(x)|^{2}V^{(2)}(x-y)|\psi(y)|^{2}\mathrm{d}x\mathrm{d}y.\end{split}$
(7)

where we introduced the spin degeneracy factor $g_{s}=2s+1$ as well as the
notation

$V^{(2)}(x)=V\ast V(x)=\int V(x-y)V(y)\mathrm{d}y\,.$ (8)

Eq. (7) is precisely the well–known Pekar functional describing the
semiclassical theory of impurities interacting with _bosonic_ fields Pekar
(1946). Thus at the semiclassical level, the Fermi polaron in 2d is
essentially equivalent to the Bose polaron in the same dimension Sup .

Other dimensions – the polaronic Droste effect. In dimension different than
two, eq. (2) for the chemical potential cannot be solved as easily. However, a
perturbative expansion may be applied in this case in the regime when the
local potential or its fluctuations are small, i.e. for $|V_{\psi}-\langle
V\rangle|\ll\epsilon_{\mathrm{F}}$. Then $\mu\approx\epsilon_{\mathrm{F}}$ and
one can solve for $\mu$ and find the energy by expanding the density in
$|V_{\psi}-\langle V\rangle|/\epsilon_{\mathrm{F}}$. We provide the details of
this calculation in Sup . The result is

$\begin{split}\mathcal{H}(\psi,\rho_{\psi})=&\frac{d}{d+2}N\epsilon_{\mathrm{F}}\\\
&+N\langle V\rangle\left(1+\frac{d\langle
V\rangle}{4\epsilon_{\mathrm{F}}}+\frac{(d-2)(d-4)\langle
V\rangle^{2}}{12\epsilon_{\mathrm{F}}^{2}}\right)\\\
&+\mathcal{E}_{(3)}^{\rm{Pek}}(\psi)\end{split}$ (9)

with the following generalized version of the Pekar functional:

$\displaystyle\mathcal{E}_{(3)}^{\rm{Pek}}(\psi)=$
$\displaystyle\frac{\hbar^{2}}{2m}\int|\nabla\psi(x)|^{2}\mathrm{d}x$
$\displaystyle-\frac{d}{4}\frac{\rho}{\epsilon_{\mathrm{F}}}\left(1+\frac{(d-2)(d-4)\langle
V\rangle}{2d\epsilon_{\mathrm{F}}}\right)W_{2}(\psi)$
$\displaystyle+\frac{(d-2)(d-4)}{24}\frac{\rho}{\epsilon_{\mathrm{F}}^{2}}W_{3}(\psi)$
(10)

with the already encountered two–body interaction

$W_{2}(\psi)=\int_{{\left(\mathbb{R}^{d}\right)}^{\times
2}}|\psi(x_{1})|^{2}V^{(2)}(x_{1}-x_{2})|\psi(x_{2})|^{2}$ (11)

and a newly emerging _three-body interaction_

$W_{3}(\psi)=\int_{{\left(\mathbb{R}^{d}\right)}^{\times
3}}|\psi(x_{1})|^{2}|\psi(x_{2})|^{2}|\psi(x_{3})|^{2}V^{(3)}(x_{1},x_{2},x_{3})$
(12)

with the three–body potential

$V^{(3)}(x_{1},x_{2},x_{3})=\int V(x_{1}-y)V(x_{2}-y)V(x_{3}-y)\mathrm{d}y.$
(13)

Note that if one sets $d=2$, the results of the preceding section are
recovered, although the latter are not perturbative. Moreover, in $d\neq 2$,
the density enters the functional explicitly as a parameter modifying the
coupling, while in $d=2$ the functional does not depend on the density. The
main observation is, however, that the three–body term appears which is not
present in the $2d$ Pekar functional for fermions, or for bosons in any
dimensionality. One can say that here the image of the impurity in the medium
creates its own image in turn, and all three objects interact with each other,
also via genuine three–body forces. Note that one can proceed with the
expansion procedure as above to higher orders. The inclusion of further terms
in the series in inverse powers of $\epsilon_{\mathrm{F}}$ yields a functional
with the general structure

$\displaystyle\mathcal{E}_{(\infty)}^{\rm{Pek}}(\psi)=$
$\displaystyle\frac{\hbar^{2}}{2m}\int|\nabla\psi(x)|^{2}\mathrm{d}x$
$\displaystyle+\sum_{k=1}^{\infty}\frac{\rho}{\epsilon_{\mathrm{F}}^{k}}\int_{\mathbb{R}^{d(k+1)}}G^{(k)}(x_{1},\cdots,x_{k+1})\prod_{i=1}^{k+1}|\psi(x_{i})|^{2}$
(14)

with appropriate $k+1$–body kernels $G^{(k)}(x_{1},\cdots,x_{k+1})$ depending
on $V$. In the pictorial language of Pekar’s theory, the fermionic
correlations lead to an emergence of infinite mirror images of the impurity
imprinted in the Fermi medium, resembling the self–similar structures employed
by artists. Borrowing from the fine arts language, we describe this phenomenon
as the _polaronic Droste effect_ 111see Droste effect Wikipedia article,
https://en.wikipedia.org/wiki/Droste_effect.

Effective mass and localization transition. In the next step, we are going to
study the predictions of the semiclassical theory of the Fermi polaron. Apart
from the computation of the ground state energy of the polaron, an important
quantity to estimate is its effective mass. We adopt an approach to the
effective mass problem inspired by Lieb and Seiringer (2014). The idea is to
place the impurity particle into a very shallow harmonic trap of low frequency
$\omega$, such that the Thomas–Fermi functional reads now

$\begin{split}\mathcal{H}(\psi,\rho;\omega)&=\frac{\hbar^{2}}{2m_{I}}\int|\nabla\psi(x)|^{2}\mathrm{d}x+\frac{d\epsilon_{\mathrm{F}}\rho^{-\frac{2}{d}}}{d+2}\int\rho(x)^{1+\frac{2}{d}}\mathrm{d}x\\\
&+\iint\rho(x)V(x-y)|\psi(y)|^{2}\mathrm{d}x\mathrm{d}y\\\
&+\frac{m_{I}\omega^{2}}{2}\int x^{2}|\psi(x)|^{2}\mathrm{d}x.\end{split}$
(15)

If the polaron is indeed formed and the impurity and the gas behave as one
entity with effective mass $M_{\rm{eff}}$, then the difference between the
ground state energies of the functional (15) at small $\omega$ and $\omega=0$
should be well described by the ground state energy of the hamiltonian
$\frac{\hbar^{2}}{2M_{\rm{eff}}}(-i\nabla_{x})^{2}+\frac{m_{I}\omega^{2}}{2}x^{2}$,
namely $\frac{d}{2}\hbar\omega\sqrt{\frac{m_{I}}{M_{\rm{eff}}}}$. Accordingly,
we define the effective mass as

$M_{\rm{eff}}=\lim_{\omega\rightarrow
0}\frac{d^{2}m_{I}\hbar^{2}\omega^{2}}{4(E^{\rm{TF}}(\omega)-E^{\rm{TF}}(0))^{2}}$
(16)

with $E^{\rm{TF}}(\omega)$ being the minimum energy of the functional (15),
assuming the limit exists. In other words, we expect a shift in the frequency
of oscillations of the impurity immersed in a Fermi gas as compared to the
motion in vacuum, and attribute this shift to the mass renormalization. This
directly corresponds to possible experimental effective mass measurements in
ultracold atomic setups. In practice, we are going to estimate
$E^{\rm{TF}}(\omega)-E^{\rm{TF}}(0)$ e.g. by variational methods, and fit the
results to a parabolic curve for sufficiently small $\omega$. The limit (16)
can then be obtained from the value of the fitted parameter in front of the
linear term.

As a first simple example, we choose a Gaussian–type potential

$V(x)=-V_{0}\exp\left(-\Gamma x^{2}\right)$ (17)

with the coupling amplitude $V_{0}>0$ and the inverse range squared $\Gamma$
as the underlying impurity–fermion two–body potential, and estimating the
ground state energies of the respective Pekar functionals in $d=2$ and $d=3$,
(7) and (The long-range interacting Fermi polaron) respectively, with a
one–parameter trial wave function

$\psi_{u}(x)=\left(\frac{2u}{\pi}\right)^{\frac{d}{4}}\exp\left(-ux^{2}\right),\quad
u>0.$ (18)

This case can be largely treated analytically Sup . The main conclusion here
is the presence of a sharp _localization transition_ : depending on the value
of the coupling $V_{0}$, the optimal $u$ is either zero, corresponding to an
essentially free particle, or finite, which marks the onset of a bound state.
The transition is of second order in $d=2$, with a continuous change in the
value of the optimal $u$ from zero to positive values, and of first order in
$d=3$ where the change is abrupt. In the ”polaronic” phase where $u=0$ the
effective mass is finite and lies always above the pure impurity mass,
pointing at the quasi–particle formation in this regime, whereas when the
bound state appears, the effective mass becomes infinite. In accordance with
the order of the transition in the respective dimensions, the divergence of
the effective mass as the coupling approaches the critical value necessary for
binding is either continuous in $d=2$ or sharp in $d=3$. The results are
summarized in Fig. 1-2. Note that in $3d$, the impact of the three–body terms
on the effective mass in negligible in the polaronic phase even for moderate
densities, as might be expected since in this regime $V_{\psi}\approx\langle
V\rangle$ due to the large spread of the wave function,

Let us emphasize that self–trapping of the impurity is not due to the specific
choice of the Gaussian potential or the trial wave function, but is a feature
of the non–linearity of the Pekar functional, and its occurrence for
sufficiently well-behaved potentials in $d=2$ and $d=3$ can indeed be proven
with the use of the Sobolev inequality Mys . However, we do not expect such a
transition to occur in a strict sense in real systems, and view it rather as
an enhancement of the crossover between the polaron regime and the formation
of cluster–like many-body bound state.

Figure 1: Two–dimensional inverse effective mass in units of the impurity mass
as a function of the coupling strength in units of
$2\hbar\Gamma/\sqrt{g_{s}m_{I}m_{F}}$. The mass diverges continuously at
critical coupling, and thus the self-trapping transition is of _second order_.
Figure 2: Three–dimensional inverse effective mass in units of the impurity
mass, plotted against the coupling. The fermions are fully polarized such that
$g_{s}=1$. The full lines present the results obtained from the Pekar
functional with three–body terms included, the dots indicate truncation to
two–body terms (no Droste effect). The densities are given in units of the
volume set by the potential range $\Gamma^{3/2}$. As the effective mass jumps
to infinity discontinuously at critical coupling, the trapping transition is
of _first order_.

Applications. We now apply the model to two simple and experimentally relevant
cases in which the impurity–fermion pair consists of a charge and a neutral
polarizable entity, such that the underlying two–body potential at long range
comes from electrostatic induction and decays as $~{}r^{-4}$. We use
regularized interaction with a finite depth

$V(r)=-\frac{C_{4}}{(r^{2}+b^{2})^{2}}\,,$ (19)

where $b>0$ is the regularizing length scale, while
$C_{4}=\frac{1}{2}q^{2}\alpha$ with $q$ being the charge and $\alpha$ is the
polarizability of the neutral object in question. This potential comes along
with the length and energy scales $R^{*}=\sqrt{2m_{r}C_{4}/\hbar^{2}}$ and
$E^{*}=\hbar^{2}/(2m_{r}R^{*2})$, respectively, with $m_{r}$ denoting the
reduced mass of the impurity–fermion pair. Using a Feshbach resonance it is
possible to tune experimentally the scattering length of the potential, which
we model by tuning the value of $b$.

First we study the exciton–polaron problem in $2d$, i.e. a mobile exciton
interacting with free electrons in a $2d$ layer, taking the functional (7)
with the potential (19), and using Gaussian trial functions. Assuming that the
exciton polarizability is independent of its mass and neglecting any external
potentials, the relevant parameter quantifying the coupling is given by the
ratio of the effective masses of the electron and the exciton in the material.
In Fig. 3 we present the effective mass of the exciton–polaron as a function
of the electron to exciton mass ratio at different values of the regularizing
scale $b$. As in the simple Gaussian model, we encounter a self–trapping
transition of second order.

Figure 3: The inverse effective mass of the exciton polaron as a function of
the electron to exciton mass ratio in the material, for different values of
the length scale $b$. Here, $R_{X}=\sqrt{C_{4}m_{X}}/\hbar^{2}$ with $m_{X}$
the exciton mass.

Second, we use the $3d$ Pekar functional with the potential (19) for the ionic
Fermi polaron problem, again with a Gaussian trial function and without the
inclusion of many–body terms. As we are interested principally in the
computation of the effective mass, the latter approximation should be enough
for our purpose, even at moderate densities, see Fig 2. We apply the theory
for the case of equal masses as well as to the experimentally more interesting
mass–imbalanced case of a $\mathrm{Ba}$ ion immersed in a cold gas of
${}^{6}\mathrm{Li}$ atoms Weckesser  _et al._ (2021). The energy of the equal
mass case is depicted in the lower panel of Fig. 4 and is compared to the
result of Christensen et al. Christensen  _et al._ (2022), who calculated the
polaron energy within the ladder approximation. We note that the two curves
agree perfectly in the large $b$ limit where the potential is very shallow and
mean field theory should be strict. For deeper potentials both approaches
provide qualitatively similar results. The effective mass follows the lines
found in the Gaussian model and diverges disconitunously for values of $b$
corresponding to dimer formation in the two–body problem, cf. the upper panel
of Fig 4, which marks the presence of the self–trapping transition of first
order. We remark that recent Monte–Carlo calculations of the effective mass of
the ionic polaron Ard  display an abrupt increase for values of $b$ in the
same range, validating the use of the semiclassical theory.

Figure 4: Upper: the inverse of the effective mass of the ionic polaron in the
case where the impurity and the (polarized) fermions are of equal mass. Lower:
the ground state energy of the polaron. The thin full lines are the results of
the ladder approximation at the same densities Christensen  _et al._ (2022),
which agree for $b$ large as both theories converge to the mean–field limit
there. Dimer formation in the two–body problem occurs at $b\sim 0.6R^{*}$
Christensen  _et al._ (2022). The densities are given in units of
$\left(R^{*}\right)^{-3}$.

The case of the ionic Fermi polaron with mass imbalance gives similar results,
as depicted in Fig. 5 where we plot the effective mass as a function of $b$
for the experimentally relevant value of the density $\rho=10^{15}$cm-3. For
this density, we predict a moderate mass increase of up to $3\%$ in the
polaron regime $b\geq b_{c}\approx 0.9R^{*}$ and its abrupt enhancement for
$b\leq b_{c}$.

Figure 5: Upper: the inverse of the effective mass of the ionic polaron made
of a $\rm{Ba}$ ion in a ${}^{6}\rm{Li}$ gas at density
$\rho=10^{15}\rm{cm}^{-3}$. The line terminates at self–trapping where the
inverse effective mass vanishes. Lower: the ground state energy of the ionic
polaron for the same parameters.

Summary. We have developed a simple theory of an impurity interacting with a
free Fermi gas via a two–body long–range potential. Under these conditions,
the local density approximation in the spirit of Thomas–Fermi can be applied
to the description of the Fermi gas, which results in a suitable semiclassical
energy functional depending on the fermion number density $\rho$ and the
impurity wave function $\psi$. We found that in two dimensions this leads to
the non–linear Pekar functional known from the bosonic counterpart of the
problem, describing a self–interacting impurity. This suggests that the
(long–ranged) Bose and Fermi polarons in 2D may have similar physical
properties not present in other dimensions.
In dimensions other than two, a perturbative expansion can be applied in the
quasi–free gas limit. It gives rise to a new Pekar functional with multiple
self–interactions of the impurity with its own images, whose number increases
with the order of the expansion. Inspired by terminology from the fine arts,
we refer to this phenomenon as the _polaronic Droste effect_. Nevertheless,
for the experimentally relevant scenario of the ion-atom system the images do
not significantly impact the value of the effective mass.
Interestingly, we found that a localization transition takes place in the
system, as particle either forms a polaron with renormalized mass or an
immobile many-body bound state. The presence of such transition marks a
boosted version of the smooth crossover between polarons and many–body
clusters which is likely to be found in the full quantum treatment and in the
laboratory.
Our approach can be viewed as a kind of the density functional theory for the
Fermi polaron problem in its simplest form, which already provides insight
into the physics involved. We expect our results to act as useful guidelines
for future experiments and theories to follow, in particular the measurements
of the effective mass in various dimensions and its dependence on the gas
density and the impurity–fermion mass ratio. All these parameters can easily
be varied within our approach, while the model itself remains open to further
analysis.

_Acknowledgements_. We thank R. Seiringer for helpful discussions on the
properties of the Pekar functional in $2d$, L.P. Ardila for the discussions of
his results in the equal mass case, and E. Christensen for sharing his results
in the ladder approximation. This work was supported by the National Science
Centre of Poland Grant 2020/37/B/ST2/00486 and the Polish National Agency for
Academic Exchange (NAWA) via the Polish Returns 2019 programme.

## References

  * Pekar (1946) S. Pekar, Zhurnal Eksperimentalnoi I Teoreticheskoi Fiziki 16, 341 (1946).
  * Landau and Pekar (1948) L. Landau and S. Pekar, Zh. Eksp. Teor. Fiz 18, 419 (1948).
  * Lee  _et al._ (1953) T. Lee, F. Low, and D. Pines, Phys. Rev. 90, 297 (1953).
  * Fröhlich (1954) H. Fröhlich, Advances in Physics 3, 325 (1954).
  * Gross (1962) E. Gross, Annals of Physics 19, 234 (1962).
  * Devreese and Alexandrov (2009) J. T. Devreese and A. S. Alexandrov, Reports on Progress in Physics 72, 066501 (2009).
  * Massignan  _et al._ (2014) P. Massignan, M. Zaccanti, and G. M. Bruun, Rep. Prog. Phys. 77, 034401 (2014).
  * Rosch (1999) A. Rosch, Advances in Physics 48, 295 (1999).
  * Gross (1959) E. P. Gross, Annals of Physics 8, 78 (1959).
  * Cucchietti and Timmermans (2006) F. Cucchietti and E. Timmermans, Phys. Rev. Lett. 96, 210401 (2006).
  * Yakaboylu  _et al._ (2018) E. Yakaboylu, B. Midya, A. Deuchert, N. Leopold, and M. Lemeshko, Phys. Rev. B 98, 224506 (2018).
  * Seiringer (2021) R. Seiringer, Reviews in Mathematical Physics 33, 2060012 (2021).
  * Schmidt  _et al._ (2018) R. Schmidt, M. Knap, D. A. Ivanov, J.-S. You, M. Cetina, and E. Demler, Rep. Prog. Phys. 81, 024401 (2018).
  * Ardila (2022) L. Ardila, Nature Reviews Physics 4, 214 (2022).
  * Levinsen  _et al._ (2015) J. Levinsen, M. M. Parish, and G. M. Bruun, Phys. Rev. Lett. 115, 125302 (2015).
  * Christianen  _et al._ (2022) A. Christianen, J. I. Cirac, and R. Schmidt, Phys. Rev. A 105, 053302 (2022).
  * Christianen  _et al._ (2023) A. Christianen, J. I. Cirac, and R. Schmidt, arXiv preprint arXiv:2306.09075 (2023).
  * Tomza  _et al._ (2019) M. Tomza, K. Jachymski, R. Gerritsma, A. Negretti, T. Calarco, Z. Idziaszek, and P. S. Julienne, Rev. Mod. Phys. 91, 035001 (2019).
  * Christensen  _et al._ (2022) E. R. Christensen, A. Camacho-Guardian, and G. M. Bruun, Phys. Rev. A 105, 023309 (2022).
  * Astrakharchik  _et al._ (2021) G. E. Astrakharchik, L. A. P. Ardila, R. Schmidt, K. Jachymski, and A. Negretti, Communications Physics 4, 94 (2021).
  * Verzelen  _et al._ (2002) O. Verzelen, R. Ferreira, and G. Bastard, Phys. Rev. Lett. 88, 146803 (2002).
  * Sidler  _et al._ (2017) M. Sidler, P. Back, O. Cotlet, A. Srivastava, T. Fink, M. Kroner, E. Demler, and A. Imamoglu, Nature Physics 13, 255 (2017).
  * Wang  _et al._ (2018) G. Wang, A. Chernikov, M. M. Glazov, T. F. Heinz, X. Marie, T. Amand, and B. Urbaszek, Rev. Mod. Phys. 90, 021001 (2018).
  * Fey  _et al._ (2020) C. Fey, P. Schmelcher, A. Imamoglu, and R. Schmidt, Phys. Rev. B 101, 195417 (2020).
  * Efimkin  _et al._ (2021) D. K. Efimkin, E. K. Laird, J. Levinsen, M. M. Parish, and A. H. MacDonald, Phys. Rev. B 103, 075417 (2021).
  * Lieb and Simon (1973) E. H. Lieb and B. Simon, Phys. Rev. Lett. 31, 681 (1973).
  * Hohenberg and Kohn (1964) P. Hohenberg and W. Kohn, Phys. Rev. 136, B864 (1964).
  * (28) See the Appendices, which contain additional references Tempere  _et al._ (2009); Myśliwy and Seiringer (2020); Jon ; Olaussen and Sudbø (2014); Napiórkowski and Piasecki (2017), for more detailed derivation of the effective energy functional, the discussion of semiclassical Bose polaron problem, and the analysis of the self-trapping transition.
  * Note (1) See Droste effect Wikipedia article, https://en.wikipedia.org/wiki/Droste_effect.
  * Lieb and Seiringer (2014) E. H. Lieb and R. Seiringer, Journal of Statistical Physics 154, 51 (2014).
  * (31) K. Myśliwy, unpublished.
  * Weckesser  _et al._ (2021) P. Weckesser, F. Thielemann, D. Wiater, A. Wojciechowska, L. Karpa, K. Jachymski, M. Tomza, T. Walker, and T. Schaetz, Nature 600, 429 (2021).
  * (33) L. P. Ardila, private communication.
  * Tempere  _et al._ (2009) J. Tempere, W. Casteels, M. K. Oberthaler, S. Knoop, E. Timmermans, and J. T. Devreese, Phys. Rev. B 80, 184504 (2009).
  * Myśliwy and Seiringer (2020) K. Myśliwy and R. Seiringer, Ann. Henri Poincare. 21, 1 (2020).
  * (36) J. Lampart and A. Triay, in preparation.
  * Olaussen and Sudbø (2014) K. Olaussen and A. Sudbø, Transactions of The Royal Norwegian Society of Sciences and Letters 3, 115 (2014).
  * Napiórkowski and Piasecki (2017) M. Napiórkowski and J. Piasecki, Phys. Rev. A 95, 063627 (2017).

##  Appendix A Perturbation expansion

We shall give a few more details about the perturbation expansion which leads
to the polaronic Droste effect described by the Pekar functional with
many–body terms.
As pointed out in the main text, the regime we are interested in corresponds
to situations where the chemical potential is close to
$\epsilon_{\mathrm{F}}$, which is its value for the free Fermi gas at zero
temperature. We assume that the impurity–fermion potential $V$ is bounded,
integrable and decays sufficiently fast such that all boundary effects are
negligible, in particular integrals involving $V$ may be extended to infinity.
For simplicity, we shift the interaction potential $V\rightarrow V-\langle
V\rangle$ and work with a $V$ that has mean value zero. When considering the
final result, one simply has to add a constant term $N\langle V\rangle$ to the
energy functional. Further, in order to control the expansion, we substitute
$V_{\psi}\rightarrow\lambda V_{\psi}$ where $\lambda>0$ is meant to be a small
characteristic energy scale resulting either from the weakness of the
potential $V$ or the large spread of the wave function $\psi$, in which case
$V_{\psi}$ is indeed small as we assume that $V$ has mean value zero. Our
expansion is then in the dimensionless parameter
$\lambda/\epsilon_{\mathrm{F}}$, and at the end $\lambda$ is set equal to
unity. Accordingly, we write

$\mu=\epsilon_{\mathrm{F}}\left(1+\frac{\lambda}{\epsilon_{\mathrm{F}}}\mu_{0}+\left(\frac{\lambda}{\epsilon_{\mathrm{F}}}\right)^{2}\mu_{1}+\left(\frac{\lambda}{\epsilon_{\mathrm{F}}}\right)^{3}\mu_{2}+o((\lambda/\epsilon_{\mathrm{F}})^{4})\right)$
(20)

and plug this into the formula for the optimal density profile given $\psi$,
i.e.,

$\rho_{\psi}(x)=\rho\left(\frac{\mu-\lambda
V_{\psi}}{\epsilon_{\mathrm{F}}}\right)^{\frac{d}{2}}$ (21)

(the Heaviside function can be dropped since in the assumed regime of validity
of the expansion it clearly holds that $\mu>\lambda V_{\psi}(x)$ for all $x$).
The application of the generalized binomial expansion

$(1+x)^{\alpha}=\sum_{k=0}^{\infty}\frac{\Gamma(\alpha+1)}{\Gamma(\alpha+1-k)k!}x^{k}\equiv\sum_{k=0}^{\infty}{\alpha\choose
k}x^{k}$ (22)

with $\Gamma(\cdot)$ denoting the Euler Gamma function yields

$\displaystyle\rho_{\psi}(x)=$
$\displaystyle\rho\left(1+\frac{\lambda}{\epsilon_{\mathrm{F}}}D(\mu_{0}-V_{\psi}(x))\right.$
$\displaystyle+\frac{\lambda}{\epsilon_{\mathrm{F}}}^{2}\left(D\mu_{1}+{D\choose
2}(\mu_{0}-V_{\psi}(x))^{2}\right)+$ $\displaystyle-$
$\displaystyle\left(\frac{\lambda}{\epsilon_{\mathrm{F}}}\right)^{3}\left(2{D\choose
2}\mu_{1}(\mu_{0}-V_{\psi}(x))+D\mu_{2}\right.+$
$\displaystyle\left.\left.{D\choose
3}(\mu_{0}-V_{\psi}(x))^{3}\right)+o((\lambda/\epsilon_{\mathrm{F}})^{4})\right)$

where we introduced $D=\frac{d}{2}$ for the sake of transparency. Then
$\int\rho_{\psi}(x)\mathrm{d}x-N$ is a polynomial in $\lambda$. Since
$\int\rho_{\psi}(x)\mathrm{d}x=N$, all coefficients of this polynomial have to
vanish, which leads to

$\displaystyle\mu_{0}=0$ $\displaystyle\mu_{1}=-\frac{1}{D}{D\choose
2}\frac{\int V_{\psi}(x)^{2}\mathrm{d}x}{L^{d}}$
$\displaystyle\mu_{2}=\frac{1}{D}{D\choose 3}\frac{\int
V_{\psi}(x)^{3}\mathrm{d}x}{L^{d}}.$

(Recall that $\int V_{\psi}(x)=\int V=0$ – the first equality follows from the
assumption that $V$ vanishes fast enough so that the integration might be
extended to infinity, and the second since we have shifted the mean value from
the potential at the beginning.) We plug these back into the density profile
and compute resulting energy from the functional $\mathcal{H}(\rho,\psi)$.
With the use of (22), the internal energy of the fermions is then evaluated to

$\displaystyle\frac{d}{d+2}\epsilon_{\mathrm{F}}\rho^{-\frac{2}{d}}\int\rho(x)^{1+\frac{2}{d}}\mathrm{d}x=$
$\displaystyle\frac{d}{d+2}N\epsilon_{\mathrm{F}}+\frac{\lambda^{2}}{\epsilon_{\mathrm{F}}}\frac{d}{d+2}{D^{\prime}\choose
2}D^{2}\int V_{\psi}(x)^{2}\mathrm{d}x$
$\displaystyle-\frac{\lambda^{3}}{\epsilon_{\mathrm{F}}^{2}}\left({D^{\prime}\choose
3}D^{3}+2{D\choose 2}{D^{\prime}\choose 2}D\right)\int
V_{\psi}(x)^{3}\mathrm{d}x$
$\displaystyle+o(\lambda^{4}\epsilon_{\mathrm{F}}^{-3})$

where $D^{\prime}=1+\frac{2}{d}$. The interaction energy at the same order
reads

$\displaystyle\lambda\int\rho(x)V_{\psi}(x)\mathrm{d}x=-\frac{\lambda^{2}}{\epsilon_{\mathrm{F}}}\rho
D\int V_{\psi}(x)^{2}\mathrm{d}x$
$\displaystyle+\frac{\lambda^{3}}{\epsilon_{\mathrm{F}}^{2}}\rho{D\choose
2}\int V_{\psi}(x)^{3}\mathrm{d}x+o(\lambda^{4}\epsilon_{\mathrm{F}}^{-3}).$

After simple manipulations, we arrive at the final expression

$\displaystyle\mathcal{H}(\psi,\rho_{\psi})=$
$\displaystyle\frac{\hbar^{2}}{2m_{I}}\int|\nabla\psi(x)|^{2}\mathrm{d}x$
$\displaystyle-\frac{d}{4}\frac{\rho}{\epsilon_{\mathrm{F}}}\int\tilde{V}_{\psi}(x)^{2}\mathrm{d}x$
$\displaystyle+\frac{(d-2)(d-4)}{24}\frac{\rho}{\epsilon_{\mathrm{F}}^{2}}\int\tilde{V}_{\psi}(x)^{3}\mathrm{d}x+N\langle
V\rangle$ (23)

where we have put $\lambda=1$ and denoted explicitly the shifted potential
$\tilde{V}_{\psi}=V_{\psi}-\langle V\rangle$. By noting that $\int
V_{\psi}^{2}\mathrm{d}x=W_{2}(\psi),\int V_{\psi}^{3}\mathrm{d}x=W_{3}(\psi)$,
it is straightforward to arrive at the form given in the main text. It is
evident from the procedure sketched that at higher orders in $\lambda$, many–
body terms will emerge from expressions like $\int V_{\psi}(x)^{k}\mathrm{d}x$
which appear at higher orders in the binomial expansion of the energy.

##  Appendix B Self–trapping within the Gaussian Ansatz

Now we turn to the analysis of the Pekar functional (given by Eq. (7) in the
main text) using the Gaussian–type potential which is commonly chosen as a
model of finite–ranged interaction

$V(x)=-V_{0}\exp\left(-\Gamma x^{2}\right)$ (24)

with the coupling amplitude $V_{0}>0$ and the inverse range squared $\Gamma$.

Then a simple calculation shows that

$V^{(2)}(x)=\frac{\pi
V_{0}^{2}}{2\Gamma}\exp\left(-\frac{\Gamma}{2}x^{2}\right).$ (25)

As indicated in the main text, we estimate the ground state energy of
$\mathcal{E}^{\rm{Pek}}$ using a one-parameter trial wave function

$\psi_{u}(x)=\left(\frac{2u}{\pi}\right)^{d/4}\exp\left(-ux^{2}\right),\quad
u>0.$ (26)

We start with $d=2$. The resulting energy is very easy to calculate, and reads

$E(u)\equiv\mathcal{E}^{\rm{Pek}}(\psi_{u})=\frac{\hbar^{2}}{m_{I}}u-\frac{g_{s}m_{F}V_{0}^{2}}{8\Gamma\hbar^{2}}\frac{u}{u+\frac{\Gamma}{2}}.$
(27)

Already this simple approach leads to the occurrence of a localisation
transition, i.e. the phenomenon that a bound state appears only for
sufficiently strong couplings. In fact, the positive root of the derivative of
$E(u)$ is given by

$u_{+}=\frac{1}{2}\left(\frac{V_{0}}{2\hbar}\sqrt{g_{s}m_{I}m_{F}}-\Gamma\right)$
(28)

and appears if and only if

$V_{0}\geq V_{c}=\frac{2\Gamma\hbar^{2}}{\sqrt{g_{s}m_{F}m_{I}}}.$ (29)

Otherwise, the lowest energy attainable by means of $\psi_{u}$ equals zero,
and corresponds to the ground state of the free particle in the box. Thus, for
$V_{0}<V_{c}$, the surrounding gas does not localise the particle, in contrast
to the situation encountered in Pekar’s theory of electrons in polar crystals.
At $V_{0}>V_{c}$ binding does occur, and the particle gets trapped by the
surrounding medium. If we now again take (24) as the particle–impurity
potential and define also

$E(u;\omega)=\mathcal{E}^{\rm{Pek}}(\psi_{u};\omega);\quad
E_{0}(\omega)=\inf_{u>0}E(u,\omega)$ (30)

with $\psi_{u}$ denoting the class of functions (26), then the effective mass
in the _Gaussian_ approach will be

$M^{\rm{G}}_{\rm{eff}}=\lim_{\omega\rightarrow
0}\frac{d^{2}m_{I}\hbar^{2}\omega^{2}}{4(E_{0}(\omega)-E_{0}(0))^{2}}.$ (31)

In $2d$, it is then straightforward to find

$E(u;\omega)=\frac{\hbar^{2}}{m_{I}}u-\frac{g_{s}m_{F}V_{0}^{2}}{8\Gamma\hbar^{2}}\frac{u}{u+\frac{\Gamma}{2}}+\frac{m_{I}\omega^{2}}{4u}.$
(32)

The condition for the zero of the derivative

$\frac{\hbar^{2}}{m}-\frac{g_{s}m_{F}V_{0}^{2}}{16\hbar^{2}}\frac{1}{(u+\frac{\Gamma}{2})^{2}}-\frac{m_{I}\omega^{2}}{4u^{2}}=0$
(33)

leads to the quartic equation

$\frac{\hbar^{2}}{m}u^{2}\left(u+\frac{\Gamma}{2}\right)^{2}-\frac{g_{s}m_{F}V_{0}^{2}}{16\hbar^{2}}u^{2}-\frac{m_{I}\omega^{2}}{4}\left(u+\frac{\Gamma}{2}\right)^{2}=0.$
(34)

whose unperturbed ($\omega=0$) nonnegative solutions are $u_{p}=0$ in the
unbound regime $(V_{0}<V_{c})$ and $u_{b}=u_{+}$ in the bound $(V_{0}>V_{c})$
regime. In the unbound regime, we solve (34) by treating $\omega$ as small and
by setting $u=u_{p}+\delta=\delta$, where $\delta$ the first order correction
to the unperturbed solution, and retaining only leading order terms in
$\delta$ and $\omega$. This leads to

$u=\frac{\frac{m_{I}\omega}{2\hbar}}{\sqrt{1-\left(\frac{V_{0}}{V_{c}}\right)^{2}}}+o(\omega)$
(35)

and the minimising energy at small $\omega$ reads

$E_{0}(\omega)=\sqrt{1-\left(\frac{V_{0}}{V_{c}}\right)^{2}}\hbar\omega$ (36)

so that the effective mass equals

$M^{\rm{G}}_{\rm{eff}}=\frac{m_{I}}{1-\left(\frac{V_{0}}{V_{c}}\right)^{2}}$
(37)

and diverges as the amplitude $V_{0}$ approaches the localization point. In
contrast, beyond the localization point and at $\omega$ small, it is easy to
see the lowest–order approximate solution of (34) is $b=b_{+}+\delta$ with
$\delta\sim\omega^{2}$, and consequently
$E_{0}(\omega)-E_{0}(0)\sim\omega^{2}$. The limit (31) is then infinite, and
one can say that the localized particle has infinite effective mass under the
definition adopted here.

Now we pass to the case $d=3$ and study the (pure) Pekar functional in $3d$

$\displaystyle\mathcal{E}_{(2)}^{\rm{Pek}}(\psi)$
$\displaystyle=\frac{\hbar^{2}}{2m}\int|\nabla\psi(x)|^{2}\mathrm{d}x$ (38)
$\displaystyle-\frac{3}{4}\frac{\rho_{0}}{\epsilon_{\mathrm{F}}}\iint|\psi(x_{1})|^{2}V^{(2)}(x_{1}-x_{2})|\psi(x_{2})|^{2}\mathrm{d}x_{1}\mathrm{d}x_{2}$

which, with our choice of the potential and wave function, evaluates to the
function

$E_{2}(b)=\frac{3}{2}\frac{\hbar^{2}}{m_{I}}u-\frac{3\rho
V_{0}^{2}}{4\epsilon_{\mathrm{F}}}\left(\frac{\pi}{2\Gamma}\right)^{\frac{3}{2}}\left(\frac{2u}{2u+\Gamma}\right)^{\frac{3}{2}}$
(39)

Now, the two–body Gaussian functional (39) admits a simple calculation of the
critical coupling strength for the localized state to appear as a global
minimum of the untrapped system. The condition for the zero of the derivative
of (39) at $\omega=0$ leads to the equation

$\frac{(2u)^{1/2}}{(2u+\Gamma)^{5/2}}=\frac{2\hbar^{2}\epsilon_{\mathrm{F}}}{3m_{I}V_{0}^{2}\rho\Gamma}\left(\frac{2\Gamma}{\pi}\right)^{\frac{3}{2}}$
(40)

that can be solved only if
$|V_{0}|\geq\sqrt{\frac{25\sqrt{5}}{24}\left(\frac{2}{\pi}\right)^{\frac{3}{2}}\frac{\hbar^{2}\Gamma^{\frac{5}{2}}\epsilon_{\mathrm{F}}}{m_{I}\rho}}$
which is the minimal value of $|V_{0}|$ for which _metastable_ localized
states appear. The minimal value of $|V_{0}|$ required for _stable_
localization can be found by using (40) together with the condition
$E_{2}(u)=0$, as the localized and delocalized state coexist as stable minima
at this value of $V_{0}$. This gives

$|V_{c}|=\sqrt{\left(\frac{6}{\pi}\right)^{\frac{3}{2}}\frac{\hbar^{2}\Gamma^{\frac{5}{2}}\epsilon_{\mathrm{F}}}{2m_{I}\rho}}.$
(41)

Unfortunately, the effective mass cannot be found explicitly, as the relevant
equation for small $\omega$ is a quintic one, rather than quartic as in $2d$,
and thus it does not admit a closed solution. It can be, however, computed
numerically from the slope of the energy difference $E_{0}(\omega)-E_{0}(0)$
at small frequencies.

##  Appendix C Semiclassical theory of the Bose polaron

In order to illustrate the point of our result in $d=2$, which we describe as
the bosonization of the semi–classical Fermi polaron, we recall the
corresponding results for the Bose polaron. We start with the Fröhlich
Hamiltonian describing an impurity of mass $m$ immersed in a superfluid Bose
gas

$\begin{split}\mathbb{H}=&\int\hbar
c|k|b^{\dagger}_{k}b_{k}\mathrm{d}k+\frac{\hbar^{2}}{2m}(-\nabla_{x}^{2})\\\
&+\int\left(\frac{|k|w_{k}}{\sqrt{2m_{B}\hbar
c|k|}}b_{k}e^{ikx}+\mathit{h.c.}\right)dk\\\ &\end{split}$ (42)

where $c$ is the critical velocity of the gas, $w_{k}$ is the Fourier
transform of the impurity–boson potential $W$ and $m_{B}$ is the mass of the
bosons. Such a Hamiltonian arises if one applies the Bogoliubov approximation
to the full quantum mechanical many–body problem, and is known to be
asymptotically correct in appropriate scaling regimes Tempere  _et al._
(2009); Myśliwy and Seiringer (2020); Jon . Its semiclassical theory is
provided by taking the expectation value of $\mathbb{H}$ on product states of
the type $|\psi\rangle\otimes|z\rangle$, where $\psi$ is some impurity wave
function and $|z\rangle$ is a coherent state. Minimisation over all possible
coherent states yields the functional

$\begin{split}\mathcal{E}_{B}^{\rm{Pek}}(\psi)=&\frac{\hbar^{2}}{2m}\int|\nabla\psi(x)|^{2}\mathrm{d}x\\\
&-\frac{1}{2m_{B}c^{2}}\iint|\psi(x)|^{2}W^{(2)}(x-y)|\psi(y)|^{2}\mathrm{d}x\mathrm{d}y\end{split}$
(43)

with $W^{(2)}=W\ast W$. As we see, the resulting semiclassical theory has the
precise same structure in as we found for our model of the semiclassical Fermi
polaron, provided that $d=2$. It is, in fact, known that in this particular
dimension the difference between free fermions and free bosons is least
prominent, for example their virial expansions are identical up to one term
Olaussen and Sudbø (2014). It is known that the two systems map exactly to
each other provided that suitable interactions are present Napiórkowski and
Piasecki (2017). Thus, our findings suggest that the Bose and Fermi polarons
in two dimensions should display certain similarities not present in other
dimensions.


# RotLSTM: rotating memories in Recurrent Neural Networks

Vlad Velici
School of Electronics and Computer Science
University of Southampton
Southampton, UK &Adam Prügel-Bennett
School of Electronics and Computer Science
University of Southampton
Southampton, UK

###### Abstract

Long Short-Term Memory (LSTM) units have the ability to memorise and use long-
term dependencies between inputs to generate predictions on time series data.
We introduce the concept of modifying the cell state (memory) of LSTMs using
rotation matrices parametrised by a new set of trainable weights. This
addition shows significant increases of performance on some of the tasks from
the bAbI dataset.

mAP

    mean average precision
CNN

    Convolutional Neural Network
RPN

    Region Proposal Network
RNN

    Recurrent Neural Network
RNN

    Recurrent Neural Network
LSTM

    Long Short-Term Memory
NLP

    Natural Language Processing
GRU

    Gated Recurrent Unit

##  1 Introduction

In the recent years, Recurrent Neural Networks have been successfully used to
tackle problems with data that can be represented in the shape of time series.
Application domains include Natural Language Processing (NLP) (translation
(Rosca & Breuel, 2016), summarisation (Nallapati et al., 2016), question
answering and more), speech recogition (Hannun et al., 2014; Graves et al.,
2013), text to speech systems (Arik et al., 2017), computer vision tasks
(Stewart et al., 2016; Wu et al., 2017), and differentiable programming
language interpreters (Riedel et al., 2016; Rocktäschel & Riedel, 2017).

An intuitive explanation for the success of RNNs in fields such as natural
language understanding is that they allow words at the beginning of a sentence
or paragraph to be memorised. This can be crucial to understanding the
semantic content. Thus in the phrase _"The cat ate the fish"_ it is important
to memorise the subject (cat). However, often later words can change the
meaning of a senstence in subtle ways. For example, _"The cat ate the fish,
didn’t it"_ changes a simple statement into a question. In this paper, we
study a mechanism to enhance a standard RNN to enable it to modify its memory,
with the hope that this will allow it to capture in the memory cells sequence
information using a shorter and more robust representation.

One of the most used RNN units is the Long Short-Term Memory (LSTM)
(Hochreiter & Schmidhuber, 1997). The core of the LSTM is that each unit has a
_cell state_ that is modified in a gated fashion at every time step. At a high
level, the cell state has the role of providing the neural network with
_memory_ to hold long-term relationships between inputs. There are many small
variations of LSTM units in the literature and most of them yield similar
performance (Greff et al., 2017).

The memory (cell state) is expected to encode information necessary to make
the next prediction. Currently the ability of the LSTMs to rotate and swap
memory positions is limited to what can be achieved using the available gates.
In this work we introduce a new operation on the memory that explicitly
enables rotations and swaps of pairwise memory elements. Our preliminary tests
show performance improvements on some of the bAbI tasks (Weston et al., 2015)
compared with LSTM based architectures.

##  2 Related work

Adding rotations to RNN cells has been previously studied in Henaff et al.
(2016). They do compare their results to LSTM models but do not directly add a
new rotation gates to the LSTM like in our work. Their results suggest that
rotations do not generalise well for multiple tasks.

Using unitary constraints for weight matrices in RNN units is shown in
Arjovsky et al. (2015), where they use the weights in the complex domain (they
show a way to only use real numbers in implementation). The EURNN (Jing et
al., 2017b) presents a more efficient method of parametrising a weights matrix
such that it is unitary and uses this parametrisation to add unitary
constraints to weights of RNN cells. A similar work is GORU (Jing et al.,
2017a) which adds the same unitary constraints on one of the gate weights in
the GRU cell.

A new RNN cell, the Rotational Unit of Memory (RUM) (Dangovski et al., 2017),
adds a rotation operation in a gated RNN cell. The rotation parametrisation is
passed through to the next cell, similar to how the memory state of an LSTM
is. They show increased performance on the bAbI tasks when compared to LSTM
and GORU based models.

##  3 The rotation gate

In this section we introduce the idea of adding a new set of parameters for
the RNN cell that enable rotation of the cell state. The following subsection
shows how this is implemented in the LSTM unit.

One of the key innovations of LSTMs was the introduction of gated modified
states so that if the gate neuron $i$ is saturated then the memory
$c_{i}(t-1)$ would be unaltered. That is, $c_{i}(t-1)\approx c_{i}(t)$ with
high accuracy. The fact that the amplification factor is very close to 1
prevents the memory vanishing or exploding over many epochs.

To modify the memory, but retain an amplification factor of 1 we take the
output after appling the forget and add gates (we call it $\mathbf{d_{t}}$),
and apply a rotation matrix $\mat{U}$ to obtain a modified memory
$\mathbf{c}_{t}=\mat{U}\mathbf{d}_{t}$. Note that, for a rotation matrix
$\mat{U}^{T}\mat{U}=\mat{I}$ so that
$\lVert\mathbf{d_{t}}\rVert=\lVert\mathbf{c_{t}}\rVert$.

We parametrise the rotation by a vector of angles

$\mathbf{u}=2\pi\sigma(\mat{W}_{\text{rot}}\mathbf{x}+\mathbf{b}_{\text{rot}}),$
(1)

where $\mat{W}_{\text{rot}}$ is a weight matrix and $\mathbf{b}_{\text{rot}}$
is a bias vector which we learn along with the other parameters. $\mathbf{x}$
is the vector of our concatenated inputs (in LSTMs given by concatenating the
input for the current timestep with the output from the previous time step).

A full rotation matrix is parametrisable by $n(n-1)/2$ parameters (angles).
Using all of these would introduce a huge number of weights, which is likely
to over-fit. Instead, we have limited ourselves to considering rotations
between pairs of inputs $d_{i}(t)$ and $d_{i+1}(t)$.

Our rotation matrix is a block-diagonal matrix of 2D rotations

$\mat{U}(\mathbf{u})=\begin{bmatrix}\ \cos u_{1}&-\sin u_{1}&&&\\\ \sin
u_{1}&\cos u_{1}&&&\\\ &&\ddots&&\\\ &&&\cos u_{n/2}&-\sin u_{n/2}\\\ &&&\sin
u_{n/2}&\cos u_{n/2}\ \end{bmatrix},$ (2)

where the cell state is of size $n$. Our choice of rotations only needs $n/2$
angles.

###  3.1 RotLSTM

In this section we show how to add memory rotation to the LSTM unit. The
rotation is applied after the forget and add gates and before using the
current cell state to produce an output.

The RotLSTM equations are as follows:

$\displaystyle\mathbf{x}$ $\displaystyle=[\mathbf{h_{t-1}},\mathbf{x_{t}}],$
(3) $\displaystyle\mathbf{f_{t}}$
$\displaystyle=\sigma(\mat{W}_{\text{f}}\mathbf{x}+\mathbf{b}_{\text{f}}),$
(4) $\displaystyle\mathbf{i_{t}}$
$\displaystyle=\sigma(\mat{W}_{\text{i}}\mathbf{x}+\mathbf{b}_{\text{i}}),$
(5) $\displaystyle\mathbf{o_{t}}$
$\displaystyle=\sigma(\mat{W}_{\text{o}}\mathbf{x}+\mathbf{b}_{\text{o}}),$
(6) $\displaystyle\mathbf{u_{t}}$
$\displaystyle=2\pi\sigma(\mat{W}_{\text{rot}}\mathbf{x}+\mathbf{b}_{\text{rot}}),$
(7) $\displaystyle\mathbf{d_{t}}$
$\displaystyle=\mathbf{f_{t}}\circ\mathbf{c_{t-1}}+\mathbf{i_{t}}\circ\tanh(\mat{W}_{\text{c}}\mathbf{x}+\mathbf{b}_{\text{c}}),$
(8) $\displaystyle\mathbf{c_{t}}$
$\displaystyle=\mat{U}(\mathbf{u_{t}})\mathbf{d_{t}},$ (9)
$\displaystyle\mathbf{h_{t}}$
$\displaystyle=\mathbf{o_{t}}\circ\tanh(\mathbf{c_{t}}),$ (10)

where $\mat{W_{\\{\text{f},\text{i},\text{o},\text{rot},\text{c}\\}}}$ are
weight matrices,
$\mathbf{b}_{\\{\text{f},\text{i},\text{o},\text{rot},\text{c}\\}}$ are biases
($\mat{W}$s and $\mathbf{b}$s learned during training), $\mathbf{h_{t-1}}$ is
the previous cell output, $\mathbf{h_{t}}$ is the output the cell produces for
the current timestep, similarly $\mathbf{c_{t-1}}$ and $\mathbf{c_{t}}$ are
the cell states for the previous and current timestep, $\circ$ is element-wise
multiplication and $[\cdot,\cdot]$ is concatenation. $\mat{U}$ as defined in
Equation 2, parametrised by $\mathbf{u_{t}}$. Figure 1 shows a RotLSTM unit in
detail.

Assuming cell state size $n$, input size $m$, the RotLSTM has $n(n+m)/2$ extra
parameters, a 12.5% increase (ignoring biases). Our expectation is that we can
decrease $n$ without harming performance and the rotations will enforce a
better representation for the cell state.

Figure 1: RotLSTM diagram. $\mathbf{x}$ is the concatenation of
$\mathbf{h_{t-1}}$ and $\mathbf{x_{t}}$ in the diagram (green and blue lines).
Note that this differs from a regular LSTM by the introduction of the network
producing angles $\mathbf{u_{t}}$ and the rotation module marked $\mat{U}$. In
the diagram input size is 4 and cell state size is 3.

##  4 Experiments and results

To empirically evaluate the performance of adding the rotation gate to LSTMs
we use the toy NLP dataset bAbI with 1000 samples per task. The bAbI dataset
is composed of 20 different tasks of various difficulties, starting from easy
questions based on a single supporting fact (for example: _John is in the
kitchen. Where is John? A: Kitchen_) and going to more difficult tasks of
reasoning about size (example: _The football fits in the suitcase. The box is
smaller than the football. Will the box fit in the suitcase? A: yes_) and path
finding (example: _The bathroom is south of the office. The bathroom is north
of the hallway. How do you go from the hallway to the office? A: north,
north_). A summary of all tasks is available in Table 1. We are interested in
evaluating the behaviour and performance of rotations on RNN units rather than
beating state of the art.

Table 1: bAbI dataset tasks.

# | Description | # | Description
---|---|---|---
1 | Factoid QA with one supporting fact | 11 | Basic coreference
2 | Factoid QA with two supporting facts | 12 | Conjunction
3 | Factoid QA with three supporting facts | 13 | Compound coreference
4 | Two argument relations: subject vs. object | 14 | Time manipulation
5 | Three argument relations | 15 | Basic deduction
6 | Yes/No questions | 16 | Basic induction
7 | Counting | 17 | Positional reasoning
8 | Lists/Sets | 18 | Reasoning about size
9 | Simple Negation | 19 | Path finding
10 | Indefinite Knowledge | 20 | Reasoning about agent’s motivation

We compare a model based on RotLSTM with the same model based on the
traditional LSTM. All models are trained with the same hyperparameters and we
do not perform any hyperparameter tuning apart from using the sensible
defaults provided in the Keras library and example code (Chollet et al.,
2015).

For the first experiment we train a LSTM and RotLSTM based model 10 times
using a fixed cell state size of 50. In the second experiment we train the
same models but vary the cell state size from 6 to 50 to assess whether the
rotations help our models achieve good performance with smaller state sizes.
We only choose even numbers for the cell state size to make all units go
through rotations.

###  4.1 The model architecture

The model architecture, illustrated in Figure 2, is based on the Keras example
implementation111Available at https://goo.gl/9wfzr5.. This model architecture,
empirically, shows better performance than the LSTM baseline published in
Weston et al. (2015). The input question and sentences are passed thorugh a
word embedding layer (not shared, embeddings are different for questions and
sentences). The question is fed into an RNN which produces a representation of
the question. This representation is concatenated to every word vector from
the story, which is then used as input to the second RNN. Intuitively, this
helps the second RNN (Query) to _focus_ on the important words to answer the
question. The output of the second RNN is passed to a fully connected layer
with a softmax activation of the size of the dictionary. The answer is the
word with the highest activation.

Figure 2: Model architecture for the bAbI dataset. The RNN is either LSTM or
RotLSTM.

The categorical cross-entropy loss function was used for training. All dropout
layers are dropping 30% of the nodes. The train-validation dataset split used
was 95%-5%. The optimizer used was Adam with learning rate 0.001, no decay,
$\beta_{1}=0.9$, $\beta_{2}=0.999$, $\epsilon=10^{-8}$. The training set was
randomly shuffled before every epoch. All models were trained for 40 epochs.
After every epoch the model performance was evaluated on the validation and
training sets, and every 10 epochs on the test set. We set the random seeds to
the same number for reproducibility and ran the experiments 10 times with 10
different random seeds. The source code is available at
https://github.com/vladvelici/swaplstm.

###  4.2 Results

In this subsection we compare the the performance of models based on the LSTM
and RotLSTM units on the bAbI dataset.

Table 2: Performance comparison on the bAbI dataset. Values are % average
accuracy on test set $\pm$standard deviation taken from training each model 10
times. For each of the trained models the test accuracy is taken for the epoch
with the best validation accuracy (picked from epoch numbers 1, 11, 21, 31, 40
since we only evaluated on the test set for those).

Task: | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10
---|---|---|---|---|---|---|---|---|---|---
LSTM | 49.8 $\pm$3.3 | 26.5 $\pm$3.8 | 21.1 $\pm$1.0 | 63.7 $\pm$9.4 | 33.6 $\pm$6.2 | 49.2 $\pm$0.8 | 62.7 $\pm$15.5 | 68.8 $\pm$8.3 | 63.9 $\pm$0.2 | 45.2 $\pm$2.0
RotLSTM | 52.3 $\pm$1.1 | 27.1 $\pm$1.3 | 22.4 $\pm$1.3 | 65.8 $\pm$3.6 | 55.7 $\pm$5.5 | 50.1 $\pm$1.9 | 76.7 $\pm$3.0 | 66.1 $\pm$6.2 | 61.5 $\pm$1.9 | 48.1 $\pm$1.1
Task: | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20
LSTM | 73.6 $\pm$2.2 | 74.3 $\pm$1.7 | 94.4 $\pm$0.2 | 20.7 $\pm$3.3 | 21.4 $\pm$0.5 | 48.0 $\pm$2.2 | 48.0 $\pm$0.0 | 70.3 $\pm$20.8 | 8.5 $\pm$0.6 | 87.7 $\pm$4.2
RotLSTM | 73.9 $\pm$1.2 | 76.5 $\pm$1.1 | 94.4 $\pm$0.0 | 19.9 $\pm$2.0 | 28.8 $\pm$8.8 | 46.7 $\pm$2.0 | 54.2 $\pm$3.1 | 90.5 $\pm$0.9 | 8.9 $\pm$1.1 | 89.9 $\pm$2.4

Applying rotations on the unit memory of the LSTM cell gives a slight
improvement in performance overall, and significant improvements on specific
tasks. Results are shown in Table 2. The most significant improvements are
faster convergence, as shown in Figure 3, and requiring smaller state sizes,
illustrated in Figure 4.

On tasks 1 (basic factoid), 11 (basic coreference), 12 (conjunction) and 13
(compound coreference) the RotLSTM model reaches top performance a couple of
epochs before the LSTM model consistently. The RotLSTM model also needs a
smaller cell state size, reaching top performance at state size 10 to 20 where
the LSTM needs 20 to 30. The top performance is, however, similar for both
models, with RotLSTM improving the accuracy with up to 2.5%.

The effect is observed on task 18 (reasoning about size) at a greater
magnitude where the RotLSTM reaches top performance before epoch 20, after
which it plateaus, while the LSTM model takes 40 epochs to fit the data. The
training is more stable for RotLSTM and the final accuracy is improved by 20%.
The RotLSTM reaches top performance using cell state 10 and the LSTM needs
size 40. Similar performance increase for the RotLSTM (22.1%) is observed in
task 5 (three argument relations), reaching top performance around epoch 25
and using a cell state of 50. Task 7 (counting) shows a similar behaviour with
an accuracy increase of 14% for RotLSTM.

Tasks 4 (two argument relations) and 20 (agent motivation) show quicker
learning (better performance in the early epochs) for the RotLSTM model but
both models reach their top performance after the same amount of traning. On
task 20 the RotLSTM performance reaches top accuracy using state size 10 while
the LSTM incremetally improves until using state size 40 to 50.

Signs of overfitting for the RotLSTM model can be observed more prominently
than for the LSTM model on tasks 15 (basic deduction) and 17 (positional
reasoning).

Our models, both LSTM and RotLSTM, perform poorly on tasks 2 and 3 (factoid
questions with 2 and 3 supporting facts, respectively) and 14 (time
manipulation). These problem classes are solved very well using models that
look over the input data many times and use an attention mechanism that allows
the model to focus on the relevant input sentences to answer a question
(Sukhbaatar et al., 2015; Kumar et al., 2016). Our models only look at the
input data once and we do not filter out irrelevant information.

Figure 3: Accuracy comparison on training, validation (val) and test sets over
40 epochs for LSTM and RotLSTM models. The models were trained 10 times and
shown is the average accuracy and in faded colour is the standard deviation.
Test set accuracy was computed every 10 epochs. Figure 4: Accuracy on the test
set for the LSTM and RotLSTM while varying the cell state size from 6 to 50.
The shown numbers are for the epochs with best validation set accuracy.

##  5 RotGRU

A popular and successful Recurrent Neural Network (RNN) cell is the Gated
Recurrent Unit (GRU) unit (Cho et al., 2014). Unlike the LSTM, the GRU unit
only has a cell state that can be interpreted as output when needed.
Intuitively we want the transformation matrix to enable the unit to learn more
compact, richer representations. The rotation is inserted at the _remember_
($\mathbf{r}$) gate to avoid applying a transformation directly on the
outputs. The RotGRU equations are the following:

$\displaystyle\mathbf{x}$ $\displaystyle=[\mathbf{h_{t-1}},\mathbf{x_{t}}],$
(11) $\displaystyle\mathbf{z_{t}}$
$\displaystyle=\sigma(\mat{W}_{\text{z}}\mathbf{x}+\mathbf{b}_{\text{z}}),$
(12) $\displaystyle\mathbf{d_{t}}$
$\displaystyle=\mathbf{h_{t-1}}\circ\sigma(\mat{W}_{\text{r}}\mathbf{x}+\mathbf{b}_{\text{r}}),$
(13) $\displaystyle\mat{U}$
$\displaystyle=2\pi\sigma(\mat{W}_{\text{rot}}\mathbf{x}+\mathbf{b}_{\text{rot}}),$
(14) $\displaystyle\mathbf{r_{t}}$ $\displaystyle=\mat{U}\mathbf{d_{t}}$ (15)
$\displaystyle\mathbf{\hat{h}_{t}}$
$\displaystyle=\tanh(\mat{W}_{\text{h}}[\mathbf{r_{t}},\mathbf{x_{t}}]+\mathbf{b}_{\text{h}}),$
(16) $\displaystyle\mathbf{\mathbf{h}_{t}}$
$\displaystyle=(1-\mathbf{z_{t}})\circ\mathbf{h_{t-1}}+\mathbf{z_{t}}\circ\mathbf{\hat{h}_{t}}.$
(17)

We show the results of training a RotGRU and a GRU model on the bAbI tasks in
Table 3 and Figure 5. Notice that models based on the GRU cell perform better
than the LSTM and RotLSTM cells. From this experiment we also observe that
RotGRU performs the same or slightly worse then the initial GRU cell. The
RotGRU also requires more computation and introduces more parameters, as a
result it takes longer (wall clock) time to train and run for inference.

Table 3: Performance comparison on the bAbI dataset. Values are % average
accuracy on test set $\pm$standard deviation taken from training each model 10
times. For each of the trained models the test accuracy is taken for the epoch
with the best validation accuracy (picked from epoch numbers 1, 11, 21, 31, 40
since we only evaluated on the test set for those).

Task: | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10
---|---|---|---|---|---|---|---|---|---|---
GRU | 51.3 $\pm$0.9 | 34.6 $\pm$3.8 | 21.5 $\pm$1.4 | 69.8 $\pm$1.9 | 67.1 $\pm$5.7 | 49.9 $\pm$1.0 | 79.2 $\pm$0.5 | 76.3 $\pm$1.2 | 62.6 $\pm$0.7 | 48.2 $\pm$1.1
RotGRU | 50.2 $\pm$1.3 | 35.7 $\pm$1.9 | 21.3 $\pm$1.1 | 70.8 $\pm$1.2 | 73.4 $\pm$2.6 | 50.1 $\pm$1.1 | 78.4 $\pm$2.6 | 75.9 $\pm$2.0 | 62.8 $\pm$0.4 | 48.4 $\pm$1.0
Task: | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20
GRU | 77.7 $\pm$3.3 | 76.8 $\pm$0.8 | 94.4 $\pm$0.0 | 31.4 $\pm$5.6 | 27.8 $\pm$6.7 | 48.0 $\pm$1.3 | 48.9 $\pm$1.7 | 91.1 $\pm$0.5 | 8.9 $\pm$0.8 | 92.3 $\pm$1.9
RotGRU | 79.2 $\pm$2.6 | 76.7 $\pm$0.7 | 94.3 $\pm$0.1 | 29.9 $\pm$4.7 | 23.3 $\pm$4.1 | 47.1 $\pm$3.7 | 48.6 $\pm$1.6 | 90.9 $\pm$0.8 | 8.9 $\pm$1.0 | 93.6 $\pm$2.9

Figure 5: Accuracy comparison on training, validation (val) and test sets over
40 epochs for GRU and RotGRU models. The models were trained 5 times and shown
is the average accuracy and in faded colour is the standard deviation. Test
set accuracy was computed every 10 epochs.

##  6 Discussion

A limitation of the models in our experiments is only applying pairwise 2D
rotations. Representations of past input can be larger groups of the cell
state vector, thus 2D rotations might not fully exploit the benefits of
transformations. Rotating groups of elements and multi-dimensional rotations
was not investigated, but could potentially also force the models to learn a
more structured representation of the world, similar to how forcing a model to
learn specific representations of scenes, as presented in Higgins et al.
(2017), yields semantic representations of the scene.

In this work we presented preliminary tests for adding rotations to simple
models but we only used a toy dataset. The bAbI dataset has certain advantages
such as being small thus easy to train many models on a single machine, not
having noise as it is generated from a simulation, and having a wide range of
tasks of various difficulties. However it is a toy dataset that has a very
limited vocabulary and lacks the complexity of real world datasets (noise,
inconsistencies, larger vocabularies, more complex language constructs, and so
on).

A brief exploration of the angles produced by $\mathbf{u}$ and the weight
matrix $\mat{W}_{\text{rot}}$ show that $\mathbf{u}$ does not saturate, thus
rotations are in fact applied to our cell states and do not converge to 0 (or
360 degress). A more in-depth qualitative analysis of the rotation gate might
give a better insight into what the rotations are actually achieving. Peeking
into the activations of our rotation gates could help understand the behaviour
of rotations and to what extent they help (or do not) better represent long-
term memory.

A very successful and popular mutation of the LSTM is the GRU unit (Cho et
al., 2014). The GRU only has an output as opposed to both a cell state and an
output and uses fewer gates. We showed that adding similar rotations to GRU
cells does not show the same performance improvements, and that the baseline
GRU model outperforms both LSTM and RotLSTM models in most cases.

##  7 Conclusion

We have introduced a novel gating mechanism for RNN units that enables
applying a parametrised transformation matrix to the cell state. We picked
pairwise 2D rotations as the transformation and shown how this can be added to
the popular LSTM units to create what we call RotLSTM.

We trained a simple model using RotLSTM units and compared them with the same
model based on LSTM units. We show that for the LSTM-based architectures
adding rotations has a positive impact on most bAbI tasks, making the training
require fewer epochs to achieve similar or higher accuracy. On some tasks the
RotLSTM model can use a lower dimensional cell state vector and maintain its
performance.

Significant accuracy improvements of approximatively 20% for the RotLSTM model
over the LSTM model are visible on bAbI tasks 5 (three argument relations) and
18 (reasoning about size).

The same improvements could not be replicated into the GRU cell when adding
our rotation gate, and models based on the GRU cell outperform both the LSTM
and RotLSTM on most tasks.

## References

  * Arik et al. (2017) Sercan O Arik, Mike Chrzanowski, Adam Coates, Gregory Diamos, Andrew Gibiansky, Yongguo Kang, Xian Li, John Miller, Jonathan Raiman, Shubho Sengupta, et al.  Deep voice: Real-time neural text-to-speech.  _arXiv preprint arXiv:1702.07825_ , 2017.
  * Arjovsky et al. (2015) Martín Arjovsky, Amar Shah, and Yoshua Bengio.  Unitary evolution recurrent neural networks.  _CoRR_ , abs/1511.06464, 2015.  URL http://arxiv.org/abs/1511.06464.
  * Cho et al. (2014) Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio.  Learning phrase representations using rnn encoder-decoder for statistical machine translation.  _arXiv preprint arXiv:1406.1078_ , 2014.
  * Chollet et al. (2015) François Chollet et al.  Keras.  https://github.com/fchollet/keras, 2015.
  * Dangovski et al. (2017) Rumen Dangovski, Li Jing, and Marin Soljacic.  Rotational unit of memory.  _CoRR_ , abs/1710.09537, 2017.  URL http://arxiv.org/abs/1710.09537.
  * Graves et al. (2013) Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton.  Speech recognition with deep recurrent neural networks.  In _Acoustics, speech and signal processing (icassp), 2013 ieee international conference on_ , pp. 6645–6649. IEEE, 2013.
  * Greff et al. (2017) Klaus Greff, Rupesh K Srivastava, Jan Koutník, Bas R Steunebrink, and Jürgen Schmidhuber.  Lstm: A search space odyssey.  _IEEE transactions on neural networks and learning systems_ , 2017\.
  * Hannun et al. (2014) Awni Y. Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, and Andrew Y. Ng.  Deep speech: Scaling up end-to-end speech recognition.  _CoRR_ , abs/1412.5567, 2014.  URL http://arxiv.org/abs/1412.5567.
  * Henaff et al. (2016) Mikael Henaff, Arthur Szlam, and Yann LeCun.  Orthogonal rnns and long-memory tasks.  _CoRR_ , abs/1602.06662, 2016.  URL http://arxiv.org/abs/1602.06662.
  * Higgins et al. (2017) Irina Higgins, Nicolas Sonnerat, Loic Matthey, Arka Pal, Christopher P Burgess, Matthew Botvinick, Demis Hassabis, and Alexander Lerchner.  Scan: Learning abstract hierarchical compositional visual concepts.  _arXiv preprint arXiv:1707.03389_ , 2017.
  * Hochreiter & Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  _Neural computation_ , 9(8):1735–1780, 1997.
  * Jing et al. (2017a) Li Jing, Çaglar Gülçehre, John Peurifoy, Yichen Shen, Max Tegmark, Marin Soljacic, and Yoshua Bengio.  Gated orthogonal recurrent units: On learning to forget.  _CoRR_ , abs/1706.02761, 2017a.  URL http://arxiv.org/abs/1706.02761.
  * Jing et al. (2017b) Li Jing, Yichen Shen, Tena Dubcek, John Peurifoy, Scott A. Skirlo, Yann LeCun, Max Tegmark, and Marin Soljacic.  Tunable efficient unitary neural networks (EUNN) and their application to rnns.  In Doina Precup and Yee Whye Teh (eds.), _Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017_ , volume 70 of _Proceedings of Machine Learning Research_ , pp. 1733–1741. PMLR, 2017b.  URL http://proceedings.mlr.press/v70/jing17a.html.
  * Kumar et al. (2016) Ankit Kumar, Ozan Irsoy, Peter Ondruska, Mohit Iyyer, James Bradbury, Ishaan Gulrajani, Victor Zhong, Romain Paulus, and Richard Socher.  Ask me anything: Dynamic memory networks for natural language processing.  In _International Conference on Machine Learning_ , pp. 1378–1387, 2016.
  * Nallapati et al. (2016) Ramesh Nallapati, Bing Xiang, and Bowen Zhou.  Sequence-to-sequence rnns for text summarization.  _CoRR_ , abs/1602.06023, 2016.  URL http://arxiv.org/abs/1602.06023.
  * Riedel et al. (2016) Sebastian Riedel, Matko Bosnjak, and Tim Rocktäschel.  Programming with a differentiable forth interpreter.  _CoRR, abs/1605.06640_ , 2016.
  * Rocktäschel & Riedel (2017) Tim Rocktäschel and Sebastian Riedel.  End-to-end differentiable proving.  _arXiv preprint arXiv:1705.11040_ , 2017.
  * Rosca & Breuel (2016) Mihaela Rosca and Thomas Breuel.  Sequence-to-sequence neural network models for transliteration.  _CoRR_ , abs/1610.09565, 2016.  URL http://arxiv.org/abs/1610.09565.
  * Stewart et al. (2016) Russell Stewart, Mykhaylo Andriluka, and Andrew Y Ng.  End-to-end people detection in crowded scenes.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 2325–2333, 2016.
  * Sukhbaatar et al. (2015) Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al.  End-to-end memory networks.  In _Advances in neural information processing systems_ , pp. 2440–2448, 2015.
  * Weston et al. (2015) Jason Weston, Antoine Bordes, Sumit Chopra, and Tomas Mikolov.  Towards ai-complete question answering: A set of prerequisite toy tasks.  _CoRR_ , abs/1502.05698, 2015.  URL http://arxiv.org/abs/1502.05698.
  * Wu et al. (2017) Qi Wu, Damien Teney, Peng Wang, Chunhua Shen, Anthony Dick, and Anton van den Hengel.  Visual question answering: A survey of methods and datasets.  _Computer Vision and Image Understanding_ , 2017.

  *[NLP]: Natural Language Processing
  *[RNN]: Recurrent Neural Network
  *[LSTM]: Long Short-Term Memory
  *[LSTMs]: Long Short-Term Memory
  *[GRU]: Gated Recurrent Unit