††thanks: email for correspondence<EMAIL_ADDRESS>
# Superconductivity from Luttinger surfaces: Emergent $\infty$-body SYK
physics

Chandan Setty  Department of Physics, University of Florida, Gainesville,
Florida, USA  Department of Physics and Astronomy, Rice University, Houston,
Texas, USA

###### Abstract

The pairing of two electrons on a Fermi surface due to an infinitesimal
attraction between them always results in a superconducting instability at
zero temperature ($T=0$). The equivalent question of pairing instability on a
Luttinger surface (LS) – a contour of zeros of the propagator – instead leads
to a quantum critical point (QCP) that separates a non-Fermi liquid (NFL) and
superconductor. A surprising and little understood aspect of pair fluctuations
at this QCP is that their thermodynamics maps to that of the Sachdev-Ye-Kitaev
(SYK) model in the strong coupling limit. Here, we offer a simple
justification for this mapping by demonstrating that (i) LS models share the
reparametrization symmetry of the $q\rightarrow\infty$ SYK model with $q$-body
interactions close to the LS, and (ii) the enforcement of gauge invariance
results in a $\frac{1}{\sqrt{\tau}}$ ($\tau\sim T^{-1}$) behavior of the
fluctuation propagator near the QCP, as is a feature of the fundamental SYK
fermion.

##  I Introduction

The theory of superconductivity by Bardeen, Cooper and Schrieffer Bardeen  _et
al._ (1957) relies on the existence of a Fermi surface (FS)– a contour of
poles of the single particle Green function where excitations are long-lived
and the notion of a quasiparticle is well-defined. The superconducting phase
then follows from a net attractive interaction that pairs two such
quasiparticles rendering the FS unstable. The question of whether such an
instability can exist, or for that matter be defined, on a Luttinger surface
(LS) Abrikosov  _et al._ (1965), or a contour of zeros of the propagator due
to a divergent self-energy Abrikosov  _et al._ (1965); Norman  _et al._
(1998); Essler and Tsvelik (2002); Konik  _et al._ (2006a); Yang  _et al._
(2006); Stanescu and Kotliar (2006); Hong and Phillips (2012); Scheurer  _et
al._ (2018), is less straightforward. The obstacle to such a generalization
stems from a complete breakdown of the quasiparticle concept on a LS – the
quasiparticle scattering lifetime is vanishingly small in this limit and the
degrees of freedom that constitute a “Cooper pair” are either ill-defined or
unknown due to lack of exactly solvable models. Yet, experiments suggest
umpteen examples of pairing in highly incoherent matter Taillefer (2010),
including the Cuprates where LSs play a prominent role in their normal state
phenomenology Norman  _et al._ (1998); Timusk and Statt (1999); Yoshida  _et
al._ (2006); Vishik  _et al._ (2010); He  _et al._ (2011); Yang  _et al._
(2011); Chakravarty (2010); Dzyaloshinskii (1996, 2003); Essler and Tsvelik
(2002); Konik  _et al._ (2006b); Yang  _et al._ (2006); Altshuler  _et al._
(1998); Stanescu and Kotliar (2006); Rosch (2007); Dave  _et al._ (2013);
Berthod  _et al._ (2006); Sakai  _et al._ (2009); Phillips (2006); Valenzuela
and Bascones (2007); Norman  _et al._ (2007); Vanacore  _et al._ (2018);
Stanescu  _et al._ (2007); Scheurer  _et al._ (2018)

In a recent attempt Setty (2020) to address the question above, it was
demonstrated phenomenologically that such a pairing instability can indeed
exist on a minimal model LS where the self-energy has a simple pole. Unlike
the case of a Fermi liquid, the pair susceptibility in this scenario diverges
even at zero temperature resulting in a superconductor to non-Fermi liquid
(NFL) quantum phase transition above a critical residue of the self-energy
pole. In addition to a power-law divergence of the spectral density, a
surprising new feature of the NFL phase close to the quantum critical point
(QCP) was uncovered – the pair fluctuation free energy resembles that of the
Sachdev-Ye-Kitaev (SYK) Sachdev and Ye (1993); Kitaev (2015); Maldacena and
Stanford (2016) model in the limit where the self-energy residue is much
greater than the temperature (strong coupling limit). The SYK model has
garnered recent interest (see Rosenhaus (2019) and references therein) due to
its connections to gravitational models describing black holes Kitaev (2015).
Motivated by these observations, Phillips and coworkers Phillips  _et al._
revived an exactly solvable but startlingly simple microscopic model by
Hatsugai and Kohmoto (HK) Hatsugai and Kohmoto (1992) that hosts finite
frequency LSs in the Mott phase. Upon doping the Mott insulator, they found
that the elementary pair excitations are formed by doublons and holons as
opposed to Bogoliubov quasiparticles. That such a mapping between the SYK
model and LS fluctuation thermodynamics should work so well, yielding an exact
but simple picture of pair excitations upon doping, is unsettling and can be
critiqued as coincidental – so far, there exists no symmetry based argument
invoked whatsoever to justify the mapping, nor is there a consistent analysis
that places interaction vertices and self-energies on an equal footing in
accordance with the Ward-Takahashi identity Nambu (1960); Schrieffer (2018).

It is the purpose of this paper to show that LS models, such as the one by HK,
share the symmetries of the $q\rightarrow\infty$ limit of the SYK model with a
$q$-body interaction, and that the propagator resulting from gauge-invariant
pair fluctuations on a LS is the SYK conformal Green function. Thus our work
gives a simple justification for the robustness of the mapping between pair
fluctuations on a LS and SYK models. More specifically, we demonstrate that LS
models where the self-energy has a simple pole acquire an infinite-$q$
reparametrization symmetry in the low-energy scaling limit. The proof we
provide follows along the lines of its original formulation in context of the
SYK model by the authors of Refs. Sachdev (2015); Kitaev (2015); Maldacena and
Stanford (2016). To contrast, however, we work with a self-energy that is
proportional to the single particle non-interacting Green function – a
characteristic of the HK or phenomenological models proposed in Refs. Yang
_et al._ (2006); Konik  _et al._ (2006b) where the original FS is converted
into a LS. This is a key feature of our analysis that distinguishes it from
the random matrix model (or the $q=2$ SYK model). In the latter scenario, the
self energy is proportional to a linear power of the total Green function in
the presence of random hopping matrix elements; therefore, there exist no
propagator zeros rendering the model effectively non-interacting.

A further exact gauge-invariant evaluation of the pair response from the model
LS Green function is essential for the theory to maintain charge conservation.
This treatment follows from the Ward-Takahashi Nambu (1960); Schrieffer (2018)
identity and ensures that interaction vertices and self-energies are placed on
an equal footing. Recent progress has been made in this direction for several
phenomenological models describing the Cuprate pseudo-gap Boyack  _et al._
(2016); Boyack (2017); Dai and Lee (2017); He  _et al._ (2017); Guo  _et al._
(2018). We find that gauge-invariance results in a $\frac{1}{\sqrt{\tau}}$
($\tau\sim T^{-1}$) behavior of the fluctuation propagator in the strong
coupling limit – a feature of the fundamental, particle-hole asymmetric, SYK
fermion. As a consequence, the fluctuation free-energy, computed exactly from
the gauge-invariant pair response, mimics leading order SYK fluctuations in
the strong coupling limit. Furthermore, we find that gauge-invariance enforces
the fluctuation density of states obtained from the partition function in the
static, long-wavelength limit to exhibit a $\omega^{-1}$ behavior at low
energies indicating small energy spacing in the fluctuation spectrum. In the
weak coupling limit, vertex corrections only have quantitative effects on the
phase diagram. The conclusions of our work point toward the existence of a
mapping between effective theories of fluctuations on model LSs such as the HK
model and models of gravity.

##  II Models and Results

Models: Various phenomenological Norman  _et al._ (1998); Yang  _et al._
(2006); Konik  _et al._ (2006b), microscopic Baskaran (1991); Hatsugai and
Kohmoto (1992); Essler and Tsvelik (2002); Konik  _et al._ (2006a); Stanescu
and Kotliar (2006); Rosch (2007); Sakai  _et al._ (2009); Eder  _et al._
(2011); Hong and Phillips (2012); Dave  _et al._ (2013); Scheurer  _et al._
(2018) and holographic Edalati  _et al._ (2011a, b) models have been used as
relevant starting points for describing LSs. Some of these models have been
used extensively to understand and interpret Dzyaloshinskii (1996, 2003);
Essler and Tsvelik (2002); Konik  _et al._ (2006a); Yang  _et al._ (2006);
Altshuler  _et al._ (1998); Stanescu and Kotliar (2006); Stanescu  _et al._
(2007); Rosch (2007); Dave  _et al._ (2013); Berthod  _et al._ (2006); Sakai
_et al._ (2009); Phillips (2006); Valenzuela and Bascones (2007); Norman  _et
al._ (2007); Vanacore  _et al._ (2018); Scheurer  _et al._ (2018) a variety of
experimental observations in the strongly interacting phase of the Cuprates
Timusk and Statt (1999); Yoshida  _et al._ (2006); Vishik  _et al._ (2010); He
_et al._ (2011); Yang  _et al._ (2011); Chakravarty (2010). The simplest among
them are models where the full interacting inverse Green function,
$G(\textbf{p},i\epsilon_{n})^{-1}=i\epsilon_{n}-\xi(\textbf{p})-\Sigma(\textbf{p},i\epsilon_{n})$,
contains a self-energy of the form Norman  _et al._ (1998); Yang  _et al._
(2006); Konik  _et al._ (2006b)

$\displaystyle\Sigma(\textbf{p},i\epsilon_{n})=\frac{u^{2}}{i\epsilon_{n}+\xi(\textbf{p})}.$
(1)

Here we have defined $\xi(\textbf{p})=\epsilon(\textbf{p})-\mu$ as the band
dispersion with chemical potential $\mu$, $\epsilon_{n}$ is the fermionic
frequency and $u$ sets the energy scale of the residue of the pole. The
microscopic HK Hamiltonian offers an interpretation of the parameter $u$ as a
four-body interaction term where only scattering process which conserve the
total center of mass are included. In Fourier transformed space, this
Hamiltonian takes a particularly simple form

$H_{\rm
HK}=\sum_{\textbf{k}}H_{\textbf{k}}=\sum_{\textbf{k}}\left[\xi_{\textbf{k}}(n_{\textbf{k}\uparrow}+n_{\textbf{k}\downarrow})+2u~{}n_{\textbf{k}\uparrow}\,n_{\textbf{k}\downarrow}\right],$
(2)

where $n_{\textbf{k}\sigma}$ is the number operator for a state with momentum
k and spin $\sigma$. Hence, the HK model is simply a Hubbard model in momentum
space where the interaction term commutes with the kinetic energy and is
exactly solvable. In the scenario where the interaction $2u$ is larger than
the bandwidth of the non-interacting bands, there exists a Mott gap between an
upper and lower Hubbard-like bands (although the model is too simple to
capture any dynamical spectral weight transfer effects which the Hubbard model
does).  A more recent analysis Yang (2020) of Fermi arcs and pseudo-gap in the
HK model lead to the interpretation of $u$ as the pseudo-gap order parameter
in the normal state. This is in line with expectation from previous works
Norman  _et al._ (1998); Konik  _et al._ (2006b); Yang  _et al._ (2006). The
propagator of the model Hamiltonian in Eq. 2 depends on the occupation numbers
of the upper and lower Hubbard bands, and a LS is not generally obtained at
all momenta for arbitrary chemical potential and occupation numbers. However,
when the occupancies of the upper and lower bands are equal and the chemical
potential lies between the bands, a LS with self-energy written in Eq. 1 with
a (renormalized) dispersion $\xi(\textbf{p})\rightarrow-\xi(\textbf{p})$ is
ensured at all momenta  (see Refs. Hatsugai and Kohmoto (1992); Phillips  _et
al._ (2018, ) for the self-energy in the HK model). This statement is exact
and non-perturbative and holds even when the parameter $u$ is taken to
infinity. Therefore, close to their respective LSs, the distinction between
the Green function of Eq. 2 and that corresponding to model Eq. 1 vanishes.

Reparametrization symmetry: We now take a closer look at the self-energy (Eq.
1) appearing in the total Green function $G(\textbf{p},i\epsilon_{n})$. This
equation can be rewritten as

$\displaystyle\Sigma(\textbf{p},i\epsilon_{n})=-u^{2}G_{0}(\textbf{p},-i\epsilon_{n})$
(3)

where $G_{0}(\textbf{p},i\epsilon_{n})$ is the non-interacting Green function.
As in the zero dimensional SYK model Kitaev (2015); Maldacena and Stanford
(2016), we will be interested in the low-energy scaling limit
$i\epsilon_{n}\rightarrow 0$ where one anticipates reparametrization
invariance for finite interactions. Equivalently, this corresponds to a Green
function contribution on the LS ($\xi(\textbf{p})=0$) where the self-energy
diverges and its spatial structure is washed out as $\beta
u\rightarrow\infty$. Therefore, the following equations hold in this limit

$\displaystyle G(\textbf{p},i\epsilon_{n})$ $\displaystyle\simeq$
$\displaystyle-\Sigma(\textbf{p},i\epsilon_{n})^{-1}\simeq-\frac{i\epsilon_{n}}{u^{2}}\equiv
G(i\epsilon_{n})$ (4) $\displaystyle\Sigma(\textbf{p},i\epsilon_{n})$
$\displaystyle\simeq$ $\displaystyle\frac{u^{2}}{i\epsilon_{n}}\equiv
u^{2}G_{0}(i\epsilon_{n})\equiv\Sigma(i\epsilon_{n}).$ (5)

Here we have defined the local quantities $G(i\epsilon_{n})$,
$G_{0}(i\epsilon_{n})$, and $\Sigma(i\epsilon_{n})$. Transforming into
imaginary time coordinate and invoking time translational invariance we have
the total Green function from Eq. 4

$\displaystyle G(\tau-\tau^{\prime})$ $\displaystyle=$
$\displaystyle\frac{\delta^{\prime}(\tau-\tau^{\prime})}{u^{2}}$ (6)

where $\delta^{\prime}(\tau)$ is the temporal derivative of the Dirac delta
function. We can also evaluate the imaginary time self-energy as well as the
convolution integral of
$G(\tau-\tau^{\prime})\Sigma(\tau^{\prime}-\tau^{\prime\prime})$ in the same
limit. Using Eqs. 3, 4, 5, this gives us

$\displaystyle\Sigma(\tau-\tau^{\prime})=u^{2}G_{0}(\tau-\tau^{\prime})$
$\displaystyle=$ $\displaystyle\frac{u^{2}}{2}\text{sgn}(\tau-\tau^{\prime})$
(7) $\displaystyle\int
d\tau^{\prime}G(\tau-\tau^{\prime})\Sigma(\tau^{\prime}-\tau^{\prime\prime})$
$\displaystyle=$ $\displaystyle-\delta(\tau-\tau^{\prime\prime}).$ (8)

The equations 7 and 8 above are written in proximity to the LS and have some
notable features. First, as in SYK type models, they are invariant under a
reparametrization symmetry transformation $\tau\rightarrow f(\tau)$ and

$\displaystyle G(\tau-\tau^{\prime})$ $\displaystyle\rightarrow$
$\displaystyle\left[f^{\prime}(\tau^{\prime})f^{\prime}(\tau)\right]G\left(f(\tau)-f(\tau^{\prime})\right)$
(9) $\displaystyle\Sigma(\tau-\tau^{\prime})$ $\displaystyle\rightarrow$
$\displaystyle\Sigma\left(f(\tau)-f(\tau^{\prime})\right).$ (10)

Figure 1: Fluctuation propagator in the Cooper channel (zig-zag lines) defined
through the Bethe-Salpeter equations. The black solid disk and the shaded
triangle denote the bare interaction vertex and vertex corrections arising
from electron correlations respectively. The thick black lines denote total
electron Green function.

The above scaling behavior of $G(\tau-\tau^{\prime})$ and
$\Sigma(\tau-\tau^{\prime})$ in the Eqs. 9 and 10 under reparametrization is
same as the $\Delta^{-1}\equiv q\rightarrow\infty$ limit of the $q$-body SYK
model Maldacena and Stanford (2016) provided the roles of
$G(\tau-\tau^{\prime})$ and $\Sigma(\tau-\tau^{\prime})$ are swapped (the
other choice of $\Delta^{-1}\equiv q=1$ is unphysical). This exchange is an
important feature that completes the map to the conformal limit of infinite-
body SYK model. More specifically, it highlights the duality between  self-
energies in gapped models (of the form appearing in Eq. 1) close to a LS and
propagators in CFTs. That is, a theory characterized by a propagator defined
by the self-energy of a gapped LS is conformal in the sense of the derivation
leading to Eq 10. It is notable that in the special limit of
$q\rightarrow\infty$, the SYK model also has a reparametrization invariant
propagator; however, the propagator in Eq 9 only transforms covariantly.
Similarly, the self-energy in Eq. 10 maintains the full symmetry of the
reparametrization group. This is unlike the SYK model where the self-energy
transforms only covariantly under reparametrizations for all $q\geq 2$. In
both cases, however, the transformation properties of the propagator and self-
energies act to leave Eqs. 7 and 8 reparametrization invariant. This is the
key reason why the generic form of fluctuations of the two aforementioned
duals cannot be distinguished from one another (as will be shown below).
Second, the forms of the Eqs. 7 and 8, while seemingly similar to the random
matrix model (or the $q=2$ SYK model), are not the same – in the current model
with a LS, the self-energy in Eq. 7 is proportional to the linear power of the
non-interacting Green function as opposed to the random matrix model where the
full Green function appears. This is an important distinction as the solution
of the random matrix model does not contain zeros of the Green function and is
therefore an effectively non-interacting in the presence of long-range
disorder. Finally, it can be verified that arguments leading to Eqs. 7, 8, 9
and 10 also hold for the diagonal elements of a BCS Green function with the
interaction parameter $u$ replaced by the superconducting order parameter. In
this sense, they belong to the same reparametrization symmetry class near the
LS. However, our focus in the following section is on fluctuations in the non-
superconducting state; hence, we will not have off-diagonal long range order
through anomalous contributions to the pair susceptibility. At this point, we
emphasize that while the parameter $u$ begs such an interpretation, it does
not play the role of a superconducting gap in our work. It is rather the
interaction (“Mott”) gap that exists even above the instability temperature.
From Eqs. 7 8 and 9 10, and the remarks that follow, we rightly anticipate –
further supported by the original derivation for the SYK model Maldacena and
Stanford (2016) and later for LS models with a simple pole in the self-energy
Setty (2020) – that the leading order free energy contribution from pair-
fluctuations in the NFL phase close to the QCP takes the form $-\beta F=\beta
u^{*}-\gamma\ln(\beta u^{*})$, where $u^{*}$ is the QCP, $\beta$ is inverse
temperature, and $\gamma$ is a constant that determines the fluctuation
density of states. This is demonstrated below.

Ward-Takahashi identity and exact vertex: We now set out to evaluate the
fluctuation free energy and density of states from the gauge-invariant
fluctuation propagator and pair susceptibility by approaching the QCP from the
NFL side. The Bethe-Salpeter equation for the fluctuation propagator is shown
in Fig. 1 in terms of the fully interacting pair bubble. An immediate
difficulty in evaluating the gauge-invariant pair bubble is that one requires
a knowledge of the exact fluctuation vertex in models of LSs. This is a-priori
not straightforward if one chooses to obtain the vertex directly from the
Hamiltonian, especially in LS ansatz models such as those espoused in Refs.
Yang  _et al._ (2006); Konik  _et al._ (2006b) where the underlying electronic
degrees of freedom are unspecified. However, and regardless of whether the
initial Hamiltonian is known, one can resort to the Ward-Takahashi identity
Nambu (1960); Schrieffer (2018) to obtain the full vertex provided the exact
self-energy is known. This is because once the self-energy is fixed, charge
conservation restricts the form of the vertex function. In fact this approach
has been recently advocated in several works describing phenomenological
models of the Cuprate pseudo-gap Boyack  _et al._ (2016); Boyack (2017); Dai
and Lee (2017); He  _et al._ (2017); Guo  _et al._ (2018).

Figure 2: Feynman diagrams defining the vertex corrections through the Ward
identity. The thick solid (dashed) lines are the interacting (non-interacting)
Green functions. The shaded triangle denotes vertex corrections arising from
electron correlations. The dotted lines denote a generic external momentum
transfer into and away from the pair bubble.

Since the self-energy is known exactly in our case, we can proceed with the
Ward-Takahashi identity. This identity relates the pair-vertex to the
interacting Green function. For a Matsubara frequency $iq_{n}$ and momentum
transfer q, it takes the form

$\displaystyle-
iq_{n}\Gamma_{0}(p+q,p)+\textbf{q}\cdot\mathbf{\Gamma}(p+q,p)=G^{-1}(p)-G^{-1}(p+q)$

where the vertex function
$\Gamma_{\mu}\equiv\left(\Gamma_{0},\mathbf{\Gamma}\right)$ for the charge and
current and we introduce the notation $q\equiv\left(iq_{n},\textbf{q}\right)$.
Using the definition $G(p)^{-1}=ip_{n}-\xi(\textbf{p})-\Sigma(p)$, we arrive
at an expression for the exact vertex

$\displaystyle\Gamma_{\mu}(p+q,p)$ $\displaystyle=$
$\displaystyle\gamma_{\mu}\left(1+\frac{u^{2}}{\left(ip_{n}+iq_{n}+\xi_{\textbf{p}+\textbf{q}}\right)\left(ip_{n}+\xi_{\textbf{p}}\right)}\right)$
(12) $\displaystyle=$ $\displaystyle\gamma_{\mu}\left(1\pm
u^{2}G_{0}(-p-q)G_{0}(-p)\right),$

where $\gamma_{\mu}=\left(1,\textbf{p}+\frac{\textbf{q}}{2}\right)$ is the
non-interacting vertex and the $\pm$ hold for the charge and current vertices
respectively. As we are interested in fluctuations in the non-superconducting
state, we have ignored collective mode contributions to the vertex that are
proportional to the superconducting order parameter Boyack  _et al._ (2016);
He  _et al._ (2017). We now use the charge vertex in Eq. 12 to obtain the
gauge-invariant pair susceptibility (shown diagrammatically in Fig. 2)

$\displaystyle\Pi(\textbf{q},iq_{n})$ $\displaystyle=$
$\displaystyle\frac{1}{\beta(2\pi)^{d}}\sum_{\epsilon_{n}}\int
d^{d}\textbf{p}~{}G(p+q)G(-p)\Gamma_{0}(-p,p+q)$ (14) $\displaystyle\equiv$
$\displaystyle\Pi_{0}(\textbf{q},iq_{n})+\Pi_{\Gamma}(\textbf{q},iq_{n}).$

Here we have decomposed the total susceptibility into the bare and interaction
corrected vertex terms as appearing on the right hand side in Fig. 2. Note
that we have defined the pair susceptibility above as well as in Ref. Setty
(2020) in a “symmetric” scheme (see for example Chen  _et al._ (2005) and
references therein) where both the Green functions are interacting. This
scheme leads to the $T=0$ ground state wave function described in Phillips
_et al._ . It is also interesting to seek solutions in the “asymmetric” Chen
_et al._ (2005) scheme (one interacting and the other non-interacting Green
function), which will generally yield a different ground state, but we will
not address this case here. We now proceed to evaluate the momentum integral
by making a similar decomposition

$\displaystyle I(q)$ $\displaystyle=$ $\displaystyle\int
d^{d}\textbf{p}~{}G(p+q)G(-p)\Gamma_{0}(-p,p+q)$ (15) $\displaystyle\equiv$
$\displaystyle I_{0}(q)+I_{\Gamma}(q)$ (16)

with the definitions $I_{0}(q)=\int d^{d}\textbf{p}~{}G(p+q)G(-p)$ and
$I_{\Gamma}(q)=u^{2}\int d^{d}\textbf{p}~{}G(p+q)G_{0}(-p-q)G(-p)G_{0}(p)$.
Below we solve for the total pair susceptibility in the strong coupling limit
where the interaction $u\gg T$. The effects of vertex corrections on the weak
coupling result ($u\ll T$) in Ref. Setty (2020) is only quantitative and
henceforth we ignore this case.

Figure 3:  A schematic of the $u$-$T$ phase diagram. The gray (blue) contours
are the strong coupling $\beta u\equiv\kappa\gg 1$ (weak coupling $\beta u\ll
1$) phase boundary and the red line denotes a Fermi liquid. The normal state
at strong coupling is a non-Fermi liquid. $T_{c0}$ is defined as $T_{c}(u=0)$
and $u_{c\infty}$ as $u_{c}(\beta\rightarrow\infty)$ (green dot).

Strongly coupled ($\beta u\gg 1$) fluctuations: In order to study the pairing
instability, we are primarily interested in deriving the static, long-
wavelength limit of the fluctuation propagator. Hence, we take the limiting
conditions $iq_{n}\rightarrow 0$ and $r\equiv\frac{p_{f}|\textbf{q}|}{m}\ll
u$, where $m$ and $p_{f}$ are the mass and Fermi momentum from the non-
interacting electronic dispersion respectively. In this regime, the expression
for $\Pi_{0}(\textbf{q},iq_{n})$ to second order in $r$ takes the form Setty
(2020) (odd powers in $r$ vanish due to the angular integral)

$\displaystyle\Pi_{0}(\textbf{q},iq_{n}\rightarrow 0)$ $\displaystyle\simeq$
$\displaystyle\Pi^{(0)}_{0}(0,0)+\Pi^{(2)}_{0}(\textbf{q},0),$ (17)
$\displaystyle\Pi^{(0)}_{0}(0,0)$ $\displaystyle=$
$\displaystyle\frac{m}{4}\left(2S_{1}-u^{2}S_{3}\right)$
$\displaystyle\Pi^{(2)}_{0}(\textbf{q},0)$ $\displaystyle=$
$\displaystyle-\frac{mr^{2}}{32}\left(2S_{3}-u^{2}S_{5}\right),$

where
$S_{\nu}=\frac{1}{\beta}\sum_{\epsilon_{n}}(\epsilon_{n}^{2}+u^{2})^{-\nu/2}$
with $\nu$ an odd integer. It should be noted that, for each order in $r$ that
is non-vanishing, there is a term proportional to the residue $u^{2}$.
Performing the small momentum expansion and using the same procedure described
in Ref. Setty (2020), on can similarly obtain the pair susceptibility from the
vertex correction term given by $\Pi_{\Gamma}(\textbf{q},iq_{n}\rightarrow
0)\simeq\Pi^{(0)}_{\Gamma}(0,0)+\Pi^{(2)}_{\Gamma}(\textbf{q},0)$ where

$\displaystyle\Pi^{(0)}_{\Gamma}(0,0)$ $\displaystyle=$
$\displaystyle\frac{mu^{2}}{4}S_{3},$ (18)
$\displaystyle\Pi^{(2)}_{\Gamma}(\textbf{q},0)$ $\displaystyle=$
$\displaystyle-\frac{mu^{2}r^{2}}{32}S_{5}.$

In this case, however, only terms proportional to $u^{2}$ survive and have the
opposite sign compared to those appearing in the bare susceptibility in Eqs.
17. Substituting the vertex correction terms back into the full expression for
the pair susceptibility
$\Pi(\textbf{q},iq_{n})\equiv\Pi_{0}(\textbf{q},iq_{n})+\Pi_{\Gamma}(\textbf{q},iq_{n})$,
the terms proportional to $u^{2}$ from the bare susceptibility and vertex
corrections cancel. The remaining Matsubara summations $S_{\nu}$ can be
performed exactly, and we obtain the strong coupling ($\beta u\gg 1$), static,
long-wavelength limit of the inverse fluctuation propagator, denoted
$L^{-1}(\textbf{q},iq_{n}\rightarrow 0)$, from Fig. 1 as (see Ref. Setty
(2020) for further details)

$\displaystyle L^{-1}(\textbf{q},iq_{n}\rightarrow 0)$ $\displaystyle=$
$\displaystyle-g^{-1}+N_{0}\left[\ln\frac{\Lambda}{u}-\frac{\sqrt{2\pi}~{}e^{-\beta
u}}{\sqrt{\beta u}}\right]$ (19) $\displaystyle-$
$\displaystyle\frac{N_{0}r^{2}}{8u^{2}}\left(1-\sqrt{2\pi\beta u}~{}e^{-\beta
u}\right).$

Here $g$ is the bare superconducting interaction, $N_{0}$ is the density of
states of the non-interacting FS and $\Lambda$ is the Matsubara cut-off for
the summation when $\nu=1$. We can now derive the QCP separating the
superconducting and NFL phases by setting $\beta u\rightarrow\infty$ and
seeking a critical $u$ for which the fluctuation propagator diverges. This
gives us $u^{*}\equiv u_{c\infty}=\Lambda e^{-1/N_{0}g}$, and the contour of
instability for low but non-zero temperatures is shown in the phase diagram in
Fig. 3. A notable feature of the strongly coupled superconductor obtained here
and in Ref. Setty (2020) is that it results from a gapped LS for a finite
interaction $g$. That is, and unlike the BCS case, pairing in the current
model is made possible by an attractive interaction $g$ that is above a
critical value set by the formula $u=\Lambda e^{-1/N_{0}g}$. Hence, doping
does not play a crucial role in this problem. This must be contrasted with
quantum critical BCS superconductors obtained from pairing gapless conformal
fermions She and Zaanen (2009); She  _et al._ (2011) or mediated by quantum
critical bosons Moon and Chubukov (2010). Fluctuation properties close to the
QCP can be further extracted by setting $u=u_{c\infty}$ in the fluctuation
propagator in Eq. 19 which yields

$\displaystyle L^{-1}\left(\textbf{q}\rightarrow
0,iq_{n}=0\right)|_{u\rightarrow u_{c\infty}}\simeq N_{0}\frac{e^{-\beta
u_{c\infty}}}{\sqrt{\beta u_{c\infty}}}.$ (20)

It is worthwhile to examine the consequences of Eq. 20 obtained by respecting
the Ward identity. First, while gauge invariance leaves the QCP unaffected in
comparison with the non-gauge invariant approximation Setty (2020), it
modifies the exponent of the $\beta u_{c\infty}$ factor in front of the
exponential in Eq. 20 from $\frac{1}{2}$ to $-\frac{1}{2}$. Therefore, the
structure of the fluctuation propagator near zero temperature in Eq. 20 takes
a form similar to the particle-hole asymmetric conformal propagator discussed
in Patel and Sachdev (2019) with the substitution $T^{-1}\rightarrow\tau$. The
$\frac{1}{\sqrt{\tau}}$ dependence is a signature of NFL transport and has
been used to describe Patel and Sachdev (2019); Sachdev (2015) Planckian
behavior Legros  _et al._ (2019); Nakajima  _et al._ (2019) and universal
scattering rates Bruin  _et al._ (2013) observed in a variety of strongly
correlated materials. Second, following the methods described in Larkin and
Varlamov (2005); Varlamov  _et al._ (2018), one can evaluate the fluctuation
free energy from Eq. 20 to obtain $-\beta F=\beta u_{c\infty}-\gamma\ln(\beta
u_{c\infty})$ with $\gamma=-\frac{1}{2}$. Thus, despite a change in sign of
the coefficient $\gamma$ when compared to the non-gauge invariant analysis,
the form of the free energy remains the same with the inclusion of vertex
corrections, as was anticipated in Ref. Setty (2020), and takes the form of
leading order $[O(N^{0})]$ SYK fluctuation terms Maldacena and Stanford
(2016). A further Laplace transform yields a fluctuation density of states
proportional to $\frac{1}{\omega}$ at low energy. In the non-gauge invariant
calculation, one instead obtains a weaker density of states divergence
$\frac{1}{\sqrt{\omega}}$. Finally, vertex corrections render a negative
fluctuation correction to the logarithmically divergent term in Eq. 19.
Therefore, the curvature of the strong coupling phase diagram is reversed at
finite temperatures away from the QCP (see Fig. 3).

##  III Discussion

Despite the fluctuation free energy acquiring the same form as the
corresponding leading order $O(N^{0})$ fluctuation contribution of the SYK
model, the constant $\gamma$ determining the fluctuation density of states is
different in the two cases. This should not come as a surprise since the
leading order fluctuation terms for the SYK model are melon diagrams as
opposed to pair bubbles that traditionally appear in the theory of fluctuation
superconductivity. As emphasized earlier, the self-energy in LS models is
proportional to a linear power of the non-interacting Green function (see Eq.
3). This is an important property of LSs that is crucial for the symmetry
mapping to the $q\rightarrow\infty$ SYK model to work. If the self-energy was,
instead, proportional to the total Green function (or the $q$=2 SYK), the
model would effectively become the non-interacting random matrix model with no
LSs. It is interesting that the form of the gauge-invariant fluctuation
propagator near the QCP in Eq. 20 has a $\frac{1}{\sqrt{\tau}}$ dependence
with the time scale $\tau$ set by $T^{-1}$. Ref. Patel and Sachdev (2019) used
such a form of the conformal propagator and showed that resonant processes
produce Planckian scattering rates Legros  _et al._ (2019); Nakajima  _et al._
(2019) with universal coefficients Bruin  _et al._ (2013) independent of
interactions. This work, therefore, motivates an evaluation of fluctuation
transport quantities such as paraconductivity in the NFL phase close to the
QCP using the Larkin-Varlamov formalism Larkin and Varlamov (2005); Setty
(2019). It is also interesting to ask whether universality of the butterfly
velocity or “information screening length” Baggioli  _et al._ (2018) at the
QCP holds in the context of Luttinger surfaces. Finally, in the HK model, the
robustness of LSs depends crucially on the ratio of interaction parameter $u$
(or the Mott gap) and bandwidth $W$ of the non-interacting bands Phillips  _et
al._ . If $W>2u$, the LS exists only in certain parts of the Brillouin zone.
Same holds true if $2u>W$ and the system is doped so that the chemical
potential is located in one of the Hubbard bands. In either of these cases,
one still has extensively many maps to the $q$$\rightarrow\infty$ SYK model –
one for each momentum point where the LS exists. However, the QCP is avoided
and the form of the free-energy mapping between the models is lost.

To conclude, we have shown there exists a low-energy reparametrization
symmetry in models which host LSs where the self-energy has a simple pole. The
transformation properties of the Green function and self-energy can be mapped
to the $q$$\rightarrow\infty$ limit of the SYK model. The corresponding
mapping of the fluctuation action is robust to inclusion of interaction
vertices through the Ward identity, and the subsequent $\frac{1}{\sqrt{\beta
u_{c\infty}}}$ behavior of the fluctuation propagator indicates NFL transport.
A LS model of particular interest is the microscopic model by HK Hatsugai and
Kohmoto (1992). In addition to the absence of random interactions, a key
simplification of the HK Hamiltonian in Eq. 2 is that the interaction terms
commute with the kinetic energy making it exactly solvable Hatsugai and
Kohmoto (1992); Phillips  _et al._ (2018, ). Despite this simplicity, the
model is sufficient to capture important physical phenomena such as LSs, Mott
gap and doublon-holon ”Cooper” pairing. Moreover, the model is not restricted
to zero dimensions and can be extended to any higher dimensions. Thus the HK
model does exactly what is expected of any minimal model – strip down the full
interacting problem to its basic ingredients for describing the most
interesting physics. Therefore the problem of pairing instabilities in model
LSs, such as those realized in simple microscopic models like HK, lays a firm
groundwork toward understanding more sophisticated models exhibiting conformal
field theory–gravity duality Zaanen  _et al._ (2015) and is worth further
exploration.

Acknowledgements: We thank P. W. Phillips, G. La Nave and L. Yeo for critical
comments. This work is supported by the DOE grant number DE-FG02-05ER46236.

## References

  * Bardeen  _et al._ (1957) J. Bardeen, L. N. Cooper, and J. R. Schrieffer, “Theory of superconductivity,” Phys. Rev. 108, 1175–1204 (1957).
  * Abrikosov  _et al._ (1965) Aleksei Alekseevich Abrikosov, Lev Petrovich Gorkov, and Igor Ekhielevich Dzyaloshinskii,  _Quantum field theoretical methods in statistical physics_ , Vol. 4 (Pergamon, 1965).
  * Norman  _et al._ (1998) M. R. Norman, M. Randeria, H. Ding, and J. C. Campuzano, “Phenomenology of the low-energy spectral function in high-${T}_{c}$ superconductors,” Phys. Rev. B 57, R11093–R11096 (1998).
  * Essler and Tsvelik (2002) Fabian HL Essler and Alexei M Tsvelik, “Weakly coupled one-dimensional mott insulators,” Physical Review B 65, 115117 (2002).
  * Konik  _et al._ (2006a) RM Konik, TM Rice, and AM Tsvelik, “Doped spin liquid: Luttinger sum rule and low temperature order,” Physical review letters 96, 086407 (2006a).
  * Yang  _et al._ (2006) Kai-Yu Yang, TM Rice, and Fu-Chun Zhang, “Phenomenological theory of the pseudogap state,” Physical Review B 73, 174501 (2006).
  * Stanescu and Kotliar (2006) Tudor D Stanescu and Gabriel Kotliar, “Fermi arcs and hidden zeros of the green function in the pseudogap state,” Physical Review B 74, 125110 (2006).
  * Hong and Phillips (2012) Seungmin Hong and Philip Phillips, “Towards the standard model for fermi arcs from a wilsonian reduction of the hubbard model,” Physical Review B 86, 115118 (2012).
  * Scheurer  _et al._ (2018) Mathias S Scheurer, Shubhayu Chatterjee, Wei Wu, Michel Ferrero, Antoine Georges, and Subir Sachdev, “Topological order in the pseudogap metal,” Proceedings of the National Academy of Sciences 115, E3665–E3672 (2018).
  * Taillefer (2010) Louis Taillefer, “Scattering and pairing in cuprate superconductors,” Annu. Rev. Condens. Matter Phys. 1, 51–70 (2010).
  * Timusk and Statt (1999) Tom Timusk and Bryan Statt, “The pseudogap in high-temperature superconductors: an experimental survey,” Reports on Progress in Physics 62, 61 (1999).
  * Yoshida  _et al._ (2006) T Yoshida, XJ Zhou, K Tanaka, WL Yang, Z Hussain, Z-X Shen, A Fujimori, S Sahrakorpi, M Lindroos, RS Markiewicz, _et al._ , “Systematic doping evolution of the underlying fermi surface of la 2- x sr x cu o 4,” Physical Review B 74, 224510 (2006).
  * Vishik  _et al._ (2010) IM Vishik, WS Lee, RH He, M Hashimoto, Z Hussain, TP Devereaux, and ZX Shen, “Arpes studies of cuprate fermiology: superconductivity, pseudogap and quasiparticle dynamics,” New Journal of Physics 12, 105008 (2010).
  * He  _et al._ (2011) Rui-Hua He, XJ Zhou, M Hashimoto, T Yoshida, K Tanaka, SK Mo, T Sasagawa, N Mannella, W Meevasana, Hong Yao, _et al._ , “Doping dependence of the ($\pi$, $\pi$) shadow band in la-based cuprates studied by angle-resolved photoemission spectroscopy,” New Journal of Physics 13, 013031 (2011).
  * Yang  _et al._ (2011) H.-B. Yang, J. D. Rameau, Z.-H. Pan, G. D. Gu, P. D. Johnson, H. Claus, D. G. Hinks, and T. E. Kidd, “Reconstructed fermi surface of underdoped ${\mathrm{bi}}_{2}{\mathrm{sr}}_{2}{\mathrm{cacu}}_{2}{\mathrm{o}}_{8+\delta}$ cuprate superconductors,” Phys. Rev. Lett. 107, 047003 (2011).
  * Chakravarty (2010) Sudip Chakravarty, “Key issues in theories of high temperature superconductors,” arXiv preprint arXiv:1006.4180 (2010).
  * Dzyaloshinskii (1996) Igor Dzyaloshinskii, “Extended van-hove singularity and related non-fermi liquids,” Journal de Physique I 6, 119–135 (1996).
  * Dzyaloshinskii (2003) Igor Dzyaloshinskii, “Some consequences of the luttinger theorem: The luttinger surfaces in non-fermi liquids and mott insulators,” Physical Review B 68, 085113 (2003).
  * Konik  _et al._ (2006b) RM Konik, TM Rice, and AM Tsvelik, “Doped spin liquid: Luttinger sum rule and low temperature order,” Physical review letters 96, 086407 (2006b).
  * Altshuler  _et al._ (1998) BL Altshuler, AV Chubukov, A Dashevskii, AM Finkel’stein, and DK Morr, “Luttinger theorem for a spin-density-wave state,” EPL (Europhysics Letters) 41, 401 (1998).
  * Rosch (2007) A Rosch, “Breakdown of luttinger’s theorem in two-orbital mott insulators,” The European Physical Journal B 59, 495–502 (2007).
  * Dave  _et al._ (2013) Kiaran B Dave, Philip W Phillips, and Charles L Kane, “Absence of luttinger’s theorem due to zeros in the single-particle green function,” Physical review letters 110, 090403 (2013).
  * Berthod  _et al._ (2006) Christophe Berthod, Thierry Giamarchi, S Biermann, and Antoine Georges, “Breakup of the fermi surface near the mott transition in low-dimensional systems,” Physical review letters 97, 136401 (2006).
  * Sakai  _et al._ (2009) Shiro Sakai, Yukitoshi Motome, and Masatoshi Imada, “Evolution of electronic structure of doped mott insulators: Reconstruction of poles and zeros of green?s function,” Physical review letters 102, 056404 (2009).
  * Phillips (2006) Philip Phillips, “Mottness,” Annals of Physics 321, 1634–1650 (2006).
  * Valenzuela and Bascones (2007) B Valenzuela and Elena Bascones, “Phenomenological description of the two energy scales in underdoped cuprate superconductors,” Physical review letters 98, 227002 (2007).
  * Norman  _et al._ (2007) MR Norman, A Kanigel, M Randeria, U Chatterjee, and JC Campuzano, “Modeling the fermi arc in underdoped cuprates,” Physical Review B 76, 174501 (2007).
  * Vanacore  _et al._ (2018) Garrett Vanacore, Srinidhi T Ramamurthy, and Philip W Phillips, “Evolution of holographic fermi arcs from a mott insulator,” Journal of High Energy Physics 2018, 9 (2018).
  * Stanescu  _et al._ (2007) Tudor D Stanescu, Philip Phillips, and Ting-Pong Choy, “Theory of the luttinger surface in doped mott insulators,” Physical Review B 75, 104503 (2007).
  * Setty (2020) Chandan Setty, “Pairing instability on a luttinger surface: A non-fermi liquid to superconductor transition and its sachdev-ye-kitaev dual,” Physical Review B 101, 184506 (2020).
  * Sachdev and Ye (1993) Subir Sachdev and Jinwu Ye, “Gapless spin-fluid ground state in a random quantum heisenberg magnet,” Physical review letters 70, 3339 (1993).
  * Kitaev (2015) A Kitaev, http://online.kitp.ucsb.edu/online/entangled15/kitaev/ (2015).
  * Maldacena and Stanford (2016) Juan Maldacena and Douglas Stanford, “Remarks on the sachdev-ye-kitaev model,” Physical Review D 94, 106002 (2016).
  * Rosenhaus (2019) Vladimir Rosenhaus, “An introduction to the syk model,” Journal of Physics A: Mathematical and Theoretical 52, 323001 (2019).
  * (35) Philip W Phillips, Luke Yeo, and Edwin W Huang, “Exact superconducting instability in a doped mott insulator,” arXiv:1912.01008 (To appear Nat. Phys) .
  * Hatsugai and Kohmoto (1992) Yasuhiro Hatsugai and Mahito Kohmoto, “Exactly solvable model of correlated lattice electrons in any dimensions,” Journal of the Physical Society of Japan 61, 2056–2069 (1992).
  * Nambu (1960) Yoichiro Nambu, “Quasi-particles and gauge invariance in the theory of superconductivity,” Physical Review 117, 648 (1960).
  * Schrieffer (2018) J Robert Schrieffer,  _Theory of superconductivity_ (CRC Press, 2018).
  * Sachdev (2015) Subir Sachdev, “Bekenstein-hawking entropy and strange metals,” Physical Review X 5, 041025 (2015).
  * Boyack  _et al._ (2016) Rufus Boyack, Brandon M Anderson, Chien-Te Wu, and K Levin, “Gauge-invariant theories of linear response for strongly correlated superconductors,” Physical Review B 94, 094508 (2016).
  * Boyack (2017) Rufus M Boyack,  _Establishing a Consistent Theory of Transport in Strongly Correlated Fermi Superfluids_ (The University of Chicago, 2017).
  * Dai and Lee (2017) Zhehao Dai and Patrick A Lee, “Optical conductivity from pair density waves,” Physical Review B 95, 014506 (2017).
  * He  _et al._ (2017) Yan He, Yan-Xiao Wang, and Hao Guo, “Establishing gauge invariant linear response of fermionic superfluids with pair fluctuations: A diagrammatic approach,” Physics Letters A 381, 1603–1610 (2017).
  * Guo  _et al._ (2018) Hao Guo, Yan He, and Lianyi He, “Dynamic density structure factor of a unitary fermi gas at finite temperature,” Journal of Physics Communications 2, 045008 (2018).
  * Baskaran (1991) G Baskaran, “An exactly solvable fermion model: Spinons, holons and a non-fermi liquid phase,” Modern Physics Letters B 5, 643–649 (1991).
  * Eder  _et al._ (2011) R Eder, K Seki, and Y Ohta, “Self-energy and fermi surface of the two-dimensional hubbard model,” Physical Review B 83, 205137 (2011).
  * Edalati  _et al._ (2011a) Mohammad Edalati, Robert G Leigh, and Philip W Phillips, “Dynamically generated mott gap from holography,” Physical review letters 106, 091602 (2011a).
  * Edalati  _et al._ (2011b) Mohammad Edalati, Robert G Leigh, Ka Wai Lo, and Philip W Phillips, “Dynamical gap and cupratelike physics from holography,” Physical Review D 83, 046012 (2011b).
  * Yang (2020) Kun Yang, “An exactly solvable model of fermi arcs and pseudogap,” arXiv preprint arXiv:2011.01680 (2020).
  * Phillips  _et al._ (2018) Philip W Phillips, Chandan Setty, and Shuyi Zhang, “Absence of a charge diffusion pole at finite energies in an exactly solvable interacting flat-band model in d dimensions,” Physical Review B 97, 195102 (2018).
  * Chen  _et al._ (2005) Qijin Chen, Jelena Stajic, Shina Tan, and Kathryn Levin, “Bcs–bec crossover: From high temperature superconductors to ultracold superfluids,” Physics Reports 412, 1–88 (2005).
  * She and Zaanen (2009) Jian-Huang She and Jan Zaanen, “Bcs superconductivity in quantum critical metals,” Physical Review B 80, 184518 (2009).
  * She  _et al._ (2011) J-H She, Bas J Overbosch, Y-W Sun, Yan Liu, KE Schalm, John A Mydosh, and Jan Zaanen, “Observing the origin of superconductivity in quantum critical metals,” Physical Review B 84, 144527 (2011).
  * Moon and Chubukov (2010) Eun-Gook Moon and Andrey Chubukov, “Quantum-critical pairing with varying exponents,” Journal of Low Temperature Physics 161, 263–281 (2010).
  * Patel and Sachdev (2019) Aavishkar A Patel and Subir Sachdev, “Theory of a planckian metal,” Physical review letters 123, 066601 (2019).
  * Legros  _et al._ (2019) A Legros, S Benhabib, W Tabis, F Laliberté, M Dion, M Lizaire, B Vignolle, D Vignolles, H Raffy, ZZ Li, _et al._ , “Universal t-linear resistivity and planckian dissipation in overdoped cuprates,” Nature Physics 15, 142–147 (2019).
  * Nakajima  _et al._ (2019) Yasuyuki Nakajima, Tristin Metz, Christopher Eckberg, Kevin Kirshenbaum, Alex Hughes, Renxiong Wang, Limin Wang, Shanta R Saha, I-Lin Liu, Nicholas P Butch, _et al._ , “Planckian dissipation and scale invariance in a quantum-critical disordered pnictide,” arXiv preprint arXiv:1902.01034 (2019).
  * Bruin  _et al._ (2013) JAN Bruin, H Sakai, RS Perry, and AP Mackenzie, “Similarity of scattering rates in metals showing t-linear resistivity,” Science 339, 804–807 (2013).
  * Larkin and Varlamov (2005) Anatoli Larkin and Andrei Varlamov,  _Theory of fluctuations in superconductors_ (Clarendon Press, 2005).
  * Varlamov  _et al._ (2018) AA Varlamov, A Galda, and A Glatz, “Fluctuation spectroscopy: From rayleigh-jeans waves to abrikosov vortex clusters,” Reviews of Modern Physics 90, 015009 (2018).
  * Setty (2019) Chandan Setty, “Glass-induced enhancement of superconducting t c: Pairing via dissipative mediators,” Physical Review B 99, 144523 (2019).
  * Baggioli  _et al._ (2018) Matteo Baggioli, Bikash Padhi, Philip W Phillips, and Chandan Setty, “Conjecture on the butterfly velocity across a quantum phase transition,” Journal of High Energy Physics 2018, 49 (2018).
  * Zaanen  _et al._ (2015) Jan Zaanen, Yan Liu, Ya-Wen Sun, and Koenraad Schalm,  _Holographic duality in condensed matter physics_ (Cambridge University Press, 2015).


# Enhanced Scene Specificity with Sparse Dynamic Value Estimation

Jaskirat Singh & Liang Zheng
College of Engineering and Computer Science
Australian National University
Canberra, Australia
<EMAIL_ADDRESS>

###### Abstract

Multi-scene reinforcement learning involves training the RL agent across
multiple scenes / levels from the same task, and has become essential for many
generalization applications. However, the inclusion of multiple scenes leads
to an increase in sample variance for policy gradient computations, often
resulting in suboptimal performance with the direct application of traditional
methods (e.g. PPO, A3C). One strategy for variance reduction is to consider
each scene as a distinct Markov decision process (MDP) and learn a joint value
function dependent on both state $s$ and MDP $\mathcal{M}$. However, this is
non-trivial as the agent is usually unaware of the underlying level at train /
test times in multi-scene RL. Recently, Singh _et al._ [1] tried to address
this by proposing a dynamic value estimation approach that models the true
joint value function distribution as a Gaussian mixture model (GMM). In this
paper, we argue that the error between the true scene-specific value function
$V(s,\mathcal{M})$ and the predicted dynamic estimate $\hat{V}(s,\mathcal{M})$
can be further reduced by progressively enforcing sparse cluster assignments
once the agent has explored most of the state space. The resulting agents not
only show significant improvements in the final reward score across a range of
OpenAI ProcGen environments, but also exhibit increased navigation efficiency
while completing a game level.

##  1 Introduction

Training on environments comprising of multiple scenes / variations from the
same domain task (_e.g._ different levels from a video game), has become a
powerful strategy for countering over-fitting in deep reinforcement learning
[2, 3, 4, 5, 6, 7, 8]. However, such an approach comes at the price of
increased sample variance in policy gradient computations [9, 10]. The high
variance necessitates using more samples [11], and thus, training high
performance agents on these environments [9, 12, 13, 14, 15] invariably
involves increasing the sample size per update step through the use of
multiple parallel actors [16, 17, 18]. While parallel sample collection helps
in stabilizing the learning process, the obvious disadvantages of lower sample
efficiency and higher hardware constraints, suggest the need for specialized
variance reduction techniques in multi-scene RL.

One such strategy is to replace the traditionally used scene-generic value
function $V(s)$ with a scene-specific estimate $V(s,\mathcal{M})$ while
computing the advantage function [1]. However, in the absence of information
about the operational scene at train / test times, learning the scene-specific
value function presents a challenging problem. Recently, Singh _et al._ [1]
showed that while a fine estimation of the joint value function is not
feasible, a coarse approximation can be obtained by dividing the value
function distribution into multiple clusters and then using episode
trajectories to predict the assignment of the current state to each cluster
(refer Section 2.2 for details).

Figure 1: (Left) The clusters
$\\{\mathcal{C}_{1},\mathcal{C}_{2}...\mathcal{C}_{N}\\}$ originating from the
original dynamic value estimation [1] can be approximated as multi-variate
gaussian functions in $(s,V(s,\mathcal{M}))$ space. The overlap regions
correspond to points of high confusion and are usually characterized by
critical states / tricky obstacles [1]. We claim that such an overlap
constitutes a waste of information available for joint value estimation and
can be avoided by increasing the spread of learned cluster means (Right).

In this paper, we show that the scene-specificity of the dynamic value
estimates can be further enhanced by enforcing sparse cluster assignments,
once the agent has explored most of the state space and thus learned a good
enough approximation of the cluster parameters. The sparse cluster
probabilities along with application of traditional value function loss, have
a combined effect of spreading out the learned clusters in
$(s,V(s,\mathcal{M}))$ space. We claim that the adjustment of dynamic clusters
in this manner reduces the overall value function prediction error and support
it with extensive testing on OpenAI ProcGen [9] environments. Fig. 1 provides
an overview of our method.

The main contributions of this paper are summarized as follows.

  * •

We introduce a novel _confusion-contribution_ loss for improving dynamic value
estimation (DVE) [1]. The proposed loss decreases the overlap between learned
dynamic clusters by progressively enforcing sparse cluster assignments.

  * •

We demonstrate that the sparse clusters divide the overall state space into
distinct sets of game skills. The collection of these skills represents a
curriculum that the agent must master for effective game play.

  * •

By comparing the game level trajectories for the non-sparse and sparse dynamic
models, we show that the high navigation efficiency of our method and its
tendency to limit unnecessary exploration, presents an effective alternative
to explicit reward-shaping [19, 20, 21], for penalizing longer episode-lengths
/ reward-horizons in multi-scene reinforcement learning.

##  2 Relevant Background

###  2.1 Problem Setup

The multi-scene learning problem is characterized by a set of MDPs
$\mathcalbf{M}:\\{\mathcal{M}_{1},\mathcal{M}_{2}...\mathcal{M}_{N}\\}$. Each
MDP $\mathcal{M}$ is defined by state space $\mathcal{S}_{\mathcal{M}}$,
transition probabilities $\mathcal{P}_{\mathcal{M}}(s_{t+1}|s_{t},a_{t})$,
reward function $r_{\mathcal{M}}(s_{t},a_{t},s_{t+1})$, discount factor
$\gamma$ and the common action space $\mathcal{A}$. The agent with policy
$\pi(a|s)$ then interacts with a randomly chosen MDP to generate a trajectory
$\tau:\\{s_{0},a_{0},s_{1},a_{1},...s_{T}\\}$ with total discounted reward
$\mathcal{R}_{\tau}=\sum_{t=0}^{T-1}\gamma^{t}r(s_{t},a_{t},s_{t+1})$. We aim
to learn a policy $\pi^{*}$ such that the expected reward over the tuples
$(\mathcal{M},\tau)$ is maximized, _i.e._ ,
$\pi^{*}=\operatorname*{arg\,max}_{\pi}\mathbf{E}_{\tau,\mathcal{M}}\left[\mathcal{R}_{\tau,\mathcal{M}}\right]$.

###  2.2 Revisiting Dynamic Value Estimation

Singh _et al._ [1] show that the true value function distribution across
different scenes resembles a Gaussian Mixture Model and thus can be divided
into clusters. The main idea of dynamic value estimation is to enforce muti-
modal distribution learning by modelling the scene-specific value function as
weighted sum over the mean value estimates for these clusters. Mathematically,

$\displaystyle\hat{V}(s_{t},\mathcal{M})=\sum_{i=1}^{N_{b}}\alpha_{i}(s_{t},\tau^{t-})\
\hat{V}_{i}(s_{t})\qquad
s.t.\quad\alpha_{i}>0,\quad\sum_{i}^{N_{b}}\alpha_{i}=1,$ (1)

where $\tau^{t-}$ is the trajectory till time $(t-1)$, $N_{b}$ is the number
of clusters and $\alpha_{i}$, $\hat{V}_{i}(s)$ represent the cluster
assignments and the value function mean for the $i^{th}$ cluster,
respectively.

From a qualitative perspective, [1] also show that the distribution of cluster
assignments ($\alpha_{i}$) provides important intuition about the nature of
states, and define two metrics to analyse the same, _confusion_ and
_contribution_. Confusion ($\delta$) is a measure of uncertainty as to which
cluster, the current state-trajectory pair $\\{s_{t},\tau^{t-}\\}$ belongs to.
On the other hand, contribution ($\rho$), as the name suggests, determines the
‘contribution’ of a cluster in the overall value function estimation across a
general trajectory sequence $\tau:\\{s_{0},a_{0},s_{1},a_{1},...s_{T}\\}$.
Formally, confusion and contribution are defined as,

$\delta(s_{t},\tau^{t-})=\frac{1}{N_{b}.\sum_{i}\alpha^{2}_{i}(s_{t},\tau^{t-})},\qquad\rho_{i}(\tau)=\frac{1}{T}\sum_{t=1}^{T}\delta(s_{t},\tau^{t-})\
\alpha_{i}(s_{t},\tau^{t-}).$ (2)

##  3 Motivation

###  3.1 Minimizing Cluster Overlap

As shown in Fig. 1, we note that the original dynamic model leads to clusters
with high overlap (high confusion) at critical states [1]. The high confusion
states are usually characterized by presence of tricky obstacles / scenarios
and are critical to the final episode reward. Given the value estimation model
from Eq. 1, it is understandable that the use mean squared error critic loss
drives multiple cluster centers towards the true value of these critical
states. However, such a behavior is undesirable as it reduces the range of
value estimates $\hat{V}(s,\mathcal{M})$ covered through interpolation among
cluster means in Eq. 1. Fig. 2 explains how the spread of dynamic cluster
means affects the prediction error
$||V(s,\mathcal{M})-\hat{V}(s,\mathcal{M})||$ across
$\mathcal{M}\in\mathcalbf{M}$. Consequently, we conjecture that the overall
prediction error can be reduced by minimizing the overlap between the learned
dynamic clusters.

Figure 2: Qualitative Analysis. The dynamic value prediction model from Eq. 1
can be interpreted as the interpolation (using $\alpha_{i}$) across the
learned cluster means $\hat{V}_{i}(s)$. Thus, as seen above, the prediction
error is usually low in the range covered by the cluster centers. The original
dynamic clusters with their high overlap are similar to the distribution shown
on left, and have a very narrow range of low prediction error. As shown on the
right, this region of low prediction error can be expanded by increasing the
spread of learned dynamic cluster means.

###  3.2 Correlation Analysis

As the clusters move far apart from each other, the cluster assignments
$\alpha_{i}$ for a given tuple $(s,\tau^{-})$ tend towards a one-hot encoding,
with the one corresponding to the closest cluster. This implies that a higher
spread in cluster means corresponds to a sparser cluster assignment
distribution and can be measured using the confusion $\delta$ (refer Eq. 2).
Hence, to test the initial validity of the above analysis, we compute the
correlation between final model performance and inverse confusion
($1/\delta$), while training on OpenAI’s ProcGen [9] environments. The samples
for this testing are collected randomly during the first 50M timesteps of
training across 4 distinct runs with the original dynamic model. The Pearson
correlation [22] coefficients for various ProcGen games are shown in Fig. 3.
The results clearly corroborate our analysis from section 3.1 and show a high
correlation between reduced confusion and improved model performance.

[capbesideposition=right,center,capbesidewidth=7cm]figure[]

Figure 3: Results showing Pearson correlation [22] coefficient $(r)$ between
inverse confusion $(1/\delta)$ and total reward score $(\mathcal{R})$. For
most games the correlation coefficient is greater than 0.5, which points to
the statistical significance of the analysis done in section 3.1. We next
demonstrate how the original dynamic training can be modified to achieve lower
confusion in cluster assignments.

##  4 Sparse Dynamic Value Estimation

Given the analysis from Section 3.2, we note that increasing the inter-cluster
mean variance leads to sparser cluster assignment distribution. We claim that
the reverse is also true, _i.e._ , an appropriate spread in learned cluster
means can be obtained by progressively enforcing sparse cluster assignments
followed by adjustment of cluster means. Fig. 4 illustrates this process on a
sample GMM distribution.

Figure 4: Illustration. Example showing how sparse cluster assignments help in
reducing the overlap between clusters learned by the dynamic model. (Left)
Each point $(s,\mathcal{M})$ is allocated to the most probable cluster based
on the cluster assignments $\alpha_{i}$. (Right) The means of each cluster
adjust to reflect the expected value estimate of all $(s,\mathcal{M})$ pairs
in the modified cluster assignments. Note that the hard sparse assignment is
for illustration purposes only. In practice, the sparsity is introduced
progressively as the new cluster means are learned through the value function
loss.

###  4.1 Enforcing Sparsity

The sparsity condition is equivalent to maximization of the L2 norm for
cluster assignments $\\{\alpha_{1},\alpha_{2},\dots\alpha_{N_{b}}\\}$ and thus
using Eq. 2, corresponds to minimal confusion ($\delta$). However, we note
that a mere enforcement of sparsity may encourage convergence to solutions
where only one of the clusters is active. We also want to ensure that each
cluster contributes equally in the $(s,\mathcal{M})$ space. To achieve this,
we propose the following _confusion-contribution loss_ ,

$L^{CC}=k_{1}\
\mathbf{E}_{s_{t},\tau^{t-}}\left[\log\delta(s_{t},\tau^{t-})\right]+k_{2}\
\mathbf{E}_{\tau}\left[\log\left(\sum_{i}^{N_{b}}\rho^{2}_{i}(\tau)\right)\right].$
(3)

We must emphasize that the state space must have already been well explored by
the agent, prior to the application of confusion-contribution loss. If applied
prematurely, due to the continuous nature of neural networks, the sparse
cluster assignment is incorrectly generalized across the entire state space.
This would lead to a detrimental impact on value function estimation for the
currently unexplored states. Also, such a mistake is hard to recover from,
because for any state $s$, the sparse assignment ensures that the gradients
for all but one cluster are approximately zero.

##  5 Evaluation on OpenAI ProcGen

###  5.1 Experimental Design

Training Details. The network design for the dynamic model is quite similar to
the one described in [1]. The states are fed through an IMPALA-CNN [23] \+
LSTM [24] network to output a joint latent representation, used for learning
both the policy and the value function. The critic network uses these latent
representations to predict cluster assignments $\alpha_{i}$ and mean value
estimates $\hat{V}_{i}(s)$. Finally, the predicted value function
$\hat{V}(s,\mathcal{M})$ is computed using Eq. 1. Similar to [9], the agent is
trained using Proximal policy optimization (PPO) [25] with 4 parallel workers.
The only point of difference with the original dynamic model is the
application of confusion-contribution loss $(L_{CC})$ at a suitable stage in
the training process. The loss coefficients $(k_{1},k_{2})$ determine the
balance between confusion and contribution, and are chosen through extensive
hyper-parameter search for each environment.

We test our method on 8 ProcGen [9] environments: CoinRun, CaveFlyer, Climber,
Jumper, Plunder, Dodgeball, FruitBot and StarPilot. Note that each game is
characterized by a different rate of state exploration and training
trajectories. Thus, depending upon the type of environment, we adopt the
following strategies for obtaining sparse boosts.

Pre-boost. For games allowing rapid state space exploration at the beginning,
the confusion-contribution loss can be applied quite early to promote
sparsity. In fact, because the policy gradient and value function loss
dominate the initial training updates, we apply the confusion-contribution
loss from the start. However, the coefficients $(k_{1},k_{2})$ are kept
moderately small so as to encourage the network to progressively converge to a
sparse cluster assignment over the first quarter timesteps. CoinRun,
CaveFlyer, Climber and Jumper belong to this set and are labelled as _class-1_
environments.

Post-boost. In contrast, other games display a much more gradual expansion of
explored state space, exhibiting a positive correlation between episode
lengths and the total reward. Sparse-boosting for such environments, can only
be applied after the rate of increase of average episode length has declined.
Thus, the application of confusion-contribution loss is usually preceded by
pre-training with the original dynamic model for 50M timesteps (per worker).
Games like Plunder, Dodgeball, FruitBot and StarPilot are part of this set and
are labelled as _class-2_ environments.

We also train the vanilla-LSTM based RL2 [24] and non-sparse dynamic models
from [1], to show a comprehensive comparison between model performances. For
consistency reasons, a pre-training procedure same as the one described above
is followed, for all class-2 environments. All results are reported as an
average across 4 distinct runs using 500 levels for training.

###  5.2 Results

Figure 5: Learning curves for RL2, sparse and non-sparse dynamic models,
illustrating differences in sample efficiency, total reward and and episode
lengths.

Class-1. The sparse model leads to consistent performance improvements in all
4 class-1 environments. Fig. 5 shows the total reward and average episode
length curves during the training process for the Caveflyer environment. We
clearly see that sparse training leads to significant gains in both final
reward and sample efficiency over the regular dynamic model. For instance, we
report an increase of 28.3% and 22.4% in the final episode reward, over the
non-sparse dynamic model, for the games of CaveFlyer and Climber respectively
(refer Table 1).

Furthermore, as shown in Fig. 5 and Table 1, the sparse model leads to better
reward scores while on average, using much fewer timesteps per episode111Note
that the ProcGen environments have no explicit penalty for longer episode
lengths.. We call this phenomenon as enhanced navigation efficiency and delve
into it in detail in Section 7.

Class-2. Fig. 5 reports the results for the FruitBot (class-2) environment
using the post-boost strategy. While the baseline RL2 and non-sparse dynamic
models show a saturation in model performance with the extended training
protocol, the sparse model loss leads to continued gains in reward scores.
Interestingly, we also see that a saturation in rate of state space
exploration is necessary for getting gains with the sparse model. This is
illustrated through the training curves for the game of FruitBot (refer Fig.
5), where relative gains over the non-sparse model occur only after a decline
in the rate of increase of average episode length.

For sake of completeness, we report the results for all class-1 and class-2
environments in Table 1.

| Total Reward | Episode Length
---|---|---
Class | Environment | RL2 | DVE | Sparse DVE | RL2 | DVE | Sparse DVE
Class 1 | CoinRun | 7.75 | 9.16 | 9.62 | 126 | 78.99 | 62.1
CaveFlyer | 6.82 | 9.02 | 11.57 | 225.6 | 275.1 | 75.2
Climber | 7.50 | 8.14 | 10.17 | 178.1 | 226.6 | 170.5
Jumper | 6.61 | 6.52 | 6.65 | 236.4 | 180.3 | 78.9
Class 2 | Plunder | 7.13 | 17.16 | 18.42 | 495.1 | 780.6 | 739.4
DodgeBall | 10.98 | 11.25 | 12.76 | 401.5 | 459.9 | 285.8
FruitBot | 7.33 | 18.32 | 23.08 | 172.1 | 364.5 | 374.2
StarPilot | 17.94 | 18.08 | 19.81 | 327.6 | 342.9 | 320.2
Table 1: Performance comparison in final reward and average episode length for
both class 1 and 2 environments. Our method achieves higher total rewards
while needing much shorter episode lengths.

##  6 What are Clusters Made of?

Figure 6: Examples of key obstacles types learned by the each cluster in the
CoinRun Environment. We note that the sparse training divides the overall
state space into a distinct sets of game skills.

Given the non-sparse nature of original cluster probability distribution,
Singh _et al._ [1] use the normalized contribution scores as a measure of
similarity between the basis MDPs [1] and a particular game level. However,
such a comparison is not helpful in understanding the key features that
differentiate each basis cluster. In this section, we use the sparse property
of our method to visualize different obstacle types characteristic of each
cluster in the CoinRun Environment.

To visualize the distinguishing features for each cluster, we first extract
the set of states $\mathcal{S}_{i}$ for which each cluster is active. The
latent representations (output of the LSTM network) for these states are used
to map each $s\in\mathcal{S}_{i}$ to a two dimensional embedding space using
TSNE [26]. This embedding is then manually analysed for clusters to the
identify the salient obstacle classes.

Fig. 6 shows some key obstacle types for each cluster. We observe that each
cluster is responsible for predicting the value function on a distinct set of
obstacles / skills. For instance, cluster-1 is responsible for value
estimation in cases like double-jump from one side to another (skill-1) and
crossing over moving enemies (skill-2). On the other hand, cluster-2 handles
landing after jumps from higher ground (skill-1) and high jumps with very
limited visibility of the coming obstacles (skill-2). Finally, cluster-3 takes
care of precision climbs (skill-1) and jumps over wide valleys (skill-2).

Thus, we see that each disjoint state space set $\mathcal{S}_{i},\
i\in[1,N_{b}]$ represents a distinct curriculum of game skills that must be
learned for mastering the overall multi-scene game environment. This division
is analogous to human learning where it is quite common to break down a
complex task into a set of manageable skills before attempting the complete
task.

##  7 Navigation Efficiency

Figure 7: Demonstrating qualitative difference between trajectories for sparse
and non-sparse dynamic agents. Our method shows higher efficiency in
navigating to the final goals (red & green spheres).

A peculiar feature arising as a result of applying sparse boosting can be seen
in terms of improved _navigation efficiency_. That is, the agent on average
uses fewer time-steps per episode while achieving similar or better reward
score. This massive difference in time-steps results from two reasons:

  * •

The tendency to use fewer time-steps is a direct consequence of optimizing the
discounted reward function with $\gamma<1$ [11]. As a result, the agent is
incentivized to minimize the number of steps between the current state and the
next reward. Hence, more accurate policy updates (lower sample variance)
should lead to fewer timesteps.

  * •

As explained in Section 4, the expansion in state space after application of
confusion-contribution loss can lead to potential errors in value function
estimation. Thus, the sparse dynamic agent learns to maximize the utilization
of already explored state space.

In this section, we will analyse the first point in greater detail. We first
note that not all critical game states correspond to a high overlap region in
the non-sparse dynamic model. At a critical state with low overlap (and
possibly high value prediction error), the computation of suboptimal value
function, can lead the agent to underestimate the _advantage_ of choosing an
action leading to a faster route to the final destination / goal. We next try
to identify these critical states by comparing episode trajectories for the
sparse and non-sparse dynamic agents on the CaveFlyer environment.

Game Description. The goal of the Caveflyer environment is to destroy the red
spheres and finally reach the green sphere while avoiding intermediate
obstacles. The agent receives a small reward of +3 on destroying a red sphere
and an end of episode reward of +10 on successfully reaching the green one.
Direct collisions with an obstacle or the red sphere cause immediate episode
termination.

Trajectories for both sparse and non-sparse dynamic agents are shown in Fig.
7. We see that the non-sparse agent after destruction of the red sphere
(critical state), effectively restarts its search for the next target, while
often revisiting already encountered states. In contrast, the sparse agent
with its more accurate value estimates, realizes that the expected value for
exploring unseen parts of the cave is much higher than revisiting previous
states. Doing so not only helps the sparse agent in reaching the end goals
much faster, but also eliminates the need for evading obstacles it has already
crossed.

We also note that, the balance between the sparse model’s reluctance towards
state space expansion and maximization of total reward can be modulated
through the coefficients of the confusion-contribution loss. In this regard,
the high navigation efficiency of our method provides an effective alternative
to designing explicit reward shaping [19] penalties for promoting reduced
episode lengths.

##  8 Conclusion

This paper introduces a novel _confusion-contribution_ loss which improves the
efficiency of the recently proposed dynamic value estimation method, by
progressively learning sparser cluster assignments. The resulting dynamic
clusters _contribute_ equally to the overall value function estimation and
display minimal inter-cluster overlap. The proposed approach consistently
outperforms the vanilla-LSTM based RL2 and non-sparse dynamic models on a
range of OpenAI ProcGen environments, while on average using much fewer
timesteps per episode to complete a game level. Additionally, the sparse
training divides the overall state space into disjoint subsets. We show that
each subset focuses on a distinct set of game-skills, which draws a strong
parallel with the human learning paradigm.

## Broader Impact

While this work is largely theoretical, we believe that in the long term, it
will have major impact in the upcoming area of AI-inspired learning [27].
Recent years have seen the field of deep reinforcement learning demonstrate
tremendous success in achieving super-human performance in complex game play.
Deepmind’s Alphazero [28], Alphastar[29] and OpenAI’s Dota-2 [30] are some
salient examples. Each such milestone is followed by an increased public
interest to analyse and break down the policy of the trained RL agent into a
set of simple skills than can be consumed by a human learner [27, 31]. This
process is often manual and involves painstaking analysis across hundreds of
game runs. As shown in Section 6, our method does this automatically by
dividing the possible game scenarios (states) into distinct sets of game
skills. While each set can be composed of other mini-skills, the broad
division achieved by our method promises great potential in the development of
semi-automatic, AI-inspired teaching tools for human players.

## References

  * [1] J. Singh and L. Zheng, “Dynamic value estimation for single-task multi-scene reinforcement learning,” 2020.
  * [2] K. Cobbe, O. Klimov, C. Hesse, T. Kim, and J. Schulman, “Quantifying generalization in reinforcement learning,” arXiv preprint arXiv:1812.02341, 2018.
  * [3] N. Justesen, R. R. Torrado, P. Bontrager, A. Khalifa, J. Togelius, and S. Risi, “Illuminating generalization in deep reinforcement learning through procedural level generation,” arXiv preprint arXiv:1806.10729, 2018.
  * [4] A. Zhang, N. Ballas, and J. Pineau, “A dissection of overfitting and generalization in continuous reinforcement learning,” arXiv preprint arXiv:1806.07937, 2018.
  * [5] C. Zhang, O. Vinyals, R. Munos, and S. Bengio, “A study on overfitting in deep reinforcement learning,” arXiv preprint arXiv:1804.06893, 2018.
  * [6] M. Igl, K. Ciosek, Y. Li, S. Tschiatschek, C. Zhang, S. Devlin, and K. Hofmann, “Generalization in reinforcement learning with selective noise injection and information bottleneck,” in Advances in Neural Information Processing Systems, pp. 13956–13968, 2019.
  * [7] C. Packer, K. Gao, J. Kos, P. Krähenbühl, V. Koltun, and D. Song, “Assessing generalization in deep reinforcement learning,” arXiv preprint arXiv:1810.12282, 2018.
  * [8] F. Sadeghi and S. Levine, “Cad2rl: Real single-image flight without a single real image,” arXiv preprint arXiv:1611.04201, 2016.
  * [9] K. Cobbe, C. Hesse, J. Hilton, and J. Schulman, “Leveraging procedural generation to benchmark reinforcement learning,” arXiv preprint arXiv:1912.01588, 2019.
  * [10] X. Song, Y. Du, and J. Jackson, “An empirical study on hyperparameters and their interdependence for rl generalization,” arXiv preprint arXiv:1906.00431, 2019.
  * [11] J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel, “High-dimensional continuous control using generalized advantage estimation,” arXiv preprint arXiv:1506.02438, 2015.
  * [12] E. Kolve, R. Mottaghi, W. Han, E. VanderBilt, L. Weihs, A. Herrasti, D. Gordon, Y. Zhu, A. Gupta, and A. Farhadi, “Ai2-thor: An interactive 3d environment for visual ai,” arXiv preprint arXiv:1712.05474, 2017.
  * [13] A. Nichol, V. Pfau, C. Hesse, O. Klimov, and J. Schulman, “Gotta learn fast: A new benchmark for generalization in rl,” arXiv preprint arXiv:1804.03720, 2018.
  * [14] A. Juliani, A. Khalifa, V.-P. Berges, J. Harper, E. Teng, H. Henry, A. Crespi, J. Togelius, and D. Lange, “Obstacle tower: A generalization challenge in vision, control, and planning,” arXiv preprint arXiv:1902.01378, 2019.
  * [15] C. Beattie, J. Leibo, D. Teplyashin, T. Ward, M. Wainwright, H. Küttler, A. Lefrancq, S. Green, V. Valdés, A. Sadik, et al., “Deepmind lab. arxiv 2016,” arXiv preprint arXiv:1612.03801, 2016.
  * [16] P. Mirowski, R. Pascanu, F. Viola, H. Soyer, A. J. Ballard, A. Banino, M. Denil, R. Goroshin, L. Sifre, K. Kavukcuoglu, et al., “Learning to navigate in complex environments,” arXiv preprint arXiv:1611.03673, 2016\.
  * [17] M. Wortsman, K. Ehsani, M. Rastegari, A. Farhadi, and R. Mottaghi, “Learning to learn how to learn: Self-adaptive visual navigation using meta-learning,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6750–6759, 2019.
  * [18] V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu, “Asynchronous methods for deep reinforcement learning,” in International conference on machine learning, pp. 1928–1937, 2016.
  * [19] A. D. Laud, “Theory and application of reward shaping in reinforcement learning,” tech. rep., 2004.
  * [20] A. Laud and G. DeJong, “The influence of reward on the speed of reinforcement learning: An analysis of shaping,” in Proceedings of the 20th International Conference on Machine Learning (ICML-03), pp. 440–447, 2003.
  * [21] A. Y. Ng, D. Harada, and S. Russell, “Policy invariance under reward transformations: Theory and application to reward shaping,” in ICML, vol. 99, pp. 278–287, 1999.
  * [22] J. Benesty, J. Chen, Y. Huang, and I. Cohen, “Pearson correlation coefficient,” in Noise reduction in speech processing, pp. 1–4, Springer, 2009.
  * [23] L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward, Y. Doron, V. Firoiu, T. Harley, I. Dunning, et al., “Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures,” in International Conference on Machine Learning, pp. 1407–1416, 2018.
  * [24] Y. Duan, J. Schulman, X. Chen, P. L. Bartlett, I. Sutskever, and P. Abbeel, “Rl2: Fast reinforcement learning via slow reinforcement learning,” arXiv preprint arXiv:1611.02779, 2016.
  * [25] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” arXiv preprint arXiv:1707.06347, 2017\.
  * [26] L. v. d. Maaten and G. Hinton, “Visualizing data using t-sne,” Journal of machine learning research, vol. 9, no. Nov, pp. 2579–2605, 2008.
  * [27] M. Sadler and N. Regan, Game Changer: AlphaZero’s Groundbreaking Chess Strategies and the Promise of AI.  New in Chess, 2019.
  * [28] D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, et al., “Mastering chess and shogi by self-play with a general reinforcement learning algorithm,” arXiv preprint arXiv:1712.01815, 2017.
  * [29] O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev, et al., “Grandmaster level in starcraft ii using multi-agent reinforcement learning,” Nature, vol. 575, no. 7782, pp. 350–354, 2019.
  * [30] C. Berner, G. Brockman, B. Chan, V. Cheung, P. Dębiak, C. Dennison, D. Farhi, Q. Fischer, S. Hashme, C. Hesse, et al., “Dota 2 with large scale deep reinforcement learning,” arXiv preprint arXiv:1912.06680, 2019\.
  * [31] Deepmind, “Alphago teach.” https://alphagoteach.deepmind.com/, 2017.