# Jet-parton inelastic interaction beyond eikonal approximation

Raktim Abir  Theory Division, Saha Institute of Nuclear Physics, 1/AF
Bidhannagar, Kolkata 700064, India.

###### Abstract

Most of the models of jet quenching generally assumes that a jet always
travels in a straight eikonal path, which is indeed true for sufficiently hard
jet but may not be a good one for moderate and low momentum jet. In this
article an attempt has been made to relax part of this approximation for
$2\rightarrow 3$ processes and found a (15-20)$\%$ suppression in the
differential cross-section for moderately hard jets because of the noneikonal
effects. In particular, for the process $qq^{\prime}\rightarrow qq^{\prime}g$
in the centre of momentum frame the scattering with an angle wider than $\pm
0.52\pi$ is literally forbidden unlike the process $gg\rightarrow ggg$ that
allows an angular range $\pm\pi$. This may have consequence on the suppression
of hadronic spectra at low transverse momenta.

##  I Introduction

Constituent quark number scaling of elliptic flow and jet quenching are
supposed to be the most prominent signatures that favour the partonic degrees
of freedom in the deconfined QCD matter. Overwhelming evidences of both
signatures coming from dedicated heavy ion experiments $viz.,$ STAR and
PHENIX@ RHIC BNL Adams:2005dq ; Adcox:2004mh , ALICE @ LHC CERN Aamodt:2010jd
; Aamodt:2010pa  established the fact that, the primordial hot-soup of nuclear
matter, produced in those experiments, indeed contain partonic degrees of
freedom, before freeze out to hadrons in later stage, instead of hadronic
degrees of freedom throughout. Observation of strong suppression of inclusive
yields of high momentum hadrons and semi-inclusive rate of azimuthal back-to-
back high momentum hadron pairs relative to p-p collisions, are expectations
from jet quenching. Both of them are extensively explored in collisions of Au-
Au nuclei at $\sqrt{s}=200$ A GeV in RHIC. ALICE, the dedicated heavy Ion
collider experiment at CERN, seems to appear as a factory of Jets. Evidence
for jet quenching has also been observed recently in Pb-Pb collision at ALICE
Aamodt:2010jd .

There are a few well known models in the literature Salgado:2003gb ;
Wicks:2005gt ; Wang:2001ifa ; Jeon:2003gi ; Mustafa:1997pm ; Mustafa:2003vh ;
Qin:2007rn ; Qin:2009gw ; Abir:2012pu ; Abir:2011jb ; Zapp:2012ak  that aim to
quantify energy loss (mainly radiative) and jet quenching phenomena within
perturbative QCD. Most of these formalisms for energy loss of a high momentum
parton through gluon radiations, have some common technical approximations in
order to make the calculation simpler. These approximations are implemented
both at the level of single emission kernel calculations and at multiple gluon
emission estimation schemes. Main kinematic approximations at the level of
single emission kernel, are listed below (also see Refs. Armesto:2011ht ;
Mehtar-Tani:2013pia ; Renk:2011aa  for a comprehensive discussion):

  * •

Eikonal parton trajectory I :  Leading parton is having energy $E$ ($i.e.$
$p_{z}=E$ and $p_{\bot}=0$, by definition to start with) is much larger than
the transverse momentum exchanged gluon $q_{\perp}$ with the medium, $E\gg
q_{\perp}$, so that it does not give sufficient transverse kick to deflect the
parton from straight trajectory along $z$ axis. To relax this approximation it
is therefore important to keep track of the terms of ${\cal O}(q_{\bot}/E)$ in
the formalism.

  * •

Eikonal parton trajectory II :  Energy of the leading parton is sufficiently
high, $E\gg k_{\perp}$ ($k_{\perp}$ being transverse component of the emitted
gluon) so that it does not get enough transverse kick also from emitted gluon.
This ensures that the leading parton is in eikonal trajectory. However it does
not fix any definite direction for gluon emission, which requires comparison
of $k_{\perp}$ with longitudinal component $k_{z}$ or with energy $\omega$ of
the emitted gluon. Therefore, it is important not to neglect terms of ${\cal
O}(k_{\bot}/E)$ in the formalism to relax this approximation in the jet
studies.

  * •

Soft gluon emission :  One often uses the additional approximation that the
gluon energy is much smaller than the leading parton energy $\omega\ll E$.
When $x$ is the fraction of energy carried out by the emitted gluon, $i.e.$,
$x=w/E$, this approximation ensures that $x\rightarrow 0$. When $x$ is typical
light cone variable, defined by the fraction of light cone $(+)ve$ momentum
carried out by the emitted gluon, $i.e.$,
$x=k^{+}/p^{+}=(k_{\bot}/\sqrt{s})e^{\eta}$, this approximation ensures
$x\rightarrow 0$, only in the mid rapidity and backward rapidity regions
$(-\infty\geq\eta\geq 0)$ but not in the forward regions where $\eta$ could be
a large positive number.

  * •

Small angle/collinear gluon emission :  Energy of the emitted gluon $\omega$
is much larger than its transverse momentum $k_{\bot}$, $\omega\gg k_{\bot}$
and $\omega\simeq k_{z}$. For any $2\rightarrow 3$ process, without loss of
generality, one can take $k_{\bot}=\omega\sin\theta_{g}$ and
$k_{z}=\omega\cos\theta_{g}$, where $\theta_{g}$ being the angle between
direction of propagation of leading parton and direction of emitted gluon.
This particular approximation therefore implies $\theta_{g}\simeq 0$.

At this point it is worth mentioning that soft gluon emission approximation is
a broader approximation as it automatically encompasses the eikonal parton
trajectory II approximation, because energy $w$ should always be more than the
transverse momentum $k_{\bot}$ for a massless emitted gluons.

In this article we make an effort to relax the eikonal parton trajectory I
approximation in some extent for the radiative/inelastic process $2\rightarrow
3$. Investigation of all the matrix elements in ${\cal{O}}(\alpha_{s}^{3})$
for $2\rightarrow 3$ radiative processes have been done keeping terms up to
$\cal{O}$ $(t/s)$. The first order in eikonal expansion, i.e., $\cal{O}$
$(t/s)$, is termed here as noneikonal, since the calculation is performed in
Feynman gauge with Mandelstum variables instead of most extensively used light
cone gauge with light cone variables111After connecting $t$ to $q_{\perp}$ and
$s$ to $E$ in the centre of momentum frame, term of $\cal{O}$ $(t/s)$ ensures
the relaxation of the approximation $E\gg q_{\perp}$ (eikonal parton
trajectory I).. We have neglected terms of ${\cal{O}}(t^{2}/s^{2})$ and higher
orders. Throughout our study we also do not use any approximation related to
small angle/collinear gluon emission. Nevertheless, we have used soft gluon
emission approximation which automatically include Eikonal parton trajectory
II assumption.

Here we relaxed the kinematic constrains associated with eikonal propagation.
Nevertheless (non)eikonal propagation is directly related to space-time tracks
that the partons take when going through the medium. While an eikonal path is
a straight line, a non-eikonal path is generically any trajectory that have
deviation from straight line and obviously always somewhat longer. This longer
path, taken by the parton may matter, in the soft sector when the problem is
embedded into a hydrodynamically evolving density distribution Renk:2011gj .
We note that there is transport model, although uses the cross sections which
are kind of eikonal, takes all the trajectories into account, so particles do
not need to move on straight lines Fochler:2010wn .

##  II  Inelastic Quark-Quark Scattering ($qq^{\prime}\rightarrow
qq^{\prime}g$)

The process $qq^{\prime}\rightarrow qq^{\prime}g$ (prime denotes different
quark flavour) in ${\cal{O}}(\alpha_{s}^{3})$ appears in five $t$ channel
Feynman diagrams, which are shown in the Fig. 1 (see also Fig. 2 for other
details). Note that $k_{1}$ and $k_{2}$ are momenta of the different quark
flavours in the entrance channel whereas $k_{3}$ and $k_{4}$ are those for
exit channel and $k_{5}$ is that of the emitted gluon. Scattering angle
between ${\bf k_{1}}$ and ${\bf k_{3}}$ is $\theta_{q}$ whereas $\theta_{g}$
is the angle between direction of emission of soft gluon $\bf\hat{k}_{5}$ and
direction of incoming projectile quark $\bf\hat{k}_{1}$. We now define the
relevant Mandelstum variables for this $2\rightarrow 3$ process as

$\displaystyle s=(k_{1}+k_{2})^{2}\,,\hskip
8.5359pts^{\prime}=(k_{3}+k_{4})^{2},$ $\displaystyle
u=(k_{1}-k_{4})^{2}\,,\hskip 8.5359ptu^{\prime}=(k_{2}-k_{3})^{2},$
$\displaystyle t=(k_{1}-k_{3})^{2}\,,\hskip
8.5359ptt^{\prime}=(k_{2}-k_{4})^{2},$ (1)

with

$\displaystyle s+t+u+s^{\prime}+t^{\prime}+u^{\prime}=0\ .$ (2)

When the emitted gluon is soft ($k_{5}\rightarrow 0$) compare to other
external legs, one can assume : $t^{\prime}\rightarrow t,\ \
s^{\prime}\rightarrow s,\ \ u^{\prime}\rightarrow u$ and we can express the
transverse component of the momentum of the emitted gluon in the centre of
momentum frame of $k_{1}$ and $k_{2}$ as

$\displaystyle k_{\perp}^{2}$ $\displaystyle=$
$\displaystyle\frac{4(k_{1}\cdot k_{5})(k_{2}\cdot k_{5})}{s}$ (3)
$\displaystyle=$ $\displaystyle\frac{(s+t+u)(s+u^{\prime}+t^{\prime})}{s}$
$\displaystyle=$ $\displaystyle\frac{(s+t+u)^{2}}{s}\ .$

\begin{overpic}[width=260.17464pt]{qQ_qQg.eps} \put(-8.0,67.0){(1)}
\put(-8.0,38.0){(2)} \put(51.0,67.0){(3)} \put(51.0,38.0){(4)}
\put(25.0,10.0){(0)} \end{overpic} Figure 1: Five tree level Feynman diagrams
for the process $qq^{\prime}\rightarrow qq^{\prime}g$.

The hierarchy among various scale of momentum, employed in the present work is
stated as

$\sqrt{s},E>\sqrt{|t|},q_{\bot}\gg w\geq k_{\bot}\geq m_{d}\,,$ (4)

where $m_{d}$ is the Debye screening mass acts as an infrared cut-off. We note
that the above hierarchy relaxes the aproximations
$\sqrt{s},E\gg\sqrt{|t|},q_{\bot}$ (Eikonal parton trajectories I ) and $w\gg
k_{\bot}$ (small angle/collinear gluon emission ) in some extent.

Figure 2: Non-zero angular deviation from eikonal trajectory. Angle between
incoming and outgoing momentum of projectile is $\theta_{q}$ and direction of
emition of gluon with that of incoming momentum of projectile parton is
$\theta_{g}$.

###  II.1 Matrix Elements, Amplitude and Cross-section

The gauge invariant amplitude summed over all the final states and averaged
over initial states for the process, $qq^{\prime}\rightarrow qq^{\prime}g$, is

$\left|{\cal M}_{qq^{\prime}\rightarrow qq^{\prime}g}\right|^{2}=\sum_{i\geq
j}{\cal M}_{ij}^{2}\ ,$ (5)

where $i$ and $j$ run from $0$ to $4$. We note that the index $0$ represents
the diagram where the soft gluon emits from the exchanged gluon line whereas
the indices $m=1,2,3,4$ represent the diagrams where emission of the soft
gluons are being from external fermion lines having momenta $k_{m}$ (see Fig.
2). Equation (5) contains total fifteen terms in which there are five self
interfering ‘genuine amplitudes’ for $i=j$ and ten cross interfering
‘interference amplitudes’ associated with gluon emissions involving two
diagrams for $i\neq j$. Here we have extensively used the FORM, REDUCE and
CalcHep Programs Belyaev:2012qa . Results are given below up to ${\cal
O}(1/k_{\bot}^{2})$ and ${\cal O}(t/s)$, for soft gluon emission.

####  II.1.1 Genuine amplitudes

By genuine amplitudes we are referring those are coming from each diagram that
interfers with itself. In the Feynman gauge 222In light cone gauge amplitudes
coming from diagrams that involve gluon emission from target partons can only
be neglected, others are not. all of them vanish within soft gluon emission
approximations and in ${\cal O}(1/k_{\bot}^{2})$:

$\displaystyle{\cal M_{\rm 11}^{\rm 2}}$ $\displaystyle=$ $\displaystyle{\cal
M_{\rm 33}^{\rm 2}}=0;\,\,{\cal M_{\rm 22}^{\rm 2}}={\cal M_{\rm 44}^{\rm
2}}=0;\,\,{\cal M_{\rm 00}^{\rm 2}}=0.$ (6)

However, they may contribute in ${\cal O}(1)$, ${\cal O}(k_{\bot}^{2})$,
${\cal O}(k_{\bot}^{4})$ etc, and in ${\cal{O}}(t^{2}/s^{2})$ and higher
orders. All of them can safely be neglected in the soft emission limit as our
aim is to go beyond the eikonal approximation-I, $E\gg q_{\perp}$.

####  II.1.2 Interference amplitudes

Within the approximations employed above the amplitudes corresponding to the
matrix elements of $({\rm 1}\bf\otimes{\rm 4})$ and $({\rm 2}\bf\otimes{\rm
3})$ are identical in the leading order (${\cal O}(1/k_{\bot}^{2})$) as well
in ${\cal O}(t/s)$ and given as

$\displaystyle{\cal M_{\rm 14}^{\rm 2}}$ $\displaystyle=$ $\displaystyle{\cal
M_{\rm 23}^{\rm
2}}=\frac{7}{8}\frac{128}{27}g^{6}\frac{s^{2}}{t^{2}}\frac{1}{k_{\bot}^{2}}\left[1+2\frac{t}{s}\right],$

where ${\cal O}(t/s)$ is purely noneikonal (i.e., the first order in eikonal
approximation) in nature as noted earlier. Also the amplitudes for $({\rm
1}\bf\otimes{\rm 2})$ and $({\rm 3}\bf\otimes{\rm 4})$ are identical and the
contribution is obtained in ${\cal O}(1/k_{\bot}^{2})$ and ${\cal O}(t/s)$ as

$\displaystyle{\cal M_{\rm 12}^{\rm 2}}$ $\displaystyle=$ $\displaystyle{\cal
M_{\rm 34}^{\rm
2}}=\frac{1}{4}\frac{128}{27}g^{6}\frac{s^{2}}{t^{2}}\frac{1}{k_{\bot}^{2}}\left[1+\frac{t}{s}\right].$

Both $({\rm 1}\bf\otimes{\rm 3})$ and $({\rm 2}\bf\otimes{\rm 4})$ are also
identical within the employed hierarchy. However, they do not contribute in
leading order but only in ${\cal O}(t/s)$. Hence, in Feynman gauge the
contribution from the interference between initial state  and final state
radiations, is exclusively noneikonal in nature and given as

$\displaystyle{\cal M_{\rm 13}^{\rm 2}}$ $\displaystyle=$ $\displaystyle{\cal
M_{\rm 24}^{\rm
2}}=\frac{1}{4}\frac{128}{27}g^{6}\frac{s^{2}}{t^{2}}\frac{1}{k_{\bot}^{2}}\left[\frac{1}{2}\frac{t}{s}\right],$

Finally any diagram interfering with ${\rm 0}$, i.e., $({\rm 0}\bf\otimes{\it
l})$ with $l=1,2,3,4$, does not contribute in ${\cal O}(1/k_{\bot}^{2})$ but
contributes in $(1/k_{\bot}\sqrt{t})$ and higher orders. In the limit
$|\sqrt{t}|\sim q_{\bot}\gg w$, amplitudes of ${\cal O}(1/k_{\bot}\sqrt{t})$
are subleading, in comparison to ${\cal O}(1/k_{\bot}^{2})$. Therefore, all of
these (${\cal M_{\rm 10}^{\rm 2}}$, ${\cal M_{\rm 20}^{\rm 2}}$, ${\cal M_{\rm
30}^{\rm 2}}$ and ${\cal M_{\rm 40}^{\rm 2}}$) do not contribute within the
approximation used in this work.

Here we note that color coherence pattern between initial and final state
radiation in the presence of a QCD medium derived in light cone gauge shows
angular distribution of the induced gluon spectrum is broadly modified when
one includes interference terms Armesto:2012qa ; MehtarTani:2012cy . In
Feynman gauge within our hierarchy only interferance terms are contributing to
the amplitude.

The gauge invariant amplitude for the process, $qq^{\prime}\rightarrow
qq^{\prime}g$, can now be obtained by summing all the subamplitudes as

$\displaystyle\left|{\cal M}_{qq^{\prime}\rightarrow
qq^{\prime}g}\right|^{2}\\!\\!\\!$ $\displaystyle=$
$\displaystyle\\!\\!12g^{2}\left|{\cal M}_{qq^{\prime}\rightarrow
qq^{\prime}}\right|^{2}_{eknl}\frac{1}{k_{\perp}^{2}}\left(1+\frac{17}{9}\frac{t}{s}\right),$
(7)

where the two body amplitude is given as

$\displaystyle\left|{\cal M}_{qq^{\prime}\rightarrow
qq^{\prime}}\right|^{2}_{eknl}=\frac{8}{9}g^{4}\frac{s^{2}}{t^{2}}.$ (8)

The three-body amplitude in (7) for the inelastic process,
$qq^{\prime}\rightarrow qq^{\prime}g$, contains the two-body amplitude for the
elastic process, an infrared factor for the emission of a soft gluon and a
noneikonal correction factor. The expression in (7) will lead to $q_{\bot}\gg
k_{\bot}$ limit of the Gunion and Bertsch formula Gunion:1981qs . If the
emitted gluon is much softer than others it can then be regulated by the Debye
screening mass, $m_{d}$. Terms within the parenthesis in (7) would correspond
to noneikonal correction over the eikonal Gunion Bertsch formula. Eq. (7) is
complete upto ${\cal O}(1/k_{\bot}^{2})$ and ${\cal O}(t/s)$, for emission of
a soft gluon in the process $qq^{\prime}\rightarrow qq^{\prime}g$. Similar
investigation have been done earlier for the process $gg\rightarrow ggg$
Das:2010hs ; Abir:2010kc ; Bhattacharyya:2011vy .

Figure 3: Dependence of inverse cross-section ${\sigma^{-1}}$ on scattering
angle $\theta_{q}$, all prefactor that are independent of $\theta_{q}$ taken
to be unity for convenience.

####  II.1.3 Rutherford scattering beyond eikonal approximation

It is interesting to note how noneikonality gives way to probe beyond
Rutherford scattering limits. In the centre of momentum frame for a typical
$2\rightarrow 2$ (or even in case of $2\rightarrow 3$ when fifth particle is
ultra soft!) process

$\displaystyle\frac{t}{s}$ $\displaystyle=$
$\displaystyle-\sin^{2}\frac{\theta_{q}}{2}$ (9)

The eikonal cross-section $(\sigma_{eknl})$ is directly connected to the
Rutherford scattering scattering cross-section as

$\displaystyle\sigma_{eknl}\propto\frac{s^{2}}{t^{2}}=\frac{1}{\sin^{4}(\theta_{q}/2)}\,.$
(10)

When one relaxes the eikonal approximation then the cross-ection can be
written as

$\displaystyle\\!\\!\\!\sigma_{ne}\\!\\!$ $\displaystyle\propto$
$\displaystyle\\!\\!\frac{s^{2}}{t^{2}}\left(1+\frac{17}{9}\frac{t}{s}\right)\\!\\!=\\!\\!\frac{1}{\sin^{4}\frac{\theta_{q}}{2}}\left(1-\frac{17}{9}\sin^{2}\frac{\theta_{q}}{2}\right),$
(11)

which puts a restriction on the scattering angle, $\theta_{q}$. In Fig.3 we
have plotted inverse cross-section $(\sigma^{-1})$ for both eikonal and
noneikonal case. Even though both behave identically with a similar plateau in
the small angle scattering but noneikonal cross-section has a very sharp fall
in comparison with the eikonal one for large angle scattering. As seen the
noneikonal inverse matrix element is bounded by the scattering angle,
$\theta_{q}=\pm 2\sin^{-1}(3/\sqrt{17})\simeq\pm 0.52\pi$, in centre of
momentum frame, in contrary to that of eikonal one having a natural bound of
$\pm\pi$. This indicates that the back scattering is forbidden for the case of
$qq^{\prime}\rightarrow qq^{\prime}g$ when the emitted gluon is soft.

####  II.1.4 Cross-section in the first order in eikonal (viz.,noneikonal)
approximation

The cross-section for the process ${qq^{\prime}\rightarrow qq^{\prime}g}$ can
be obtained as

$\displaystyle\sigma_{qq^{\prime}\rightarrow qq^{\prime}g}$ $\displaystyle=$
$\displaystyle\frac{1}{2s}\int\prod_{i=3}^{5}\frac{d^{3}k_{i}}{(2\pi)^{3}2E_{i}}\left|{\cal
M}_{qq^{\prime}\rightarrow
qq^{\prime}g}\right|(2\pi)^{4}\delta^{4}(k_{1}+k_{2}-k_{3}+k_{4}+k_{5})\,.$
(12)

In the centre of momentum frame, ${\bf k_{1}}+{\bf k_{2}}={\bf k_{3}}+{\bf
k_{4}}+{\bf k_{5}}={\bf 0}$, and one obtains

$\displaystyle\sigma_{qq^{\prime}\rightarrow qq^{\prime}g}$ $\displaystyle=$
$\displaystyle\frac{1}{2s}\int\frac{d^{3}k_{3}}{(2\pi)^{3}2E_{3}}\frac{1}{(2\pi)^{3}2E_{4}}\frac{d^{3}k_{5}}{(2\pi)^{3}2E_{5}}\left|{\cal
M}_{qq^{\prime}\rightarrow
qq^{\prime}g}\right|(2\pi)^{4}\delta(E_{1}+E_{2}-E_{3}+E_{4}+E_{5})$ (13)
$\displaystyle=$
$\displaystyle\frac{1}{2s}\left[-\frac{1}{2}\frac{1}{(2\pi)^{2}}\int\frac{dq_{\bot}^{2}dq_{z}}{E_{3}}\right]\frac{1}{(2\pi)^{3}2E_{4}}\left[\frac{1}{4}\frac{1}{(2\pi)^{2}}\int\frac{dk_{\bot}^{2}d\theta_{g}}{\sin\theta_{g}}\right]$
$\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}\times
12g^{2}\frac{8}{9}g^{4}\frac{s^{2}}{t^{2}}\frac{1}{k_{\bot}^{2}}\left(1+\frac{17}{9}\frac{t}{s}\right)(2\pi)^{4}\delta(E_{1}+E_{2}-E_{3}+E_{4}+\omega)$
$\displaystyle=$
$\displaystyle\frac{1}{2s}\left[-\frac{1}{2}\frac{1}{(2\pi)^{2}}\int\frac{dq_{\bot}^{2}}{E_{1}}\right]\frac{1}{(2\pi)^{3}2E_{1}}\left[-\frac{1}{4}\frac{1}{(2\pi)^{2}}\int
dk_{\bot}^{2}d\eta\right]$
$\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}\times
12g^{2}\frac{8}{9}g^{4}\frac{s^{2}}{(q_{\bot}^{2})^{2}}\left(1+\frac{q_{\bot}^{2}}{s}\right)^{-2}\frac{1}{k_{\bot}^{2}}\left(1-\frac{17}{9}\frac{q_{\bot}^{2}}{s}\right)(2\pi)^{4}\,,$

where we have used rapidity
$\eta=-\ln\left[\tan\left(\theta_{g}/2\right)\right]$, $\bf q=k_{1}-k_{3}$,
$\bf q_{\bot}=q~{}{\sin\theta_{q}}$. The cross-section contains factors,
having term like $q_{\bot}^{2}/s$, that are responsible for non-eikonal
effects.

In thermal medium taking debye mass as infrared regulator, the differential
cross-section can be expressed as

$\displaystyle\frac{d~{}\sigma^{qq^{\prime}\rightarrow
qq^{\prime}g}}{dq_{\bot}^{2}dk_{\bot}^{2}d\eta}=2C_{A}C_{qq^{\prime}}\alpha^{3}\frac{\Gamma_{ab}}{(q_{\bot}^{2}+m_{d}^{2})^{2}}\frac{1}{k_{\bot}^{2}+m_{d}^{2}}\,,$
(14)

where $C_{A}=3$ and $C_{qq^{\prime}}=8/9$ are Casimir factors, and
$\Gamma_{ab}=\zeta_{\it a}\zeta_{\it b}$, with various factors are,
explicitly, given as

$\displaystyle\zeta_{\it a}$ $\displaystyle=$
$\displaystyle\left(1+\frac{q_{\bot}^{2}}{s}\right)^{-2}\,,$
$\displaystyle\zeta_{\it b}$ $\displaystyle=$
$\displaystyle\left(1-\frac{17}{9}\frac{q_{\bot}^{2}}{s}\right)\ .$ (15)

The factor $\zeta_{\it a}$ comes from eikonal part of the matrix elements, and
$\zeta_{\it b}$ is the noneikonal factor originated from noneikonal part of
matrix elements. The differential cross-section for the process
$qq^{\prime}\rightarrow qq^{\prime}g$ as given in Eq.(14) correctly reproduces
the result of Biro:1993qt  in the limit $q_{\bot}^{2}\gg k_{\bot}^{2}$ and in
the eikonal limit, $\sqrt{s},E\gg q_{\bot}^{2}$, as all the noneikonal
factors, $viz.,$ $\zeta_{\it a}$ and $\zeta_{\it b}$ become identically unity
and so as $\Gamma_{ab}$.

##  III Inelastic gluon-gluon fusion ($gg\rightarrow ggg$)

The three gluon production via gluon-gluon fusion $gg\rightarrow ggg$ is
extremely important in the context of heavy-ion phenomenology. For a sequence
of events: hot glue scenario of glasma field, thermal equilibration, gluon
chemical equilibration in later time, parton matter viscosity, radiative
energy-loss of high energy partons jet propagating through thermalised QGP,
this process plays a crucial role. Matrix elements for the process
$gg\rightarrow ggg$ have been computed up to ${\cal O}(t^{3}/s^{3})$ in
Abir:2010kc . Considering ${\cal O}(t/s)$ result it is now straightforward to
evaluate the differential cross-section for this process in first order in
eikonal approximation as

$\displaystyle\frac{d~{}\sigma^{gg\rightarrow
ggg}}{dq_{\bot}^{2}dk_{\bot}^{2}d\eta}=2C_{A}C_{gg}\alpha^{3}\frac{\Gamma_{ab}}{(q_{\bot}^{2}+m_{d}^{2})^{2}}\frac{1}{k_{\bot}^{2}+m_{d}^{2}}\,$
(16)

where $C_{gg}=9/2$ and the factor, $\Gamma_{ab}=\zeta_{a}\zeta_{b}$, with its
various components

$\displaystyle\zeta_{\it a}$ $\displaystyle=$
$\displaystyle\left(1+\frac{q_{\bot}^{2}}{s}\right)^{-2}\,,$
$\displaystyle\zeta_{\it b}$ $\displaystyle=$
$\displaystyle\left(1-\frac{1}{2}\frac{q_{\bot}^{2}}{s}\right)\ .$ (17)

Figure 4: Typical estimation of noneikonal factors at $T=300MeV$ with
$\alpha=0.3$ for the process $qq^{\prime}\rightarrow qq^{\prime}g$. First
order noneikonal factors $\zeta_{a}=\left(1+{q_{\bot}^{2}}/{s}\right)^{-2}$,
$\zeta_{b}=\left(1-{17q_{\bot}^{2}}/{9s}\right)$, and the full contribution
${\Gamma}_{ab}=\zeta_{a}\zeta_{b}$. Figure 5: Typical estimation of noneikonal
factors at $T=300MeV$ with $\alpha=0.3$ for the process $gg\rightarrow ggg$.
First order noneikonal factors
$\zeta_{a}=\left(1+{q_{\bot}^{2}}/{s}\right)^{-2}$,
$\zeta_{b}=\left(1-{q_{\bot}^{2}}/{2s}\right)$ and the full contribution
${\Gamma}_{ab}=\zeta_{a}\zeta_{b}$.

The factor comming from eikonal part of matrix element $\zeta_{\it a}$ is same
for both processes $qq^{\prime}\rightarrow qq^{\prime}g$ and $gg\rightarrow
ggg$ whereas the noneikonal factor $\zeta_{\it b}$ is different. This
noneikonal factor, obviously, does not put any restriction on the scattering
angle ($\theta=\pm\pi$) for the process $gg\rightarrow ggg$, and allows it to
go in full natural range $\pm\pi$ as compared to the process
$qq^{\prime}\rightarrow qq^{\prime}g$.

Unlike $gg\rightarrow ggg$ where a Park-Taylor type formula Parke:1986gb  is
available to compute the matrix element, the computation of matrix elements up
to ${\cal O}(t/s)$ is quite cumbersome in case of $qg\rightarrow qgg$. Also in
this article we have performed our study on inelastic quark-quark scattering
but with different flavours. In case of same flavour $qq\rightarrow qqg$
things would be a more involved one. We leave them for future study.

##  IV Results and Discussion

For quantitative estimation of the noneikonal effects, we have taken the
average value of the momentum transfer squared which can be obtained
Abir:2012pu  as

$\displaystyle\langle q_{\bot}^{2}\rangle$ $\displaystyle\simeq$
$\displaystyle\left({\int\limits_{m_{g}^{2}}^{E^{2}}dq_{\bot}^{2}\
q_{\bot}^{2}\ \frac{d\sigma_{2\rightarrow
3}}{dq_{\bot}^{2}}}\right)/\left({\int\limits_{m_{g}^{2}}^{E^{2}}dq_{\bot}^{2}\
\frac{d\sigma_{2\rightarrow 3}}{dq_{\bot}^{2}}}\right)$ (18)
$\displaystyle\simeq$ $\displaystyle 2g^{2}T^{2}\ln\left(E/gT\right).$

$\langle q_{\bot}^{2}\rangle$ has then been embedded in $\zeta_{a}$,
$\zeta_{b}$ to have a qualitative estimation of noneikonal effects over
eikonal crosssections.

In Fig.4 and Fig.5 the first order noneikonal factors: $\zeta_{a}$,
$\zeta_{b}$ and the full contribution $\Gamma_{ab}$ for both processes
$qq^{\prime}\rightarrow qq^{\prime}g$ and $gg\rightarrow ggg$, respectively,
displayed. It can be seen that the noneikonal effects are $\sim(15-20)\%$ over
eikonal one for moderately hard jets. However, the noneikonal effect gradually
becomes mild for very high energetic jets. Typically for charged hadrons in
the momentum range of $8$ to $50$ GeV, i.e. typical parton kinematics of $16$
to $150$ GeV, way off the scale of Fig.4 and Fig.5, indicate that non-eikonal
effects are largely absent above $(10-15)$ GeV parton kinematics. In the
literature attempts have already been made to address the noneikonal
propagation of partons for collision/elastic processes in a monte-carlo
approach Auvinen:2009qm  by considering full ${\cal{O}}(\alpha_{s}^{2})$
matrix elements for relevant $2\rightarrow 2$ processes. Eikonal propagation
approximation was found to be good on the $10\%$ level. Present study also
revels that for radiative/inelastic process, Eikonal parton trajectory I
approximation seems to be within $(15-20)\%$ level. This approximation should
be crude only in soft and moderate momentum regimes. We also note that
splitting kernels for partons produced in large virtuality scattering
processes that subsequently traverse a region of strongly-interacting matter
have been investigated early in the literature within effective theory
formalism Ovanesyan:2011kn .

There has always been a quest for large angle radiations. In the present study
we do not assume small angle/collinear emission approximation either in the
course of calculating matrix elements or in the calculations of kinematics. In
the present calculation the kinematic relation $k_{\bot}=\omega\sin\theta_{g}$
ensures that one can go safely to the limit where $k_{\bot}\simeq\omega$.

Most of the pQCD based model calculations for describing the medium is not
able to account the almost ideal fluid behaviour, which seems to be a
manifestation of long range correlations. It also would lead to a large
elastic contribution to energy loss Auvinen:2010yt  for reasonable values of
coupling which is not supported by the data. Sort of single hard scenario
Gyulassy:2000fs ; Wicks:2005gt  has been discussed here at the level of single
emission karnel in which gluons are induced by a single hard scattering with
the medium, but many widely used quenching models, for instance Salgado:2003gb
, make the opposite assumption of multiple soft interactions with the medium
leading to induced gluon radiation. Such classes of models can probably be
expected to have somewhat milder non-eikonal corrections due to scattering
with the medium.

The kinematic constraints, $E\gg\omega\gg k_{\bot},q_{\bot}$ referred in the
literature as soft eikonal approximation that neglects any change in parton
trajectory due to multiple scatterings but assumes a straight line tragectory
throughout. The diffusion of partons in a hot and dense medium can have an
unavoidable link beyond the eikonal approximation and it is worth to relax
eikonal approximation. In this work an attempt has been made to relax part of
this approximation for some of the inelastic processes and their differential
cross-sections in first order noneikonal approximation have been obtained.
Primary estimation indicates $15-20\%$ reduction in the cross section due to
first order noneikonal effect for both the processes in the soft and
intermediate parton energies. These cross-sections naturally reproduce
eikonally approximated results in the eikonal limit for soft emission, i.e.,
$\sqrt{s},E\gg q_{\bot}$ and $q_{\bot}\gg k_{\bot}$. QGP produced at LHC,
where large virtuality scattering processes may be dominant one, is seems to
be $`$less opaque to jets than predicted’ by constrained extrapolations from
RHIC Horowitz:2011gd . There are however other views also, for instance
Renk:2011aa ; Renk:2011gj ; Renk:2012wr , where another set of constrained
extrapolations show considerable variation in the postdictions of RHIC-
constrained scenarios with LHC data. Here this have been taken as a constraint
and cause to disregard class of models which fail to predict/postdict
correctly the uprising behaviour of nuclear modification factor rather than
assigning it as generic surprising feature of LHC data.

Our results indicate some reductions in interaction strengths of jets due to
non-eikonal effects, in soft and intermediate sector. In the soft sector when
the problem is embedded into a hydrodynamically evolving density distribution
this could lead to non-trivial effects. We also show that wide back scattering
with scattering angle more than $\simeq\pm 0.52\pi$ is forbidden in case of
$qq^{\prime}\rightarrow qq^{\prime}g$ when the emitted gluon in soft. This,
however, is not the case for $gg\rightarrow ggg$. In future study we intend
quantitative estimation of this noneikonal effect in jet quenching and other
consequences in heavy-ion collisions phenomenology.

###### Acknowledgements.

Acknowledgments :  I thank Munshi G. Mustafa for valuable discussions with
numerous help during the course of this work and critically reading the
manuscript. I also thank Jan Uphoff for his valuable suggestions and comments.

## References

  * (1) J. Adams et al. [STAR Collaboration], “Experimental and theoretical challenges in the search for the quark gluon plasma: The STAR Collaboration’s critical assessment of the evidence from RHIC collisions,” Nucl. Phys. A 757, 102 (2005) [nucl-ex/0501009].
  * (2) K. Adcox et al. [PHENIX Collaboration], “Formation of dense partonic matter in relativistic nucleus-nucleus collisions at RHIC: Experimental evaluation by the PHENIX collaboration,” Nucl. Phys. A 757, 184 (2005) [nucl-ex/0410003].
  * (3) K. Aamodt et al. [ALICE Collaboration], “Elliptic flow of charged particles in Pb-Pb collisions at 2.76 TeV,” Phys. Rev. Lett. 105, 252302 (2010) [arXiv:1011.3914 [nucl-ex]].
  * (4) K. Aamodt et al. [ALICE Collaboration], “Suppression of Charged Particle Production at Large Transverse Momentum in Central Pb–Pb Collisions at $\sqrt{s_{NN}}=2.76$ TeV,” Phys. Lett. B 696, 30 (2011) [arXiv:1012.1004 [nucl-ex]].
  * (5) C. A. Salgado and U. A. Wiedemann, “Calculating quenching weights,” Phys. Rev. D 68, 014008 (2003) [hep-ph/0302184].
  * (6) S. Wicks, W. Horowitz, M. Djordjevic and M. Gyulassy, “Elastic, inelastic, and path length fluctuations in jet tomography,” Nucl. Phys. A 784, 426 (2007) [nucl-th/0512076].
  * (7) X. -N. Wang and X. -f. Guo, “Multiple parton scattering in nuclei: Parton energy loss,” Nucl. Phys. A 696, 788 (2001) [hep-ph/0102230].
  * (8) S. Jeon and G. D. Moore, “Energy loss of leading partons in a thermal QCD medium,” Phys. Rev. C 71, 034901 (2005) [hep-ph/0309332].
  * (9) M. G. Mustafa, D. Pal, D. K. Srivastava and M. Thoma, “Radiative energy loss of heavy quarks in a quark gluon plasma,” Phys. Lett. B 428, 234 (1998) [nucl-th/9711059].
  * (10) M. Gyulassy, P. Levai and I. Vitev, “Non Abelian energy loss at finite opacity,” Phys. Rev. Lett. 85, 5535 (2000) [nucl-th/0005032].
  * (11) M. G. Mustafa and M. H. Thoma, “Quenching of hadron spectra due to the collisional energy loss of partons in the quark gluon plasma,” Acta Phys. Hung. A 22, 93 (2005) [hep-ph/0311168].
  * (12) G. -Y. Qin, J. Ruppert, C. Gale, S. Jeon, G. D. Moore and M. G. Mustafa, “Radiative and collisional jet energy loss in the quark-gluon plasma at RHIC,” Phys. Rev. Lett. 100, 072301 (2008) [arXiv:0710.0605 [hep-ph]].
  * (13) G. -Y. Qin and A. Majumder, “A pQCD-based description of heavy and light flavor jet quenching,” Phys. Rev. Lett. 105, 262301 (2010) [arXiv:0910.3016 [hep-ph]].
  * (14) R. Abir, U. Jamil, M. G. Mustafa and D. K. Srivastava, “Heavy quark energy loss and D-Mesons at RHIC and LHC energies,” Phys. Lett. B 715, 183 (2012) [arXiv:1203.5221 [hep-ph]]. Zapp:2012ak
  * (15) K. C. Zapp, F. Krauss and U. A. Wiedemann, arXiv:1212.1599 [hep-ph].
  * (16) R. Abir, C. Greiner, M. Martinez, M. G. Mustafa and J.Uphoff, “Soft gluon emission off a heavy quark revisited,” Phys. Rev. D 85, 054012 (2012) [arXiv:1109.5539 [hep-ph]].
  * (17) N. Armesto, B. Cole, C. Gale, W. A. Horowitz, P. Jacobs, S. Jeon, M. van Leeuwen and A. Majumder et al., “Comparison of Jet Quenching Formalisms for a Quark-Gluon Plasma ’Brick’,” arXiv:1106.1106 [hep-ph].
  * (18) Y. Mehtar-Tani, J. G. Milhano and K. Tywoniuk, “Jet physics in heavy-ion collisions,” arXiv:1302.2579 [hep-ph].
  * (19) T. Renk, “Constraining the Physics of Jet Quenching,” Phys. Rev. C 85, 044903 (2012) [arXiv:1112.2503 [hep-ph]].
  * (20) T. Renk, H. Holopainen, R. Paatelainen and K. J. Eskola, “Systematics of the charged-hadron $P_{T}$ spectrum and the nuclear suppression factor in heavy-ion collisions from $\sqrt{s}=200$ GeV to $\sqrt{s}=2.76$ TeV,” Phys. Rev. C 84, 014906 (2011) [arXiv:1103.5308 [hep-ph]].
  * (21) O. Fochler, Z. Xu and C. Greiner, “Energy loss in a partonic transport model including bremsstrahlung processes,” Phys. Rev. C 82, 024907 (2010) [arXiv:1003.4380 [hep-ph]].
  * (22) A. Belyaev, N. D. Christensen and A. Pukhov, “CalcHEP 3.4 for collider physics within and beyond the Standard Model,” arXiv:1207.6082 [hep-ph].
  * (23) N. Armesto, H. Ma, M. Martinez, Y. Mehtar-Tani and C. A. Salgado, “Interference between initial and final state radiation in a QCD medium,” Phys. Lett. B 717, 280 (2012) [arXiv:1207.0984 [hep-ph]].
  * (24) Y. Mehtar-Tani, C. A. Salgado and K. Tywoniuk, “The Radiation pattern of a QCD antenna in a dense medium,” JHEP 1210, 197 (2012) [arXiv:1205.5739 [hep-ph]].
  * (25) J. F. Gunion and G. Bertsch, “Hadronization By Color Bremsstrahlung,” Phys. Rev. D 25, 746 (1982).
  * (26) S. K. Das and J.-e Alam, “Soft gluon multiplicity distribution revisited,” Phys. Rev. D 82, 051502 (2010) [arXiv:1007.4405 [nucl-th]].
  * (27) R. Abir, C. Greiner, M. Martinez and M. G. Mustafa, “Generalisation of Gunion-Bertsch Formula for Soft Gluon Emission,” Phys. Rev. D 83, 011501 (2011) [arXiv:1011.4638 [nucl-th]].
  * (28) T. Bhattacharyya, S. Mazumder, S. K. Das and J.-e Alam, “Examination of the Gunion-Bertsch formula for soft gluon radiation,” Phys. Rev. D 85, 034033 (2012) [arXiv:1106.0609 [nucl-th]].
  * (29) T. S. Biro, E. van Doorn, B. Muller, M. H. Thoma and X. N. Wang, “Parton equilibration in relativistic heavy ion collisions,” Phys. Rev. C 48, 1275 (1993) [nucl-th/9303004].
  * (30) S. J. Parke and T. R. Taylor, “An Amplitude for $n$ Gluon Scattering,” Phys. Rev. Lett. 56, 2459 (1986).
  * (31) J. Auvinen, K. J. Eskola and T. Renk, “A Monte-Carlo model for elastic energy loss in a hydrodynamical background,” Phys. Rev. C 82, 024906 (2010) [arXiv:0912.2265 [hep-ph]].
  * (32) J. Auvinen, K. J. Eskola, H. Holopainen and T. Renk, “Elastic energy loss with respect to the reaction plane in a Monte-Carlo model,” Phys. Rev. C 82, 051901 (2010) [arXiv:1008.4657 [hep-ph]].
  * (33) G. Ovanesyan and I. Vitev, “Medium-induced parton splitting kernels from Soft Collinear Effective Theory with Glauber gluons,” Phys. Lett. B 706, 371 (2012) [arXiv:1109.5619 [hep-ph]].
  * (34) W. A. Horowitz and M. Gyulassy, “The Surprising Transparency of the sQGP at LHC,” Nucl. Phys. A 872, 265 (2011) [arXiv:1104.4958 [hep-ph]].
  * (35) T. Renk, “Understanding LHC jets in the light of RHIC data,” [arXiv:1209.1930 [hep-ph]].


#  ACL 2023
When Not to Trust Language Models: Investigating Effectiveness of Parametric
and Non-Parametric Memories

Alex Mallen∗♢ Akari Asai∗♢ Victor Zhong♢ Rajarshi Das♢ Daniel Khashabi♠
Hannaneh Hajishirzi♢♡ ♢University of Washington ♠Johns Hopkins University
♡Allen Institute for AI
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Despite their impressive performance on diverse tasks, large language models
(LMs) still struggle with tasks requiring rich world knowledge, implying the
difficulty of encoding a wealth of world knowledge in their parameters. This
paper aims to understand LMs’ strengths and limitations in memorizing factual
knowledge, by conducting large-scale knowledge probing experiments on two
open-domain entity-centric QA datasets: PopQA, our new dataset with 14k
questions about long-tail entities, and EntityQuestions, a widely used open-
domain QA dataset. We find that LMs struggle with less popular factual
knowledge, and that retrieval augmentation helps significantly in these cases.
Scaling, on the other hand, mainly improves memorization of popular knowledge,
and fails to appreciably improve memorization of factual knowledge in the long
tail. Based on those findings, we devise a new method for retrieval
augmentation that improves performance and reduces inference costs by only
retrieving non-parametric memories when necessary.111Our code and data are
available at https://github.com/AlexTMallen/adaptive-retrieval.

##  1 Introduction

Large language models (LMs; Brown et al. 2020; Raffel et al. 2020) have been
shown to be competitive on diverse NLP tasks, including knowledge-intensive
tasks that require fine-grained memorization of factual knowledge Chowdhery et
al. (2022); Yu et al. (2022). Meanwhile, LMs have also been shown to have
limited memorization for less frequent entities Kandpal et al. (2022), are
prone to hallucinations Shuster et al. (2021), and suffer from temporal
degradation Kasai et al. (2022); Jang et al. (2022). Incorporating _non-
parametric knowledge_ (i.e., retrieved text chunks) largely helps address
those issues stemming from reliance on LMs’ _parametric knowledge_ —knowledge
stored in their parameters Izacard et al. (2022b)—but it is unclear whether it
is strictly superior or complementary to parametric knowledge. Understanding
when we should _not_ trust LMs’ outputs is also crucial to safely deploying
them in real-world applications Kadavath et al. (2022).

Figure 1:  Relationship between subject entity popularity in a question and
GPT-3 performance in open-domain QA, with and without retrieved passages.
Adaptive Retrieval only retrieves when necessary (orange bars) based on the
heuristically-decided threshold (red line).

This work conducts a large-scale knowledge probing of LMs on factual knowledge
memorization, to understand when we should and should _not_ rely on LMs’
parametric knowledge, and how scaling and non-parametric memories (e.g.,
retrieval-augmented LMs) can help. In particular, we aim to address the
following research questions:

  1. ($RQ_{1}$)

How much factual knowledge is memorized by LMs and what factors affect the
memorization? (Section 4)

  2. ($RQ_{2}$)

To what extent can non-parametric memories alleviate the shortcomings of
parametric memories of LMs? (Section 5)

  3. ($RQ_{3}$)

Can we build a system to adaptively combine non-parametric and parametric
memories? (Section 6)

We hypothesize that factual knowledge frequently discussed on the web is
easily memorized by LMs, while the knowledge that is less discussed may not be
well captured and thus they require retrieving external non-parametric
memories. We evaluate ten large LMs of three families (i.e., GPT-Neo, OPT, and
GPT-3) with varying scales on the open-domain question answering (QA) task in
a zero- or few-shot prompting manner. We construct a new dataset, PopQA,
consisting of 14k questions to cover factual information in the long tail that
might have been missed in popular QA datasets Kwiatkowski et al. (2019). We
use Wikipedia page views as a measure of popularity and convert knowledge
triples from Wikidata, with diverse levels of popularity, into natural
language questions, anchored to the original entities and relationship types.
We also use EntityQuestions Sciavolino et al. (2021), an open-domain QA
dataset with a long-tail distribution.

On both datasets, LMs’ memorization ($RQ_{1}$) is often limited to the popular
factual knowledge and even GPT-3 davinci-003 fails to answer the majority of
the long-tail questions. Moreover, on such questions, scaling up models does
_not_ significantly improve the performance (e.g., for the 4,000 least popular
questions in PopQA, GPT-j 6B has 16% accuracy and GPT-3 davinci-003 has 19%
accuracy). This also suggests that we can predict if LMs memorize certain
knowledge based on the information presented in the input question only.

We next investigate whether a semi-parametric approach that augments LMs with
retrieved evidence can mitigate the low performance on questions about less
popular entities ($RQ_{2}$). Non-parametric memories largely improve
performance on long-tail distributions across models. Specifically, we found
that retrieval-augmented LMs are particularly competitive when subject
entities are not popular: a neural dense retriever Izacard et al.
(2022a)-augmented GPT-neo 2.7B outperforms GPT-3 davinci-003 on the 4,000
least popular questions. Surprisingly, we also find that retrieval
augmentation can hurt the performance of large LMs on questions about popular
entities as the retrieved context can be misleading.

As a result, we devise a simple-yet-effective retrieval-augmented LM method,
Adaptive Retrieval, which adaptively combines parametric and non-parametric
memories based on popularity ($RQ_{3}$). This method further improves
performance on PopQA by up to 10%, while significantly reducing the inference
costs, especially with larger LMs (e.g., reducing GPT-3 API costs by half),
indicating the potential for future research in more efficient and powerful
retrieval-augmented LMs.

##  2 Related Work

#### Parametric and non-parametric knowledge.

Petroni et al. (2019) demonstrate that large pre-trained LMs such as BERT
Devlin et al. (2019) memorize the significant amount of world knowledge in
their parameters (parametric knowledge), and Roberts et al. (2020) show that
fine-tuned T5 without any reference documents (closed-book QA) can achieve
competitive performance on open-domain QA. More recent and powerful LMs Brown
et al. (2020); Chowdhery et al. (2022) further improve performance on diverse
knowledge-intensive tasks, leveraging their strong parametric memories Kandpal
et al. (2022); Yu et al. (2022). However, relying solely on their parameters
to encode a wealth of world knowledge requires a prohibitively large number of
parameters and the knowledge can become obsolete quickly Kasai et al. (2022);
Jang et al. (2022). Recent work shows that augmenting LMs with non-parametric
memories (i.e., retrieved text chunks) enables much smaller models to match
the performance of larger models Izacard et al. (2022b); Khandelwal et al.
(2020); Min et al. (2022), although Chen et al. (2022) and Longpre et al.
(2021) show that even those models can ignore non-parametric knowledge and
rely on parametric knowledge.

#### Understanding memorization.

Several prior work establishes a positive relationship between string
frequency in pre-training corpora and memorization Carlini et al. (2022);
Razeghi et al. (2022). Concurrent to our work, Kandpal et al. (2022) show that
the co-occurrence of the question and answer entities in pretraining corpora
has a positive correlation with models’ QA accuracy on popular open-domain QA
benchmarks such as Natural Questions Kwiatkowski et al. (2019). This work,
instead, attempts to predict memorization using the variables available in the
input question only and uses popularity to obtain a proxy for how frequently
an entity is likely to be discussed on the web. Importantly, by constructing a
new dataset, we can conduct fine-grained controlled experiments across a wide
range of popularities, allowing the investigation of hypotheses that might
have been missed in prior analysis using existing open QA datasets. We further
analyze the effectiveness and limitations of retrieval-augmented LMs and
introduce Adaptive Retrieval. Prior work investigates the effectiveness of
deciding when to use non-parametric memories at the token level in $k$NN LM He
et al. (2021); Drozdov et al. (2022). This work is the first work to study the
effectiveness of deciding whether to retrieve for each query and show their
effectiveness in retrieval-augmented LM prompting.

Figure 2: PopQA is created by sampling knowledge triples from Wikidata and
converting them to natural language questions, followed by popularity
calculation.

##  3 Evaluation Setup

We evaluate LMs’ ability to memorize factual knowledge through closed-book QA
tasks with few-shot samples. We evaluate LMs on our new dataset, PopQA (Figure
2), and EntityQuestions, both of which have long-tail distributions (Figure
3).

Figure 3: Distribution of subject entity popularity for EntityQuestions,
PopQA, and for NQ-open for reference. Details on NQ entities can be found in
Appendix A.

###  3.1 Focus and Task

Focus: factual knowledge.  Among diverse types of world knowledge, this work
focuses on factual knowledge Adams (2015) of entities—knowledge about specific
details of the target entities. We define factual knowledge as a triplet of
(subject, relationship, object) as in Figure 2 left.

Task format: open-domain QA.  We formulate the task as open-domain QA Roberts
et al. (2020): given a question, a model predicts an answer without any pre-
given ground-truth paragraph.222Some work conducts knowledge probing of
encoder-only models by filling out [MASK] tokens Petroni et al. (2019). We use
decoder-only models and thus do not use this fill-in-the-blank scheme. As in
Kandpal et al. (2022), we study few-shot settings and prompt LMs without any
parameter updates, instead of fine-tuning them on QA datasets such as in
Roberts et al. (2020).

Metrics: accuracy. We mark a prediction as correct if any substring of the
prediction is an exact match of any of the gold answers.

###  3.2 Dimensions of Analysis

We hypothesize that factual knowledge that is less frequently discussed on the
web may not be well-memorized by LMs. Previous research often uses the term
frequency of object entities in pretraining corpora to understand memorization
Févry et al. (2020); Kandpal et al. (2022); Razeghi et al. (2022). Instead, we
investigate whether it’s possible to predict memorization based on the input
information only, and then apply the findings for modeling improvements,
unlike prior analyses. Therefore, our work focuses on the other two variables
in a factual knowledge triple: the subject entity and the relationship type.

Subject entity popularity. We use the popularity of the entities measured by
Wikipedia monthly page views as a proxy for how frequently the entities are
likely to be discussed on the web, instead of using the occurrence of entities
or strings in the pretraining corpus Carlini et al. (2022); Kandpal et al.
(2022); Razeghi et al. (2022). Calculating frequencies over large pretraining
corpora requires massive computations to link entities over billions of
tokens, or can result in noisy estimations.333Moreover, several recent models
like GPT-3 do not release their pretraining corpora, and it is an open
question whether the frequencies in pretraining corpora reflect the
frequencies in their private corpora.  Our initial studies show that this is
much cheaper444We can get page views by calling Wikipedia API.  and aligns
well with our intuition.

Relationship type. We also consider the relationship types as key factors for
factual knowledge memorization. For example, even given the same combinations
of the subject and object entities, model performance can depend on the
relationship types; relationship types widely discussed can be easier to be
memorized, while types that are less discussed may not be memorized much.

###  3.3 Benchmarks

PopQA. In our preliminary studies, we found that existing common open-domain
QA datasets such as Natural Questions (NQ; Kwiatkowski et al. 2019) are often
dominated by subject entities with high popularity, and it is often hard to
identify relationship types due to diverse question surface forms. To enable a
fine-grained analysis of memorization based on the aforementioned analysis
dimensions, we construct PopQA, a new large-scale entity-centric open-domain
QA dataset about entities with a wide variety of popularity, as shown in
Figure 3.

To construct PopQA, we randomly sample knowledge triples of 16 diverse
relationship types from Wikidata and convert them into natural language
questions, using a natural language template (depicted in Figure 2). We
verbalize a knowledge triple $(S,R,O)$ into a question that involves
substituting the subject $S$ into a template manually written for the
relationship type $R$. The full list of templates is found in Table 2 of the
Appendix. The set of acceptable answers to the question is the set of entities
$E$ such that $(S,R,E)$ exists in the knowledge graph. We tried various
templates and found that the results were fairly robust to the templates.
Since PopQA is grounded to a knowledge base, links to Wikidata entities allow
for reliable analysis of popularity and relationship types.

EntityQuestions. We test on another popular open-domain QA dataset,
EntityQuestions Sciavolino et al. (2021), which also covers a long-tail entity
distribution. They use Wikipedia hyperlink counts as a proxy of the frequency
of entities and sample knowledge triples from WikiData, from the frequency
distributions. Unlike PopQA, EntityQuestions doesn’t provide entity
annotations, so we only use 82% of the questions, where the mention of the
subject entity has a unique match with a Wikidata entity.

##  4 Memorization Depends on Popularity and Relationship Type

We evaluate a range of LMs with varying numbers of parameters, to quantify how
much factual knowledge they memorize and how different factors affect those
memorization behaviors ($RQ_{1}$).

###  4.1 Experimental Setup

#### Models.

We evaluate ten models with a varying scale of model size: OPT (Zhang et al.
2022; 1.3, 2.7, 6.7, and 13 billion), GPT-Neo (Black et al. 2022; 1.3, 2.7, 6,
and 20 billion), and GPT-3 (Brown et al. 2020; davinci-002, davinci-003) on
our benchmark without any fine-tuning.555We did not explore widely-used
encoder-decoder models such as T5, as their supervised pretraining consists of
QA.

#### Instructions and demonstrations.

We use a simple template “Q: <question> A:” to format all of our questions for
generative prediction. More sophisticated instructions were attempted in
preliminary experiments but they did not improve upon the simple template
significantly enough to merit using them, especially given that they may
overfit to the model. While we use zero-shot prompting for GPT-3 to reduce API
costs,666Using 15-shot prompts for GPT-3 would cost upwards of $3000 for the
combination of vanilla, Contriever, BM25, and GenRead evaluations on
davinci-002 and davinci-003. we use 15-shot prompting for all GPT-neo and OPT
models.

Figure 4: Per relationship type ($n=$ the number of questions) results on
PopQA by model, showing overall accuracy and the correlation between accuracy
and log popularity. We report the correlation between the question’s subject
entity log-popularity and the binary indicator of whether the question was
answered correctly. We see that both subject entity popularity and
relationship type are strong predictors of memorization across models. The
correlation with popularity exists across relationship types and is stronger
for larger LMs. We show a representative subset of relationship types and the
complete results are in Figures 16 and 17 in Appendix C.1, including results
on EntityQuestions.

###  4.2 Results

#### Overall model performance.

The top left column of Figure 4 illustrates the overall performance on PopQA.
As shown, even without using in-context examples, larger LMs exhibit
reasonable performance: GPT-3 achieves 35% accuracy, and GPT-Neo 20B achieves
25% accuracy. This indicates that large LMs memorize factual knowledge in
their parameters to some extent. This section examines which types of
knowledge are better memorized and what factors influence memorization.

#### Subject entity popularity predicts memorization.

Figure 4 (bottom) shows that there is a positive correlation between subject
entity popularity and models’ accuracy for almost all relationship types. This
supports our hypothesis that subject entity popularity can be a reliable
indicator of LMs’ factual knowledge memorization. In general, the correlations
between subject entity popularity and accuracy are stronger for larger LMs;
GPT-3 003 shows the highest positive correlation (roughly 0.4) while GPT-
Neo-1.3B shows relatively weak positive correlations (approximately 0.1).

#### Relationship types affects memorization.

We find that models have a higher average performance for some relationship
types than for others. While this is evidence that factual knowledge of some
relationship types are more easily memorized than others, we also observe that
questions of certain relationship types can be easily guessed without
memorizing the knowledge triple. Specifically, certain relationship types
(e.g., nationalities) allow models to exploit surface-level artifacts in
subject entity names Poerner et al. (2020); Cao et al. (2021). Additionally,
models often output the most dominant answer entities for questions about
relationship types with fewer answer entities (e.g., red for the color
relationship type). In Figure 4, relationships with lower correlation (e.g.,
country, sport) often shows higher accuracy, indicating that on those
relationship types, models may exploit surface-level clues. On the other hand,
for relationship types with relatively low accuracy (e.g., occupation, author,
director), larger LMs often show a high correlation. Further details are in
Appendix C.1.

#### Scaling may not help with tail knowledge.

As seen in the left column of Figure 4, there are clear overall performance
improvements with scale on the PopQA dataset. However, Figure 5 shows that on
both PopQA and EntityQuestions, most of scaling’s positive effect on
parametric knowledge comes from questions with high popularity. Specifically,
for the questions about the entities whose $\log_{10}{(\rm popularity)}$ is
larger than 4, there is an improvement in accuracy as model size increases
(red and yellow lines), while performance on questions with lower popularity
remains relatively constant (blue and green lines). For the 4,000 least
popular questions, GPT-Neo 6B, 20B, and GPT-3 davinci-003 have 15%, 16%, and
19% accuracy, respectively.

Figure 5: PopQA scaling results, broken down by question popularity level.
Scaling mostly improves memorization of more popular factual knowledge. Error
bars are 95% confidence intervals.

This somewhat dampens prior works’ findings that scaling up models
significantly improves their factual knowledge memorization Roberts et al.
(2020); Kandpal et al. (2022). We hypothesize that this is because their
evaluations are often conducted on QA datasets with popular entities.
66footnotetext: 30 PopQA and 26 EntityQuestions questions had popularity less
than the smallest popularity bin, and are excluded to avoid showing results
for small sample sizes.  In sum, scaling lowers the threshold of popularity
for knowledge to be reliably memorized, but is not projected to move the
threshold far into the long tail for practical model scales.

#### Relationship type results breakdown.

Figure 6 provides a closer look at the relationship between popularity,
accuracy, and relationship type; it shows model accuracy over the popularity
distributions for director and country. For the first two types, we can see a
clear positive trend between popularity and accuracy across models, and as the
model size gets larger, the LMs memorize more. On the other hand, in the
“country” relationship type, no models show trends, while overall the accuracy
is high, indicating the LMs often exploit artifacts to answer less popular
questions. We show example models’ predictions in Appendix Section C.3.

##  5 Non-parametric Memory Complements Parametric Memory

Our analysis indicates that even the current state-of-the-art LMs struggle
with less popular subjects or certain relationship types, and increasing the
model size does not lead to further performance improvements. In light of
this, we extend our analysis to non-parametric sources of knowledge, as
outlined in Section ($RQ_{2}$). Specifically, we investigate the effectiveness
of retrieval-augmented LMs Borgeaud et al. (2022); Lewis et al. (2020), which
leverage non-parametric memories (i.e., retrieved text) to improve
performance.

###  5.1 Experimental Setup

Augmenting input. In this work, we try a simple retrieval-augmented LM
approach, where we run an off-the-shelf retrieval system off-line to retrieve
context from Wikipedia relevant to a question,777We use Wikipedia dump from
December 2018. and then we concatenate the retrieved context with the original
question. Although increasing the context size often leads to performance
gains Izacard and Grave (2021); Asai et al. (2022), we only use the top one
retrieved paragraph for simplicity.

Figure 6:  Memorization versus popularity for three models and the
relationship types with the largest and smallest correlations. Within a
relationship type, generally, there is a monotonically increasing link between
popularity and performance, except for “country”. Error bars show Wilson 95%
confidence intervals.

Retrieval models. We use two widely-used retrieval systems: BM25 Robertson et
al. (2009) and Contriever Izacard et al. (2022a). BM25 is a static term-based
retriever without training, while Contriever is pretrained on large unlabeled
corpora, followed by fine-tuning on MS MARCO Bajaj et al. (2016). We also
experiment with a _parametric_ augmentation method, GenRead Yu et al. (2022),
which prompts LMs to generate rather than retrieve a contextual document to
answer a question. We use the ten LMs in Section 4, resulting in 40 LMs and
retrieval-augmented LMs.

Figure 7: PopQA accuracy of LMs augmented with BM25, Contriever, GenRead, and
unassisted (vanilla). Retrieving non-parametric memories significantly
improves the performance of smaller models.  Complete results on PopQA are
found in Figure 13. EntityQuestions results are in Figure 14 of the Appendix.

###  5.2 Results

#### Retrieval largely improves performance.

Figure 7 shows that augmenting LMs with non-parametric memories significantly
outperforms unassisted vanilla LMs. A much smaller LM (e.g., GPT-Neo 2.7B)
augmented by the Contriever retrieval results outperforms vanilla GPT-3. Large
LMs such as GPT-3 also enjoy the benefits of non-parametric memories.
Contriever gives 7% accuracy gains on top of GPT-3 davinci-003. GenRead shows
little-to-no performance improvement over vanilla parametric knowledge for
smaller models, while the technique shows sizeable gains for GPT-3, especially
davinci-003. In addition to its limited effectiveness with smaller LMs,
GenRead has potentially prohibitive inference time costs, with GPT-NeoX 20B
taking 70 seconds per query.

Figure 8:  GPT-3 davinci-003 accuracy versus relative popularity (how popular
a question is relative to other questions of its relationship type).
Retrieval-augmented LMs (dashed) outperform LMs’ parametric memory (solid) for
less popular entities, while parametric memory is competitive for more popular
entities. Relative popularity is defined as the log-popularity of a question,
normalized by the mean and standard deviation of log-popularity for the
question’s relationship type (smaller for less popular entities).999Error bars
show Wilson 95% confidence intervals. Bins with less than 40 samples have been
excluded to avoid showing results with exceedingly wide errorbars. Figure 17
shows per-relationship results.

#### Non-parametric memories are effective for less popular facts.

How does retrieval augmentation lead to such significant improvements? Figure
9 shows the relationship between the entity popularity and models’ QA
performance. It can be seen that retrieval-augmented LMs guided by Contriever
or BM25 have a clear advantage over unassisted vanilla LMs, especially on less
popular entities, resulting in a significant performance gain. Overall,
Contriever-guided LMs outperform BM25-based ones on PopQA, while the
BM25-based models perform better on the least popular entities, consistent
with the findings from Sciavolino et al. (2021). On the other hand, for more
popular entities, parametric knowledge shows equal or higher accuracy,
indicating that the state-of-the-art LMs have already memorized the answers,
and augmenting input with retrieved-context doesn’t help much or even hurts
the performance. Interestingly, GenRead generally outperforms vanilla LMs
despite relying on LMs’ parametric memory. This demonstrates the effectiveness
of elicitive prompting Wei et al. (2022); Sun et al. (2022) as observed in
prior work. However, like vanilla LMs, GenRead shows low performance on less
popular entities.

| Contriever-augmented LM
---|---
| succeeded | failed
LM succeeded | 0.83 (24%) | 0.14 (10%)
LM failed | 0.88 (17%) | 0.11 (49%)
Table 1:  The recall@1 of Contriever for questions that GPT-3 davinci-003
answered correctly and incorrectly with and without retrieval on PopQA. The
percent of questions falling in each category is shown in parentheses. For 10%
of questions, retrieval is _harmful_ due to low-quality retrieved text (0.14
recall@1).

#### Non-parametric memories can mislead LMs.

We conduct an in-depth analysis of why retrieval-augmented models suffer in
more popular entities. We hypothesize that retrieval results may not always be
correct or helpful, and can mislead LMs. To test this hypothesis, we group the
questions based on two axes: whether unassisted GPT-3 davinci-003 predict
correctly or not, and whether retrieval-augmented predictions are correct or
not. For each of the four categories, we calculate recall@1 (whether a gold
answer is included in the top 1 document; Karpukhin et al. 2020).

Table 1 shows recall@1 for each group with percentages of the questions
falling into each of the categories. For 10% of questions, retrieval-
augmentation causes the LM to incorrectly answer a question it could otherwise
answer correctly. We found that on those questions, recall@1 is significantly
lower than the overall recall@1 (0.14 vs 0.42 overall), indicating that failed
retrieval can result in performance drops. Conversely, for the 17% of
questions for which retrieval causes the LM to correctly answer a question it
would otherwise have failed to answer, the recall@1 is 0.88. We include
examples of both cases in Appendix Section C.3.

##  6 Adaptive Retrieval: Using Retrieval Only Where It Helps

While incorporating non-parametric memories helps in long-tail distributions,
powerful LMs have already memorized factual knowledge for popular entities,
and retrieval augmentation can be harmful. As outlined in ($RQ_{3}$), can we
achieve the best of both worlds? We propose a simple-yet-effective method,
Adaptive Retrieval, which decides when to retrieve passages only based on
input query information and augments the input with retrieved non-parametric
memories only when necessary. We show that this is not only more powerful than
LMs or retrieval-augmented LMs always retrieving context, but also more
efficient than the standard retrieval-augmented setup.

###  6.1 Method

Adaptive Retrieval is based on our findings: as the current best LMs have
already memorized more popular knowledge, we can use retrieval only when they
do not memorize the factual knowledge and thus need to find external non-
parametric knowledge. In particular, we use retrieval for questions whose
popularity is lower than a threshold (popularity threshold), and for more
popular entities, do not use retrieval at all.

Using a development set, the threshold is chosen to maximize the adaptive
accuracy, which we define as the accuracy attained by taking the predictions
of the retrieval-augmented system for questions below the popularity threshold
and the predictions based on parametric knowledge for the rest. We determine
the popularity threshold independently for each relationship type.

Figure 9: PopQA performance of GPT-neo models and GPT3 davinci-003, with
different retrieval methods. Adaptive Retrieval robustly outperforms
approaches that always retrieve, especially for larger LMs.

###  6.2 Results

#### Adaptive Retrieval improves performance.

Figure 9 shows the results when we adaptively retrieve non-parametric memories
based on the per-relationship type thresholds. We can see that adaptively
retrieving non-parametric memories is effective for larger models. The best
performance on PopQA is using GPT-3 davinci-003 adaptively with GenRead and
Contriever, yielding 46.5% accuracy, 5.3% higher than any non-adaptive method.

Figure 10: The proportion of questions for which various models use retrieval
in the Adaptive Retrieval setup on PopQA. When using Adaptive Retrieval, small
models must still rely on non-parametric memory for most questions, while
larger models have more reliable parametric memories enabling them to use
retrieval less often.

#### The threshold shifts with LM scale.

While Adaptive Retrieval shows performance gains for larger models, smaller
models do not realize the same benefits; as shown in Figure 9, the performance
gain from Adaptive Retrieval is much smaller when we use models smaller than
10 billion. Why does this happen? Figure 10 shows that smaller LMs almost
always retrieve, indicating that there are not many questions for which small
LMs’ parametric knowledge is more reliable than non-parametric memory. In
contrast, large models typically retrieve much less. For example, GPT-3
davinci-003 only retrieves for 40% of questions when paired with BM25, and
even the much smaller GPT-neox 20B does not retrieve documents on more than
20% of the questions. On EntityQuestions (Appendix Figure 15) all of the LMs
retrieve much more, as the questions are mostly about less popular entities.

#### Adaptive Retrieval reduces inference-time costs.

We also found that Adaptive Retrieval improves efficiency; if we know we do
not need to retrieve documents, we can skip retrieval components and the input
length becomes shorter, which improves latency in both retrieval and language
model components. Figure 11 shows the inference latency of GPT-J 6B and GPT-
neox 20B, and API costs of GPT-3. Especially for larger LMs, concatenating
retrieved context results in significantly increased latency (e.g., for GPT-J
6B, the inference time latency almost doubles). Adaptive retrieval enables
reducing inference time up to 9% from standard retrieval. We also observe cost
reduction on EntityQuestions, as shown in Figure 12.

Figure 11: PopQA latency for large GPT-neo models that were run on our
machines, and API costs for GPT3. Adaptive retrieval reduces latency and API
costs. Figure 12: Accuracy and cost savings of Adaptive Retrieval for
EntityQuestions. Despite EntityQuestions’s lack of popular entities (see
Figure 3), Adaptive Retrieval is able to reduce API costs by 15% while
maintaining equivalent performance to retrieval only.

##  7 Discussion and Conclusions

This work conducts large-scale knowledge probing to examine the effectiveness
and limitations of relying on LMs’ parameters to memorize factual knowledge
and to understand what factors affect factual knowledge memorization. Our
results show that memorization has a strong correlation with entity popularity
and that scaling up models on long-tail distributions may only provide
marginal improvements. We also demonstrate that non-parametric memories can
greatly aid LMs on these long-tail distributions, but can also mislead LMs on
questions about well-known entities, as powerful LMs have already memorized
them in their parameters. Based on those findings, we devise simple-yet-
effective Adaptive Retrieval, which only retrieves when necessary, using a
heuristic based on entity popularity and relationship types. Our experimental
results show that this method is not only more powerful than LMs or previous
retrieval-augmented LMs but also more efficient.

## Limitations

This work focuses on entity-centric factual knowledge and demonstrates that
LMs’ memorization is heavily affected by the popularity of the entities and
the aspect of the entities being asked in the questions. It is important to
emphasize that for running controlled experiments, we have relied on two
synthetic datasets, and the extent to which our results apply to naturally
occurring factual knowledge has not been firmly established. While we can be
fairly confident about the relationship between scaling, retrieval,
popularity, relationship type, and performance for the kinds of knowledge
studied here, the effectiveness of Adaptive Retrieval will depend on many
details of the question answering pipeline. Moreover, our work depends on a
definition of popularity that is time-dependent and may not perfectly reflect
how frequently entities are discussed on the web. Wikipedia page views are one
possible definition of popularity for which we observe our results, and we
invite others to improve upon it in future work. Further research can expand
upon this simple approach, perhaps drawing on insights from Kadavath et al.
(2022) to improve the effectiveness of Adaptive Retrieval.

It is an open question if the same findings are applicable to other types of
world knowledge such as commonsense. We conjecture that the concept of the
subject topic (entity), as well as the aspect (relationship type), can be
applied with some minor modifications, which future work can quantify
memorization following our scheme.

## Ethical Considerations

Recent work Huang et al. (2022) shows that LMs memorize personal information
available on the web, which has significant security issues. Our evaluation
focuses on the memorization of general entity-centric knowledge, but our
findings can be applicable to those areas. Our findings suggest that LMs are
likely to have less reliable knowledge of minority groups. Parrish et al.
(2022) established that models often rely on stereotypes to answer in
uncertain cases, so our results indicate that LMs are likely to rely on
stereotypes disproportionately for minority groups. Future work could
investigate whether retrieval augmentation reduces bias in these cases.

## Acknowledgements

We thank the UW NLP group members for their helpful discussions, and Joongwon
Kim, Wenya Wang, and Sean Welleck for their insightful feedback on this paper.
This research was supported by NSF IIS-2044660, ONR N00014-18-1-2826, ONR MURI
N00014- 18-1-2670, and Allen Distinguished Award. AM is funded by a Goldwater
Scholarship and AA is funded by the IBM PhD Fellowship.

## References

  * Adams (2015) Nancy E Adams. 2015.  Bloom’s taxonomy of cognitive learning objectives.  _Journal of the Medical Library Association_.
  * Asai et al. (2022) Akari Asai, Matt Gardner, and Hannaneh Hajishirzi. 2022.  Evidentiality-guided generation for knowledge-intensive NLP tasks.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_.
  * Bajaj et al. (2016) Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. 2016.  MS MARCO: A human generated machine reading comprehension dataset.
  * Black et al. (2022) Sidney Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, Usvsn Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. 2022.  GPT-NeoX-20B: An open-source autoregressive language model.  In _Proceedings of BigScience Episode #5 – Workshop on Challenges & Perspectives in Creating Large Language Models_.
  * Borgeaud et al. (2022) Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego De Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack Rae, Erich Elsen, and Laurent Sifre. 2022\.  Improving language models by retrieving from trillions of tokens.  In _Proceedings of the 39th International Conference on Machine Learning_.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing systems_.
  * Cao et al. (2021) Boxi Cao, Hongyu Lin, Xianpei Han, Le Sun, Lingyong Yan, Meng Liao, Tong Xue, and Jin Xu. 2021.  Knowledgeable or educated guess? revisiting language models as knowledge bases.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing_.
  * Carlini et al. (2022) Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramèr, and Chiyuan Zhang. 2022.  Quantifying memorization across neural language models.
  * Chen et al. (2022) Hung-Ting Chen, Michael JQ Zhang, and Eunsol Choi. 2022.  Rich knowledge sources bring complex knowledge conflicts: Recalibrating models to reflect conflicting evidence.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022.  PaLM: Scaling language modeling with pathways.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_.
  * Drozdov et al. (2022) Andrew Drozdov, Shufan Wang, Razieh Rahimi, Andrew McCallum, Hamed Zamani, and Mohit Iyyer. 2022.  You can’t pick your neighbors, or can you? when and how to rely on retrieval in the kNN-LM.  In _Findings of EMNLP_.
  * Ferragina and Scaiella (2010) Paolo Ferragina and Ugo Scaiella. 2010.  TAGME: on-the-fly annotation of short text fragments (by wikipedia entities).  In _Proceedings of the 19th ACM international conference on Information and knowledge management_.
  * Févry et al. (2020) Thibault Févry, Livio Baldini Soares, Nicholas FitzGerald, Eunsol Choi, and Tom Kwiatkowski. 2020.  Entities as experts: Sparse memory access with entity supervision.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing_.
  * He et al. (2021) Junxian He, Graham Neubig, and Taylor Berg-Kirkpatrick. 2021.  Efficient nearest neighbor language models.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_.
  * Huang et al. (2022) Jie Huang, Hanyin Shao, and Kevin Chen-Chuan Chang. 2022.  Are large pre-trained language models leaking your personal information? In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing_.
  * Izacard et al. (2022a) Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022a.  Unsupervised dense information retrieval with contrastive learning.  _Transactions on Machine Learning Research_.
  * Izacard and Grave (2021) Gautier Izacard and Edouard Grave. 2021.  Leveraging passage retrieval with generative models for open domain question answering.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics_.
  * Izacard et al. (2022b) Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2022b.  Few-shot learning with retrieval augmented language models.
  * Jang et al. (2022) Joel Jang, Seonghyeon Ye, Changho Lee, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, and Minjoon Seo. 2022.  Temporalwiki: A lifelong benchmark for training and evaluating ever-evolving language models.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing_.
  * Kadavath et al. (2022) Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield Dodds, Nova DasSarma, Eli Tran-Johnson, et al. 2022.  Language models (mostly) know what they know.
  * Kandpal et al. (2022) Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2022\.  Large language models struggle to learn long-tail knowledge.
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020.  Dense passage retrieval for open-domain question answering.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing_.
  * Kasai et al. (2022) Jungo Kasai, Keisuke Sakaguchi, Yoichi Takahashi, Ronan Le Bras, Akari Asai, Xinyan Yu, Dragomir Radev, Noah A Smith, Yejin Choi, and Kentaro Inui. 2022.  Realtime QA: What’s the answer right now?
  * Khandelwal et al. (2020) Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020\.  Generalization through memorization: Nearest neighbor language models.  In _International Conference on Learning Representations_.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.  Natural questions: A benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_.
  * Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  In _Advances in Neural Information Processing Systems_.
  * Longpre et al. (2021) Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. 2021.  Entity-based knowledge conflicts in question answering.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 7052–7063, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Min et al. (2022) Sewon Min, Weijia Shi, Mike Lewis, Xilun Chen, Wen-tau Yih, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022.  Nonparametric masked language model.
  * Parrish et al. (2022) Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman. 2022.  BBQ: A hand-built bias benchmark for question answering.  In _Findings of the Association for Computational Linguistics: ACL 2022_.
  * Petroni et al. (2019) Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019.  Language models as knowledge bases? In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing_.
  * Poerner et al. (2020) Nina Poerner, Ulli Waltinger, and Hinrich Schütze. 2020.  E-BERT: Efficient-yet-effective entity embeddings for BERT.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_.
  * Razeghi et al. (2022) Yasaman Razeghi, Robert L Logan IV, Matt Gardner, and Sameer Singh. 2022.  Impact of pretraining term frequencies on few-shot reasoning.
  * Roberts et al. (2020) Adam Roberts, Colin Raffel, and Noam Shazeer. 2020.  How much knowledge can you pack into the parameters of a language model? In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing_.
  * Robertson et al. (2009) Stephen Robertson, Hugo Zaragoza, et al. 2009.  The probabilistic relevance framework: Bm25 and beyond.  _Foundations and Trends in Information Retrieval_.
  * Sciavolino et al. (2021) Christopher Sciavolino, Zexuan Zhong, Jinhyuk Lee, and Danqi Chen. 2021.  Simple entity-centric questions challenge dense retrievers.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_.
  * Shuster et al. (2021) Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021.  Retrieval augmentation reduces hallucination in conversation.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_.
  * Sun et al. (2022) Zhiqing Sun, Xuezhi Wang, Yi Tay, Yiming Yang, and Denny Zhou. 2022.  Recitation-augmented language models.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. 2022.  Chain of thought prompting elicits reasoning in large language models.  In _Advances in Neural Information Processing Systems_.
  * Yu et al. (2022) Wenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. 2022.  Generate rather than retrieve: Large language models are strong context generators.
  * Zeng et al. (2022) Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. 2022.  Glm-130b: An open bilingual pre-trained model.
  * Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022.  Opt: Open pre-trained transformer language models.

## Appendix

##  Appendix A Details of PopQA Constructions

#### List of the relationship types and templates.

In this work, we use the following 16 relationship types, and the authors of
this paper manually annotated templates to verbalize knowledge triple to
natural language questions. We show the final list of the templates used to
create PopQA in Table 2.

Figure 3 shows the distribution of subject popularity of PopQAand
EntityQuestions versus the popular NQ benchmark. NQ may have multiple entities
so the distribution of the least popular entity per question is shown. Subject
entities from NQ were extracted using TagMe Ferragina and Scaiella (2010) on
the NQ-open development set with a score threshold of 0.22. TagMe returns the
title of a Wikidata entity which can be directly used to find popularity.

Relationship | Template
---|---
occupation | What is [subj] ’s occupation?
place of birth | In what city was [subj] born?
genre | What genre is [subj]?
father | Who is the father of [subj] ?
country | In what country is [subj] ?
producer | Who was the producer of [subj] ?
director | Who was the director of [subj] ?
capital of | What is [subj] the capital of?
screenwriter | Who was the screenwriter for [subj] ?
composer | Who was the composer of [subj] ?
color | What color is [subj] ?
religion | What is the religion of [subj] ?
sport | What sport does [subj] play?
author | Who is the author of [subj] ?
mother | Who is the mother of [subj] ?
capital | What is the capital of [subj] ?
Table 2: Full list of the manually annotated templated used for
PopQAcreations. [subj] denotes a placeholder for subject entities.

#### Knowledge triples sampling.

In the construction of the PopQAdataset, knowledge triples are sampled with
higher weight given to more popular entities, otherwise, the distribution
would be dominated by the tail and we would not have enough high-popularity
entities to complete our analysis. Specifically, when considering whether to
sample a particular knowledge triple, we include the knowledge triple if and
only if $f>\exp(8R-6)$, where $R\sim U(0,1)$ is a unit uniform pseudo-random
number and $f$ is the exact match term frequency of the subject entity’s
aliases in an 800 MB random sample of C4. To increase diversity, once 2000
knowledge triples of a particular relation type have been sampled, they are no
longer sampled.

##  Appendix B Experimental Details

#### Computational resources and API costs.

GPT-3 API usage totaled to $275. We ran 14,282 questions through two GPT-3
davinci models using four different methods: vanilla experiments cost $13
($0.46 per 1000 questions), Contriever-augmented experiments cost $88 ($3.08
per 1000 questions), BM25-augmented experiments cost $81 ($2.80 per 1000
questions), and GenRead experiments cost $93 ($3.25 per 1000 questions).

To run experiments using LMs larger than two billion parameters, we use a
single V100 Volta GPU with 32GB GPU memories. We use int8bit Zeng et al.
(2022) quantization with OPT 13 billion and GPT-Neo 20 billion models to make
them fit our GPUs. In our preliminary experiments using GPT-Neo 6 billion, we
did not observe a notable performance drop by using the quantization.

#### Constructing few-shot contexts.

For PopQA, we sample few-shot examples stratified by relationship type to
diversify the samples: for each of the 15 relationship types other than the
one in the test question, we sample one random question-answer pair to include
in the context. For EntityQuestions, we take a simple random sample of 15
question-answer pairs because there are more than 16 relationship types.

#### Details of deciding thresholds.

We 75% of PopQAto determine a popularity threshold for each relation type.
Using brute force search, we select the threshold to maximize the adaptive
accuracy, which we define as the accuracy attained by taking the predictions
of the retrieval-augmented system for questions below the popularity threshold
and the predictions based on parametric knowledge for the rest.

We then evaluate adaptive accuracy using the learned thresholds on the
remaining 25% of PopQA, and repeat with 100 different random splits and take
the mean to obtain the reported adaptive accuracy measurement.

##  Appendix C Detailed Results

###  C.1 LM results

#### Full results of per-relationship type accuracy and correlation.

Figure 16 shows the full result of per-relationship type accuracy for all
relationship types in PopQA. Figure 17 shows the correlations for all relation
types. Figures 19 and 18 show the same results for the EntityQuestions
dataset.

Figure 13: Accuracy by LMs and retrieval-augmented LMs on PopQA. This is an
extension of Figure 7 Figure 14: Accuracy by LMs and retrieval-augmented LMs
on EntityQuestions. Figure 15: The proportion of questions for which Adaptive
Retrieval uses retrieval versus model size for EntityQuestions. Figure 16:
Accuracy on PopQA for all relationship types and models. This is an extension
of Figure 4. Figure 17: Correlations on PopQA for all relationship types and
models. This is an extension of Figure 4. Figure 18: Accuracy on
EntityQuestions for all relationship types and models. Figure 19: Correlations
on EntityQuestions for all relationship types and models.

#### Negative correlations of capital on EntityQuestions.

As shown in Figure 19, the capital relationship types on in EntityQuestions,
while on PopQA, this relationship shows relatively high correlations. We found
that in EntityQuestions, this capital relationship type has many low-
popularity questions whose answers are included in subject entity names (e.g.,
subject="canton of Marseille-Belsunce", object="Marseille"). This causes
performance to have a U-shaped relationship with popularity for the capital
relationship type, so if most of the questions sampled come from the top half
of popularity, the linear correlation will be positive, and vice versa.

###  C.2 Retrieval-augmented LM results

#### Overall performance of retrieval-augmented LMs.

Figure 13 shows the overall performance of 40 LMs and retrieval-augmented LMs
on PopQA. Retrieval-augmentation largely improves performance across different
LMs, and much smaller models (GPT-Neo 1.3B) can perform on per with GPT-3.
Figure 14 shows the results on EntityQuestions. Due to computational and time
constraints, we were only able to run vanilla and Contriever results for most
models.

#### Adaptive Retrieval for EntityQuestions.

Figure 15 shows the proportion of questions above the retrieval threshold for
various models using Adaptive Retrieval on EntityQuestions. Because
EntityQuestions has a large quantity of low-popularity questions, models
(especially smaller ones) must rely heavily on retrieval.

#### Full results on all relationship types.

Figure 20 shows the full results on PopQA of the retrieval-augmented LMs and
unassisted LMs on 16 relationship types using three different LMs as
backbones. Figure 21 shows these results for GPT-3 davinci-003 on
EntityQuestions.

\begin{overpic}[width=411.93767pt]{figures/appendix_PopQA_popularity_linecharts.pdf}
\put(100.5,70.0){\rotatebox{-90.0}{{GPT-neo 1.3B}}}
\put(100.5,44.0){\rotatebox{-90.0}{{OPT 13B}}}
\put(100.5,22.0){\rotatebox{-90.0}{{GPT-3 DaVinci 003}}} \end{overpic} Figure
20: Accuracy for 3 models on PopQA versus popularity as shown in Figure 9
broken down by relationship type. Popularity bins with less than 5 samples are
excluded to avoid cluttering the figures with noisy results that have wide
error bars. Figure 21: Accuracy versus popularity for GPT-3 davinci-003 on
EntityQuestions broken down by relationship type. Popularity bins with less
than 5 samples are excluded.

###  C.3 Qualitative Results

Table 3 shows several examples on PopQA, where GPT-3 davinci-003 answers
correctly while the Contriever-augmented version fails to answer. Along with
the low recall@1 of 0.14 for this group, Table 3 suggests that the most common
reason retrieval can be harmful is that it retrieves a document about a
mistaken entity, such as a person with the same name as the subject, or an
entity that simply is not relevant to the question (as in the case of “Noel
Black”).

Table 4 shows several examples on PopQA, where GPT-3 davinci-003 answers
correctly only when augmented with Contriever. The recall@1 for this case is
0.88, which is significantly higher than the overall recall. Note that in the
second example, the retrieval caused the LM to answer correctly, but only by
coincidence: the subject entity “Pierre” actually refers to the city in South
Dakota, not the Basketball player. Otherwise, retrieval appears to be helpful
because it provides the relevant information directly.

Question (answer) |  LM |  Retrieval-LM |  Retrieval
---|---|---|---
Who was the director of Black? (Sanjay Leela Bhansali) |  The director of Black is Sanjay Leela Bhansali. ✓ |  Noel Black ✗ |  Noel Black (June 30, 1937 – July 5, 2014) was an American film and television director, screenwriter, and producer. Black was born in Chicago, Illinois. He won awards at the 1966 … and subsequently concentrated on directing for television, occasionally directing films such as "Private School".
Who was the producer of The Faculty? (Robert Rodriguez) |  The Faculty was produced by Elizabeth Avellan and Robert Rodriguez. ✓ |  The producer of The Faculty was Elizabeth Avellan. ✗ |  The Faculty is a 1998 American science fiction teen horror film written by Kevin Williamson, directed by Robert Rodriguez, and starring Elijah Wood,… The film was released on Christmas Day, 1998, by Miramax Films through Dimension Films. … The film has developed a cult following in the years since its release.
What is Michael Shelley’s occupation? (singer-songwriter) |  Michael Shelley is a singer-songwriter and musician. ✓ |  Michael Shelley is an American applied mathematician. ✗ |  Michael Shelley (mathematician) Michael J. Shelley (born August 17, 1959) is an American applied mathematician who works on the modeling and simulation of complex systems arising in physics and biology. This has included free-boundary problems in fluids and materials science… He is also the co-founder and co-director of the Courant Institute’s Applied Mathematics Lab.
In what city was Zijah Sokolović born? (Sarajevo) |  Zijah Sokolović was born in Sarajevo ✓ |  Zijah Sokolović was born in Orahovac, Kingdom ✗ |  Ali Sokol … (born 8 May 1921 in Orahovac, Kingdom of Serbs, Croats and Slovenes, died 23 September 1974) was a Yugoslav pulmonologist . Ali Sokol was born into an agricultural family. He was the third of four children of father and mother Hatixhes Solomon. It is unknown the exact date of birth but the most reliable date is May 8 year in 1921.
Table 3: Qualitative examples of the questions where only retrieval-augmented LMs fail to answer correctly. The blue underlined text indicates the sub-strings matching the gold answers in the retrieved context.  Question (answer) |  LM |  Retrieval-LM |  Retrieval
---|---|---|---
What genre is Unknown? (fantasy) |  Unknown is not a specific genre of music. It could refer to ✗ |  Unknown is a pulp fantasy fiction magazine. ✓ |  Unknown (magazine) Unknown (also known as Unknown Worlds) was an American pulp fantasy fiction magazine, published from 1939 to 1943 by Street & Smith, and edited by John W. Campbell. … The leading fantasy magazine in the 1930s was "Weird Tales", which focused on shock and horror.
In what country is Pierre? (United States) |  Pierre is in France. ✗ |  Pierre is from the United States. ✓ |  Jonas Farley Pierre (born May 30, 1980) is an American professional basketball player. A center, he has played with three teams in the National Basketball League of Canada (NBL). Pierre last played with the Saint John Mill Rats in Canada. … He then joined Lincoln University in Missouri for his third year. In the summer of 2003, he signed a letter of intent to play with Nicholls State in the NCAA Division I.
Who was the producer of The Cocoanuts? (Walter Wanger) |  The Cocoanuts was produced by Florenz Ziegfeld. ✗ |  The Cocoanuts was produced for Paramount Pictures by Walter Wanger, who ✓ |  The Cocoanuts is a 1929 musical comedy film starring the Marx Brothers. Produced for Paramount Pictures by Walter Wanger, who is not credited, the film stars the four Marx Brothers, Oscar Shaw, Mary Eaton, and Margaret Dumont. It was the first sound film to credit more than one director (Robert Florey and Joseph Santley), and was adapted to the screen by Morrie Ryskind from the George S. Kaufman Broadway musical play. …
Who was the director of The White Suit? (Lazar Ristovski) |  The White Suit was directed by Sachin Kundalkar. ✗ |  Lazar Ristovski ✓ |  In 1999 "The White Suit" an auteur film by Ristovski (director, writer, lead actor, and producer) was at the Cannes Film Festival in the Critics Week program. "The White Suit" was the Serbian entry for the 1999 Academy Awards. Lazar Ristovski is the sole owner of Zillion Film Company In 2006, he made a small appearance in the James Bond film "Casino Royale". He played Caruso in the 2004 movie "King of Thieves". He starred as Đorđe in the award-winning 2009 film "St. George Shoots the Dragon".
Table 4: Qualitative examples of the questions where only retrieval-augmented
LMs successfully answer correctly. The blue underlined text indicates the sub-
strings matching the gold answers in the retrieved context.