††thanks<EMAIL_ADDRESS>corresponding author

#  Mixed-state Entanglement for AdS Born-Infeld Theory

Peng Liu 1<EMAIL_ADDRESS>Zhe Yang 1<EMAIL_ADDRESS>Chao
Niu 1<EMAIL_ADDRESS>Cheng-Yong Zhang 1<EMAIL_ADDRESS>Jian-
Pin Wu 2,3 1 Department of Physics and Siyuan Laboratory, Jinan University,
Guangzhou 510632, China
2 Center for Gravitation and Cosmology, College of Physical Science and
Technology, Yangzhou University, Yangzhou 225009, China
3 School of Aeronautics and Astronautics, Shanghai Jiao Tong University,
Shanghai 200240, China

###### Abstract

We study the mixed-state entanglement for AdS Born-Infeld (BI) theory. We
calculate the mixed-state entanglement and investigate the relationship
between it and the system parameters. We find that the holographic
entanglement entropy (HEE) and mutual information (MI) exhibit monotonically
increasing and decreasing behavior with BI factor $b$. However, the
entanglement wedge cross-section (EWCS) exhibits a very rich set of phenomena
about system parameters. EWCS always increases with $b$ when $b$ is small and
then monotonically decreases with $b$. These behaviors suggest that increasing
the BI factor, which is essentially enhancing the coupling between the
background geometry and the transport properties can always enhance the EWCS.
The coupling between the entanglement and the transport behaviors has also
been studied in condensed matter theories and is important to construct a
stable quantum circuit. We also provide analytical understanding of the above
phenomenon.

###### Contents

  1. I Introduction
  2. II Holographic Born-Infeld Theory And Information-Related Quantities
    1. II.1 The AdS Born-Infeld Model
    2. II.2 Holographic information-related quantities
    3. II.3 Computations of holographic geometric quantities
      1. II.3.1 The minimum surface
      2. II.3.2 The EWCS
  3. III The Holographic Entanglement Entropy And The Holographic Mutual Information
  4. IV The Holographic Entanglement Wedge Cross-Section
  5. V Discussion

##  I Introduction

Quantum entanglement is the most distinguishing characteristic between quantum
and classical systems. Holographic gravity, condensed matter theory, quantum
information, and other areas have recently overlapped with each other on
quantum entanglement. Numerous quantum entanglement measurements have been
discovered to be capable of diagnosing the quantum phase transition of strong
correlation systems and the topological quantum phase transitions, as well as
playing a key role in the emergence of spacetime Osterloh:2002na ;
Amico:2007ag ; Wen:2006topo ; Kitaev:2006topo ; Ryu:2006bv ; Hubeny:2007xt ;
Lewkowycz:2013nqa ; Dong:2016hjy .

There are numerous types of quantum entanglement measurements, including
entanglement entropy (EE), mutual information (MI), Rényi entanglement
entropy, and negativity. Among these quantum entanglement measurements, EE is
commonly considered a useful measure of pure state entanglement. However, EE
is not applicable to measure the more common mixed-state entanglement. To
measure mixed-state entanglement, numerous new entanglement measurements, such
as the entanglement of purification, non-negativity, and the entanglement of
formation, have been proposed vidal:2002 ; Horodecki:2009review . On the other
hand, calculating the mixed-state entanglement measures is extremely
difficult.

Gauge/gravity duality is a powerful tool for analyzing strongly correlated
systems because it connects entanglement-related physical quantities to
geometric objects in dual gravity systems. In the dual gravity system, the
holographic entanglement entropy (HEE) connects the EE of a subregion on the
boundary with the area of the minimum surface Ryu:2006bv . HEE has been
demonstrated to be able to detect quantum phase transitions and thermodynamic
phase transitions Nishioka:2006gr ; Klebanov:2007ws ; Pakman:2008ui ;
Zhang:2016rcm ; Zeng:2016fsb . Recently, the Rényi entropy was proposed to be
proportional to the minimal area of cosmic branes Dong:2016fnf . Moreover, the
butterfly effect that reflects the dynamic properties of quantum systems, has
been extensively studied in holographic theories Shenker:2013pqa ;
Sekino:2008he ; Maldacena:2015waa ; Donos:2012js ; Blake:2016wvh ;
Blake:2016sud ; Ling:2016ibq ; Ling:2016wuy ; Wu:2017mdl ; Liu:2019npm . In
addition, holographic duality of quantum complexity, a new information-related
quantity from the EE, was also proposed Brown:2015lvg ; Brown:2015bva ;
Chapman:2016hwi ; Ling:2018xpc ; Chen:2018mcc ; Yang:2019gce ; Ling:2019ien .
More recently, the EWCS was associated with the area of the minimum cross-
section of the entanglement wedge Takayanagi:2017knl ; Nguyen:2017yqw . The
geometric prescription of EWCS provides a novel and powerful tool for studying
the mixed-state entanglement in holographic theories.

Among all the models in holographic theories, the Born-Infeld (BI) model is a
special class of models for nonlinear electromagnetic field theories. It was
first proposed to eliminate the divergent self-energy of Maxwell theory.
Later, it was found that the BI theory can be naturally derived from the
string theory under the low-energy approximation. The BI model under the
holographic theories can be dual to the quantum chromodynamics (QCD) systems
Kundu:2013eba ; Kundu:2019ull , and some condensed matter systems with novel
transport behaviors, such as the quantum liquid Karch:2009zz , the Mott-
insulator Baggioli:2016oju , and the novel magneto-resistance phenomenon
Kiritsis:2016cpm ; Cremonini:2017qwq , which is consistent with the
experimental phenomenon in Hayers:2014zz ; Hayes:2016 . Various properties of
the BI model, such as its thermodynamic properties, transport properties
Wu:2018zdc , the complexity Bakhtiarizadeh:2020kav , have been extensively
investigated. However, the question of how exactly the BI factor $b$, which
embodies the nonlinearity of this nonlinear electromagnetic field theory,
affects the properties of the system, especially the mixed-state entanglement
properties, remains to be answered.

This paper focuses on the effect of the BI factor on two measures of mixed-
state entanglement - MI and EWCS. When $b\to 0$, the background geometry is
AdS-Schwarzschild solution, and the entanglement property of the system is
decoupled from the transport property of the system; while for non-zero $b$,
the transport behaviors can affect the entanglement property. Therefore, we
interpret $b$ as the degree of correlation between the entanglement and
transport properties of the metric when $b$ increases from zero. Remind also
that the coupling between the transport properties and the entanglement is
also an important topic in condensed matter field theory, and is crucial for
the construction of a stable quantum circuit nanowire:2011 ; plasom:2002 ;
plasmonics:2013 . For $b\to\infty$, the system goes to the AdS-RN black brane
system with a linear Maxwell field. Therefore, the range $b\in(0,\infty)$
represents the process that the Maxwell field turns on and converges to a
linear Maxwell field case. Our main goal is to explore how BI factor $b$
affects the MI and EWCS.

We organize this paper as follows: we introduce the holographic BI model in
Sec. II.1, entanglement measures (HEE, MI, EWCS) and their holographic duality
in Sec. II.2. We discuss the properties of HEE, MI (III) and EWCS (IV)
systematically. Finally, we summarize in Sec. V.

##  II Holographic Born-Infeld Theory And Information-Related Quantities

First, we review the holographic BI model. Following that, we review the
concepts of the HEE, MI, and EWCS with their holographic dual. Then, we
elaborate upon our algorithms proposed to calculate minimum surfaces and
minimum cross-sections.

###  II.1 The AdS Born-Infeld Model

The action of the 4-dimensional holographic BI model is,

$S=\int d^{4}x\sqrt{-g}\left[\frac{R-3\Lambda}{16\pi G}+\frac{b^{2}}{4\pi
G}\left(1-\sqrt{1+\frac{2F}{b^{2}}}\right)\right].$ (1)

The parameter $b$ is the BI factor, and $\Lambda=-\frac{3}{l^{2}}$ with $l$
the AdS radius. The solution of the BI theory is,

$ds^{2}=-f(r)dt^{2}+\frac{1}{f(r)}dr^{2}+r^{2}h_{ij}dx^{i}dx^{j},$ (2)

with

$f(r)=\frac{r^{2}}{l^{2}}-\frac{2M}{r}+\frac{4Q^{2}{}_{2}F_{1}\left(\frac{1}{4},\frac{1}{2};\frac{5}{4};-\frac{Q^{2}}{b^{2}r^{4}}\right)}{3r^{2}}+\frac{2b^{2}r^{2}}{3}\left(1-\sqrt{\frac{Q^{2}}{b^{2}r^{4}}+1}\right),$
(3)

$Q$ is the electric charge and $M$ is the mass of the black brane. For
$l^{2}<0$ and $l^{2}>0$ the system is asymptotically dS and AdS, respectively.
Here, we fix $l^{2}=1$ for concreteness. For $k=1,0,-1$ the $h_{ij}$ denotes a
sphere, a Ricci flat surface, and a hyperbolic surface, respectively. Here, we
focus on the planar case, i.e., $k=0$.

At the horizon $r=r_{h}$ we have $f(r_{h})=0$, and hence we arrive at the ADM
mass

$M=\frac{4l^{2}Q^{2}\,_{2}F_{1}\left(\frac{1}{4},\frac{1}{2};\frac{5}{4};-\frac{Q^{2}}{b^{2}r_{h}^{4}}\right)-2b^{2}l^{2}r_{h}^{4}\sqrt{\frac{Q^{2}}{b^{2}r_{h}^{4}}+1}+2b^{2}l^{2}r_{h}^{4}+3r_{h}^{4}}{6l^{2}r_{h}}.$
(4)

The Hawking temperature is,

$T=\frac{r_{h}}{4\pi}\left(3-2b^{2}\left(\sqrt{\frac{Q^{2}}{b^{2}r_{h}^{4}}+1}-1\right)\right).$
(5)

The planar case is always thermodynamically stable Cai:2004eh . Therefore, in
this BI black brane system, there is no thermodynamic phase transition.

The system is invariant under the rescaling,

$(t,1/r,x,y)\to\alpha(t,1/r,x,y),\;Q\to Q/\alpha^{2},\,T\to
T/\alpha,\,r_{h}\to\alpha r_{h}.$

Other parameters such as $b,\,\beta$ are all dimensionless. Therefore, we can
fix $r_{h}=1$. Here, we adopt $\sqrt{Q}$ as the scaling unit, consequently, we
need to divide physical quantity with scaling dimension $[n]$ by $Q^{n/2}$.

For numerical convenience, we transform $r$ into $z\equiv r_{h}/r$ such that
the semi-infinite domain $r\in(r_{h},\infty)$ becomes a finite domain
$z\in[0,1]$. Then the metric becomes,

$ds^{2}=\frac{1}{z^{2}}\left(-hdt^{2}+\frac{r_{h}^{2}dz^{2}}{h}+r_{h}^{2}dx^{2}+r_{h}^{2}dy^{2}\right),$
(6)

with

$\displaystyle h(z)\equiv$
$\displaystyle\frac{4}{3}Q^{2}z^{3}\left(z\,_{2}F_{1}\left(\frac{1}{4},\frac{1}{2};\frac{5}{4};-\frac{Q^{2}z^{4}}{b^{2}}\right)-\,_{2}F_{1}\left(\frac{1}{4},\frac{1}{2};\frac{5}{4};-\frac{Q^{2}}{b^{2}}\right)\right)$
(7)
$\displaystyle-\frac{2}{3}b^{2}\left(z^{3}\left(1-\sqrt{\frac{Q^{2}}{b^{2}}+1}\right)+\sqrt{\frac{Q^{2}z^{4}}{b^{2}}+1}-1\right)-z^{3}+1.$

And the dimensionless Hawking temperature becomes,

$T=\frac{b^{2}\left(2-2\sqrt{\frac{Q^{2}}{b^{2}}+1}\right)+3}{4\pi\sqrt{Q}}.$
(8)

From the dimensionless Hawking temperature (8) we can find that,

$\lim_{Q\to 0}T\to\infty,\quad\lim_{Q\to\frac{\sqrt{12b^{2}+9}}{2b}}T\to 0.$
(9)

Also, we can find that,

$\partial_{Q}T=2b\left(b-\sqrt{b^{2}+Q^{2}}\right)-3\sqrt{\frac{Q^{2}}{b^{2}}+1}-2Q^{2}<0.$
(10)

Therefore, the quantity $Q$ is restricted to the range
$[0,\frac{\sqrt{12b^{2}+9}}{2b}]$ and that the temperature $T$ decreases as
$Q$ increases. This system is described by three variables $(T,\,b\,,r_{h})$,
with only two of them being independent. We have also observed that for any
given value of $b$, the temperature $T$ always increases with increasing
$r_{h}$, thus, the value of $r_{h}$ is uniquely determined by a given
temperature $T$. This can be seen in the Fig. 1. Therefore, we can simplify
the system to a two-parameter system $(b,T)$.

Figure 1: The contour plot of the Hawking temperature in the plane
$(b,r_{h})$, where the temperature is only positive in the shaded region.

When the parameter $b\to\infty$, the background solution of our system
converges to the AdS-RN solution, and when $b\to 0$, it becomes the AdS-
Schwarzschild solution. When $b$ is zero, there is an electromagnetic field
present, but the background solution is still the AdS-Schwarzschild solution.
This means that the entanglement-related physical quantities are not affected
by the conductivity of the system. However, as $b$ increases, the
electromagnetic fields starts to affect the background solution, and thus has
an impact on the entanglement structure of the system. Therefore, we refer to
increasing $b$ from zero to infinity as the process of turning on the coupling
between the background and the conductivity, and finally resulting in an AdS-
RN system.

It is worth noting that the relationship between conductivity and
entanglement-related quantities is of great importance in condensed matter
theories. Recent experiments have shown that entanglement between quantum dots
can persist despite the influence of surface plasmon polariton (SPPs)
transmission plasmonics:2013 ; plasmon:2004 ; plasom:2002 . These findings are
crucial for the development of stable quantum circuits. Additionally, it has
been found that at specific values of the inter-dot distance $d$ or detuning
$\delta$, the two-quantum-dot system can be in a highly entangled state and be
separate from the transmission of SPPs nanowire:2011 . However, when $d$ or
$\delta$ deviate from these values, the entanglement of quantum dots becomes
highly correlated with the transmission of SPPs. This suggests that decoupling
of entanglement and transport can exist in real physical systems and can be
characterized by certain parameters.

Next, we will focus on how the entanglement-related physical quantities change
as we vary the parameter $b$.

###  II.2 Holographic information-related quantities

Entanglement is a fundamental and intriguing aspect of quantum mechanics. One
way to quantify entanglement is through entanglement entropy (EE), which
measures the degree of entanglement between a subset of a system and the rest
of the system. Specifically, the entanglement entropy $S_{A}$ between subsets
$A$ and $B$ of a system $A\cup B$ is defined as the von Neumann entropy in
terms of the reduced density matrix $\rho_{A}$.

$S_{A}(|\psi\rangle)=-\text{Tr}\left[\rho_{A}\log\rho_{A}\right],\quad\rho_{A}=\text{Tr}_{B}\left(|\psi\rangle\langle\psi|\right).$
(11)

It is easy to find that $S_{A}=S_{B}$ for pure states Chuang:2002book .
Holographic duality theory relates the holographic entanglement entropy (HEE)
to the area of the minimum surface in dual gravity systems Ryu:2006bv  (see
the left plot of Fig. 2).

EE is often used to measure the degree of entanglement in pure states, but it
is not as effective in measuring mixed state entanglement. For example, even
when subsystems $A$ and $B$ are not entangled, they can still have non-zero EE
in a system composed of direct product of the density matrices of $\rho_{A}$
and $\rho_{B}$. This is because EE takes into account both quantum
entanglement and classical correlation, so it does not always provide a
accurate measure of the entanglement. As a result, other measures for mixed-
state entanglement have been proposed in the literature vidal:2002 ;
Horodecki:2009review . The most direct mixed-state entanglement measure is MI.

For the subsystem $A\cup C$ separated by $B$, the mutual information (MI) is
defined as:

$I\left(A,B\right):=S\left(A\right)+S\left(B\right)-S\left(A\cup B\right),$
(12)

This measures the mixed-state entanglement between $A$ and $B$. It can be
easily verified that $I\left(A,B\right)=0$ when
$\rho_{AB}=\rho_{A}\otimes\rho_{B}$, therefore MI have the property that
direct product states have zero entanglement. However, MI is not a perfect
measure of mixed-state entanglement, as it is closely related to EE, and it’s
properties are sometimes dominated by EE or thermal entropy in certain
situations. This indicates that other measures of mixed-state entanglement
should be used.

The entanglement wedge cross-section (EWCS) has been associated with the
duality of certain mixed-state entanglement measures, such as entanglement of
purification, logarithmic negativity, and reflect entropy Kudler-Flam:2018qjo
; Kusuki:2019zsp ; Dutta:2019gen . Takayanagi proposed that EWCS
$E_{W}\left(\rho_{AB}\right)$ is the area of the minimum cross-section
$\Sigma_{AB}$ in connected entanglement wedge Takayanagi:2017knl , i.e. (see
the right plot in Fig. 2),

$E_{W}\left(\rho_{AB}\right)=\min_{\Sigma_{AB}}\left(\frac{\text{Area}\left(\Sigma_{AB}\right)}{4G_{N}}\right).$
(13)

It is important to note that if the entanglement wedge is disconnected,
meaning the minimum cross-section does not exist, the EWCS will be zero, it
corresponds to cases with vanishing MI. Additionally, the EWCS also satisfies
some important inequalities as its quantum information counterparts
Takayanagi:2017knl ; Bao:2017nhh

$x$$y$$z$

$x$$y$$z$

Figure 2: The left plot: The minimum surface for a given width $w$. The right
plot: The minimum cross-section (green surface) of the entanglement wedge.

Next, we present the algorithm for obtaining the minimum surfaces and EWCS.

###  II.3 Computations of holographic geometric quantities

We examine the EWCS of an infinite strip with a homogeneous background for
numerical simplicity. For a generic homogeneous background

$ds^{2}={g_{tt}}dt^{2}+g_{zz}dz^{2}+g_{xx}dx^{2}+g_{yy}dy^{2},$ (14)

where $z=0$ represents the boundary of the asymptotic AdS spacetime. The left
plot in Fig. 3 is a visual representation of the minimum surface for an
infinite strip along the $y$-axis. Since the background is homogeneous, all
metric components $g_{\mu\nu}$ only depend on the coordinate $z$.

####  II.3.1 The minimum surface

The minimum surface near the AdS boundary is perpendicular to the boundary,
making the spatial direction $x$ an unsuitable parameter for finding the
minimum surface. Ref. Ling:2019tbi  adopted the angle $\theta$ with
$\tan\theta=z/x$, as the parameter for the minimum surface (see Fig. 3). Using
this method, we can parametrize a surface as $(x(\theta),z(\theta))$ with area
$A$ given by

$A=2\int_{0}^{\pi/2}\sqrt{x^{\prime}(\theta)^{2}g_{xx}g_{yy}+z^{\prime}(\theta)^{2}g_{yy}g_{zz}}d\theta.$
(15)

The resultant equations of motion read,

$\displaystyle
x^{\prime}(\theta)z^{\prime}(\theta)^{2}\left(\frac{g_{xx}^{\prime}}{2g_{xx}}+\frac{g_{yy}^{\prime}}{g_{yy}}-\frac{g_{zz}^{\prime}}{2g_{zz}}\right)+\frac{x^{\prime}(\theta)^{3}\left(g_{yy}g_{xx}^{\prime}+g_{xx}g_{yy}^{\prime}\right)}{2g_{xx}g_{zz}}+x^{\prime\prime}(\theta)z^{\prime}(\theta)-x^{\prime}(\theta)z^{\prime\prime}(\theta)$
$\displaystyle=0,$ (16) $\displaystyle z(\theta)-\tan(\theta)x(\theta)$
$\displaystyle=0.$

where $g_{\\#\\#}^{\prime}\equiv g_{\\#\\#}^{\prime}(z)$. The boundary
conditions are,

$z(0)=0,\quad x(0)=w,\quad z^{\prime}(\pi/2)=0,\quad x(\pi/2)=0,$ (17)

where $w$ is the width of the strip.

####  II.3.2 The EWCS

Given a biparty subsystem with minimum surfaces
$C_{1}(\theta_{1}),\,C_{2}(\theta_{2})$, we solve the minimum surface
$C_{p_{1},p_{2}}$ connecting $p_{1}\in C_{1}$ and $p_{2}\in C_{2}$. We
parametrize $C_{p_{1},p_{2}}$ with $z$, then the area of $C_{p_{1},p_{2}}$
reads,

$A=\int_{C_{p_{1},p_{2}}}\sqrt{g_{xx}g_{yy}x^{\prime}(z)^{2}+g_{xx}g_{zz}}dz.$
(18)

The resultant equation of motion becomes,

$x^{\prime}(z)^{3}\left(\frac{g_{xx}g_{yy}^{\prime}}{2g_{yy}g_{zz}}+\frac{g_{xx}^{\prime}}{2g_{zz}}\right)+x^{\prime}(z)\left(\frac{g_{xx}^{\prime}}{g_{xx}}+\frac{g_{yy}^{\prime}}{2g_{yy}}-\frac{g_{zz}^{\prime}}{2g_{zz}}\right)+x^{\prime\prime}(z)=0,$
(19)

with boundary conditions,

$x(z(\theta_{1}))=x(\theta_{1}),\quad x(z(\theta_{2}))=x(\theta_{2}).$ (20)

To obtain the EWCS, we need to locate the global minimum of the minimum
surfaces connecting $C_{1}(\theta_{1}),\,C_{2}(\theta_{2})$, i.e., the minimum
cross-section.

Finding the minimum cross-section is a challenging task as it involves
searching through a two-dimensional parameter space $(\theta_{1},\theta_{2})$.
However, it can be noted that the globally minimum cross-section must be
perpendicular to the minimum surfaces at the point of intersection. This
observation serves as a local constraint, which can greatly speed up the
search process. We demonstrate the methods of solving the EWCS in Fig. 3.

Figure 3:  The demonstration of the EWCS. The $p_{1}$ and $p_{2}$ are the
intersection points of the minimum surface connecting those two minimum
surfaces. The solid blue curve (parametrized with $\theta_{1}$) and solid
orange curve (parametrized with $\theta_{2}$) are minimum surfaces. The thick
red curve is the minimum surface connecting $p_{1}$ and $p_{2}$. The blue
arrows at the $p_{1}$ and $p_{2}$ are the tangent vector
$\left.\left(\frac{\partial}{\partial z}\right)^{a}\right|_{p_{1}}$ and
$\left.\left(\frac{\partial}{\partial z}\right)^{a}\right|_{p_{2}}$ along the
$C_{p_{1},p_{2}}$, while the purple arrows are the tangent vectors
$\left.\left(\frac{\partial}{\partial\theta_{1}}\right)^{a}\right|_{p_{1}}$
and
$\left.\left(\frac{\partial}{\partial\theta_{2}}\right)^{a}\right|_{p_{2}}$
along $C_{1},\,C_{2}$, respectively. The dark dashed horizontal line is the
horizon.

For numerical stability, it is better to implement the perpendicular
conditions with normalized vectors as,

$\displaystyle Q_{1}(\theta_{1},\theta_{2})$
$\displaystyle\equiv\left.\frac{g_{ab}\left(\frac{\partial}{\partial
z}\right)^{a}\left(\frac{\partial}{\partial\theta_{1}}\right)^{b}}{\sqrt{g_{cd}\left(\frac{\partial}{\partial
z}\right)^{c}\left(\frac{\partial}{\partial
z}\right)^{d}}\sqrt{g_{mn}\left(\frac{\partial}{\partial\theta_{1}}\right)^{m}\left(\frac{\partial}{\partial\theta_{1}}\right)^{n}}}\right|_{p_{1}}=0,$
(21) $\displaystyle Q_{2}(\theta_{1},\theta_{2})$
$\displaystyle\equiv\left.\frac{g_{ab}\left(\frac{\partial}{\partial
z}\right)^{a}\left(\frac{\partial}{\partial\theta_{2}}\right)^{b}}{\sqrt{g_{cd}\left(\frac{\partial}{\partial
z}\right)^{c}\left(\frac{\partial}{\partial
z}\right)^{d}}\sqrt{g_{mn}\left(\frac{\partial}{\partial\theta_{2}}\right)^{m}\left(\frac{\partial}{\partial\theta_{2}}\right)^{n}}}\right|_{p_{2}}=0.$

Note that $Q_{1}$ and $Q_{2}$ are both functions of the $\theta_{1}$ and
$\theta_{2}$. Now, the search of the EWCS is equivalent to finding the minimum
surface ending at $(\theta_{1},\theta_{2})$ where (21) is satisfied.

To determine the correct EWCS, we first select an initial seed
$(\theta_{1},\theta_{2})$ and use the Newton iterative method to obtain
feedback $(\delta\theta_{1},\delta\theta_{2})$. By repeating this process, we
can find the minimum cross-section, which is the EWCS. It is crucial to
carefully choose the initial values of $(\theta_{1},,\theta_{2})$ for the
iterations to converge. The numerical reliability is ensured by the
convergence of results when using different initial values or increasing the
density of discretization. For more technical details, refer to reference
Boyd:2001 .

Using the techniques outlined above, we will now examine mixed-state
entanglement measures for the BI model. Additionally, we will examine the
correlation between the BI factor $b$ and information-related quantities.

##  III The Holographic Entanglement Entropy And The Holographic Mutual
Information

Figure 4: HEE vs $T$ and $b$ at width $w=0.8$, respectively.

We begin by examining the relationship between HEE, system parameters $b$ and
$T$. As shown in Fig. 4, HEE, represented by $S$, increases monotonically with
both $b$ and $T$, but their rate of increase is different. Initially, $S$
increases slowly with $T$ and its growth rate with $T$ becomes more pronounced
as $T$ increases. On the other hand, $S$ increases quickly with $b$ at first
and then slows down as $b$ decreases. Next, we explain the behavior of $S$
with $b$ and $T$, respectively.

When the horizon radius of the black brane increases, the minimum surface
tends to be closer to the horizon of the black brane, which makes the
thermodynamic entropy dominate the behavior of the HEE. Therefore, the growth
of HEE with $T$ as well as $b$, can be understood from the relation between
$r_{h}$ and $T$ or $b$. According to (5) we can deduce that $r_{h}$ increases
with increasing temperature and $b$, this can be seen by taking the derivative
of $r_{h}$ with respect to $T$ and $b$. The results are,

$\displaystyle\partial_{T}r_{h}$
$\displaystyle=\frac{r_{h}^{4}\sqrt{\frac{Q^{2}}{b^{2}r_{h}^{4}}+1}}{r_{h}^{4}\left(2b^{2}\left(\sqrt{\frac{Q^{2}}{b^{2}r_{h}^{4}}+1}-1\right)+3\sqrt{\frac{Q^{2}}{b^{2}r_{h}^{4}}+1}\right)+2Q^{2}},$
(22) $\displaystyle\partial_{b}r_{h}$
$\displaystyle=\frac{2r_{h}\left(Q^{2}-2b^{2}r_{h}^{4}\left(\sqrt{\frac{Q^{2}}{b^{2}r_{h}^{4}}+1}-1\right)\right)}{b\left(r_{h}^{4}\left(2b^{2}\left(\sqrt{\frac{Q^{2}}{b^{2}r_{h}^{4}}+1}-1\right)+3\sqrt{\frac{Q^{2}}{b^{2}r_{h}^{4}}+1}\right)+2Q^{2}\right)}.$

From the above equation, it is clear that $\partial_{T}r_{h}$ is always
positive, indicating that $r_{h}$ increases as $T$ increases. However,
$\partial_{b}r_{h}$ can be positive or negative, depending on the specific
parameter range. Further examination shows that $\partial_{b}r_{h}$ is always
greater than zero when $r_{h}$ is relatively large. This means that $r_{h}$
increases with $b$ when $r_{h}$ is large, or when the minimum surface is
closer to the horizon of the black brane.

When $b$ is relatively large, the system is approximately the AdS-RN system.
The argument presented in Liu:2019qje  can be applied to prove that
$\partial_{T}S>0$. Furthermore, for small subregions, it can be inferred from
the equations in Liu:2019qje  that $\partial_{T}S$ is close to $0$, which
explains the flat behavior of $S$ along $T$ for small temperatures.

Figure 5:  MI as a function of $b$ and $T$ for different configurations.

After studying HEE, we proceed to investigate the behavior of MI with $T$ and
$b$. In the BI model, the configurations for MI and EWCS are subsystems
composed of $a$ and $b$ separated by region $p$. As seen in Fig. 5, MI
decreases with increasing temperature and $b$. This is in contrast to the
behavior of HEE. Moreover, it is worth noting that MI can decrease to zero,
which is an indication of a disentanglement phase transition. We have also
plotted the MI for smaller configurations (see Fig. 6), and the qualitative
phenomena remain the same.

Figure 6: MI as a function of $b$ and $T$ for different configurations.

As the subsystem $c$ and the separation $p$ change, the system undergoes a
disentangling phase transition, at which point the entanglement of two
subsystems $a$ and $c$ vanishes. The critical value of subsystem $c_{c}$ and
separation $p_{c}$ are shown in Fig. 7. The left plot of Fig. 7 shows that the
critical value of subsystem $c_{c}$ increases with $b$ and $T$; however, the
right plot of Fig. 7 shows that the critical value of the separation $p_{c}$
decreases with $b$ and $T$. This is as expected since increasing the
temperature or $b$ will tends to destroy the entanglement, resulting in a
larger subregion $c_{c}$ or a smaller separation $p_{c}$.

Figure 7: Critical configurations of $c_{c}$ and $p_{c}$.

Next, we explore the mixed-state entanglement through the EWCS.

##  IV The Holographic Entanglement Wedge Cross-Section

In Fig. 8, we present the minimum surfaces and the corresponding minimum
cross-sections. It can be observed that the minimum surface is flatter when
the temperature is lower. This is due to the fact that the coordinate $z$ is
related to the horizon radius $r_{h}$, and at lower temperatures, a small
$r_{h}$ will rescale $z$ to $zr_{h}$, resulting in a flatter minimum surface.
This makes it challenging to obtain precise enough solutions for the minimum
surface since the flat case is more singular in the $\theta$ coordinate. To
overcome this issue, we redefine the angle as $z=\eta x\tan(\theta)$, where
$\eta$ is a number related to the temperature. Only with this technique, we
can achieve precise enough solutions.

Figure 8: The illustration of EWCS. At the same configuration
$(a,p,c)=(0.1,0.05,0.06925)$ we see that the minimum surface becomes flatter
when decreasing the temperature. Meanwhile, the minimum cross-section always
ends at the point near the tops of the inner minimum surface, while ends at
the point away from the tops of the outer minimum surface.

Figure 9: EWCS vs $T$. This plot is obtained at $(a,p,c)=(0.1,0.05,0.06925)$.
When $b$ is relatively large, the $E_{W}$ converges to certain fixed values.
For $T=0.2971$ it can first increase, and later decreases, and after that
increases with $b$. Therefore, for very small $b$ the EWCS increases with $b$,
irrespective of the values of the $T$ and the configurations.

We show the EWCS vs $b$ in Fig. 9, from which we can find that the EWCS can
show very delicate behaviors. The EWCS increases with $b$ at first in a very
narrow range of $b$, however, it starts to decrease with $b$ when $b$ is
relatively large and monotonically decreases with $b$. This is in sharp
contrast to the behavior of the HEE and MI, that only shows monotonical
behaviors (see Fig. 4 and Fig. 5). In addition, the $E_{W}$ changes slower
with $b$ than that with $T$. The typical change is of order $10^{-4}$ and
$10^{-3}$, respectively. This delicate behavior can be captured precisely
because the precision of our numerical methods can be up to $10^{-7}$.

Notice that the background is an AdS-Schwarzschild solution when $b$ is 0,
meanwhile, its electromagnetic field is non-zero. At this point, the charge
transport behavior of the system is significantly different from that of the
genuine AdS-Schwarzschild system. Moreover, since its geometry is still AdS-
Schwarzschild, the entanglement-related geometric quantities will be decoupled
from the charge transport. As $b$ gradually increases, the background geometry
will receive back reactions from the Maxwell field. At this time, the
entanglement-related geometry starts to couple with the charge transports.
Therefore, $b$ can play a role in measuring the relationship between
entanglement and transport when $b$ is small. As we have pointed out, the EWCS
increases with $b$ when $b$ is very small, i.e., when the coupling has just
occurred. And when $b$ increases further, the EWCS gradually shows a
decreasing behavior. Notice that simpler geometric quantities such as HEE, and
MI only show a very flat monotonic behavior. This indicates that EWCS, as a
mixed-state entanglement, captures very different properties from HEE and MI.

To understand the above behavior more clearly, we implement the following
analytical treatments. For small values of $b$, we can expand the expression
of the $E_{W}$ (18) integral with respect to $b$ as,

$E_{W}=\int_{\Sigma}\left(\frac{1}{z^{2}}\sqrt{{dx}^{2}+\frac{{dz}^{2}}{\left(1-z^{3}\right)}}+\frac{b{dz}^{2}\left(\Gamma\left(\frac{1}{4}\right)+8\Gamma\left(\frac{5}{4}\right)\right)\left({dz}^{2}+{dx}^{2}\left(1-z^{3}\right)\right)^{-1/2}}{2\sqrt{3\pi
T\left(1-z^{3}\right)}\left(z^{2}+z+1\right)\Gamma\left(\frac{1}{4}\right)}+O(b^{2})\right),$
(23)

where the second term shows us that $\frac{dE_{W}}{db}>0$ for small values of
$b$. This explains the ubiquitous existence of the monotonically increasing
behavior of $E_{W}$ vs $b$ for small values of $b$. From the holographic dual
picture, it means that when the Maxwell field starts to turn on from the BI
case, the $E_{W}$ is increased. However, when further increasing $b$ we find
that $E_{W}$ reaches local maximums and starts to decrease. When $b$ is large,
it can be expected that the background system approaches the AdS-RN, a fixed
background geometry. Therefore, the $E_{W}$ will starts to converge to some
fixed value.

Next, we show the EWCS in larger configurations in Fig. 10.

Figure 10: The EWCS vs $b$ for a larger configuration
$(a,p,c)=(0.5,0.2,0.3875)$.

As seen in Fig. 10, the non-monotonicity of $E_{W}$ with $b$ becomes more
pronounced as the width of the configuration increases. This means that the
non-monotonicity exists over a wider interval. The reason for this is that
when the width is relatively small, the minimum surface and the minimal cross-
section are only slightly different from the properties of AdS. However, as
the width increases, they deviate more significantly from AdS.

Next, we examine the behavior of EWCS with temperature. When the configuration
is relatively small in BI systems, EWCS decreases monotonically with
temperature, as shown in the left plot of Fig. 11. It is worth noting that the
non-monotonic behavior of EWCS at extremely small temperatures has been
studied in Liu:2019qje  for AdS-RN systems. Additionally, we illustrate the
behavior of EWCS with temperature for larger configurations in the right plot
of Fig. 11, which also shows that EWCS decreases monotonically with
temperature. Although the monotonic decreasing behaviors are similar, the EWCS
curves for small configurations differ from those for large configurations. By
comparing the two plots in Fig. 11, crossovers of the EWCS curves with
temperature can be observed in the larger configuration, which reflects the
non-monotonic behavior of EWCS with $b$. These findings suggest that the
behavior of EWCS is generally monotonically decreasing with temperature, and
this behavior is consistent with that of MI.

Figure 11: EWCS vs $T$. The left plot is obtained at
$(a,p,c)=(0.1,0.05,0.06925)$; while the right plot is obtained for a larger
configuration $(a,p,c)=(0.5,0.2,0.3875)$.

In order to more clearly demonstrate the relationship between the EWCS and
variables $b$ and $T$, a contour plot of EWCS as a function of $b$ and $T$ is
presented in Figure 12. This plot illustrates the non-monotonic nature of EWCS
with respect to $b$ and the monotonic decrease of EWCS as $T$ increases.

Figure 12: The EWCS vs $b$ for a larger configuration
$(a,p,c)=(0.5,0.2,0.3875)$.

##  V Discussion

In this paper, we study the behavior of HEE, MI, and the mixed-state
entanglement measure EWCS in the BI model. Our results shows that HEE
increases monotonically with both $b$ and $T$, while MI decreases
monotonically with both $b$ and $T$. Interestingly, the behavior of EWCS with
respect to $b$ shows a non-monotonic trend. Specifically, when $b$ is small,
EWCS increases with $b$, but it begins to decrease as $b$ increases further.
In contrast, EWCS exhibits a consistent monotonically decreasing trend with
$T$.

Moreover, we provide analytical explanations for the non-monotonic behavior of
EWCS with respect to $b$. Note that when $b$ is small, $b$ serves as a measure
of the coupling between the entanglement-related quantities and the charge
transport of the system. Based on this observation, we conjecture that
increasing the coupling between the entanglement-related quantities and the
transport properties can enhance the EWCS of the system. This coupling between
transport behaviors and entanglement is also a topic of significant interest
in condensed matter theory, as seen in previous studies on nanowires
nanowire:2011 , plasmonics plasmonics:2013 ; plasmon:2004 , and plasmons
plasom:2002 .

Next, we point out the issues that deserve further investigation. To begin, we
can examine other BI-like theories, such as the BI theory with massive
gravity, the BI theory with Axions, and so on, to see if the non-monotonic
behavior observed in this paper is general. Furthermore, we can examine the
effect of more general nonlinear EM field theories on the entanglement-related
physical quantities of the system, such as the more general nonlinear EM
fields Guo:2017bru ; Baggioli:2016oju . We are working on these directions.

## Acknowledgments

Peng Liu would like to thank Yun-Ha Zha for her kind encouragement during this
work. Zhe Yang would like to express appreciation to Feng-Ying Deng. This work
is supported by the Natural Science Foundation of China under Grant No.
11805083, 11905083, 12005077, 12147209, the Science and Technology Planning
Project of Guangzhou (202201010655) and Guangdong Basic and Applied Basic
Research Foundation (2021A1515012374). J.-P.W. is also supported by Top Talent
Support Program from Yangzhou University.

## References

  * (1) A. Osterloh, L. Amico, G. Falci, R. Fazio, “Scaling of Entanglement close to a Quantum Phase Transitions” Nature 416, 608 (2002) [arXiv:0202029 [quant-ph]]
  * (2) L. Amico, R. Fazio, A. Osterloh and V. Vedral, “Entanglement in many-body systems” Rev.Mod.Phys. 80, 517 (2008) [arXiv:0703044 [quant-ph]]
  * (3) Levin, Michael, and Xiao-Gang Wen. “Detecting topological order in a ground state wave function”. Physical review letters 96.11 (2006): 110405.
  * (4) Kitaev, Alexei, and John Preskill. “Topological entanglement entropy”. Physical review letters 96.11 (2006): 110404.
  * (5) S. Ryu and T. Takayanagi, “Holographic derivation of entanglement entropy from AdS/CFT,” Phys. Rev. Lett. 96, 181602 (2006) [hep-th/0603001].
  * (6) V. E. Hubeny, M. Rangamani and T. Takayanagi, “A Covariant holographic entanglement entropy proposal,” JHEP 0707, 062 (2007) [arXiv:0705.0016 [hep-th]].
  * (7) A. Lewkowycz and J. Maldacena, “Generalized gravitational entropy,” JHEP 1308, 090 (2013) [arXiv:1304.4926 [hep-th]].
  * (8) X. Dong, A. Lewkowycz and M. Rangamani, “Deriving covariant holographic entanglement,” JHEP 1611, 028 (2016) [arXiv:1607.07506 [hep-th]].
  * (9) Vidal, G. and Werner, R.F., 2002. “A computable measure of entanglement”, Physical Review A, 65(3), p.032314. quant-ph:0102117
  * (10) Horodecki, R., Horodecki, P., Horodecki, M., Horodecki, K. (2009). “Quantum entanglement.” Reviews of modern physics, 81(2), 865.
  * (11) T. Nishioka and T. Takayanagi, “AdS Bubbles, Entropy and Closed String Tachyons,” JHEP 0701, 090 (2007) [hep-th/0611035].
  * (12) I. R. Klebanov, D. Kutasov and A. Murugan, “Entanglement as a probe of confinement,” Nucl. Phys. B 796, 274 (2008) [arXiv:0709.2140 [hep-th]].
  * (13) A. Pakman and A. Parnachev, “Topological Entanglement Entropy and Holography,” JHEP 0807, 097 (2008) [arXiv:0805.1891 [hep-th]].
  * (14) S. J. Zhang, “Holographic entanglement entropy close to crossover/phase transition in strongly coupled systems,” Nucl. Phys. B 916, 304 (2017) [arXiv:1608.03072 [hep-th]].
  * (15) X. X. Zeng and L. F. Li, “Holographic Phase Transition Probed by Nonlocal Observables,” Adv. High Energy Phys. 2016, 6153435 (2016) [arXiv:1609.06535 [hep-th]].
  * (16) X. Dong, “The Gravity Dual of Renyi Entropy,” Nature Commun. 7, 12472 (2016) [arXiv:1601.06788 [hep-th]].
  * (17) S. H. Shenker and D. Stanford, “Black holes and the butterfly effect,” JHEP 1403, 067 (2014) [arXiv:1306.0622 [hep-th]].
  * (18) Y. Sekino and L. Susskind, “Fast Scramblers,” JHEP 0810, 065 (2008) [arXiv:0808.2096 [hep-th]].
  * (19) J. Maldacena, S. H. Shenker and D. Stanford, “A bound on chaos,” JHEP 1608, 106 (2016) [arXiv:1503.01409 [hep-th]].
  * (20) A. Donos and S. A. Hartnoll, “Metal-insulator transition in holography”, Nature Phys. 9, 649 (2013) [arXiv:1212.2998].
  * (21) M. Blake, “Universal Charge Diffusion and the Butterfly Effect in Holographic Theories,” Phys. Rev. Lett. 117, no. 9, 091601 (2016) [arXiv:1603.08510 [hep-th]].
  * (22) M. Blake, “Universal Diffusion in Incoherent Black Holes,” Phys. Rev. D 94, no. 8, 086014 (2016) [arXiv:1604.01754 [hep-th]].
  * (23) Y. Ling, P. Liu and J. P. Wu, “Holographic Butterfly Effect at Quantum Critical Points,” JHEP 1710, 025 (2017) [arXiv:1610.02669 [hep-th]].
  * (24) Y. Ling, P. Liu and J. P. Wu, “Note on the butterfly effect in holographic superconductor models,” Phys. Lett. B 768, 288 (2017) [arXiv:1610.07146 [hep-th]].
  * (25) S. F. Wu, B. Wang, X. H. Ge and Y. Tian, “Collective diffusion and quantum chaos in holography,” Phys. Rev. D 97, no. 10, 106018 (2018) [arXiv:1702.08803 [hep-th]].
  * (26) P. Liu, C. Niu and J. P. Wu, “The Effect of Anisotropy on Holographic Entanglement Entropy and Mutual Information,” Phys. Lett. B 796, 155 (2019) [arXiv:1905.06808 [hep-th]].
  * (27) A. R. Brown, D. A. Roberts, L. Susskind, B. Swingle and Y. Zhao, “Complexity, action, and black holes”, Phys. Rev. D 93, no. 8, 086006 (2016) [arXiv:1512.04993 [hep-th]].
  * (28) A. R. Brown, D. A. Roberts, L. Susskind, B. Swingle and Y. Zhao, “Holographic Complexity Equals Bulk Action?” Phys. Rev. Lett. 116, no. 19, 191301 (2016) [arXiv:1509.07876 [hep-th]].
  * (29) S. Chapman, H. Marrochio and R. C. Myers, “Complexity of Formation in Holography”, JHEP 1701, 062 (2017) [arXiv:1610.08063 [hep-th]].
  * (30) Y. Ling, Y. Liu and C. Y. Zhang, “Holographic Subregion Complexity in Einstein-Born-Infeld theory,” Eur. Phys. J. C 79, no. 3, 194 (2019) [arXiv:1808.10169 [hep-th]].
  * (31) B. Chen, W. M. Li, R. Q. Yang, C. Y. Zhang and S. J. Zhang, “Holographic subregion complexity under a thermal quench,” JHEP 1807, 034 (2018) [arXiv:1803.06680 [hep-th]].
  * (32) R. Q. Yang, H. S. Jeong, C. Niu and K. Y. Kim, “Complexity of Holographic Superconductors,” JHEP 1904, 146 (2019) [arXiv:1902.07586 [hep-th]].
  * (33) Y. Ling, Y. Liu, C. Niu, Y. Xiao and C. Y. Zhang, “Holographic Subregion Complexity in General Vaidya Geometry,” JHEP 1911, 039 (2019) [arXiv:1908.06432 [hep-th]].
  * (34) T. Takayanagi and K. Umemoto, “Holographic entanglement wedge cross-section,” arXiv:1708.09393 [hep-th].
  * (35) P. Nguyen, T. Devakul, M. G. Halbasch, M. P. Zaletel and B. Swingle, “entanglement wedge cross-section: from spin chains to holography,” JHEP 1801, 098 (2018) [arXiv:1709.07424 [hep-th]].
  * (36) A. Kundu and S. Kundu, “Steady-state Physics, Effective Temperature Dynamics in Holography,” Phys. Rev. D 91 (2015) no.4, 046004 [arXiv:1307.6607 [hep-th]].
  * (37) A. Kundu, “Steady States, Thermal Physics, and Holography,” Adv. High Energy Phys. 2019 (2019), 2635917 doi:10.1155/2019/2635917
  * (38) A. Karch, D. T. Son and A. O. Starinets, “Holographic Quantum Liquid,” Phys. Rev. Lett. 102 (2009), 051602
  * (39) M. Baggioli and O. Pujolas, “On Effective Holographic Mott Insulators,” JHEP 12 (2016), 107 doi:10.1007/JHEP12(2016)107 [arXiv:1604.08915 [hep-th]].
  * (40) E. Kiritsis and L. Li, “Quantum Criticality and DBI Magneto-resistance,” J. Phys. A 50 (2017) no.11, 115402 [arXiv:1608.02598 [cond-mat.str-el]].
  * (41) S. Cremonini, A. Hoover and L. Li, JHEP 10, 133 (2017) doi:10.1007/JHEP10(2017)133 [arXiv:1707.01505 [hep-th]].
  * (42) I. M. Hayers, N. P. Breznay, T. Helm, P. Moll, M. Wartenbe, R. D. McDonald, A. Shekhter, J. G. Analytis, Magnetoresistance near a quantum critical point, [ArXiv:1412.6484][cond-mat.str-el];
  * (43) I. M. Hayes, R. D. McDonald, N. P. Breznay, T. Helm, P. J. W. Moll, M. Wartenbe, A. Shekhter, J. G. Analytis, Scaling between magnetic field and temperature in the high-temperature superconductor BaFe2(As1-xPx)2, Nature Physics (2016).
  * (44) J. P. Wu, X. M. Kuang and Z. Zhou, “Holographic transports from Born–Infeld electrodynamics with momentum dissipation,” Eur. Phys. J. C 78, no.11, 900 (2018) [arXiv:1805.07904 [hep-th]].
  * (45) H. R. Bakhtiarizadeh and G. Jafari, “Holographic complexity of Born–Infeld gravity,” Eur. Phys. J. C 80, no.3, 208 (2020) [arXiv:2002.09974 [hep-th]].
  * (46) Chen, Guang-Yin, et al. “Surface Plasmons in a Metal Nanowire Coupled to Colloidal Quantum Dots: Scattering Properties and Quantum Entanglement.“ Physical Review B, vol. 84, no. 4, July 2011.
  * (47) Tame, M., McEnery, K., Özdemir, . et al. “Quantum plasmonics.” Nature Phys 9, 329-340 (2013). https://doi.org/10.1038/nphys2615
  * (48) Altewischer, E., van Exter, M. P. & Woerdman, J. P. Plasmon-assisted transmission of entangled photons. Nature 418, 304-306 (2002).
  * (49) R. G. Cai, D. W. Pang and A. Wang, “Born-Infeld black holes in (A)dS spaces,” Phys. Rev. D 70 (2004), 124034 [arXiv:hep-th/0410158 [hep-th]].
  * (50) Moreno, E., García, F. J., Erni, D., Ignacio Cirac, J. & Martín-Moreno, L. “Theory of plasmon-assisted transmission of entangled photons.” Phys. Rev. Lett. 92, 236801 (2004).
  * (51) Nielsen, Michael A., and Isaac Chuang. “Quantum computation and quantum information.” (2002): 558-559.
  * (52) Y. f. Huang, Z. j. Shi, C. Niu, C. y. Zhang and P. Liu, “mixed-state Entanglement for Holographic Axion Model,” Eur. Phys. J. C 80 (2020) no.5, 426 [arXiv:1911.10977 [hep-th]].
  * (53) J. Kudler-Flam and S. Ryu, “Entanglement negativity and minimal entanglement wedge cross sections in holographic theories,” Phys. Rev. D 99 (2019) no.10, 106014 doi:10.1103/PhysRevD.99.106014 [arXiv:1808.00446 [hep-th]].
  * (54) Y. Kusuki, J. Kudler-Flam and S. Ryu, “Derivation of holographic negativity in AdS3/CFT2,” Phys. Rev. Lett. 123 (2019) no.13, 131603 doi:10.1103/PhysRevLett.123.131603 [arXiv:1907.07824 [hep-th]].
  * (55) S. Dutta and T. Faulkner, “A canonical purification for the entanglement wedge cross-section,” JHEP 03 (2021), 178 doi:10.1007/JHEP03(2021)178 [arXiv:1905.00577 [hep-th]].
  * (56) N. Bao and I. F. Halpern, “Holographic Inequalities and entanglement wedge cross-section,” JHEP 1803, 006 (2018) [arXiv:1710.07643 [hep-th]].
  * (57) John P Boyd. Chebyshev and Fourier spectral methods. Courier Corporation, 2001.
  * (58) Y. Ling, Y. Liu and Z. Y. Xian, “Entanglement Entropy of Annulus in Holographic Thermalization,” arXiv:1911.03716 [hep-th].
  * (59) P. Liu, Y. Ling, C. Niu and J. P. Wu, “entanglement wedge cross-section in Holographic Systems,” JHEP 09 (2019), 071 [arXiv:1902.02243 [hep-th]].
  * (60) X. Guo, P. Wang and H. Yang, “Membrane Paradigm and Holographic DC Conductivity for Nonlinear Electrodynamics,” Phys. Rev. D 98 (2018) no.2, 026021 [arXiv:1711.03298 [hep-th]].


# Few-Shot Question Answering by Pretraining Span Selection

Ori Ram$\ \hskip 1.42271pt{}^{,1}$ Yuval Kirstain∗,1 Jonathan Berant1,2 Amir
Globerson1 Omer Levy1

Blavatnik School of Computer Science, Tel Aviv University1
Allen Institute for AI2
<EMAIL_ADDRESS>Equal
contribution.

###### Abstract

In several question answering benchmarks, pretrained models have reached human
parity through fine-tuning on an order of 100,000 annotated questions and
answers. We explore the more realistic few-shot setting, where only a few
hundred training examples are available, and observe that standard models
perform poorly, highlighting the discrepancy between current pretraining
objectives and question answering. We propose a new pretraining scheme
tailored for question answering: recurring span selection. Given a passage
with multiple sets of recurring spans, we mask in each set all recurring spans
but one, and ask the model to select the correct span in the passage for each
masked span. Masked spans are replaced with a special token, viewed as a
question representation, that is later used during fine-tuning to select the
answer span. The resulting model obtains surprisingly good results on multiple
benchmarks (e.g., 72.7 F1 on SQuAD with only 128 training examples), while
maintaining competitive performance in the high-resource setting.111Our code,
models, and datasets are publicly available:
https://github.com/oriram/splinter.

##  1 Introduction

Figure 1: Performance of SpanBERT (red) and RoBERTa (yellow) base-size models
on SQuAD, given different amounts of training examples. Our model (Splinter,
green) dramatically improves performance. SpanBERT-base trained on the _full_
training data of SQuAD (blue, dashed) is shown for reference.  Figure 2: An
example paragraph before (a) and after (b) masking recurring spans. Each color
represents a different cluster of spans. After masking recurring spans
(replacing each with a single [QUESTION] token), only one span from each
cluster remains unmasked, and is considered the correct answer to the masked
spans in the cluster. The pretraining task is to predict the correct answer
for each [QUESTION].

The standard approach to question answering is to pretrain a masked language
model on raw text, and then fine-tune it with a span selection layer on top
Devlin et al. (2019); Joshi et al. (2020); Liu et al. (2019). While this
approach is effective, and sometimes exceeds human performance, its success is
based on the assumption that large quantities of annotated question answering
examples are available. For instance, both SQuAD Rajpurkar et al. (2016, 2018)
and Natural Questions Kwiatkowski et al. (2019) contain an order of 100,000
question and answer pairs in their training data. This assumption quickly
becomes unrealistic as we venture outside the lab conditions of English
Wikipedia, and attempt to crowdsource question-answer pairs in other languages
or domains of expertise Tsatsaronis et al. (2015); Kembhavi et al. (2017). How
do question answering models fare in the more practical case, where an in-
house annotation effort can only produce a couple hundred training examples?

We investigate the task of few-shot question answering by sampling small
training sets from existing question answering benchmarks. Despite the use of
pretrained models, the standard approach yields poor results when fine-tuning
on few examples (Figure 1). For example, RoBERTa-base fine-tuned on 128
question-answer pairs from SQuAD obtains around 40 F1. This is somewhat
expected, since the pretraining objective is quite different from the fine-
tuning task. While masked language modeling requires mainly local context
around the masked token, question answering needs to align the question with
the global context of the passage. To bridge this gap, we propose (1) a novel
self-supervised method for pretraining span selection models, and (2) a
question answering layer that aligns a representation of the question with the
text.

We introduce Splinter (span-level pointer), a pretrained model for few-shot
question answering. The challenge in defining such a self-supervised task is
how to create question-answer pairs from unlabeled data. Our key observation
is that one can leverage recurring spans: n-grams, such as named entities,
which tend to occur multiple times in a given passage (e.g., “Roosevelt” in
Figure 2). We emulate question answering by masking all but one instance of
each recurring span with a special [QUESTION] token, and asking the model to
select the correct span for each such token.

To select an answer span for each [QUESTION] token in parallel, we introduce a
question-aware span selection (QASS) layer, which uses the [QUESTION] token’s
representation to select the answer span. The QASS layer seamlessly integrates
with fine-tuning on real question-answer pairs. We simply append the
[QUESTION] token to the input question, and use the QASS layer to select the
answer span (Figure 3). This is unlike existing models for span selection,
which do not include an explicit question representation. The compatibility
between pretraining and fine-tuning makes Splinter an effective few-shot
learner.

Splinter exhibits surprisingly high performance given only a few training
examples throughout a variety of benchmarks from the MRQA 2019 shared task
Fisch et al. (2019). For example, Splinter-base achieves 72.7 F1 on SQuAD with
only 128 examples, outperforming all baselines by a very wide margin. An
ablation study shows that the pretraining method and the QASS layer itself
(even without pretraining) both contribute to improved performance. Analysis
indicates that Splinter’s representations change significantly less during
fine-tuning compared to the baselines, suggesting that our pretraining is more
adequate for question answering. Overall, our results highlight the importance
of designing objectives and architectures in the few-shot setting, where an
appropriate inductive bias can lead to dramatic performance improvements.

Figure 3: An example of our fine-tuning setup, taken from the development set
of SQuAD. The question, followed by the [QUESTION] token, is concatenated to
the context. The [QUESTION] token’s representation is then used to select the
answer span.

##  2 Background

Extractive question answering is a common task in NLP, where the goal is to
select a contiguous span $a$ from a given text $T$ that answers a question
$Q$. This format was popularized by SQuAD Rajpurkar et al. (2016), and has
since been adopted by several datasets in various domains Trischler et al.
(2017); Kembhavi et al. (2017) and languages Lewis et al. (2020); Clark et al.
(2020), with some extensions allowing for unanswerable questions Levy et al.
(2017); Rajpurkar et al. (2018) or multiple answer spans Dua et al. (2019);
Dasigi et al. (2019). In this work, we follow the assumptions in the recent
MRQA 2019 shared task Fisch et al. (2019) and focus on questions whose answer
is a single span.

The standard approach uses a pretrained encoder, such as BERT Devlin et al.
(2019), and adds two parameter vectors $\mathbf{s},\mathbf{e}$ to the
pretrained model in order to detect the start position $s$ and end position
$e$ of the answer span $a$, respectively. The input text $T$ and question $Q$
are concatenated and fed into the encoder, producing a contextualized token
representation $\mathbf{x}_{i}$ for each token in the sequence. To predict the
start position of the answer span, a probability distribution is induced over
the entire sequence by computing the inner product of a learned vector
$\mathbf{s}$ with every token representation (the end position is computed
similarly using a vector $\mathbf{e}$):

$\displaystyle P(s=i\mid T,Q)$
$\displaystyle=\frac{\exp(\mathbf{x}_{i}^{\top}\mathbf{s})}{\sum_{j}\exp(\mathbf{x}_{j}^{\top}\mathbf{s})},$
$\displaystyle P(e=i\mid T,Q)$
$\displaystyle=\frac{\exp(\mathbf{x}_{i}^{\top}\mathbf{e})}{\sum_{j}\exp(\mathbf{x}_{j}^{\top}\mathbf{e})}.$

The parameters $\mathbf{s},\mathbf{e}$ are trained during fine-tuning, using
the cross-entropy loss with the start and end positions of the gold answer
span.

This approach assumes that each token representation $\mathbf{x}_{i}$ is
contextualized with respect to the question. However, the masked language
modeling objective does not necessarily encourage this form of long-range
contextualization in the pretrained model, since many of the masked tokens can
be resolved from local cues. Fine-tuning the attention patterns of pretrained
masked language models may thus entail an extensive learning effort, difficult
to achieve with only a handful of training examples. We overcome this issue by
(1) pretraining directly for span selection, and (2) explicitly representing
the question with a single vector, used to detect the answer in the input
text.

##  3 Splinter

We formulate a new task for pretraining question answering from unlabeled
text: recurring span selection. We replace spans that appear multiple times in
the given text with a special [QUESTION] token, except for one occurrence,
which acts as the “answer” span for each (masked) cloze-style “question”. The
prediction layer is a modification of the standard span selection layer, which
replaces the static start and end parameter vectors, $\mathbf{s}$ and
$\mathbf{e}$, with dynamically-computed boundary detectors based on the
contextualized representation of each [QUESTION] token. We reuse this
architecture when fine-tuning on question-answer pairs by adding a [QUESTION]
token at the end of the actual question, thus aligning the pretraining
objective with the fine-tuning task. We refer to our pretrained model as
Splinter.

###  3.1 Pretraining: Recurring Span Selection

Given an input text $T$, we find all recurring spans: arbitrary n-grams that
appear more than once in the same text. For each set of identical recurring
spans $R$, we select a single occurrence as the answer $a$ and replace all
other occurrences with a single [QUESTION] token.222In practice, only some
sets of recurring spans are processed; see Cluster Selection below. The goal
of recurring span selection is to predict the correct answer $a$ for a given
[QUESTION] token $q\in R\setminus\\{a\\}$, each $q$ thus acting as an
independent cloze-style question.

Figure 2 illustrates this process. In the given passage, the span “Roosevelt”
appears three times. Two of its instances (the second and third) are replaced
with [QUESTION], while one instance (the first) becomes the answer, and
remains intact. After masking, the sequence is passed through a transformer
encoder, producing contextualized token representations. The model is then
tasked with predicting the start and end positions of the answer given each
[QUESTION] token representation. In Figure 2b, we observe four instances of
this prediction task: two for the “Roosevelt” cluster, one for the “Allied
countries” cluster, and one for “Declaration by United Nations”.

Taking advantage of recurring _words_ in a passage (restricted to nouns or
named entities) was proposed in past work as a signal for coreference Kocijan
et al. (2019); Ye et al. (2020). We further discuss this connection in Section
7.

#### Span Filtering

To focus pretraining on semantically meaningful spans, we use the following
definition for “spans”, which filters out recurring spans that are likely to
be uninformative: (1) spans must begin and end at word boundaries, (2) we
consider only maximal recurring spans, (3) spans containing only stop words
are ignored, (4) spans are limited to a maximum of 10 tokens. These simple
heuristic filters do not require a model, as opposed to masking schemes in
related work Glass et al. (2020); Ye et al. (2020); Guu et al. (2020), which
require part-of-speech taggers, constituency parsers, or named entity
recognizers.

#### Cluster Selection

We mask a random subset of recurring span clusters in each text, leaving some
recurring spans untouched. Specifically, we replace up to $30$ spans with
[QUESTION] from each input passage.333In some cases, the last cluster may have
more than one unmasked span. This number was chosen to resemble the 15% token-
masking ratio of Joshi et al. (2020). Note that in our case, the number of
masked tokens is greater than the number of questions.

###  3.2 Model: Question-Aware Span Selection

Our approach converts texts into a set of questions that need to be answered
simultaneously. The standard approach for extractive question answering Devlin
et al. (2019) is inapplicable, because it uses fixed start and end vectors.
Since we have multiple questions, we replace the standard parameter vectors
$\mathbf{s},\mathbf{e}$ with dynamic start and end vectors
$\mathbf{s}_{q},\mathbf{e}_{q}$, computed from each [QUESTION] token $q$:

$\displaystyle\mathbf{s}_{q}=\mathbf{S}\mathbf{x}_{q}\qquad\mathbf{e}_{q}=\mathbf{E}\mathbf{x}_{q}$

Here, $\mathbf{S},\mathbf{E}$ are parameter matrices, which extract ad hoc
start and end position detectors $\mathbf{s}_{q},\mathbf{e}_{q}$ from the
given [QUESTION] token’s representation $\mathbf{x}_{q}$. The rest of our
model follows the standard span selection model by computing the start and end
position probability distributions. The model can also be viewed as two
bilinear functions of the question representation $\mathbf{x}_{q}$ with each
token in the sequence $\mathbf{x}_{i}$, similar to Dozat and Manning (2017):

$\displaystyle P(s=i\mid T,q)$
$\displaystyle=\frac{\exp(\mathbf{x}_{i}^{\top}\mathbf{S}\mathbf{x}_{q})}{\sum_{j}\exp(\mathbf{x}_{j}^{\top}\mathbf{S}\mathbf{x}_{q})}$
$\displaystyle P(e=i\mid T,q)$
$\displaystyle=\frac{\exp(\mathbf{x}_{i}^{\top}\mathbf{E}\mathbf{x}_{q})}{\sum_{j}\exp(\mathbf{x}_{j}^{\top}\mathbf{E}\mathbf{x}_{q})}$

Finally, we use the answer’s gold start and end points $(s_{a},e_{a})$ to
compute the cross-entropy loss:

$\displaystyle-\log P(s=s_{a}\mid T,q)-\log P(e=e_{a}\mid T,q)$

We refer to this architecture as the question-aware span selection (QASS)
layer.

###  3.3 Fine-Tuning

After pretraining, we assume access to labeled examples, where each training
instance is a text $T$, a question $Q$, and an answer $a$ that is a span in
$T$. To make this setting similar to pretraining, we simply append a
[QUESTION] token to the input sequence, immediately after the question $Q$
(see Figure 3). Selecting the answer span then proceeds exactly as during
pretraining. Indeed, the advantage of our approach is that in both pretraining
and fine-tuning, the [QUESTION] token representation captures information
about the question that is then used to select the span from context.

##  4 A Few-Shot QA Benchmark

To evaluate how pretrained models work when only a small amount of labeled
data is available for fine-tuning, we simulate various low-data scenarios by
sampling subsets of training examples from larger datasets. We use a subset of
the MRQA 2019 shared task Fisch et al. (2019), which contains extractive
question answering datasets in a unified format, where the answer is a single
span in the given text passage.

Split I of the MRQA shared task contains 6 large question answering datasets:
SQuAD Rajpurkar et al. (2016), NewsQA Trischler et al. (2017), TriviaQA Joshi
et al. (2017), SearchQA Dunn et al. (2017), HotpotQA Yang et al. (2018), and
Natural Questions Kwiatkowski et al. (2019). For each dataset, we sample
smaller training datasets from the original training set with sizes changing
on a logarithmic scale, from 16 to 1,024 examples. To reduce variance, for
each training set size, we sample 5 training sets using different random seeds
and report average performance across training sets. We also experiment with
fine-tuning the models on the full training sets. Since Split I of the MRQA
shared task does not contain test sets, we evaluate using the official
development sets as our test sets.

We also select two datasets from Split II of the MRQA shared task that were
annotated by domain experts: BioASQ Tsatsaronis et al. (2015) and TextbookQA
Kembhavi et al. (2017). Each of these datasets only has a development set that
is publicly available in MRQA, containing about 1,500 examples. For each
dataset, we sample 400 examples for evaluation (test set), and follow the same
protocol we used for large datasets to sample training sets of 16 to 1,024
examples from the remaining data.

To maintain the few-shot setting, every dataset in our benchmark has well-
defined training and test sets. To tune hyperparameters, one needs to extract
validation data from each training set. For simplicity, we do not perform
hyperparameter tuning or model selection (see Section 5), and thus use all of
the available few-shot data for training.

Figure 4: Performance (F1) of Splinter-base (green), compared to all baselines
as a function of the number of training examples on two datasets. Each point
reflects the average performance across 5 randomly-sampled training sets of
the same size.

##  5 Experimental Setup

We describe our experimental setup in detail, including all models and
baselines.

###  5.1 Baselines

Splinter-base shares the same architecture (transformer encoder Vaswani et al.
(2017)), vocabulary (cased wordpieces), and number of parameters (110M) with
SpanBERT-base Joshi et al. (2020). In all experiments, we compare Splinter-
base to three baselines of the same capacity:

#### RoBERTa

Liu et al. (2019) A highly-tuned and optimized version of BERT, which is known
to perform well on a wide range of natural language understanding tasks.

#### SpanBERT

Joshi et al. (2020) A BERT-style model that focuses on span representations.
SpanBERT is trained by masking contiguous spans of tokens and optimizing two
objectives: (a) masked language modeling, which predicts each masked token
from its own vector representation; (b) the span boundary objective, which
predicts each masked token from the representations of the unmasked tokens at
the start and end of the masked span.

#### SpanBERT (Reimpl)

Our reimplementation of SpanBERT, using exactly the same code, data, and
hyperparameters as Splinter. This baseline aims to control for implementation
differences and measures the effect of replacing masked language modeling with
recurring span selection. Also, this version does not use the span boundary
objective, as Joshi et al. (2020) reported no significant improvements from
using it in question answering.

###  5.2 Pretraining Implementation

We train Splinter-base using Adam Kingma and Ba (2015) for 2.4M training steps
with batches of 256 sequences of length 512.444We used this setting to
approximate SpanBERT’s hyperparameter setting in terms of epochs. That said,
SpanBERT-base was trained for a quarter of the steps (600k steps) using four
times as many examples per batch (1024 sequences). See Section 5.1 for
additional baselines that control for this difference. The learning rate is
warmed up for 10k steps to a maximum value of $10^{-4}$, after which it decays
linearly. As in previous work, we use a dropout rate of 0.1 across all layers.

We follow Devlin et al. (2019) and train on English Wikipedia (preprocessed by
WikiExtractor as in Attardi (2015)) and the Toronto BookCorpus Zhu et al.
(2015). We base our implementation on the official TensorFlow implementation
of BERT, and train on a single eight-core v3 TPU (v3-8) on the Google Cloud
Platform.

###  5.3 Fine-Tuning Implementation

For fine-tuning, we use the hyperparameters from the default configuration of
the HuggingFace Transformers package Wolf et al. (2020).555We did rudimentary
tuning on the number of steps only, using a held-out portion of the SQuAD
training set, since our training sets can be too small for the default values
(e.g., running 10 epochs on 16 examples results in 20 update steps).
Specifically, we train all models using Adam Kingma and Ba (2015) with bias-
corrected moment estimates for few-shot learning Zhang et al. (2021). When
fine-tuning on 1024 examples or less, we train for either 10 epochs or 200
steps (whichever is larger). For full-size datasets, we train for 2 epochs. We
set the batch size to 12 and use a maximal learning rate of $3\cdot 10^{-5}$,
which warms up in the first 10% of the steps, and then decays linearly.

An interesting question is how to fine-tune the QASS layer parameters (i.e.,
the $\mathbf{S}$ and $\mathbf{E}$ matrices in Section 3.2). In our
implementation, we chose to discard the pretrained values and fine-tune from a
random initialization, due to the possible discrepancy between span statistics
in pretraining and fine-tuning datasets. However, we report results on fine-
tuning without resetting the QASS parameters as an ablation study (Section
6.3).

Model | SQuAD | TriviaQA | NQ | NewsQA | SearchQA | HotpotQA | BioASQ | TextbookQA
---|---|---|---|---|---|---|---|---
16 Examples |  |  |  |  |  |  |  |
RoBERTa |  7.7 |  7.5 | 17.3 |  1.4 |  6.9 | 10.5 | 16.7 |  3.3
SpanBERT | 12.5 | 12.8 | 19.7 |  6.0 | 13.0 | 12.6 | 22.0 |  5.6
SpanBERT (Reimpl) | 18.2 | 11.6 | 19.6 |  7.6 | 13.3 | 12.5 | 15.9 |  7.5
Splinter | 54.6 | 18.9 | 27.4 | 20.8 | 26.3 | 24.0 | 28.2 | 19.4
128 Examples |  |  |  |  |  |  |  |
RoBERTa | 43.0 | 19.1 | 30.1 | 16.7 | 27.8 | 27.3 | 46.1 |  8.2
SpanBERT | 48.5 | 24.2 | 32.2 | 17.4 | 34.3 | 35.1 | 55.3 |  9.4
SpanBERT (Reimpl) | 55.8 | 26.3 | 36.0 | 29.5 | 26.3 | 36.6 | 52.2 | 20.9
Splinter | 72.7 | 44.7 | 46.3 | 43.5 | 47.2 | 54.7 | 63.2 | 42.6
1024 Examples |  |  |  |  |  |  |  |
RoBERTa | 73.8 | 46.8 | 54.2 | 47.5 | 54.3 | 61.8 | 84.1 | 35.8
SpanBERT | 77.8 | 50.3 | 57.5 | 49.3 | 60.1 | 67.4 | 89.3 | 42.3
SpanBERT (Reimpl) | 77.8 | 55.5 | 59.5 | 52.2 | 58.9 | 64.6 | 89.0 | 45.7
Splinter | 82.8 | 64.8 | 65.5 | 57.3 | 67.3 | 70.3 | 91.0 | 54.5
Full Dataset |  |  |  |  |  |  |  |
RoBERTa | 90.3 | 74.0 | 79.6 | 69.8 | 81.5 | 78.7 | - | -
SpanBERT | 92.0 | 77.2 | 80.6 | 71.3 | 80.1 | 79.6 | - | -
SpanBERT (Reimpl) | 92.0 | 75.8 | 80.5 | 71.1 | 81.4 | 79.7 | - | -
Splinter | 92.2 | 76.5 | 81.0 | 71.3 | 83.0 | 80.7 | - | -
Table 1: Performance (F1) across all datasets when the number of training
examples is 16, 128, and 1024. We also show performance when training on the
full-sized large datasets (MRQA version). All models have the same capacity to
BERT-base (110M parameters). NQ stands for Natural Questions.

##  6 Results

Our experiments show that Splinter dramatically improves performance in the
challenging few-shot setting, unlocking the ability to train question
answering models with only hundreds of examples. When trained on large
datasets with an order of 100,000 examples, Splinter is competitive with (and
often better than) the baselines. Ablation studies demonstrate the
contributions of both recurring span selection pretraining and the QASS layer.

###  6.1 Few-Shot Learning

Figure 4 shows the F1 score Rajpurkar et al. (2016) of Splinter-base, plotted
against all baselines for two datasets, TriviaQA and TextbookQA, as a function
of the number of training examples (see Figure 6 in the appendix for the
remaining datasets). In addition, Table 1 shows the performance of individual
models when given 16, 128, and 1024 training examples across all datasets (see
Table 3 in the appendix for additional performance and standard deviation
statistics). It is evident that Splinter outperforms all baselines by large
margins.

Let us examine the results on SQuAD, for example. Given 16 training examples,
Splinter obtains 54.6 F1, significantly higher than the best baseline’s 18.2
F1. When the number of training examples is 128, Splinter achieves 72.7 F1,
outperforming the baselines by 17 points (our reimplementation of SpanBERT) to
30 (RoBERTa). When considering 1024 examples, there is a 5-point margin
between Splinter (82.8 F1) and SpanBERT (77.8 F1). The same trend is seen in
the other datasets, whether they are in-domain sampled from larger datasets
(e.g. TriviaQA) or not; in TextbookQA, for instance, we observe absolute gaps
of 9 to 23 F1 between Splinter and the next-best baseline.

###  6.2 High-Resource Regime

Table 1 also shows the performance when fine-tuning on the entire training
set, when an order of 100,000 examples are available. Even though Splinter was
designed for few-shot question answering, it reaches the best result in five
out of six datasets. This result suggests that when the target task is
extractive question answering, it is better to pretrain with our recurring
span selection task than with masked langauge modeling, regardless of the
number of annotated training examples.

Figure 5: Ablation studies on SQuAD and BioASQ datasets. We examine the role
of the QASS layer by fine-tuning it on top of our reimplementation of
SpanBERT. In addition, we test whether it is beneficial to keep the pretrained
parameters of the QASS layer when fine-tuning Splinter.

###  6.3 Ablation Study

We perform an ablation study to better understand the independent
contributions of the pretraining scheme and the QASS layer. We first ablate
the effect of pretraining on recurring span selection by applying the QASS
layer to pretrained masked language models. We then test whether the QASS
layer’s pretrained parameters can be reused in Splinter during fine-tuning
without reinitializion.

#### Independent Contribution of the QASS Layer

While the QASS layer is motivated by our pretraining scheme, it can also be
used without pretraining. We apply a randomly-initialized QASS layer to our
implementation of SpanBERT, and fine-tune it in the few-shot setting. Figure 5
shows the results of this ablation study for two datasets (see Figure 7 in the
appendix for more datasets). We observe that replacing the static span
selection layer with QASS can significantly improve performance on few-shot
question answering. Having said that, most of Splinter’s improvements in the
extremely low data regime do stem from combining the QASS layer with our
pretraining scheme, and this combination still outperforms all other variants
as the amount of data grows.

#### QASS Reinitialization

Between pretraining and fine-tuning, we randomly reinitialize the parameters
of the QASS layer. We now test the effect of fine-tuning with the QASS layer’s
pretrained parameters; intuitively, the more similar the pretraining data is
to the task, the better the pretrained layer will perform. Figure 5 shows that
the advantage of reusing the pretrained QASS is data-dependent, and can result
in both performance gains (e.g. extremely low data in SQuAD) and stagnation
(e.g. BioASQ with 256 examples or more). Other datasets exhibit similar trends
(see appendix). We identify three conditions that determine whether keeping
the pretrained head is preferable: (1) when the number of training examples is
extremely low, (2) when the target domain is similar to that used at
pretraining (e.g. Wikipedia), and (3) when the questions are relatively simple
(e.g. SQuAD versus HotpotQA). The latter two conditions pertain to the
compatibility between pretraining and fine-tuning tasks; the information
learned in the QASS layer is useful as long as the input and output
distribution of the task are close to those seen at pretraining time.

Model | Representation Similarity
---|---
RoBERTa | 0.29
SpanBERT | 0.23
SpanBERT (Reimpl) | 0.19
Splinter | 0.89
Table 2: Cosine similarity of the representations produced by the transformer
encoder before and after fine-tuning on 128 SQuAD examples.

###  6.4 Analysis

The recurring span selection objective was designed to emulate extractive
question answering using unlabeled text. How similar is it to the actual
target task? To answer this question, we measure how much each pretrained
model’s functionality has changed after fine-tuning on 128 examples of SQuAD.
For the purpose of this analysis, we measure change in functionality by
examining the vector representation of each token as produced by the
transformer encoder; specifically, we measure the cosine similarity between
the vector produced by the pretrained model and the one produced by the fine-
tuned model, given exactly the same input. We average these similarities
across every token of 200 examples from SQuAD’s test set.

Table 2 shows that Splinter’s outputs are very similar before and after fine-
tuning (0.89 average cosine similarity), while the other models’
representations seem to change drastically. This suggests that fine-tuning
with even 128 question-answering examples makes significant modifications to
the functionality of pretrained masked language models. Splinter’s
pretraining, on the other hand, is much more similar to the fine-tuning task,
resulting in much more modest changes to the produced vector representations.

##  7 Related Work

The remarkable results of GPT-3 Brown et al. (2020) have inspired a renewed
interest in few-shot learning. While some work focuses on classification tasks
Schick and Schütze (2020); Gao et al. (2021), our work investigates few-shot
learning in the context of extractive question answering.

One approach to this problem is to create synthetic text-question-answer
examples. Both Lewis et al. (2019) and Glass et al. (2020) use the traditional
NLP pipeline to select noun phrases and named entities in Wikipedia paragraphs
as potential answers, which are then masked from the context to create pseudo-
questions. Lewis et al. (2019) use methods from unsupervised machine
translation to translate the pseudo-questions into real ones, while Glass et
al. (2020) keep the pseudo-questions but use information retrieval to find new
text passages that can answer them. Both works assume access to language- and
domain-specific NLP tools such as part-of-speech taggers, syntactic parsers,
and named-entity recognizers, which might not always be available. Our work
deviates from this approach by exploiting the natural phenomenon of recurring
spans in order to generate multiple question-answer pairs per text passage,
without assuming any language- or domain-specific models or resources are
available beyond plain text.

Similar ideas to recurring span selection were used for creating synthetic
coreference resolution examples Kocijan et al. (2019); Varkel and Globerson
(2020), which mask single words that occur multiple times in the same context.
CorefBERT Ye et al. (2020) combines this approach with a copy mechanism for
predicting the masked word during pretraining, alongside the masked language
modeling objective. Unlike our approach, which was designed to align well with
span selection, CorefBERT masks only single-word nouns (rather than arbitrary
spans) and replaces each token in the word with a separate mask token (rather
than a single mask for the entire multi-token word). Therefore, it does not
emulate extractive question answering. We did not add CorefBERT as a baseline
since the performance of both CorefBERT-base and CorefBERT-large was lower
than SpanBERT-base’s performance on the full-data MRQA benchmark, and
pretraining CorefBERT from scratch was beyond our available computational
resources.

##  8 Conclusion

We explore the few-shot setting of extractive question answering, and
demonstrate that existing methods, based on fine-tuning large pretrained
language models, fail in this setup. We propose a new pretraining scheme and
architecture for span selection that lead to dramatic improvements, reaching
surprisingly good results even when only an order of a hundred examples are
available. Our work shows that choices that are often deemed unimportant when
enough data is available, again become crucial in the few-shot setting,
opening the door to new methods that take advantage of prior knowledge on the
downstream task during model development.

## Acknowledgements

This project was funded by the European Research Council (ERC) under the
European Unions Horizon 2020 research and innovation programme (grant ERC HOLI
819080), the Blavatnik Fund, the Alon Scholarship, the Yandex Initiative for
Machine Learning and Intel Corporation. We thank Google’s TPU Research Cloud
(TRC) for their support in providing TPUs for this research.

## References

  * Attardi (2015) Giusepppe Attardi. 2015.  WikiExtractor.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_.
  * Clark et al. (2020) Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020.  TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages.  _Transactions of the Association for Computational Linguistics_ , 8:454–470.
  * Dasigi et al. (2019) Pradeep Dasigi, Nelson F. Liu, Ana Marasović, Noah A. Smith, and Matt Gardner. 2019.  Quoref: A reading comprehension dataset with questions requiring coreferential reasoning.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5925–5932, Hong Kong, China. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dozat and Manning (2017) Timothy Dozat and Christopher D. Manning. 2017.  Deep biaffine attention for neural dependency parsing.  In _ICLR 2017_.
  * Dua et al. (2019) Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019.  DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 2368–2378, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dunn et al. (2017) Matthew Dunn, Levent Sagun, Mike Higgins, V. Ugur Guney, Volkan Cirik, and Kyunghyun Cho. 2017.  SearchQA: A new Q&A dataset augmented with context from a search engine.
  * Fisch et al. (2019) Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. 2019\.  MRQA 2019 shared task: Evaluating generalization in reading comprehension.  In _Proceedings of the 2nd Workshop on Machine Reading for Question Answering_ , pages 1–13, Hong Kong, China. Association for Computational Linguistics.
  * Gao et al. (2021) Tianyu Gao, Adam Fisch, and Danqi Chen. 2021.  Making pre-trained language models better few-shot learners.  In _Association for Computational Linguistics (ACL)_.
  * Glass et al. (2020) Michael Glass, Alfio Gliozzo, Rishav Chakravarti, Anthony Ferritto, Lin Pan, G P Shrivatsa Bhargav, Dinesh Garg, and Avi Sil. 2020.  Span selection pre-training for question answering.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 2773–2782, Online. Association for Computational Linguistics.
  * Guu et al. (2020) Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020.  Retrieval augmented language model pre-training.  In _Proceedings of the 37th International Conference on Machine Learning_ , volume 119 of _Proceedings of Machine Learning Research_ , pages 3929–3938. PMLR.
  * Joshi et al. (2020) Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020.  SpanBERT: Improving pre-training by representing and predicting spans.  _Transactions of the Association for Computational Linguistics_ , 8:64–77.
  * Joshi et al. (2017) Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017.  TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1601–1611, Vancouver, Canada. Association for Computational Linguistics.
  * Kembhavi et al. (2017) Aniruddha Kembhavi, Minjoon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hannaneh Hajishirzi. 2017.  Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _ICLR 2015_.
  * Kocijan et al. (2019) Vid Kocijan, Oana-Maria Camburu, Ana-Maria Cretu, Yordan Yordanov, Phil Blunsom, and Thomas Lukasiewicz. 2019.  WikiCREM: A large unsupervised corpus for coreference resolution.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4303–4312, Hong Kong, China. Association for Computational Linguistics.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.  Natural questions: A benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:453–466.
  * Levy et al. (2017) Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. 2017.  Zero-shot relation extraction via reading comprehension.  In _Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017)_ , pages 333–342, Vancouver, Canada. Association for Computational Linguistics.
  * Lewis et al. (2019) Patrick Lewis, Ludovic Denoyer, and Sebastian Riedel. 2019.  Unsupervised question answering by cloze translation.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4896–4910, Florence, Italy. Association for Computational Linguistics.
  * Lewis et al. (2020) Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2020\.  MLQA: Evaluating cross-lingual extractive question answering.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7315–7330, Online. Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A robustly optimized bert pretraining approach.
  * Rajpurkar et al. (2018) Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018.  Know what you don’t know: Unanswerable questions for SQuAD.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 784–789, Melbourne, Australia. Association for Computational Linguistics.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
  * Schick and Schütze (2020) Timo Schick and Hinrich Schütze. 2020.  Exploiting cloze questions for few shot text classification and natural language inference.
  * Trischler et al. (2017) Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017.  NewsQA: A machine comprehension dataset.  In _Proceedings of the 2nd Workshop on Representation Learning for NLP_ , pages 191–200, Vancouver, Canada. Association for Computational Linguistics.
  * Tsatsaronis et al. (2015) George Tsatsaronis, Georgios Balikas, Prodromos Malakasiotis, Ioannis Partalas, Matthias Zschunke, Michael R. Alvers, Dirk Weissenborn, Anastasia Krithara, Sergios Petridis, Dimitris Polychronopoulos, Yannis Almirantis, John Pavlopoulos, Nicolas Baskiotis, Patrick Gallinari, Thierry Artiéres, Axel-Cyrille Ngonga Ngomo, Norman Heino, Eric Gaussier, Liliana Barrio-Alvers, Michael Schroeder, Ion Androutsopoulos, and Georgios Paliouras. 2015.  An overview of the BIOASQ large-scale biomedical semantic indexing and question answering competition.  _BMC Bioinformatics_ , 16(1):138.
  * Varkel and Globerson (2020) Yuval Varkel and Amir Globerson. 2020.  Pre-training mention representations in coreference models.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 8534–8540, Online. Association for Computational Linguistics.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems 30_ , pages 5998–6008.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018.  HotpotQA: A dataset for diverse, explainable multi-hop question answering.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2369–2380, Brussels, Belgium. Association for Computational Linguistics.
  * Ye et al. (2020) Deming Ye, Yankai Lin, Jiaju Du, Zhenghao Liu, Peng Li, Maosong Sun, and Zhiyuan Liu. 2020.  Coreferential Reasoning Learning for Language Representation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7170–7186, Online. Association for Computational Linguistics.
  * Zhang et al. (2021) Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q. Weinberger, and Yoav Artzi. 2021\.  Revisiting few-sample BERT fine-tuning.  In _ICLR_.
  * Zhu et al. (2015) Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015.  Aligning books and movies: Towards story-like visual explanations by watching movies and reading books.  In _The IEEE International Conference on Computer Vision (ICCV)_.

##  Appendix A Additional Results

#### Few-Shot Results

Figure 6 shows the results on the six few-shot question answering datasets not
included in Figure 4. In addition, we give the full raw results (including
standard deviation) in Table 3.

#### Ablation Studies

Figure 7 shows results of ablation studies on the six question answering
datasets not included in Figure 5.

Figure 6: Results complementary to Table 1. Performance (F1) of Splinter-base (green line, triangular points), compared to all baselines as a function of the number of training examples on 4 datasets. Each point reflects the average performance across 5 randomly-sampled training sets of the same size. Model | SQuAD | TriviaQA | NQ | NewsQA | SearchQA | HotpotQA | BioASQ | TBQA
---|---|---|---|---|---|---|---|---
16 Examples |  |  |  |  |  |  |  |
RoBERTa |  7.7 (4.3) |  7.5 (4.4) | 17.3 (3.3) |  1.4 (0.8) |  6.9 (2.7) | 10.5 (2.5) | 16.7 (7.1) |  3.3 (2.1)
SpanBERT | 12.5 (5.7) | 12.8 (5.4) | 19.7 (3.6) |  6.0 (1.6) | 13.0 (4.2) | 12.6 (4.3) | 22.0 (4.6) |  5.6 (2.5)
(Reimpl) | 18.2 (6.7) | 11.6 (2.1) | 19.6 (3.0) |  7.6 (4.1) | 13.3 (6.0) | 12.5 (5.5) | 15.9 (4.4) |  7.5 (2.9)
Splinter |  54.6 (6.4) |  18.9 (4.1) |  27.4 (4.6) |  20.8 (2.7) |  26.3 (3.9) |  24.0 (5.0) |  28.2 (4.9) |  19.4 (4.6)
32 Examples |  |  |  |  |  |  |  |
RoBERTa | 18.2 (5.1) | 10.5 (1.8) | 22.9 (0.7) |  3.2 (1.7) | 13.5 (1.8) | 10.4 (1.9) | 23.3 (6.6) | 4.3 (0.9)
SpanBERT | 19.0 (4.6) | 19.0 (4.8) | 23.5 (0.9) |  7.5 (1.3) | 20.1 (3.9) | 14.4 (2.9) | 32.5 (3.5) |  7.4 (1.1)
(Reimpl) | 25.8 (7.7) | 15.1 (6.4) | 25.1 (1.6) |  7.2 (4.6) | 14.6 (8.5) | 13.2 (3.5) | 25.1 (3.3) |  7.6 (2.3)
Splinter |  59.2 (2.1) |  28.9 (3.1) |  33.6 (2.4) |  27.5 (3.2) |  34.8 (1.8) |  34.7 (3.9) |  36.5 (3.2) |  27.6 (4.3)
64 Examples |  |  |  |  |  |  |  |
RoBERTa | 28.4 (1.7) | 12.5 (1.4) | 24.2 (1.0) |  4.6 (2.8) | 19.8 (2.4) | 15.0 (3.9) | 34.0 (1.8) |  5.4 (1.1)
SpanBERT | 33.6 (4.3) | 22.8 (2.6) | 28.4 (1.8) |  8.8 (2.4) | 26.7 (2.9) | 21.8 (1.5) | 43.9 (4.5) |  7.4 (1.2)
(Reimpl) | 45.8 (3.3) | 15.9 (6.4) | 29.7 (1.5) | 12.5 (4.3) | 18.0 (4.6) | 23.3 (1.1) | 35.3 (3.1) | 13.0 (6.9)
Splinter |  65.2 (1.4) |  35.5 (3.7) |  38.2 (2.3) |  37.4 (1.2) |  39.8 (3.6) |  45.4 (2.3) |  49.5 (3.6) |  35.9 (3.1)
128 Examples |  |  |  |  |  |  |  |
RoBERTa | 43.0 (7.1) | 19.1 (2.9) | 30.1 (1.9) | 16.7 (3.8) | 27.8 (2.5) | 27.3 (3.9) | 46.1 (1.4) |  8.2 (1.1)
SpanBERT | 48.5 (7.3) | 24.2 (2.1) | 32.2 (3.2) | 17.4 (3.1) | 34.3 (1.1) | 35.1 (4.2) | 55.3 (3.8) |  9.4 (3.0)
(Reimpl) | 55.8 (3.7) | 26.3 (2.1) | 36.0 (1.9) | 29.5 (7.3) | 26.3 (4.3) | 36.6 (3.4) | 52.2 (3.2) | 20.9 (5.1)
Splinter |  72.7 (1.0) |  44.7 (3.9) |  46.3 (0.8) |  43.5 (1.3) |  47.2 (3.5) |  54.7 (1.4) |  63.2 (4.1) |  42.6 (2.5)
256 Examples |  |  |  |  |  |  |  |
RoBERTa | 56.1 (5.2) | 26.9 (3.5) | 36.0 (3.2) | 31.2 (2.4) | 37.5 (1.7) | 42.7 (3.1) | 63.5 (1.8) | 13.5 (1.9)
SpanBERT | 55.2 (8.8) | 34.0 (5.7) | 41.3 (2.2) | 34.7 (4.1) | 42.3 (4.1) | 49.4 (4.0) | 67.5 (3.9) | 18.2 (4.5)
(Reimpl) | 67.1 (2.1) | 39.4 (4.0) | 44.4 (3.2) | 41.8 (1.8) | 41.5 (3.2) | 51.5 (2.8) | 66.4 (2.8) | 31.1 (3.4)
Splinter |  76.8 (0.6) |  57.2 (2.2) |  54.6 (1.2) |  49.0 (0.4) |  55.7 (1.9) |  62.0 (1.6) |  77.4 (2.0) |  48.5 (2.2)
512 Examples |  |  |  |  |  |  |  |
RoBERTa | 67.3 (0.7) | 38.7 (3.8) | 46.7 (2.2) | 41.5 (2.2) | 46.9 (1.6) | 56.7 (1.3) | 77.0 (1.9) | 27.0 (2.2)
SpanBERT | 70.0 (4.3) | 44.2 (2.9) | 51.5 (1.8) | 42.4 (2.6) | 53.9 (3.2) | 61.6 (1.7) | 80.3 (3.0) | 33.7 (3.4)
(Reimpl) | 73.4 (0.4) | 50.4 (2.8) | 52.5 (1.9) | 47.6 (1.3) | 48.8 (4.1) | 59.5 (1.5) | 79.0 (1.9) | 40.2 (0.8)
Splinter |  80.1 (0.4) |  61.9 (1.8) |  61.4 (1.1) |  53.2 (0.9) |  63.1 (1.6) |  66.2 (0.6) |  84.8 (0.9) |  54.2 (1.7)
1024 Examples |  |  |  |  |  |  |  |
RoBERTa | 73.8 (0.8) | 46.8 (0.9) | 54.2 (1.1) | 47.5 (1.1) | 54.3 (1.2) | 61.8 (1.3) | 84.1 (1.1) | 35.8 (2.0)
SpanBERT | 77.8 (0.9) | 50.3 (4.0) | 57.5 (0.9) | 49.3 (2.0) | 60.1 (2.2) | 67.4 (1.6) | 89.3 (0.6) | 42.3 (1.9)
(Reimpl) | 77.8 (0.6) | 55.5 (1.9) | 59.5 (1.7) | 52.2 (1.2) | 58.9 (1.9) | 64.6 (1.2) | 89.0 (1.8) | 45.7 (1.5)
Splinter |  82.8 (0.8) |  64.8 (0.9) |  65.5 (0.5) |  57.3 (0.8) |  67.3 (1.3) |  70.3 (0.8) |  91.0 (1.0) |  54.5 (1.5)
Table 3: Average performance (F1) across all datasets and training set sizes.
We add the standard deviation over the five seeds for each setting in
parentheses. NQ and TBQA stand for Natural Questions and TextbookQA
respectively. (Reimpl) stands for the SpanBERT (Reimpl) baseline (see Section
5.1).

Figure 7: Results complementary to ablation studies (Section 6.3). We examine
the role of QASS layer by fine-tuning it on top of our SpanBERT. In addition,
we test whether it is beneficial to keep the parameters of QASS from
pretraining (Splinter with Head).