#  STUDY OF THE PROCESS $e^{+}e^{-}\to
K_{S}^{0}K^{\pm}\pi^{\mp}\pi^{+}\pi^{-}$ IN THE C.M. ENERGY RANGE 1.6–2.0 GEV
WITH THE CMD-3 DETECTOR

R.R. Akhmetshin  A.N. Amirkhanov  A.V. Anisenkov  V.M. Aulchenko  N.S.
Bashtovoy  D.E. Berkaev  A.E. Bondar  A.V. Bragin  S.I. Eidelman D.A. Epifanov
L.B. Epshteyn  A.L. Erofeev  G.V. Fedotovich  A.O. Gorkovenko  F.J.
Grancagnolo  A.A. Grebenuk  S.S. Gribanov  D.N. Grigoriev  F.V. Ignatov  V.L.
Ivanov  S.V. Karpov  V.F. Kazanin  A.N. Kirpotin  I.A. Koop  A.A. Korobov
A.N. Kozyrev  E.A. Kozyrev  P.P. Krokovny  A.S. Kuzmin  I.B. Logashenko  P.A.
Lukin  K.Yu. Mikhailov  I.V. Obraztsov  V.S. Okhapkin  A.V. Otboev  Yu.N.
Pestov  A.S. Popov  G.P. Razuvaev  Yu.A. Rogovsky  A.A. Ruban  N.M. Ryskulov
A.E. Ryzhenenkov  A.V. Semenov  A.I. Senchenko  P.Yu. Shatunov  Yu.M. Shatunov
V.E. Shebalin  D.N. Shemyakin  B.A. Shwartz  D.B. Shwartz  A.L. Sibidanov
E.P. Solodov111Corresponding author<EMAIL_ADDRESS>V.M. Titov  A.A.
Talyshev  S.S. Tolmachev  A.I. Vorobiov  I.M. Zemlyansky  D.S. Zhadan  A.S.
Zubakin  Yu.V. Yudin  Budker Institute of Nuclear Physics, SB RAS,
Novosibirsk, 630090, Russia  Novosibirsk State University, Novosibirsk,
630090, Russia  Novosibirsk State Technical University, Novosibirsk, 630092,
Russia  University of Victoria, Victoria, BC, Canada V8W 3P6  Lebedev Physical
Institute RAS, Moscow, 119333, Russia  Instituto Nazionale di Fisica Nucleare,
Sezione di Lecce, Lecce, Italy

###### Abstract

A cross section of the process $e^{+}e^{-}\to
K_{S}^{0}K^{\pm}\pi^{\mp}\pi^{+}\pi^{-}$ has been measured for the first time
using a data sample of 185.4 pb-1 collected with the CMD-3 detector at the
VEPP-2000 $e^{+}e^{-}$ collider. With the $K_{S}^{0}\to\pi^{+}\pi^{-}$ decay
detection, 373$\pm$20 and 514$\pm$28 signal events have been selected with six
and five reconstructed tracks, respectively, in the center-of-mass energy
range 1.6–2.0 GeV. The total systematic uncertainty of the cross section is
about 15%. A study of the production dynamics allows us to extract a
contribution from the $e^{+}e^{-}\to f_{1}(1285)\pi^{+}\pi^{-}$ intermediate
state and to measure the corresponding cross section. The intermediate states
with the $f_{1}(1420)$ and $f_{1}(1510)$ resonances have been observed.

##  1  Introduction

$e^{+}e^{-}$ annihilation into hadrons below 2 GeV is rich for various multi-
particle final states. Their detailed studies are important for the
development of the phenomenological models describing strong interactions at
low energies. The contributions from the different intermediate states are
particularly important for the calculations of the hadronic vacuum
polarization (HVP), and the calculation of the hadronic contribution to the
muon anomalous magnetic moment [1, 2, 3]. The reaction $e^{+}e^{-}\to
K\bar{K}3\pi$ has been studied before only in the
$K^{+}K^{-}\pi^{+}\pi^{-}\pi^{0}$ mode by the BaBar collaboration [4], based
on the Initial-State Radiation (ISR) method. This mode is dominated by the
$\phi(1020)\eta$ and $\omega(782)K^{+}K^{-}$ intermediate states, and possible
contributions from the other states below 2 GeV are not observed with the
available data. From the other hand, the $e^{+}e^{-}\to K\bar{K}3\pi$ process
includes many isospin combinations of the kaons and pions, and a small signal
in each mode can give a sizable effect in the total cross section value. A
measurement of the $e^{+}e^{-}\to K\bar{K}3\pi$ cross section for the
different isospin combinations of the kaons and pions, and a study of the
production dynamics can further improve the accuracy of the HVP calculations.

In this paper we report an analysis of the $e^{+}e^{-}\to
K_{S}^{0}K^{\pm}\pi^{\mp}\pi^{+}\pi^{-}$ reaction using a data sample of 185.4
pb-1 collected with the CMD-3 detector in the Ec.m.=1.6–2.0 GeV center-of-mass
energy range. These data were collected during six energy scans, with 5–10 MeV
c.m. energy step each, performed at the VEPP-2000 $e^{+}e^{-}$ collider [5, 6,
7, 8] in 2011–2021 experimental runs. Starting in 2017 a beam energy was
monitored by the back-scattering laser-light system [9, 10], providing an
absolute beam-energy measurement with better than 0.1 MeV uncertainty at every
10–20 minutes of the data taking. In earlier runs the beam energy was
determined using measurements of the charged track momenta in the detector
magnetic field with an about 1 MeV uncertainty.

The general-purpose detector CMD-3 has been described in detail elsewhere
[11]. Its tracking system consists of a cylindrical drift chamber (DC) [12]
and double-layer multi-wire proportional Z-chamber, both are also used for a
charged track trigger, and both inside a thin (0.2 X0) superconducting
solenoid with a field of 1.3 T. The tracking system provides the (96–99)%
tracking efficiency in about 70% of the solid angle. The ionisation losses for
the charged tracks in the DC are measured with the 15% accuracy. A liquid
xenon (LXe) barrel calorimeter with a 5.4 X0 thickness has a fine electrode
structure, providing 1–2 mm spatial resolution for the photons independently
of their energy [13], and shares a cryostat vacuum volume with the solenoid. A
barrel CsI crystal calorimeter with a thickness of 8.1 X0 surrounds the LXe
calorimeter, while an end cap BGO calorimeter with a thickness of 13.4 X0 is
placed inside the solenoid [14]. Altogether, the calorimeters cover 90% of the
solid angle and their amplitude signals provide information for the neutral
trigger. A charged trigger requirement of at least one track in DC is quite
loose that provides almost 100% trigger efficiency for the process under study
with five or six detected tracks. About 80% of these events have a sufficient
energy deposition in the calorimeter for the independent neutral trigger:
these events are used to control the charged trigger efficiency. A luminosity
is measured using the Bhabha scattering events at large angles with about 1.5%
systematic uncertainty [15].

To understand the detector response to the processes under study and to obtain
a detection efficiency, we have developed Monte Carlo (MC) simulation of our
detector based on the GEANT4 [16] package, in which all simulated events pass
the whole reconstruction and selection procedure. The MC simulation uses
primary generators with the matrix elements for the
$K_{S}^{0}K^{\pm}\pi^{\mp}\pi^{+}\pi^{-}$ final state with the
$f_{1}(1285)\rho(770)$ and $f_{1}(1420,1510)\rho(770)$ intermediate states. We
simulate the $f_{1}(1285)$ resonance decaying to $a_{0}(980)\pi$, which gives
about 9% of $f_{1}(1285)$ decay rate to $K\bar{K}\pi$, while
$f_{1}(1420,1510)$ predominantly decays to $K^{*}(892)K$ final state [17]. The
primary generator with the $K_{S}^{0}K^{\pm}\pi^{\mp}\pi^{+}\pi^{-}$ events
uniformly distributed in the phase-space (PS) has been also tested. The
primary generators include radiation of the photons by an initial electron and
positron, calculated according to Ref. [18].

##  2 Selection of $e^{+}e^{-}\to K_{S}^{0}K^{\pm}\pi^{\mp}\pi^{+}\pi^{-}$
events

The analysis procedure is similar to our study of the production of six
charged pions or $K_{S}^{0}K_{S}^{0}\pi^{+}\pi^{-}$ final state described in
Refs. [19, 20]. Candidate events are required to have six or five tracks with
the total charge zero or $\pm 1$, each having:

  * 1.

more than five hits in the DC;

  * 2.

a transverse momentum larger than 40 MeV/c;

  * 3.

a minimum distance from a track to the beam axis in the transverse plane of
less than 6 cm, that allows reconstruction of a decay point of $K_{S}^{0}$ up
to this distances;

  * 4.

a minimum distance from a track to the center of the interaction region along
the beam axis Z of less than 15 cm.

Reconstructed momenta and angles of the tracks for the five- and six-track
events are used for the further selection.

In our reconstruction procedure we create a list of the
$K_{S}^{0}\to\pi^{+}\pi^{-}$ candidates which includes every pair of the
oppositely charged tracks, assuming them to be pions, with an invariant mass
within $\pm 80$ MeV/c2 from the $K_{S}^{0}$ mass [17] and a common vertex
point within a spacial uncertainty of the DC. We calculate momentum and energy
for the $K_{S}^{0}$ candidate taking the values of the pions mass from Ref.
[17].

(a)(b)
(c)(d)

Figure 1:  (a) The invariant mass of the $K_{S}^{0}$ candidate for the data
(dots) and simulation (histogram). The lines show selections for the signal
events and for the control region to estimate the background level. (b) The
transverse distance for the $K_{S}^{0}$ candidate vertices from the beam axis.
Solid histogram is for simulation, dashed histogram is for the background. (c)
The dE/dX distribution vs momentum for all events with six tracks. Black
squares indicate selected charged kaon candidates. (d) The transverse distance
for the charged kaon candidate tracks from the beam axis for the data (dots)
and simulation (histogram).

At the first stage of the signal event selection we require at least one
$K_{S}^{0}$ candidate with two independent tracks plus three or four
additional charged tracks, which are required to be within 0.35 cm from the
beam axis. If there are more than one $K_{S}^{0}$ candidate, a candidate pair
with a minimal deviation from the $K_{S}^{0}$ mass is retained. A charged kaon
candidate is selected from these beam-originated tracks using the ionization
loss measurement in the DC as described below.

The signal event candidate is required to have at least one $K_{S}^{0}$ and at
least one $K^{\pm}$ for the next selection stage.

Figure 1(a) shows the invariant mass of the $K_{S}^{0}\to\pi^{+}\pi^{-}$
candidates for the data (points) and MC simulation (histogram). The data from
the energy intervals above E${}_{\rm c.m.}>$1950 MeV are combined for the
presented histograms. The vertical lines show selections for the signal events
and events for a background level estimate by using the side bands with equal
to the signal mass intervals. The radial distances of the $K_{S}^{0}$ decay
vertices from the beam axis are shown in Fig. 1(b) for the data (points) and
MC simulation (histogram) for the events in the signal region. The dashed
histogram represents a background contribution, estimated from the side bands
of Fig. 1(a). Figure 1(c) shows a scatter plot for the dE/dX values vs
momentum for the six tracks of the events from Fig. 1(a). Clear bands for the
pions and kaons are seen. The signals from the kaons and pions are overlapped
at large momenta, and we use a selection boundary formed by the two linear
functions. The most critical is the horizontal boundary above the 300 MeV/c
momentum. This boundary is set at the two sigma level above the average dE/dX
value for the pions, and rejects about 50% of the kaons in this momentum
range. The black dots show the selected charged kaon candidates. The radial
distance from the beam axis for the charged kaon tracks is shown in Fig. 1(d).
Our simulation well describes the experimental distributions.

(a)(b)

Figure 2:  The scatter plot of the difference between the energy of
$K_{S}^{0}K^{\pm}\pi^{\mp}\pi^{+}\pi^{-}$ candidates and c.m. energy vs total
momentum, $\Delta\rm E$, for events with six tracks (a) and events with five
tracks (b). The stars are for data, while the signal simulation is shown by
red (in color version) dots; the line shows the applied selection.

For the six- and five-track $K_{S}^{0}K^{\pm}\pi^{\mp}\pi^{+}\pi^{-}$
candidates we calculate the total energy of two kaons and three pions: for the
five-track candidates a missing momentum is used to calculate an energy of the
lost pion. Figure 2 shows a scatter plot of the difference between the total
energy and c.m. energy, $\rm\Delta E=E_{tot}-E_{\rm c.m.}$, vs the total
momentum, $\rm P_{\rm tot}$, of six- (a) and five-track (b) candidates. A
clear signal of the $e^{+}e^{-}\to K_{S}^{0}K^{\pm}\pi^{\mp}\pi^{+}\pi^{-}$
reaction is seen in Fig. 2(a) as a cluster of the stars near zero, in
agreement with the expectation from the simulation shown by (red in the color
version) dots. We require $\rm P_{\rm tot}$ to be less than 180 MeV/c, thus
reducing a number of signal events with the hard radiative photons.

The expected signal of the five-track candidates has the $\rm\Delta E_{5tr}$
value near zero, and the $\rm P_{tot}$ value is distributed up to about 500
MeV/c, as shown by the (red) dots from the signal MC simulation in Fig. 2(b).
The (black) stars show our data: signal events are clearly seen.

(a)(b)

Figure 3:  (a) The difference between the energy of the
$K_{S}^{0}K^{\pm}\pi^{\mp}\pi^{+}\pi^{-}$ candidates and c.m. energy,
$\Delta\rm E$, after selection shown by the line in Fig. 2(a) for the six-
track (a) and the five-track (b) events. All the energy intervals are summed.
The histograms show the background contribution.

The background contribution from the $e^{+}e^{-}\to 3(\pi^{+}\pi^{-})$
reaction, seen as a cluster at $\Delta\rm E=$ 200 MeV in Fig. 2(a), is
effectively reduced by a requirement of the minimum distance between any track
from the $K_{S}^{0}$ decay and the beam axis to be larger than 0.1 cm. A
signal loss due to this requirement is small as seen in Fig. 1(b). Another
significant background from the $e^{+}e^{-}\to
K_{S}^{0}K^{\pm}\pi^{\mp}\pi^{0}$ reaction with additional tracks from the
photon conversion is reduced by a requirement of missing mass to any of the
$K_{S}^{0}K^{\pm}\pi^{\mp}$ combination to be larger than two pion masses.

Figure 3 shows the projection plots of Fig. 2, $\rm\Delta E$, for the six-
track (a) and the five-track (b) events with all above selection applied: the
dots present events from the signal region, while the histograms are the
events from the side bands of Fig. 1(a), which are used to subtract the
remaining background contribution. All energy intervals are summed. The
background contribution is small: almost negligible for the six-track events
and is about 15% for the five-track events.

As seen in Fig. 1(c), the dE/dX values for the kaons and pions are
significantly overlapped at large momentum, and above the selection boundary
we observe about 20% of the events with more than one charged kaon candidate.
In this case we apply kaon mass to only one candidate, assume all the other
tracks to be pions, test all combinations, and retain a combination with the
$\rm\Delta E$ closest to zero. The MC simulation shows that the procedure
relatively well recover the leakage of the pions to the kaon selections.

(a)(b)

Figure 4:  (a) The difference between the energy of the
$K_{S}^{0}K^{\pm}\pi^{\mp}\pi^{+}\pi^{-}$ candidates and c.m. energy,
$\Delta\rm E$, after background subtraction for six-track (a) and five-track
events (b) for eight c.m. energy intervals (dots): left to right, top to
bottom according to the last eight lines of Table 1. Histograms show expected
signals from simulation, normalized to the total number of events in each
plot.

The observed signal is small, and to get a reasonable number of events we
combine our scanned points from the early runs into eight energy intervals, as
shown in Table 1, while the latest scans with the larger integrated luminosity
are presented as individual energy points. To obtain the number of signal
events, we use $\rm\Delta E$ distributions of Fig. 3 after the background
subtraction for each energy interval for the six- and the five-track events.
We count remaining events in the $\pm$70 MeV region for the six-track events,
and in the $\pm$150 MeV region for the five-track events. The obtained
differences are shown in Fig. 4 by dots for the six- (a) and the five-track
(b) events: from left to right, from top to bottom according to the last eight
energy points of Table 1. The histograms show expected signals from the
simulation. In total, we obtain 373$\pm$20 and 514$\pm$28 six- and five-track
signal events, respectively. The numbers of selected events in each energy
interval are listed in Table 1.

##  3 Study of the production dynamics

(a)(b)(c)

Figure 5: (a) The $m(K_{S}^{0}K^{\pm}\pi^{\mp})$ invariant mass distribution
(dots) (two entries per event, sum for all energy intervals). The curve and
the histogram show the fit to the combinatorial background, estimated from a
wrong-sign combination, $m(K_{S}^{0}K^{\pm}\pi^{\pm})$ (squares). (b) The
$m(K_{S}^{0}K^{\pm}\pi^{\mp})$ invariant mass distribution (dots) for
$\mathrm{E}_{\mathrm{c}.m.}=\mathrm{2007~{}MeV}$. Blue, green and magenta
histograms are simulated contribution from the $f_{1}(1285)$, $f_{1}(1420)$
and $f_{1}(1510)$ resonances, while red histogram is a sum of them. (c) The
same as (b) after combinatorial background subtraction. The histogram is the
sum of the three resonances from the simulation.

The events from the side bands of Fig. 1(a) have no structures, and are used
to subtract the background in the other distributions. The background-
subtracted invariant mass for the two kaons and one pion,
$m(K_{S}^{0}K^{\pm}\pi^{\mp})$, (two entries per event) is shown in Fig. 5(a)
by points: all energy intervals are summed. We use both six- and five-track
events assigning the missing four-momenta to the lost pion for the latter
case. This distribution indicates a presence of a narrow resonance which is
interpreted as $f_{1}(1285)$ from the $e^{+}e^{-}\to f_{1}(1285)\rho(770)$
reaction. A bump at 1400 MeV/c2 can be interpreted as $f_{1}(1420)$ resonance.
These resonances were previously studied in proton-proton interactions, see
for example Ref. [21], and have relatively well determined parameters [17].
The curve and the histogram show the fit to the combinatorial background,
obtained from a wrong-sign combination, $m(K_{S}^{0}K^{\pm}\pi^{\pm})$.

Figure 5(b) shows the same distribution for the events from the E${}_{\rm
c.m.}=2007~{}\rm MeV$ energy point in comparison with the MC simulation. We
simulate production of the $f_{1}(1285)$ and $f_{1}(1420)$ resonances
separately and show the expected contribution by blue and green histograms:
here and below we perform an approximate weight of the simulated events from
the different channels to compare with the data for a demonstration. It is
seen, that in addition to the $f_{1}(1285)$ and $f_{1}(1420)$ we need to add
the $e^{+}e^{-}\to f_{1}(1510)\rho$ reaction with approximately the same
weight as $f_{1}(1420)$, pink histogram, to describe the events at higher
masses. The $f_{1}(1510)$ resonance has the same decay modes as $f_{1}(1420)$,
and it has some inconsistency in the parameters [17], so its mass and width
can be varied in a relatively wide range. The solid red histogram shows a sum
of these three simulated reactions with approximately equal weights,
demonstrating that the above reactions describe the observed mass
distribution.

Figure 5(c) shows the events from Fig. 5(b) when the combinatorial background
is subtracted by using the wrong-sign mass distribution: the signal from
$f_{1}(1285)$ is seen almost isolated, while the bump at higher masses can be
explained as a sum of the $f_{1}(1510)$ and $f_{1}(1420)$ resonances, which
are wide and highly overlapped.

(a)(b)

Figure 6:  The experimental $m(K^{\pm}\pi^{\mp})$ (a) and
$m(K_{S}^{0}\pi^{\pm})$ (b) invariant mass distributions (two entries per
event) in comparison with simulation (red histogram)). The blue and green
histograms are the MC simulation contributions from the $e^{+}e^{-}\to
f_{1}(1285)\rho$ and $e^{+}e^{-}\to f_{1}(1420,1510)\rho$ reactions,
respectively.

The existence of the $f_{1}(1510)$ in addition to $f_{1}(1420)$ is supported
by a study of the $K\pi$ invariant mass distributions. Figure 6 shows the
$m(K^{\pm}\pi^{\mp})$(a) and $m(K_{S}^{0}\pi^{\pm})$ (b) invariant mass
distributions, where signals from the $K^{*}(892)^{0}$ and $K^{*}(892)^{\pm}$
are well seen. The final state with the $f_{1}(1285)$ resonance has no
$K^{*}(892)$ signal (blue histogram), while the sum of the signals from
$f_{1}(1420)$ and $f_{1}(1510)$, shown by green histogram, explains the
observed experimental distribution. Red histograms present the MC simulation
with the sum of the three resonances.

(a)(b)(c)

Figure 7:  The experimental $m(\pi^{+}\pi^{-})$ (a),
$m(K_{S}^{0}\pi^{+}\pi^{-})$ (b), and $m(K_{S}^{0}K^{\pm})$ (c) invariant mass
distributions in comparison with the simulation (red histogram). The blue and
green histograms are the MC simulation contributions from the $e^{+}e^{-}\to
f_{1}(1285)\rho$ and $e^{+}e^{-}\to f_{1}(1420,1510)\rho$ reactions,
respectively.

Figure 7 shows the background subtracted $m(\pi^{+}\pi^{-})$ (a),
$m(K_{S}^{0}\pi^{+}\pi^{-})$ (b), and $m(K_{S}^{0}K^{\pm})$ (c) invariant mass
distributions. The blue and green histograms show the MC simulated
contributions from the $e^{+}e^{-}\to f_{1}(1285)\rho$ reaction, and the sum
of $e^{+}e^{-}\to f_{1}(1420)\rho$ and $e^{+}e^{-}\to f_{1}(1510)\rho$
reactions, respectively. An enhancement at the large values of
$m(\pi^{+}\pi^{-})$ is due to a presence of the $\rho(770)$ from the
$e^{+}e^{-}\to f_{1}(1285)\rho$ reaction, while in the case of $f_{1}(1420)$
and $f_{1}(1510)$ the signal from $\rho(770)$ is suppressed kinematically at
our energies due to a large energy fraction taken by the $KK^{*}$ pair. Note,
that the $f_{1}(1285)$ resonance decays to $K\bar{K}\pi$ via the
$a_{0}(980)\pi$ intermediate state, while the $f_{1}(1510)$ and $f_{1}(1420)$
resonances both predominantly decay to the $KK^{*}(892)$ mode. A dominance of
the $f_{1}(1285)\to a_{0}(980)\pi\to K\bar{K}\pi$ intermediate reaction is
demonstrated in Fig. 7(c), where the $K_{S}^{0}K^{\pm}$ invariant masses are
concentrated at the threshold and well described by the simulation.

From the above study we conclude that the $e^{+}e^{-}\to
K_{S}^{0}K^{\pm}\pi^{\mp}\pi^{+}\pi^{-}$ process dominates by the
$e^{+}e^{-}\to f_{1}\rho(770)$ reaction with a combination of the
$f_{1}(1285)$, $f_{1}(1420)$, and $f_{1}(1510)$ resonances. A small
contribution from the other non-resonant processes is not excluded.

The above reactions have the same final state and an interference of them
could change the observed mass distributions. The influence of the
interference to the $m(K_{S}^{0}K^{\pm}\pi^{\mp})$ distribution is studied
with the MC simulation. Because of the relatively narrow width and the
difference in the intermediate state, the $f_{1}(1285)$ signal has practically
no influence from the interference with the $f_{1}(1420)$ and $f_{1}(1510)$
resonances, while the latter two are highly overlapped and the mass shape
depends on the relative phase between the amplitudes.

(a)(b)

Figure 8: (a) The background-subtracted experimental (dots) polar angle,
$\Theta$, distribution in comparison with the simulated distribution
(histograms, $f_{1}\rho(770)$ model) for the missing pion (a) and all detected
pions (b).

(a)(b)

Figure 9:  (a) The ratio of the number of five- to six-track events for the
data (dots) and simulation for the different intermediate states: PS model
(triangles), $f_{1}(1285)\rho$ (up-down triangles), $f_{1}(1420)\rho$ (open
circles), and $f_{1}(1510)\rho$ (open squares) intermediate states. (b) The
detection efficiency obtained from the MC simulation for the $e^{+}e^{-}\to
K_{S}^{0}K^{\pm}\pi^{\mp}\pi^{+}\pi^{-}$ reaction for the different
intermediate states (symbols legend is the same as for (a)).

##  4 Detection efficiency

Since DC acceptance in our experiment is about 70%, the detection efficiency
depends on the particle angular distribitions determined by the hadron
production dynamics. In addition, we have to take into account some minor
track reconstruction inefficiency.

To obtain the detection efficiency value, we simulate
$K_{S}^{0}K^{\pm}\pi^{\mp}\pi^{+}\pi^{-}$ production in the primary
generators, 50000 events for each c.m. energy interval for each model, trace
simulated particles through the CMD-3 detector using the GEANT4 [16] package,
and reconstruct them with the same software as experimental data. We calculate
the detection efficiency from the MC-simulated events as a ratio of events
after the selections described in Secs. 2, 3 to the total number of generated
events.

Our selection of the six- and the five-track signal events allows us to
estimate a difference in the tracking efficiency in the data and simulation,
and perform a test of the model used for the efficiency calculation. Figure 8
shows by dots the background-subtracted polar angle for a missing pion (a) and
for all detected charged tracks (b). Solid histograms represent the simulated
distribution for the $f_{1}\rho$ intermediate state, normalized to the number
of experimental events in Fig. 8(b). The events inside the DC acceptance, seen
in the 1.0–2.1 radians region of Fig. 8(a), are used to estimate the
difference in the track resonstruction efficiency for data and simulation (see
below).

The calculated ratio of the number of five- to six-track events at each c.m.
energy interval is shown in Fig. 9(a) by points for the data. The values of
the ratio for the PS model (triangles), and for the
$f_{1}(1285,1420,1510)\rho$ intermediate states (up-down triangles, open
squares, and open circles, respectively) are shown in Fig. 9(a) for a
comparison. The PS model is not compatible with our data.

We calculate the detection efficiency for a sum of the events with the six and
five detected tracks. Figure 9(b) shows the detection efficiency obtained for
the $e^{+}e^{-}\to K_{S}^{0}K^{\pm}\pi^{\mp}\pi^{+}\pi^{-}$ reaction for
different intermediate states: markers are the same as for Fig. 9(a).

(a)(b)

Figure 10:  (a) The $e^{+}e^{-}\to K_{S}^{0}K^{\pm}\pi^{\mp}\pi^{+}\pi^{-}$
cross section measured with the CMD-3 detector at VEPP-2000 (filled circles
for combined scans, filled squares for individual energy points). The cross
section for the $e^{+}e^{-}\to f_{1}(1285)\pi^{+}\pi^{-}\to
K_{S}^{0}K^{\pm}\pi^{\mp}\pi^{+}\pi^{-}$ reaction is shown by open circles.
(b) The $e^{+}e^{-}\to f_{1}(1285)\pi^{+}\pi^{-}$ cross section with the
correction for the missing decay modes (open circles) in comparison with BaBar
data [4] measured in the $f_{1}(1285)\to\eta\pi^{+}\pi^{-}$ mode.

##  5 Cross section calculation

In each energy interval the cross section is calculated as

$\sigma=\frac{N_{6tr}+N_{5tr}}{L\cdot\epsilon\cdot(1+\delta)},$

where $N_{6tr}$, $N_{5tr}$ are the background-subtracted numbers of the signal
events with six and five tracks, $L$ is the integrated luminosity for this
energy interval, $\epsilon$ is the detection efficiency, and $(1+\delta)$ is a
radiative correction calculated according to Ref. [18, 22]. To calculate the
radiative correction, we use a procedure with the iterations of the observed
cross section in the radiative integral, and obtain $(1+\delta)=0.85$ with a
weak energy dependence.

As shown above the observed events are the mixture of the different
intermediate states with slightly different detection efficiency. For the
inclusive $e^{+}e^{-}\to K_{S}^{0}K^{\pm}\pi^{\mp}\pi^{+}\pi^{-}$ cross
section we use an averaged value of the efficiency for the three observed
modes. We assign $\pm$5% systematic uncertainty to this procedure, which is
close to the difference between calculated efficiencies for the observed
reactions. We also apply some corrections due to the difference in the data
and MC simulation for the tracking efficiency (see Sec. 7). The cross section
is shown in Fig. 10(a) by filled circles for earlier runs, combined into the
eight energy intervals, and filled squares for the latest scans. There are no
other measurements. We also calculate the cross section by using only events
with the six detected tracks: a less than 5% difference within statistical
fluctuation is observed.

Energy interval, integrated luminosity, the number of six- and five-track
events, and the obtained cross section for each energy interval are listed in
Table 1.

Table 1: Energy interval, integrated luminosity, number of signal 6-track events, number of signal 5-track events, and the obtained cross section for the $e^{+}e^{-}\to K_{S}^{0}K^{\pm}\pi^{\mp}\pi^{+}\pi^{-}$ reaction. Only statistical uncertainties are shown.  Ec.m., MeV |  $L$, nb-1 | $N_{6tr}$ | $N_{5tr}$ |  $\sigma_{K_{S}^{0}K^{\pm}\pi^{\mp}\pi^{+}\pi^{-}}$, nb
---|---|---|---|---
2000–2007 | 4259 | 22.0 $\pm$ 4.7 | 34.0 $\pm$ 6.6 | 0.147 $\pm$ 0.021
1975–1980 | 4640 | 14.0 $\pm$ 4.0 | 20.0 $\pm$ 5.5 | 0.082 $\pm$ 0.016
1940–1962 | 9653 | 30.0 $\pm$ 5.7 | 37.0 $\pm$ 7.7 | 0.079 $\pm$ 0.011
1890–1925 | 15158 | 21.0 $\pm$ 4.8 | 24.0 $\pm$ 6.9 | 0.034 $\pm$ 0.006
1870–1884 | 19333 | 25.0 $\pm$ 5.0 | 42.0 $\pm$ 7.7 | 0.041 $\pm$ 0.006
1800–1860 | 11428 | 10.0 $\pm$ 3.2 | 14.0 $\pm$ 5.1 | 0.025 $\pm$ 0.006
1700–1780 | 12783 | 1.0 $\pm$ 1.7 | 7.0 $\pm$ 3.6 | 0.008 $\pm$ 0.004
1600–1680 | 13193 | 0.0 $\pm$ 0.0 | –1.0 $\pm$ 1.7 | –0.001 $\pm$ 0.002
2007 | 21643 | 107.0 $\pm$ 10.5 | 152.0 $\pm$ 14.2 | 0.133 $\pm$ 0.009
1980 | 10093 | 39.0 $\pm$ 6.6 | 48.0 $\pm$ 8.6 | 0.097 $\pm$ 0.012
1960 | 11065 | 27.0 $\pm$ 5.9 | 43.0 $\pm$ 7.9 | 0.072 $\pm$ 0.010
1940 | 14416 | 35.0 $\pm$ 5.9 | 38.0 $\pm$ 7.6 | 0.058 $\pm$ 0.008
1920 | 9904 | 14.0 $\pm$ 3.7 | 21.0 $\pm$ 6.6 | 0.041 $\pm$ 0.009
1900 | 9675 | 13.0 $\pm$ 3.6 | 19.0 $\pm$ 5.4 | 0.038 $\pm$ 0.008
1890 | 8912 | 8.0 $\pm$ 2.8 | 14.0 $\pm$ 4.9 | 0.029 $\pm$ 0.007
1870 | 9286 | 6.0 $\pm$ 2.4 | 21.0 $\pm$ 5.2 | 0.034 $\pm$ 0.007

##  6 Cross section of the $e^{+}e^{-}\to f_{1}(1285)\pi^{+}\pi^{-}$

The distributions shown in Fig. 5(c) can be used to extract the number of
events associated with the $e^{+}e^{-}\to f_{1}(1285)\pi^{+}\pi^{-}$ reaction.
After the combinatorial background subtraction, the $f_{1}(1285)$ signal is
well seen and has low remaining non-resonant background. Simulation shows that
a possible interference between $f_{1}(1285)$ and $f_{1}(1420)$–$f_{1}(1510)$
resonances is negligible because of the difference in the width and in the
decay modes. The numbers of events are extracted in the $\pm 30$ MeV/c2 mass
interval around $f_{1}(1285)$ peak of Fig. 5(c) with a background subtraction
from side bands. The numbers of events are small therefore we combine the
results into the six energy intervals. The results for the number of events in
the energy intervals and the integrated luminosity are shown in Table 2. The
contribution of the $e^{+}e^{-}\to f_{1}(1285)\pi^{+}\pi^{-}$ reaction to the
inclusive $e^{+}e^{-}\to K_{S}^{0}K^{\pm}\pi^{\mp}\pi^{+}\pi^{-}$ cross
section is shown in Fig. 10(a) and listed in Table 2.

We correct the number of events for the missing decay modes of the
$f_{1}(1285)$ resonance, containing $K_{L}^{0}$ and $\pi^{0}$ (factor of 3),
and using the branching fraction of $f_{1}(1285)\to K\bar{K}\pi$ from Ref.
[17] we obtain the cross section for the $e^{+}e^{-}\to
f_{1}(1285)\pi^{+}\pi^{-}$ reaction. The obtained cross section for the
$e^{+}e^{-}\to f_{1}(1285)\pi^{+}\pi^{-}$ reaction is shown in Fig. 10(b) by
open circles and listed in Table 2.

This cross section can be compared with the only available measurement by
BaBar [4], shown in Fig. 10(b), in which the $f_{1}(1285)$ resonance was
observed in the $\eta\pi^{+}\pi^{-}$ decay mode. Our measurement demonstrates
a faster rise of the cross section from the threshold.

Because of the relatively large widths, large uncertainty in the parameters,
and unknown influence of the interference we cannot extract separately the
events for the $f_{1}(1420)$ and $f_{1}(1510)$ resonances from our data.

Table 2: Energy interval, integrated luminosity, number of signal events, the obtained cross section for the $e^{+}e^{-}\to f_{1}(1285)\pi^{+}\pi^{-}\to K_{S}^{0}K^{\pm}\pi^{\mp}\pi^{+}\pi^{-}$ reaction, and for the $e^{+}e^{-}\to f_{1}(1285)\pi^{+}\pi^{-}$ process. Only statistical uncertainties are shown.  Ec.m., MeV |  $L$, nb-1 | $N_{f_{1}}$ |  $\sigma_{K_{S}^{0}K^{\pm}\pi^{\mp}\pi^{+}\pi^{-}}$, nb |  $\sigma_{f_{1}(1285)\pi^{+}\pi^{-}}$, nb
---|---|---|---|---
1700–1780 | 12783 | 2.0 $\pm$ 40.0 | 0.002 $\pm$ 0.003 | 0.07 $\pm$ 0.11
1800–1860 | 11428 | 10.0 $\pm$ 30.0 | 0.012 $\pm$ 0.007 | 0.38 $\pm$ 0.23
1860–1900 | 36903 | 44.0 $\pm$ 20.0 | 0.016 $\pm$ 0.004 | 0.52 $\pm$ 0.12
1900–1935 | 34738 | 58.0 $\pm$ 17.5 | 0.022 $\pm$ 0.004 | 0.72 $\pm$ 0.15
1935–1965 | 35134 | 64.0 $\pm$ 15.0 | 0.024 $\pm$ 0.005 | 0.79 $\pm$ 0.16
1980–2010 | 40675 | 113.0 $\pm$ 15.0 | 0.036 $\pm$ 0.006 | 1.20 $\pm$ 0.21

##  7 Systematic uncertainties

The following sources of systematic uncertainties and corrections are
considered for the cross section measurement.

  * 1.

The major uncertainty in the event number comes from the separation of the
kaons and pions using the dE/dX values. As shown in Fig. 1(c) signals from the
kaons and pions are highly overlapped, and the applied boundary rejects about
50% of the kaons with momentum above 300 MeV/c, corresponding to about 20% of
losses in the total number of the events. By changing the boundary we either
loosing the signal events or rapidly increase the pion leakage to the kaons.
We vary the boundary and only half of these losses is properly described by
the simulation. It corresponds to about $\pm$10% systematic uncertainty in the
result.

  * 2.

The tracking efficiency was studied in detail in our previous papers [19, 23].
A similar estimate is made using angular distribution in Figs. 8(a,b) where we
observe a difference for the data and MC simulation in the number of missing
tracks inside the DC acceptance. The correction for the track reconstruction
efficiency compared to the MC simulation is estimated as about (2.0$\pm$1.0)%
per track. Since we add events with one missing pion track, the MC-simulated
detection efficiency is corrected by (–7$\pm$5)%: the uncertainty is taken as
the corresponding systematic uncertainty.

  * 3.

The model dependence of the acceptance is determined by a comparing
efficiencies calculated for the different production dynamics. Since we assume
a mixture of the $f_{1}(1285,1420,1510)\rho$ intermediate states with the
approximately equal contributions, we average the efficiences shown in Fig. 9,
and assign a 5% uncertainty to the calculation.

  * 4.

Since only one charged track is sufficient for a trigger (98–99% single track
efficiency), and using a cross check with the independent neutral trigger, we
conclude that the trigger inefficiency for the multitrack events gives a
negligible contribution to the systematic uncertainty.

  * 5.

The systematic uncertainty due to the selection criteria is studied by varying
the requirements described above and doesn’t exceed 5%.

  * 6.

The uncertainty on the the integrated luminosity comes from the selection
criteria of the Bhabha events, the radiative corrections, the detector
calibrations, and does not exceed 1.5% [15].

  * 7.

The radiative correction uncertainty is estimated as about 2%, mainly due to
the uncertainty on the maximum allowed energy of the emitted photon, as well
as from the uncertainty on the cross section.

The above systematic uncertainties summed in quadrature give an overall
systematic uncertainty of about 15%.

##  Conclusion

The total cross section of the process $e^{+}e^{-}\to
K_{S}^{0}K^{\pm}\pi^{\mp}\pi^{+}\pi^{-}$ has been measured using 185.4 pb-1 of
integrated luminosity collected by the CMD-3 detector at the VEPP-2000
$e^{+}e^{-}$ collider in the 1.6–2.0 GeV c.m. energy range. The systematic
uncertainty is about 15%. At the present statistical accuracy we do not
observe any influence of the $N\bar{N}$ threshold to the cross section. From
our study we can conclude that the observed final state can be described by
the $e^{+}e^{-}\to f_{1}\pi^{+}\pi^{-}$ reaction with the contribution from
the $f_{1}(1285)$, $f_{1}(1420)$, and $f_{1}(1510)$ resonances. We extracted
the number of events associated with the $f_{1}(1285)$ resonance, and the
measured cross section for the $e^{+}e^{-}\to f_{1}(1285)\pi^{+}\pi^{-}$
reaction is compatible with the only available measurement by BaBar.

### Acknowledgments

We thank the VEPP-2000 team for the excellent machine operation. The work is
partially supported by the Russian Foundation for Basic Research grant
20-02-00496.

## References

  * [1] M. Davier, A. Hoecker, B. Malaescu, and Z. Zhang, Eur. Phys. J. C 77, 827 (2017).
  * [2] F. Jegerlehner, Acta Phys. Polon. B49, 1157 (2018).
  * [3] A. Keshavarzi, D. Nomura, T. Teubner, Phys. Rev. D 97, 114025 (2018).
  * [4] B. Aubert et al. (BaBar Collaboration), Phys. Rev. D 76, 092005 (2007).
  * [5] V. V. Danilov et al., Proceedings EPAC96, Barcelona, p.1593 (1996).
  * [6] I. A. Koop, Nucl. Phys. B (Proc. Suppl.) 181-182, 371 (2008).
  * [7] P. Yu. Shatunov et al., Phys.Part.Nucl.Lett. 13, 995 (2016).
  * [8] D. Shwartz et al., PoS ICHEP2016, 054 (2016).
  * [9] E. V. Abakumova et al., Phys. Rev. Lett. 110, 140402 (2013).
  * [10] E. V. Abakumova et al., JINST 10, T09001 (2015).
  * [11] B. I. Khazin, Nucl. Phys. B (Proc. Suppl.) 181-182, 376 (2008).
  * [12] F. Grancagnolo et al., Nucl. Instr. Meth. A623, 114 (2010).
  * [13] A. V. Anisyonkov et al., Nucl. Instr. Meth. A598, 266 (2009).
  * [14] D. Epifanov (CMD-3 Collaboration), J. Phys. Conf. Ser. 293, 012009 (2011).
  * [15] A. E. Ryzhenenkov et al., EPJ Web Conf. 212 (2019) 04011.
  * [16] S. Agostinelli et al. (GEANT4 Collaboration), Nucl. Instr. Meth. A506, 250 (2003).
  * [17] P. A. Zyla et al. (Particle Data Group), Prog. Theor. Exp. Phys. 2020, 083C01 (2020) and 2021 update.
  * [18] E. A. Kuraev and V. S. Fadin, Sov. J. Nucl. Phys. 41, 466 (1985).
  * [19] R. R. Akhmetshin et al. (CMD-3 Collaboration), Phys. Lett. B723, 82 (2013).
  * [20] R. R. Akhmetshin et al. (CMD-3 Collaboration), Phys. Lett. B804, 135380 (2020).
  * [21] D. Barberis et al. (WA102 Collaboration), Phys. Lett. B413, 225-231 (1997).
  * [22] S. Actis et al., Eur. Phys. J. C66, 585 (2010).
  * [23] R. R. Akhmetshin et al. (CMD-3 Collaboration), Phys. Lett. B768, 345 (2017).


# On the Influence of Masking Policies in Intermediate Pre-training

Qinyuan Ye1† Belinda Z. Li2‡ Sinong Wang3 Benjamin Bolte3
Hao Ma3 Wen-tau Yih3 Xiang Ren1 Madian Khabsa3
1University of Southern California 2MIT CSAIL 3Facebook AI
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Current NLP models are predominantly trained through a two-stage “pre-train
then fine-tune” pipeline. Prior work has shown that inserting an intermediate
pre-training stage, using heuristic masking policies for masked language
modeling (MLM), can significantly improve final performance. However, it is
still unclear (1) in what cases such intermediate pre-training is helpful, (2)
whether hand-crafted heuristic objectives are optimal for a given task, and
(3) whether a masking policy designed for one task is generalizable beyond
that task. In this paper, we perform a large-scale empirical study to
investigate the effect of various masking policies in intermediate pre-
training with nine selected tasks across three categories. Crucially, we
introduce methods to automate the discovery of optimal masking policies via
direct supervision or meta-learning. We conclude that the success of
intermediate pre-training is dependent on appropriate pre-train corpus,
selection of output format (i.e., masked spans or full sentence), and clear
understanding of the role that MLM plays for the downstream task. In addition,
we find our learned masking policies outperform the heuristic of masking named
entities on TriviaQA, and policies learned from one task can positively
transfer to other tasks in certain cases, inviting future research in this
direction.

##  1 Introduction

Large, neural language models (LMs) pre-trained with masked language modeling
Devlin et al. (2019); Raffel et al. (2020) have achieved impressive results
over a variety of NLP tasks.†††Work partially done while interning at Facebook
AI. Studies show that an additional intermediate pre-training stage between
general pre-training and task-specific fine-tuning further improves downstream
performance (Fig. 1).††‡Work partially done while working at Facebook AI. For
example, intermediate pre-training by masking and recovering named entities or
dates, known as salient span masking (SSM, Guu et al. 2020), significantly
improves a model’s performance of answering factoid questions in a closed-book
setting Roberts et al. (2020). However, there is a lack of systematic study on
how intermediate pre-training works, whether heuristic masking policies like
SSM are near-optimal, or whether they generalize to different NLP tasks.
Additionally, it is unclear that for tasks other than closed-book QA, whether
intermediate pre-training is helpful, or what masking strategy should be
adopted.

Figure 1: Analysis Setup. We investigate the influence brought by different
masking policies during intermediate pre-training, a stage between general
pre-training and task-specific fine-tuning. We apply three types of policies
(heuristic, supervised, meta-learned) on three categories of tasks (closed-
book QA, knowledge-intensive language tasks, multiple-choice QA).

In this paper, we offer a large-scale, systematic study on the effects and
transferability of masking strategies during intermediate pre-training, while
we carefully control all other aspects (§3). We first begin our analysis with
a focus on three heuristic masking policies (§4.1). We fine-tune the models
resulting from intermediate pre-training on nine selected tasks covering three
categories (closed-book QA, knowledge-intensive language tasks, and multi-
choice QA). Our results suggest that successful intermediate pre-training is
dependent on the selection of appropriate corpus. Moreover, heuristic-based
approaches are effective only when we have a precise understanding of the role
masked language modeling (MLM) plays in downstream task. For example, MLM
serves as a sort of memorization step Petroni et al. (2019), whereby learning
to unmask spans in context is analogous to memorizing facts about the span. In
the absence of such understanding, heuristic policies may be sub-optimal.

This motivates us to explore whether automating the discovery of optimal
masking policies is possible. We design methods to learn a masking policy with
supervised learning (§4.2) or meta-learning (§4.3), and compare downstream
task performance using the same protocol in our previous analysis. Notably, we
observe that masking policies learned with supervised learning and meta-
learning outperforms the SSM policy for TriviaQA Joshi et al. (2017), and
these policies learned from TriviaQA also help improve performance on Web
Questions Berant et al. (2013). We also discuss the pros and cons of learned
masking policies, such as downstream task learning efficiency, risks of over-
fitting and learning instability.

Finally, in hopes to better understand the heuristic and learned masking
policies, we provide quantitative analysis on the masks produced by these
policies. We visualize the distribution of part-of-speech tags among masked
tokens, and their relation to token frequency in the corpus (§5.3). We find
that the masking policies learned from TriviaQA tend to mask more proper nouns
and tend to mask less frequent words when compared to SSM.

Overall, our empirical analysis provides useful suggestions for NLP
researchers who aim to improve downstream task performance using intermediate
pre-training and heuristic masking strategies. In addition, our experiments
reveal that infusing task-specific knowledge into LMs with learned masking
policies is a promising way to improve downstream task performance, and invite
future research in this direction.

##  2 Preliminary: Masked Language Modeling

In this section, we revisit MLM objective with the notation that we will use
throughout the paper. MLM is a predominant pre-training objective for large-
scale transformers in NLP Devlin et al. (2019). MLM and its variants can be
characterized with two key components: a masking policy $g(.;\phi)$,
parameterized by $\phi$, which decides the collection of tokens to be masked,
and a language model $f(.;\theta)$, parameterized by $\theta$.

Formally, given a sequence of tokens $\mathbf{x}=[x_{1},x_{2},...,x_{m}]$,
$g(\mathbf{x};\phi)$ generates a sequence of binary decisions
$\mathbf{d}=[d_{1},d_{2},...,d_{m}]$, where $d_{i}=1$ indicates the token
$x_{i}$ will be masked. The source sequence for pre-training,
$\mathbf{x}^{(\text{src})}$, is formulated by replacing the selected tokens
with a special <mask> token, i.e.,
$\mathbf{x}^{(\text{src})}=[x^{(\text{src})}_{1},x^{(\text{src})}_{2},...,x^{(\text{src})}_{m}]$,
where $x^{(\text{src})}_{i}=x_{i}$ if $d_{i}=0$ and
$x^{(\text{src})}_{i}=\texttt{<mask>}$ if $d_{i}=1$. We denote this operation
as $\mathbf{x}^{(\text{src})}=\mathbf{x}\oplus\mathbf{d}$. The target sequence
$\mathbf{x}^{(\text{tar})}$ can be either the full original sequence
$\mathbf{x}$ (BART, Lewis et al. 2020), or the sequence of masked tokens (T5,
Raffel et al. 2020).

##  3 Analysis Setup

In this section we introduce the analysis pipeline (§3.1) and downstream
datasets we use (§3.2). We defer the details of learned masking policies to
§4.

###  3.1 Experiment Procedure

Our goal is to analyze the influence in downstream task performance brought by
different masking policies $g(.;\phi)$ during intermediate pre-training.
Towards this goal, we ensure that the only variable is the masking policy,
while all other aspects are controlled, so that the downstream performance
reveal the influence we aim to study. We first initialize with a BART-base
model Lewis et al. (2020); then for each masking policy, we conduct
experiments following a two-stage pipeline:

#### Stage 1. Intermediate Pre-training.

We perform intermediate pre-training with a given masking policy $g(.;\phi)$.
All intermediate pre-training is done with input sequence length of 128, batch
size of 2048, learning rate of $0.0001$, up to a total number of $100,000$
updates, using Wikipedia snapshot from December 20, 2018111The snapshot is
available at https://archive.org/download/enwiki-20181220. Wikipedia is
licensed under CC BY-SA 3.0..

#### Stage 2. Task-specific Fine-tuning.

We fine-tune each resulting checkpoint from Stage 1 on related downstream
tasks, and evaluate their performance. We follow the same routine of
hyperparameter search for each checkpoint. We then run the fine-tuning
experiments with the best hyperparameter setting and three different random
seeds. See Appendix B for details.

###  3.2 Downstream Tasks and Datasets

We focus our study on nine downstream tasks across three categories. We
introduce their details and explain the rationale behind our selection in the
following.

#### Closed-book QA.

Closed-book QA is a task that requires a language model to directly answer
questions without access to external knowledge Roberts et al. (2020). This
paradigm assumes that the model memorizes large amounts of knowledge from its
pre-training data, which gets “packed” into its parameters, and can
subsequently be “retrieved” to answer questions. Notably, Roberts et al.
(2020) reported 9%+ improvement in exact match on TriviaQA when intermediate
pre-training with salient span masking (i.e., masking and recovering named
entities or dates) is performed on a T5-11B model. This observation inspired
our work. Our study considers three datasets for closed-book QA: Natural
Questions (NQ, Kwiatkowski et al. 2019), WebQuestions (WQ) and TriviaQA (TQA).

#### Knowledge-Intensive Tasks from KILT.

Extending from closed-book QA, we select three tasks from the KILT benchmark
Petroni et al. (2020) that also aims to test a model’s implicit knowledge
capacity, while having different task formats and goals. Aidayago2 (AY2,
Hoffart et al. 2011) is an entity linking task that requires the model to
assign a Wikipedia page to an entity mention in the text. The output is the
unique name of the Wikipedia page in text format. Zero-shot relation
extraction (ZSRE, Levy et al. 2017) is a slot filling task that aims to
predict the object when given the subject and the relation. The relations in
the train/dev/test splits are non-overlapping. Wizard of Wikipedia (WoW, Dinan
et al. 2019) is a dataset of dialogue histories relevant to knowledge in
Wikipedia. The model is required to act like a chatbot and generate the
response given previous dialogue history.

#### Knowledge-Intensive Multiple-choice QA.

We select three multiple-choice QA datasets, in which the questions can be
answered with commonsense/background knowledge without any context, but the
dataset provides additional context paragraphs to explicitly state the
background knowledge used. We use WIQA Tandon et al. (2019) which focuses on
procedural text, QuaRTz Tafjord et al. (2019) which focuses on qualitative
relationship, and ROPES Lin et al. (2019) which focuses on causes and effects.
We reformat these tasks into sequence-to-sequence format, following UnifiedQA
Khashabi et al. (2020).

To summarize, all tasks above can be treated as sequence-to-sequence tasks,
where each example is a source-target pair $(\mathbf{s},\mathbf{t})$,
accompanied with a context paragraph $\mathbf{c}$ provided by the dataset.
Details for dataset splitting are in Appendix D.1.

##  4 Compared Masking Policies

We experiment with three categories of masking policies: heuristic policies,
where $g$ is a fixed heuristic function (§4.1); supervised policies, where $g$
is a model whose weights are learned from direct supervision on downstream
tasks (§4.2); and meta-learned policies, where $g$ is a model whose weights
are learned through meta-learning on downstream tasks (§4.3).

###  4.1 Heuristic Policy

We experiment with the following three heuristic masking policies: (1) BART’s
original denoising objective (+Orig); (2) Masking and recovering 15% randomly
selected tokens (+Rand)22215% is borrowed from BERT and T5. “+Orig” and
“+Rand” are different in that $\mathbf{x}^{\text{(tar)}}=\mathbf{x}$ for
“+Orig”, while $\mathbf{x}^{\text{(tar)}}$ contains the masked 15% tokens for
“+Rand”.; (3) Salient span masking, i.e., masking and recovering one named
entity Roberts et al. (2020); Guu et al. (2020) (+SSM).

###  4.2 Supervised Policy

When students prepare for closed-book exams, they are likely to review and
memorize what they perceive as most important in the text book. Such
perception is learned from their prior experience of taking closed-book exams.
Following this intuition, Ye et al. (2020) proposed to learn a masking policy
for closed-book QA tasks to help the model focus on likely answers during
intermediate pre-training. The masking policy is trained with (answer,
context) examples, and the policy is an extractive model that extracts the
answer span from the context. For example, if the context $\mathbf{x}$ is
[Charles, Schulz, was, the, creator, of, Snoopy] and the answer is “Charles
Schulz”, the label for the answer start index will be [1,0,0,0,0,0,0]; for end
index it will be [0,1,0,0,0,0,0]. In the following, we briefly recap the
method with our notations.

#### Model.

Given context paragraph tokens $\mathbf{x}=[x_{1},x_{2},...,x_{m}]$, we first
use an embedding matrix $\mathbf{E}$ to embed each token:
$[\mathbf{e}_{1},\mathbf{e}_{2},...,\mathbf{e}_{n}]$. Then, we use a 2-layer
bi-directional LSTM model to compute the hidden representation at each
position.333Though the masking policy can theoretically take any form, we opt
for a lightweight architecture (2-layer Bi-LSTM) as we need to apply it to
millions of pre-training instances.

$[\mathbf{h}_{1},\mathbf{h}_{2},...,\mathbf{h}_{n}]=\textrm{Bi-
LSTM}([\mathbf{e}_{1},\mathbf{e}_{2},...,\mathbf{e}_{n}])$ (1)

Finally, we use two learned vectors $(\mathbf{w}_{st},b_{st})$ and
$(\mathbf{w}_{ed},b_{ed})$ to compute the logits for each position being the
start or end position of the potential answer/target span. For example, the
logit of position $j$ being a start/end position is computed as follows.

$\displaystyle y_{j,st}=\mathbf{w}_{st}\mathbf{h}_{j}+b_{st},\ \
y_{j,ed}=\mathbf{w}_{ed}\mathbf{h}_{j}+b_{ed}.$ (2)

#### Policy Inference.

When deploying the policy to intermediate pre-training, we select the
potential answer spans by ranking the sum of start and end logits of each
potential spans, in accordance to the inference step in machine reading
comprehension models. That is, we rank the spans $(i,j)$ according to
$y_{i,st}+y_{j,ed}$. We consider two variants when deploying the policy: (a)
masking the top 1 span or (b) sampling 1 span from the top 5 spans.

#### Applicability and Limitation.

Supervised policy is designed for closed-book QA, and one limitation of this
method is that the target span $\mathbf{t}$ must appear as is in the context
paragraph $\mathbf{c}$. Within all other knowledge intensive tasks, only ZSRE
satisfies this constraint. To sum up, we apply supervised policy method to
TQA, NQ, ZSRE.

###  4.3 Meta-learned Policy

Conceptually, what the learned masking policy captures is closely related to
the concept of “learning to learn” Schmidhuber (1987); Thrun and Pratt (1998).
At a high level, the masking policy should provide the model with the desired
initialization for the downstream task, such that the model can better learn
the downstream task in only a few fine-tuning updates. Therefore, we construct
a meta-learning approach, which we describe below.

#### Overview.

We formulate each $(\mathbf{c},\mathbf{s},\mathbf{t})$ example as a small
“task”. For each task, the goal is to improve the performance of generating
target sequence $\mathbf{t}$ given input $\mathbf{s}$, immediately after
learning from the context $\mathbf{c}$. This is similar to taking quizzes,
where a student first learns from a passage $\mathbf{c}$ and then is
immediately tested on it by trying to answer $\mathbf{t}$ given $\mathbf{s}$.
Studying from $\mathbf{c}$ strategically with an optimal masking policy will
result in better performance (i.e., smaller loss in generating $\mathbf{t}$).

Following work in gradient-based meta-learning Finn et al. (2017);
Grefenstette et al. (2019), we set up an inner and outer loop. We briefly
sketch the procedure in Fig. 2. In the inner loop, we focus on the current
$(\mathbf{c},\mathbf{s},\mathbf{t})$ examples by applying the current masking
policy $g(.;\phi)$ and performing pre-train/fine-tune updates to
$f(.;\theta)$. In the outer loop, we update the policy $g(.;\phi)$ with the
signal at the end of inner loop training. We denote $\phi^{(p)}$ as the
masking policy parameters after $p$ outer loop optimization steps, and
$\theta^{(p,q)}$ as the LM parameters after $p$ outer loop optimization steps
and $q$ inner loop optimization steps.

Figure 2: Update masking policy by learning from one context-query-answer
example. (1) Inner Loop: (a) A context paragraph $c$ is first masked with
current policy $g(.;\phi)$, and the language model is trained to recover
masked tokens for one step; (b) the language model is trained on $(q,a)$ pair
for one step. (2) Outer Loop: We use the validation loss on the same $(q,a)$
pair to update the masking policy, by directly taking the gradient of loss $L$
w.r.t policy parameters $\phi$.

#### Inner Loop.

In one inner-loop curriculum, we first take the context as a pre-training
sentence, i.e., $\mathbf{x}=\mathbf{c}$, and use the current masking policy
$g(.;\phi^{(p)})$ to determine the masks $\mathbf{d}$ and the implied
perturbed input $\mathbf{x}^{(\text{src})}$, i.e.,
$\mathbf{d}=g(\mathbf{x};\phi^{(p)})$,
$\mathbf{x}^{(\text{src})}=\mathbf{x}\oplus\mathbf{d}$.

We pre-train $\theta^{(p,0)}$ for one step to recover $\mathbf{x}$ from
$\mathbf{x}^{(\text{src})}$:

$\theta^{(p,1)}=\theta^{(p,0)}-\alpha_{0}\nabla_{\theta^{(p,0)}}\mathcal{L}(f(\mathbf{x}^{(\text{src})};\theta^{(p,0)}),\mathbf{x}),$
(3)

where $\alpha_{0}$ is the learning rate, and $\mathcal{L(.,.)}$ is the cross
entropy loss for recovering $\mathbf{x}$ using disturbed input
$\mathbf{x}^{(\text{src})}$ and model parameters $\theta^{(p,0)}$.

Next, we take $\theta^{(p,1)}$ as initialization and fine-tune it for one step
on the downstream objective of predicting $\mathbf{t}$ given $\mathbf{s}$:

$\theta^{(p,2)}=\theta^{(p,1)}-\alpha_{1}\nabla_{\theta^{(p,1)}}\mathcal{L}(f(\mathbf{s};\theta^{(p,1)}),\mathbf{t}).$
(4)

#### Outer Loop.

In outer loop, we update the masking policy $g(.;\phi^{(p)})$. We aim to
answer query $\mathbf{s}$ correctly after the inner-loop curriculum. We define
the meta-loss $\mathcal{L}^{\prime}$ as the decrease in losses after the one
fine-tuning update, i.e.,

$\mathcal{L}^{\prime}=\mathcal{L}(f(\mathbf{s};\theta^{(p,2)}),\mathbf{t})-\mathcal{L}(f(\mathbf{s};\theta^{(p,1)}),\mathbf{t}).$
(5)

$\mathcal{L}^{\prime}$ characterizes how fast the model has adapted itself to
answer ($\mathbf{s}$, $\mathbf{t}$) within one step of optimization. Since all
computations in Eq. (3-5) are continuous444We use Gumbel Softmax Jang et al.
(2017) to discretize the output of $g(.;\phi^{(p)})$ and formulate masking
decision $\mathbf{d}$, and we use embedding mixture for
$\mathbf{x}^{(\text{src})}=\mathbf{x}\oplus\mathbf{d}$, we optimize $\phi$ by
directly taking gradients from $\mathcal{L}^{\prime}$,

$\phi^{(p+1)}=\phi^{(p)}-\alpha_{2}\nabla_{\phi^{(p)}}\mathcal{L}^{\prime}.$
(6)

#### Controlling Masking Budget.

Higher order optimization is known to be unstable Antoniou et al. (2019). In
early stages of the study, we found the policy to be flipping between masking
none or all of the tokens. To stabilize, we add a softened L2 loss to control
the portion of mask/not-mask decisions output by $g(.;\phi)$. Denoting
$l(\mathbf{x})$ as the input sequence length, $l(\mathbf{d})$ as the number of
mask decisions; we define a budget $\gamma$ and a tolerance factor $\epsilon$,
and compute the regularization term $\mathcal{L}_{reg}$,

$\mathcal{L}_{reg}(\mathbf{x},\mathbf{d})=\left\\{\begin{aligned}
&0,\quad|\gamma l(\mathbf{x})-l(\mathbf{d})|\leq\epsilon l(\mathbf{x})\\\
&(\gamma
l(\mathbf{x})-l(\mathbf{d}))^{2},\quad\text{otherwise}\end{aligned}\right.$
(7)

For example, when $\gamma=15\%$, $\epsilon=5\%$ and the input sequence
$\mathbf{x}$ contains 100 tokens, the policy will not be penalized if it’s
masking $15\pm 5$ of all tokens in the sequence. We modify the optimization
step in Eq. (6) as follows, where $\beta$ is a co-efficient balancing the
regularization intensity.

$\phi^{(p+1)}=\phi^{(p)}-\alpha_{2}\nabla_{\phi^{(p)}}(\mathcal{L}^{\prime}+\beta\mathcal{L}_{reg}(\mathbf{x},\mathbf{d}))\\\
$ (8)

#### Post-processing.

When we deploy a learned policy to pre-training, we are no longer constrained
by differentiability. Based on useful techniques in previous work, we apply
post-processing to predicted masking decisions $\mathbf{d}$. (1) Whole-word
masking and text infilling Liu et al. (2019); Lewis et al. (2020): whenever
one subword $x_{i}$ within a whole word is masked ($d_{i}=1$), we expand the
mask and always mask the whole word. When consecutive tokens are masked, we
replace the sequence of <mask> in the input sequence with exactly one <mask>
token. (2) Additional budget control: Even with our budget regularization loss
(Eq. 7), we find some input sequences get too many masks ($>50\%$). This
creates extremely challenging pre-train examples that may prevent the model
from learning useful information. For these sentences we randomly “unmask”
tokens to keep the portion of masks below $30\%$.

For space concerns, we leave pseudo-code and other implementation details in
Appendix A.2.

##  5 Results and Discussion

Following our analysis setup (§3), we present the results for closed-book QA
in Table 1, knowledge-intensive language tasks (KILT) in Table 2 and multiple-
choice QA in Table 3. In the following, we aim to understand the influence
brought by different masking policies through these results. We also introduce
several ad-hoc experiments to verify our hypotheses raised in our analysis.

###  5.1 Comparison of Heuristic Policies

Continue pre-training with the original objective is helpful in general. Prior
work has shown that intermediate pre-training on encoder models (i.e.,
RoBERTa, Liu et al. 2019) with in-domain corpora helps to improve downstream
classification tasks performance Gururangan et al. (2020). Our experiments
help to examine whether similar conclusion holds for text-to-text models and
tasks beyond classification. From our results, we found intermediate pre-
training with Wikipedia and BART’s original objective (+Orig) improves
performance of two closed-book QA tasks (TQA and WQ), one entity linking task
(AY2), and two multiple-choice QA tasks (WIQA and QuaRTz); maintains
performance on NQ and ZSRE; leads to worse performance on ROPES. Overall,
intermediate pre-training leads to improved performance; this may be due to
the common observation that language models tend to improve with further pre-
training even after validation perplexity have plateaued, or that Wikipedia as
a general knowledge-intensive corpus, is more closely related to our
downstream tasks, compared to the mixture of corpus555Similar to RoBERTa, BART
uses the combination of BookCorpus Zhu et al. (2015), CC-News Nagel (2016),
OpenWebText Gokaslan and Cohen (2019), and Stories Trinh and Le (2018) as pre-
training corpus. used to pre-train BART.

A closer look at ROPES, the exception. We notice that the context paragraphs
in ROPES are from science textbooks and Wikipedia. We hypothesize that
intermediate pre-training on only Wikipedia may cause catastrophic forgetting
of some scientific knowledge obtained during BART’s general pre-training. To
verify this, we randomly mask 15% tokens in ROPES context paragraphs and
computed MLM loss. The BART-Base checkpoint achieves 1.97 in NLL Loss, while
+Orig achieves 2.02. This supports our hypothesis that intermediate pre-
training on a smaller corpus (e.g., Wikipedia) may make the model forget
knowledge in general pre-training (e.g., scientific textbooks). We conclude
that it is important to pay attention to the corpus from which the downstream
dataset is created.

| TQA | WQ | NQ
---|---|---|---
BART-Base | $21.82_{\pm.15}$ | $26.23_{\pm.05}$ | $23.72_{\pm.25}$
+Orig | $22.91_{\pm.16}$ | $27.17_{\pm.56}$ | $23.85_{\pm.37}$
+Rand | $22.93_{\pm.14}$ | $27.25_{\pm.68}$ | $24.64_{\pm.44}$
+SSM | $23.62_{\pm.29}$ | $28.17_{\pm.04}$ | $24.80_{\pm.06}$
+Supervised-NQ(Top1) | $23.48_{\pm.10}$ | $27.43_{\pm.38}$ | $24.58_{\pm.10}$
+Supervised-NQ(Top5) | $23.73_{\pm.21}$ | $28.15_{\pm.05}$ | $24.86_{\pm.28}$
+Supervised-TQA(Top1) | $24.71_{\pm.21}$ | $27.84_{\pm.03}$ | $24.58_{\pm.19}$
+Supervised-TQA(Top5) | $24.43_{\pm.09}$ | $28.35_{\pm.73}$ | $24.66_{\pm.22}$
+Meta-learned-NQ | $23.50_{\pm.28}$ | $27.07_{\pm.20}$ | $24.83_{\pm.18}$
+Meta-learned-TQA | $23.88_{\pm.04}$ | $27.49_{\pm.17}$ | $24.85_{\pm.21}$
BART-Large | $24.28_{\pm.51}$ | $28.82_{\pm.33}$ | $24.72_{\pm.16}$
+Orig | $24.34_{\pm.35}$ | $28.28_{\pm.35}$ | $24.91_{\pm.68}$
+SSM | $26.29_{\pm.43}$ | $29.79_{\pm.47}$ | $25.34_{\pm.23}$
+Supervised-TQA(Top1) | $27.18_{\pm.34}$ | $29.71_{\pm.74}$ | $24.28_{\pm.28}$

Table 1: Performance of Closed-book QA Tasks. We report average and standard
deviation of exact match over three runs with different random seeds. Dark
blue highlights the best performing model. Light blue highlights models that
are not significantly worse than the best performing model ($p$>0.1 in paired
t-test).

| AY2 | ZSRE | WoW
---|---|---|---
Metric | EM | EM | F1
BART-Base | $81.07_{\pm.15}$ | $1.89_{\pm.15}$ | $15.14_{\pm.22}$
+Orig | $81.38_{\pm.06}$ | $1.67_{\pm.15}$ | $15.20_{\pm.13}$
+Rand | $81.67_{\pm.13}$ | $2.29_{\pm.19}$ | $14.69_{\pm.21}$
+SSM | $81.74_{\pm.19}$ | $3.52_{\pm.03}$ | $14.68_{\pm.16}$
+Supervised-ZSRE(Top1) | $81.57_{\pm.03}$ | $2.84_{\pm.15}$ | $14.58_{\pm.01}$
+Supervised-ZSRE(Top5) | $81.90_{\pm.22}$ | $2.90_{\pm.03}$ | $14.50_{\pm.38}$
+Meta-learned-ZSRE | $81.31_{\pm.22}$ | $1.99_{\pm.21}$ | $15.07_{\pm.09}$
+Meta-learned-WoW | $80.90_{\pm.23}$ | $1.64_{\pm.05}$ | $15.32_{\pm.05}$

Table 2: Performance of KILT Tasks.

| ROPES | WIQA | QuaRTz
---|---|---|---
BART-Base | $46.60_{\pm 0.48}$ | $71.18_{\pm 1.12}$ | $62.80_{\pm 1.16}$
+Orig | $43.68_{\pm 0.67}$ | $73.06_{\pm 0.72}$ | $63.35_{\pm 0.52}$
+Rand | $44.59_{\pm 1.15}$ | $70.55_{\pm 0.42}$ | $63.31_{\pm 1.74}$
+SSM | $50.51_{\pm 1.15}$ | $69.31_{\pm 0.77}$ | $64.41_{\pm 1.04}$
+Meta-learned-ROPES | $53.71_{\pm 2.33}$ | $73.05_{\pm 0.98}$ | $62.93_{\pm 1.28}$
+Meta-learned-WIQA | $48.30_{\pm 0.69}$ | $72.38_{\pm 0.37}$ | $63.14_{\pm 1.26}$
+Meta-learned-QuaRTz | $49.01_{\pm 1.92}$ | $72.65_{\pm 0.53}$ | $63.69_{\pm 0.48}$

Table 3: Performance of Multiple-choice QA Tasks. We report accuracy for each
task.

Select the heuristic masking policy that resemble the downstream task most. So
far, we limit the focus to the +Orig objective. Now we further add +Rand and
+SSM into the comparison. From the results in Table 1, we first confirm that
salient span masking (SSM) is indeed very beneficial for closed-book QA
Roberts et al. (2020). In addition, SSM helps improve performance for two
entity-centric knowledge intensive tasks (AY2 and ZSRE, see Table 2) and two
multiple-choice QA tasks (ROPES and QuaRTz, see Table 3). Note that ROPES
focus on causal relationships between entities and QuaRTz focus on qualitative
relations (involving numbers); both can be considered entity-centric. We
conclude that using heuristic masking policies that resemble the downstream
tasks, or masking information known to be important for the downstream task,
tend to improve downstream performance. When it’s difficult to design a
heuristic that satisfy these needs, using random masking may be helpful. In
this case, we recommend to decide whether to generate full sequence (+Orig) or
only masked tokens (+Rand) based on the task output length. If the downstream
tasks requires generating long sentences, generating full sequence is more
helpful. This is supported by the observation that +Orig is better than +Rand
for WoW. On the other hand, if the target sequences in the downstream dataset
are shorter, generating masked tokens is more helpful, as shown by experiments
on NQ, AY2, ZSRE and ROPES.

###  5.2 How Do Learned Policies Perform?

We have introduced two ways to automate the discovery of better masking
policies, with supervised learning (§4.2) and meta-learning (§4.3). We now
extend our analysis to these learned policies.

Successful Cases. We observe that learned policies are most successful on
TriviaQA, with both the supervised policy and the meta-learned policy
outperforming SSM. We attribute its success to the following reasons: (1)
(context, source, target) examples are abundant, so the masking policy has
sufficient supervision. TriviaQA dataset is accompanied with large-scale
context paragraphs created with distant supervision, so the scale of
$(\mathbf{c},\mathbf{s},\mathbf{t})$ examples is larger than other datasets.
(2) The heuristic masking policy does not “perfectly” resemble the downstream
task, and it still has room for improvement. SSM masks one random named entity
in the context. However, the answer to trivia questions are not necessarily
named entities, and one named entity may be more important than another.
Therefore the learned policies can better capture the characteristics of
TriviaQA than SSM. Apart from TriviaQA, meta-learned policies outperforms
+Orig on NQ, ZSRE and ROPES, demonstrating the effectiveness of the method.
This also opens up a promising direction for downstream tasks whose heuristic
masking policy is not intuitive (e.g., dialogue response generation, multiple-
choice QA).

Improved learning efficiency. We additionally consider a low-resource setting
for TriviaQA, where we use 0.1% and 1% of its training set for fine-tuning. We
present the results in Table 4. We observe that the supervised policy has
better sample efficiency than SSM. We also observe that intermediate pre-
training by generating full sequence (a/d) is worse than generating spans
(b/c), supporting our previous conclusion that the choice of target sequences
should be based on the downstream task output format (span or sentence).

Training Data Used | 0.1% | 1%
---|---|---
(a) BART-Base | $3.69\%$ | $5.54\%$
(b) +SSM | $5.56\%$ | $7.31\%$
(c) +Supervised-TQA(Top1) | $6.49\%$ | $8.40\%$
(b) +Meta-learned-TQA | $4.50\%$ | $6.44\%$

Table 4: Performance of TriviaQA in low-resource settings. Exact match is
reported. Supervised policy outperforms other masking policies in low-resource
setting, consistent with the full-dataset setting.

Overfitting on ZSRE. ZSRE dataset has a unique setting: it is a slot filling
task similar to close-book QA; however it adds additional challenge as the
relations in train/dev/test splits are non-overlapping. We hypothesize that
this train/test discrepancy leads to unsatisfactory behavior of learned ZSRE
policies, and we conduct a set of controlled experiment to validate this
hypothesis. Concretely, we use 90% of its original train set as the new train
set, use the 10% remaining training examples as a “matched” dev set, and the
original dev set as a “mismatched” dev set. In our experiments, SSM achieves
$20.02\%_{\pm.16\%}$ EM on match-dev, and $3.21\%_{\pm.15\%}$ on mismatch-dev.
Supervised-ZSRE(Top5) achieves $20.37\%_{\pm.04\%}$ on match-dev (outperforms
SSM, p$<$0.05), and $2.94\%_{\pm.11\%}$ on mismatch-dev. These experiments
show that our supervised policy is learning useful information, but has
overfitted to the training data and becomes less robust to distribution shift
during inference. In comparison, SSM is agnostic to train-test discrepancy and
thus achieves the strongest performance.

Generalization of learned policies. We observe several cases where a policy
learned from one dataset positively transfer to another downstream tasks. That
includes Supervised-TQA(Top5) bringing improvement to WQ, +Supervised-NQ(Top5)
bringing improvement to TQA, and +Supervised-ZSRE(Top5) bringing improvement
to AY2, compared to random masking baselines. This is reasonable since all
these tasks are entity-centric and are similar in nature. For tasks with
significantly different formats and goals, e.g., ZSRE and WoW, policies
learned on one does not benefit the other. Here we only exhibit the evidence
supporting that learned masking policies can positively transfer, and we leave
the question of “when and why does it work” as future work.

Remarks. Supervised/meta-learned masking policies are our initial attempt
towards the idea of “learning to mask”. While being successful and exhibiting
the evidence for positive transfer in certain cases, we recognize the
potential risks of over-fitting, or suffering from high instability in meta-
learning. We hope future work can investigate these issues and design novel
methods to learn better masking policies.

###  5.3 Quantitative Analysis: What are Masked? How are policies different?

Figure 3: Part-of-speech tag distribution for masked produced by different
masking policies.

In this section we aim to understand how masking policies are different from
each other in terms of their masking decisions. We analyze the relation
between masking decisions, part-of-speech tags and token frequency.
Specifically, we take 1% of the pre-train corpus and compare the masking
decisions made by each policy to facilitate our analysis.

#### Relation to Part-of-speech Tags.

In Fig. 3, we plot the stacked bar chart of part-of-speech tags to visualize
their distribution. Each bar represent the portion of masks having the part-
of-speech tag, amongst all masks produced by this policy. Notably, most
supervised policies learns to focus more on proper nouns, and less on common
nouns. This is consistent with the goal of the entity-centric downstream
tasks. Comparing Supervised-TQA and SSM, Supervised-TQA focuses less on nouns,
numbers and adjectives, and it focuses even more on proper nouns. This
suggests that Supervised-TQA better characterizes the property of TQA, and
thus outperforms SSM by learning to mask task-specific information. Due to the
differences in learning procedures, the meta-learned policies has
distributions different from supervised policies. Still, meta-learned policies
for NQ and TQA masks more proper nouns compared to random masking, similar to
their supervised counterparts.

#### Relation to Token Frequency.

In Fig. 4, we plot the relation between mask frequency and token frequency for
masking policies learned from TQA, along with random masking and SSM for
reference. Mask frequency is computed as the number of occurrences that a
token was masked divided by the number of all masked tokens. For random
masking, the datapoints approximate a Zipfian distribution Zipf (1999), with
some noise due to random sampling of words. Secondly, for SSM, most datapoints
fall on a curve above the random masking line, while a small portion of tokens
are less likely to be masked, formulating line segments in the bottom area.
These observations indicate that SSM tend to mask less frequent tokens, but
its behavior is not fully explained away with token frequency. The two learned
policies, Supervised-TQA (Fig. 4(a)) and Meta-TQA (Fig. 4(b)) are in general
similar to SSM, while the curve for Supervised-TQA is more scattered,
indicating a weaker preference for Zipfian behavior.

(a) Supervised-TriviaQA

(b) Meta-learned-TriviaQA

Figure 4: Relation between token frequency rank in the corpus and the token’s
mask frequency. Best viewed in color.

##  6 Related Work

#### Implicit Knowledge in Pre-trained Language Models.

Petroni et al. (2019) discovered that pre-trained language models can
implicitly store relational knowledge in their parameters, and such knowledge
can be accessed with cloze-style queries. Roberts et al. (2020) introduced the
task of closed-book QA, which breaks the convention of retriever-reader
strategy for open-domain QA, and requires the model to directly generate
answers with its implicit knowledge. Closed-book QA performance is boosted
significantly when salient span masking Guu et al. (2020) is used. Guu et al.
(2020) maintained that SSM helps the model to “focus on problems that require
world knowledge”.

#### Self-supervised Pre-training.

Pre-trained language models has shown its capability on a wide variety of NLP
tasks. Current self-supervised objectives are mostly heuristic, including
masked language modeling Devlin et al. (2019), span boundary representation
learning Joshi et al. (2020), corrupted sentence reconstruction Lewis et al.
(2020), etc. Raffel et al. (2020) systematically studied the self-supervised
objectives used in previous literature. Related to our goal of exploring pre-
training objectives, ELECTRA Clark et al. (2020) propose a replaced token
prediction task which improves pre-training efficiency. Chen et al. (2020)
propose to reduce the variance of gradients in SGD and expedite model pre-
training. Levine et al. (2020) propose to mask n-grams according to Pointwise
Mutual Information (PMI). These works typically consider the efficiency of an
objective when pre-training from scratch and without preconceived focus on a
given problem; while we focus on encoding knowledge or adapting the model
during intermediate pre-training with a given task in mind.

#### Domain/Task-specific Pre-training.

Gururangan et al. (2020) experiment on four domains (bio-medical, computer
science, news, reviews) and eight different datasets, where they discover that
pre-training with in-domain corpus leads to better downstream performance.
Kang et al. (2020) propose to learn a mask generator via reinforcement
learning. Closely related to us, Gu et al. (2020) propose task-guided pre-
training by learning to predict importance score for each token in pre-train
corpus. Vu et al. (2020); Pruksachatkun et al. (2020) studies knowledge
transfer from intermediate-task fine-tuning, while we focus on a different
problem setting of intermediate pre-training with generic corpus (e.g.,
Wikipedia). We believe both settings have practical utility in real-world
applications.

##  7 Conclusion

In this paper, we study the influence brought by different masking policies
used during intermediate pre-training, and offer two methods as our initial
attempts towards automating the discovery of optimal masking policy. From
extensive experiments with heuristic and learned masking policies across three
categories of tasks, we have identified several successful cases of
intermediate pre-training, offered in-depth analysis and insights for the
masking policies we used, discussed the risks of learned masking policies, and
summarized several suggestions for researchers who wish to adopt intermediate
pre-training in their applications.

We also acknowledge that, despite our additional efforts and experiments,
several observations still cannot be explained away. We invite future research
into this challenging and under-explored problem, to expand on our methods,
and to search the space of pre-training objectives beyond masked language
modeling. Furthermore, we hope our work encourages researchers to consider the
type of downstream applications they wish to deploy their LMs in, before
investing resources into large-scale pre-training.

## Acknowledgments

We would like to thank anonymous reviewers for their constructive feedback. We
would like to thank Sebastian Riedel, Edward Grefenstette, Douwe Kiela,
Patrick Lewis, Xian Li, Sewon Min, and USC INK Lab members for the insightful
discussions. Qinyuan Ye and Xiang Ren are supported in part by the Office of
the Director of National Intelligence (ODNI), Intelligence Advanced Research
Projects Activity (IARPA), via Contract No. 2019-19051600007; the DARPA MCS
program under Contract No. N660011924033; the Defense Advanced Research
Projects Agency with award W911NF-19-20271; NSF IIS 2048211.

## References

  * Antoniou et al. (2019) Antreas Antoniou, Harrison Edwards, and Amos Storkey. 2019.  How to train your MAML.  In _International Conference on Learning Representations_.
  * Berant et al. (2013) Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013.  Semantic parsing on Freebase from question-answer pairs.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 1533–1544, Seattle, Washington, USA. Association for Computational Linguistics.
  * Chen et al. (2020) Liang Chen, Tianyuan Zhang, Di He, Guolin Ke, Liwei Wang, and Tie-Yan Liu. 2020\.  Variance-reduced language pretraining via a mask proposal network.  _arXiv preprint arXiv:2008.05333_.
  * Clark et al. (2020) Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020.  ELECTRA: Pre-training text encoders as discriminators rather than generators.  In _ICLR_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dinan et al. (2019) Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, M. Auli, and J. Weston. 2019\.  Wizard of wikipedia: Knowledge-powered conversational agents.  _ArXiv_ , abs/1811.01241.
  * Finn et al. (2017) Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017.  Model-agnostic meta-learning for fast adaptation of deep networks.  volume 70 of _Proceedings of Machine Learning Research_ , pages 1126–1135, International Convention Centre, Sydney, Australia. PMLR.
  * Gokaslan and Cohen (2019) Aaron Gokaslan and Vanya Cohen. 2019.  Openwebtext corpus.  _URl: https://skylion007. github. io/OpenWebTextCorpus_.
  * Grefenstette et al. (2019) Edward Grefenstette, Brandon Amos, Denis Yarats, Phu Mon Htut, Artem Molchanov, Franziska Meier, Douwe Kiela, Kyunghyun Cho, and Soumith Chintala. 2019.  Generalized inner loop meta-learning.  _arXiv preprint arXiv:1910.01727_.
  * Gu et al. (2020) Yuxian Gu, Zhengyan Zhang, Xiaozhi Wang, Zhiyuan Liu, and Maosong Sun. 2020.  Train no evil: Selective masking for task-guided pre-training.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6966–6974, Online. Association for Computational Linguistics.
  * Gururangan et al. (2020) Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020.  Don’t stop pretraining: Adapt language models to domains and tasks.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8342–8360, Online. Association for Computational Linguistics.
  * Guu et al. (2020) Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020.  REALM: Retrieval-augmented language model pre-training.  In _International Conference on Machine Learning (ICML)_.
  * Hoffart et al. (2011) Johannes Hoffart, Mohamed Amir Yosef, Ilaria Bordino, Hagen Fürstenau, Manfred Pinkal, Marc Spaniol, Bilyana Taneva, Stefan Thater, and Gerhard Weikum. 2011.  Robust disambiguation of named entities in text.  In _Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing_ , pages 782–792, Edinburgh, Scotland, UK. Association for Computational Linguistics.
  * Jang et al. (2017) Eric Jang, Shixiang Gu, and Ben Poole. 2017.  Categorical reparameterization with gumbel-softmax.  _ArXiv_ , abs/1611.01144.
  * Joshi et al. (2020) Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020.  SpanBERT: Improving pre-training by representing and predicting spans.  _Transactions of the Association for Computational Linguistics_ , 8:64–77.
  * Joshi et al. (2017) Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017.  TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1601–1611, Vancouver, Canada. Association for Computational Linguistics.
  * Kang et al. (2020) Minki Kang, Moonsu Han, and Sung Ju Hwang. 2020.  Neural mask generator: Learning to generate adaptive word maskings for language model adaptation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6102–6120, Online. Association for Computational Linguistics.
  * Khashabi et al. (2020) Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. 2020.  UNIFIEDQA: Crossing format boundaries with a single QA system.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 1896–1907, Online. Association for Computational Linguistics.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.  Natural questions: A benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:452–466.
  * Levine et al. (2020) Yoav Levine, Barak Lenz, Opher Lieber, Omri Abend, Kevin Leyton-Brown, Moshe Tennenholtz, and Yoav Shoham. 2020.  PMI-masking: Principled masking of correlated spans.  _ArXiv_ , abs/2010.01825.
  * Levy et al. (2017) Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. 2017.  Zero-shot relation extraction via reading comprehension.  In _Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017)_ , pages 333–342, Vancouver, Canada. Association for Computational Linguistics.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Lin et al. (2019) Kevin Lin, Oyvind Tafjord, Peter Clark, and Matt Gardner. 2019.  Reasoning over paragraph effects in situations.  In _Proceedings of the 2nd Workshop on Machine Reading for Question Answering_ , pages 58–62, Hong Kong, China. Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Nagel (2016) Sebastian Nagel. 2016.  Cc-news.  _URL: http://web. archive. org/save/http://commoncrawl. org/2016/10/newsdatasetavailable_.
  * Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019.  fairseq: A fast, extensible toolkit for sequence modeling.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)_ , pages 48–53, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Petroni et al. (2020) Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vassilis Plachouras, Tim Rocktäschel, et al. 2020.  Kilt: a benchmark for knowledge intensive language tasks.  _arXiv preprint arXiv:2009.02252_.
  * Petroni et al. (2019) Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019.  Language models as knowledge bases? In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2463–2473, Hong Kong, China. Association for Computational Linguistics.
  * Pruksachatkun et al. (2020) Yada Pruksachatkun, Jason Phang, Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann, and Samuel R. Bowman. 2020\.  Intermediate-task transfer learning with pretrained language models: When and why does it work? In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5231–5247, Online. Association for Computational Linguistics.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67.
  * Roberts et al. (2020) Adam Roberts, Colin Raffel, and Noam Shazeer. 2020.  How much knowledge can you pack into the parameters of a language model? In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 5418–5426, Online. Association for Computational Linguistics.
  * Schmidhuber (1987) Jürgen Schmidhuber. 1987.  _Evolutionary principles in self-referential learning, or on learning how to learn: the meta-meta-… hook_.  Ph.D. thesis, Technische Universität München.
  * Tafjord et al. (2019) Oyvind Tafjord, Matt Gardner, Kevin Lin, and Peter Clark. 2019.  QuaRTz: An open-domain dataset of qualitative relationship questions.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5941–5946, Hong Kong, China. Association for Computational Linguistics.
  * Tandon et al. (2019) Niket Tandon, Bhavana Dalvi, Keisuke Sakaguchi, Peter Clark, and Antoine Bosselut. 2019.  WIQA: A dataset for “what if…” reasoning over procedural text.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 6076–6085, Hong Kong, China. Association for Computational Linguistics.
  * Thrun and Pratt (1998) Sebastian Thrun and Lorien Pratt. 1998.  Learning to learn: Introduction and overview.  In _Learning to learn_ , pages 3–17. Springer.
  * Trinh and Le (2018) Trieu H Trinh and Quoc V Le. 2018.  A simple method for commonsense reasoning.  _arXiv preprint arXiv:1806.02847_.
  * Vu et al. (2020) Tu Vu, Tong Wang, Tsendsuren Munkhdalai, Alessandro Sordoni, Adam Trischler, Andrew Mattarella-Micke, Subhransu Maji, and Mohit Iyyer. 2020.  Exploring and predicting transferability across NLP tasks.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7882–7926, Online. Association for Computational Linguistics.
  * Ye et al. (2020) Qinyuan Ye, Belinda Z. Li, Sinong Wang, Benjamin Bolte, Hao Ma, X. Ren, Wen tau Yih, and Madian Khabsa. 2020.  Studying strategically: Learning to mask for closed-book qa.  _ArXiv_ , abs/2012.15856.
  * Zhu et al. (2015) Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015.  Aligning books and movies: Towards story-like visual explanations by watching movies and reading books.  In _Proceedings of the IEEE international conference on computer vision_ , pages 19–27.
  * Zipf (1999) George Kingsley Zipf. 1999.  _The psycho-biology of language: An introduction to dynamic philology_ , volume 21.  Psychology Press.

##  Appendix A Additional Training Details

###  A.1 Supervised Policy

#### Training Details.

The embedding matrix $\mathbf{E}$ is initialized with the weights in BART-base
model. We optimize cross entropy loss between the logits outputted by the
model and the gold annotations. For each source of supervision stated above,
we train the policy for 30 epochs with learning rate of 1e-5 and batch size of
512, and select the best checkpoint according to validation loss.

###  A.2 Meta-learned Policy

0: Dataset $\mathcal{S}=\\{(\mathbf{c},\mathbf{s},\mathbf{t})\\}$

0: Masking Policy $g(.;\phi)$, A Pre-trained Language Model $f(.;\theta)$

1: for $p=1..T$ do

2: $\\{(\mathbf{c},\mathbf{s},\mathbf{t})\\}=\text{SampleBatch}(\mathcal{S})$

3: $\mathbf{x}\mathbf{c}$; $\mathbf{d}=g(\mathbf{x};\phi^{(p)})$;
$\mathbf{x}^{\prime}=\mathbf{x}\oplus\mathbf{d}$ // Apply current masking
policy $g(.;\phi)$

4:
$\theta^{(p,1)}=\theta^{(p,0)}-\alpha_{0}\nabla_{\theta^{(p,0)}}\mathcal{L}(f(\mathbf{x}^{\prime};\theta^{(p,0)}),\mathbf{x})$
// Inner Loop Update 1: Pre-train on $(\mathbf{x},\mathbf{x}^{\prime})$

5:
$\theta^{(p,2)}=\theta^{(p,1)}-\alpha_{1}\nabla_{\theta^{(p,1)}}\mathcal{L}(f(\mathbf{s};\theta^{(p,1)}),\mathbf{t})$
// Inner Loop Update 2: Fine-tune on $(\mathbf{s},\mathbf{t})$

6:
$\mathcal{L}^{\prime}=\mathcal{L}(f(\mathbf{s};\theta^{(p,2)}),\mathbf{t})-\mathcal{L}(f(\mathbf{s};\theta^{(p,1)}),\mathbf{t})$
// Compute loss measuring how fast the model adapts to
$(\mathbf{s},\mathbf{t})$

7: $\mathcal{L}_{reg}=\delta[|\gamma l(\mathbf{x})-l(\mathbf{d})|>\epsilon
l(\mathbf{x})](\gamma l(\mathbf{x})-l(\mathbf{d}))^{2}$ // Compute
regularization loss to control masking budget

8:
$\phi^{(p+1)}=\phi^{(p)}-\alpha_{2}\nabla_{\phi^{(p)}}(\mathcal{L}^{\prime}+\beta\mathcal{L}_{reg}(\mathbf{x},\mathbf{d}))$
// Outer Loop Update for $\phi$

9: $\theta^{(p+1,0)}=\theta^{(p,1)}$ // Maintain pre-train progress at
timestamp $p$

Algorithm 1 Meta-learning Policy $g(.;\phi)$

#### Design choices.

We use a 1D convolution layer with two additional linear layers as our policy
network $g(.;\phi)$. The linear layers output two logits for each token in
input sequence $\mathbf{x}$. The two logits for each tokens go through Gumbel
Softmax Jang et al. (2017) to decide whether it should be masked ($d_{i}=1$)
or not ($d_{i}=0$). We’ve also experimented with Bi-LSTM as the encoder, but
find meta-learning with LSTMs to be extremely unstable.

#### Intuitive Example.

The PTLM $f$ is given a piece of context $\mathbf{c}$ “Charles Schulz
(November 26, 1922 – February 12, 2000) was an American cartoonist” and is
expected to take an upcoming “closed-book exam” based on this piece of
context. In the pre-train step, the current policy $g$ predicts masks (e.g.,
Charles Schulz (<mask> – February 12, 2000) was an American cartoonist) and
take one step of optimization, implicitly encoding this piece of knowledge
into its parameters. After this, the PTLM “transit to closed-book exam mode”
by fine-tuning on $(\mathbf{s},\mathbf{t})$ for one step. Finally the language
model “takes the closed-book exam” and the loss for generating $\mathbf{t}$
given $\mathbf{s}$ as input can be interpreted as the supervision for the
masking decisions (i.e., whether masking “November 26, 1922” is helpful).

#### Pseudo-code.

We provide pseudo-code for our method in Algorithm 1.

##  Appendix B Hyperparameters

For downstream task fine-tuning, we first select the learning rate from
$\\{$5e-6, 1e-5, 2e-5, 3e-5$\\}$ and then fix learning rate to select batch
size from $\\{$32, 64, 128, 256$\\}$. See Table 5 for more details.

Parameter Name | Value
---|---
Max Epoch | 100
Validation Interval | 2 or 5
Warmup Updates | 500
Learning Rate |  $\\{$5e-6, 1e-5, 2e-5, 3e-5$\\}$
Batch Size |  $\\{$32, 64, 128, 256$\\}$
Label Smoothing | 0.1
Dropout | 0.1
Weight Decay | 0.01
Clip Norm | 0.1
Generation Beam Size | 4
Generation Min/Max Length | 1/20
Generation Length Penalty | 1.0

Table 5: Hyperparameters for Downstream Task Fine-tuning.

##  Appendix C Discussion on NQ

In Table 1 we observe that performances on NQ are close for all BART-base
models; therefore it is hard to rank all compared methods. We argue that
multiple factors leads to this phenomenon, including dataset characteristics
and evaluation protocol. Specifically, NQ may not be an ideal testbed for our
study due to three reasons.

Firstly, intermediate pre-training in general might not be as beneficial for
this particular task. For instance, Roberts et al. (2020) reports only 2% EM
gain on NQ using T5-11B. In our experiments, we use significantly smaller pre-
trained models (BART-base/large), so the effect brought by intermediate pre-
training will be even smaller. In our case we believe the effect is hidden in
the variance brought by random seeds.

Secondly, performance on NQ may not represent the real implicit knowledge
capacity of a LM. For reference, we observe a 20% dev set EM when fine-tuning
a randomly initialized BART-base model on NQ. The general pre-training stage
brings merely 4-5% EM improvement, and therefore the improvement brought by
intermediate pre-training can be marginal.

And finally, evaluation based on exact match may substantially underestimate
the model capability, as suggested in Roberts et al. (2020).

Category | Dataset | #Train | #Dev | #Test
---|---|---|---|---
Closed-book QA | Natural Questions Kwiatkowski et al. (2019) | 79,168 | 8,757 | 3,610
WebQuestions Berant et al. (2013) | 3,417 | 361 | 2,032
TriviaQA Joshi et al. (2017) | 78,785 | 8,837 | 11,313
Knowledge-Intensive Tasks (KILT) | Aidayago2 Hoffart et al. (2011) | 18,395 | 4,784 | 4,463
Zero-shot Relation Extraction Levy et al. (2017) | 147,909 | 3,724 | 4,966
Wizard of Wikipedia Dinan et al. (2019) | 94,577 | 3,058 | 2,944
Knowledge-Intensive Multiple-choice QA | ROPES Lin et al. (2019) | 10,924 | 844 | 844
WIQA Tandon et al. (2019) | 29,808 | 6,894 | 3,003
QuaRTz Tafjord et al. (2019) | 2,696 | 384 | 784

Table 6: Details of Datasets Used in This Study.

##  Appendix D Reproducibility

###  D.1 Dataset Details

We obtain closed-book QA datasets from
https://github.com/facebookresearch/DPR/blob/master/data/download_data.py,
knowledge-intensive language tasks from
https://github.com/facebookresearch/KILT/blob/master/scripts/donwload_all_kilt_data.py.
We obtain ROPES, WIQA and QuaRTz from huggingface datasets
(https://huggingface.co/datasets). For more details, see Table 6. KILT hosts
the test set evaluation on its leaderboard and the test set annotations are
not publicly available; therefore we report performance on dev set in Table 2.
The test set annotations for ROPES is not publicly available, so we take 50%
of original dev set as the new dev set, and the other 50% as the new test set.

###  D.2 Training Details

#### Implementation.

All our experiments are implemented with fairseq Ott et al. (2019). For
higher-order optimization in the meta-learning approach optimization, we use
higher library Grefenstette et al. (2019). Our code will be released upon
acceptance.

#### Infrastructure and Runtime.

Intermediate pre-training experiments are done with NVIDIA Quadro GP100 or
NVIDIA Tesla V100 GPUs, based on availability. For BART-Base, we use 32 GPUs
in parallel; For BART-Large, we use 64 GPUs in parallel. Pre-train job takes
less than 24 hours for BART-Base models and less than 48 hours for BART-Large
models. The checkpoints from intermediate pre-training will be released upon
acceptance. Fine-tuning jobs are all done with one single GPU, with either
NVIDIA Quadro GP100, NVIDIA Quadro RTX 8000, NVIDIA Quadro RTX 6000, NVIDIA
GeForce RTX 1080 Ti, or NVIDIA GeForce RTX 2080 Ti, based on availability. The
list the estimated maximum training time in the following: NQ (4h), WQ (2h),
TQA (40h), AY2 (4h), ZSRE (2h), ROPES (1h), WIQA (1h), QuaRTz (1h).

#### Number of Parameters.

BART-Base model contains 140 million parameters, BART-Large model contains 406
million parameters. Supervised policies contain 43 million parameters (where
the word embeddings take 39 millions parameters). Meta-learned policies
contain 40 million parameters (where the word embeddings take 39 millions
parameters).