# Spatial Analysis of Physical Reservoir Computers

Jake Love  Faculty of Physics, University of Duisburg-Essen, 47057 Duisburg,
Germany  Jeroen Mulkers  DyNaMat, Department of Solid State Sciences, Ghent
University, Ghent, Belgium  Robin Msiska  Faculty of Physics, University of
Duisburg-Essen, 47057 Duisburg, Germany  George Bourianoff  Senior Principle
Engineer, Intel Corp. (Retired)  Jonathan Leliaert  DyNaMat, Department of
Solid State Sciences, Ghent University, Ghent, Belgium  Karin Everschor-Sitte
Faculty of Physics and Center for Nanointegration Duisburg-Essen (CENIDE),
University of Duisburg-Essen, 47057 Duisburg, Germany

###### Abstract

Physical reservoir computing is a computational framework that implements
spatiotemporal information processing directly within physical systems. By
exciting nonlinear dynamical systems and creating linear models from their
state, we can create highly energy-efficient devices capable of solving
machine learning tasks without building a modular system consisting of
millions of neurons interconnected by synapses. To act as an effective
reservoir, the chosen dynamical system must have two desirable properties:
nonlinearity and memory. We present task agnostic spatial measures to locally
measure both of these properties and exemplify them for a specific physical
reservoir based upon magnetic skyrmion textures. In contrast to typical
reservoir computing metrics, these metrics can be resolved spatially and in
parallel from a single input signal, allowing for efficient parameter search
to design efficient and high-performance reservoirs. Additionally, we show the
natural trade-off between memory capacity and nonlinearity in our reservoir’s
behaviour, both locally and globally. Finally, by balancing the memory and
nonlinearity in a reservoir, we can improve its performance for specific
tasks.

##  I Introduction:

Reservoir computing (RC) is a unified supervised learning framework derived
from recurrent neural networks (RNN) that utilizes dynamical systems to
perform computations [1, 2]. A reservoir computer consists of two components;
a fixed nonlinear dynamical system called a reservoir and a trainable linear
readout layer. Input signals are fed into the reservoir, and the system’s
resulting transient states are observed. The observed states, referred to as
readout values, are then fed through the linear readout layer, which can be
trained to solve specific problems. By separating the complex nonlinear
dynamics of the reservoir from the trainable readout layer, the connecting
weights can be solved inexpensively using a least-squares model, often
requiring far less training data than conventional RNNs [3]. Typically,
reservoir computers are suited to pattern recognition problems dealing with
temporally or spatially correlated information, such as audio or image data
[4].

The RC framework is not limited to mathematical models and can also be
implemented with physical dynamical systems [5]. In physical RC, the input
signal is fed into a physical substrate via time-dependent excitations. A
finite representation of the system’s transient states over the excitation is
then observed via experimental methods, representing the readout values. Like
the non-physical case, the readout values are put through a trained linear
model to perform the desired computation. By implementing reservoir computers
physically, also one takes advantage of physics’s inherently parallel nature
and opens doors to low cost, energy-efficient computations that are not
constrained by the limitations of conventional CMOS hardware,

Although one can use any physical dynamical system for reservoir computing,
some are better than others. Solving complex tasks requires that the system
has several fundamental properties: high dimensionality, nonlinearity, and
fading memory [6]. A diverse set of physical dynamical systems based upon
optics, spintronics [7] and even a bucket of water [8] have already been
proposed to implement the reservoir computing paradigm including magnetic [9]
and magnetic skyrmion based systems [10, 11, 12].

Existing literature provides global metrics to quantify the total memory
capacity [13] and nonlinearity [6, 14] of reservoir computers. While these
global metrics are useful for the characterisation of reservoir computers,
they provide limited guidance in interpreting where the required properties,
memory and nonlinearity, arise within a dynamical system.

This work introduces methods that measure the memory and nonlinearity locally
for physical reservoirs with spatially correlated readouts. One can use these
methods to understand which physical structures lead to memory and
nonlinearity in the reservoir. By creating conditions that replicate such
structures, one can parameterise future reservoirs in a directed manner to
optimise a balance between nonlinearity and memory as required by possible
tasks.

##  II  Spatially Resolved Metrics:

We introduce two spatially resolved metrics for memory and nonlinearity. Both
metrics can be calculated in parallel from the same set of input and output
data and are task agnostic, hence can be applied to various input sequences.
This allows one to analyse how different input sequences, such as a sequence
of uniform random numbers or series generated from a Mackey-Glass equation,
couple to different structures in physical reservoirs.

###  II.1 Nonlinearity

(a)  (b)

Figure 1: Sketches comparing the metrics for (a) nonlinearity and (b) memory
capacity. For nonlinearity, we use multiple past inputs to predict the present
value of individual readout nodes. For memory capacity, we use multiple
readout nodes to predict the inputs at an individual time delay in the past.

As a reservoir is a nonlinear system with fading memory, its readout node
values $\bm{y}$ can be modelled as a function of input to the system $u$ using
a Volterra series [15] such that

$\bm{y}(t)=\bm{h}^{(0)}+\int_{\tau=0}^{\infty}\bm{h}^{(1)}(\tau)u(t-\tau)d\tau+\bm{y}^{\mathrm{NL}}(t).$
(1)

Here, $\bm{h}^{(0)}$ and $\bm{h}^{(1)}$ are zeroth and first order Volterra
kernels and $\bm{y}^{\mathrm{NL}}(t)$ represents truncated nonlinear terms of
the Volterra series. For a linear system, the truncated terms of Eq. (1) are
zero and for a highly nonlinear system these truncated terms dominate. To
measure the nonlinearity of each readout node in the reservoir, we quantify
how important these truncated terms are when modelling the individual readout
as a Volterra series.

We first model each readout node $y_{n}(t)$ in the reservoir as a discrete-
time truncated Volterra series that can be estimated by the history of inputs
$u(t)$.

$y_{n}^{L}(t)=h^{(0)}+\sum_{\tau=0}^{\infty}h^{(1)}_{\tau}u(t-\tau),$ (2)

This model is then implemented as a linear estimator of the form,

$\hat{y}_{n}(t)=c+\sum_{\tau=0}^{k}w_{\tau}u(t-\tau),$ (3)

where weights $w$ and constant $c$ are trained on the known input and output
data, see Fig. 1a. $k$ is the time delay at which we truncate the Volterra
kernel, picked to be longer than the relaxation time of the reservoir. The
quality of the estimator in Eq. (3) is measured using the R2 correlation
coefficient (see Appendix A for details) and can be considered a direct
measure of how linear the response of each reservoir node is. To convert this
to a measure of nonlinearity we subtract the correlation coefficient from one.

$\mathrm{NL_{n}}=1-\mathrm{R}^{2}[\hat{y}_{n},y_{n}]$ (4)

The result is a measure of nonlinearity $\mathrm{NL_{n}}$ for each individual
node of the reservoir bound by the interval $[0,1]$, where zero indicates a
linear relationship between reservoir input and a readout node’s value, and
conversely values close to one indicate a highly nonlinear relationship. The
definition of nonlinearity proposed here is designed to be decoupled from the
measure of memory in the system, hence it is possible to have systems with
high memory and zero nonlinearity (a linear time-invariant filter) and vice
versa (a high-order polynomial). This decoupling of memory and nonlinearity is
in contrast to other methods such as the quadratic memory capacity task seen
in Refs [16, 17] and the $n$-bit parity task [18], that measure nonlinearity
as a form of memory.

###  II.2 Memory

To spatially measure memory in our system, we use an adaption of the linear
memory capacity metric [13]. In the original metric, a linear estimator is
trained to recall the history of a reservoir’s inputs $u$ from all the readout
values of all readout nodes. We use this metric to measure memory on a per-
node basis by considering the memory capacity of only the readout nodes within
a threshold distance, in physical space, of the chosen node. The threshold
distance and the definition of distance and the between nodes is chosen
depending on the system being analysed.

For a chosen $n$th readout node we create a vector $\gamma_{n}$ of all other
nodes of the reservoir that fall within a threshold distance, measured from
$n$. We then use $\gamma_{n}$ to create an estimator $\hat{u}_{n}(t-\tau)$
that can recall the input at a time delay of $\tau$,

$\hat{u}_{n}(t-\tau)=c+\sum_{i}w_{i}\gamma_{n,i}(t),$ (5)

see Fig. 1b.

The memory capacity of the nodes in $\gamma_{n}$ is then calculated by summing
the correlation coefficient of all estimators up to a cutoff threshold, $k$,
that is larger than the relaxation time of the reservoir.

$\mathrm{MC_{n}}=\sum_{\tau=1}^{k}\mathrm{R}^{2}[u(t-\tau),\hat{u_{n}}(t-\tau)]$
(6)

###  II.3 Stability

We also consider the system’s stability an additional metric for physical
systems as it is a prerequisite for its functional principle. Being stable is
essential for physical reservoirs to ensure reproducibility and prevent the
reservoir from going into a chaotic regime. For stable reservoirs, the system
relaxes back to a consistent (meta)stable state in the absence of inputs.

To measure the stability, we first pass a random input random signal into the
reservoir. After the input fizzles out, the reservoir is relaxed until it
reaches a (meta)stable state. The difference between the system’s initial and
final state characterises stability, where a small (high) difference refers to
a high (low) stability. The details of the difference measure can be adjusted
to the specific system.

##  III Spatial analysis of Magnetic Skyrmion Reservoir:

Figure 2: Example distribution of grain inhomogeneities across a magnetic thin
film. Left panel: each grain has a distinct magnetic anisotropy strength as
depicted by their greyscale colour. Center panel: the non-uniform anisotropy
creates pinning sites for skyrmions, resulting in their aperiodic
arrangements. The color code depicts the out-of-plane component of the local
normalized magnetization. Right panel: to readout values from the skyrmion
reservoir, the out-of-plane projection of the magnetisation field is taken and
then discretised by taking it’s mean value over an area.

To demonstrate the application of the above-introduced metrics, we apply them
to a specific implementation of physical reservoir computing [12, 10].
Magnetic skyrmions are magnetic whirls with non-trivial topology. They occur,
for example, in magnetic thin films where spins favour a canted alignment,
often caused by Dzyaloshinskii-Moriya interaction (DMI). Skyrmions host
complex interactions with spin-polarized electric currents, defects in their
host material, and also other skyrmions, resulting in deformations with highly
nonlinear behaviour [11]. Skyrmions elastically deform back to their ground
state upon removal of external forces; a condition which is necessary for
fading memory.

In addition to the basic requirements, magnetic skyrmions are a competitive
system for reservoir computing due to their nanoscale size, responsiveness to
low-power excitations, and their ability to integrate with CMOS technology
[19] and other potential magnetic devices [20, 7, 10, 21].

For our specific model, we consider an inhomogeneous magnetic thin-film with
an interfacial DMI populated with magnetic skyrmions. We introduce grain
inhomogeneities with varying magnetic anisotropy, resulting in a complex
energy landscape. This causes specific locations on the film to be
energetically favourable for skyrmions, referred to as pinning sites. As a
result, the local energetic minimum state of the magnetic texture becomes an
aperiodic arrangement of skyrmions, see Fig. 2.

We implement the input $u(t)$ to the reservoir as a time-varying
unidirectional electric current across the film characterised by the current
density $\bm{j}(t)$,

$\bm{j}(t)=j_{\mathrm{in}}u(ft)\hat{\bm{x}}.$ (7)

Here $\hat{\bm{x}}$ is the current direction and $j_{\mathrm{in}}u(ft)$ is the
time-dependent current strength. $j_{\mathrm{in}}$ provides control over input
scaling, and the factor $f$ controls time scaling. Discrete-time inputs are
fed into the reservoir as a step function such that the reciprocal of $f$ is
the duration of each step. The electrical current interacts with the magnetic
skyrmions via spin-torques, resulting in a time-dependent nonlinear
deformation of the magnetic texture. Provided the current is within a certain
threshold, the structure will relax back to its original state when the
current is removed.

For readout nodes, we observe the changes in magnetic texture by taking a low-
resolution snapshot of the film’s out-of-plane magnetisation component. We
choose the resolution to be of similar magnitude to the sizes of individual
skyrmions as indicated in Fig. 2 (right panel).

We implement the model in simulation using the MuMax3 software [22], with
precise details on the simulation setup given in the appendix B.

###  III.1 Results

(a)

(b)

Figure 3: Spatial analysis of two skyrmion reservoirs with (a) high DMI and
(b) low DMI. The relaxed magnetisation shows the state of the reservoir when
no input is applied. The nonlinearity and memory plots show the distribution
said quantities over the reservoir. The stability shows the difference between
the reservoir’s initial and final state when measuring the nonlinearity and
memory metrics.

We first analyse two distinct skyrmion reservoirs with different DMI
strengths, $3.3\times 10^{-3}$ Jm-2 and $3.6\times 10^{-3}$ Jm-2, but
otherwise identical dynamics. These reservoirs can be visually distinguished
by the sizes of the individual skyrmion present on the thin-films as seen in
the magnetisation shown in Fig 3. Due to the effect of DMI on the possible
metastable states, the two reservoirs also have inherently different initial
configurations.

Both reservoirs are tested against a discrete-time random input signal,
$u(t)$. The signal has 1500 elements drawn from a random distribution with the
bounds $[-1,+1]$. The input is coupled to the reservoirs with input-scaling
$j_{\mathrm{in}}=300\times 10^{9}$ Am-2 and time-scaling $f=4$ GHz. The
memory, nonlinearity and stability are spatially analysed. The results are
shown in Fig 3. For the case of the skyrmion reservoir, we define stability as
the difference between the magnetic states before and after the random input
signal has been applied.

For both reservoirs, we can observe the well-studied trade-off between memory
and nonlinearity [14, 23, 2] both spatially and globally.

Figure 4: Resolved memory, nonlinearity and stability for system with a DMI
gradient  Figure 5: Mean squared error (MSE) for prediction of the Mackey-
glass series for $k$ steps in the future. The results for the high, low and
gradient DMI reservoirs are shown.

Locally, regions of high nonlinearity correspond closely to regions of low
memory, as seen predominantly in Fig 3(a) and less so in Fig 3(b). However,
the metrics are not entirely orthogonal as regions or with both memory and
nonlinearity do coexist. Physically, these regions of high nonlinearity appear
where changes to the magnetic texture are greatest. This is reflected in the
stability plots where the change in initial and final states of the reservoir
is observed. Although this results in the fading memory property being
violated, we argue that the material can still act as a reservoir provided the
global structure of the texture is maintained. In the skyrmion reservoir case,
changes to local structure can be due to individual skyrmions switching
between a small group of distinct pinning sites. However, this is not
necessarily a problem as, once the reservoir is ’warmed up’, the dynamics of
the reservoir may lead to skyrmions constantly switching between pinning
sites. The final pinning site upon removal of input is dependent on the local
history of the input and hence, the time at which the signal is removed from
the reservoir.

Globally, our analysis suggests that the high DMI reservoir has an overall
higher mean memory capacity and lower mean nonlinearity for input $u(t)$ than
the low DMI reservoir. As both high memory capacity and high nonlinearity are
desirable for reservoir computing, we next analyse a skyrmion ’mixture
reservoir’ with a low to high DMI gradient. By mixing nonlinear and linear
dynamics, we find that the skyrmion mixture reservoir has improved task
performance over the low and high reservoirs. This complements the existing
idea of an echo-state network-based ’mixture reservoir’, where replacing small
numbers of nonlinear nodes with linear nodes in the network improves their
performance for time series forecasting tasks [14].

We implement the skyrmion mixture reservoir by modifying our original
reservoir to possess a DMI gradient from low ($3.3\times 10^{-3}$ Jm-2) to
high ($3.6\times 10^{-3}$ Jm-2) DMI along the axis parallel to the input
current direction. We extend the reservoir geometrically along the same axis
by a factor of four so that the DMI gradient is shallow. In Fig 4 we see that
as a result of the DMI gradient, there is a also a gradient in the
nonlinearity and memory metrics.

We next apply the reservoir to a chaotic time-series prediction task. We use a
Mackey-Glass [24] system to generate a chaotic time series, see Appendix C for
details. We input this series into the reservoir and train the readout to
predict the value of the series at $k$ steps in the future. The results for
the prediction task are compared to the reservoirs with constant low and high
DMI, extended to the same physical size as the gradient reservoir. The results
of predicting up to 50 future steps are shown in Fig. 5 in which a minor
performance improvement is obtained when using the gradient DMI ’mixture
reservoir’.

##  IV Summary

In this work, we have demonstrated how two independent, spatial metrics can be
employed to locally quantify the properties of a reservoir computer in terms
of nonlinearity and memory. Both metrics can be calculated in parallel using a
small random dataset. This allows for a rapid evaluation of reservoir models
and efficient screening of reservoir parameters for ’good’ reservoir behaviour
and tuning reservoir performance to purpose. For the particular reservoir
computing model studied, the skyrmion reservoir, we find that dynamics in
which both memory and nonlinear effects are present lead to improved task
performance for the Mackey-Glass series prediction task.

The metrics can also be used to better understand which environments lead to
memory and nonlinearity in physical systems. By creating conditions that lead
to desirable dynamics, one can parameterise reservoirs in a guided manner to
optimise a balance between nonlinearity and memory. For example, in the case
of the skyrmion reservoir we achieve this be introducing a gradient in the DMI
parameter. The spatial metrics also provide a path to effectively choose a
readout structure guided by local contributions to memory and nonlinearity.
For example, having fewer effectively placed readout nodes can reduce the
number of model parameters, making the engineering of such devices more
feasible.

###### Acknowledgements.

We acknowledge funding from the Emergent AI Centre funded by the Carl-Zeiss-
Stiftung and the Deutsche Forschungsgemeinschaft under Project No. 320163632.
J.L. was supported by the Fonds Wetenschappelijk Onderzoek (FWO-Vlaanderen)
with senior postdoctoral fellowship No12W7622N. Part of the computational
resources and services used in this work were provided by the VSC (Flemish
Supercomputer Center), funded by Ghent University, the Research Foundation
Flanders (FWO) and the Flemish Government – department EWI.

##  Appendix A Estimator Quality

To measure the quality of the linear estimators for both nonlinearity, Eq. (3)
and memory Eq. (5), we use the R2 coefficient of determination. Consider the
linear estimator

$\hat{y}(t)=\sum_{i}w_{i}u(t).$ (8)

We train the weights $w$ of the estimator using the first 75% of our input
$u(t)$ and output $y(t)$ samples, minimising the residual sum of squares
error. Using the remaining 25% of input samples, we compare the corresponding
estimated values $\hat{y}(t)$ to the true value $y(t)$ with

$\mathrm{R}^{2}[\hat{y},y]=\frac{\mathrm{cov}^{2}(\hat{y}(t),y(t))}{\sigma^{2}(\hat{y}(t))\sigma^{2}(y(t))}.$
(9)

Here $\sigma^{2}$ is the variance and ’cov’ is the covariance.
$\mathrm{R}^{2}[\hat{y},y]$ is a measure for the estimators quality, ranging
from zero (no correlation) to one (perfect prediction). In this work we use a
total of 1000 data points for training and testing to measure estimator
quality.

##  Appendix B Simulation Details for Skyrmion Reservoir

The skyrmion reservoirs considered in this study were simulated using MuMax3
[22]. The homogeneous DMI reservoirs for the initial spatial analysis, were
modelled with a 1024 by 1024 cell grid with a cell size of 1 nm, smaller than
the exchange length of the simulated material. The gradient reservoir, along
with additional homogeneous DMI reservoirs for comparing Mackey-Glass
prediction performance, were modelled with a 4096 by 1024 cell grid.

We simulated the Co layer of Pt/Co/MgO [25], with the following material
parameters: saturation magnetisation $M_{\mathrm{sat}}=580$ kAm-1, exchange
coupling constant $A_{\mathrm{ex}}=15$ pJm-1, mean uniaxial anisotropy
$K_{u}=0.8$ MJm-3, temperature $T=0$ K, and we used an electrical current
polarisation of 1. The DMI parameter is reservoir dependant, either $3.3\times
10^{-3}$ Jm-2 or $3.6\times 10^{-3}$ Jm-2. The anisotropy grain
inhomogeneities were generated using a Voronoi tessellation [26] and have
average size of 40 nm and anisotropy variance of 20% (see Fig. 2(a) of the
main text), similar to the experimentally observed grains in [27]. In the
simulations, we do not include anisotropic magneto-resistance effects and
instead assume the material resistance to be constant.

We initialise the magnetic texture by first creating a skyrmion lattice and
letting it relax to a metastable state. To further stabilise the system, we
excite the texture with a random input current for $200$ ns, then allow it to
relax. This is to remove any low energy, initial instabilities in the
reservoir and to obtain a metastable state with a higher energy barrier
suitable for RC.

##  Appendix C Mackey-Glass Details

The Mackey-Glass system is described by the equation [24]

$\frac{dx}{dt}=\frac{ax(t-\tau)}{1+x^{n}(t-\tau)}-bx(t),$ (10)

where $a=0.2$, $b=0.1$, $n=10.0$ and $\tau=23.0$. Since the dynamics of the
series generated by Eq. 10 are of a different timescale to the dynamics of the
reservoir, we also scale time before using it as an input. This is done such
that one unit of $t$ in Eq. 10 is scaled to $1$ ns of physical input. The
input is coupled to the reservoir with input-scaling $j_{\mathrm{in}}=3\times
10^{11}$ Am-2.

## References

  * Lukoševičius and Jaeger [2009] M. Lukoševičius and H. Jaeger, Reservoir computing approaches to recurrent neural network training, Computer Science Review 3, 127 (2009).
  * Verstraeten  _et al._ [2007] D. Verstraeten, B. Schrauwen, M. d’Haene, and D. Stroobandt, An experimental unification of reservoir computing methods, Neural networks 20, 391 (2007).
  * Jaeger [2002a] H. Jaeger, Tutorial on training recurrent neural networks, covering BPPT, RTRL, EKF and the ”echo state network” approach (GMD - German National Research Institute for Computer Science, 2002).
  * Jaeger and Haas [2004] H. Jaeger and H. Haas, Harnessing nonlinearity: Predicting chaotic systems and saving energy in wireless communication, Science (2004).
  * Tanaka  _et al._ [2019] G. Tanaka, T. Yamane, J. B. Héroux, R. Nakane, N. Kanazawa, S. Takeda, H. Numata, D. Nakano, and A. Hirose, Recent advances in physical reservoir computing: A review, Neural Networks 115, 100 (2019).
  * Dambre  _et al._ [2012] J. Dambre, D. Verstraeten, B. Schrauwen, and S. Massar, Information processing capacity of dynamical systems, Scientific Reports 2, 1 (2012).
  * Nakane  _et al._ [2019] R. Nakane, G. Tanaka, and A. Hirose, In a spin-wave reservoir for machine learning, in  _2019 International Joint Conference on Neural Networks IJCNN_ (IEEE, 2019) pp. 1–9.
  * Fernando and Sojakka [2003] C. Fernando and S. Sojakka, Pattern recognition in a bucket, in  _European conference on artificial life_ (Springer, 2003) pp. 588–597.
  * Furuta  _et al._ [2018] T. Furuta, K. Fujii, K. Nakajima, S. Tsunegi, H. Kubota, Y. Suzuki, and S. Miwa, Macromagnetic simulation for reservoir computing utilizing spin dynamics in magnetic tunnel junctions, Physical Review Applied 10, 034063 (2018).
  * Prychynenko  _et al._ [2018] D. Prychynenko, M. Sitte, K. Litzius, B. Krüger, G. Bourianoff, M. Kläui, J. Sinova, and K. Everschor-Sitte, Magnetic skyrmion as a nonlinear resistive element: a potential building block for reservoir computing, Physical Review Applied 9, 014034 (2018).
  * Bourianoff  _et al._ [2018] G. Bourianoff, D. Pinna, M. Sitte, and K. Everschor-Sitte, Potential implementation of reservoir computing models based on magnetic skyrmions, AIP Advances 8, 055602 (2018).
  * Pinna  _et al._ [2020] D. Pinna, G. Bourianoff, and K. Everschor-Sitte, Reservoir computing with random skyrmion textures, Physical Review Applied 14, 054020 (2020).
  * Jaeger [2002b] H. Jaeger, Short term memory in echo state networks, in  _GMD Technical Report 152_ (GMD - German National Research Institute for Computer Science, 2002).
  * Inubushi and Yoshimura [2017] M. Inubushi and K. Yoshimura, Reservoir computing beyond memory-nonlinearity trade-off, Scientific Reports 7, 1 (2017).
  * Boyd and Chua [1985] S. Boyd and L. Chua, Fading memory and the problem of approximating nonlinear operators with volterra series, IEEE Transactions on Circuits and Systems 32, 1150 (1985).
  * Duport  _et al._ [2012] F. Duport, B. Schneider, A. Smerieri, M. Haelterman, and S. Massar, All-optical reservoir computing, Optics Express 20, 22783 (2012).
  * Grigoryeva  _et al._ [2015] L. Grigoryeva, J. Henriques, L. Larger, and J.-P. Ortega, Optimal nonlinear information processing capacity in delay-based reservoir computers, Scientific Reports 5, 1 (2015).
  * Tsunegi  _et al._ [2019] S. Tsunegi, T. Taniguchi, K. Nakajima, S. Miwa, K. Yakushiji, A. Fukushima, S. Yuasa, and H. Kubota, Physical reservoir computing based on spin torque oscillator with forced synchronization, Applied Physics Letters 114, 164101 (2019).
  * Finocchio  _et al._ [2020] G. Finocchio, M. Di Ventra, K. Y. Camsari, K. Everschor-Sitte, P. K. Amiri, and Z. Zeng, The promise of spintronics for unconventional computing, Journal of Magnetism and Magnetic Materials , 167506 (2020).
  * Torrejon  _et al._ [2017] J. Torrejon, M. Riou, F. A. Araujo, S. Tsunegi, G. Khalsa, D. Querlioz, P. Bortolotti, V. Cros, K. Yakushiji, A. Fukushima, _et al._ , Neuromorphic computing with nanoscale spintronic oscillators, Nature 547, 428 (2017).
  * Romera  _et al._ [2018] M. Romera, P. Talatchian, S. Tsunegi, F. A. Araujo, V. Cros, P. Bortolotti, J. Trastoy, K. Yakushiji, A. Fukushima, H. Kubota, _et al._ , Vowel recognition with four coupled spin-torque nano-oscillators, Nature 563, 230 (2018).
  * Vansteenkiste  _et al._ [2014] A. Vansteenkiste, J. Leliaert, M. Dvornik, M. Helsen, F. Garcia-Sanchez, and B. Van Waeyenberge, The design and verification of mumax3, AIP advances 4, 107133 (2014).
  * Verstraeten  _et al._ [2010] D. Verstraeten, J. Dambre, X. Dutoit, and B. Schrauwen, Memory versus non-linearity in reservoirs, in  _The 2010 international joint conference on neural networks IJCNN_ (IEEE, 2010) pp. 1–8.
  * Mackey and Glass [1977] M. C. Mackey and L. Glass, Oscillation and chaos in physiological control systems, Science 197, 287 (1977).
  * Wang  _et al._ [2018] X. Wang, H. Yuan, and X. Wang, A theory on skyrmion size, Communications Physics 1, 1 (2018).
  * Leliaert  _et al._ [2014] J. Leliaert, B. Van de Wiele, A. Vansteenkiste, L. Laurson, G. Durin, L. Dupré, and B. Van Waeyenberge, Current-driven domain wall mobility in polycrystalline permalloy nanowires: A numerical study, Journal of Applied Physics 115, 233903 (2014).
  * Baćani  _et al._ [2019] M. Baćani, M. A. Marioni, J. Schwenk, and H. J. Hug, How to measure the local dzyaloshinskii-moriya interaction in skyrmion thin-film multilayers, Scientific Reports 9, 1 (2019).


✓✓ ✗✗

# Drop your Decoder: Pre-training with Bag-of-Word Prediction for Dense
Passage Retrieval

Guangyuan Ma1,2, Xing Wu1,2, Zijia Lin3, Songlin Hu1,2 1Institute of
Information Engineering, Chinese Academy of Sciences, Beijing, China
2School of Cyber Security, University of Chinese Academy of Sciences, Beijing,
China
3Kuaishou Technology, Beijing, China
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract.

Masked auto-encoder pre-training has emerged as a prevalent technique for
initializing and enhancing dense retrieval systems. It generally utilizes
additional Transformer decoder blocks to provide sustainable supervision
signals and compress contextual information into dense representations.
However, the underlying reasons for the effectiveness of such a pre-training
technique remain unclear. The usage of additional Transformer-based decoders
also incurs significant computational costs. In this study, we aim to shed
light on this issue by revealing that masked auto-encoder (MAE) pre-training
with enhanced decoding significantly improves the term coverage of input
tokens in dense representations, compared to vanilla BERT checkpoints.
Building upon this observation, we propose a modification to the traditional
MAE by replacing the decoder of a masked auto-encoder with a completely
simplified Bag-of-Word prediction task. This modification enables the
efficient compression of lexical signals into dense representations through
unsupervised pre-training. Remarkably, our proposed method achieves state-of-
the-art retrieval performance on several large-scale retrieval benchmarks
without requiring any additional parameters, which provides a 67% training
speed-up compared to standard masked auto-encoder pre-training with enhanced
decoding. 111Our code will be available at
https://github.com/ma787639046/bowdpr.

Dense Retrieval, Masked auto-encoder pre-training, Bag-of-Word prediction

††conference: Conference Title; Month xx–xx, xxxx; Conference, City††ccs:
Information systems Language models

##  1\. Introduction

Dense passage retrieval utilizes pre-trained language models (PLMs), e.g.,
BERT (Devlin et al., 2019), to encode the lexical information as
representations in latent spaces and retrieve relevant passages of a given
query based on the similarity search (Mussmann and Ermon, 2016; Malkov and
Yashunin, 2020). Pre-training with masked auto-encoder (MAE) has recently
become a hot topic to boost the initialization ability of PLMs for downstream
retrieval tasks. Such a technique appends carefully designed Transformers-
based decoder blocks to the PLM encoder, forming encoder-decoder architecture,
and bringing supervision signals to the representations.

Figure 1.  Comparison of Masked Auto-Encoder Pre-training and Bag-of-Word
Prediction Pre-training.

Recent studies have explored several pre-training strategies, such as pre-
training with the weakened decoder (Lu et al., 2021), extreme mask ratio (Liu
and Shao, 2022), replace token detection (Wang et al., 2022), context
reconstruction (Wu et al., 2023), and multi-task decoding (Zhou et al., 2022).
As is shown in Figure 1 A), these MAE-style pre-trainings commonly preserve
several traits: 1) Asymmetric structure: The encoder is a full-sized BERT for
generating discriminative representations based on input sentences. While the
decoder is typically a shallow Transformers-based block with a few or just one
layer. 2) Bottlenecked connection. The connection between the encoder and the
decoder is typically a single representation vector. 3) Reconstruction
signals. Carefully designed decoding tasks are applied to the decoder side,
which decodes context information through auto-encoding (Gao and Callan, 2021;
Liu and Shao, 2022), auto-regression (Lu et al., 2021), enhanced decoding (Liu
and Shao, 2022) or other curated decoding tasks (Wang et al., 2022; Zhou et
al., 2022; Wu et al., 2022). By optimizing a weak decoder, the lexical
information is compressed into the sentence representations, thus providing
good initialization ability for retrieval tasks. Notably, enhanced decoding
from (Liu and Shao, 2022) is proven effective for dense retrieval pre-
training, which utilizes sentence representations as query streams and masked
context embeddings as content streams.

However, we argue that existing MAE-style pre-training paradigms have several
drawbacks, which may hinder the development of dense retrieval pre-training:
1) Lack of interpretability. Existing studies focus on creating carefully
designed tasks, but still neglect to reveal the underlying reason for why such
a pre-training method is effective. 2) Additional complexity. MAE-style pre-
training involves optimization with millions of document collections. The
usage of additional Transformers-based decoders brings considerable GPU memory
usage and additional $O(n^{2})$ computational complexity. Some studies also
need additional data preprocessing (Wu et al., 2022; Zhou et al., 2022) or
extensive data masking (Liu and Shao, 2022; Wu et al., 2023).

In this study, we aim to solve the above issues and develop a more
computationally efficient and more interpretable pre-training technique. The
semantic meaning of a dense representation is typically condensed in latent
spaces and hard to directly interpret. In our preliminary studies, we project
the low-dimensional representation to the vocabulary space with the BERT
language model heads and compare the changes in MAE-style pre-training.
Notably, we discover that MAE-style pre-training with enhanced decoding (Liu
and Shao, 2022) significantly enhances the term coverage of original input
tokens, which implies that the stronger retrieval ability may come from the
compression of input lexicon terms into dense representations.

Building upon this observation, we propose a new pre-training schema, the Bag-
of-Word prediction task, for directly compressing the lexicon terms into dense
representations. As is shown in Figure 1 B), it first projects dense
representations to the vocabulary space, then directly predicts the input
token bags with multi-label cross-entropy loss. As a result, the lexical
information of input tokens is directly compressed into the dense
representations, which will give superior initialization ability to the PLMs
for downstream retrieval tasks. This method is a straightforward replacement
for existing MAE-style pre-training paradigms. There are several benefits to
utilizing a Bag-of-Word prediction task, instead of previous MAE-style pre-
training.

$\bullet$ Zero computational complexity. This method requires NO extra
Transformers-based decoder layers, GPU memory consumption, data preprocessing,
or extensive data masking. It is easy to implement and greatly reduces the
additional computational cost of the pre-training.

$\bullet$ High interpretability. Compared to complex modifications to the
decoder tasks (Liu and Shao, 2022; Wang et al., 2022; Wu et al., 2023; Zhou et
al., 2022) of MAE-style pre-training, our method removes the need for adding
decoders and directly optimizes a multi-label prediction task of input
lexicons.

To verify the effectiveness of our method, we conduct experiments on several
large-scale web search and open-domain QA datasets, including MS-MARCO passage
ranking (Nguyen et al., 2016), Natural Questions (NQ) (Kwiatkowski et al.,
2019), TriviaQA (Joshi et al., 2017) and heterogeneous zero-shot BEIR
benchmarks (Thakur et al., 2021). Our proposed method achieves state-of-the-
art retrieval performance and provides a 67% training speed-up compared to
masked auto-encoder with enhanced decoding (Liu and Shao, 2022).

In summary, our contributions are as follows.

$\bullet$ We first reveal that masked auto-encoder (MAE) pre-training with
enhanced decoding significantly improves the term coverage of input tokens in
dense representations.

$\bullet$ Building upon this observation, we propose a direct Bag-of-Word
prediction task for compressing the information of input lexicon terms into
dense representations.

$\bullet$ Our method achieves considerable training speed-up without
additional computational complexity, and meanwhile keeping full
interpretability as well as state-of-the-art retrieval performances on
multiple retrieval benchmarks.

##  2\. Related Works

#### Passage Retrieval.

Passage retrieval aims to find relevant information from large passage
collections based on a given search query. Traditional techniques for ad-hoc
retrieval, like TF-IDF and BM25 (Robertson et al., 2009), perform relevant
searches based on lexicon terms and inverted document frequencies. Such
methods have efficient retrieval abilities, but they suffer from vocabulary
mismatch problems (Karpukhin et al., 2020). Although there are continuous
techniques like query expansions (Nogueira et al., 2019; Wang et al., 2023;
Jagerman et al., 2023) and query rewrite (Gollapudi et al., 2011) to alleviate
such mismatch issues, additional preprocess of queries or passage collections
make the retrieval pipelines much more complicated and hard to maintain.

Dense retrieval (Karpukhin et al., 2020; Qu et al., 2021) has become popular
in mainstream retrieval applications recently, because it maps the textual
information into learnable representations and naturally alleviates the
lexicon mismatch problems by learning the similarity traits of queries and
passages through end-to-end contrastive learning. Dense retrieval encodes
queries and passage collections as condensed low-dimension sentence
representations with pre-trained language models (PLMs), and performs
retrieval based on the similarity of representations with efficient search
algorithms like maximum inner-product search (MIPS) (Mussmann and Ermon,
2016).

#### Pre-training for Dense Retrieval.

There are continuous efforts (Gao and Callan, 2021, 2022; Ren et al., 2021; Lu
et al., 2022) to improve the retrieval quality of sentence representations
with specifically designed pre-training techniques, where masked auto-encoder
(MAE) (Gao and Callan, 2021; Lu et al., 2021; Gao and Callan, 2022; Liu and
Shao, 2022; Wu et al., 2023; Wang et al., 2022; Zhou et al., 2022) has gained
popularity to boost the initialization of PLMs for downstream retrieval tasks
in recent years. Masked auto-encoder (MAE) pre-training seeks to compress the
lexicon information of contextual texts into sentence representations through
various auxiliary pre-training tasks and tailored bottlenecked architectures.
It usually appends a shallow decoder to the PLM encoder and connects the
encoder-decoder with merely a sentence representation. Through the process of
MAE-style pre-training, the information of input texts is first condensed as
sentence representations and then decoded in the appended decoder with auto-
encoding (Gao and Callan, 2021, 2022; Liu and Shao, 2022; Wu et al., 2023;
Zhou et al., 2022), auto-regression (Lu et al., 2021) or enhanced decoding
(Liu and Shao, 2022). Such an information encoding-decoding process brings
sustainable supervision signals to the sentence representations and thus gives
better initialization to the PLMs for downstream retrievals.

Recent studies focus on designing various decoding tasks for MAE-based pre-
trainings. (Lu et al., 2021) proposes to pre-train an encoder with merely a
single-layer auto-regressive decoder (Radford et al., 2019) and apply
attention window restriction to enforce textual information compression to the
representations. (Gao and Callan, 2021, 2022) divides the MAE into early-late-
head layers and proposes to optimize with condensed representations and
structural readiness. Furthermore, (Liu and Shao, 2022; Liu et al., 2023)
proposes pre-training with an auto-encoding decoder and an extreme mask ratio.
They further propose an enhanced decoding paradigm (Yang et al., 2019) for
improving token usage ability in MAE-style pre-training. It applies sentence
representations with positional information as query streams and performs
cross-attention with masked context embeddings as content streams in the
decoder. In our studies below, we observe significantly increased term
coverage of input lexicon tokens by projecting the pre-trained sentence
representations of enhanced decoding to the vocabulary space. This implies
that the stronger retrieval ability may come from the mediate compression of
lexicon terms into dense representations.

Although the bloom of MAE-style pre-training gives better retrieval
performances to dense systems, several obvious drawbacks hinder the further
development of such pre-training techniques. Firstly, existing MAE pre-
trainings focus on creating carefully designed auxiliary pre-training tasks,
but it’s hard to interpret the underlying reason why such an auxiliary pre-
training task is effective. Furthermore, MAE-style pre-training mandatory
needs the additional Transformers-based decoders, which brings considerable
GPU memory cost and additional $O(n^{2})$ computational complexity. Complex
data preprocessing (Wu et al., 2022; Zhou et al., 2022) or extensive data
masking (Liu and Shao, 2022; Wu et al., 2023) are also needed in several
implementations. In contrast, our method breaks the constraints of using any
additional decoders, preprocessing, or masking, by directly optimizing a Bag-
of-Word decoding task. Such a method achieves state-of-the-art retrieval
performances on various datasets. It is highly interpretable and has nearly
zero additional computational complexity in the whole pre-training process.

| Eval | MS-MARCO
---|---|---
Model |  MLM Loss $\downarrow$ | MRR@10 $\uparrow$ | R@1 $\uparrow$ | R@50 $\uparrow$ | R@1k $\uparrow$
BERT | 2.9389 | 36.05 | 23.55 | 83.48 | 96.93
Further Pre-train | 1.8863 | 37.33 $\dagger$ | 24.90 $\dagger$ | 84.81 $\dagger$ | 97.66 $\dagger$
Auto-Encoding | 1.9499 | 37.72 | 25.06 | 86.16 | 98.02
Auto-Regression | 1.9765 | 37.06 | 24.71 | 84.76 | 97.56
Enhanced Decoding | 2.0027 |  38.12 * | 25.19 |  86.49 * |  98.22 *

Table 1.  MLM Loss and retrieval performances of MAE-style pre-training models
on the MS-MARCO passage ranking dataset. Better results are marked as bold.
$\dagger$ means significant improvement over “BERT” baselines. * means
significant improvement over “Further Pre-train” settings. ($p\leq 0.05$)

| Eval | NQ | TriviaQA
---|---|---|---
Model |  MLM Loss $\downarrow$ | R@20 $\uparrow$ | R@100 $\uparrow$ | R@20 $\uparrow$ | R@100 $\uparrow$
BERT | 2.9498 | 83.13 | 88.78 | 82.21 | 86.83
Further Pre-train | 1.7682 | 83.99 $\dagger$ | 89.67 $\dagger$ | 83.79 $\dagger$ | 87.62 $\dagger$
Auto-Encoding | 1.8320 |  84.52 * | 90.00 | 84.02 | 87.61
Auto-Regression | 1.8609 | 84.40 * | 90.06 |  84.63 * | 87.93
Enhanced Decoding | 1.8884 | 84.46 * |  90.30 * | 84.51 * | 88.08

Table 2.  MLM Loss and retrieval performances of MAE-style pre-training models
on Natural Questions (NQ) (Kwiatkowski et al., 2019), TriviaQA (Joshi et al.,
2017) datasets.

##  3\. Method

In this section, we first introduce the preliminary of dense retrieval and the
drawbacks of existing masked auto-encoder pre-training. Then we analyze the
interpretability of MAE pre-training in detail. Next, we introduce a new Bag-
of-Word pre-training schema for efficient dense retrieval pre-training.

###  3.1. Preliminary of Dense Retrieval Pre-training

Dense retrieval utilizes an encoder $Enc$, e.g., BERT (Devlin et al., 2019),
to generate the sentence representations $v_{q}$ and $v_{p}$ of a query $q$
and a passage $p$, respectively. Typically, the representations are derived
from the last hidden states at the [CLS] token position, i.e.,
$h_{last}^{CLS}$. Then the similarity $Sim(q,p)$ of $v_{q}$ and $v_{p}$ is
calculated with dot product or cosine similarity.

(1) $Sim(q,p)=Enc(q)\cdot Enc(p)=v_{q}\cdot v_{p}$

Masked auto-encoder (MAE) pre-training is an effective schema to bring
sustainable supervision signals to sentence representations of the encoder and
boost the retrieval performance for subsequent fine-tuning. Given a tokenized
text $t\in T$ from training corpus collections, MAE pre-training first selects
a proportion of tokens, with a pre-defined mask ratio of $r_{Enc}$, and
replaces each with a mask token [m].

(2) $mask(t)=\\{[CLS],t_{1},t_{2},[m],t_{4},...,t_{n},[SEP]\\}$

Then the masked texts $mask(t)$ are forwarded through the encoder $Enc$ to
optimize a Masked Language Model (MLM) task loss $\mathcal{L}_{enc}$ with
Cross-Entropy (CE) loss.

(3) $\displaystyle\mathcal{L}_{Enc}$
$\displaystyle=CE(Enc(mask({t})),t_{i}),i\in M$ $\displaystyle=-\sum_{t\in
T}\sum_{i\in{M}}\log p(t_{i}|Enc(mask({t})))$

where $M$ denotes the set of masked tokens.

Decoder designs are key innovations of MAE-style pre-training. There are
mainly three categories of decoder designs, including auto-encoding (Gao and
Callan, 2021; Liu and Shao, 2022), auto-regression (Lu et al., 2021), and
enhanced decoding (Liu and Shao, 2022).

Algorithm 1 Input Token Coverage Ratio

0: Projected Representation $h_{proj}$, Tokenized Inputs $T$

1: $T_{topk}\leftarrow$ Top-$k$ dominating tokens from $h_{proj}$

2: $\mathrm{r}_{dominate}\leftarrow$ Numbers of $(T_{topk}\cap T)$ / Numbers
of $T$

2: Input Token Coverage Ratio $\mathrm{r}_{dominate}$

Figure 2.  Compositions of Top-k tokens of dense representation.  Figure 3.
Examples of the compositions of Top-20 tokens of dense representation. We
encode the input texts with various encoders and project them to the
vocabulary space to interpret dominating tokens. ✓ means a token hits the
input text, while ✗ means miss of the input text.

#### Auto-Encoding Decoder

Given the text $t_{Dec}$ as a decoder input, the auto-encoding decoder first
masks it with a large mask ratio $r_{Dec}(r_{Dec}>r_{Enc})$, and replaces the
first token with the encoder representations.

(4) $mask(t_{Dec})=\\{h_{last}^{CLS},t_{1},t_{2},[m],t_{4},...,t_{n},[SEP]\\}$

Then the decoder also applies the MLM loss as reconstruction signals and
optimizes the encoder representation through the gradient flow of the
bottleneck-connected encoder-decoder architecture.

(5) $\mathcal{L}_{Dec}=CE(Dec(mask(t_{Dec})),t_{i}),i\in M$

#### Auto-Regression Decoder

Given the decoder input text $t_{Dec}$, it also replaces the first token with
the encoder representations.

(6) $t_{Dec}=\\{h_{last}^{CLS},t_{1},t_{2},t_{3},t_{4},...,t_{n},[SEP]\\}$

Then Casual Language Model (CLM) loss is used as the generation signal for the
decoder, implemented with the cross-entropy (CE) loss. The encoder
representation will be optimized via the gradient flow of decoder generation
from the decoder to the encoder.

(7) $\displaystyle\mathcal{L}_{Dec}$
$\displaystyle=CE(Dec(t_{Dec}[:i-1]),t_{i}),i\in(0,n]$
$\displaystyle=-\sum_{t_{i}\in{T}_{Dec}}\log p(t_{i}|Dec(t_{Dec}[:i-1]))$

where $t_{Dec}[:i-1]$ denotes all the tokens before $t_{i}$.

#### Enhanced Decoding Decoder

Unlike self-attention schemas with auto-encoding or auto-regression, enhanced
decoding (Liu and Shao, 2022; Yang et al., 2019) develops a cross-attention
pattern with an extreme masked ratio for improving token usage. The decoder
inputs $t_{Dec}$ are first forwarded through the embedding layer and converted
to input embeddings $h_{Dec}^{Emb}$. Then enhanced decoding utilizes encoder
representations $h_{last}^{CLS}$ with positional embeddings $p$ as the query
stream $q$, and input embeddings $h_{Dec}^{Emb}$ as the content stream $k$ and
$v$.

(8) $\displaystyle q=h_{last}^{CLS}+p$ $\displaystyle k=v=h_{Dec}^{Emb}$

Afterward, a cross-attention is performed with the streams $q$, $k$, and $v$,
then forwarded through feed-forward network (FFN) chunks with post-layer norm
(LN).

(9) $\displaystyle A=softmax(q^{T}k/\sqrt{d}+M)\cdot v$ $\displaystyle
h_{Dec}=LN(FFN(A)+A)$

Specifically, a position-specific attention mask $M_{ij}\in[L,L]$ ($L$ is the
sequence length) is applied to the cross-attention to increase the difficulty
of the decoder optimization objective. This mask $M_{ij}$ is composed of
masking diagnose positions $i=j$ and a randomly sampled token mask $i,j\in
sample([0...L])$. After projecting the decoder’s hidden states $h_{Dec}$ with
BERT language-model heads (i.e., its embedding matrix $E^{T}$), a cross-
entropy loss is used with all corresponding input tokens as labels.

(10) $\displaystyle\mathcal{L}_{Dec}=CE(h_{Dec}E^{T},t_{i}),i\in[0,n]$

The above three decoder designs are popular schemas for masked auto-encoder
(MAE) pre-training of dense retrieval. They provide rich supervision signals
to the encoder representation and enable better initialization for downstream
retrieval tasks. However, MAE pre-training lacks interpretability for its
effectiveness and brings additional $O(n^{2})$ complexity because of the
mandatory need for extra Transformers-based decoder layers. These drawbacks
hinder the further development of the existing MAE-style pre-training.

Algorithm 2 Bag-of-word Prediction

0: Tokenized Inputs $T$

1: $h_{last}^{CLS}\leftarrow Enc(mask(T))$ $\triangleright$ Encoder Forward

2: $h_{proj}\leftarrow h_{last}^{CLS}\cdot E^{T}$ $\triangleright$ Projection

3: $\mathcal{L}_{Dec}\leftarrow CrossEntropy(h_{proj},T)$ $\triangleright$
Equation 12

3: Bag-of-word Prediction Loss $\mathcal{L}_{Dec}$

###  3.2. Interprete MAE Pre-training

#### Adaption Abilities and Retrieval Performances

The MLM loss on target corpora reflects the modeling abilities of various pre-
trained checkpoints. To evaluate the domain adaption abilities of the PLMs on
the target corpus, we randomly sample 10,000 corpora from MS-MARCO (Nguyen et
al., 2016) and DPR-Wiki (Karpukhin et al., 2020) collections and calculate the
corresponding average MLM loss.

Here we involve 5 models for performance comparison. 1) Original BERT
checkpoint (Devlin et al., 2019); 2) The BERT further pre-trained with MLM on
the original training corpus, i.e., Wikipedia and Bookcorpus collections; 3)
MAE Pre-training with Auto-Encoding Decoder; 4) MAE Pre-training with Auto-
Regression Decoder; 5) MAE Pre-training with Enhanced Decoding Decoder. For
fair comparisons, we mainly reproduce the pre-training settings in (Liu and
Shao, 2022) and directly take the enhanced decoding checkpoint from (Liu and
Shao, 2022). The MLM loss on evaluation sets of MS-MARCO, DPR-Wiki, and the
corresponding retrieval performances are listed in Table 1 and 2.

Here we can observe some interesting phenomena. 1) Both further pre-training
with pure MLM loss and MAE-style pre-training significantly lower the MLM loss
on the target evaluation corpus, and improve the downstream retrieval
performances. 2) Although MAE-style pre-training has higher MLM loss than the
further pre-trained checkpoint, they tend to achieve higher retrieval
performances. Thus we can conclude that pre-training for dense retrieval is
effective for better domain adaption abilities and retrieval performances, but
such abilities are not always positively correlated to lower MLM losses on
target corpora when using MAE-style pre-training. Thus, other critical traits
should contribute to better retrieval abilities for MAE-style pre-training.

#### Token Compositions of Dense Representation

MAE-style pre-training optimizes the sentence representations through
bottleneck-connected encoder-decoder architecture. Although dense
representations are typically highly feature-condensed and hard to interpret
as human-readable format, inspired by (Ram et al., 2023), we can still try to
visualize the dominating tokens contained in dense representations by
projecting with original BERT language model heads (LM-head). To inspect the
changes in sentence representations after MAE-style pre-training, we project
the representations at the [CLS] token positions $h_{last}^{CLS}$ to the
vocabulary space $h_{proj}\in vocab\\_size$ and analyze the composition of
top-$k$ tokens.

(11) $h_{proj}=h_{last}^{CLS}\cdot E^{T}$

Specifically, as is demonstrated in Algorithm 1, we first take the top-$k$
dominating tokens from $h_{proj}$ and calculate the ratio of the above tokens
appearing in the tokenized input token set $T=\\{t_{0},t_{1},..,t_{n}\\}$.
This ratio is called the input token coverage ratio. The statistics of the
above pre-training checkpoints are plotted in Figure 2. Compared to the
original BERT checkpoint, auto-encoding and further pre-training do not
contribute to the input token coverage ratio. However, MAE-style pre-training
with enhanced decoding significantly enhances the dominating token coverage of
input texts. Our interpretation pipeline and an example of all pre-training
schemas are displayed in Figure 3. Here we can also observe higher input token
coverage of pre-training with enhanced decoding, i.e. more hits ✓ on the
inputs, compared to all other pre-training methods. Considering the highest
retrieval performances of enhanced decoding, which are shown in Table 1 and 2,
we hypothesize that the increased retrieval abilities may come from enhanced
dominating token coverage of input texts.

###  3.3. Pre-training with Bag-of-Word Prediction

Building upon the above observations about token compositions and domain
adaption abilities, here we propose a new pre-training schema that explicitly
removes the mandatory needed decoder in existing MAE-style pre-training. As is
shown in Algorithm 2, the sentence representations at the [CLS] token position
are first projected to the vocabulary space with the BERT language-model heads
(i.e., its embedding matrix $E^{T}$). Then we apply a multi-label cross-
entropy loss to predict the input token set $T$ directly. Bag-of-Word means
the label $T$ is a set of tokenized input text. For simplicity, it is not
related to positional information, similar to the traits of sparse retrieval
(Robertson et al., 2009).

(12) $\displaystyle\mathcal{L}_{Dec}=CE(h_{last}^{CLS}E^{T},T)$

Then the final objective function of our method comprises both the encoder MLM
loss and the Bag-of-Word prediction loss.

(13) $\displaystyle\mathcal{L}=\mathcal{L}_{Enc}+\mathcal{L}_{Dec}$

There are several benefits of the design behind our method. First of all, Bag-
of-Word prediction is a direct compression of input lexicon information into
dense representations with extreme simplicity, which merely utilizes the
existing BERT language model head for projection. Secondly, it removes the
need for extra Transformers-based decoder layers, data preprocessing, or
extensive data masking, which introduces ZERO extra computational complexity,
compared to other expensive MAE-style pre-training. Moreover, different from
previous MAE-style pre-training modifications, which mostly focus on extensive
modifications to the decoder tasks (Liu and Shao, 2022; Wang et al., 2022; Wu
et al., 2023; Zhou et al., 2022), Bag-of-Word pre-training only involves a
simple lexicon prediction task, which bridges the condensed representations
and lexicon traits. Such a pre-training schema is highly interpretable and
easy to implement.

| Additional | MS-MARCO | Natural Question | Trivia QA
---|---|---|---|---
Model | Decoder | MRR@10 | R@50 | R@1k | R@5 | R@20 | R@100 | R@5 | R@20 | R@100
BM25 (Robertson et al., 2009) |  | 18.7 | 59.2 | 85.7 | - | 59.1 | 73.7 | - | 66.9 | 76.7
SPLADE (Formal et al., 2021) |  | 32.2 | - | 95.5 | - | - | - | - | - | -
DPR (Karpukhin et al., 2020) |  | 31.7 | 58.0 | 85.7 | - | 74.4 | 85.3 | - | 79.3 | 84.9
ANCE (Xiong et al., 2020) |  | 33.0 | - | 95.9 | - | 81.9 | 87.5 | - | 80.3 | 85.3
SEED (Lu et al., 2021) | $\checkmark$ | 33.9 | - | 96.1 | - | 83.1 | 88.7 | - | - | -
Condenser (Gao and Callan, 2021) | $\checkmark$ | 36.6 | 85.2 | 97.4 | 73.5 | 83.2 | 88.4 | 74.3 | 81.9 | 86.2
RocketQA (Qu et al., 2021) |  | 37.0 | - | 97.9 | 74.0 | 82.7 | 88.5 | - | - | -
coCondenser (Gao and Callan, 2022) | $\checkmark$ | 38.2 | 87.3 | 98.4 | 75.8 | 84.3 | 89.0 | 76.8 | 83.2 | 87.3
SimLM (Wang et al., 2022) | $\checkmark$ | 39.1 | 87.3 | 98.6 | 73.6 | 84.3 | 89.0 | 78.1 | 83.8 | 87.6
RetroMAE (Liu and Shao, 2022) | $\checkmark$ | 39.3 | 87.0 | 98.5 | 74.4 | 84.4 | 89.4 | 78.9 | 84.5 | 88.0
BoW Prediction |  | 40.1* | 88.7* | 98.9 | 75.3 | 84.6 | 90.4* | 79.4* | 84.9* | 88.0
Table 3.  Results of Bag-of-Word (BoW) Prediction on MS-MARCO passage ranking,
Natural Question, and Trivia QA tasks. The best scores are marked in bold. *
means significant improvement over the best results of previous works. ($p\leq
0.05$)

|  | Data Process | Additional Decoder | Training Speed
---|---|---|---|---
Pre-train Task | Architecture | Complexity |  Time(s) $\downarrow$ | Complexity |  GPU Time(s) $\downarrow$ |  Sample per second $\uparrow$ |  Degeneration $\downarrow$
Further Pre-train | Encoder | $O(n)$ | 0.0476 | - | - | 269.708 | -
Auto-Encoding (Gao and Callan, 2021, 2022; Liu and Shao, 2022; Wu et al., 2023) | Encoder-Decoder | $O(n)$ | 0.0940 | $O(n^{2})$ | 0.0013 | 222.658 | 17.4%
Auto-Regression (Lu et al., 2021) | Encoder-Decoder | $O(n)$ | 0.0636 | $O(n^{2})$ | 0.0030 | 215.136 | 20.2%
Enhanced Decoding (Liu and Shao, 2022) | Encoder-Decoder | $O(n^{2})$ | 5.6261 | $O(n^{2})$ | 0.0012 | 85.797 | 68.2%
BoW Prediction | Encoder | $O(n)$ | 0.0533 | - | 0.0002 | 266.359 | 1.2%

Table 4.  Comparisons of architecture, data process, additional decoder, and
training speed of different pre-training methods. Note that Bag-of-Word (BoW)
prediction does not involve additional decoders, thus we report the GPU
computation time of the prediction task. The original implementation of
enhanced decoding (Liu and Shao, 2022) randomly generates 256 attention masks
for each input text, and picks one as the final mask used in the pre-training.
And thus it has $O(n^{2})$ data process complexity.

##  4\. Experiments

This section introduces the experiment settings and performances of Bag-of-
Word prediction.

###  4.1. Experiment Settings

#### Pre-training

Wikipedia (Foundation, [n. d.]) and Bookcorpus (Zhu et al., 2015), the
original BERT training sets, are used as pre-training corpora, which are
commonly adopted in previous retrieval pre-trainings (Gao and Callan, 2021;
Liu and Shao, 2022; Lu et al., 2021). The BERT-base-uncased checkpoint is used
to initialize the encoder for saving computation. The model is pre-trained
using the AdamW optimizer for 20 epochs, with a learning rate of 3e-4, a batch
size of 2048, and a maximum sequence length of 512. Following previous in-
domain pre-training settings (Gao and Callan, 2021; Liu and Shao, 2022; Lu et
al., 2021; Wang et al., 2022), MS-MARCO corpus (Nguyen et al., 2016) is also
used when evaluating the MS-MARCO benchmark. All pre-training hyper-parameters
are kept the same, except the maximum sequence length is cut to 144.

#### Evaluation

We conduct supervised evaluations on several large-scale web search and open-
domain QA datasets, including MS-MARCO passage ranking (Nguyen et al., 2016),
Natural Questions (NQ) (Kwiatkowski et al., 2019) and TriviaQA (Joshi et al.,
2017). The metrics of Mean Reciprocal Rank (MRR@k) and Recall (R@k) are
reported, following (Gao and Callan, 2022). We fine-tune MS-MARCO with a batch
size of 64, a learning rate of 2e-5, and a passage group size of 16 for 3
epochs. Following (Karpukhin et al., 2020), we utilize the DPR-Wiki versions
of NQ and TriviaQA benchmarks and evaluate retrieval performances. We fine-
tuned NQ and TriviaQA with the same hyper-parameters, except that the learning
rate is 5e-6, and the number of training epochs is 20. Retrievers are fine-
tuned with ANCE manners (Xiong et al., 2020) by following (Gao and Callan,
2021, 2022; Liu and Shao, 2022). Following (Gao and Callan, 2021; Liu and
Shao, 2022), both BM25 negatives and retriever-mined negatives are used in
fine-tuning stages for fair comparisons. A large-scale heterogeneous zero-shot
benchmark BEIR (Thakur et al., 2021) is also used to evaluate the out-of-
domain transferability of our method. Normalized Discounted Cumulative Gain
(NDCG@10) is used as the metric for BEIR.

| MS-MARCO | Natural Question | Trivia QA
---|---|---|---
Model | MRR@10 | R@50 | R@1k | R@5 | R@20 | R@100 | R@5 | R@20 | R@100
Fine-tuned with BM25 Negatives
BERT | 33.7 | 80.9 | 96.4 | 68.2 | 79.8 | 87.0 | 70.9 | 80.2 | 85.7
Further Pre-train | 37.2 | 86.0 | 98.2 | 71.4 | 81.9 | 88.2 | 75.1 | 82.3 | 86.8
Auto-Encoding | 37.6 | 86.3 | 98.5 | 71.6 | 82.5 | 88.8 | 76.4 | 83.2 | 87.2
Auto-Regression | 38.0 | 86.9 | 98.7 | 71.4 | 82.8 | 88.8 | 76.7 | 83.6 | 87.4
Enhanced Decoding | 37.0 | 86.7 | 98.4 | 72.3 | 83.0 | 88.9 | 77.2 | 83.5 | 87.4
BoW Prediction | 38.4 | 87.5 | 98.7 | 72.3 | 82.9 | 89.3 | 77.7 | 83.9 | 87.6
Fine-tuned with Mined Negatives
BERT | 36.1 | 83.5 | 96.9 | 72.5 | 83.1 | 88.8 | 75.0 | 82.2 | 86.8
Further Pre-train | 39.2 | 87.7 | 98.5 | 74.0 | 84.0 | 89.7 | 77.4 | 83.8 | 87.6
Auto-Encoding | 39.3 | 87.9 | 98.7 | 74.9 | 84.5 | 90.0 | 77.5 | 84.0 | 87.6
Auto-Regression | 39.7 | 88.3 | 98.9 | 74.8 | 84.4 | 90.1 | 78.8 | 84.6 | 87.9
Enhanced Decoding | 39.3 | 87.0 | 98.6 | 74.4 | 84.5 | 90.3 | 79.0 | 84.5 | 88.1
BoW Prediction | 40.1 | 88.7 | 98.9 | 75.3 | 84.6 | 90.4 | 79.4 | 84.9 | 88.0
Table 5.  Retrieval performances of different pre-training schema with BM25
negatives and mined negatives.

#### Baselines

We include multiple public sparse and dense retrieval baselines for
comparisons. Sparse retrieval baselines include BM25 (Robertson et al., 2009)
and SPLADE (Formal et al., 2021). Dense retrieval baselines are composed of
multiple well-known pre-training designs, including MAE-style pre-training
SEED (Lu et al., 2021), Condenser (Gao and Callan, 2021), coCondenser (Gao and
Callan, 2022), SimLM (Wang et al., 2022), RetroMAE (Liu and Shao, 2022), and
other dense retrieval modifications, like DPR (Karpukhin et al., 2020), ANCE
(Xiong et al., 2020), RocketQA (Qu et al., 2021). Particularly, we reproduce
multiple MAE-style pre-training methods and compare with their performances
and training efficiencies, including Auto-Encoding (Liu and Shao, 2022), Auto-
Regression (Lu et al., 2021), and Enhanced Decoding (Liu and Shao, 2022).

Other data augmentation techniques, such as context-based (Wu et al., 2023),
expansion-based (Wu et al., 2022), mutual information-based (Li et al., 2023)
or multi-task (Zhou et al., 2022) pre-training, are not included here for fair
comparisons. Our proposed Bag-of-Word prediction is orthogonal to them, and we
will leave the combinations of our method with various augmentation techniques
to further work.

Figure 4.  Input token coverage about Top-k tokens of dense representation
after Bag-of-Word prediction pre-training.  Figure 5.  Examples of the
compositions of Top-20 tokens of dense representation after pre-training with
Bag-of-Word prediction.

###  4.2. Main Results

#### Retrieval Performances

Pre-training with Bag-of-Word prediction achieves state-of-the-art retrieval
performances on several large-scale retrieval benchmarks. As is displayed in
Table 3, our method outperforms the strongest baseline by +0.8% (MRR@10) and
+1.7% (R@50) on MS-MARCO, +1.0% (R@100) on NQ and +0.5% (R@5 and R@20) on
TriviaQA. Note that compared with the existing MAE-style pre-training, our
method explicitly removes the mandatory need for the additional decoder and
outperforms several MAE retrieval baselines. The performance improvements
completely come from the incorporation of the Bag-of-Word pre-training task,
other than data augmentation or task ensembles. Our method works with extreme
simplicity and is robust to various retrieval domains.

#### Pre-training Efficiency

Bag-of-Word prediction pre-trains dense retrieval models with higher speed and
fewer CPU/GPU complexities. As is shown in Table 4, compared with MAE-based
encoder-decoder structures, our method removes the need for the additional
decoder, which saves ¿ 83% GPU computation time on the decoding side.
Additionally, our method does not involve extensive masking strategies, which
saves 99% CPU time for the data process compared to enhanced decoding.
Overall, the Bag-of-Word prediction has a very slight training speed
degeneration and achieves 67%, 19%, and 16.2% training speedup compared with
enhanced decoding, auto-regression, and auto-encoding.

|  | Encoder Mask Ratios |
---|---|---|---
Datasets | Baseline | 15% | 30% | 45% | wo/ BoW
TREC-COVID | 65.69 | 68.82 | 59.40 | 64.60 | 67.01
BioASQ | 27.51 | 39.44 | 40.06 | 39.43 | 34.99
NFCorpus | 27.41 | 31.19 | 31.45 | 30.85 | 29.41
NQ | 46.75 | 50.52 | 48.68 | 48.18 | 47.59
HotpotQA | 48.28 | 64.10 | 63.64 | 62.47 | 55.25
FiQA-2018 | 24.43 | 30.87 | 30.77 | 30.48 | 28.31
Signal-1M | 18.86 | 28.07 | 29.33 | 28.97 | 26.71
TREC-NEWS | 36.61 | 41.10 | 41.17 | 40.20 | 40.01
Robust04 | 37.51 | 44.14 | 44.46 | 43.67 | 42.04
ArguAna | 33.33 | 41.93 | 44.94 | 44.29 | 49.63
Touche-2020 | 25.55 | 21.22 | 17.18 | 18.39 | 20.59
CQADupStack | 27.28 | 32.74 | 32.96 | 32.59 | 31.38
Quota | 78.43 | 85.85 | 85.75 | 85.84 | 86.29
DBPedia | 31.95 | 38.72 | 38.27 | 38.84 | 36.27
SCIDOCS | 12.14 | 15.92 | 15.70 | 15.55 | 14.36
FEVER | 69.86 | 74.61 | 72.27 | 72.75 | 70.72
Climate-FEVER | 20.56 | 24.01 | 25.78 | 25.56 | 24.39
SciFact | 50.66 | 64.63 | 65.52 | 64.84 | 58.81
Average | 37.93 | 44.33 | 43.74 | 43.75 | 42.43
Relative Gains | - | +6.39 | +5.81 | +5.81 | +4.50

Table 6.  Ablation study on the heterogeneous BEIR benchmark for our Bag-of-
Word prediction pre-training method, with various encoder mask ratios or
removing the proposed Bag-of-Word prediction task.

##  5\. Ablation Studies

###  5.1. Performances with Different Negatives

We compare the retrieval performances of different pre-training schemas and
report the results with different negative training data in Table 5. The
retrievers are trained with BM25 negatives and retriever-mined negatives,
respectively, by following previous supervised fine-tuning pipelines (Xiong et
al., 2020; Gao and Callan, 2021). It can be seen that even with different
negatives, Bag-of-Word prediction pre-training surpasses all MAE-style pre-
training, MLM further pre-training, and BERT baselines, under the same pre-
training and fine-tuning pipelines. That further verifies our pre-training
method is simple but more effective than other MAE pre-training schemas.

###  5.2. Input Token Coverage

As is presented in Section 3.2, we visualize the input token coverage about
top-k tokens by projecting dense representations to the vocabulary space via
the BERT language model heads. As is shown in Figure 4, Bag-of-Word has a
higher input token coverage ratio than other pre-training methods. It implies
that Bag-of-Word prediction has higher compression efficiency of lexicon
information than MAE-style pre-training with its direct prediction task. The
higher retrieval performances are also attributed to the direct compression of
the input token into dense representations. Examples in Figure 5 further
verify that Bag-of-Word prediction enables dense representation to give higher
frequencies to input tokens, especially representative tokens, thus achieving
higher retrieval performances.

###  5.3. Transferability and Encoder Mask Ratios

Previous study (Liu and Shao, 2022; Wu et al., 2023) reported that properly
increasing the mask ratio for the encoder can lead to higher retrieval
performances. Hence we perform pre-training with Bag-of-Word prediction under
different encoder mask ratios and report the performances on the heterogeneous
BEIR benchmarks. BEIR benchmarks include 18 out-of-domain retrieval datasets
across different domains, sizes, and topics, which is suitable for evaluating
the transferability of retriever models. As is shown in Table 6, pre-training
with Bag-of-Word prediction achieves relatively +6.39% performance gains over
the BERT baseline. On the contrary, removing the Bag-of-Word prediction task
and applying further pre-training with pure MLM loss will incur +1.89% (6.39%
$\rightarrow$ 4.50%) performance degeneration. Here we also observe that
increasing the encoder mask ratio from 15% to 30% or 45% does not yield higher
retrieval performances.

##  6\. Conclusion

Masked auto-encoder pre-training has become a prevalent technique for
initializing and enhancing dense retrieval systems. However, the common
utilization of additional Transformers-based decoders incurs significant
computational costs. And the underlying reasons for its effectiveness are not
well explored. In this paper, we first reveal that MAE pre-training with
enhanced decoding significantly increases the input lexicon coverage, via
visualizing the dense representations. Then we propose a direct modification
to traditional MAE by replacing the decoder with merely a Bag-of-Word
prediction task. The Bag-of-Word prediction task is extremely simple but
effective for the pre-training of dense retrieval systems. Our method achieves
state-of-the-art retrieval performances without requiring any additional
Transformers decoders, data preprocessing, or extensive data masking
strategies. It is also a direct compression of lexicon information to dense
representations, which is highly interpretable and easy to implement.

## Limitations

In this study, we explore the direct application of Bag-of-Word prediction in
dense retrieval pre-training. There are also effective modifications about
data augmentation or task-ensembling techniques, such as context sampling (Wu
et al., 2023), document expansion (Nogueira et al., 2019; Wu et al., 2023),
mutual information masking (Li et al., 2023) and multi-task (Zhou et al.,
2022). Bag-of-Word prediction focuses on pre-training paradigm innovation,
which is orthogonal to them. Combinations of Bag-of-Word prediction with those
data augmentation or task-ensembling techniques will be likely to further
boost the downstream retrieval performances, and we leave this to our future
work.

## References

  * (1)
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_. Association for Computational Linguistics, Minneapolis, Minnesota, 4171–4186.  https://doi.org/10.18653/v1/N19-1423
  * Formal et al. (2021) Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. 2021.  SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking. In _SIGIR ’21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021_ , Fernando Diaz, Chirag Shah, Torsten Suel, Pablo Castells, Rosie Jones, and Tetsuya Sakai (Eds.). ACM, 2288–2292.  https://doi.org/10.1145/3404835.3463098
  * Foundation ([n. d.]) Wikimedia Foundation. [n. d.].  _Wikimedia Downloads_.  https://dumps.wikimedia.org
  * Gao and Callan (2021) Luyu Gao and Jamie Callan. 2021.  Condenser: a Pre-training Architecture for Dense Retrieval. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 981–993.  https://doi.org/10.18653/v1/2021.emnlp-main.75
  * Gao and Callan (2022) Luyu Gao and Jamie Callan. 2022.  Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrieval. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, Dublin, Ireland, 2843–2853.  https://doi.org/10.18653/v1/2022.acl-long.203
  * Gollapudi et al. (2011) Sreenivas Gollapudi, Samuel Ieong, Alexandros Ntoulas, and Stelios Paparizos. 2011.  Efficient query rewrite for structured web queries. In _Proceedings of the 20th ACM Conference on Information and Knowledge Management, CIKM 2011, Glasgow, United Kingdom, October 24-28, 2011_ , Craig Macdonald, Iadh Ounis, and Ian Ruthven (Eds.). ACM, 2417–2420.  https://doi.org/10.1145/2063576.2063981
  * Jagerman et al. (2023) Rolf Jagerman, Honglei Zhuang, Zhen Qin, Xuanhui Wang, and Michael Bendersky. 2023.  Query Expansion by Prompting Large Language Models.  _CoRR_ abs/2305.03653 (2023).  https://doi.org/10.48550/arXiv.2305.03653 arXiv:2305.03653
  * Joshi et al. (2017) Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017.  TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers_ , Regina Barzilay and Min-Yen Kan (Eds.). Association for Computational Linguistics, 1601–1611.  https://doi.org/10.18653/V1/P17-1147
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020.  Dense Passage Retrieval for Open-Domain Question Answering. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Association for Computational Linguistics, Online, 6769–6781.  https://doi.org/10.18653/v1/2020.emnlp-main.550
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.  Natural Questions: a Benchmark for Question Answering Research.  _Trans. Assoc. Comput. Linguistics_ 7 (2019), 452–466.  https://doi.org/10.1162/TACL_A_00276
  * Li et al. (2023) Zehan Li, Yanzhao Zhang, Dingkun Long, and Pengjun Xie. 2023.  Challenging Decoder helps in Masked Auto-Encoder Pre-training for Dense Passage Retrieval.  _CoRR_ abs/2305.13197 (2023).  https://doi.org/10.48550/ARXIV.2305.13197 arXiv:2305.13197
  * Liu and Shao (2022) Zheng Liu and Yingxia Shao. 2022.  RetroMAE: Pre-training Retrieval-oriented Transformers via Masked Auto-Encoder.  _arXiv preprint arXiv:2205.12035_ (2022).
  * Liu et al. (2023) Zheng Liu, Shitao Xiao, Yingxia Shao, and Zhao Cao. 2023.  RetroMAE-2: Duplex Masked Auto-Encoder For Pre-Training Retrieval-Oriented Language Models. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023_ , Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, 2635–2648.  https://aclanthology.org/2023.acl-long.148
  * Lu et al. (2021) Shuqi Lu, Di He, Chenyan Xiong, Guolin Ke, Waleed Malik, Zhicheng Dou, Paul Bennett, Tie-Yan Liu, and Arnold Overwijk. 2021.  Less is More: Pretrain a Strong Siamese Encoder for Dense Text Retrieval Using a Weak Decoder. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_. 2780–2791.
  * Lu et al. (2022) Yuxiang Lu, Yiding Liu, Jiaxiang Liu, Yunsheng Shi, Zhengjie Huang, Shikun Feng Yu Sun, Hao Tian, Hua Wu, Shuaiqiang Wang, Dawei Yin, et al. 2022\.  Ernie-search: Bridging cross-encoder with dual-encoder via self on-the-fly distillation for dense passage retrieval.  _arXiv preprint arXiv:2205.09153_ (2022).
  * Malkov and Yashunin (2020) Yury A. Malkov and Dmitry A. Yashunin. 2020.  Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs.  _IEEE Trans. Pattern Anal. Mach. Intell._ 42, 4 (2020), 824–836.  https://doi.org/10.1109/TPAMI.2018.2889473
  * Mussmann and Ermon (2016) Stephen Mussmann and Stefano Ermon. 2016.  Learning and Inference via Maximum Inner Product Search. In _Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016_ _(JMLR Workshop and Conference Proceedings, Vol. 48)_ , Maria-Florina Balcan and Kilian Q. Weinberger (Eds.). JMLR.org, 2587–2596.  http://proceedings.mlr.press/v48/mussmann16.html
  * Nguyen et al. (2016) Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016.  MS MARCO: A Human Generated MAchine Reading COmprehension Dataset. In _Proceedings of the Workshop on Cognitive Computation: Integrating neural and symbolic approaches 2016 co-located with the 30th Annual Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain, December 9, 2016_ _(CEUR Workshop Proceedings, Vol. 1773)_ , Tarek Richard Besold, Antoine Bordes, Artur S. d’Avila Garcez, and Greg Wayne (Eds.). CEUR-WS.org.  https://ceur-ws.org/Vol-1773/CoCoNIPS_2016_paper9.pdf
  * Nogueira et al. (2019) Rodrigo Frassetto Nogueira, Wei Yang, Jimmy Lin, and Kyunghyun Cho. 2019.  Document Expansion by Query Prediction.  _CoRR_ abs/1904.08375 (2019).  arXiv:1904.08375 http://arxiv.org/abs/1904.08375
  * Qu et al. (2021) Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. 2021.  RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Association for Computational Linguistics, Online, 5835–5847.  https://doi.org/10.18653/v1/2021.naacl-main.466
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019\.  Language models are unsupervised multitask learners.  _OpenAI blog_ 1, 8 (2019), 9.
  * Ram et al. (2023) Ori Ram, Liat Bezalel, Adi Zicher, Yonatan Belinkov, Jonathan Berant, and Amir Globerson. 2023.  What Are You Token About? Dense Retrieval as Distributions Over the Vocabulary. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023_ , Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, 2481–2498.  https://doi.org/10.18653/V1/2023.ACL-LONG.140
  * Ren et al. (2021) Ruiyang Ren, Yingqi Qu, Jing Liu, Wayne Xin Zhao, QiaoQiao She, Hua Wu, Haifeng Wang, and Ji-Rong Wen. 2021.  RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 2825–2835.  https://doi.org/10.18653/v1/2021.emnlp-main.224
  * Robertson et al. (2009) Stephen Robertson, Hugo Zaragoza, et al. 2009\.  The probabilistic relevance framework: BM25 and beyond.  _Foundations and Trends® in Information Retrieval_ 3, 4 (2009), 333–389.
  * Thakur et al. (2021) Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021.  BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models.  _CoRR_ abs/2104.08663 (2021).  arXiv:2104.08663 https://arxiv.org/abs/2104.08663
  * Wang et al. (2022) Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2022.  SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrieval.  _CoRR_ abs/2207.02578 (2022).  https://doi.org/10.48550/arXiv.2207.02578 arXiv:2207.02578
  * Wang et al. (2023) Liang Wang, Nan Yang, and Furu Wei. 2023.  Query2doc: Query Expansion with Large Language Models.  _CoRR_ abs/2303.07678 (2023).  https://doi.org/10.48550/arXiv.2303.07678 arXiv:2303.07678
  * Wu et al. (2022) Xing Wu, Guangyuan Ma, and Songlin Hu. 2022.  Query-as-context Pre-training for Dense Passage Retrieval.  _CoRR_ abs/2212.09598 (2022).  https://doi.org/10.48550/arXiv.2212.09598 arXiv:2212.09598
  * Wu et al. (2023) Xing Wu, Guangyuan Ma, Meng Lin, Zijia Lin, Zhongyuan Wang, and Songlin Hu. 2023.  ConTextual Masked Auto-Encoder for Dense Passage Retrieval. In _Thirty-Seventh AAAI Conference on Artificial Intelligence, AAAI 2023, Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence, IAAI 2023, Thirteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2023, Washington, DC, USA, February 7-14, 2023_ , Brian Williams, Yiling Chen, and Jennifer Neville (Eds.). AAAI Press, 4738–4746.  https://ojs.aaai.org/index.php/AAAI/article/view/25598
  * Xiong et al. (2020) Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. 2020.  Approximate nearest neighbor negative contrastive learning for dense text retrieval.  _arXiv preprint arXiv:2007.00808_ (2020).
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019.  XLNet: Generalized Autoregressive Pretraining for Language Understanding. In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett (Eds.). 5754–5764.  https://proceedings.neurips.cc/paper/2019/hash/dc6a7e655d7e5840e66733e9ee67cc69-Abstract.html
  * Zhou et al. (2022) Kun Zhou, Xiao Liu, Yeyun Gong, Wayne Xin Zhao, Daxin Jiang, Nan Duan, and Ji-Rong Wen. 2022.  MASTER: Multi-task Pre-trained Bottlenecked Masked Autoencoders are Better Dense Retrievers.  _arXiv preprint arXiv:2212.07841_ (2022).
  * Zhu et al. (2015) Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015.  Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books. In _The IEEE International Conference on Computer Vision (ICCV)_.