# Asymptotic Persistence of Langmuir modes in Kinematically Complex Plasma
Flows

Ketevan Arabuli  Andria Rogava

###### Abstract

Dynamics of Langmuir modes - Langmuir waves (LW) and shear Langmuir vortices
(SLV) - in kinematically complex astrophysical plasma flows is studied. It is
found that they exhibit a number of peculiar, velocity shear induced,
asymptotically persistent phenomena: efficient energy exchange with the
background flow, various kinds of instabilities, leading to their exponential
growth; echoing solutions with persistent wave-vortex-wave conversions.
Remarkable similarity of these phenomena with ones happening with compressible
acoustic modes, revealed in (Mahajan and Rogava, 1999) is pointed out. The
relevance and possible importance of these phenomena for different types of
astrophysical plasma flow patterns with kinematic complexity is discussed. In
particular, we argue that these physical processes may account for the
persistent appearance of plasma oscillations in the heliosphere and in
interstellar plasma flows. In particular, we believe that kinematically
complex motion of plasma may naturally lead to asymptotically persistent
appearance of Langmuir modes born, grown, fed, sustained and maintained by
these flows.

00footnotetext: KU Leuven, Leuven, Belgium00footnotetext: Institute for
Theoretical Physics, Ilia State University, G. Tsereteli 3, 0162 Tbilisi,
Georgia

Keywords astrophysical plasmas; shear flows; plasma waves; heliosphere;
interstellar plasma

##  1 Introduction

It is widely known that both the wave and the vortical modes of motion are
common in many different kinds of astronomical objects. Since the visible
universe is predominantly made of plasma it is obvious that the study of both
the wave and the vortical motions in plasma flows is of key importance in
theoretical astrophysics. In a larger perspective, for several decades, vortex
dynamics is widely recognized as a fundamental iterdisciplinary scientific
problem (Saffman, 1993). Remarkably, there is a considerable degree of
similarity between the vortices in plasmas and neutral fluids (Petviashvili,
1980; Horton and Hasegawa, 1994). Usually vortical motions in plasmas are
associated with complicated, highly nonlinear systems. But with the gradual
disclosure of so called nonmodal phenomena in neutral fluid shear flows
(Kelvin, 1887; Goldreich and Lynden-Bell, 1965; Trefethen et al., 1993) it
became increasingly clear that even on the linear level there exists a number
of of interesting shear-induced effects associated with the vortex and wave
linear dynamics both in neutral fluid and plasma shear flows (Mahajan and
Rogava, 1999). These phenomena are characterized by intense energy exchange
between the wave and the vortical modes and the background flow.

The importance of velocity shear induced phenomena in astrophysics is widely
known, acknowledged and extensively discussed for several decades in a broad
variety of important classes of astrophysical flows. The incomplete list of
astronomical situations, where these phenomena have been considered includes:
density waves in galaxies (Goldreich and Lynden-Bell, 1965; Toomre, 1969;
Goldreich and Tremaine, 1978; Fan and qing Lou, 1997; Rogava et al., 1999;
Poedts and Rogava, 2002); transiently growing incompressible modes in
accretion discs (Lominadze et al., 1988; Balbus and Hawley, 1992; Tagger et
al., 1992); reciprocal transformations of solar magnetohydrodynamic (MHD)
waves and their role in the acceleration of the solar wind (Poedts et al.,
1998; Rogava et al., 2000). It was also found out that within relativistic
pulsar magnetospheric $e^{+}e^{-}$ plasmas transformation of non-escaping
longitudinal Langmuir modes into escaping electromagnetic waves (Mahajan et
al., 1997) may happen and contribute to the generation of pulsar radio
emission. Another large class of astrophysical flows with highly nontrivial
velocity inhomogeneity are the galactic and extragalactic jets (Lovelace et
al., 1987; Ferrari, 1998; Chen, 2021). Their kinematic complexity is caused
and ensured by complex spatial geometry of ordered magnetic fields,
responsible for the jet collimation and efficient acceleration of matter up to
relativistic velocities (Lovelace et al., 1991; Rogava and Khujadze, 1997).

As a rule spatial inhomogeneity of astrophysical flows is quite complicated,
with different components of the velocity field possessing different, often
nonlinear gradients. Sometimes, when considered physical processes have length
scales $l$ that are sufficiently small compared with the outward spatial
length-scales $L$, the flows may be assumed to have locally linear field-
profile geometries. It is well-known that the $l\ll L$ approximation is
elemental for the efficacy of commonly used Kelvin’s method (Kelvin, 1887) of
change of variables. However the validity of the approximation is often
questionable in astrophysical situations. In many cases of astrophysical
importance shear flows hardly ever are that simple.

The dazzling variety of flow patterns in the solar atmosphere and the
heliosphere, for example, is characterized by a notable degree of kinematic
complexity. (Mahajan and Rogava, 1999). Convective motions within the solar
photosphere, for instance, are characterized by highly nontrivial, non-
potential velocity fields coupled with the additional effect of the solar
differential rotation. Further up, according to Parker (1974), magnetic
filaments being rooted in the convective zone, with their footpoints
continuously squeezed by the convective granules, leads to kinematically
complex, variable plasma motion within these flux tubes (Rogava et al., 2000).
There is also an abundant and ever-increasing observational evidence (Pneuman
and Orral, 1980; Pike and Mason, 1998) for the presence of significantly
complex transient and/or oscillatory motions within coronal plasma loops,
spicules and solar tornadoes. These motions are superimposed with the motions
within horizontal fibrils and with downflowing streams, constituting ”complex
flow networks” (Athay, 1986) in the transition region. It was repeatedly
argued (Poedts et al., 1998; Mahajan and Rogava, 1999; Rogava et al., 2000)
that presence of these kinematically complex patterns of plasma flows, coupled
with the presence of magnetic fields, often governing these motions, should
heavily influence modes of collective behavior, generation and propagation of
waves and vortices in the solar atmosphere, possibly contributing to the
coronal heating, and to the acceleration of the solar wind.

Bearing in mind the necessity and importance of nonmodal studies of modes of
collective behavior in astrophysical shear flows with multidimensional
geometry and nontrivial kinematics (Mahajan and Rogava, 1999) developed a non-
asymptotic method apt to inspect the response of such flows to small-scale
perturbations. It allows to reduce the initial value problem to a set of
manageable ordinary differential equations in time. A similar method, designed
for flows with spatially uniform shearing rates, had been used in
hydrodynamics (Lagnado et al., 1984; Craik and Criminale, 1986). These studies
showed a remarkable richness in the spectrum of both incompressible and
compressible fluctuations sustained by these flow. The most impressive feature
of revealed exotic phenomena was their asymptotic persistence in the absence
of viscosity. The study revealed the presence of “Echoing”, strongly unstable
solutions, efficient energy exchange between the “parent” flow and the waves
and instabilities sustained by it, mode conversions and beat wave phenomena
(Mahajan and Rogava, 1999). Further application of this method to
kinematically complex electrostatic (Osmanov et al., 2015) and
magnetohydrodynamic (Osmanov et al., 2012) flows showed that these phenomena
could be of relevance in cosmic plasma flows as well.

The presence of the velocity shear brings to existence new modes of collective
behavior; Kelvin modes (Kelvin, 1887) are the earliest example discovered more
than a century ago. Considering linear evolution of electrostatic
perturbations in a cold, unmagnetized, two-component plane-parallel plasma
shear flow it was found out (Rogava et al., 1998) that similar modes may exist
in electrostatic plasma flows. It was shown that the velocity shear induces,
due to the non-normality of the linear dynamics, a new elementary mode of
plasma non-periodic collective behavior the SLV, associated with
nonoscillatory motion of plasma species. The study showed that SLV may
efficiently exchange energy with the flow. It turned out that the two-
dimensional SLV could extract the flow energy only transiently, while the
three-dimensional ones could exhibit asymptotic persistence. It was also
discovered that in moderate and high velocity inhomogeneity (velocity shear)
regimes, the remarkable non-resonant conversion of SLV into LW could take
place.

The aim of this paper is to study dynamics of LW and SLV in the context of
astrophysical shear flows with kinematic complexity. Recent observational
developments with the study of plasma waves and instabilities within and
beyond the heliosphere strengthens our confidence that theoretical studies of
Langmuir modes in complex flows may find interesting applications in the
astronomical context. These groundbreaking observations showed that even
beyond the limits of the solar system, beyond the heliosphere and outside the
heliopause boundary, LW are present. In particular, Voyager 1, the first ever
in situ probe of the interstellar medium, very recently discovered the
distinct presence of LW; it was reported (Ocker et al., 2021) that from 2017
onwards the probe’s Plasma Wave System detected a class of weak but
persistent, narrowband plasma wave emission in its data. Most remarkably
large-scale density fluctuations were discovered, tracing possible presence of
interstellar turbulence between episodes of previously detected plasma
oscillations. The persistence of this emission indicated that the Voyager 1
was able to detect the interstellar plasma density variations even in the
absence of shock-generated LW events.

The paper is arranged in the following manner: In the Main Consideration,
based on results of (Rogava et al., 1998) and (Mahajan and Rogava, 1999), we
develop the linear theory of Langmuir modes in kinematically complex plasma
flows and derive an explicit second-order ODE governing the evolution of these
modes. In the Discussion we study the evolution of these modes and find out
that there is a remarkable similarity between the evolution of Langmuir modes
and acoustic modes (Mahajan and Rogava, 1999). Our study makes clear that
Langmuir modes in kinematically complex flows exhibit an impressive variety of
peculiar, shear-induced, asymptotically persistent phenomena, involving (but
not limited to) various kinds of, including parametrically driven,
instabilities, efficient energy exchange processes with the background flow,
interesting “echoing” dynamics of perturbations, wave-vortex-wave conversion
events and appearance of beat wave phenomena. In the concluding section of the
paper we discuss the possible relevance and connection of the ”asymptotic
persistence” with the surprisingly persistent appearance of Langmuir modes
within the heliosphere.

##  2 Main Consideration

In this paper we employ the general approach developed in (Mahajan and Rogava,
1999): we consider only small-scale perturbations with their length-scales
$l_{i}$ much smaller than the characteristic scales $L_{i}$ of the flow
($i=x,y,z$). When $l_{i}{\ll}L_{i}$ the spatial inhomogeneity of an arbitrary
background velocity field ${\bf U}({\bf r})$ in the close neighbourhood of a
point $A$ with a radius vector ${\bf r}_{A}$ ($|{\bf r}-{\bf r}_{A})|/|{{\bf
r}_{A}}|{\ll}1$) may be approximated by the linear terms in its Taylor
expansion: $U_{i}(x_{i})\simeq{U_{A}}_{i}+a_{ik}x_{k}$. It leads to the
definition of the Shear Matrix (Mahajan and Rogava, 1999):

${\cal S}\equiv(a_{ik})=\begin{pmatrix}U_{x,x}&U_{x,y}&U_{x,z}\\\
U_{y,x}&U_{y,y}&U_{y,z}\\\ U_{z,x}&U_{z,y}&U_{z,z}\end{pmatrix}$ (1)

For flows with uniform equilibrium density, the velocity field is solenoidal
($\nabla\cdot{\bf U}=0$), implying that the Shear Matrix has to be traceless.
The linearized convective derivative is reduced to ${\cal
D}u_{i}+a_{ik}u_{k}$, where ${\cal
D}\equiv\partial_{t}+U_{i}(x,y,z)\partial_{i}$. The essential idea of this
approach is to find a proper form of an unknown function that would facilitate
solution, viz. that would eliminate the spatial dependence in the operator
${\cal D}$. It turns out that for the ansatz:

$F({\bf r},t)\equiv\hat{F}({\bf k}(t),t)e^{i\varphi},$ (2) ${\varphi}({\bf
r},t)\equiv{\bf k}\cdot{\bf r}-{\bf U_{A}}\cdot{\int}_{0}^{t}{\bf
k}(t^{\prime})dt^{\prime},$ (3)

the elimination does happen, the ansatz ensures that the linearized convective
derivative reduces to an ordinary derivative in time for the amplitude of the
perturbation:

${\cal D}F=e^{i\varphi}\partial_{t}\hat{F}^{,}$ (4)

but it occurs if and only if the wave vector ${\bf k}$ acquires the time-
dependence given by:

$\partial_{t}{\bf k}+{\cal S}^{T}\cdot{\bf k}=0,$ (5)

where ${\cal S}^{T}$ stands for the transposed shear matrix (Craik and
Criminale, 1986; Mahajan and Rogava, 1999).

Consequently the set of partial differential equations (PDE’s), describing the
evolution of linear perturbations reduces to the set of non-autonomous
ordinary differential equations (ODE’s). It dramatically simplifies the task,
reducing mathematically challenging sets of PDE’s (often with complicated and
obscure boundary conditions) to the initial value problem formulated for the
set of non-autonomous ODE’s. It was repeatedly shown (Mahajan and Rogava,
1999; Rogava et al., 2001, 2003a, 2003b, 2007; Osmanov et al., 2012, 2015)
that the latter can be inspected and reveal previously overlooked modes of
collective behaviour, sustained and modified by the velocity fields with
kinematic complexity.

Following (Rogava et al., 1998) we consider nonrelativistic, quasineutral,
cold and unmagnetized fluid plasma. That is, our main assumptions are standard
for the study of plasma oscillations. We do assume there is no magnetic field
$(\bf B=0)$; there are no thermal motions $(T=0)$; we adopt “no boundaries”
condition: the plasma is infinite in extent; and the oscillations are
electrostatic: $\vec{E}=-\nabla\varphi$.

For the mean flow velocity we adopt the two-dimensional kinematically complex
flow model used in (Mahajan and Rogava, 1999): ${\bf U}(x,y)\equiv
U_{x}(x,y){\bf e}_{x}+U_{y}(x,y){\bf e}_{y}$ with
$a_{11}=-a_{22}\equiv\sigma$, $a_{12}\equiv a$ and $a_{21}\equiv b$. Applying
the ansatz (2-5) we derive the set of linearized, first order ODE’s:

$\varrho^{(1)}={\cal K}_{x}v_{x}+{\cal K}_{y}v_{y},$ (6)
$v_{x}^{(1)}=-\varepsilon v_{x}-R_{1}v_{y}-(W/{\cal K})^{2}{\cal
K}_{x}\varrho,$ (7) $v_{y}^{(1)}=-R_{2}v_{x}+\varepsilon v_{y}-(W/{\cal
K})^{2}{\cal K}_{y}\varrho.$ (8)

while the dimensionless components of the wavevector ${\cal K}_{x,y}\equiv
k_{x,y}/k_{x}(0)$ vary in time as:

${\cal K}_{x}^{(1)}=-\varepsilon{\cal K}_{x}-R_{2}{\cal K}_{y},$ (9) ${\cal
K}_{y}^{(1)}=-R_{1}{\cal K}_{x}+\varepsilon{\cal K}_{y}.$ (10)

In these equations we introduced the dimensionless notation:
$W\equiv\omega/ck_{x}(0)$, ($\omega\equiv(4\pi q^{2}N/m)^{1/2}$ is the plasma
frequency); $\varepsilon\equiv\sigma/ck_{x}(0)$, $R_{1}\equiv a/ck_{x}(0)$,
$R_{2}\equiv b/ck_{x}(0)$, , $\varrho\equiv
i({\hat{\rho}}^{\prime}/\rho_{0})$, and $v_{x,y}\equiv\hat{u}_{x,y}/c$. Note
that $f^{(n)}$ denotes n-th order time derivative of $f$ by the dimensionless
time variable $\tau\equiv ck_{x}(0)t$.

These equations, after taking one more time derivative, readily lead to:

$\begin{Bmatrix}{\cal K}_{x}\\\ {\cal
K}_{y}\end{Bmatrix}^{(2)}={\Lambda}^{2}\begin{Bmatrix}{\cal K}_{x}\\\ {\cal
K}_{y}\end{Bmatrix},$ (11)

where ${\Lambda}^{2}\equiv{\varepsilon}^{2}+R_{1}R_{2}$.

Another interesting and important property of the dimensionless wavevector is
that the z-component of the vector ${\bf k}^{(1)}\times{\bf k}$ is a conserved
quantity:

$\Delta={\cal K}_{x}^{(1)}{\cal K}_{y}-{\cal K}_{y}^{(1)}{\cal K}_{x}=const.$
(12)

Introducing in the k-space the polar angle $\varphi(t)$, implying ${\cal
K}^{2}\equiv{\cal K}_{x}^{2}+{\cal K}_{x}^{2}$, ${\cal K}_{x}={\cal
K}cos{\varphi}$, and K${}_{y}={\cal K}sin{\varphi}$, we can easily find out
that:

$\Delta=-{\cal K}^{2}\varphi^{(1)}\equiv-{\cal K}^{2}\omega=const,$ (13)

where, obviously, $\omega=\varphi^{(1)}$ is the angular velocity of the
wavevector rotation in the k-space!

Evidently ${\cal K}$ plays an important role in the mathematical description
of the Langmuir mode dynamics. Therefore, it is important to study in detail
its properties. We can easily show that:

$({\cal K}^{2})^{(1)}=2\left[\varepsilon({\cal K}_{y}^{2}-{\cal
K}_{x}^{2})-(R_{1}+R_{2}){\cal K}_{x}{\cal K}_{y}\right].$ (14)

Furthermore, with some additional calculations and taking into account (11)
and (12) we derive:

$({\cal K}^{2})^{(2)}=4\Lambda^{2}{\cal K}^{2}+2\Delta(R_{1}-R_{2}).$ (15)
${\cal K}^{(1)}=\sqrt{2\Lambda^{2}{\cal K}^{2}+\Delta(R_{1}-R_{2})-{\cal
K}{\cal K}^{(2)}},$ (16) $\frac{{\cal K}^{(2)}}{{\cal
K}}=\Lambda^{2}+\frac{\Delta^{2}}{{\cal K}^{4}},$ (17)

while combining the last two equations we finally derive an explicit first
order ODE for the ${\cal K}^{2}$ function:

$({\cal K}^{2})^{(1)}=\sqrt{{\Lambda^{2}{\cal K}^{4}+\Delta(R_{1}-R_{2}){\cal
K}^{2}-\Delta^{2}}}.$ (18)

This is important equation because it allows to determine the function
$X(t)\equiv{\cal K}^{2}$ by calculating analytically the well-known integral:

$\int\frac{dX}{\sqrt{AX^{2}+BX+C}}=2\int dt,$ (19)

where $A\equiv\Lambda^{2}$, $B\equiv(R_{1}-R_{2})\Delta$, and
$C\equiv-\Delta^{2}$. Depending on values of these parameters the function
$\cal K$ shall vary in time in a polynomial, periodic or exponential way. The
similar result, of course, is achieved if one solves for ${\cal K}_{x}$ and
${\cal K}_{y}$ and determines the variability of the $\cal K$ via its
definition.

Turning our attention now to the set (6-8) we can easily prove that the
following combination of variables is a conserved quantity:

${\cal C}\equiv{\cal K}_{y}v_{x}-{\cal K}_{x}v_{y}+(R_{1}-R_{2})\varrho=const$
(20)

Subsequently, calculating ${\varrho}^{(2)}$. We find out that:

${\varrho}^{(2)}+W^{2}\varrho=2({\cal K}_{x}^{(1)}v_{x}+{\cal
K}_{y}^{(1)}v_{y})$ (21)

It is possible to calculate the velocity perturbation components in terms of
the density perturbation and its first derivative. It is straightforward to
derive that:

$v_{x}=\left({\cal K}_{y}[{\cal C}+(R_{2}-R_{1}){\varrho}]+{\cal
K}_{x}{\varrho}^{(1)}\right)/{\cal K}^{2},$ (22) $v_{y}=\left(-{\cal
K}_{x}[{\cal C}+(R_{2}-R_{1}){\varrho}]+{\cal
K}_{y}{\varrho}^{(1)}\right)/{\cal K}^{2}.$ (23)

Obviously (21-23) can be used to derive explicit, second-order nonautonomous
ODE for the density perturbation. Taking into account the properties of the
$\cal K$ function we come to the result:

$\varrho^{(2)}-{{({\cal K}^{2})^{(1)}}\over{{\cal
K}^{2}}}\varrho^{(1)}+\left[W^{2}+{{2\Delta(R_{1}-R_{2})}\over{{\cal
K}^{2}}}\right]\varrho={{2{\cal C}\Delta}\over{{\cal K}^{2}}}$ (24)

By eliminating the first derivative term $\varrho^{(1)}$ this equation can be
rewritten in a more elegant and concise way. The task is fulfilled by
introducing yet another auxiliary variable $\Psi\equiv\varrho/{\cal K}$. It is
straightforward to see that for this function we will have the following
equation:

${\Psi}^{(2)}+{\Omega}^{2}{\Psi}={{2{\cal C}{\Delta}}\over{{{\cal K}^{3}}}}$
(25)

where

${\Omega}^{2}\equiv W^{2}+{{{\cal K}^{(2)}}\over{{\cal K}}}-{{2({\cal
K}^{(1)})^{2}}\over{{\cal K}^{2}}}+{{2\Delta(R_{1}-R_{2})}\over{{\cal
K}^{2}}}.$ (26)

Fig. 1 : Exponentially growing density perturbation $\varrho$, for
$R_{1}=2\cdot 10^{-3}$, $R_{2}=0.2$, $\varepsilon=0$, $K_{y}(0)=0.1$, $C=0$,
$W=1$, ${\Psi}(0)=10^{-2}$, and ${{\Psi}^{{}^{\prime}}}(0)=0$. Fig. 2 :
Asymptotic persistence of echoing SLV interacting with LW for the density
$\varrho$ perturbations. $\varepsilon=0$, $R_{1}=0.1$, $R_{2}=-0.01$, ${\cal
C}=1.2$, $W=1$, ${\cal K}_{y}(0)=1$, $\Psi(0)=9\cdot 10^{-2}$, and
${{\Psi}^{{}^{\prime}}}(0)=8.66\cdot 10^{-3}$.

Which can be further simplified by taking into account (16) and (17). The
final result is:

${\Psi}^{(2)}+{\left[W^{2}-\Lambda^{2}+\frac{3\Delta^{2}}{{\cal
K}^{4}}\right]}\Psi=\frac{2{\cal C}\Delta}{{\cal K}^{3}}$ (27)

This equation is very convenient for inspecting and classifying possible
regimes of Langmuir mode dynamics. It is important to note that it features a
striking similarity with the equation (7) from (Mahajan and Rogava, 1999):

${\Psi}^{(2)}+{\left[{\cal K}^{2}-\Lambda^{2}+\frac{3\Delta^{2}}{{\cal
K}^{4}}\right]}\Psi=\frac{2{\cal C}\Delta}{{\cal K}^{3}},$ (28)

where dynamics of acoustic modes in kinematically complex flows was
considered. The only, but quite important, difference between these two
equations is related with the ”generic” frequency of oscillations: for
acoustic perturbations it is equal to ${\cal K}^{2}$, while for Langmuir
modes, obviously, instead we do have the square of the normalized plasma
oscillation (Langmuir) frequency $W^{2}$.

The analogy suggests classes of probable evolution scenarios for the evolution
of Langmuir modes in kinematically complex flows. It is evident that,
similarly to the acoustic case, the value and the magnitude of the
$\Lambda^{2}$ parameter would determine temporal evolution of the $\cal K$
function, determining, in its turn, the time-dependence of the effective
frequency $\Omega$ and the inhomogeneity term on the right hand side of the
(27). When $\Lambda^{2}=0$ the evolution is fully identical to the case of
plane-parallel flows studied in detail in (Rogava et al., 1998). When
$\Lambda^{2}>0$, the time-dependence of the $\cal K$ function is exponential,
transcribing into shear-induced, exponentially growing waves within
kinematically complex flows (Mahajan and Rogava, 1999). We have found
analogous, exponentially growing LW in our case, represented by the 1. The
values of parameters here are: $R_{1}=2\cdot 10^{-3}$, $R_{2}=0.2$,
$\varepsilon=0$, $K_{y}(0)=0.1$, $C=0$, $W=1$, $\Psi(0)=10^{-2}$, and
${{\Psi}^{{}^{\prime}}}(0)=0$. Of course viscous damping would eventually
”kill” these perturbations on a nonlinear stage of their evolution, but the
existence of robust, shear-induced linear exponential growth indicates at the
important ability of LW to be asymptotically persistent in flows with
kinematic complexity. When $\Lambda^{2}<0$ we found that nonperiodic modes of
plasma collective behaviour - shear Langmuir vortices (SLV) - exhibit
remarkable “echoing” behaviour, reminiscent again of the analogous behaviour
of Kelvin modes in hydrodynamic flows (Mahajan and Rogava, 1999). It is
featured on 2. The values of parameters here are: $\varepsilon=0$,
$R_{1}=0.1$, $R_{2}=-0.01$, ${\cal C}=1.2$, $W=1$, ${\cal K}_{y}(0)=1$,
$\Psi(0)=9\cdot 10^{-2}$, and ${{\Psi}^{{}^{\prime}}}(0)=8.66\cdot 10^{-3}$.
And again and again, this is clear example of asymptotic persistence - SLV and
LW keep converting into one another and it persists for a long intervals of
time. In the $\Lambda^{2}<0$ the existence of parametrically unstable LW is
also expected, resembling similar instability disclosed for acoustic waves in
(Mahajan and Rogava, 1999). It is one of topics of our future research to
determine those isolated areas of the k-space, where parametrically unstable
LS are expected to exist.

Evidently the problem is quite complex, the equations contain a number of free
parameters related to the physics of the Langmuir modes and the kinematic
portrait of the background flow. The thorough and systematic exploration of
all interesting regimes, with certain cases allowing exactly or approximately
analytic solutions, is currently undergoing and the results will be published
in a forthcoming, separate publication.

##  3 Discussion and Conclusions

Within last few decades studies of the phenomenon of asymptotic persistence
(Mahajan and Rogava, 1999) of shear-induced mode dynamics in kinematically
complex flows amply demonstrated that these processes take place in a wide
variety of physical or astrophysical systems. At the other hand, even in
plane-parallel flows Langmuir modes are known to exhibit impressive variety of
nonmodal phenomena (Rogava et al., 1998), including exponential growth of LW
and onset of totally new, shear-flow-generated nonperiodic modes of collective
behaviour - Shear Langmuir Vortexes (SLV). It was found out that LW and SLV
efficiently interact with background flows and may convert into one another.
The main purpose of the current study was to fuse these two issues, to make
one step further and check what happens with Langmuir modes (LW and SLV) in
plasma flows with kinematic complexity. Naturally the main field of
application that we have for starters in mind is solar physics, namely highly
probable relevance of this physical study to the dynamics of plasma modes in
the heliosphere and beyond the heliopause, in the interstellar medium.

The relevance and importance of this context seems very well-established.
Empirical study of LW in the heliosphere has undergone significant
advancements in recent years. Recently it became increasingly clear that these
waves, intrinsically connected to electron dynamics, may play an important
role in providing a channel of the energy transfer from the Sun to the
outermost regions of the heliosphere (Briand, 2015). Besides, the measurable
growth rates and saturation levels of LW, processes of fundamental nature,
potentially provide credible diagnostic capabilities. In addition, in the
heliosphere, an almost collisionless medium, LW may play the role of
collisions, participating in the exchange of energy between different species.

On the basis of this understanding recent empirical research has expanded the
study objectives of LW to a number of contexts, in both quiescent conditions
of the solar wind and during transient events, including planetary
environments such as the ionosphere, the magnetotail, and the foreshock,
interaction of the heliospheric plasma with ion and dust tails of rapidly
moving comets, as well as the processes well beyond the heliopause, in the
interplanetary medium. Namely, several spacecrafts, including IMP-6, IMP-7,
IMP-8, Helios, STEREO, and Wind, have observed electron events, associated
with LW and type III radio bursts as far out as 1 AU. These observations have
been critical in understanding the characteristics and dynamics of LW (Lorfing
et al., 2023).

Furthermore, Solar electron beams responsible for type III radio emission
generate LW as they propagate out from the Sun. They are observed via in-situ
electric field measurements. These LW are not smoothly distributed but appear
in discrete clumps, commonly attributed to the turbulent nature of the solar
wind electron density (Reid and Kontar, 2017). Still there is no exact
knowledge of the interaction between the density turbulence and the LW
electric fields, the issue is only qualitatively understood. Namely, on the
basis of the weak turbulence simulations, it was checked out how solar wind
density turbulence might changes the probability distribution functions, mean
value, and variance of the beam-driven electric field distributions.

Concluding the paper we can only repeat that our main purpose in this study
was to see whether asymptotic persistence of Langmuir modes in kinematically
complex shear plasma flows could be related to the observed persistency of
plasma waves in the heliosphere and beyond, in the interstellar space
immediately beyond the heliopause. Our first results seem very encouraging and
pointing out at the highly probable relevance of the assumption. Needless to
say further study is required to prove the validity of our conjecture in a
more comprehensive way.

Acknowledgements Our research was supported in part by Shota Rustaveli
National Science Foundation of Georgia (SRNSFG) grant FR-18-14747. Andria
Rogava wishes to thank for hospitality Centre for mathematical Plasma-
Astrophysics, KU Leuven, where a part of this work was completed.

## References

  * Athay (1986) Athay, R.G.: In: Sturrock, P.A. (ed.) Physics of the Sun vol. 2, p. 51 (1986)
  * Balbus and Hawley (1992) Balbus, S.A., Hawley, J.F.: The Astrophysical Journal 400, 610 (1992)
  * Briand (2015) Briand, C.: Journal of Plasma Physics 81(2) (2015)
  * Chen (2021) Chen, B. L.and Zhang: The Astrophysical Journal 906(2), 105 (2021)
  * Craik and Criminale (1986) Craik, A.D.D., Criminale, W.O.: Proc. R. Soc. Lond. A 406, 13 (1986)
  * Fan and qing Lou (1997) Fan, Z., Lou, Y.-q.: Monthly Notices of the Royal Astronomical Society 291, 91 (1997)
  * Ferrari (1998) Ferrari, A.: Annual Review of Astronomy and Astrophysics 36(1), 539 (1998)
  * Goldreich and Lynden-Bell (1965) Goldreich, P., Lynden-Bell, D.: Monthly Notices of the Royal Astronomical Society 130, 125 (1965)
  * Goldreich and Tremaine (1978) Goldreich, P., Tremaine, S.: Astrophysical Journal 222, 850 (1978)
  * Horton and Hasegawa (1994) Horton, W., Hasegawa, A.: Chaos 4, 227 (1994)
  * Kelvin (1887) Kelvin, L.: Philosophical Magazine 24(5), 188 (1887)
  * Lagnado et al. (1984) Lagnado, R., Phan-Thien, N., Leal, L.: Physics of Fluids 27 (1984)
  * Lominadze et al. (1988) Lominadze, J.G., Chagelishvili, G.D., Chanishvili, R.G.: Pisma v Astronomicheskii Zhurnal 14, 856 (1988)
  * Lorfing et al. (2023) Lorfing, C.Y., Reid, H.A.S., Gómez-Herrero, R., Maksimovic, M., Nicolaou, G., Owen, C.J., Rodriguez-Pacheco, J., Ryan, D.F., Trotta, D., Verscharen, D.: The Astrophysical Journal 959(2), 128 (2023)
  * Lovelace et al. (1991) Lovelace, R.V.E., Berk, H.L., Contopoulos, J.: The Astrophysical Journal 379, 696 (1991)
  * Lovelace et al. (1987) Lovelace, R.V.E., Wang, J.C.L., Sulkanen, M.E.: The Astrophysical Journal 315, 504 (1987)
  * Mahajan and Rogava (1999) Mahajan, S.M., Rogava, A.D.: The Astrophysical Journal 518(2), 814 (1999)
  * Mahajan et al. (1997) Mahajan, S.M., Machabeli, G.Z., Rogava, A.D.: The Astrophysical Journal 479(2), 129 (1997)
  * Ocker et al. (2021) Ocker, S.K., Cordes, J.M., Chatterjee, S., Gurnett, D.A., Kurth, W.S., Spangler, S.R.: Nature Astronomy 5, 761 (2021)
  * Osmanov et al. (2012) Osmanov, Z., Rogava, A., Poedts, S.: Astronomy and Astrophysics 19 (2012)
  * Osmanov et al. (2015) Osmanov, Z., Rogava, A., Poedts, S.: New Journal of Physics 17 (2015)
  * Parker (1974) Parker, E.N.: Astronomy and Astrophysics 190, 429 (1974)
  * Petviashvili (1980) Petviashvili, V.I.: Soviet Journal of Experimental and Theoretical Physics Letters 32, 619 (1980)
  * Pike and Mason (1998) Pike, C.D., Mason, H.E.: Solar Physics 182(2), 333 (1998)
  * Pneuman and Orral (1980) Pneuman, G., Orral, F.: Physics of the Sun. D. Reidel Publ. Co., Dordrecht, Holland (1980)
  * Poedts and Rogava (2002) Poedts, S., Rogava, A.D.: The Astrophysical Journal 385, 32 (2002)
  * Poedts et al. (1998) Poedts, S., Rogava, A.D., Mahajan, S.M.: The Astrophysical Journal 505(1), 369 (1998)
  * Reid and Kontar (2017) Reid, H., Kontar, E.: Astronomy & Astrophysics 598 (2017)
  * Rogava et al. (2003b) Rogava, A.D., Bodo, G., Massaglia, S., Osmanov, Z.: Astronomy & Astrophysics 408(2), 401 (2003b)
  * Rogava et al. (2000) Rogava, A.D., Poedts, S., Mahajan, S.M.: Astronomy & Astrophysics 354, 749 (2000)
  * Rogava et al. (2001) Rogava, A.D., Poedts, S., Mahajan, S.M.: Journal of Computational Acoustics 09(03), 869 (2001)
  * Rogava et al. (2003a) Rogava, A.D., Mahajan, S.M., Bodo, G., S., M.: Astronomy & Astrophysics 399, 421 (2003a)
  * Rogava et al. (2007) Rogava, A., Gogoberidze, G., Poedts, S.: The Astrophysical Journal 664(2), 1221 (2007)
  * Rogava and Khujadze (1997) Rogava, A.D., Khujadze, G.R.: General Relativity and Gravitation 29, 345 (1997)
  * Rogava et al. (1998) Rogava, A.D., Chagelishvili, G.D., Mahajan, S.M.: Physical Reiew E 57, 7103 (1998)
  * Rogava et al. (1999) Rogava, A.D., Poedts, S., Heirman, S.: Monthly Notices of the Royal Astronomical Society 307(4), 31 (1999)
  * Saffman (1993) Saffman, P.G.: Vortex Dynamics. Cambridge Monographs on Mechanics. Cambridge University Press, ??? (1993)
  * Tagger et al. (1992) Tagger, M., Pellat, R., Coroniti, F.V.: Astrophysical Journal 393, 708 (1992)
  * Toomre (1969) Toomre, A.: Astrophysical Journal 158, 899 (1969)
  * Trefethen et al. (1993) Trefethen, L.N., Trefethen, A.E., Reddy, S.C., Driscoll, T.A.: Science 261(5121), 578 (1993)


# Unsupervised Dense Retrieval with
Relevance-Aware Contrastive Pre-Training

Yibin Lei1 , Liang Ding2 , Yu Cao3, Chantong Zan4, Andrew Yates1, Dacheng Tao2
1University of Amsterdam 2JD Explore Academy
3Tencent IEG 4China University of Petroleum (East China)
{y.lei<EMAIL_ADDRESS>{liangding.liam<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>Work done when Yibin Lei was
interning at JD Explore Academy. Corresponding author

###### Abstract

Dense retrievers have achieved impressive performance, but their demand for
abundant training data limits their application scenarios. Contrastive pre-
training, which constructs pseudo-positive examples from unlabeled data, has
shown great potential to solve this problem. However, the pseudo-positive
examples crafted by data augmentations can be irrelevant. To this end, we
propose relevance-aware contrastive learning. It takes the intermediate-
trained model itself as an imperfect oracle to estimate the relevance of
positive pairs and adaptively weighs the contrastive loss of different pairs
according to the estimated relevance. Our method consistently improves the
SOTA unsupervised Contriever model (Izacard et al., 2022) on the BEIR and
open-domain QA retrieval benchmarks. Further exploration shows that our method
can not only beat BM25 after further pre-training on the target corpus but
also serves as a good few-shot learner. Our code is publicly available at
https://github.com/Yibin-Lei/ReContriever.

##  1 Introduction

Dense retrievers, which estimate the relevance between queries and passages in
the dense embedding space, have achieved impressive performance in various
applications, including web search (Liu et al., 2021) and open-domain question
answering (Karpukhin et al., 2020). One key factor for the success of dense
retrievers is a large amount of human-annotated training data, e.g., MS-MARCO
(Bajaj et al., 2016) with above 500,000 examples. However, a recent study
(Thakur et al., 2021) shows that even trained with enormous labeled data,
dense retrievers still suffer from a generalization issue, where they perform
relatively poorly on novel domains in comparison to BM25. Meanwhile,
collecting human-annotated data for new domains is always hard and expensive.
Thus improving dense retrievers with limited annotated data becomes essential,
considering the significant domain variations of practical retrieval tasks.

Contrastive pre-training, which first generates pseudo-positive examples from
a universal corpus and then utilizes them to contrastively pre-train
retrievers, has shown impressive performance without any human annotations Lee
et al. (2019); Gao et al. (2021); Gao and Callan (2022); Ram et al. (2022);
Izacard et al. (2022). For instance, Contriever Izacard et al. (2022) crafts
relevant query-passage pairs by randomly cropping two random spans within the
same document. However, owing to the high information density of texts, even
nearby sentences in a document can be very irrelevant, as shown in Figure 1.
These false positive samples may mislead the model to pull unrelated texts
together in the embedding space and further harm the validity of
representations.

Figure 1: A text snippet from Wikipedia, where two nearby sentences are quite
irrelevant. Random cropping may lead to a false positive query-passage pair.

Motivated by recent findings in computer vision that pre-training performance
can be greatly boosted by reducing the effect of such false positives Peng et
al. (2022); Mishra et al. (2022), we propose Relevance-Aware Contrastive
Retriever (ReContriever). At each training step, we utilize the trained models
at the current step itself to estimate the relevance of all the positives.
Then the losses of different positive pairs are adaptively weighed using the
estimated relevance, i.e., the pairs that receive higher relevance scores
obtain higher weight. Moreover, simply applying lower weights to irrelevant
pairs will result in insufficient usage of data, since many documents will
contribute less to training. Therefore, we also introduce a one-document-
multiple-pair strategy that generates multiple positive pairs from a single
document, with a pair-weighting process conducted among samples originating
from a single document. Such an operation makes sure that the model can learn
positive knowledge from every document in the corpus.

To summarize, our contributions in this paper are three-fold: 1) We propose
relevance-aware contrastive learning for dense retrieval pre-training, which
aims to reduce the false positive problem. 2) Experiments show our method
brings consistent improvements to the SOTA unsupervised Contriver model on
10/15 tasks on the BEIR benchmark and three representative open-domain QA
retrieval datasets. 3) Further explorations show that our method works well
given no or limited labeled data. Specifically, on 4 representative domain-
specialized datasets it outperforms BM25 when only unsupervised pre-training
on the target corpora, and with only a few annotated samples its accuracy can
be on par with DPR Karpukhin et al. (2020) which is trained on thousands of
annotated examples.

DATASET | BM25 | BERT | SimCSE | RetroMAE | coCondenser | Contriever |  Contriever (reproduced) | ReContriever
---|---|---|---|---|---|---|---|---
MS MARCO | 22.8 | 0.6 | 8.8 | 4.5 | 7.7 | 20.6 | 21.1 |  $21.8^{\dagger}$
Trec-COVID | 65.6 | 16.6 | 38.6 | 20.4 | 17.3 | 27.4 | 42.0 | 40.5
NFCorpus | 32.5 | 2.5 | 14.0 | 15.3 | 14.4 | 31.7 | 30.0 |  $31.9^{\dagger}$
NQ | 32.9 | 2.7 | 12.6 | 3.4 | 3.9 | 25.4 | 29.5 |  $31.0^{\dagger}$
HotpotQA | 60.3 | 4.9 | 23.3 | 25.0 | 24.4 | 48.1 | 44.1 |  $50.1^{\dagger}$
FiQA-2018 | 23.6 | 1.4 | 14.8 | 9.3 | 5.2 | 24.5 | 26.2 | 26.2
ArguAna | 31.5 | 23.1 | 45.6 | 37.6 | 34.5 | 37.9 | 43.4 | 39.8
Touche-2020 | 36.7 | 3.4 | 11.6 | 1.9 | 3.0 | 16.7 | 16.7 | 16.6
CQADupStack | 29.9 | 2.5 | 20.2 | 17.0 | 9.8 | 28.4 | 28.4 |  $28.7^{\dagger}$
Quora | 78.9 | 3.9 | 81.5 | 69.0 | 66.7 | 83.5 | 83.6 |  $\textbf{84.3}^{\dagger}$
DBPedia | 31.3 | 3.9 | 13.7 | 4.6 | 15.1 | 29.2 | 27.6 |  $29.3^{\dagger}$
SCIDOCS | 15.8 | 2.7 | 7.4 | 7.4 | 1.9 | 14.9 | 15.0 |  $15.6^{\dagger}$
FEVER | 75.3 | 4.9 | 20.1 | 7.1 | 25.3 | 68.2 | 66.9 |  $68.9^{\dagger}$
Climate-fever | 21.3 | 4.1 | 17.6 | 4.4 | 9.8 | 15.5 | 15.6 | 15.6
SciFact | 66.5 | 9.8 | 38.5 | 53.1 | 48.1 | 64.9 | 65 | 66.4
Avg | 41.7 | 8.7 | 24.6 | 18.7 | 8.9 | 35.8 | 37.0 | 37.8
Avg Rank | 1.9 | 7.9 | 4.9 | 6.1 | 6.3 | 3.4 | 2.7 | 2.2

Table 1: NDCG@10 of BEIR Benchmark. All models are unsupervised trained
without any human-annotated data. Bold indicates the best result. The average
and rank across the entire benchmark are included. Four datasets are excluded
because of their licenses. “†” means ReContriever performs significantly
better than our reproduced Contriever, as determined by a t-test with p-value
0.05 as threshold.

##  2 Method

###  2.1 Preliminary

In this section, we briefly describe the bi-encoder structure used in dense
retrieval and the SOTA Contriever model, on which we build our model.

##### Bi-Encoder Structure

Dense retrievers are always a bi-encoder composed of two separate encoders to
transform the query and document into a single vector each. The relevance
score is obtained by computing the similarity (e.g., inner-product) between
the encoded vectors of queries and documents. The typical way to train a dense
retriever is using a contrastive loss that aims to pull relevant passages
closer to the query and irrelevant passages farther in the embedding space.
For each query, the training data involves one positive passage labeled by
annotators and a pool of negative passages, which are usually random passages
in the corpus.

##### Contriever

It crafts pseudo-positive pairs by randomly cropping two spans of the same
document. As negative texts have shown to be a key to the success of retrieval
training (Xiong et al., 2021), Contriever also applies the MoCo mechanism He
et al. (2020) to utilize negatives in the previous batches to increase the
number of negatives. These two factors make Contriever obtain significant
decent performance without any human annotations.

###  2.2 Relevance-Aware Contrastive Learning

We start by 1) producing a larger number of positives (one-document-multi-
pair) and 2) forcing the model to pay more attention to the ones with higher
relevance (relevance-aware contrastive loss).

##### One-Document-Multi-Pair

Given a text snippet $T$, previous pre-training methods always craft only one
positive (query-passage) pair $(q,d^{+})$. To exploit $T$ more effectively,
our one-document-multi-pair strategy generates $n$ positive pairs, denoting as
$\\{(q,d^{+}_{1}),(q,d^{+}_{2}),\ldots,(q,d^{+}_{n})\\}$, from $T$ by
repeating the procedure several times. We keep the query $q$ unchanged to
ensure the relevance comparison is fair among pairs within the same snippet,
which is used in our following step. Building upon Contirever, we craft $n$
pairs by random cropping $n+1$ spans and setting 1 span as the fixed query for
the left $n$ spans. And it is easy to extend this strategy to other
contrastive pre-training methods.

##### Relevance-Aware Contrastive Loss

The ordinary contrastive loss for training dense retrievers is the InfoNCE
loss. Given a positive pair $(q,d^{+})$ and a negative pool
$\\{d^{-}_{i}\\}_{i=1..D}$, $\text{InfoNCE}\left(q,d^{+}\right)$ is computed
by:

$-\ \text{log}\
\frac{\exp\left(s\left(q,d^{+}\right)/\tau\right)}{\exp\left(s\left(q,d^{+}\right)/\tau\right)+\sum_{i=1}^{D}\exp\left(s\left(q,d^{-}_{i}\right)/\tau\right)},$
(1)

where $s(\cdot)$ and $\tau$ denote the similarity function and temperature
parameter. Then the overall loss of a batch is usually the average across all
the $m\times n$ positive pairs from $m$ snippets:
$L={\frac{1}{mn}}\sum_{i=1}^{m}\sum_{j=1}^{n}\text{InfoNCE}(q_{i},d^{+}_{ij})$.

The relevance-aware contrastive loss aims to force the model to focus more on
true positive pairs by 1) utilizing trained model $\theta$ at present itself
as an imperfect oracle to compute the relevance score $s_{\theta}(q,d^{+})$
between all pairs; and 2) adaptively assigning weights to different pairs
according to the estimated relevance. Then the relevance-aware contrastive
loss $L_{\text{relevance}}$ can be expressed as:111Equation (2) will be
invalid when $s_{\theta}\left(q_{i},d_{ij}^{+}\right)$ is negative. In the
preliminary study, we found the value is always positive and thus ignore this
special case for simplicity.

${\frac{1}{m}}\sum_{i=1}^{m}\sum_{j=1}^{n}\frac{s_{\theta}\left(q_{i},d_{ij}^{+}\right)}{\sum_{k=1}^{n}s_{\theta}\left(q_{i},d_{ik}^{+}\right)}\text{InfoNCE}(q_{i},d^{+}_{ij}).$
(2)

In this way, for each text snippet, positive pairs with more confidence to be
relevant will thus be more focused on by the model, or vice versa.

##  3 Experiments

In this section, we evaluate our model in several settings after describing
our experimental setup. We consider unsupervised retrieval performance and two
practical use cases: further pre-training on the target domain and few-short
retrieval. We then conduct an ablation study to separate the impact of our
method’s two components.

###  3.1 Setup

  * •

Datasets We evaluate retrieval models on the BEIR Thakur et al. (2021)
benchmark and three representative open-domain QA retrieval benchmarks:
Natural Questions (NQ; Kwiatkowski et al. (2019)), TriviaQA Joshi et al.
(2017) and WebQuestions Berant et al. (2013).

  * •

Baselines We compare our model with two types of unsupervised models, namely
models based on contrastive pre-training and on auto-encoding pre-training.
The former models include SimCSE Gao et al. (2021), coCondenser Gao and Callan
(2022), Spider Ram et al. (2022) and Contriever Izacard et al. (2022). The
latter category includes the recently proposed RetroMAE Xiao et al. (2022).
BM25 Robertson and Zaragoza (2009) and uncased BERT-base model Devlin et al.
(2019) are also involved for reference. We use the official checkpoints for
evaluation.

  * •

Implementation Details We apply our method to the SOTA Contriever model and
use its default settings. The pre-training data is a combination of Wikipedia
and CCNet (Wenzek et al., 2020), same as Contriever. We generate 4 positive
pairs for each document. Refer to Appendix A for more details.

We conduct a t-test with p-value 0.05 as threshold to compare the performance
of ReContriever and our reproduced Contriever.

Model | NQ | TriviaQA | WQ
---|---|---|---
Top-5 | Top-20 | Top-100 | Top-5 | Top-20 | Top-100 | Top-5 | Top-20 | Top-100
| Supervised Model
DPR | - | 78.4 | 85.4 | - | 79.4 | 85.0 | - | 73.2 | 81.4
| Unsupervised Models
BM25 | 43.8 | 62.9 | 78.3 | 66.3 | 76.4 | 83.2 | 41.8 | 62.4 | 75.5
RetroMAE | 23.0 | 40.1 | 58.8 | 47.0 | 61.4 | 74.2 | 25.8 | 43.8 | 62.3
SimCSE | 5.4 | 11.5 | 23.0 | 3.7 | 7.6 | 17.0 | 3.3 | 8.7 | 19.4
coCondenser | 28.9 | 46.8 | 63.5 |  7.5 | 13.8 | 24.3 | 30.2 | 50.7 | 68.7
Spider | 49.6 | 68.3 | 81.2 | 63.6 | 75.8 | 83.5 | 46.8 | 65.9 | 79.7
Contriever | 47.3 | 67.8 | 80.6 | 59.5 | 73.9 | 82.9 | 43.5 | 65.7 | 80.1
Contriever (reproduced) | 48.9 | 68.3 | 81.4 | 61.2 | 74.6 | 83.4 | 47.0 | 67.0 | 80.5
ReContriever |  50.3† |  69.4† |  82.6† |  63.4† |  75.9† |  84.1† | 48.3 | 68.0 | 81.1
Table 2: Recall of open-domain retrieval benchmarks. Bold: the best results
across unsupervised models. “†” means ReContriever performs significantly
better than our reproduced Contriever, as determined by a t-test with p-value
0.05 as threshold.

###  3.2 Main Results

####  3.2.1 BEIR

The NDCG@10 of ReContriever and other fully unsupervised models across 15
public datasets of BEIR are shown in Table 1. ReContriever achieves consistent
improvements over Contriever on 10/15 datasets, with a significant improvement
observed in 9 of those datasets. Notably, it also only sees very slight
decreases on datasets without promotion (e.g., FiQA, Touche and Climate-Fever
with at most -0.1 decrease). Moreover, our method obtains an average rank of
2.2, proving our method to be the best unsupervised dense retriever. BM25 is
still a strong baseline under the fully unsupervised scenario, but
ReContriever greatly narrows the gap between dense retrievers and it.

####  3.2.2 Open-Domain QA Retrieval

Table 2 shows the Recall performance of ReContriever on open-domain QA
retrieval benchmarks, where supervised DPR Karpukhin et al. (2020) is involved
for reference. Obviously, ReContriever outperforms BM25 by a large margin
except for Recall@5 and Recall@10 on TriviaQA with relatively smaller
differences, verifying the effect of our method. Moreover, among all
unsupervised methods, ReContriever obtains the best performance in nearly all
cases, especially substantial improvement over Contriever. Our ReContriever
promisingly narrows the gaps between supervised and unsupervised models,
making it more valuable.

Model | SciF | SCID | Arg | CQA | Avg.
---|---|---|---|---|---
BM25 | 66.5 | 15.8 | 31.5 | 29.9 | 35.9
Contriever | 64.9 | 14.9 | 43.4 | 28.4 | 37.9
\+ corpus pretrain | 66.3 | 17.1 | 52.4 | 30.6 | 41.6⇑+3.7
ReContriever | 66.4 | 15.6 | 39.8 | 28.4 | 37.6
\+ corpus pretrain | 67.1 | 16.6 |  54.6† | 30.7 | 42.3⇑+4.7
Table 3: NDCG@10 after further pre-training on the target domain corpus. “⇑”
denotes the gains of further pre-training. “†” means ReContriever performs
significantly better than our reproduced Contriever.

###  3.3 Practical Use Cases

In this section, we explore the applicability of ReContriever in more
practical scenarios222We report results of our reproduced Contriever as they
are slightly better than the original ones (Izacard et al., 2022)., where only
texts in the target corpus (pre-training on the target domain) or very limited
annotated training data (few-shot retrieval) are available.

Model | NQ
---|---
Top-5 | Top-20 | Top-100
Reference
DPR | - | 78.4 | 85.4
BM25 | 43.8 | 62.9 | 78.3
| 8 examples
Spider | 49.7 | 68.3 | 81.4
Contriever | 51.7 | 70.6 | 83.1
ReContriever | 52.9 | 71.6 |  84.2†
| 32 examples
Spider | 50.2 | 69.4 | 81.7
Contriever | 52.6 | 70.9 | 83.1
ReContriever | 53.5 |  71.9† |  84.7†
| 128 examples
Spider | 57.0 | 74.3 | 85.3
Contriever | 55.1 | 72.4 | 83.7
ReContriever | 55.9 |  74.1† |  85.1†
Table 4: Few-shot Retrieval on NQ. Results are report with Recall. “†” means
ReContriever performs significantly better than our reproduced Contriever.

Model | MS MARCO | NFCoprus | NQ | Hotpot | FiQA | Touche | Quora | SCIDOCS | Avg.
---|---|---|---|---|---|---|---|---|---
Contriever | 19.1 | 25.1 | 26.7 | 43.2 | 23.2 | 18.6 | 82.3 | 14.6 | 31.6
\+ relevance-aware loss | 0.2 | 2.5 | 0.0 | 0.2 | 0.5 | 0.6 | 57.6 | 14.5 | 9.5
\+ one-document-multiple-pair | 19.9 | 29.5 | 27.5 | 44.2 | 21.9 | 15.9 | 82.8 | 14.5 | 32.0
ReContriever | 20.8 | 28.1 | 29.6 | 49.9 | 23.4 | 18.2 | 83.3 | 14.7 | 33.5

Table 5: Ablation Study. Results are reported with NDCG@10.

##### Pre-Training on the Target Domain

Four domain-specialized datasets (SciFact (Wadden et al., 2020) (SciF;
citation-prediction), SCIDOCS Cohan et al. (2020) (SCID; fact checking),
ArguAna Wachsmuth et al. (2018) (Arg; argument retrieval) and CQADupStack
Hoogeveen et al. (2015) (CQA; StackExchange retrieval) with only corpus
available are picked as a testbed, shown in Table 3. Before further pre-
training on the corresponding corpus, ReContriever underperforms BM25 on 3 of
4 datasets. Surprisingly, after pre-training, ReContriever is able to
consistently beat BM25. Moreover, our model obtains an average +4.7
improvement after further pre-training, which is substantially better than
Contriever (+3.7).

##### Few-Shot Retrieval.

Results on training with limited annotated data of NQ are shown in Table 4.
Following the same setting, our model trained on 128 samples can perform on
par on Recall@100 with DPR which has seen thousands of annotated samples. In
addition, when training data is scarce (below 100 examples), ReContriever
still shows stronger few-shot performance compared to Spider and Contriever.

###  3.4 Ablation Study

We conduct an ablation study to investigate the contributions of our proposed
loss and pairing strategies within ReContriever, using 100,000 training steps.
Solely adding relevance-aware loss means estimating the relevance of N pairs
from N documents and then normalizing the relevance among the N pairs within a
batch, which slightly differs from equation (2) that normalizes over 4 pairs
from the same document. As shown in Table 5, solely adding relevance-aware
contrastive loss to Contriever will lead to a noticeable degeneration, owing
to the missing information from the documents with low adjusted weights and
the unstable relevance comparison without a fixed query. Applying the one-
document-multi-pair strategy can obtain a slight improvement which can be
attributed to the effective usage of the unlabeled data. Combining both
strategies (i.e., ReContriever) can lead to an obvious improvement, which
demonstrates the necessity of both components in our method.

##  4 Conclusion

In this work, we propose ReContriever to further explore the potential of
contrastive pre-training to reduce the demand of human-annotated data for
dense retrievers. Benefiting from multiple positives from the same document as
well as relevance-aware contrastive loss, our model achieves remarkable
performance under zero-shot cases. Additional results on low data resources
further verify its value under various practical scenarios.

## Limitations

Although ReContriever narrows the gap between BM25 and unsupervised dense
retrievers, it still lags behind BM25 when acting as a general-purpose
retriever. This issue may make ReContriever not directly usable when facing a
new domain, thus limiting its practicality. Also, as ReContriever is
initialized from the language model BERT${}_{\text{base}}$, there may exist
social biases Zhao et al. (2017) in ReContriever and thus have the risk of
offending people from under-represented groups.

## Ethics Statement

We strictly adhere to the ACL Ethics Policy. This paper focuses on reducing
the false positives problem of unsupervised dense retrieval. The datasets used
in this paper are publicly available and have been widely adopted by
researchers. We ensure that the findings and conclusions of this paper are
reported accurately and objectively.

## References

  * Bajaj et al. (2016) Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, et al. 2016.  Ms marco: A human generated machine reading comprehension dataset.  _arXiv preprint_.
  * Berant et al. (2013) Jonathan Berant, Andrew K. Chou, Roy Frostig, and Percy Liang. 2013.  Semantic parsing on freebase from question-answer pairs.  In _EMNLP_.
  * Cohan et al. (2020) Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel Weld. 2020.  SPECTER: Document-level representation learning using citation-informed transformers.  In _ACL_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _NAACL_.
  * Gao and Callan (2022) Luyu Gao and Jamie Callan. 2022.  Unsupervised corpus aware language model pre-training for dense passage retrieval.  In _ACL_.
  * Gao et al. (2021) Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021.  SimCSE: Simple contrastive learning of sentence embeddings.  In _EMNLP_.
  * He et al. (2020) Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020.  Momentum contrast for unsupervised visual representation learning.  In _CVPR_.
  * Hoogeveen et al. (2015) Doris Hoogeveen, Karin M. Verspoor, and Timothy Baldwin. 2015.  Cqadupstack: A benchmark data set for community question-answering research.  In _ADCS_.
  * Izacard et al. (2022) Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022.  Unsupervised dense information retrieval with contrastive learning.  _TMLR_.
  * Joshi et al. (2017) Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017.  TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension.  In _ACL_.
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020.  Dense passage retrieval for open-domain question answering.  In _EMNLP_.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.  Natural questions: A benchmark for question answering research.  _TACL_.
  * Lee et al. (2019) Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019.  Latent retrieval for weakly supervised open domain question answering.  In _ACL_.
  * Liu et al. (2021) Yiding Liu, Weixue Lu, Suqi Cheng, Daiting Shi, Shuaiqiang Wang, Zhicong Cheng, and Dawei Yin. 2021.  Pre-trained language model for web-scale retrieval in baidu search.  In _KDD_.
  * Mishra et al. (2022) Shlok Kumar Mishra, Anshul Shah, Ankan Bansal, Janit K Anjaria, Abhyuday Narayan Jagannatha, Abhishek Sharma, David Jacobs, and Dilip Krishnan. 2022.  Object-aware cropping for self-supervised learning.  _TMLR_.
  * Peng et al. (2022) Xiangyu Peng, Kai Wang, Zheng Zhu, Mang Wang, and Yang You. 2022.  Crafting better contrastive views for siamese representation learning.  In _CVPR_.
  * Ram et al. (2022) Ori Ram, Gal Shachaf, Omer Levy, Jonathan Berant, and Amir Globerson. 2022.  Learning to retrieve passages without supervision.  In _NAACL_.
  * Robertson and Zaragoza (2009) Stephen Robertson and Hugo Zaragoza. 2009.  The probabilistic relevance framework: Bm25 and beyond.  _Found. Trends Inf. Retr._
  * Thakur et al. (2021) Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021.  BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models.  In _NeurIPS Datasets and Benchmarks_.
  * Wachsmuth et al. (2018) Henning Wachsmuth, Shahbaz Syed, and Benno Stein. 2018.  Retrieval of the best counterargument without prior topic knowledge.  In _ACL_.
  * Wadden et al. (2020) David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020.  Fact or fiction: Verifying scientific claims.  In _EMNLP_.
  * Wenzek et al. (2020) Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, et al. 2020.  CCNet: Extracting high quality monolingual datasets from web crawl data.  In _LREC_.
  * Xiao et al. (2022) Shitao Xiao, Zheng Liu, Shao Yingxia, and Cao Zhao. 2022.  Retromae: Pre-training retrieval-oriented language models via masked auto-encoder.  In _EMNLP_.
  * Xiong et al. (2021) Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. 2021.  Approximate nearest neighbor negative contrastive learning for dense text retrieval.  In _ICLR_.
  * Zhao et al. (2017) Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2017\.  Men also like shopping: Reducing gender bias amplification using corpus-level constraints.  In _EMNLP_.

##  Appendix A Implementation Details

##### Basic Infrastructure

Basic backbones of our implementation involve Pytorch333https://pytorch.org/
and Huggingface-Transformers444https://github.com/huggingface/transformers. We
build our code upon the released code of
Contriever555https://github.com/facebookresearch/contriever. Models are
evaluated using evaluation scripts provided by the
BEIR666https://github.com/beir-cellar/beir (for BEIR evaluation) and
Spider777https://github.com/oriram/spider (for open-domain QA retrieval
evaluation) GitHub repositories. The pre-training experiments are conducted on
16 NVIDIA A100 GPUs and the few-shot experiments are conducted on a single
NVIDIA A100 GPU. We report the results on a single run with a fixed random
seed 0 (same as the setting of Contriever).

##### Details of ReContriever

Following the default settings of Contriever, we pre-train ReContriever for
500,000 steps with a batch size of 2048, initializing from the uncased
BERT${}_{\text{base}}$ model with 110 million parameters. The pre-training
data is a combination of Wikipedia and CCNet (Wenzek et al., 2020). The
learning rate is set to $5\cdot 10^{-5}$ with a warm-up for the first 20,000
steps and a linear decay for the remaining steps. Average pooling over the
whole sequence is used for obtaining the final representation of the query or
document. For each document, we generate 4 positive pairs.

For experiments on target domain pre-training, we initialize the model from
our pre-trained Contriever/ReContriever checkpoints. To avoid overfitting, the
models are further pre-trained with 5000 warm-up steps to a learning rate of
$1.25\cdot 10^{-7}$ on all 4 picked datasets with a batch size of 1024 on 8
NVIDIA A100 GPUs.

For few-shot retrieval experiments, we adopt the training procedure from
Karpukhin et al. (2020): exploiting BM25 negatives and not including negatives
mined by the model itself Xiong et al. (2021) for few-shot fine-tuning. The
hyper-parameters are directly borrowed from Karpukhin et al. (2020) except for
the batch size and number of training epochs. We fine-tune all the models with
80 epochs. For 8 examples, the batch size is set to 8. The batch size is 32
when there are 32 or 128 examples.

##  Appendix B Dataset Statistics

Details about the number of examples in the there open-domain QA retrieval
datasets are shown in Table 6.

Dataset | Train | Dev | Test
---|---|---|---
NQ | 58880 | 8757 | 3610
TriviaQA | 60413 | 8837 | 11313
WQ | 2474 | 361 | 2032
Table 6: Statistics of Open-Domain QA Retrieval Datasets