# Bound-state dark matter and Dirac neutrino mass

M. Reig 1<EMAIL_ADDRESS>D. Restrepo 2<EMAIL_ADDRESS>J. W. F.
Valle 1<EMAIL_ADDRESS>URL: de http://astroparticles.es/ O. Zapata 2
<EMAIL_ADDRESS>1 AHEP Group, Institut de Física Corpuscular –
C.S.I.C./Universitat de València, Parc Cientific de Paterna.
C/Catedratico José Beltrán, 2 E-46980 Paterna (València) - SPAIN  2 Instituto
de Física, Universidad de Antioquia, Calle 70 No. 52-21, Apartado Aéreo 1226,
Medellín, Colombia.

###### Abstract

We propose a simple theory for the idea that cosmological dark matter (DM) may
be present today mainly in the form of stable neutral hadronic thermal relics.
In our model neutrino masses arise radiatively from the exchange of colored DM
constituents, giving a common origin for both dark matter and neutrino mass.
The exact conservation of $B-L$ symmetry ensures dark matter stability and the
Dirac nature of neutrinos. The theory can be falsified by dark matter nuclear
recoil direct detection experiments, leading also to possible signals at a
next generation hadron collider.

###### pacs:

13.15.+g, 14.60.Pq, 14.60.St, 95.35.+d

##  I Introduction

The common lore concerning particle dark matter candidates has long been that
they must be electrically neutral and carry no color. This view has been
challenged by the authors in Ref. DeLuca:2018mzn , who suggested that dark
matter (DM) may be the lightest hadron made of two stable color octet Dirac
fermions $\cal Q$ with mass below $10~{}\text{TeV}$. The interest of this idea
may go well beyond QCD, since analogous bound-state DM candidates emerge in
models with a new confining hypercolor interaction Reig:2017nrz .

We argue that neutrino mass and cosmological dark matter may have a common
origin, with the underlying DM physics acting as messenger of neutrino mass
generation Ma:2006km ; Hirsch:2013ola ; Merle:2016scw ; Bonilla:2016diq . We
propose a simple implementation of the bound-state dark matter scenario in
which DM constituents induce calculable neutrino mass at the radiative level.
Dark matter is a QCD bound state $\mathcal{Q}\mathcal{Q}$ stabilized by the
same conserved $B-L$ symmetry associated to the Dirac nature of neutrinos
Chulia:2016ngi .

In addition to the heavy Dirac fermion, our model introduces extra scalars, in
order to ensure that at least two neutrino masses are nonzero, required by the
neutrino oscillation data in order to account both for solar and atmospheric
mass scales deSalas:2017kay . This simple picture can account for current
neutrino oscillation and dark matter phenomena, and can be falsified
relatively soon, in nuclear recoil studies at XENON1T Aprile:2015uzo .
Moreover, the extra colored states, including scalar bosons, may lead to new
phenomena at a next generation hadron collider.

##  II The model

As a theory preliminary, we recall that, within the type-I seesaw mechanism
with a single right-handed neutrino, two neutrinos remain massless after the
seesaw Schechter:1980gr . This degeneracy is lifted by calculable loop
corrections 111An analogous situation happens in supersymmetry with bilinear
breaking of R parity Hirsch:2000ef ; diaz:2003as ; hirsch:2004he , which
induces the solar scale at one-loop, once atmospheric scale is taken as tree
level input.. Here we propose a variant radiative seesaw scheme, in which a
single colored fermion Dirac messenger $\cal Q$ is introduced, charged under
the $\operatorname{U}(1)_{B-L}$ symmetry, plus two sets of colored scalars,
labeled by $a=1,2$, see table 1.

Particles | $\operatorname{U}(1)_{B-L}$ | $\left(\operatorname{SU}(3)_{c},\operatorname{SU}(2)_{L}\right)_{Y}$ | $Z_{2}$
---|---|---|---
$Q_{i}=\begin{pmatrix}u_{L}&d_{L}\end{pmatrix}_{i}^{\operatorname{T}}$ | $+1/3$ | $\left(\mathbf{3},\mathbf{2}\right)_{1/6}$ | $+$
$\overline{u_{Ri}}$ | $-1/3$ | $\left(\overline{\mathbf{3}},\mathbf{1}\right)_{-2/3}$ | $+$
$\overline{d_{Ri}}$ | $-1/3$ | $\left(\overline{\mathbf{3}},\mathbf{1}\right)_{1/3}$ | $+$
$L_{i}=\begin{pmatrix}\nu_{L}&e_{L}\end{pmatrix}_{i}^{\operatorname{T}}$ | $-1$ | $\left(\mathbf{1},\mathbf{2}\right)_{-1/2}$ | $+$
$\overline{e_{Ri}}$ | $+1$ | $\left(\mathbf{1},\mathbf{1}\right)_{1}$ | $+$
$\overline{\nu_{Ri}}$ | $+1$ | $\left(\mathbf{1},\mathbf{1}\right)_{0}$ | $-$
$\mathcal{Q}_{L}$ | $-r$ | $\left(\mathbf{N}_{c},\mathbf{1}\right)_{0}$ | $+$
$\overline{\mathcal{Q}_{R}}$ | $r$ | $\left(\mathbf{N}_{c},\mathbf{1}\right)_{0}$ | $+$
$H$ | 0 | $\left(\mathbf{1},\mathbf{2}\right)_{1/2}$ | $+$
$\sigma_{a}$ | $1-r$ | $\left(\mathbf{N}_{c},\mathbf{1}\right)_{0}$ | $-$
$\eta_{a}$ | $1-r$ | $\left(\mathbf{N}_{c},\mathbf{2}\right)_{1/2}$ | $+$
Table 1: Left-handed fermions and scalars. When the vector-like quark $\cal Q$
is taken as a color octet, $\mathbf{N}_{c}=\mathbf{8}$, one can realize bound-
state dark matter scenario proposed in Ref. DeLuca:2018mzn .

Apart from the right-handed neutrinos, all of the new particles are colored.
For definiteness, we assign them to the octet $\operatorname{SU}(3)_{c}$
representation 222 Notice, however, that our neutrino mass discussion also
holds if they had different $\operatorname{SU}(3)_{c}$ transformation
properties.. We also impose that the $B-L$ symmetry holds, together with a
$Z_{2}$ symmetry. The former ensures that $\cal Q$ has only a Dirac-type mass
term, while the latter forbids the tree-level Dirac neutrino mass terms from
$\overline{\nu_{Ri}}\widetilde{H}^{\dagger}L_{j}$. The Lagragian contains the
following new terms (summation is implied over repeated indices, and trace
over $\mathbf{N}_{c}=\mathbf{8}$ is implicit.)

$\displaystyle\mathcal{L}\supset$
$\displaystyle-\left[h^{a}_{i}\overline{\mathcal{Q}_{R}}\widetilde{\eta}^{\dagger}_{a}L_{i}+M_{\mathcal{Q}}\,\overline{\mathcal{Q}_{R}}\mathcal{Q}_{L}+y^{a}_{i}\overline{\nu_{Ri}}\
\sigma_{a}^{*}\mathcal{Q}_{L}+\text{h.c}\right]$
$\displaystyle-\mathcal{V}(H,\eta_{a},\sigma_{a})\,.$ (1)

The condition $r\neq 1$ forbids Higgs-like Yukawa couplings of $\eta^{\alpha}$
to the Standard Model fermions (for $r=1$ one would need an additional $Z_{2}$
symmetry, as in Farzan:2012sa ). The new part of the scalar potential can be
cast as

$\displaystyle\mathcal{V}(H,\eta_{a},\sigma_{a})$
$\displaystyle=\mathcal{V}(\eta_{a})+\mathcal{V}(\sigma_{a})+\mathcal{V}(\eta_{a},\sigma_{a})$
$\displaystyle+\mathcal{V}(H,\eta_{a})+\mathcal{V}(H,\sigma_{a}),$ (2)

where the various terms in the Higgs potential are

$\displaystyle\mathcal{V}(\eta_{a},\sigma_{a})$
$\displaystyle=\kappa^{ab}\,\operatorname{Tr}\left(\sigma_{a}\eta_{b}^{\dagger}\right)H+\lambda_{\sigma\eta}^{abcd}\operatorname{Tr}\left(\sigma_{a}^{*}\sigma_{b}\right)\operatorname{Tr}\left(\eta^{\dagger}_{c}\eta_{d}\right)+\tilde{\lambda}_{\sigma\eta}^{abcd}\operatorname{Tr}\left(\sigma_{a}^{*}\eta_{b}\right)\operatorname{Tr}\left(\sigma_{c}\eta_{d}^{\dagger}\right)+\text{h.c.},$
(3) $\displaystyle\mathcal{V}(H,\eta_{a})$ $\displaystyle=\lambda_{3\eta
H}^{ab}\left(H^{\dagger}H\right)\operatorname{Tr}\left(\eta_{a}^{\dagger}\eta_{b}\right)+\lambda_{4\eta
H}^{ab}\operatorname{Tr}\left[\left(\eta_{a}^{\dagger}H\right)\left(H^{\dagger}\eta_{b}\right)\right]+\text{h.c.},$
(4) $\displaystyle\mathcal{V}(H,\sigma_{a})$ $\displaystyle=\lambda_{\sigma
H}^{ab}\operatorname{Tr}\left(\sigma_{a}^{*}\sigma_{b}\right)H^{\dagger}H+\text{h.c.}.$
(5)

Since CP conservation is assumed, the CP-even and CP-odd scalars do not mix.
Moreover, terms like $\left(\eta^{\dagger}H\right)^{2}$ are also forbidden
and, as a consequence, the real and imaginary parts of the scalars with
nonzero $B-L$ charges are degenerate. Note, however, that the cubic scalar
coupling terms $\kappa^{ab}$ breaking the $Z_{2}$ symmetry softly allow for
the mixing between the $\sigma_{a}$ and $\eta_{a}$. At the end the $4\times 4$
mass matrices for the CP-odd and CP-even scalars are equal, since the
$\kappa^{ab}$ terms do not break such a degeneracy.

In order to illustrate the neutrino mass generation mechanism we consider the
following block-diagonal mass matrix for the the CP-even scalars (in the basis
$S_{R}=(\eta_{1R}^{0},\sigma_{1R}^{0},\eta_{2R}^{0},\sigma_{2R}^{0})^{T}$):

$\displaystyle\mathcal{M}^{2}_{R}=\begin{pmatrix}(\mu_{\eta}^{11})^{2}&\frac{\kappa_{11}v}{\sqrt{2}}&0&0\cr\frac{\kappa_{11}v}{\sqrt{2}}&(\mu_{\sigma}^{11})^{2}&0&0\cr
0&0&(\mu_{\eta}^{22})^{2}&\frac{\kappa_{22}v}{\sqrt{2}}\cr
0&0&\kappa_{22}\frac{v}{\sqrt{2}}&(\mu_{\sigma}^{22})^{2}\end{pmatrix}.$ (6)

Here we have used the parametrization
$\eta_{a}=(\eta_{a}^{+},(\eta_{aR}^{0}+i\eta_{aI}^{0})/\sqrt{2})^{T}$,
$\sigma_{a}=(\sigma_{aR}^{0}+i\sigma_{aI}^{0})/\sqrt{2}$ and
$H=(0,(h+v)/\sqrt{2})^{T}$, with $h$ denoting the standard model Higgs boson,
and $v=246$ GeV. The parameters $\mu_{\eta(\sigma)}^{ab}$ are the quadratic
mass terms after electroweak symmetry breaking in
$\mathcal{V}\left(\eta_{a}\right)$ and $\mathcal{V}\left(\sigma_{a}\right)$.

Since the tree-level Dirac mass term is forbidden by symmetry, calculable
neutrino masses are generated at one–loop order, by the Feynman diagram
displayed in Fig. 1. One finds the following effective mass matrix

Figure 1: $B-L$ flux in the one-loop Dirac neutrino mass.

$\displaystyle\left(\mathcal{M}_{\nu}\right)_{ij}=$ $\displaystyle
N_{c}\frac{M_{\mathcal{Q}}}{64\pi^{2}}\sum_{a=1}^{2}h_{i}^{a}y_{j}^{a}\frac{\sqrt{2}\kappa_{aa}v}{m^{2}_{S_{2R}^{a}}-m^{2}_{S_{1R}^{a}}}$
$\displaystyle\left[F\left(\frac{m^{2}_{S_{2R}^{a}}}{M_{\mathcal{Q}}^{2}}\right)-F\left(\frac{m^{2}_{S_{1R}^{a}}}{M_{\mathcal{Q}}^{2}}\right)\right]+(R\to
I)$ (7)

where
$F(m_{S_{\beta}}^{2}/M_{Q}^{2})=m_{S_{\beta}}^{2}\log(m_{S_{\beta}}^{2}/M_{Q}^{2})/(m_{S_{\beta}}^{2}-M_{Q}^{2})$
and the $\operatorname{SU}(3)_{c}$ color factor $N_{c}$ is assumed to be $8$,
since the new particles running in the loop transform as octets. The four CP-
even mass eigenstates are denoted as
$S_{1R}^{1},S_{2R}^{1},S_{1R}^{2},S_{2R}^{2}$, with a similar notation for the
CP-odd ones. Notice that the above effective neutrino mass matrix in does not
have the projective rank one structure
$\left(\mathcal{M}_{\nu}\right)_{ij}^{\text{tree-level}}\propto
h_{i}^{a}y^{b}_{j}$ characterizing the Dirac variant Chulia:2016ngi  of the
“incomplete” tree-level seesaw mechanism, involving the exchange of a single
heavy messenger, called (3,1) in Ref. Schechter:1980gr  333Here one would
recover the rank-one situation characterizing the type-I seesaw in the limit
where one set of scalars decouples..

Here the effective one-loop induced neutrino mass matrix has in general rank
two, as in Hehn:2012kz , implying two non-vanishing Dirac neutrino masses. As
a simple numerical estimate, let’s consider the case
$\mu_{\eta}^{aa}=\mu_{\sigma}^{aa}\gg\sqrt{2}\kappa^{aa}v$. Taking
$m_{S_{2R}^{a}}^{2}-m_{S_{1R}^{a}}^{2}=\sqrt{2}\kappa^{aa}v$ and
$m_{S_{2R}^{a}}^{2}+m_{S_{1R}^{a}}^{2}=2(\mu_{\eta}^{aa})^{2}$ one finds

$\displaystyle\left(\mathcal{M}_{\nu}\right)_{ij}=$ $\displaystyle
N_{c}\frac{M_{\mathcal{Q}}}{32\pi^{2}}\sqrt{2}\kappa^{aa}v\sum_{a=1}^{2}\frac{h_{i}^{a}y_{j}^{a}}{(\mu_{\eta}^{aa})^{2}-M_{\mathcal{Q}}^{2}}$
$\displaystyle\left[1-\frac{M_{\mathcal{Q}}^{2}}{(\mu_{\eta}^{aa})^{2}-M_{\mathcal{Q}}^{2}}\log\left(\frac{(\mu_{\eta}^{aa})^{2}}{M_{\mathcal{Q}}^{2}}\right)\right],$

so that if $(\mu_{\eta}^{aa})^{2}\gg M_{\mathcal{Q}}^{2}$ one has

$\displaystyle\left(\mathcal{M}_{\nu}\right)_{ij}=N_{c}\frac{M_{\mathcal{Q}}}{32\pi^{2}}\sqrt{2}v\sum_{a=1}^{2}\kappa^{aa}\frac{h_{i}^{a}y_{j}^{a}}{(\mu_{\eta}^{aa})^{2}}$
(8) $\displaystyle\sim
0.03\,\mbox{eV}\left(\frac{M_{\mathcal{Q}}}{9.5\,\mbox{TeV}}\right)\left(\frac{\kappa^{aa}}{1\,\mbox{GeV}}\right)\left(\frac{50\,\mbox{TeV}}{\mu_{\eta}^{aa}}\right)^{2}\left(\frac{h_{i}^{a}y_{j}^{a}}{10^{-6}}\right).$

One sees that, indeed, small neutrino masses arise naturally by taking
reasonable values for the Yukawa couplings, small value for the soft breaking
parameter $\kappa^{ab}$, as well as sufficiently large values for the scalar
masses. Notice that the smallness of $\kappa^{ab}$ is natural, as the theory
attains a larger symmetry when $\kappa^{ab}\to 0$, i.e. the smallness of
neutrino mass is symmetry-protected.

In short, concerning neutrino mass generation, our model provides a colored
variant of the one suggested in Farzan:2012sa . However, although the neutrino
mass generation is similar in both models, the details of the associated
physics differ substantially.

##  III Bound-state dark matter stability from Dirac neutrinos

The Dirac nature of neutrinos may ensure dark matter stability, as suggested
in Chulia:2016ngi . Here we clone this idea with the proposal that dark matter
may be present today mainly in the form of stable neutral hadronic thermal
relics. For definiteness we assume DM is a neutral bound-state of colored
constituents, such as $\mathcal{Q}\mathcal{Q}$, where $\cal Q$ is a vector-
like color octet isosinglet fermion. It was claimed that a necessary and
suficient condition for dark matter stability in this case is the presence of
a global $U(1)_{D}$ dark baryon number, under which the $\cal Q$ is charged
DeLuca:2018mzn . In our present model construction the role of such apparently
ad-hoc symmetry is played by the usual ${B-L}$ symmetry present in the
Standard Model. In fact, in our model dark matter stability, and the Dirac
nature of the exotic fermion $\cal Q$ and of the neutrinos are all equivalent,
and result from $B-L$ conservation.

An adequate thermal relic density of bound-state dark matter requires the
lightest constituent vector-like color octet Dirac fermion, $\mathcal{Q}$, to
have a mass $\approx 9.5\ \text{TeV}$, so that the $\mathcal{Q}\mathcal{Q}$
hadron weighs approximately $19\ \text{TeV}$ DeLuca:2018mzn .

The set of scalars $\eta^{a}$, $\sigma^{a}$ in Fig. 1 can be either neutral or
charged under ${B-L}$, depending on the baryon number assignment. If ${B-L}$
neutral, both $\eta^{a}$ and $\sigma^{a}$ are expected to be unstable,
decaying to quarks Manohar:2006ga  and two gluons respectively Bityukov:1997dh
; Bai:2018jsr . In contrast, in the second case, if the lightest octet
particle were a scalar, then it would be strictly stable.

Bound-state dark matter will impart nuclear recoil in underground dark matter
search experiments. The spin-independent direct detection cross-section is
given as

$\sigma_{\text{SI}}\approx 2\times 10^{-45}\ \text{cm}^{2}\left(\frac{20\
\text{TeV}}{M_{\mathcal{Q}\mathcal{Q}}}\right)^{6}\frac{\Omega_{\mathcal{Q}\mathcal{Q}}}{\Omega_{\text{Planck}}}\,,$
(9)

sharply correlated to the dark matter mass
$M_{\mathcal{Q}\mathcal{Q}}=2M_{\mathcal{Q}}$, as shown in the red line in
Fig. 2.

Figure 2: The red line gives the spin independent cross section as a function
of $M_{\mathcal{Q}\mathcal{Q}}=2M_{\mathcal{Q}}$. The star denotes the mass
required for a thermal bound state DM (19 TeV). Smaller values can be probed
by direct searches, the blue line gives the current bound, while the black
(dashed, dotted and dot-dashed) lines represent future sensitivities.

In this figure the star corresponds to the case where our bound-state DM makes
up 100% of the cosmological dark matter. In the presence of an additional dark
matter particle, such as the axion, bound-state dark matter masses below 19
TeV can be envisaged, as indicated by the red line. In this case their
contribution to the relic density will be correspondingly smaller, while the
spin-independent cross section would be correspondingly larger. The blue line
represents the current limit of PandaX Cui:2017nnn . The black (dashed, and
dot-dashed) lines represent the future sensitivities expected at XENON1T
Aprile:2015uzo  and LZ Akerib:2018lyp . On the other hand we note that, within
the standard thermal cosmological scenario, DM masses above 19 TeV are ruled
out by current observations by the Planck collaboration Ade:2015xua  (gray
band).

Notice that the current LHC limit of 2 TeV (next section) implies that the
cross section is always small enough so as to have the bound-state dark matter
candidate reaching underground detectors. For more detailed discussion and
general aspects of the cosmology of a stable colored relic see Geller:2018biy
.

##  IV Color octets at hadron colliders

In our model the messengers of neutrino mass generation are the colored
constituents of bound-state dark matter. Given enough energy, the $\cal Q$ ’s
are copiously pair produced at hadron colliders, through the processes
$q_{i}\bar{q}_{i}\to\mathcal{Q}\bar{\mathcal{Q}}$ and
$gg\to\mathcal{Q}\bar{\mathcal{Q}}$, and are expected to hadronize. In
contrast to WIMP dark matter scenarios, which engender only missing-energy
signals, the bound-state dark matter scenario gives rise to very visible
signals at hadron colliders, as they can form either neutral or charged bound
states DeLuca:2018mzn , e.g. neutral hybrid states $\mathcal{Q}g$ (detected as
neutral hadrons, presumably stable) or charged $\mathcal{Q}q\bar{q}^{\prime}$
states, or more exotic $\mathcal{Q}qqq$ states, expected to be long-lived on
collider time-scales.

Current LHC data place a limit to the fermion color octet mass,
$M_{\mathcal{Q}}>2$ TeV CMS:2016ybj . Since the the cosmological relic
abundance requires $M_{\mathcal{Q}}\approx 9.5$ TeV, this scenario offers an
attractive benchmark for future collider experiments beyond the energies
attainable at the LHC. In fact, from the estimate in Ref. diCortona:2016fsn
one finds that a hadron collider of at least 65 TeV center-of-mass energy
would be required to probe the full cosmologically allowed range of masses of
our bound-state DM model. This will allow a cross-check of the DM search
results of XENON1T, expected quite soon, in just one year or so. Concerning
the scalar messengers, we have two pairs of these, the
$\sigma_{a}\sim(\mathbf{8},1,0)$, which are singlets under
$\operatorname{SU}(2)_{L}$, and the $\eta_{a}$, which transform as weak
doublets, $\eta_{a}\sim(\mathbf{8},\mathbf{2},1/2)$. As color octets, these
would also be copiously produced at a hadron collider of sufficient energy
Hayreter:2017wra . However, their masses are expected to lie well above the
reach of the LHC. Moreover, in our model these scalars carry non-trivial $B-L$
charges, see Table 1. This makes them relatively inert with respect to the
standard model fermions. This, in addition to their heavy masses, makes them
very difficult to probe directly.

##  V Lepton Flavor Violation

Our model may also lead to indirect virtual effects, such as charged lepton
flavor violation. For example, the Yukawa interactions in Eq. (II) lead to
radiative lepton flavour violation processes, as seen in Fig. 3, mediated by
the charged scalar $\eta^{+}_{a}$.

Figure 3: Feynman diagram for the process $l_{i}\to l_{j}\gamma$

The corresponding decay rate is given as Lavoura:2003xp ,

$\displaystyle\Gamma(l_{i}\to l_{j}\gamma)=$
$\displaystyle\frac{e^{2}m_{i}^{5}}{16\pi}\Big{[}\sum_{a}N_{c}h^{a}_{i}h^{a*}_{j}\frac{i}{16\pi^{2}M_{\eta_{a}^{+}}^{2}}$
$\displaystyle\left[\frac{-t^{2}\log
t}{2(t-1)^{4}}+\frac{2t^{2}+5t-1}{12(t-1)^{3}}\right]\Big{]}^{2}\,,$ (10)

with $t={M_{\mathcal{Q}}^{2}}/{M_{\eta^{+}}^{2}}$ and $N_{c}=8$. In the limit
of heavy scalars, $t\to 0$, the decay width reads

$\displaystyle\Gamma(l_{i}\to
l_{j}\gamma)=\frac{e^{2}m_{i}^{5}}{16\pi}\Big{[}\sum_{a}N_{c}h^{a}_{i}h^{a*}_{j}\frac{i}{16\pi^{2}M_{\eta_{a}^{+}}^{2}}\left[-\frac{1}{12}\right]\Big{]}^{2}.$

One sees that the current experimental constraint $\text{BR}(\mu\to
e\gamma)=\frac{\Gamma(\mu\to e\gamma)}{\Gamma_{\text{total}}}<5.7\times
10^{-13}$, can be fullfilled provided

$\left(\sum_{a}\frac{h^{a}_{\mu}h^{a*}_{e}}{M_{\eta_{a}^{+}}^{2}}\right)^{2}\leq
5.7\times
10^{-13}\frac{G_{F}^{2}}{\alpha_{\text{EM}}}\frac{768\pi}{N_{c}^{2}}~{},$ (11)

which leads to a relatively mild requirement,

$\left|\sum_{a}h^{a}_{\mu}h^{a*}_{e}\left(\frac{50\,\mbox{TeV}}{M_{\eta_{a}^{+}}}\right)^{2}\right|\lesssim
1.5.$ (12)

##  VI Summary and outlook

We have proposed a consistent viable theory for the recently proposed idea
that the cosmological dark matter may be made up of stable colored relics
forming neutral hadronic bound states of QCD. In our model we have taken up at
face value the suggestion in Ref. DeLuca:2018mzn , employing an exotic vector-
like Dirac color octet fermion $\cal Q$ with mass below 10 TeV as the dark
matter constituent. In our construction dark matter and neutrino mass
generation both have a common origin. Our minimum particle content leads to
two non-zero neutrino masses, that can be associated to the solar and
atmospheric scale. Bound-state dark matter stability is directly associated
with the Dirac nature of neutrinos, and reflects the presence of an underlying
exact $B-L$ symmetry. The scheme can account for both neutrino physics and
dark matter phenomena, within a consistent ultraviolet complete setup, free of
Landau poles up to the Planck scale, provided the scalars are heavy enough.
Our model can be falsified relatively soon by dark matter searches, and could
also be cross-checked later by a next generation hadron collider. Variants of
our construction may be envisaged, in which the dark matter is bound by a new
hypercolor interaction Mitridate:2017oky , instead of QCD, as suggested in
Ref. Reig:2017nrz .

###### Acknowledgements.

We thank Martin Hirsch for very useful discussions. Work supported by the
Spanish grants FPA2017-85216-P and SEV-2014-0398 (MINECO), Sostenibilidad-
UdeA, and by COLCIENCIAS through the Grants 11156584269 and 111577657253.

## References

  * (1) V. De Luca, A. Mitridate, M. Redi, J. Smirnov, and A. Strumia, “Colored Dark Matter,” arXiv:1801.01135 [hep-ph].
  * (2) M. Reig, J. W. F. Valle, C. A. Vaquera-Araujo, and F. Wilczek, “A Model of Comprehensive Unification,” Phys. Lett. B774 (2017) 667–670, arXiv:1706.03116 [hep-ph].
  * (3) E. Ma, “Verifiable radiative seesaw mechanism of neutrino mass and dark matter,” Phys.Rev. D73 077301, arXiv:hep-ph/0601225 [hep-ph].
  * (4) M. Hirsch et al., “WIMP dark matter as radiative neutrino mass messenger,” JHEP 1310 149, arXiv:1307.8134 [hep-ph].
  * (5) A. Merle et al., “Consistency of WIMP Dark Matter as radiative neutrino mass messenger,” JHEP 07 (2016) 013, arXiv:1603.05685 [hep-ph].
  * (6) C. Bonilla, E. Ma, E. Peinado, and J. W. F. Valle, “Two-loop Dirac neutrino mass and WIMP dark matter,” Phys. Lett. B762 (2016) 214–218, arXiv:1607.03931 [hep-ph].
  * (7) S. Centelles Chuliá et al., “Dirac Neutrinos and Dark Matter Stability from Lepton Quarticity,” Phys. Lett. B767 209–213, arXiv:1606.04543 [hep-ph].
  * (8) P. F. de Salas et al., “Status of neutrino oscillations 2017,” arXiv:1708.01186 [hep-ph].
  * (9) XENON Collaboration, E. Aprile et al., “Physics reach of the XENON1T dark matter experiment,” JCAP 1604 no. 04, (2016) 027, arXiv:1512.07501 [physics.ins-det].
  * (10) J. Schechter and J. W. F. Valle, “Neutrino Masses in SU(2) x U(1) Theories,” Phys. Rev. D22 (1980) 2227.
  * (11) M. Hirsch et al., “Neutrino masses and mixings from supersymmetry with bilinear R parity violation: A Theory for solar and atmospheric neutrino oscillations,” Phys. Rev. D62 (2000) 113008, arXiv:hep-ph/0004115 [hep-ph]. [Erratum: Phys. Rev.D65,119901(2002)].
  * (12) M. A. Diaz et al., “Solar neutrino masses and mixing from bilinear R-parity broken supersymmetry: Analytical versus numerical results,” Phys. Rev. D68 013009, hep-ph/0302021.
  * (13) M. Hirsch and J. W. F. Valle, “Supersymmetric origin of neutrino mass,” New J. Phys. 6 76, hep-ph/0405015.
  * (14) Y. Farzan and E. Ma, “Dirac neutrino mass generation from dark matter,” Phys. Rev. D86 (2012) 033007, arXiv:1204.4890 [hep-ph].
  * (15) D. Hehn and A. Ibarra, “A radiative model with a naturally mild neutrino mass hierarchy,” Phys. Lett. B718 (2013) 988–991, arXiv:1208.3162 [hep-ph].
  * (16) A. V. Manohar and M. B. Wise, “Flavor changing neutral currents, an extended scalar sector, and the Higgs production rate at the CERN LHC,” Phys. Rev. D74 (2006) 035009, arXiv:hep-ph/0606172 [hep-ph].
  * (17) S. I. Bityukov and N. V. Krasnikov, “The Search for new physics by the measurement of the four jet cross-section at LHC and FNAL,” Mod. Phys. Lett. A12 (1997) 2011–2028, arXiv:hep-ph/9705338 [hep-ph].
  * (18) Y. Bai and B. A. Dobrescu, “Collider Tests of the Renormalizable Coloron Model,” arXiv:1802.03005 [hep-ph].
  * (19) PandaX-II Collaboration, X. Cui et al., “Dark Matter Results From 54-Ton-Day Exposure of PandaX-II Experiment,” Phys. Rev. Lett. 119 no. 18, (2017) 181302, arXiv:1708.06917 [astro-ph.CO].
  * (20) LUX-ZEPLIN Collaboration, D. S. Akerib et al., “Projected WIMP sensitivity of the LUX-ZEPLIN (LZ) dark matter experiment,” arXiv:1802.06039 [astro-ph.IM].
  * (21) Planck Collaboration, P. A. R. Ade et al., “Planck 2015 results. XIII. Cosmological parameters,” Astron. Astrophys. 594 (2016) A13, arXiv:1502.01589 [astro-ph.CO].
  * (22) M. Geller, S. Iwamoto, G. Lee, Y. Shadmi, and O. Telem, “Dark quarkonium formation in the early universe,” arXiv:1802.07720 [hep-ph].
  * (23) CMS Collaboration, C. Collaboration, “Search for heavy stable charged particles with $12.9~{}\mathrm{fb}^{-1}$ of 2016 data,”.
  * (24) G. Grilli di Cortona, E. Hardy, and A. J. Powell, “Dirac vs Majorana gauginos at a 100 TeV collider,” JHEP 08 (2016) 014, arXiv:1606.07090 [hep-ph].
  * (25) A. Hayreter and G. Valencia, “LHC constraints on color octet scalars,” Phys. Rev. D96 no. 3, (2017) 035004, arXiv:1703.04164 [hep-ph].
  * (26) L. Lavoura, “General formulae for $f(1)\to f(2)\gamma$,” Eur. Phys. J. C29 (2003) 191–195, arXiv:hep-ph/0302221 [hep-ph].
  * (27) A. Mitridate, M. Redi, J. Smirnov, and A. Strumia, “Dark Matter as a weakly coupled Dark Baryon,” arXiv:1707.05380 [hep-ph].


Ginpagebox=artbox

11institutetext: Martin-Luther-Universität Halle-Wittenberg 22institutetext:
Leipzig University

# How Train–Test Leakage Affects Zero-shot Retrieval

Maik Fröbe  Christopher Akiki 11 Martin Potthast 22 Matthias Hagen 2211

###### Abstract

Neural retrieval models are often trained on (subsets of) the millions of
queries of the MS MARCO / ORCAS datasets and then tested on the 250 Robust04
queries or other TREC benchmarks with often only 50 queries. In such setups,
many of the few test queries can be very similar to queries from the huge
training data—in fact, 69% of the Robust04 queries have near-duplicates in MS
MARCO / ORCAS. We investigate the impact of this unintended train–test leakage
by training neural retrieval models on combinations of a fixed number of MS
MARCO / ORCAS queries that are highly similar to the actual test queries and
an increasing number of other queries. We find that leakage can improve
effectiveness and even change the ranking of systems. However, these effects
diminish the smaller and, thus, the more realistic the amount of leakage is
among all training instances.

###### Keywords:

Neural information retrieval; Train–test leakage; BERT; T5

##  1 Introduction

Training transformer-based retrieval models requires large amounts of data
unavailable in many traditional retrieval benchmarks [34]. Data-hungry
training regimes became possible with the 2019 release of MS MARCO [10] and
its 367,013 queries that were subsequently enriched by the ORCAS click log [8]
with 10 million queries. Fine-tuning models trained on MS MARCO to other
benchmarks or using them without fine-tuning in zero-shot scenarios is often
very effective [34, 36, 47]. For example, monoT5 [36], which has been trained
only on MS MARCO data, is currently the most effective model for the Robust04
document ranking task.111https://paperswithcode.com/sota/ad-hoc-information-
retrieval-on-trec-robust04 Furthermore, the reference implementations of
monoT5 and monoBERT [37] in retrieval frameworks such as PyTerrier [32] or
Pyserini / PyGaggle [26] all use models trained only on MS MARCO by default.
However, when MS MARCO was officially split into train and test data, cross-
benchmark use was not anticipated, so that MS MARCO’s training queries may
overlap with the test queries of other much smaller datasets (e.g., Robust04).
In this paper, we investigate the impact of such a train–test leakage by
training neural models on MS MARCO document ranking data with different
proportions of controlled leakage to Robust04 and the TREC 2017 and 2018
Common Core tracks as test datasets.

Figure 1: MS MARCO / ORCAS queries with high Sentence-BERT (SBERT) similarity
to Robust04 Topic 441.

To identify probably leaking queries, we run a semantic nearest-neighbor
search using Sentence-BERT [38] and compare each MS MARCO / ORCAS query to the
title, description, and manual query variants [3, 4] of the topics in Robust04
and the TREC 2017 and 2018 Common Core tracks. Figure 1 illustrates this
procedure for Topic 441 (lyme disease) from Robust04. Our manual review of the
leakage candidates shows that 69% to 76% of the topics have near-duplicates in
MS MARCO / ORCAS. To analyze the effect of this potential train–test leakage
on neural retrieval models, we create three types of training datasets per
test corpus, in variants with 1,000 to 128,000 training instances (query +
(non-)relevant document): (1) a fixed number of instances derived from test
queries from the test corpora (1000 for Robust04 and 200 for each of the two
Common Core tracks), augmented by other random non-leaking MS MARCO / ORCAS
instances to simulate an upper bound on train–test leakage effects, (2) a
fixed number of leaking MS MARCO / ORCAS instances (1000 for Robust04 and 200
each for the two Common Core tracks) supplemented by other random non-leaking
instances, and (3) random MS MARCO / ORCAS instances, ensuring that no
train–test leakage candidates are included.

In our experiments, we observe leakage-induced improvements in effectiveness
for Robust04 and the two Common Core tracks, which can even change the ranking
of systems. However, the average improvements in overall effectiveness are
often not significant and decrease as the proportion of leakage in the
training data becomes smaller and more representative of realistic training
scenarios. Nonetheless, our experiments on the effects of leaked instances on
search results and the resulting system rankings show that leakage effects
occur even when improvements in effectiveness are statistically negligible—a
strong argument that train–test leaks should be avoided in academic
experiments.222All code and data is publicly available at
https://github.com/webis-de/SPIRE-22.

##  2 Background and Related Work

Disjoint training, validation, and test datasets are essential to properly
evaluate the effectiveness of machine learning models [7]. Duplication between
training and test data can lead to incorrectly high “effectiveness” by
memorizing instances rather than learning the target concept. In practice,
though, train and test data often still contain redundancies. For text data,
paraphrases, synonyms, etc., can be especially problematic, resulting in
train–test leaks [19, 24, 29]. For instance, the training and test sets of the
ELI5 dataset [13] for question answering were created using TF-IDF as a
heuristic to eliminate redundancies between them. This proved insufficient as
81% of the test questions turned out to be paraphrases of training questions,
which clearly favored models that memorized the training data [24]. Recently,
Zhan et al. [46] found that 79% of the TREC 2019 Deep Learning track topics
have similar or duplicated queries in the training data and proposed new data
splits to evaluate the interpolation and extrapolation effectiveness of
models. However, not all types of train–test leaks are unintentional. The TREC
2017 and 2018 Common Core tracks [1] intentionally reused topics from Robust04
to allow participants to use the relevance judgments for training. Indeed,
approaches trained on the Robust04 judgments were more effective than others
[1]. In this paper, we study whether a similar effect can be observed for
unintentional leakage from the large MS MARCO and ORCAS datasets.

Training retrieval models on MS MARCO and applying them to another corpus is a
form of transfer learning [34]. Transfer learning is susceptible to train–test
leakage since the train and test data are often generated independently
without precautions to prevent leaks [6]. Research on leakage in transfer
learning focuses on membership inference [35, 41] (predicting if a model has
seen an instance during training) and property inference [2, 17] (predicting
properties of the training data). Both inferences rely on the observation that
neural models may memorize some training instances to generalize through
interpolation [5, 7] and to similar test instances [15, 16]. It is unclear
whether and how neural retrieval models in a transfer learning scenario are
affected by leakage. Memorized relevant instances might reduce effectiveness
for different test queries while improving it for similar queries, like the
examples in Figure 1. We take the first steps to investigate the effects of
such a train–test leakage.

When the target corpus contains only few training instances, transferred
retrieval models are often more effective without fine-tuning, in a zero-shot
setting [47]; for instance, when training on MS MARCO and testing on TREC
datasets [34, 36, 47]. A frequently used target TREC dataset is Robust04 [42]
with 250 topics and a collection of 528,155 documents published between 1989
and 1996 by the Financial Times, the Federal Register, the Foreign Broadcast
Information Service, and the LA
Times.333https://trec.nist.gov/data/cd45/index.html Later, the TREC Common
Core track 2017 [1] reused 50 of the 250 Robust04 topics on the New York Times
Annotated Corpus [39]444https://catalog.ldc.upenn.edu/LDC2008T19 (1,864,661
documents published between 1987 and 2007) and the Common Core track 2018
reused another 25 Robust04 topics (and introduced 25 new topics) on the
Washington Post Corpus555https://trec.nist.gov/data/wapost/ (595,037 documents
published between 2012 and 2017). A total of 311,410 relevance judgments were
collected for the Robust04 topics, 30,030 for the TREC 2017 Common Core track,
and 26,233 for the TREC 2018 Common Core track. Interestingly, every Robust04
topic and every topic from the Common Core tracks 2017 and 2018 was augmented
with at least eight query variants compiled by expert searchers, and made
available as an additional resource [3, 4].

Research on paraphrase detection [12, 43] and semantic question matching [40]
is of great relevance to the identification of potentially leaking queries
between training and test data. Reimers and Gurevych [38] and Lin et al. [28]
showed that pooling or averaging the output of contextual word embeddings of
pre-trained transformer encoders like BERT [11] is not suited for paraphrase
detection—both, with respect to efficiency and accuracy. Sentence-BERT [38]
solves this issue by adopting a BERT-based triplet network structure and a
contrastive loss that attempts to learn a global and a local structure suited
for detecting semantically related sentences. We therefore use Sentence-BERT
in a version specifically trained for paraphrase detection to identify leaking
queries.

##  3 Identifying Leaking Queries

To examine the impact of possible leaks from MS MARCO / ORCAS to the TREC
datasets Robust04 and Common Core 2017 and 2018, we compare the former’s
queries (367,013 plus 10 million) to the 275 topics of the latter three. Since
lexical similarity may not be sufficient, as indicated by the ELI5 issue [24]
mentioned above, we compute semantic similarity scores using Sentence-BERT
[38].666Of the available pre-trained Sentence-BERT models, we use the
paraphrase detection model: https://huggingface.co/sentence-
transformers/paraphrase-MiniLM-L6-v2 We store the Sentence-BERT embeddings of
all MS MARCO and ORCAS queries in two Faiss indexes [23] and query them for
the 100 nearest neighbors (exact; cosine similarity) of each topic’s title,
description, and query variants.

To determine the threshold for the Sentence-BERT similarity score beyond which
we consider a query a source of leakage for a topic, one human annotator
assessed whether a query is leaking for a TREC topic (title, description,
query variants) for a stratified sample of 100 pairs of queries and topics
with a similarity above 0.8. Against these manual judgments, a similarity
threshold of 0.91 is the lowest that yields a 0.9 precision for deciding that
a query is leaking for a topic. Table 3 shows the number of topics for which
queries above this threshold can be found. From MS MARCO and ORCAS combined,
3,960 queries are leakage candidates for one of 181 Robust04 topics (72% of
the 250 topics). From the two Common Core tracks, 37 and 38 topics have
leakage candidates (76% of the 50 topics, respectively)—high similarities
mostly against the query variants.

Table 1: Number of topics (T) in Robust04 and the TREC 2017 and 2018 Common
Core tracks for which similar queries (number as Q) in MS MARCO (MSM) and the
union of MSM and ORCAS (+ORC) exist in terms of the query having a Sentence-
BERT score > 0.91 against the topic’s title, description, or a query variant.

Candidates | Robust04 | Core 2017 | Core 2018
---|---|---|---
| MSM | \+ ORC | MSM | \+ ORC | MSM | \+ ORC
| T | Q | T | Q | T | Q | T | Q | T | Q | T | Q
Title | 33 |  183 | 140 | 1,775 | 2 | 12 | 23 | 176 | 2 |  22 | 21 | 110
Description |  42 |  113 |  168 |  3,350 | 0 |  10 |  30 |  600 | 0 |  20 |  21 |  972
Variants | 45 | 116 | 167 | 3,356 | 6 | 16 | 38 | 602 | 9 | 26 | 38 | 973
5pt. Union | 53 | 151 | 181 | 3,960 | 7 | 18 | 38 | 645 | 9 | 26 | 38 | 973

Some of these leakage candidates still are false positives (threshold
precision of 0.9). To only use actual leaking queries in our train–test
leakage experiments, two annotators manually reviewed the 5 most similar
candidates per topic above the 0.91 threshold (a total of 827 candidates; not
all topics had 5 candidates). Given the title, description, and narrative of a
topic, the annotators labeled the similarity of a query to the topic title
according to Jansen et al.’s reformulation types [22]: a query can be
_identical_ to the topic title (differences only in inflection or word order),
be a _generalization_ (subset of words), a _specialization_ (superset of
words), a _reformulation_ (some synonymous terms), or it can be on a
_different topic_. An initial kappa test on 103 random of the 827 candidates
showed moderate agreement (Cohen’s kappa of 0.59; 103 queries: we aimed for
100 but included all queries for a topic when one was selected). After
discussing the 103 cases with the annotators, they agreed on all cases and we
had them each independently label half of the remaining 724 candidates. Table
2 shows the annotation results: 172 topics of Robust04 (i.e., 69%) have
manually verified leaking queries (648 total), as well as 37 topics of Common
Core 2017 (74%) and 38 of Common Core 2018 (76%). A large portion of the true-
positive leaking queries are identical to or specializations of a topic’s
title (57.5% of 721). In our below train–test leakage experiments, we only use
manually verified true-positive leaking queries as the source of leakage from
MS MARCO / ORCAS.

Table 2: Statistics of the 827 manually annotated leakage candidate queries.
(a) Number of true and false candidates. (b) Annotated query reformulation
types.

(a) Manually annotated candidates.
---
Corpus | Candidates | Queries | Topics
Robust04 | true | 648 | 172
false |  093 |  053
Core 2017 | true | 138 |  037
false |  021 |  011
Core 2018 | true | 157 |  038
false |  019 |  007

(b) Reformulation types.
---
Type | Queries
Identical | 187
Generalization | 124
Specialization | 228
Reformulation | 182
Different Topic | 106

##  4 Experimental Analysis

Focusing on zero-shot settings, we train neural retrieval models on
specifically designed datasets to assess the effect of train–test leakage from
MS MARCO / ORCAS to Robust04 and TREC 2017 and 2018 Common Core. We analyze
the models’ effectiveness in five-fold cross-validation experiments, report
detailed results for varying training set sizes for monoT5 (which has the
highest effectiveness in our experiments), and study the effects of leaked
instances on the retrieval scores and the resulting rankings.

### Training Datasets.

For each of the three test scenarios (Robust04 and the two Common Core
scenarios), we construct three types of training datasets: (1) ‘No Leakage’
with random non-leaking queries (balanced between MS MARCO and ORCAS as in
previous experiments [8]), (2) ‘MSM Leakage’ with a fixed number of random
manually verified leaking queries from MS MARCO / ORCAS (500 queries for
Robust04, 100 queries for Common Core) supplemented by no-leakage queries till
a desired size is reached, and (3) ‘Test Leakage’ with a fixed number of
queries from the actual test data (500 for Robust04, 100 for Common Core;
oversampling: each topic twice (but different documents) to match ‘MSM
Leakage’) supplemented by no-leakage queries till a desired size is reached.
‘Test Leakage’ is meant as an “upper bound” for any train–test leakage effect.

For each type, we construct datasets with 1,000 to 128,000 instances (500 to
64,000 queries; each with one relevant and one non-relevant document). Since
MS MARCO / ORCAS queries only have annotated relevant documents, we follow
Nogueira et al. [36] and sample “non-relevant” instances from the top-100 BM25
results for such queries. For the ‘Test Leakage’ scenario, we use the actual
TREC judgments to sample the non-/relevant instances. In our 72 training
datasets (3 test scenarios, 3 types, 8 sizes), the number of leaked instances
is held constant to analyze the effect of a decreasing (and thus more
realistic) ratio of leakage. Larger training data would result in more costly
training, but our chosen sizes already suffice to observe a diminishing effect
of train–test leakage.

### Trained Models.

For our experimental analyses, we use models based on monoBERT [37] and monoT5
[36] as implemented in PyGaggle [26], and models based on Duet [33], KNRM
[44], and PACRR [21] as implemented in Capreolus [45] (default configurations
each). In pilot experiments with 32,000 ‘No Leakage’ instances, these models
had higher nDCG@10 scores on Robust04 than CEDR [30], HINT [14], PARADE [25],
and TK [20]. Following Nogueira et al. [36], we use the base versions of BERT
and T5 to spend the computational budget on training many models instead of a
single large one. Since the training is not deterministic, each model is
trained on each of the 72 training sets five times for one epoch with varying
seeds (used to shuffle the training queries; configured in PyTorch). We use
ir_datasets [31] for data wrangling and, following previously suggested
training regimes [36, 37, 45], pass the relevant and the non-relevant document
of a query consecutively to a model in the same batch during training. During
inference, all models re-rank the top-100 BM25 results (Capreolus, default
configuration) and we break potential score ties in rankings via alphanumeric
ordering by document ID (with random IDs, this leads to a random distribution
for other document properties such as text length [27]).

Figure 2: Effectiveness of monoT5 measured as nDCG@10 on the topics with
leakage (172 topics for Robust04, 37 and 38 for the 2017 and 2018 editions of
the Common Core track). Models trained on datasets of varying size with no
leakage (No), leakage from MS MARCO / ORCAS (MSM), or leakage from the test
data (Test).

### Leakage-Induced nDCG Improvements for MonoT5.

Figure 2 shows the average nDCG@10 of monoT5, the most effective model in our
experiments, for different training set sizes, tested only on topics with
leaked queries. For small training sets, monoT5 achieves rather low nDCG@10
values and cannot exploit the leakage. The nDCG@10 increases with more
training data on all benchmarks, peaking at 16,000 or 32,000 instances. At the
peaks, monoT5 trained with leakage is more effective than without, and
training on test leakage leads to a slightly higher nDCG@10 than leakage from
MS MARCO / ORCAS (MSM). However, the difference between test and MSM leakage
is larger for Robust04 (with some documents published as early as 1989)
compared to the newer Common Core tracks (with documents published closer to
the publication date of MS MARCO). On the Common Core data, MSM leakage is
almost as effective as test leakage.

### Leakage-Induced Effectiveness Improvements for Other Models.

We employ a five-fold cross-validation setup for Duet, KNRM, monoBERT, monoT5,
and PACRR to study whether leakage-induced effectiveness improvements can also
be observed for other models when a grid search in the cross-validation setup
can choose the training set size with the highest leakage effect for each
model. We report the effectiveness of the models as nDCG@10, Precision@1, and
the mean first rank of a relevant document (MFR) [18].777We use MFR instead of
the mean reciprocal rank (MRR) as suggested by Fuhr [18]. His criticism of MRR
was recently supported by further empirical evidence [48]. While effectiveness
scores measured via nDCG@10 and Precision@1 have the property that higher
values are better (a score of 1 indicates “best” effectiveness), for MFR,
lower scores are better—but still a score of 1 is the best case indicating
that the document on rank 1 always is relevant. In all effectiveness
evaluations, we conduct significance tests via Student’s t-test ($p=0.05$)
with Bonferroni correction for multiple testing as implemented in PyTerrier
[32].

Table 3: Effectiveness on Robust04 (R04) as nDCG@10, mean first rank of a relevant document (MFR), and Precision@1 (Prec@1) in a five-fold cross-validation setup on all test topics. Models are trained with no leakage (No), leakage from MS MARCO / ORCAS (MSM), or leakage from the test data (Test). Highest scores in bold; ${\dagger}$ marks Bonferroni-corrected significant differences to the no-leakage baseline (Student’s t-test, $p=0.05$). Model order swaps induced by MSM leakage in red. Model | nDCG@10 on R04 | MFR on R04 | Prec@1 on R04
---|---|---|---
| No | MSM | Test | No | MSM | Test | No | MSM | Test
Duet [33] | 0.201† | 0.198† | 0.224† | 2.420† | 2.682† | 2.340† | 0.297† | 0.261† | 0.304
KNRM [44] | 0.194† | 0.214† | 0.309† | 2.348† | 2.309† | 1.976† | 0.293† | 0.313† | 0.329
monoBERT [37] | 0.394† | 0.373† | 0.396† | 1.688† | 1.725† | 1.639† | 0.434† | 0.454† | 0.414
monoT5 [36] | 0.461† | 0.457† | 0.478† | 1.443† | 1.416† | 1.417† | 0.562† | 0.578† | 0.590
PACRR [21] | 0.382† | 0.364† | 0.391† | 1.663† | 1.604† | 1.579† | 0.458† | 0.462† | 0.502

Table 3 shows the five-fold cross-validated effectiveness on Robust04 for the
five models when optimizing each fold for nDCG@10, MFR, or Precision@1 in a
grid search. Models trained on test leakage almost always are more effective
than their no-leakage counterparts (exception: Precision@1 of monoBERT) and
actual test leakage usually helps more than leakage from MS MARCO / ORCAS
(MSM; exceptions: MFR of monoT5 and Precision@1 of monoBERT). Overall, on
Robust04, models trained with MSM leakage are often less effective than their
no-leakage counterparts (e.g., the nDCG@10 of monoBERT). A possible
explanation might be the large time gap between the Robust04 document
publication dates (documents published between 1987 and 2007) and the MS MARCO
data (crawled in 2018). A similar observation was made during the TREC 2021
Deep Learning track [9]. The transition from Version 1 of MS MARCO to Version
2 (crawled three years later) caused some models to prefer older documents
since they saw old documents as positive instances and newer ones as negative
instances during training. Still, MSM leakage can lead to swaps in model
ranking on Robust04. For instance, KNRM trained with MSM leakage achieves a
higher nDCG@10 and Precision@1 than Duet without leakage, while KNRM trained
without leakage is less effective than Duet.

Table 4: Effectiveness on Common Core 2017 (CC17) as nDCG@10, mean first rank of a relevant document (MFR), and Precision@1 (Prec@1) in a five-fold cross-validation setup on all test topics. Models are trained with no leakage (No), leakage from MS MARCO / ORCAS (MSM), or leakage from the test data (Test). Highest scores in bold; ${\dagger}$ marks Bonferroni-corrected significant differences to the no-leakage baseline (Student’s t-test, $p=0.05$). Model order swaps induced by MSM leakage in red. Model | nDCG@10 on CC17 | MFR on CC17 | Prec@1 on CC17
---|---|---|---
| No | MSM | Test | No | MSM | Test | No | MSM | Test
Duet [33] | 0.374† | 0.373† | 0.376 | 1.620 | 1.512 | 1.485† | 0.500 | 0.480 | 0.540
KNRM [44] | 0.316† | 0.343† | 0.330† | 1.587 | 1.512 | 1.568† | 0.480 | 0.520 | 0.480
monoBERT [37] | 0.402† | 0.407† | 0.419† | 1.625 | 1.605 | 1.634† | 0.480 | 0.460 | 0.460
monoT5 [36] | 0.445† | 0.464† | 0.490† | 1.363† | 1.384 | 1.359 | 0.660 | 0.620 | 0.680
PACRR [21] | 0.406† | 0.403† | 0.413† | 1.390 | 1.515 | 1.546† | 0.540 | 0.520 | 0.580

Table 4 shows the five-fold cross-validated effectiveness on the TREC 2017
Common Core track for the five models when optimizing each fold for nDCG@10,
MFR, or Precision@1 in a grid search. In contrast to Robust04, more models
improve their effectiveness when trained with MSM leakage as the time gap
between the New York Times Annotated Corpus and MS MARCO is smaller than for
Robust04. MonoT5 with actual test leakage is the most effective model for all
three measures, and monoT5 trained on MSM leakage is more effective than the
no-leakage counterpart in nDCG@10 and MFR. MSM leakage also may cause model
order swaps at higher positions in the systems’ nDCG@10 ordering: monoBERT
with MSM leakage would slightly overtake PACRR. Still, most of the
effectiveness improvements on this dataset caused by MSM leakage or test
leakage are not significant (exception: the nDCG@10 differences for monoT5
with test leakage and KNRM with MSM leakage to the no-leakage counterparts).

Table 5: Effectiveness on Common Core 2018 (CC18) as nDCG@10, mean first rank of a relevant document (MFR), and Precision@1 (Prec@1) in a five-fold cross-validation setup on all test topics. Models are trained with no leakage (No), leakage from MS MARCO / ORCAS (MSM), or leakage from the test data (Test). Highest scores in bold; ${\dagger}$ marks Bonferroni-corrected significant differences to the no-leakage baseline (Student’s t-test, $p=0.05$). Model order swaps induced by MSM leakage in red. Model | nDCG@10 on CC18 | MFR on CC18 | Prec@1 on CC18
---|---|---|---
| No | MSM | Test | No | MSM | Test | No | MSM | Test
Duet [33] | 0.285† | 0.301† | 0.295† | 1.993† | 1.812† | 2.231† | 0.320† | 0.380† | 0.260
KNRM [44] | 0.201† | 0.256† | 0.238† | 3.099† | 2.768† | 3.125† | 0.100† | 0.160† | 0.140
monoBERT [37] | 0.364† | 0.380† | 0.366† | 1.810† | 1.683† | 1.719† | 0.460† | 0.560† | 0.460
monoT5 [36] | 0.417† | 0.448† | 0.445† | 1.577† | 1.503† | 1.512† | 0.480† | 0.540† | 0.540
PACRR [21] | 0.376† | 0.406† | 0.393† | 1.649† | 1.383† | 1.485† | 0.520† | 0.560† | 0.540

Table 5 shows the five-fold cross-validated effectiveness on the TREC 2018
Common Core track for the five models when optimizing each fold for nDCG@10,
MFR, or Precision@1 in a grid search. In contrast to Robust04 and the 2017
edition of the Common Core track, training with MSM leakage improves the
effectiveness in all cases for all three measures. While most of the leakage-
induced effectiveness improvements are not statistically significant, the
model order even changes on the top MFR position, where PACRR with MSM leakage
would overtake monoT5 without leakage.

### Discussion.

The results in Tables 3–5 show that leakage from MS MARCO / ORCAS (MSM) can
have an impact on the retrieval effectiveness, even when only a small number
of instances are leaked, as in our experiments. While the changes on Robust04
are rather negligible, the impact is larger for the Common Core tracks with
document publication dates closer to the ones from MS MARCO. Interestingly,
MSM leakage-induced nDCG@10 improvements sometimes can lead to swaps in model
ordering despite the improvements not being significant in most cases. This
exemplifies that experimental effectiveness comparisons might be invalid when
some models had access to leaked instances during training.

Table 6: Mean rank of the (leaked) relevant training documents ($\mathbf{\pm}$ standard deviation) for models trained with and without leakage from MS MARCO / ORCAS (MSM leakage) or from the test data (test leakage). Ranks macro-averaged over all topics for test leakage and over all topics with leaking queries for MSM leakage. | Model | Robust04 | Common Core 17 | Common Core 18
---|---|---|---|---
|  | With | Without | With | Without | With | Without
MSM leak. | Duet | 41.70 $\mathbf{\pm}$45.88 | 46.79 $\mathbf{\pm}$46.51 | 34.52 $\mathbf{\pm}$32.67 | 35.98 $\mathbf{\pm}$32.93 | 43.39 $\mathbf{\pm}$33.52 | 45.67 $\mathbf{\pm}$32.99
KNRM | 82.36 $\mathbf{\pm}$31.88 | 84.74 $\mathbf{\pm}$30.15 | 43.24 $\mathbf{\pm}$31.74 | 43.68 $\mathbf{\pm}$31.50 | 53.12 $\mathbf{\pm}$32.14 | 53.45 $\mathbf{\pm}$32.14
monoBERT | 23.08 $\mathbf{\pm}$28.71 | 23.58 $\mathbf{\pm}$28.22 | 46.97 $\mathbf{\pm}$34.95 | 47.11 $\mathbf{\pm}$35.49 | 41.79 $\mathbf{\pm}$36.16 | 42.48 $\mathbf{\pm}$36.39
monoT5 | 20.13 $\mathbf{\pm}$26.77 | 20.15 $\mathbf{\pm}$26.64 | 35.68 $\mathbf{\pm}$31.69 | 36.46 $\mathbf{\pm}$31.88 | 29.86 $\mathbf{\pm}$28.24 | 30.31 $\mathbf{\pm}$28.27
PACRR | 42.41 $\mathbf{\pm}$44.86 | 42.43 $\mathbf{\pm}$44.67 | 35.79 $\mathbf{\pm}$33.71 | 36.28 $\mathbf{\pm}$33.83 | 34.76 $\mathbf{\pm}$36.45 | 35.70 $\mathbf{\pm}$36.87
Test leak. | Duet | 90.04 $\mathbf{\pm}$26.98 | 90.65 $\mathbf{\pm}$26.41 | 45.78 $\mathbf{\pm}$30.03 | 46.55 $\mathbf{\pm}$30.34 | 46.31 $\mathbf{\pm}$29.85 | 46.35 $\mathbf{\pm}$30.13
KNRM | 89.95 $\mathbf{\pm}$26.43 | 91.20 $\mathbf{\pm}$25.24 | 47.37 $\mathbf{\pm}$32.80 | 47.49 $\mathbf{\pm}$32.81 | 50.53 $\mathbf{\pm}$32.40 | 50.13 $\mathbf{\pm}$32.26
monoBERT | 47.01 $\mathbf{\pm}$31.84 | 47.39 $\mathbf{\pm}$31.80 | 46.64 $\mathbf{\pm}$31.51 | 47.12 $\mathbf{\pm}$31.51 | 43.19 $\mathbf{\pm}$31.51 | 44.04 $\mathbf{\pm}$31.66
monoT5 | 45.28 $\mathbf{\pm}$32.09 | 45.37 $\mathbf{\pm}$31.96 | 46.35 $\mathbf{\pm}$31.47 | 47.45 $\mathbf{\pm}$31.83 | 40.16 $\mathbf{\pm}$31.18 | 40.95 $\mathbf{\pm}$31.24
PACRR | 80.89 $\mathbf{\pm}$34.05 | 82.60 $\mathbf{\pm}$33.07 | 53.59 $\mathbf{\pm}$31.49 | 52.91 $\mathbf{\pm}$31.26 | 52.25 $\mathbf{\pm}$32.69 | 52.28 $\mathbf{\pm}$32.32

### Memorization of Leaked Instances.

To analyze whether the models memorize leaked instances, we compare the
retrieval scores and resulting ranks of leaked documents in the test rankings
when training includes or does not include leakage. For leaked documents not
returned in the top-100 BM25 results—the models only re-rank these—, we
determine a hypothetical rank by calculating the score of this document for
the query and inserting the document at the corresponding rank in the to-be-
re-ranked 100 documents (including breaking score-ties by document ID). Each
leaked document thus has a maximal rank of 101.

Table 6 shows the mean rank of relevant documents when they were included
during training (with leakage) or not (without leakage). Models perfectly
memorizing their positive training instances (i.e., relevant documents for
test queries) would rank these documents at substantially higher positions
than models that did not see the same instance during training. However, while
the mean rank of leaked relevant documents improves for most cases, the
improvement is mostly negligible. For instance, the mean rank of leaked
relevant documents for the very effective monoT5 and monoBERT models improves
only slightly compared to their no-leakage counterparts on all three corpora.
But the difference increases (still rather negligibly, though) on the corpora
on which leakage was more effective. In combination with the high standard
deviations, one can hardly see memorization effects for the positions of
leaked relevant documents in the final rankings. We thus also inspect the
retrieval scores of the leaked documents.

Table 7: Mean retrieval score of the (leaked) relevant training documents ($\mathbf{\pm}$ standard deviation; higher scores = “more relevant”) for models trained with / without leakage from MS MARCO / ORCAS (MSM) or the test data (Test). Scores macro-averaged over all topics for test leakage and over all topics with leaking queries for MSM leakage. | Model | Robust04 | Common Core 17 | Common Core 18
---|---|---|---|---
|  | With | Without | With | Without | With | Without
MSM leak. | Duet |  -0.89 $\mathbf{\pm}$1.22 |  -0.78 $\mathbf{\pm}$1.18 |  -0.52 $\mathbf{\pm}$1.18 |  -0.47 $\mathbf{\pm}$1.17 |  -0.16 $\mathbf{\pm}$0.79 |  -0.09 $\mathbf{\pm}$0.75
KNRM | -2.06 $\mathbf{\pm}$3.64 | -2.58 $\mathbf{\pm}$3.43 | -2.53 $\mathbf{\pm}$3.75 | -3.08 $\mathbf{\pm}$3.52 | -2.32 $\mathbf{\pm}$3.24 | -2.78 $\mathbf{\pm}$3.01
monoBERT | -0.75 $\mathbf{\pm}$0.44 | -0.72 $\mathbf{\pm}$0.41 | -0.88 $\mathbf{\pm}$0.49 | -0.85 $\mathbf{\pm}$0.47 | -0.92 $\mathbf{\pm}$0.50 | -0.89 $\mathbf{\pm}$0.48
monoT5 | -1.05 $\mathbf{\pm}$1.14 | -1.19 $\mathbf{\pm}$1.20 | -1.32 $\mathbf{\pm}$1.26 | -1.48 $\mathbf{\pm}$1.31 | -1.51 $\mathbf{\pm}$1.34 | -1.65 $\mathbf{\pm}$1.38
PACRR |  -2.59 $\mathbf{\pm}$3.25 |  -2.29 $\mathbf{\pm}$3.11 |  -2.78 $\mathbf{\pm}$3.35 |  -2.46 $\mathbf{\pm}$3.20 |  -2.25 $\mathbf{\pm}$3.08 |  -1.95 $\mathbf{\pm}$2.96
Test leak. | Duet |  -0.07 $\mathbf{\pm}$0.61 | -0.11 $\mathbf{\pm}$0.56 |  -0.22 $\mathbf{\pm}$0.68 | -0.01 $\mathbf{\pm}$0.66 |  -0.30 $\mathbf{\pm}$0.69 |  -0.09 $\mathbf{\pm}$0.68
KNRM | -2.71 $\mathbf{\pm}$3.79 | -3.41 $\mathbf{\pm}$3.71 | -2.78 $\mathbf{\pm}$3.65 | -3.28 $\mathbf{\pm}$3.63 | -3.08 $\mathbf{\pm}$4.14 | -3.59 $\mathbf{\pm}$4.14
monoBERT | -0.91 $\mathbf{\pm}$0.45 | -1.04 $\mathbf{\pm}$0.53 | -0.85 $\mathbf{\pm}$0.44 | -0.90 $\mathbf{\pm}$0.48 | -0.85 $\mathbf{\pm}$0.46 | -0.92 $\mathbf{\pm}$0.49
monoT5 | -1.70 $\mathbf{\pm}$1.24 | -2.37 $\mathbf{\pm}$1.53 | -1.47 $\mathbf{\pm}$1.09 | -1.98 $\mathbf{\pm}$1.37 | -1.52 $\mathbf{\pm}$1.24 | -2.01 $\mathbf{\pm}$1.50
PACRR |  -2.31 $\mathbf{\pm}$4.27 |  -1.92 $\mathbf{\pm}$3.09 |  -1.83 $\mathbf{\pm}$4.40 |  -1.98 $\mathbf{\pm}$3.13 |  -2.66 $\mathbf{\pm}$3.31 |  -2.26 $\mathbf{\pm}$3.23

Table 7 shows the mean retrieval score of the relevant documents when they
were included during training (with leakage) or not (without leakage). Models
that memorize the leaked relevant training documents should increase their
score, and we indeed observe that the retrieval score of leaked relevant
documents increases in most cases compared to their no-leakage counterpart
(exception: monoBERT for MSM leakage and PACRR for test leakage from Common
Core 2017). The difference between the score differences of leakage models and
non-leakage models is larger for leakage from the test data than for MSM
leakage in 13 of the 15 cases (with a maximum difference for monoT5 from a
test leakage difference of $0.67=2.37-1.70$ to an MSM leakage difference of
$0.14=1.19-1.05$). To investigate the “full picture” with respect to also
negative leaked instances (i.e., non-relevant documents), we next also study
the rank offsets between the positive and the negative leaked instances.

Table 8: Macro-averaged increase of the rank-offset between the leaked relevant and non-relevant documents ($\mathbf{\pm}$ standard deviation) for models trained on MSM leakage ($\Delta$ on MSM) or on test leakage ($\Delta$ on Test) over the no-leakage variants. Model |  $\Delta$ on MSM |  $\Delta$ on Test
---|---|---
| R04 | C17 | C18 | R04 | C17 | C18
Duet | 6.378 $\mathbf{\pm}$32.15 | 3.119 $\mathbf{\pm}$19.17 | 2.647 $\mathbf{\pm}$19.23 | 0.809 $\mathbf{\pm}$17.69 | 1.430 $\mathbf{\pm}$19.33 | 1.023 $\mathbf{\pm}$20.10
KNRM | 0.640 $\mathbf{\pm}$19.22 | 0.979 $\mathbf{\pm}$15.23 | 0.398 $\mathbf{\pm}$14.55 | 1.335 $\mathbf{\pm}$11.75 | 0.012 $\mathbf{\pm}$14.92 | 0.140 $\mathbf{\pm}$15.18
monoBERT | 0.692 $\mathbf{\pm}$17.97 | 0.076 $\mathbf{\pm}$17.19 | 0.369 $\mathbf{\pm}$20.04 | 3.886 $\mathbf{\pm}$20.39 | 0.980 $\mathbf{\pm}$17.44 | 3.497 $\mathbf{\pm}$25.98
monoT5 | 0.443 $\mathbf{\pm}$8.600 | 0.390 $\mathbf{\pm}$9.280 | 0.789 $\mathbf{\pm}$9.910 | 3.443 $\mathbf{\pm}$19.96 | 2.242 $\mathbf{\pm}$9.840 | 1.819 $\mathbf{\pm}$10.98
PACRR | 0.043 $\mathbf{\pm}$19.30 | 0.764 $\mathbf{\pm}$10.93 | 0.452 $\mathbf{\pm}$12.38 | 1.952 $\mathbf{\pm}$17.71 | 0.271 $\mathbf{\pm}$16.96 | 0.753 $\mathbf{\pm}$14.16

Table 8 shows the macro-averaged increase in the rank difference of the leaked
relevant and non-relevant documents between models trained with and without
leakage. The leakage increases the rank offset for all five analyzed models
(e.g., 6.4 ranks for Duet on Robust04 with MSM leakage). Interestingly, an in-
depth inspection showed that most of the increased differences are caused by
lowered ranks of the leaked non-relevant documents (e.g., 2 ranks lower for
monoT5) while the leaked relevant documents improve their ranks only slightly
(e.g., 0.3 ranks higher for monoT5).

### Discussion.

Overall, our results in Tables 6–8 indicate that memorization happens but has
little impact. Larger memorization effects might be desirable in practical
scenarios where a retrieval system that memorizes good results can simply
present them when the same query is submitted again. However, for empirical
evaluations in scientific publications or at shared tasks, (unintended)
leakage memorization at a larger scale might still lead to unwanted outcomes.

##  5 Conclusion

Our study of train–test leakage effects for neural retrieval models was
inspired by the observation that 69% of the Robust04 topics, a dataset often
used to test neural models, have very similar queries in the MS MARCO / ORCAS
datasets, that are often used to train neural models. At first glance, this
overlap might seem alarming since train–test leakage is known to invalidate
experimental evaluations. We thus analyzed train–test leakage effects for five
neural models (Duet, KNRM, monoBERT, monoT5, and PACRR) in scenarios with
different amounts of leakage. While our experiments show leakage-induced
effectiveness improvements that may even lead to swaps in model ranking, our
overall results are reassuring: the effects on nDCG@10 are rather small and
not significant in most cases. They also become smaller the smaller (and more
realistic) the amount of leakage among all training instances is. Still, even
if only a few nDCG@10 differences were significant, we noticed a memorization
effect: the rank offset between leaked relevant and non-relevant documents
increased on all scenarios.

Train–test leakage should thus still be avoided in academic experiments but
the practical consequences for real search engines might be different. The
observed increased rank offset might be a highly desirable effect when
presuming that queries already seen during training are submitted again after
a model has been deployed to production. An interesting direction for future
research is to enlarge our experiments to investigate more of the few cases
where train–test leakage slightly reduced the effectiveness.

## References

  * Allan et al. [2017] Allan, J., Harman, D., Kanoulas, E., Li, D., Gysel, C., Voorhees, E.: TREC 2017 Common Core track overview. In: Proc. of TREC 2017, vol. 500-324, NIST (2017)
  * Ateniese et al. [2015] Ateniese, G., Mancini, L., Spognardi, A., Villani, A., Vitali, D., Felici, G.: Hacking smart machines with smarter ones: How to extract meaningful data from machine learning classifiers. Int. J. Secur. Networks 10(3), 137–150 (2015)
  * Benham et al. [2017] Benham, R., Gallagher, L., Mackenzie, J., Damessie, T., Chen, R., Scholer, F., Moffat, A., Culpepper, J.: RMIT at the 2017 TREC CORE track. In: Proc. of TREC 2017, NIST Special Publication, vol. 500-324, NIST (2017)
  * Benham et al. [2018] Benham, R., Gallagher, L., Mackenzie, J., Liu, B., Lu, X., Scholer, F., Culpepper, J., Moffat, A.: RMIT at the 2018 TREC CORE track. In: Proc. of TREC 2018, NIST Special Publication, vol. 500-331, NIST (2018)
  * Berthelot et al. [2019] Berthelot, D., Raffel, C., Roy, A., Goodfellow, I.: Understanding and improving interpolation in autoencoders via an adversarial regularizer. In: Proc. of ICLR 2019, OpenReview.net (2019)
  * Chen et al. [2020] Chen, C., Wu, B., Qiu, M., Wang, L., Zhou, J.: A comprehensive analysis of information leakage in deep transfer learning. CoRR abs/2009.01989 (2020)
  * Chollet [2021] Chollet, F.: Deep Learning with Python. Simon and Schuster (2021)
  * Craswell et al. [2020] Craswell, N., Campos, D., Mitra, B., Yilmaz, E., Billerbeck, B.: ORCAS: 20 million clicked query-document pairs for analyzing search. In: Proc. of CIKM 2020, pp. 2983–2989, ACM (2020)
  * Craswell et al. [2021] Craswell, N., Mitra, B., Yilmaz, E., Campos, D.: Overview of the TREC 2021 Deep Learning Track. In: Voorhees, E.M., Ellis, A. (eds.) Notebook, NIST (2021)
  * Craswell et al. [2019] Craswell, N., Mitra, B., Yilmaz, E., Campos, D., Voorhees, E.: Overview of the TREC 2019 Deep Learning Track. In: Proc. of TREC 2019, NIST Special Publication, NIST (2019)
  * Devlin et al. [2019] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of deep bidirectional transformers for language understanding. In: Proc. of NAACL 2019, pp. 4171–4186, Association for Computational Linguistics, Minneapolis, Minnesota (2019)
  * Dolan and Brockett [2005] Dolan, W.B., Brockett, C.: Automatically constructing a corpus of sentential paraphrases. In: Proc. of the Third International Workshop on Paraphrasing (IWP 2005) (2005)
  * Fan et al. [2019] Fan, A., Jernite, Y., Perez, E., Grangier, D., Weston, J., Auli, M.: ELI5: Long form question answering. In: Proc. of ACL 2019, pp. 3558–3567, ACL (2019)
  * Fan et al. [2018] Fan, Y., Guo, J., Lan, Y., Xu, J., Zhai, C., Cheng, X.: Modeling diverse relevance patterns in ad-hoc retrieval. In: Proc. of SIGIR 2018, pp. 375–384, ACM (2018)
  * Feldman [2020] Feldman, V.: Does learning require memorization? A short tale about a long tail. In: Proc. of STOC 2020, pp. 954–959, ACM (2020)
  * Feldman and Zhang [2020] Feldman, V., Zhang, C.: What neural networks memorize and why: Discovering the long tail via influence estimation. In: Proc. of NeurIPS 2020 (2020)
  * Fredrikson et al. [2015] Fredrikson, M., Jha, S., Ristenpart, T.: Model inversion attacks that exploit confidence information and basic countermeasures. In: Proc. of CCS 2015, pp. 1322–1333, ACM (2015)
  * Fuhr [2017] Fuhr, N.: Some common mistakes in IR evaluation, and how they can be avoided. SIGIR Forum 51(3), 32–41 (2017)
  * He and Garcia [2009] He, H., Garcia, E.: Learning from imbalanced data. IEEE Trans. Knowl. Data Eng. 21(9), 1263–1284 (2009)
  * Hofstätter et al. [2020] Hofstätter, S., Zlabinger, M., Hanbury, A.: Interpretable & time-budget-constrained contextualization for re-ranking. In: Proc. of ECAI 2020, Frontiers in Artificial Intelligence and Applications, vol. 325, pp. 513–520, IOS Press (2020)
  * Hui et al. [2017] Hui, K., Yates, A., Berberich, K., Melo, G.: PACRR: A position-aware neural IR model for relevance matching. In: Proc. of EMNLP 2017, pp. 1049–1058, ACL (2017)
  * Jansen et al. [2009] Jansen, B., Booth, D., Spink, A.: Patterns of query reformulation during web searching. J. Assoc. Inf. Sci. Technol. 60(7), 1358–1371 (2009)
  * Johnson et al. [2021] Johnson, J., Douze, M., Jégou, H.: Billion-scale similarity search with GPUs. IEEE Trans. Big Data 7(3), 535–547 (2021)
  * Krishna et al. [2021] Krishna, K., Roy, A., Iyyer, M.: Hurdles to progress in long-form question answering. In: Proc. of NAACL 2021, pp. 4940–4957, ACL (2021)
  * Li et al. [2020] Li, C., Yates, A., MacAvaney, S., He, B., Sun, Y.: PARADE: Passage representation aggregation for document reranking. CoRR abs/2008.09093 (2020)
  * Lin et al. [2021] Lin, J., Ma, X., Lin, S., Yang, J., Pradeep, R., Nogueira, R.: Pyserini: A Python toolkit for reproducible information retrieval research with sparse and dense representations. In: Proc. of SIGIR 2021, pp. 2356–2362, ACM (2021)
  * Lin and Yang [2019] Lin, J., Yang, P.: The impact of score ties on repeatability in document ranking. In: Proc. of SIGIR 2019, pp. 1125–1128, ACM (2019)
  * Lin et al. [2020] Lin, S., Yang, J., Lin, J.: Distilling dense representations for ranking using tightly-coupled teachers. CoRR abs/2010.11386 (2020)
  * Linjordet and Balog [2020] Linjordet, T., Balog, K.: Sanitizing synthetic training data generation for question answering over knowledge graphs. In: Proc. of ICTIR 2020, pp. 121–128, ACM (2020)
  * MacAvaney et al. [2019] MacAvaney, S., Yates, A., Cohan, A., Goharian, N.: CEDR: Contextualized embeddings for document ranking. In: Proc. of SIGIR 2019, pp. 1101–1104, ACM (2019)
  * MacAvaney et al. [2021] MacAvaney, S., Yates, A., Feldman, S., Downey, D., Cohan, A., Goharian, N.: Simplified data wrangling with ir_datasets. In: Proc. of SIGIR 2021, pp. 2429–2436, ACM (2021)
  * Macdonald et al. [2021] Macdonald, C., Tonellotto, N., MacAvaney, S., Ounis, I.: PyTerrier: Declarative experimentation in Python from BM25 to dense retrieval. In: Proc. of CIKM 2021, pp. 4526–4533, ACM (2021)
  * Mitra et al. [2017] Mitra, B., Diaz, F., Craswell, N.: Learning to match using local and distributed representations of text for web search. In: Proc. of WWW 2017, pp. 1291–1299, ACM (2017)
  * Mokrii et al. [2021] Mokrii, I., Boytsov, L., Braslavski, P.: A systematic evaluation of transfer learning and pseudo-labeling with BERT-based ranking models. In: Proc. of SIGIR 2021, pp. 2081–2085, ACM (2021)
  * Nasr et al. [2019] Nasr, M., Shokri, R., Houmansadr, A.: Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning. In: Proc. of SP 2019, pp. 739–753, IEEE (2019)
  * Nogueira et al. [2020] Nogueira, R., Jiang, Z., Pradeep, R., Lin, J.: Document ranking with a pretrained sequence-to-sequence model. In: Findings of EMNLP 2020, vol. EMNLP 2020, pp. 708–718, ACL (2020)
  * Nogueira et al. [2019] Nogueira, R., Yang, W., Cho, K., Lin, J.: Multi-stage document ranking with BERT. CoRR abs/1910.14424 (2019)
  * Reimers and Gurevych [2019] Reimers, N., Gurevych, I.: Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In: Proc. of EMNLP 2019, pp. 3980–3990, ACL (2019)
  * Sandhaus [2008] Sandhaus, E.: The New York Times Annotated Corpus. Linguistic Data Consortium, Philadelphia 6(12), e26752 (2008)
  * Sharma et al. [2019] Sharma, L., Graesser, L., Nangia, N., Evci, U.: Natural language understanding with the Quora question pairs dataset. CoRR abs/1907.01041 (2019)
  * Shokri et al. [2017] Shokri, R., Stronati, M., Song, C., Shmatikov, V.: Membership inference attacks against machine learning models. In: Proc. of SP 2017, pp. 3–18, IEEE (2017)
  * Voorhees [2005] Voorhees, E.: The TREC Robust Retrieval track. SIGIR Forum 39(1), 11–20 (2005)
  * Wahle et al. [2021] Wahle, J.P., Ruas, T., Meuschke, N., Gipp, B.: Are neural language models good plagiarists? A benchmark for neural paraphrase detection. In: Proc. of JCDL 2021, pp. 226–229 (2021)
  * Xiong et al. [2017] Xiong, C., Dai, Z., Callan, J., Liu, Z., Power, R.: End-to-end neural ad-hoc ranking with kernel pooling. In: Proc. of SIGIR 2017, pp. 55–64, ACM (2017)
  * Yates et al. [2020] Yates, A., Arora, S., Zhang, X., Yang, W., Jose, K., Lin, J.: Capreolus: A toolkit for end-to-end neural ad hoc retrieval. In: Proc. of WSDM 2020, pp. 861–864, ACM (2020)
  * Zhan et al. [2022] Zhan, J., Xie, X., Mao, J., Liu, Y., Zhang, M., Ma, S.: Evaluating extrapolation performance of dense retrieval. CoRR abs/2204.11447 (2022)
  * Zhang et al. [2020] Zhang, X., Yates, A., Lin, J.: A little bit is worse than none: Ranking with limited training data. In: Proc. of SustaiNLP 2020, pp. 107–112, Association for Computational Linguistics (2020)
  * Zobel and Rashidi [2020] Zobel, J., Rashidi, L.: Corpus bootstrapping for assessment of the properties of effectiveness measures. In: Proc. of CIKM 2020, pp. 1933–1952, ACM (2020)