# LoL: A Comparative Regularization Loss over Query Reformulation Losses for
Pseudo-Relevance Feedback

Yunchang Zhu  Data Intelligence System Research Center, Institute of Computing
Technology, CASUniversity of Chinese Academy of SciencesBeijingChina
<EMAIL_ADDRESS>,  Liang Pang  Data Intelligence System Research
Center, Institute of Computing Technology, CASBeijingChina<EMAIL_ADDRESS>,  Yanyan Lan  Institute for AI Industry Research, Tsinghua
UniversityBeijingChina<EMAIL_ADDRESS>,  Huawei Shen  Data
Intelligence System Research Center, Institute of Computing Technology,
CASUniversity of Chinese Academy of SciencesBeijingChina<EMAIL_ADDRESS>and  Xueqi Cheng  CAS Key Lab of Network Data Science and Technology,
Institute of Computing Technology, CASUniversity of Chinese Academy of
SciencesBeijingChina<EMAIL_ADDRESS>
(2022)

###### Abstract.

Pseudo-relevance feedback (PRF) has proven to be an effective query
reformulation technique to improve retrieval accuracy. It aims to alleviate
the mismatch of linguistic expressions between a query and its potential
relevant documents. Existing PRF methods independently treat revised queries
originating from the same query but using different numbers of feedback
documents, resulting in severe query drift. Without comparing the effects of
two different revisions from the same query, a PRF model may incorrectly focus
on the additional irrelevant information increased in the more feedback, and
thus reformulate a query that is less effective than the revision using the
less feedback. Ideally, if a PRF model can distinguish between irrelevant and
relevant information in the feedback, the more feedback documents there are,
the better the revised query will be. To bridge this gap, we propose the Loss-
over-Loss (LoL) framework to compare the reformulation losses between
different revisions of the same query during training. Concretely, we revise
an original query multiple times in parallel using different amounts of
feedback and compute their reformulation losses. Then, we introduce an
additional regularization loss on these reformulation losses to penalize
revisions that use more feedback but gain larger losses. With such comparative
regularization, the PRF model is expected to learn to suppress the extra
increased irrelevant information by comparing the effects of different revised
queries. Further, we present a differentiable query reformulation method to
implement this framework. This method revises queries in the vector space and
directly optimizes the retrieval performance of query vectors, applicable for
both sparse and dense retrieval models. Empirical evaluation demonstrates the
effectiveness and robustness of our method for two typical sparse and dense
retrieval models.

query reformulation; pseudo-relevance feedback; regularization

††journalyear: 2022††copyright: rightsretained††conference: Proceedings of the
45th International ACM SIGIR Conference on Research and Development in
Information Retrieval; July 11–15, 2022; Madrid, Spain.††booktitle:
Proceedings of the 45th Int’l ACM SIGIR Conference on Research and Development
in Information Retrieval (SIGIR ’22), July 11–15, 2022, Madrid, Spain††doi:
10.1145/3477495.3532017††isbn: 978-1-4503-8732-3/22/07††ccs: Information
systems Query reformulation††ccs: Information systems Retrieval models and
ranking

##  1\. Introduction

Figure 1. An example of pseudo-relevant feedback. When the top 1 document
(potentially relevant) is used as the feedback, the reformulated query is
better than the original. But if more documents, such as the second
(potentially irrelevant), are added to the feedback set, it may cause query
drift.

In information retrieval (IR), users often formulate short and ambiguous
queries due to the reluctance and the difficulty in expressing their
information needs precisely in words (Carpineto and Romano, 2012). This may
specifically arise from several reasons, such as the use of inconsistent
terminology, commonly known as vocabulary mismatch (Furnas et al., 1987), or
the lack of knowledge in the area in which information is sought. Decades of
IR research demonstrate that such casual queries prevent a search engine from
correctly and completely satisfying users’ information needs (Croft et al.,
2010). To mitigate the mismatch of expressions between a query and its
potential relevant documents, many query reformulation approaches leveraging
on external resources (such as thesaurus and relevance feedback) have been
proposed to revise a better query – one that ranks relevant documents higher.

Pseudo-Relevance Feedback (PRF) (Attar and Fraenkel, 1977) has shown to be one
of effective query reformulation technique in various information retrieval
settings (Manning et al., 2008; Zhu et al., 2021; Fan et al., 2022). As the
name implies, no real relevant feedback from users is required in PRF, which
makes it more convenient and and widely studied. In the first-pass retrieval
of PRF, a small set of top-retrieved documents for an original query, called
the feedback set, is assumed to contain relevant information (Attar and
Fraenkel, 1977; CROFT and HARPER, 1979). The “pseudo” feedback set is then
exploited as external resources in the query reformulation process to form a
query revision, which is then run to retrieve the final list of documents
presented to the user. An example is shown in Figure 1 where the first
document introduces the synonymous term ‘COVID-19’ into the original query to
clarify the original query that contains ambiguous ‘Omicron’. Early PRF was
widely studied for sparse retrieval like vector space models (Rocchio, 1971),
probabilistic models (Robertson and Jones, 1976), and language modeling
methods (Lafferty and Zhai, 2001; Lavrenko and Croft, 2001; Zhai and Lafferty,
2001; Jaleel et al., 2004; Tao and Zhai, 2006; Lv and Zhai, 2014). Recently,
some work has shifted to apply PRF in dense retrieval of single-representation
(Yu et al., 2021; Li et al., 2021a, b) and multi-representation (Wang et al.,
2021).

Although PRF methods are generally accepted to improve average retrieval
effectiveness (Carpineto and Romano, 2012; Lv and Zhai, 2009b; Clinchant and
Gaussier, 2013; Yu et al., 2021), their performance is sometimes inferior to
the original query (Mitra et al., 1998; Tao and Zhai, 2006; Zighelnic and
Kurland, 2008). One of the causes for this robustness problem is query drift:
the topic of the revision drifts away from the original intent (Mitra et al.,
1998). This is not surprising, considering that many top-ranked documents can
be irrelevant and misleading, and relevant documents may contain irrelevant
information. For example in Figure 1, adding more feedback documents, e.g. the
second document, leads to a worse reformulated query, because an irrelevant
document introduces the noise term ‘Greek’ into the reformulated query which
totally changes the meaning of the original query. Therefore, PRF models need
to learn to suppress the irrelevant information in the feedback set and make
the most of the relevant information. Imagine an ideal PRF model, given a
larger feedback set in which both relevant and irrelevant information
increase, the model should form a better query revision.

Previous studies cope with query drift mainly by adding pre-processing or
post-processing (Mitra et al., 1998; Amati et al., 2004; Lv and Zhai, 2009a;
Collins-Thompson, 2009) or leveraging state-of-the-art pre-trained language
models (Wang et al., 2021; Yu et al., 2021). However, additional processing
brings more computational cost, and pre-trained language models may not
necessarily learn to suppress irrelevant information for retrieval without
particular supervision (Ma et al., 2021). Moreover, existing PRF methods
optimize different revisions of the same query independently by minimizing
their own reformulation losses, ignoring the comparison principle between
these revisions: the more feedback, the better the revision, a necessary
condition for an ideal PRF model.

Thus, to explicitly pursue this principle during training, we propose a
conceptual framework, namely Loss-over-Loss (LoL). This is a general
optimization framework applicable to any supervised PRF method. First, to
enable performance comparisons across revisions, the original query is revised
multiple times in a batch using feedback sets of different sizes. Then, we
impose a comparative regularization loss on all reformulation losses derived
from the same original query to penalize those revisions that use more
feedback but obtain larger losses. Specifically, the comparative
regularization is a pairwise ranking loss of these reformulation losses, where
the ascending order of reformulation losses is expected to coincide with the
descending order of the sizes of the feedback sets they use. With such
comparative regularization, we expect the PRF model to learn to suppress those
extra increased irrelevant in more feedback by comparing the effects of
different revisions. Furthermore, we present a differentiable PRF method as a
simple implementation of LoL. The method revises queries in the vector space,
thus avoiding the hassle of natural language generation and gradient back-
propagation, which makes it applicable for sparse retrieval as well as dense
retrieval. Besides, this method uses a ranking loss as the query reformulation
loss, which ensures the consistency of PRF with its ultimate goal, i.e.,
improving retrieval effectiveness.

To verify the effectiveness of our method, we evaluate two implemented LoL
models, one for sparse retrieval and the other for dense retrieval, on
multiple benchmarks based on MS MARCO passage collection. Experimental results
show that the retrieval performance of LoL models is significantly better than
their base models and other PRF models. Furthermore, we prove the critical
role of comparative regularization through ablation studies and visualization
of learning curves. Moreover, our analysis demonstrates that LoL is more
robust to the number of feedback documents compared to PRF baselines and is
not sensitive to the training hyper-parameters.

The main contributions can be summarized as follows:

  * •

A comparison principle is pointed out: the more feedback documents, the better
the effect of the reformulated query. Ignoring this principle during training
may cause PRF models to be misled by irrelevant information that appears in
more feedback, leading to query drift.

  * •

A comparative regularization is proposed to enhance the irrelevant information
suppression ability of PRF models, applicable for both sparse and dense
retrieval.

  * •

With the help of comparative regularization, our PRF models outperform their
base retrieval models and state-of-the-art PRF baselines on multiple
benchmarks based on MS MARCO. We release the code at
https://github.com/zycdev/LoL.

##  2\. Related Work

Query reformulation is the process of refining a query to improve ranking
effectiveness. According to the external resources used in the reformulation
process, there are two categories of methods: global and local (Xu and Croft,
1996). The first category of methods typically expands the query based on
global resources, such as WordNet (Gong et al., 2005), thesauri (Shiri and
Revie, 2006), Wikipedia (Aggarwal and Buitelaar, 2012), Freebase (Xiong and
Callan, 2015), and Word2Vec (Diaz et al., 2016). While the second category,
the so-called relevance feedback (Rocchio, 1971), is usually more popular. It
leverages local relevance feedback for the original query to reformulate a
query revision. Relevance feedback information can be obtained through
explicit feedback (e.g., document relevance judgments (Rocchio, 1971)),
implicit feedback (e.g., click-through data (Joachims, 2002)), or pseudo-
relevance feedback (assuming the top-retrieved documents contain information
relevant to the user’s information need (Attar and Fraenkel, 1977; CROFT and
HARPER, 1979)). Of these, pseudo-relevance feedback is the most common, since
no user intervention is required. Although pseudo-relevance feedback (PRF) is
also used for re-ranking (Li et al., 2018; Zheng et al., 2020), we next focus
on PRF methods in sparse and dense retrieval. Finally, we review existing
efforts to mitigate query drift.

###  2.1. PRF for Sparse Retrieval

Pseudo-relevance feedback methods for sparse retrieval have a long history,
dating back to Rocchio (Rocchio, 1971). The Rocchio algorithm is originally a
relevance feedback method proposed for vector space models (Salton et al.,
1975) but is also applicable to PRF. It constructs the refined query
representation through the linear combination of the sparse vectors of the
query and feedback documents. After that, many other heuristics were proposed.
For probabilistic models (Maron and Kuhns, 1960), the feedback documents are
naturally treated as examples of relevant documents to improve the estimation
of model parameters (Robertson and Jones, 1976). Whereas for language modeling
approaches (Ponte and Croft, 1998), PRF can be implemented by exploiting the
feedback set to estimate a query language model (Zhai and Lafferty, 2001; Lv
and Zhai, 2014) or relevance model (Lavrenko and Croft, 2001; Jaleel et al.,
2004). Overall, these methods expand new terms to the original query or/and
reweight query terms by exploiting statistical information on the feedback set
and the whole collection. Besides, some recent methods expand the query using
static (Zamani and Croft, 2016) or contextualized embeddings (Naseri et al.,
2021). For instance, CEQE (Naseri et al., 2021) uses BERT (Devlin et al.,
2019) to compute contextual representations of terms in the query and feedback
documents and then selects those closest to query embeddings as extension
terms according to some measure. But these methods are still heuristic because
they make strong assumptions to estimate the feedback weight for each term.
For example, the divergence minimization model (Zhai and Lafferty, 2001)
assumes that a term with high frequency in the feedback documents and low
frequency in the collection should be assigned a high feedback weight.
However, these assumptions are not necessarily in line with the ultimate
objective of PRF, i.e., improving retrieval performance.

Due to the discrete nature of natural language, the reformulated query text is
non-differentiable, making it difficult for supervised learning to optimize
retrieval performance directly. Therefore, (Montazeralghaem et al., 2020)
proposes a general reinforcement learning framework to directly optimize
retrieval metrics. To escape the expensive and unstable training of
reinforcement learning, a few supervised methods (Cao et al., 2008; Qi et al.,
2019) are optimized to generate oracle query revisions by selecting good terms
or spans from the feedback documents. However, these “oracle” query revisions
are constructed heuristically and do not necessarily achieve optimal ranking
results. Unlike all the above methods, our introduced method for sparse
retrieval refines the query in the vector space, enabling direct optimization
of retrieval performance in an end-to-end fashion.

###  2.2. PRF for Dense Retrieval

Dense retrieval has made great progress in recent years. Since dense
retrievers (Karpukhin et al., 2020; Xiong et al., 2020; Khattab and Zaharia,
2020) use embedding vectors to represent queries and documents, a few methods
(Wang et al., 2021; Yu et al., 2021; Li et al., 2021a, b) have been studied to
integrate pseudo-relevance information into reformulated query vectors.
ColBERT-PRF (Wang et al., 2021) first verified the effectiveness of PRF in
multi-representation dense retrieval (Khattab and Zaharia, 2020).
Specifically, it expands multi-vector query representations with
representative feedback embeddings extracted by KMeans clustering. (Li et al.,
2021a) investigated two simple methods, Average and Rocchio (Rocchio, 1971),
to utilize feedback documents in single-representation dense retrievers (e.g.,
ANCE (Xiong et al., 2020)) without introducing new neural models or further
training. Instead of refining the query vector heuristically, ANCE-PRF (Yu et
al., 2021) uses RoBERTa (Liu et al., 2019) to consume the original query and
the top-retrieved documents from ANCE (Xiong et al., 2020). Keeping the
document index unchanged, ANCE-PRF is trained end-to-end with relevance labels
and learns to optimize the query vector by exploiting the relevant information
in the feedback documents. Further, (Li et al., 2021b) investigate the
generalisability of the strategy underlying ANCE-PRF (Yu et al., 2021) to
other dense retrievers (Lin et al., 2020; Hofstätter et al., 2021). Although
our presented PRF method for dense retrieval uses the same strategy, all the
above methods are optimized to perform well on average, ignoring the
performance comparison between different versions of a query.

###  2.3. Coping with Query Drift

Query drift is a long-standing problem in PRF, where the topic of the
reformulated query drifts in an unintended direction mainly due to the
introduced irrelevant information from the feedback set (CROFT and HARPER,
1979; Mitra et al., 1998). There have been many studies on coping with query
drift. The strategies they typically employ include: (i) selectively
activating query reformulation to avoid performance damage to some queries
(Amati et al., 2004; Cronen-Townsend et al., 2004); (ii) refining the feedback
set to increase the proportion of relevant information in it (Mitra et al.,
1998; Zheng et al., 2020); (iii) varying the impact of the original query and
feedback documents to highlight query-relevant information (Tao and Zhai,
2006; Lv and Zhai, 2009a); (iv) post-processing the reformulated query to
eliminate risky expansions (Collins-Thompson, 2009); (v) model ensemble to
fuse results from multiple models (Collins-Thompson and Callan, 2007;
Zighelnic and Kurland, 2008); (vi) leveraging an advanced pre-trained language
model (Devlin et al., 2019; Liu et al., 2019) with the expectation that the
model itself to be immune to noise (Wang et al., 2021; Yu et al., 2021); (vii)
introducing a regularization term in the optimization objective to constrain
some principles (Tao and Zhai, 2006; Lv and Zhai, 2014). Our presented method
belongs to the last two strategies, introducing no additional processing
during inference. Unlike the other approaches in strategy (vi) that count on
the model to naturally learn to distinguish irrelevant information when
learning query reformulation, LoL also provides additional supervision on
comparing the effects of revisions. Moreover, to the best of our knowledge, we
are the first to impose comparative regularization on multiple revisions of
the same query.

##  3\. Methodlogy

This section describes our proposed framework for pseudo-relevant feedback
(PRF) and its implementation method. We first formalize the process of PRF and
introduce its traditional optimization paradigm. Then, we propose a general
conceptual framework for PRF, namely Loss-over-Loss (LoL). Finally, we present
an end-to-end query reformulation method based on vector space as an instance
of this framework and introduce its special handling for sparse and dense
retrieval.

###  3.1. Preliminary

Given an original textual query $q$ and a document collection $C$, a retrieval
model returns a ranked list of documents $D=(d_{1},d_{2},\cdots,d_{|D|})$. Let
$F^{(k)}=D_{\leq k}$ denote the feedback set containing the first $k$
documents, where $k\geq 0$ is often referred to as the PRF depth. The goal of
pseudo-relevant feedback is to reformulate the original query $q$ into a new
representation $q^{(k)}$ using the query-relevant information in the feedback
set $F^{(k)}$,

(1) $q^{(k)}=\mathrm{QR}(q,F^{(k)}),$

where $\mathrm{QR}$ is a query reformulation model based on PRF.

Denoting the reformulation loss of the revision $q^{(k)}$ as
$\mathcal{L}_{\mathrm{rf}}(q^{(k)})$, the general form of $\mathrm{QR}$ is to
optimize by minimizing the following loss function, which take multiple depths
of PRF into consideration:

(2) $\mathcal{L}(q)=\frac{1}{|K|}\sum_{k\in
K}\mathcal{L}_{\mathrm{rf}}(q^{(k)}),$

where $K$ is the set of PRF depths that $\mathrm{QR}$ needs to learn in one
epoch. For example, $K=\\{1,3,5\\}$ means that the loss considers the top-1,
top-3, and top-5 documents in the ranked list as the feedback set,
respectively.

However, $|K|$ is always set to 1 in many existing methods (Yu et al., 2021).
Specifically, existing PRF models are trained separately at each PRF depth,
where $K=\\{k\\}$ is a constant set and

(3) $\mathcal{L}(q)=\mathcal{L}_{\mathrm{rf}}(q^{(k)}).$

Taking it a little further, let $A\supseteq K$ be the set of all PRF depths
that a PRF model is designed to handle, e.g., $A=\\{1,2,3,4,5\\}$. If the PRF
model needs to be trained jointly at all PRF depths, we can sample a random
subset from $A$ as $K$ in each epoch.

###  3.2. Loss-over-loss Framework for PRF

Figure 2. The overview of the LoL framework. In the initial stage, the
retrieval model first generates a ranking list based on the original query. In
the reformulate stage, a list of top-k documents is selected to reformulate
the query. In the loss-over-loss regularization stage, a constrain is
constructed to ensure more feedback documents lead to small query
reformulation loss.

To prevent query drift caused by the increase of (irrelevant) feedback
information, we propose the Loss-over-Loss (LoL) framework. We first discover
a comparison principle that an ideal PRF model should guarantee but was
neglected in previous work. This principle describes the ideal comparative
relationship between revisions derived from the same query but using different
amounts of feedback. Therefore, we regularize the reformulation losses of
these revisions.

Suppose $\mathrm{RI}(q,F)$ denotes the information relevant to the query $q$
in the feedback set $F$, while $\mathrm{NRI}(q,F)$ represents the information
irrelevant to $q$ in $F$. Normally, as the PRF depth $k$ increases, both
relevant and irrelevant information increase, i.e.,
$\mathrm{RI}(q,F^{(k+1)})\supseteq\mathrm{RI}(q,F^{(k)})$ and
$\mathrm{NRI}(q,F^{(k+1)})\supseteq\mathrm{NRI}(q,F^{(k)})$. In this way, an
ideal PRF model immune to irrelevant information should be able to generate a
better revision due to more relevant information, which implies a smaller
reformulation loss. In short, the principle can be expressed as follows:

##### Comparison Principle

Given a larger feedback set $F^{(k+1)}\supseteq F^{(k)}$, an ideal PRF model
should generate a better revision $q^{(k+1)}$ whose reformulation loss is
less, i.e.,
$\mathcal{L}_{\mathrm{rf}}(q^{(k+1)})\leq\mathcal{L}_{\mathrm{rf}}(q^{(k)})$.

The above principle describes a necessary condition for an ideal PRF model,
i.e., a regular comparative relationship. Therefore, we try to constrain this
comparative relationship, which was ignored by the previous work, by means of
regularization. Instead of optimizing different revisions of the same query
independently, we optimize them collectively with a comparative
regularization. First, to allow comparison between multiple revisions, at each
epoch, we randomly sample $|K|>1$ distinct PRF depths from the full set $A$
without replacement. Then, $|K|$ revisions $\\{q^{(k)}|k\in K\\}$ of the same
query $q$ are reformulated in parallel by the PRF model in the same batch, and
their reformulation losses are calculated as $\\{\mathcal{L}(q^{(k)})|k\in
K\\}$. Finally, we regularize these losses to pursue the above principle
during training. Specifically, we add the following comparative regularization
term to the reformulation losses,

(4) $\mathcal{L}_{\mathrm{cr}}(q)=\frac{1}{{|K|\choose 2}}\sum_{j,k\in K\atop
j<k}\max\big{(}0,\mathcal{L}_{\mathrm{rf}}(q^{(k)})-\mathcal{L}_{\mathrm{rf}}(q^{(j)})\big{)}.$

As shown in Figure 2, the regularization term $\mathcal{L}_{\mathrm{cr}}(q)$
can be viewed as a pairwise hinge (Herbrich et al., 2000) ranking loss of
these reformulation losses, where the ascending order of these losses is
expected to coincide with the descending order of the feedback amounts they
use. Intuitively, it is reasonable if the revision $q^{(k)}$ using a larger
feedback set obtains no larger reformulation loss than the revision $q^{(j)}$
using a smaller feedback set, and we should not penalize revision $q^{(k)}$.
Otherwise, we penalize $q^{(k)}$ with the loss difference
$\mathcal{L}_{\mathrm{rf}}(q^{(k)})-\mathcal{L}_{\mathrm{rf}}(q^{(j)})$
between them, while rewarding $q^{(j)}$ at the same time. With such
comparative regularization, we expect the PRF model to learn to keep the
reformulation loss non-increasing with respect to the size of the feedback set
by comparing the losses (effects) of different revisions and consequently
learn to suppress increased irrelevant information from a larger feedback set.

In summary, the loss function of LoL consists of the conventional
reformulation loss and the newly proposed comparative regularization term,
formally written as follows,

(5) $\mathcal{L}(q)=\frac{1}{|K|}\sum_{k\in
K}\mathcal{L}_{\mathrm{rf}}(q^{(k)})+\lambda\mathcal{L}_{\mathrm{cr}}(q),$

where $\lambda$ is a weight that adjusts the strength of the comparative
regularization. When we set $\lambda$ to 0 and $|K|$ to 1, Equation (5) can
degenerate to Equation (3).

###  3.3. A Differentiable PRF Method under LoL

The LoL framework can be used for the training of any PRF model as long as the
query reformulation loss is differentiable. Here, we present a simple LoL
implementation for both dense and sparse retrieval.

####  3.3.1. Query Reformulation Loss

The ultimate goal of PRF is to improve retrieval effectiveness. Generally,
given a textual query $q$ and a document $d$, the similarity score between
them in a single-representation retrieval model is calculated as the dot
product of their vectors:

(6) $f(q,d)=\boldsymbol{q}\cdot\boldsymbol{d},$

where $\boldsymbol{q}$ and $\boldsymbol{d}$ are their encoded vector
representations. In dense retrieval, $\boldsymbol{q}$ and $\boldsymbol{d}$ are
dense low-dimensional vectors, while in sparse retrieval, they are sparse
high-dimensional vectors whose dimensions are the size of the vocabulary.
Notably, PRF only improves the representation of the query, while the vector
representation of all documents in the collection keeps invariant.

To ensure that the training objective of query reformulation is consistent
with the ultimate objective of PRF, we define the reformulation loss for a
revision $q^{(k)}$ derived from the original query $q$ as a ranking loss:

(7)
$\mathcal{L}_{\mathrm{rf}}(q^{(k)})=-\log\frac{e^{f(q^{(k)},d^{+})}}{e^{f(q^{(k)},d^{+})}+\sum_{d^{-}\in
D^{-}}e^{f(q^{(k)},d^{-})}},$

where $d^{+}$ is the positive document relevant to $q$ and $q^{(k)}$, and
$D^{-}$ is the collection of negative documents for them. Optimizing this
reformulation loss is trivial for dense retrieval, which inherently operates
in the vector space. However, since natural language queries are non-
differentiable, optimizing this loss for sparse retrieval is tricky.

Considering that the query text will eventually be encoded as a vector at
retrieval time, we skip the generation of the query text and directly refine
the hidden representation of the query in the vector space as in dense
retrieval approaches (Karpukhin et al., 2020; Xiong et al., 2020). In other
words, the vector $q^{(k)}$ refined by the PRF model $\mathrm{QR}$, hereafter
we call it $\boldsymbol{q}^{(k)}$, will serve as the vector of the
reformulated query in the second-pass retrieval. In this way, we eliminate
both the natural language generator that generates the textual revision and
the query encoder that encodes the revised text. More importantly, we
guarantee the differentiability of the reformulation loss in Equation (7),
which allows us to train the PRF model end-to-end.

####  3.3.2. PRF Model

In the following, we describe a simple instance of the PRF model $\mathrm{QR}$
in Equation (1), which encodes the original query and the feedback set into a
sparse or dense revision vector.

In general, the PRF model consists of an encoder, a vector projector, and a
pooler. We first leverage a BERT-style encoder to encode all tokens in the
original query $q$ and the feedback set $F^{(k)}$ into contextualized
embeddings:

(8) $\boldsymbol{H}=\mathrm{BERT}(\verb|[CLS]|\circ q\circ\verb|[SEP]|\circ
d_{1}\circ\verb|[SEP]|\circ\ldots\circ d_{k}\circ\verb|[SEP]|).$

Then, contextualized embeddings $\boldsymbol{H}$ are projected to vectors with
the same dimension as indexed document vectors:

(9) $\boldsymbol{V}=\mathrm{projector}(\boldsymbol{H}).$

Finally, all vectors are pooled into a single vector representation:

(10) $\boldsymbol{q^{(k)}}=\mathrm{pooler}(\boldsymbol{V}).$

For sparse retrieval, the projector is a MLP with GeLU activation and layer
normalization, initialized from a pre-trained masked language model layer. And
the pooler is composed of a max pooling operation and a L2 normalization111We
experimentally find that L2 normalization helps the model train stably, and it
does not change the relevance ranking of documents to a query.. While for
dense retrieval, the projector is a linear layer, and the pooler applies a
layer normalization on the first vector (`[CLS]`) in the sequence, as in the
previous work (Xiong et al., 2020; Yu et al., 2021).

##  4\. Experimental Setup

This section describes the datasets, evaluation metrics, baselines, and
details of our implementations.

###  4.1. Datasets

Experiments are conducted on MS MARCO passage (Nguyen et al., 2016)
collection, which includes 8.8M English passages from web pages gathered from
Bing’s results to 1M real-world queries. We train our models with the MS MARCO
Train set, which includes 530K queries with shallow annotation (~1.1 relevant
passages per query in average). The trained models are first evaluated on the
MS MARCO Dev set containing 6980 queries to tune hyper-parameters and select
model checkpoints. We then evaluate the selected models on the MS MARCO online
Eval set and three offline benchmarks (DL-HARD (Mackie et al., 2021), TREC DL
2019 (Craswell et al., 2020) and TREC DL 2020 (Craswell et al., 2021)). MS
MARCO Eval222https://microsoft.github.io/MSMARCO-Passage-Ranking-
Submissions/leaderboard/, TREC DL 2019, TREC DL 2020 and DL-HARD contain 6837,
43, 54 and 50 labeled queries, respectively. Unlike MS MARCO, whose relevance
judgments are binary, the other three benchmarks provide fine-grained
annotations (relevance grades from 0 to 3) for each query. Among them, DL-HARD
(Mackie et al., 2021) is a recent evaluation dataset focusing on challenging
and complex queries.

###  4.2. Evaluation Metrics

The official metric of MS MARCO (Nguyen et al., 2016) is MRR@10, and the main
metric of TREC DL (Craswell et al., 2020, 2021) and DL-HARD (Craswell et al.,
2021) is NDCG@10. MRR@10 is also the criterion used to select our models.
Since we focus on PRF for first-stage retrieval, we report Recall@1K for all
benchmarks. To compute the recall metric for TREC DL and DL-HARD, we treat
documents with relevance grade 1 as irrelevant following (Craswell et al.,
2020, 2021). To evaluate the robustness of PRF methods, we use the robustness
index (RI) (Collins-Thompson, 2009). RI is defined as
$\frac{N_{+}-N_{-}}{|Q|}$, where $|Q|$ is the total number of queries and
$N_{+}$ and $N_{-}$ are the number of queries that are improved or degraded by
the PRF method. The value of RI is always in the [-1, 1] interval, and methods
with higher values are more robust. Statistically significant differences in
performance are determined using the paired t-test.

###  4.3. Baselines

Since in this paper we only provide one implementation of LoL for single-
representation retrieval, we do not consider baselines of re-ranking and
multi-representation retrieval.

For base retrieval models without feedback, we mainly consider BM25 (Robertson
and Zaragoza, 2009), uniCOIL + docT5query (Lin and Ma, 2021), and ANCE (Xiong
et al., 2020). The first two are lexical sparse retrieval models, while ANCE
is a single-representation dense retrieval model.

  * •

BM25 (Robertson and Zaragoza, 2009) is heuristic bag-of-words retrieval model
with frequency-based signals to estimate the weights of terms in a text.

  * •

uniCOIL + docT5query (Lin and Ma, 2021) is a state-of-the-art trainable BERT-
based term weighting model that encodes both queries and documents as
30522-dimensional sparse vectors, where each dimension is a term tokenized by
WordPiece. Instead of PRF for query expansion, it expands all documents before
indexing through docT5query (Nogueira et al., 2019), which leverages the
powerful T5 (Raffel et al., 2020) language model to generate queries for
document expansion.

  * •

ANCE (Xiong et al., 2020) is a popular RoBERTa-based (Liu et al., 2019) dense
retrieval model that learns to generate a single-representation vector for
each query and document via mining hard negatives from asynchronously updated
document index built by the latest model checkpoint.

For the PRF models, we consider three heuristic methods (RM3, Rocchio and
Average) and one supervised learning method (ANCE-PRF) based on the retrieval
model described above.

  * •

RM3 (Jaleel et al., 2004) is an effective relevance model for traditional
sparse retrieval. We apply RM3 on BM25, serving as a representative method for
classic PRF.

  * •

Rocchio (Rocchio, 1971) and Average are the other two heuristic PRF methods,
and have been investigated for ANCE by (Li et al., 2021a).

  * •

ANCE-PRF (Yu et al., 2021) is currently the strongest PRF method for single-
representation retrieval. Keeping the document index of ANCE unchanged, ANCE-
PRF is trained end-to-end with relevance labels and learns to optimize the
revised query vector by exploiting the relevant information in the feedback
documents.

We also evaluate two variants of standard LoL ($\lambda>0,|K|>1$):

  * •

LoL w/o Reg ($\lambda=0,|K|>1$) does not impose the comparative regularization
in Equation (4) but has multiple parallel revisions derived from the same
query in each batch.

  * •

LoL w/o Par ($\lambda=0,|K|=1$) does not revise the same original query
multiple times in parallel in each batch, but unlike ANCE-PRF, it is still
trained jointly for all PRF depths.

Table 1. Overall retrieval results. The best results in each group are marked
in bold. We reproduce all baseline results, except for ANCE-PRF, Rocchio, and
Average, whose results are reported in previous work and not available for
significance tests. Superscript $\ast$ indicates statistically significant
improvements over its base retrieval model with $p\leq 0.05$.

Model | MARCO Dev | MARCO Eval | TREC DL 2019 | TREC DL 2020 | DL-HARD
---|---|---|---|---|---
Retrieval | PRF | NDCG@10 | MRR@10 | R@1K | MRR@10 | NDCG@10 | R@1K | NDCG@10 | R@1K | NDCG@10 | R@1K
BM25 (Robertson and Zaragoza, 2009) | - | 23.40 | 18.74 | 85.73 | 18.60 | 49.73 | 74.50 | 48.76 | 80.31 | 28.97 | 67.83
RM3 (Jaleel et al., 2004) | 21.35 | 16.68 | 86.86∗ | - | 52.31∗ | 77.92∗ | 48.08 | 82.86∗ | 26.63 | 69.47∗
uniCOIL+ docT5query (Lin and Ma, 2021) | - | 41.21 | 35.13 | 95.81 | 34.42 | 70.09 | 82.83 | 67.35 | 84.42 | 35.96 | 76.85
LoL(3) | 42.02∗ | 35.75∗ | 96.91∗ | 35.14 | 70.10 | 83.58 | 69.70 | 84.51 | 36.90 | 77.67
ANCE (Xiong et al., 2020) | - | 38.76 | 33.01 | 95.84 | 31.70 | 64.76 | 75.70 | 64.58 | 77.64 | 33.39 | 76.65
Average(3) (Li et al., 2021a) | - | - | 94.90 | - | - | 77.39 | - | 79.09 | - | -
Rocchio(5) (Li et al., 2021a) | - | - | 95.45 | - | - | 78.25 | - | 79.57 | - | -
ANCE-PRF(3) (Yu et al., 2021) | 40.10 | 34.40 | 95.90 | 33.00 | 68.10 | 79.10 | 69.50 | 81.50 | 36.50 | 76.10
LoL(3) | 40.68∗ | 34.84∗ | 96.94∗ | - | 68.42 | 80.10∗ | 69.58∗ | 81.77∗ | 35.61 | 79.39∗
LoL(5) | 41.01∗ | 35.14∗ | 97.03∗ | 34.17 | 69.58∗ | 80.81∗ | 70.44∗ | 82.77∗ | 37.44∗ | 79.55∗

###  4.4. Implementation Details

To ensure that gradients can be back-propagated during training, we perform
real-time retrieval by multiplying the query matrix and the document
matrix333For sparse retrieval, the document matrix is stored in sparse formats
of PyTorch.. The query matrix consists of all revised query vectors in a
batch, and the document matrix contains pre-computed vectors of all positive
and mined negative documents. These negative documents are the union of the
top-ranked documents retrieved by BM25, uniCOIL + docT5query, and ANCE. At
training time, $D^{-}$ in Equation (7) contains all documents in the document
matrix except the relevant documents for that query. Since document vectors do
not need to be updated, we can mine as many negative documents as possible, as
long as it does not exceed the memory limit of GPUs or retrieval is too slow.

We train two PRF models using the presented LoL implementation on 4 Tesla V100
GPUs with 32GB memory for up to $\frac{12}{|K|}$ epochs444Regardless of $|K|$,
all original queries are revised at most 12 times during training., one model
for sparse retrieval with document expansion (uniCOIL + docT5query) and the
other for dense retrieval (ANCE). During training, one GPU is dedicated to
retrieval, and the other three are used for the PRF model to revise query
vectors. The document matrices are converted from the pre-built inverted or
dense indexes provided by pyserini555https://github.com/castorini/pyserini, a
wrapper of the Anserini IR toolkit (Yang et al., 2017) for Python. We
optimized the PRF models using the AdamW optimizer with the learning rate
selected from $\\{2\times 10^{-5},1\times 10^{-5},5\times 10^{-6}\\}$ for all
PRF depths in $A=\\{0,1,2,3,4,5\\}$666Since the maximum input length of a
BERT-style PLM is 512, we consider up to 5 feedback documents in this work..
We set the feedback weight $\lambda$ to 1 and the number of comparative
revisions $|K|$ to 2 if not specified. For uniCOIL + docT5query, the PRF model
is initialized from $\mathrm{BERT}_{\mathrm{base}}$, and the document matrix
contains 3,738,207 documents. We set the batch size (number of original
queries) to $\frac{96}{|K|}$, which means the total number of revisions in a
batch is always 96. For ANCE, the PRF model is initialized from
$\mathrm{ANCE}_{\mathrm{FirstP}}$777https://github.com/microsoft/ANCE, the
document matrix contains 5,284,422 documents, and the batch size is set to
$\frac{108}{|K|}$. We keep the model checkpoint with the best MRR@10 score on
the MS MARCO Dev set. In inference, we first obtain top-ranked documents using
the base retrieval model. Then we feed them into the trained PRF model in
Equation (1) to get a revised query vector, and perform the second-pass
retrieval for the final results.

For baselines BM25 and BM25 with RM3, we set $k_{1}$ to 0.82 and $b$ to 0.6
and reproduce them via pyserini. For uniCOIL + docT5query or ANCE, we convert
its pre-built document index to a sparse or dense matrix and reproduce its
retrieval results through our matrix multiplication on GPUs.

##  5\. Experimental Results

In this section, we discuss our experimental results and analysis. We first
compare LoL with typical base retrieval models and state-of-the-art PRF
models; Then, we verify the role of comparative regularization through
ablation studies. Furthermore, we investigate the robustness of LoL to PRF
depths and its sensitivity to training hyper-parameters. Finally, we visualize
the impact of LoL in training through learning curves.

For simplicity, we hereafter refer to the LoL model as the PRF model optimized
under the LoL framework, $\mathrm{LoL}_{\mathrm{uniCOIL}}$ as the LoL model
for uniCOIL + docT5query, and $\mathrm{LoL}_{\mathrm{ANCE}}$ as the LoL model
for ANCE.

###  5.1. Main Results

Table 1 shows the overall retrieval results of baselines and LoL models on
MARCO Dev, MARCO Eval, TREC DL 2019, TREC DL 2020 and DL-HARD. For both sparse
retrieval and dense retrieval, we report the results of LoL models at their
best-performing PRF depths (numbers in superscript brackets). For a fair
comparison with ANCE-PRF(3), we also report the results of
$\mathrm{LoL}_{\mathrm{ANCE}}^{(3)}$, both of which use the top 3 feedback
documents.

In the first group in Table 1, we can see that RM3 improves Recall@1K of BM25
at the expense of MRR@10 and NDCG@10, which reflects the problem of query
drift.

From the last two groups in Table 1, we find that all LoL models outperform
their base retrieval models, i.e., uniCOIL + docT5query and ANCE, across all
evaluation benchmarks and metrics. This proves the availability of our
differentiable PRF implementation of the LoL.

Compared with recent PRF baseline models for ANCE,
$\mathrm{LoL}_{\mathrm{ANCE}}$ also achieve better retrieval performance,
except for the NDCG@10 metric of $\mathrm{LoL}_{\mathrm{ANCE}}^{(3)}$ on the
DL-HARD benchmark is lower than that of ANCE-PRF(3). However, the Recall@1K of
$\mathrm{LoL}_{\mathrm{ANCE}}^{(3)}$ on DL-HARD is improved by 4.3% compared
to ANCE-PRF(3), and outperforms the base ANCE without PRF. Moreover, when five
feedback documents are fed into $\mathrm{LoL}_{\mathrm{ANCE}}$,
$\mathrm{LoL}_{\mathrm{ANCE}}^{(5)}$ achieves a go-ahead over ANCE-PRF(3) on
the NDCG@10 metric. Considering that ANCE-PRF can be regarded as a special
case ($\lambda=0$ and $|K|=|A|=1$) under the LoL framework, the above results
demonstrate the effectiveness of the Loss-over-Loss framework.

It is worth noting that even though the documents are expanded with
T5-generated queries in advance, which to some extent mitigates the expression
mismatch problem, $\mathrm{LoL}_{\mathrm{uniCOIL}}$ still improves on uniCOIL
+ docT5query. This phenomenon demonstrates the powerful query reformulation
capability of LoL and shows that document expansion cannot completely supplant
query reformulation.

###  5.2. Ablation Studies

Table 2. Ablation on comparative regularization for ANCE at all PRF depths.
The best results in each group are marked in bold. Superscripts $\dagger$,
$\ddagger$ and $\S$ indicate statistically significant improvements over LoL
w/o Par, LoL w/o Reg and LoL at the same PRF depth with $p\leq 0.1$,
respectively.

$k$ | Method | NDCG@10 | MRR@10 | R@1K
---|---|---|---|---
- | ANCE | 38.76 | 33.01 | 95.84
0 | ANCE-PRF | 36.40 | 30.70 | 94.30
2-5 | LoL w/o Par | 38.21 | 32.47 | 96.11
| LoL w/o Reg | 38.25 | 32.49 | 96.10
| LoL | 38.68†‡ | 32.85†‡ | 96.22
1 | ANCE-PRF | 39.30 | 33.40 | 96.30
2-5 | LoL w/o Par | 39.59 | 33.77 | 96.67
| LoL w/o Reg | 39.56 | 33.73 | 96.82
| LoL | 39.82†‡ | 33.98†‡ | 96.77
2 | ANCE-PRF | 40.10 | 34.30 | 96.20
2-5 | LoL w/o Par | 40.27 | 34.46 | 96.83
| LoL w/o Reg | 40.16 | 34.33 | 96.79
| LoL | 40.39‡ | 34.51 | 96.86
3 | ANCE-PRF | 40.10 | 34.40 | 95.90
2-5 | LoL w/o Par | 40.58 | 34.71 | 96.94
| LoL w/o Reg | 40.40 | 34.62 | 96.83
| LoL | 40.68‡ | 34.84‡ | 96.94
4 | ANCE-PRF | 40.30 | 34.60 | 96.10
2-5 | LoL w/o Par | 40.59 | 34.72 | 96.93
| LoL w/o Reg | 40.53 | 34.66 | 96.90
| LoL | 40.83†‡ | 34.95‡ | 97.01
5 | ANCE-PRF | 40.00 | 34.40 | 96.00
2-5 | LoL w/o Par | 40.72 | 34.84 | 96.96
| LoL w/o Reg | 40.77 | 34.85 | 96.93
| LoL | 41.01†‡ | 35.14†‡ | 97.03

Table 3. Ablation on comparative regularization for uniCOIL + docT5query at
all PRF depths. The best results in each group are marked in bold.
Superscripts $\dagger$, $\ddagger$ and $\S$ indicate statistically significant
improvements over LoL w/o Par, LoL w/o Reg and LoL at the same PRF depth with
$p\leq 0.1$, respectively.

$k$ | Method | NDCG@10 | MRR@10 | R@1K
---|---|---|---|---
- | uniCOIL+docT5query | 41.21 | 35.13 | 95.81
0 | LoL w/o Par | 41.31 | 35.03 | 96.69
LoL w/o Reg | 41.13 | 34.92 | 96.79
LoL | 41.36‡ | 35.08‡ | 96.80†
1 | LoL w/o Par | 41.73 | 35.46 | 96.79
LoL w/o Reg | 41.73 | 35.50 | 96.95
LoL | 41.86 | 35.62 | 96.98†
2 | LoL w/o Par | 41.83 | 35.56 | 96.82
LoL w/o Reg | 41.81 | 35.59 | 97.06
LoL | 41.94‡ | 35.68 | 97.01†
3 | LoL w/o Par | 41.76 | 35.48 | 96.85
LoL w/o Reg | 41.75 | 35.51 | 97.03§
LoL | 42.02†‡ | 35.75†‡ | 96.91
4 | LoL w/o Par | 41.61 | 35.28 | 96.85
LoL w/o Reg | 41.74 | 35.43 | 97.03
LoL | 41.94†‡ | 35.67†‡ | 96.96†
5 | LoL w/o Par | 41.68 | 35.37 | 96.87
LoL w/o Reg | 41.74 | 35.44 | 97.04
LoL | 41.94†‡ | 35.67†‡ | 96.89

In this part, we conduct ablation studies on MARCO Dev for both sparse and
dense retrieval to further explore the roles of comparative regularization and
multiple parallel revisions in LoL.

A standard LoL ($\lambda=1,|K|=2$) and two LoL variants, i.e., LoL w/o Reg
($\lambda=0,|K|=2$) and LoL w/o Par ($\lambda=0,|K|=1$), are measured at all
PRF depths in $A$. We compare the evaluation results of the standard LoL and
LoL w/o Reg to show the role of the comparative regularization in Equation
(4). We further introduce LoL w/o Par to eliminate the effect of parallel
revision multiple times in one batch. For dense retrieval, we also compare LoL
models to ANCE-PRF models, which are equivalent to LoL w/o Par trained
separately at each PRF depth. Note that we use different checkpoints for the
model of the same type at different PRF depths, which are selected for each
PRF depth separately.

As shown in Table 2, at each PRF depth, the standard LoL outperforms its two
variants and ANCE-PRF in all metrics, with the one exception of recall@1K at
$k=1$, where LoL w/o Reg is slightly better than LoL. The conclusions of the
sparse search in Table 3 are similar, although there are four slight drops in
recall compared to LoL w/o Reg. We speculate this may be because the ranking
loss function in Equation (7) is closer to the shallower metrics like NDCG@10
and MRR@10. And the comparative regularization further increases LoL models’
attention to these shallow ranking metrics. Therefore, it is sufficient to
show the effectiveness of the comparative regularization. In addition, we find
that LoL w/o Reg and LoL w/o Par are generally competitive with each other,
which indicates the impact of parallel multiple revisions is not significant
and highlights the role of comparative regularization.

Moreover, as shown in Table 2, LoL w/o Par also outperforms the ANCE-PRF
across the board, especially the Recall@1K metric. We believe this may be
attributed to joint training and the computation of reformulation loss on the
entire (mined) document matrix.

###  5.3. Robustness to PRF Depth

(a) ANCE

(b) uniCOIL + docT5query

Figure 3. The MRR@10 curve of the same PRF model using different numbers of
documents. The horizontal dotted line represents the MRR@10 value of the base
retrieval model.

At the beginning of the design, we expect LoL to alleviate query drift, i.e.,
make the model more robust to the increasing number of feedback documents. In
this part, we verify this expectation.

Figure 3 shows the performance of the best checkpoint for multiple PRF models
at all PRF depths. Different from using different model checkpoints at
different PRF depths in Table 2 and Table 3, each curve of LoL in Figure 3 is
drawn from the performance of the same model checkpoint. Therefore, the MRR@10
values in Table 2 and Table 3 can be viewed as the upper bound of the values
in Figure 3(a) and Figure 3(b), respectively. As we can see in Figure 3(a) and
Table 2, only LoL and LoL w/o Reg are monotonically increasing with respect to
the number of documents. ANCE-PRF reaches peak performance at PRF depth 4 and
then suffers performance degradation, and LoL w/o Par. encounters a
performance dip when the number of feedback documents increased from 3 to 4.
As for PRF models applied in sparse retrieval in Figure 3(b) and Table 3, LoL
w/o Par. and LoL w/o Reg reach peak performance at PRF depth 2, while LoL
continues to grow until the PRF depth approaches 4.

To quantify the robustness of LoL, we report the robustness indices (RI) of
$\mathrm{LoL}_{\mathrm{ANCE}}^{(k)}$ with respect to ANCE and
$\mathrm{LoL}_{\mathrm{ANCE}}^{(k-1)}$ in Table 4 and Table 5, respectively.
From Table 4, we can find that $\mathrm{LoL}_{\mathrm{ANCE}}$ reformulates
more revisions that are better than original queries compared to its variant
baselines at all PRF depths. Similarly, as shown in Table 5, when the number
of feedback documents increases from $k-1$ to $k$, compared to LoL w/o Par and
LoL w/o Reg, LoL can more robustly revise better queries than before.

From these observations, we may draw two conclusions. (1) Compared to these
baselines, LoL is more robust to PRF depths. That is, as the number of
feedback documents increases, LoL-reformulated queries have less drift and are
less prone to performance degradation. (2) LoL for dense retrieval is more
robust than LoL for sparse retrieval. We conjecture that this is because dense
query vectors are more fine-grained and are more likely to prevent the
introduction of irrelevant information, while sparse query vectors are term-
grained and may introduce relevant polysemous terms when reformulating the
query, which in turn leads to query drift.

Table 4. RI of $\mathrm{LoL}_{\mathrm{ANCE}}^{(k)}$ with respect to ANCE on MARCO Dev at all PRF depths. $k$ | 0 | 1 | 2 | 3 | 4 | 5
---|---|---|---|---|---|---
LoL w/o Par | 0.34 | 0.42 | 0.41 | 0.42 | 0.41 | 0.41
LoL w/o Reg | 0.32 | 0.40 | 0.41 | 0.41 | 0.41 | 0.40
LoL | 0.36 | 0.43 | 0.43 | 0.44 | 0.44 | 0.44
Table 5. RI of $\mathrm{LoL}_{\mathrm{ANCE}}^{(k)}$ with respect to $\mathrm{LoL}_{\mathrm{ANCE}}^{(k-1)}$ on MARCO Dev at all PRF depths. $k$ | 1 | 2 | 3 | 4 | 5
---|---|---|---|---|---
LoL w/o Par | 0.51 | 0.54 | 0.58 | 0.58 | 0.61
LoL w/o Reg | 0.52 | 0.53 | 0.58 | 0.59 | 0.61
LoL | 0.54 | 0.56 | 0.63 | 0.63 | 0.66

###  5.4. Sensitivity to Training Hyper-parameters

To capture the sensitivity of LoL to the number of comparative revisions $|K|$
and the regularization weight $\lambda$, we evaluate multiple
$\mathrm{LoL}_{\mathrm{ANCE}}$ models trained with different $|K|$ and
$\lambda$ on MARCO Dev set. As shown in Table 6, all
$\mathrm{LoL}_{\mathrm{ANCE}}$ models trained with $|K|>1$ perform better than
that with $|K|=1$, which indicates that is not sensitive to $|K|>1$. Comparing
the last two rows that share the regularization weight $\lambda=0.5$, we can
find that the smaller $|K|$ seems to be better trained than the larger $|K|$.
We speculate that this may be because larger $|K|$ leads to smaller training
batch size under the GPU memory limitation. Using the default setting of
$|K|=2$, although the variance of the values in rows 2 to 4 is not large,
setting $\lambda$ to 1 performs best at most PRF depths.

Table 6. MRR@10 of $\mathrm{LoL}_{\mathrm{ANCE}}$ with different training hyper-parameters at all PRF depths on MARCO Dev. $|K|$ | $\lambda$ | $k$
---|---|---
1 | 2 | 3 | 4 | 5
1 | 0 | 33.74 | 33.38 | 34.57 | 34.51 | 34.59
2 | 1.5 | 34.02 | 34.58 | 34.75 | 34.89 | 35.08
2 | 1 | 33.98 | 34.51 | 34.84 | 34.95 | 35.14
2 | 0.5 | 33.99 | 34.59 | 34.89 | 34.98 | 35.10
3 | 0.5 | 33.97 | 34.61 | 34.77 | 34.79 | 35.05

###  5.5. Analysis of Loss Curves

(a) Reformulation loss on Dev

(b) Reformulation loss on Train

(c) Regularization loss on Train

Figure 4. The loss curves of standard LoL, LoL w/o Reg and LoL w/o Par for
ANCE on MS MARCO.

To visualize the impact of LoL in training, we show the loss curves of
$\mathrm{LoL}_{\mathrm{ANCE}}$ on the MARCO Train and Dev sets in Figure 4.
Figure 4(a) and 4(b) plot the query reformulation losses in Equation (7) on
Train and Dev sets at training time. We can find that the training
reformulation loss of LoL w/o Par drops the fastest and lowest, followed by
LoL w/o Reg, and LoL the slowest. And their performance on Dev set is just the
opposite, where the evaluation reformulation loss of LoL w/o Par and LoL w/o
Reg starts to increase successively after reaching the lowest point, while the
evaluation loss of LoL rises slightly at the end. This indicates that both
comparative regularization and multiple parallel revisions have the effect of
mitigating overfitting, and comparative regularization has more impact. Figure
4(c) shows the comparative regularization terms of LoL and LoL w/o Reg. They
both revise a query multiple times in parallel. Without regularizing the
reformulation losses of these parallel revisions, the regularization loss of
LoL w/o Reg also drops but not to a level as low as LoL. This implies that a
PRF model cannot naturally learn to guarantee the normal comparison
relationship among multiple revisions without explicitly imposing
regularization. Even with regularization imposed, we can see that guaranteeing
this normal comparison relationship is not easy for the model, because this
regularization loss drops slowly in the middle and late stages of training.

##  6\. Discussion

Further deriving the final loss in Equation (5), we can find that LoL can be
viewed as re-weighting multiple reformulation losses of the same query. For
simplicity, we denote $\mathcal{L}_{\mathrm{rf}}(q^{(k)}))$ as $L^{k}$.
Speicially, the loss can be rewrited as follow:

$\displaystyle\mathcal{L}(q)$ $\displaystyle=\frac{1}{|K|}\sum_{k\in
K}L^{k}+\lambda\mathcal{L}_{\mathrm{cr}}(q)$
$\displaystyle=\frac{1}{|K|}\left[\sum_{k\in
K}L^{k}+\frac{2\lambda}{|K|-1}\sum_{j,k\in K\atop
j<k}\max(0,L^{k}-L^{j})\right]$
$\displaystyle=\frac{1}{|K|}\sum_{k}\left[L^{k}+\frac{2\lambda}{|K|-1}\sum_{j<k}\max(0,L^{k}-L^{j})\right]$
$\displaystyle=\frac{1}{|K|}\sum_{k}\left[\left(1+2\lambda\frac{\sum_{j<k}\mathds{1}(L^{k}>L^{j})}{|K|-1}\right)L^{k}-2\lambda\frac{\sum_{i<k}\mathds{1}(L^{k}>L^{i})}{|K|-1}L^{i}\right]$
$\displaystyle=\frac{1}{|K|}\sum_{k}\left[1+\frac{2\lambda}{{|K|-1}}\left(\sum_{j<k}\mathds{1}(L^{k}>L^{j})-\sum_{i>k}\mathds{1}(L^{k}<L^{i})\right)\right]L^{k}$
$\displaystyle=\frac{1}{|K|}\sum_{k}\left[1+2\lambda\frac{\sum_{j\neq
k}\mathrm{CMP}(k,j,L^{k},L^{j})}{|K|-1}\right]L^{k},$

where $\mathds{1}(\cdot)$ is a indicator function and $\mathrm{CMP}$ is a
function to compare the sizes of the feedback sets and evaluation losses of
two revisions derived from the same query. Formally, the $\mathrm{CMP}$
function is defined as:

$\mathrm{CMP}(k,j,L^{k},L^{j})=\begin{cases}1,&\text{if }j<k\text{ and
}L^{j}<L^{k}\\\ -1,&\text{if }j>k\text{ and }L^{j}>L^{k}\\\
0,&\text{otherwise}.\end{cases}$

From this re-weighting perspective, given the size of the PRF depth set $|K|$,
the training complexity of LoL is the same as LoL w/o Reg ($\lambda=0$), and
the additional comparison overhead is a small constant and negligible.
Besides, since LoL is just an optimization framework, PRF models trained under
LoL do not have any increase in computational cost at inference time.

Essentially, comparative regularization aims to guarantee the normal order of
a set of objects. This normal order is usually supposed to be maintained, i.e.
unsupervised, but ignored by the model. Therefore, from this perspective, LoL
can be seen as an unsupervised application of leaning-to-rank. As such, one
future direction is to explore the application of other leaning-to-rank losses
here. Furthermore, these objects should be able to be mapped to differentiable
values, such as evaluation metrics or losses. Therefore, future work can also
replace the mapping function (reformulation loss) in our method. Moreover, if
there are similar neglected normal orders in other tasks, then the comparative
regularization may also be used for other tasks.

##  7\. Conclusion

In this paper, we find that the query drift problem in pseudo-relevance
feedback is mainly caused by irrelevant information when more pseudo-relevant
documents are involved as feedback information to reformulate the query.
Ideally, a good pseudo-relevance feedback model should have the ability to use
more feedback documents that contain irrelevant information. That is, the more
pseudo-relevant documents provided, the better quality of the reformulated
query. Armed with this intuition, we design a novel comparative regularization
loss based on multiple query reformulation losses to ensure that more feedback
documents lead to smaller query reformulation losses. The proposed comparative
regularization loss over query reformulation losses (LoL) framework can be
used in any pseudo-relevance feedback model with any retrieval framework,
e.g., sparse retrieval or dense retrieval. Experiments on publicly large-scale
dataset MS MARCO and its variant evaluation sets demonstrate that our plug-
and-play regularization can bring improvements compared to the baseline
methods.

###### Acknowledgements.

This work was supported by the National Natural Science Foundation of China
(NSFC) under Grants No.61906180, U21B2046 and Liang Pang, Huawei Shen, Yanyan
Lan are also supported by Beijing Academy of Artificial Intelligence (BAAI).

## References

  * (1)
  * Aggarwal and Buitelaar (2012) Nitish Aggarwal and P. Buitelaar. 2012.  Query Expansion Using Wikipedia and Dbpedia. In _CLEF_.
  * Amati et al. (2004) Giambattista Amati, Claudio Carpineto, and Giovanni Romano. 2004.  Query Difficulty, Robustness, and Selective Application of Query Expansion. In _Advances in Information Retrieval_ _(Lecture Notes in Computer Science)_ , Sharon McDonald and John Tait (Eds.). Springer, Berlin, Heidelberg, 127–137.  https://doi.org/10.1007/978-3-540-24752-4_10
  * Attar and Fraenkel (1977) R. Attar and A. S. Fraenkel. 1977.  Local Feedback in Full-Text Retrieval Systems.  _J. ACM_ 24, 3 (July 1977), 397–417.  https://doi.org/10.1145/322017.322021
  * Cao et al. (2008) Guihong Cao, Jian-Yun Nie, Jianfeng Gao, and Stephen Robertson. 2008.  Selecting Good Expansion Terms for Pseudo-Relevance Feedback. In _Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval_ _(SIGIR ’08)_. Association for Computing Machinery, New York, NY, USA, 243–250.  https://doi.org/10.1145/1390334.1390377
  * Carpineto and Romano (2012) Claudio Carpineto and Giovanni Romano. 2012.  A Survey of Automatic Query Expansion in Information Retrieval.  _Comput. Surveys_ 44, 1 (Jan. 2012), 1:1–1:50.  https://doi.org/10.1145/2071389.2071390
  * Clinchant and Gaussier (2013) Stéphane Clinchant and Eric Gaussier. 2013.  A Theoretical Analysis of Pseudo-Relevance Feedback Models. In _Proceedings of the 2013 Conference on the Theory of Information Retrieval_ _(ICTIR ’13)_. Association for Computing Machinery, New York, NY, USA, 6–13.  https://doi.org/10.1145/2499178.2499179
  * Collins-Thompson (2009) Kevyn Collins-Thompson. 2009\.  Reducing the Risk of Query Expansion via Robust Constrained Optimization. In _Proceedings of the 18th ACM Conference on Information and Knowledge Management_ _(CIKM ’09)_. Association for Computing Machinery, New York, NY, USA, 837–846.  https://doi.org/10.1145/1645953.1646059
  * Collins-Thompson and Callan (2007) Kevyn Collins-Thompson and Jamie Callan. 2007.  Estimation and Use of Uncertainty in Pseudo-Relevance Feedback. In _Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval_ _(SIGIR ’07)_. Association for Computing Machinery, New York, NY, USA, 303–310.  https://doi.org/10.1145/1277741.1277795
  * Craswell et al. (2021) Nick Craswell, Bhaskar Mitra, Emine Yilmaz, and Daniel Campos. 2021.  Overview of the TREC 2020 Deep Learning Track.  _arXiv:2102.07662 [cs]_ (Feb. 2021).  arXiv:2102.07662 [cs] http://arxiv.org/abs/2102.07662
  * Craswell et al. (2020) Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M. Voorhees. 2020.  Overview of the TREC 2019 Deep Learning Track.  _arXiv:2003.07820 [cs]_ (March 2020).  arXiv:2003.07820 [cs] http://arxiv.org/abs/2003.07820
  * CROFT and HARPER (1979) W.B. CROFT and D.J. HARPER. 1979.  Using Probabilistic Models of Document Retrieval without Relevance Information.  _Journal of Documentation_ 35, 4 (1979), 285–295.  https://doi.org/10.1108/eb026683
  * Croft et al. (2010) W. Bruce Croft, Donald Metzler, and Trevor Strohman. 2010\.  _Search Engines: Information Retrieval in Practice_. Vol. 520.  Addison-Wesley Reading.
  * Cronen-Townsend et al. (2004) Steve Cronen-Townsend, Yun Zhou, and W. Bruce Croft. 2004\.  A Framework for Selective Query Expansion. In _Proceedings of the Thirteenth ACM International Conference on Information and Knowledge Management_ _(CIKM ’04)_. Association for Computing Machinery, New York, NY, USA, 236–237.  https://doi.org/10.1145/1031171.1031220
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_. Association for Computational Linguistics, Minneapolis, Minnesota, 4171–4186.  https://doi.org/10.18653/v1/N19-1423
  * Diaz et al. (2016) Fernando Diaz, Bhaskar Mitra, and Nick Craswell. 2016\.  Query Expansion with Locally-Trained Word Embeddings. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, Berlin, Germany, 367–377.  https://doi.org/10.18653/v1/P16-1035
  * Fan et al. (2022) Yixing Fan, Xiaohui Xie, Yinqiong Cai, Jia Chen, Xinyu Ma, Xiangsheng Li, Ruqing Zhang, Jiafeng Guo, and Yiqun Liu. 2022.  Pre-Training Methods in Information Retrieval.  _arXiv:2111.13853 [cs]_ (April 2022).  http://arxiv.org/abs/2111.13853
  * Furnas et al. (1987) G. W. Furnas, T. K. Landauer, L. M. Gomez, and S. T. Dumais. 1987.  The Vocabulary Problem in Human-System Communication.  _Commun. ACM_ 30, 11 (Nov. 1987), 964–971.  https://doi.org/10.1145/32206.32212
  * Gong et al. (2005) Zhiguo Gong, Chan Wa Cheang, and U Leong Hou. 2005\.  Web Query Expansion by Wordnet. In _Proceedings of the 16th International Conference on Database and Expert Systems Applications_ _(DEXA’05)_. Springer-Verlag, Berlin, Heidelberg, 166–175.  https://doi.org/10.1007/11546924_17
  * Herbrich et al. (2000) Ralf Herbrich, Thore Graepel, and Klaus Obermayer. 2000\.  Large Margin Rank Boundaries for Ordinal Regression.  _Advances in large margin classifiers_ 88, 2 (2000), 115–132.
  * Hofstätter et al. (2021) Sebastian Hofstätter, Sheng-Chieh Lin, Jheng-Hong Yang, Jimmy Lin, and Allan Hanbury. 2021\.  Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling.  In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_. Association for Computing Machinery, New York, NY, USA, 113–122.  https://doi.org/10.1145/3404835.3462891
  * Jaleel et al. (2004) N. A. Jaleel, James Allan, W. Croft, Fernando Diaz, L. Larkey, Xiaoyan Li, M. Smucker, and C. Wade. 2004\.  UMass at TREC 2004: Novelty and HARD. In _TREC_.  https://doi.org/10.21236/ada460118
  * Joachims (2002) Thorsten Joachims. 2002\.  Optimizing Search Engines Using Clickthrough Data. In _Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ _(KDD ’02)_. Association for Computing Machinery, New York, NY, USA, 133–142.  https://doi.org/10.1145/775047.775067
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020\.  Dense Passage Retrieval for Open-Domain Question Answering. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Association for Computational Linguistics, Online, 6769–6781.  https://doi.org/10.18653/v1/2020.emnlp-main.550
  * Khattab and Zaharia (2020) Omar Khattab and Matei Zaharia. 2020.  ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT.  In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_. Association for Computing Machinery, New York, NY, USA, 39–48.  https://doi.org/10.1145/3397271.3401075
  * Lafferty and Zhai (2001) John Lafferty and Chengxiang Zhai. 2001.  Document Language Models, Query Models, and Risk Minimization for Information Retrieval. In _Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval_ _(SIGIR ’01)_. Association for Computing Machinery, New York, NY, USA, 111–119.  https://doi.org/10.1145/383952.383970
  * Lavrenko and Croft (2001) Victor Lavrenko and W. Bruce Croft. 2001.  Relevance Based Language Models. In _Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval_ _(SIGIR ’01)_. Association for Computing Machinery, New York, NY, USA, 120–127.  https://doi.org/10.1145/383952.383972
  * Li et al. (2018) Canjia Li, Yingfei Sun, Ben He, Le Wang, Kai Hui, Andrew Yates, Le Sun, and Jungang Xu. 2018.  NPRF: A Neural Pseudo Relevance Feedback Framework for Ad-hoc Information Retrieval. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Brussels, Belgium, 4482–4491.  https://doi.org/10.18653/v1/D18-1478
  * Li et al. (2021a) Hang Li, Ahmed Mourad, Shengyao Zhuang, Bevan Koopman, and Guido Zuccon. 2021a.  Pseudo Relevance Feedback with Deep Language Models and Dense Retrievers: Successes and Pitfalls.  _arXiv:2108.11044 [cs]_ (Aug. 2021).  arXiv:2108.11044 [cs] http://arxiv.org/abs/2108.11044
  * Li et al. (2021b) Hang Li, Shengyao Zhuang, Ahmed Mourad, Xueguang Ma, Jimmy Lin, and Guido Zuccon. 2021b.  Improving Query Representations for Dense Retrieval with Pseudo Relevance Feedback: A Reproducibility Study.  _arXiv:2112.06400 [cs]_ (Dec. 2021).  arXiv:2112.06400 [cs] http://arxiv.org/abs/2112.06400
  * Lin and Ma (2021) Jimmy Lin and Xueguang Ma. 2021.  A Few Brief Notes on DeepImpact, COIL, and a Conceptual Framework for Information Retrieval Techniques.  _arXiv:2106.14807 [cs]_ (June 2021).  arXiv:2106.14807 [cs] http://arxiv.org/abs/2106.14807
  * Lin et al. (2020) Sheng-Chieh Lin, Jheng-Hong Yang, and Jimmy Lin. 2020\.  Distilling Dense Representations for Ranking Using Tightly-Coupled Teachers.  _arXiv:2010.11386 [cs]_ (Oct. 2020).  arXiv:2010.11386 [cs] http://arxiv.org/abs/2010.11386
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A Robustly Optimized BERT Pretraining Approach.  _arXiv:1907.11692 [cs]_ (July 2019).  arXiv:1907.11692 [cs] http://arxiv.org/abs/1907.11692
  * Lv and Zhai (2009a) Yuanhua Lv and ChengXiang Zhai. 2009a.  Adaptive Relevance Feedback in Information Retrieval. In _Proceedings of the 18th ACM Conference on Information and Knowledge Management_ _(CIKM ’09)_. Association for Computing Machinery, New York, NY, USA, 255–264.  https://doi.org/10.1145/1645953.1645988
  * Lv and Zhai (2009b) Yuanhua Lv and ChengXiang Zhai. 2009b.  A Comparative Study of Methods for Estimating Query Language Models with Pseudo Feedback. In _Proceedings of the 18th ACM Conference on Information and Knowledge Management_ _(CIKM ’09)_. Association for Computing Machinery, New York, NY, USA, 1895–1898.  https://doi.org/10.1145/1645953.1646259
  * Lv and Zhai (2014) Yuanhua Lv and ChengXiang Zhai. 2014.  Revisiting the Divergence Minimization Feedback Model. In _Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management_ _(CIKM ’14)_. Association for Computing Machinery, New York, NY, USA, 1863–1866.  https://doi.org/10.1145/2661829.2661900
  * Ma et al. (2021) Xinyu Ma, Jiafeng Guo, Ruqing Zhang, Yixing Fan, Xiang Ji, and Xueqi Cheng. 2021\.  PROP: Pre-Training with Representative Words Prediction for Ad-Hoc Retrieval. In _Proceedings of the 14th ACM International Conference on Web Search and Data Mining_ _(WSDM ’21)_. Association for Computing Machinery, New York, NY, USA, 283–291.  https://doi.org/10.1145/3437963.3441777
  * Mackie et al. (2021) Iain Mackie, Jeffrey Dalton, and Andrew Yates. 2021\.  How Deep Is Your Learning: The DL-HARD Annotated Deep Learning Dataset.  In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_. Association for Computing Machinery, New York, NY, USA, 2335–2341.  https://doi.org/10.1145/3404835.3463262
  * Manning et al. (2008) Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze. 2008.  _Introduction to Information Retrieval_.  Cambridge University Press, Cambridge.  https://doi.org/10.1017/CBO9780511809071
  * Maron and Kuhns (1960) M. E. Maron and J. L. Kuhns. 1960.  On Relevance, Probabilistic Indexing and Information Retrieval.  _J. ACM_ 7, 3 (July 1960), 216–244.  https://doi.org/10.1145/321033.321035
  * Mitra et al. (1998) Mandar Mitra, Amit Singhal, and Chris Buckley. 1998\.  Improving Automatic Query Expansion. In _Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval_ _(SIGIR ’98)_. Association for Computing Machinery, New York, NY, USA, 206–214.  https://doi.org/10.1145/290941.290995
  * Montazeralghaem et al. (2020) Ali Montazeralghaem, Hamed Zamani, and James Allan. 2020\.  A Reinforcement Learning Framework for Relevance Feedback.  In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_. Association for Computing Machinery, New York, NY, USA, 59–68.  https://doi.org/10.1145/3397271.3401099
  * Naseri et al. (2021) Shahrzad Naseri, Jeffrey Dalton, Andrew Yates, and James Allan. 2021.  CEQE: Contextualized Embeddings for Query Expansion. In _Advances in Information Retrieval_ _(Lecture Notes in Computer Science)_ , Djoerd Hiemstra, Marie-Francine Moens, Josiane Mothe, Raffaele Perego, Martin Potthast, and Fabrizio Sebastiani (Eds.). Springer International Publishing, Cham, 467–482.  https://doi.org/10.1007/978-3-030-72113-8_31
  * Nguyen et al. (2016) Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016.  MS MARCO: A Human Generated Machine Reading Comprehension Dataset. In _CoCo@ NIPS_.
  * Nogueira et al. (2019) Rodrigo Nogueira, Jimmy Lin, and A. I. Epistemic. 2019\.  From Doc2query to docTTTTTquery.  _Online preprint_ (2019).
  * Ponte and Croft (1998) Jay M. Ponte and W. Bruce Croft. 1998.  A Language Modeling Approach to Information Retrieval. In _Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval_ _(SIGIR ’98)_. Association for Computing Machinery, New York, NY, USA, 275–281.  https://doi.org/10.1145/290941.291008
  * Qi et al. (2019) Peng Qi, Xiaowen Lin, Leo Mehr, Zijian Wang, and Christopher D. Manning. 2019.  Answering Complex Open-domain Questions Through Iterative Query Generation. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. Association for Computational Linguistics, Hong Kong, China, 2590–2602.  https://doi.org/10.18653/v1/D19-1261
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.  _Journal of Machine Learning Research_ 21, 140 (2020), 1–67.
  * Robertson and Zaragoza (2009) Stephen Robertson and Hugo Zaragoza. 2009.  The Probabilistic Relevance Framework: BM25 and Beyond.  _Foundations and Trends® in Information Retrieval_ 3, 4 (Dec. 2009), 333–389.  https://doi.org/10.1561/1500000019
  * Robertson and Jones (1976) S. E. Robertson and K. Sparck Jones. 1976.  Relevance Weighting of Search Terms.  _Journal of the American Society for Information Science_ 27, 3 (1976), 129–146.  https://doi.org/10.1002/asi.4630270302
  * Rocchio (1971) Joseph Rocchio. 1971\.  Relevance Feedback in Information Retrieval.  _The Smart retrieval system-experiments in automatic document processing_ (1971), 313–323.
  * Salton et al. (1975) G. Salton, A. Wong, and C. S. Yang. 1975.  A Vector Space Model for Automatic Indexing.  _Commun. ACM_ 18, 11 (Nov. 1975), 613–620.  https://doi.org/10.1145/361219.361220
  * Shiri and Revie (2006) Ali Shiri and Crawford Revie. 2006.  Query Expansion Behavior within a Thesaurus-Enhanced Search Environment: A User-Centered Evaluation.  _Journal of the American Society for Information Science and Technology_ 57, 4 (2006), 462–478.  https://doi.org/10.1002/asi.20319
  * Tao and Zhai (2006) Tao Tao and ChengXiang Zhai. 2006.  Regularized Estimation of Mixture Models for Robust Pseudo-Relevance Feedback. In _Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval_ _(SIGIR ’06)_. Association for Computing Machinery, New York, NY, USA, 162–169.  https://doi.org/10.1145/1148170.1148201
  * Wang et al. (2021) Xiao Wang, Craig Macdonald, Nicola Tonellotto, and Iadh Ounis. 2021.  Pseudo-Relevance Feedback for Multiple Representation Dense Retrieval. In _Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval_ _(ICTIR ’21)_. Association for Computing Machinery, New York, NY, USA, 297–306.  https://doi.org/10.1145/3471158.3472250
  * Xiong and Callan (2015) Chenyan Xiong and Jamie Callan. 2015.  Query Expansion with Freebase. In _Proceedings of the 2015 International Conference on The Theory of Information Retrieval_ _(ICTIR ’15)_. Association for Computing Machinery, New York, NY, USA, 111–120.  https://doi.org/10.1145/2808194.2809446
  * Xiong et al. (2020) Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. 2020\.  Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=zeFrfgyZln
  * Xu and Croft (1996) Jinxi Xu and W. Bruce Croft. 1996.  Query Expansion Using Local and Global Document Analysis. In _Proceedings of the 19th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval_ _(SIGIR ’96)_. Association for Computing Machinery, New York, NY, USA, 4–11.  https://doi.org/10.1145/243199.243202
  * Yang et al. (2017) Peilin Yang, Hui Fang, and Jimmy Lin. 2017.  Anserini: Enabling the Use of Lucene for Information Retrieval Research. In _Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval_ _(SIGIR ’17)_. Association for Computing Machinery, New York, NY, USA, 1253–1256.  https://doi.org/10.1145/3077136.3080721
  * Yu et al. (2021) HongChien Yu, Chenyan Xiong, and Jamie Callan. 2021\.  Improving Query Representations for Dense Retrieval with Pseudo Relevance Feedback. In _Proceedings of the 30th ACM International Conference on Information & Knowledge Management_ _(CIKM ’21)_. Association for Computing Machinery, New York, NY, USA, 3592–3596.  https://doi.org/10.1145/3459637.3482124
  * Zamani and Croft (2016) Hamed Zamani and W. Bruce Croft. 2016.  Embedding-Based Query Language Models. In _Proceedings of the 2016 ACM International Conference on the Theory of Information Retrieval_ _(ICTIR ’16)_. Association for Computing Machinery, New York, NY, USA, 147–156.  https://doi.org/10.1145/2970398.2970405
  * Zhai and Lafferty (2001) Chengxiang Zhai and John Lafferty. 2001.  Model-Based Feedback in the Language Modeling Approach to Information Retrieval. In _Proceedings of the Tenth International Conference on Information and Knowledge Management_ _(CIKM ’01)_. Association for Computing Machinery, New York, NY, USA, 403–410.  https://doi.org/10.1145/502585.502654
  * Zheng et al. (2020) Zhi Zheng, Kai Hui, Ben He, Xianpei Han, Le Sun, and Andrew Yates. 2020.  BERT-QE: Contextualized Query Expansion for Document Re-ranking. In _Findings of the Association for Computational Linguistics: EMNLP 2020_. Association for Computational Linguistics, Online, 4718–4728.  https://doi.org/10.18653/v1/2020.findings-emnlp.424
  * Zhu et al. (2021) Yunchang Zhu, Liang Pang, Yanyan Lan, Huawei Shen, and Xueqi Cheng. 2021.  Adaptive Information Seeking for Open-Domain Question Answering. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 3615–3626.  https://doi.org/10.18653/v1/2021.emnlp-main.293
  * Zighelnic and Kurland (2008) Liron Zighelnic and Oren Kurland. 2008.  Query-Drift Prevention for Robust Query Expansion. In _Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval_ _(SIGIR ’08)_. Association for Computing Machinery, New York, NY, USA, 825–826.  https://doi.org/10.1145/1390334.1390524