# HYRR: Hybrid Infused Reranking for Passage Retrieval

Jing Lu, Keith Hall, Ji Ma  Jianmo Ni
Google Research
<EMAIL_ADDRESS>

###### Abstract

We present Hybrid Infused Reranking for Passages Retrieval (HYRR), a framework
for training rerankers based on a hybrid of BM25 and neural retrieval models.
Retrievers based on hybrid models have been shown to outperform both BM25 and
neural models alone. Our approach exploits this improved performance when
training a reranker, leading to a robust reranking model. The reranker, a
cross-attention neural model, is shown to be robust to different first-stage
retrieval systems, achieving better performance than rerankers simply trained
upon the first-stage retrievers in the multi-stage systems. We present
evaluations on a supervised passage retrieval task using MS MARCO and zero-
shot retrieval tasks using BEIR. The empirical results show strong performance
on both evaluations.

##  1 Introduction

Information Retrieval (and subsequent retrieve-and-read approaches to open-
domain question answering (QA)) have generally seen pipelined retrieve-and-
rank approaches achieve the best accuracies. This paradigm has become so
common that many of the TREC IR evaluations have placed an emphasis on
reranking the top N (often 1000) results from a first-pass retriever Craswell
et al. (2022); Ekstrand et al. (2022). There has been significant work
recently showing that neural retrieval models can improve recall and produce
state-of-the-art retrievers Karpukhin et al. (2020); Qu et al. (2021); Ren et
al. (2021), but this is not true across the board; there are still many tasks
where simple term-based techniques such as BM25 Robertson and Zaragoza (2009)
significantly outperform sophisticated neural retrieval models.

This motivates recent exploration in hybrid retrievers in order to join the
best from each world. Techniques which combine term-based models and neural
models in either early stage Ma et al. (2021) or later stage Karpukhin et al.
(2020); Chen et al. (2022) outperform each individual model. No matter how
much improvement achieved from first-stage retrievers, the reranking model
continues to achieve an additional performance boost.

In this work, we revisit the question of how best to train a reranking model
for retrieval. We study two different settings: the supervised setting where
we have training data in the target domain, and the zero-shot setting where we
have no training data in the target domain. The traditional wisdom is that you
should train the reranker on data that is similar to the distribution that you
will observe at inference time, and the best means to do so is to use the same
retriever for inference as you do for training the reranker. That is to say,
the first-stage retriever produces candidates for the reranker for both
training and inference Nogueira et al. (2019); Pradeep et al. (2021a).
However, recent work shows that this does not always guarantee an effective
reranker Gao et al. (2021). While we do see this approach achieving strong
results in general, we show that by training a robust reranker which has been
exposed to training data from a hybrid of term-based and neural retrieval
models, we are able to achieve high performance on MS MAMRCO passage ranking
task and a large set of out-of-domain datasets (BEIR Thakur et al. (2021))
even when reranking candidates are generated by a BM25 retriever.

The primary contributions of this paper are:

  * •

We present HYRR, a training paradigm for training rerankers based on hybrid
term-based and neural retrievers.

  * •

We show that this approach is effective in both supervised setting and zero-
shot setting.

  * •

We show that this approach results in a robust reranker which performs well
across different retrievers, domains, and tasks; though there are still
limitations which appear to be based in the query generation approach utilized
in the zero-shot setting.

##  2 Related Work

###  2.1 Neural ranking models

Many prior works explore using neural models for text ranking, most recently
the focus as been on transformer-based models Vaswani et al. (2017). Even
through it is computationally expensive, a BERT-based cross-attention model is
one of the most dominant models for text ranking Nogueira and Cho (2019); Gao
et al. (2021) because of its capability to model the interaction between the
query and passage. Concretely, queries and passages are concatenated and fed
into the BERT model, a pairwise score is then obtained by projecting the
encoding of [CLS] token. The text ranking problem is cast as a binary
classification problem. Recently, encoder-decoder language models, such as T5
Raffel et al. (2020), have been adapted for text ranking. Nogueira et
al.Nogueira et al. (2020) proposed a model that takes a query and passage pair
as input of encoder, and the decoder produces the tokens “true” or “false” to
indicate the relevance of a query and a given passage. Pradeep et al. Pradeep
et al. (2021b) further proposed a pairwise ranking model that takes a query
and two passages as input and the decoder produces the token “true” if the
first passage is more relevant then the second passage, and “false” otherwise.
Zhuang et al. (2022) proposed T5 encoder-only and encoder-decoder rerankers
that optimize ranking performance directly by outputting real-value scores and
using ranking losses. Muennighoff (2022) proposed SGPT that uses pre-trained
language model as reranking model directly; and Sachan et al. (2022) proposed
UPR that uses a zero-shot question generation model via prompting a large
language model in order to directly rerank passages.

###  2.2 Multi-stage retrieval pipelines

Passage retrieval systems commonly adopt the multi-stage pipelined approach
where in the first stage, an initial ranking is produced by an efficient model
which searches through the entire corpus which may contain millions of
passages and in the later stages, the ranking is refined by more precise
models. Many existing works use the model in the previous stage to generate
training examples for the current stage Nogueira and Cho (2019); Nogueira et
al. (2019, 2020); Pradeep et al. (2021a); Bonifacio et al. (2022). Gao et al.
Gao et al. (2021) point out sometimes an effective first-stage model may
introduce false positives for the next stages and propose localized
contrastive estimation learning.

##  3 Model

Given a query $q_{i}$ and a list of candidate passages
$C(i)={c_{1},c_{2},...,c_{n}}$ in a document collection $D$, the ranking task
aims to sort passages in the $C(i)$ such that more relevant passages have
higher scores. More formally, we aim to learn a scoring function
$s(q_{i},c_{j})$ such that ${c^{*}=argmax_{j\in C(i)}s(q_{i},c_{j})}$ is the
most relevant passage to the query.

###  3.1 Model structure

We follow Zhuang et al. (2022) to use a T5-based cross-attention model.
Specifically, we represent the query-passage pair as input sequence “Query:
{Query} Document: {Title. Passage}” and feed it into the encoder. The output
of the encoder is the encodings of the input sequence. We then apply a
projection layer on the encoding of the first token and the output is used as
the score. We use the encoder and discard the decoder allowing us to exploit
the encoder-decoder pretraining while not requiring a decoder for inference.
During inference, we pair query $q_{i}$ with each passage in $C(i)$ and
compute scores. The ranking result is obtained by sorting the passages based
on their scores. The structure is shown in Figure 1.

The loss function we use is a listwise softmax cross entropy loss Bruch et al.
(2019) and is defined as follows:

$\ell=\sum_{i=1}^{m}\hat{y}_{ij}\log\Big{(}\frac{e^{s_{ij}}}{\sum_{j^{\prime}}e^{s_{ij^{\prime}}}}\Big{)}$

where $s_{ij}$ is the predicted ranking score on query $q_{i}$ and passage
$c_{j}$, and $\hat{y}_{ij}$ is the relevance label.

Figure 1: The architecture of T5-based reranking model.

###  3.2 Training data generation

Usually the number of passages in $C(i)$ is much smaller than the total number
of passages in the document collection. How to construct $C(i)$ is important
and affects the performance of the ranking model. The widely used strategy is
to sample passages returned by a retriever, such as BM25 Muennighoff (2022);
Bonifacio et al. (2022). In the pipelined multi-stage retrieval system
consisting of retrieval and reranking stages, $C(i)$ is usually formed by
using the first-stage retriever. However, a better first-stage retriever not
always guarantee a better training set for reranking models. In this work, we
present a strategy to train robust rerankers.

We use a hybrid retriever to generate passage lists, inspired by Ma et al.’s
Ma et al. (2021) hybrid first-stage retriever. Specifically, we use a BM25
model as the sparse retrieval model and a T5-based dual encoder model Ni et
al. (2022) as the dense retrieval model. For each passage (and query), we
concatenate the encodings from the two models to create a hybrid encoding. We
perform maximal inner-product search (MIPS) using approximate nearest neighbor
search over these hybrid encodings. This results in a different set of
neighbors than independently selecting neighbors from BM25 and the dual
encoder.

For the BM25 model, we represent each query as a $|V|$-dimensional binary
encoding $\textbf{q}^{\text{bm25}}$, where $\textbf{q}^{\text{bm25}}[i]$ is
$k$ if the i-th entry of vocabulary $V$ is seen $k$ times in the query, 0
otherwise (e.g., this is a vector of query term counts). We represent each
passage as a sparse real-valued vector $\textbf{c}^{\text{bm25}}$:

$\textbf{c}^{\text{bm25}}[i]=\frac{\text{IDF}(t_{i})*\text{cnt}(t_{i},c)*(k+1)}{\text{cnt}(t_{i},c)+k*(1-b+b*\frac{m}{m_{\text{avg}}})}$

where $t_{i}$ are tokens from passage $c$, cnt($t_{i}$, $c$) is $t_{i}$’s term
frequency in $c$, $k$/$b$ are BM25 hyperparameters, IDF is the term’s inverse
document frequency from the document collection, $m$ are the number of tokens
in $c$, and $m_{\text{avg}}$ is the collection’s average passage length. We
can recover the original BM25 score via dot-product between the query vector
and the passage vector.

Our dual encoder is based on T5 Raffel et al. (2020) as well. We adapt the
encoder-only structure from Ni et al.Ni et al. (2022)’s GTR model. To encode a
query, we feed the query text to T5 encoder and use the mean pooling of the
encoder output as the query encoding $\textbf{q}^{\text{de}}$. A passage
encoding $\textbf{c}^{\text{de}}$ is generated in a similar way but we
concatenate the passage and corresponding document title as the input to T5
encoder. The query tower and the passage tower share parameters. The query to
passage relevance is computed by the cosine similarity of their encodings. We
optimize the model using the in-batch sampled softmax loss Henderson et al.
(2017). The structure is shown in Figure 2.

Figure 2: The architecture of T5 dual encoder model.

To benefit from both sparse model and dense neural model, we create the hybrid
model by combining the encodings from two models in a principled way:

$\begin{split}\text{sim}(\textbf{q}^{\text{hyb}},\textbf{c}^{\text{hyb}})&=\langle\textbf{q}^{\text{hyb}},\textbf{c}^{\text{hyb}}\rangle\\\
&=\langle[\textbf{q}^{\text{bm25}},\lambda\textbf{q}^{\text{de}}],[\textbf{c}^{\text{bm25}},\textbf{c}^{\text{de}}]\rangle\\\
&=\langle\textbf{q}^{\text{bm25}},\textbf{c}^{\text{bm25}}\rangle+\lambda\langle\textbf{q}^{\text{de}},\textbf{c}^{\text{de}}\rangle\end{split}$

where $\textbf{q}^{\text{hyb}}$ and $\textbf{c}^{\text{hyb}}$ are the hybrid
encodings that concatenate the BM25
($\textbf{q}^{\text{bm25}}$/$\textbf{c}^{\text{bm25}}$) and the dual encoder
encodings ($\textbf{q}^{\text{de}}$/$\textbf{c}^{\text{de}}$) described above;
and $\lambda$ is an interpolation hyperparameter that trades-off the relative
weight of BM25 versus the dual encoder models.

To generate the training data for reranking model, we apply the hybrid
retriever to the queries in the training set and retrieve top-K passages. We
then sample $N$ negatives from retrieved result. In result, $C(i)$ is a
passage list of size $N+1$ with one positive and $N$ negatives.

##  4 Experimental Setup

We evaluate our proposed approach on two tasks: first is the MS MARCO passage
ranking task to understand how our approach performs in supervised setting
when labeled training data is available. Second is the zero-shot retrieval on
BEIR where no labeled data is available in the target domains.

###  4.1 MS MARCO Passage Ranking

The MS MARCO passage ranking task aims to retrieve passages from a collection
of web documents containing about 8.8 million passages. All questions in this
dataset are sampled from real and anonymized Bing queries Nguyen et al.
(2016). The dataset contains 532,761 and 6980 examples in the training and
development set respectively. We report our results using MRR@10 metric on the
development set. We use the GTR-Large from Ni et al. (2022) as the dual
encoder used in the hybrid retriever.

###  4.2 Zero-shot Retrieval

We also perform evaluation on the BEIR corpus Thakur et al. (2021), a
benchmark for zero-shot evaluation, to understand how our approach generalizes
to out-of-domain setting. BEIR contains 18 evaluation datasets across 9
domains and no training data is available for those datasets.

To train the dual encoder used for hybrid retriever, we follow Ma et al. Ma et
al. (2021) to generate synthetic training data from a query generator and
extend with iterative training. Specifically, following Dai et al. (2022), we
pretrain the dual encoder on C4 dataset Raffel et al. (2020) with the
independent cropping task Izacard et al. (2021). We then fine-tune the dual
encoder using synthetically generated queries. We apply a T5-based query
generation model (QGen) on the passages in the target corpus to generate
(synthetic query, passage) pairs. For each dataset, we iterate over every
passage and treat every sentence as the target to generate synthetic queries.
For large datasets, such as BioASQ and Climate-fever, we randomly sample 2
million passages for query generation. The statistics of BEIR and the number
of synthetic data can be found in the Table 1. We then use these data to train
a dual encoder model $DE_{0}$. To filter low quality questions, we apply
round-trip consistency for retrievalAlberti et al. (2019); Lewis et al.
(2021); Dai et al. (2022). Specifically, given a synthetic query in the
training data, we run 1-nearest neighbor search based on scores between the
query and all passages using $DE_{0}$. If the neighbor is the one from which
the query is generated, we keep that (query, passage). Otherwise, the (query,
passage) pair is filtered. With the filtered data, we continue fine tune
$DE_{0}$ to get the final dual encoder model $DE_{1}$.

The QGen model is created by fine-tuning a general T5 model using question and
passage pairs from Natural Question (NQ) Kwiatkowski et al. (2019). This is
the only place that we use supervised data in the pipeline. Particularly, we
form the input of the encoder as “Generate question `>>>` title.passage `>>>`
target sentence”, and the output of the decoder is the corresponding question.
Here “target sentence” is the sentence that contains the short answer span,
and “passage” corresponds to long answer and the passage of NQ. At inference
time, given a passage, we iterate over every sentence as the target to
generate diverse queries. Similar to PAQ Lewis et al. (2021), we perform
targeted generation, where knowing the location of the answer in a passage is
important.

Datasets | Domain |  | # Test
---
Query
| # Corpus
---
Doc.
| # Synth.
---
Ques.
| # Synth.
---
Ques.
After
Filter
NQ | Wiki | 3452 | 2.68M | 2.62M | 976K
MS MARCO | Misc. | 6980 | 8.84M | 3.74M | 1.08M
Trec-Covid | BioMed | 50 | 171K | 753K | 352K
BioASQ | BioMed | 500 | 14.91M | 10.75M | 3.44M
NFCorpus | BioMed | 323 | 3.6K | 22K | 13K
HotpotQA | Wiki | 7405 | 5.23M | 4.78M | 3.19M
FiQA-2018 | Finance | 648 | 57K | 255K | 157K
Signal-1M | Twitter | 97 | 2.86M | 2.58M | 1.03M
Trec-News | News | 57 | 595K | 1.64M | 618K
Robust04 | News | 249 | 528K | 2.65M | 1.05M
ArguAna | Misc. | 1406 | 8.67K | 41K | 21K
Touché-2020 | Misc. | 49 | 382K | 1.34M | 348K
Quora | Quora | 10000 | 523K | 552K | 400K
DBPedia-entity | Wiki | 400 | 4.63M | 4.76M | 3.25M
SCIDOCS | Science | 1000 | 25K | 136K | 114K
Fever | Wiki | 6666 | 5.42M | 6.41M | 4.22M
Climate-Fever | Wiki | 1535 | 5.42M | 6.41M | 4.23M
SciFact | Science | 300 | 5K | 27.3K | 20K
CQADupStack | StackEx. | 13145 | 457K | 1.55M | 1.18M

Table 1: The statistics of BEIR benchmark and number of synthetic questions
before and after filtering.

Results are obtained using the official TREC evaluation
tool111https://github.com/usnistgov/trec_eval. We report normalised cumulative
discount gain (nDCG@10) and recall@100 for all datasets.

###  4.3 Implementation Details

The query generation model is initialized from T5 Version 1.1 XL model and we
fine-tune using question and passage pairs from NQ Kwiatkowski et al. (2019)
for 20,000 steps with a learning rate of 0.01, batch size 256 and dropout rate
0.1.

The BM25 model in our hybrid retriever is a unigram model. We use the
WordPiece tokenizer and vocabulary from uncased BERTbase of size 30522. We use
K=0.9 and b=0.8 on all datasets. We only index the passages. The dual encoder
model is initialized from the public pre-trained T5 Version 1.1 large model
and pre-train on C4 for 200k steps with batch size 4096. We encode queries and
passages into vectors of size 64 and 512 for all datasets except Climate-Fever
and ArguAna. We use query length 100 for Climate-Fever and 512 for ArguAna. We
use batch size of 5120 and train $DE_{0}$ for 100 epochs. $DE_{1}$ is
initialized from $DE_{0}$ and we train 10 epochs. For the hybrid retrieval
model, we use $\lambda=600$ in searching from 50 to 750 with step size 50.

The reranking models are initialized from T5 Version 1.1 models: Base and
Large. Since we only use the encoders, the number of parameters are about 125M
for Base model and 400M for Large model. We sample 50 negative examples from
top 250 retrieved passage for MS MARCO and top retrieved passage from rank 10
to rank 210 for BEIR. We use input sequence length as 512 for all datasets
except ArguAna, for which we use 1024. We train the models for 20000 steps
with batch size 64.

All of the above hyperparameters were tuned on the MS MARCO development set
and then applied to all other datasets, allowing for a true zero-shot setting
(not to mention that not every dataset has a development set). From
preliminary experiments, we found that tuning parameters for each dataset may
achieve improved results due to the nature of each dataset. For example, when
tuning the weight $\lambda$ between BM25 and dual encoder model in the hybrid
retriever, the value should be smaller if we observe more term overlapping
between questions and passage collection, such as SciFact and BioASQ, to
emphasize the BM25 model. However, for the results in this paper, we have use
the parameters which were selected based on the MS MARCO dataset.

We implement the neural models using T5X222https://github.com/google-
research/t5x and we also use RAX Jagerman et al. (2022), a learning-to-rank
framework for implementing the ranking losses in reranking models. For
training, it takes about 6 hours to train a dual encoder model and 6.5 hours
to train a reranking model of T5-Large size using Cloud TPU-V3.

| Model size | MRR@10
---|---|---
BM25 Anserini |  | 0.1874
HLATR | RoBERTaLarge | 0.3680
MiniLM | Distilled BERT | 0.3901
monoT5 | T53B | 0.3980
RankT5-EncDec | T5Large | 0.3986
RankT5 (Ours) | T5Large 1.1 | 0.4222
HYRR | T5Large 1.1 | 0.4235
Table 2: Reranking performance on MS MARCO Dev set in MRR@10.

| Retriever | Reranker
---|---|---
|  | Supervised Baselines | Unsupervised Baselines | Ours
| BM25 | MiniLM | NQRR | MSHYRR | SGPT | UPR | HYRR | HYRR | HYRRft | HYRRft
Rerank Top K | (Anserini) | 100 | 100 | 100 | 100 | 1000 | 100 | 100 | 100 | 1000
Model Size |  | 22M | 400M | 400M | 6.1B | 3B | 125M | 400M | 400M | 400M
NQ | 0.329 | 0.533 | 0.624‡ | 0.569 | 0.401 | 0.454 | 0.532 | 0.555 | 0.573 | 0.626
MS MARCO | 0.228 | 0.413‡ | 0.330 | 0.435‡ | 0.290 | 0.302 | 0.307 | 0.309 | 0.319 | 0.344
Trec-Covid | 0.656 | 0.757 | 0.787 | 0.798 | 0.791 | 0.688 | 0.796 | 0.820 | 0.822 | 0.817
BioASQ | 0.465 | 0.523 | 0.507 | 0.554 | 0.547 | - | 0.551 | 0.549 | 0.554 | 0.565
NFCorpus | 0.325 | 0.350 | 0.345 | 0.371 | 0.347 | 0.348 | 0.379 | 0.382 | 0.391 | 0.396
HotpotQA | 0.603 | 0.707 | 0.665 | 0.717 | 0.699 | 0.733 | 0.706 | 0.707 | 0.708 | 0.730
FiQA-2018 | 0.236 | 0.347 | 0.397 | 0.411 | 0.401 | 0.444 | 0.408 | 0.437 | 0.437 | 0.470
Signal-1M | 0.330 | 0.338 | 0.271 | 0.264 | 0.323 | - | 0.307 | 0.318 | 0.304 | 0.280
Trec-News | 0.398 | 0.431 | 0.419 | 0.452 | 0.466 | - | 0.437 | 0.453 | 0.441 | 0.418
Robust04 | 0.407 | 0.475 | 0.400 | 0.505 | 0.480 | - | 0.501 | 0.544 | 0.534 | 0.552
ArguAna | 0.414 | 0.311 | 0.107 | 0.351 | 0.286 | 0.372 | 0.344 | 0.342 | 0.382 | 0.326
Touché-2020 | 0.367 | 0.271 | 0.363 | 0.467 | 0.234 | 0.206 | 0.368 | 0.384 | 0.366 | 0.287
Quora | 0.789 | 0.825 | 0.807 | 0.637 | 0.794 | 0.831 | 0.861 | 0.867 | 0.869 | 0.868
DBPedia-entity | 0.313 | 0.409 | 0.391 | 0.402 | 0.370 | 0.534 | 0.385 | 0.403 | 0.408 | 0.443
SCIDOCS | 0.158 | 0.166 | 0.182 | 0.184 | 0.196 | 0.170 | 0.183 | 0.187 | 0.198 | 0.201
Fever | 0.753 | 0.819 | 0.871 | 0.825 | 0.725 | 0.591 | 0.868 | 0.861 | 0.851 | 0.856
Climate-Fever | 0.213 | 0.253 | 0.321 | 0.262 | 0.161 | 0.117 | 0.272 | 0.294 | 0.275 | 0.271
SciFact | 0.665 | 0.688 | 0.696 | 0.745 | 0.682 | 0.703 | 0.734 | 0.754 | 0.755 | 0.767
CQADupStack | 0.299 | 0.370 | 0.397 | 0.368 | 0.420 | 0.416 | 0.398 | 0.416 | 0.421 | 0.444
Average | 0.418 | 0.473 | 0.467 | 0.490 | 0.453 | 0.461 | 0.491 | 0.504 | 0.506 | 0.508
Average w/o NQ | 0.423 | 0.470 | 0.459 | 0.486 | 0.456 | 0.461 | 0.489 | 0.501 | 0.502 | 0.502
Avg. improvement on BM25 | 4.63% | 3.54% | 6.26% | 3.29% | 3.11% | 6.58% | 7.81% | 7.86% | 7.87%

Table 3: Reranking performance on BEIR in NDCG@10. ‡ indicates the in-domain
performances. The results of baseline models are copied verbatim from the
original papers. “-” indicate results that are not reported. Baseline models
rerank the top-100 passages from BM25 except UPR that reranks the top-1000
passages.

##  5 Results and Discussion

###  5.1 Results on MS MARCO

To understand the effectiveness of our proposed approach, we fix the first-
stage retrieval system and compare the reranking performance. Table 2 shows
the performance of our proposed reranker on reranking BM25 top-1000 results.
The BM25 results in row 1 is obtained from Anserini Yang et al. (2017)
toolkit333https://github.com/castorini/anserini with parameters: k=0.82,
b=0.68 following other baselines. The results of several strong baselines are
shown in row 2-6. HLATR Zhang et al. (2022) extends the retrieval-and-rerank
pipeline with an additional ranking module by using the features from
retrieval and reranking stages. It achieves top performance on MS MARCO
leaderboard. MiniLM Wang et al. (2020), which is a cross-encoder reranking
model distilled from an ensemble of three teacher models: BERT-base, BERT-
large and ALBERT-large. The other baselines are T5-based models: monoT5 Raffel
et al. (2020); Rosa et al. (2022) and RankT5-EncDec Zhuang et al. (2022) adopt
the encoder-decoder architecture. Our model adopts RankT5’s encoder-only
variant as described in Section 3.1. To compare with RankT5 model fairly, we
implement our version of RankT5 (ours), which shares the same architecture and
parameter settings as HYRR. They only differ from the training data
generation. Our reproduced RankT5 generates training data from dual encoder
retriever. From row 7, we can see that HYRR improves over BM25 performance by
23.6% in MRR@10, and outperforms other baselines. This demonstrates that our
proposed training framework is effective in supervised setting.

###  5.2 Results on BEIR

Similar as evaluation on MS MARCO, we fix the first-stage retrieval system and
compare the reranking performance. Table 3 shows the reranking performance of
our proposed reranker. The BM25 results in Col.1 are obtained from Anserini
toolkit with parameters:k=0.9 and b=0.4 following other baselines.

We consider several reranking models that can be categorized in two zero-shot
settings as baselines. The first one is to train a supervised reranker and
perform inference on the target domains directly, and the results are shown in
Col. 2-4. We choose MiniLM Wang et al. (2020), which reranks the top 100
retrieved passages from BM25. The result on MS MARCO is considered as in-
domain for this model. We train a T5-based reranker NQRR using supervised NQ
data and the model structure is the same as the one described in Section 3.1.
The result on the NQ dataset is considered in-domain for the NQRR. We also use
our reranker MSHYRR trained using our proposed approach on MS MARCO from
Section 4.1 as baseline.

The second setting is unsupervised, and we choose two models to compare. SGPT
Muennighoff (2022) uses a pre-trained GPT model in a cross-encoder setting for
reranking. Specifically, query text and passage text are concatenated and fed
into pre-trained GPT model. The log probability is used as the score for
reranking. Col. 5 shows the results of reranking top 100 retrieved passages
with model size of 6.1 billion parameters. UPR Sachan et al. (2022) also uses
a pre-trained language model for reranking. UPR uses a T5 model by taking the
passage and a simple prompt as input and uses the average likelihood of query
tokens conditioned on the passage as the score for reranking. Col. 6 shows the
results of reranking top 1000 retrieved passages using T0 model Sanh et al.
(2022) with 3 billion parameters.

Retriever $\downarrow$ | No Reranker | BM25RR | DERR | HYRR | No Reranker | BM25RR | DERR | HYRR
---|---|---|---|---|---|---|---|---
| MS MARCO
| MRR@10 | nDCG@10
BM25 | 0.187 | 0.375 | 0.422 | 0.424 | 0.234 | 0.438 | 0.485 | 0.486
DE | 0.378 | 0.350 | 0.440 | 0.440 | 0.445 | 0.411 | 0.507 | 0.507
Hybrid | 0.390 | 0.351 | 0.438 | 0.440 | 0.457 | 0.413 | 0.506 | 0.508
| SciFact
| nDCG@10 | Recall@100
BM25 | 0.677 | 0.750 | 0.742 | 0.752 | 0.918 | 0.916 | 0.923 | 0.946
DE | 0.597 | 0.755 | 0.745 | 0.752 | 0.903 | 0.919 | 0.923 | 0.946
Hybrid | 0.706 | 0.753 | 0.744 | 0.759 | 0.941 | 0.931 | 0.931 | 0.958
Table 4: Ablation results on MS MARCO and SciFact.

For our models, we show results of two training settings. HYRR is a fine-tuned
vanilla T5 model, while HYRRft is a fine-tuned NQRR model. The results are
shown in Col. 7-10. As we can see, our proposed model outperform the baseline
models in 10 of 18 out-of-domain datasets. The best setting is reranking the
top 1000 retrieved passages using HYRRft and the improvement of nDCG@10 over
BM25 is from 2% - 23.4%, and in average 7.87%. Comparing Col. 8 and Col. 9, we
can see that HYRR already provides strong performance on many datasets, and
fine-tuning an out-of-domain supervised reranker achieves an additional
performance boost. We note that the results on NQ cannot be considered
completely out-of-domain even for our HYRR as the question generation model
used to generate training data for the hybrid retriever is trained on the NQ
dataset. Similarly, UPR’s results on HotpotQA cannot be considered completely
out-of-domain since T0 uses HotpotQA for training. The evaluation on BEIR
demonstrates the effectiveness of our proposed method in zero-shot settings.

Query |  what mlb team play in
---|---
HYRR |  P1 major league baseball (mlb) is a professional baseball organization, … a total of 30 teams now play in the american league (al) and national league (nl)…
|  P2 in the united states and canada, professional major league baseball
(mlb) teams are divided into the national league (nl) and american league
(al), each with three divisions…"
BM25RR |  P3 major league baseball. the 16 teams were located in ten cities, all in the northeastern and midwestern united states: new york city had three teams and boston, chicago, ….
|  P4 list of current major league baseball stadiums. the newest mlb stadium
is suntrust park in cumberland, georgia, home of the atlanta braves, which
opened for the 2017 season.
DERR |  P5 major league baseball. the arizona diamondbacks, …, are a major league baseball (mlb) franchise that play in the west division….
|  P6 boston red sox. the yankees compete in major league baseball (mlb) as a
member club of the american league (al) east division….

Table 5: Reranker predictions on MS MARCO

###  5.3 Ablation

To show the robustness of HYRR, we conduct an ablation experiment. We train
rerankers using the training data generated from the BM25 or the dual encoder
model, namely BM25RR and DERR. Those two variants are commonly seen in many
pipelined retrieval systems, where rerankers are simply trained upon the
first-stage retriever. We train them using the same training setting for HYRR
and then apply them on three retrievers: the BM25 model, the dual encoder
model (DE) and the hybrid retriever, respectively. We experiment on both
supervised setting and zero-shot setting. The results on MS MARCO are shown in
the top section of Table 4. As we can see HYRR provides the most performance
gain over all three retrievers on both MRR@10 and nDCG@10. The BM25RR improves
the performance on BM25 while hurts the other two. DERR achieves best
performance when we apply it to DE. It also improves the other two retrievers
but not as much as HYRR. This shows that HYRR not only outperforms the other
two rerankers but also is effective on different retrievers.

Corpus |  Query Type |  Test Queries |  Synthetic Questions
---|---|---|---
FiQA-2018 |  Financial Question |  How does unemployment insurance work? |  what is the advantage of stocks over bonds
BioASQ |  Biomedical Question |  List types of DNA lesions caused by UV light. |  how many subjects in the silent lacunar lesion study
|  |  Does deletion of cohesin change gene expression? |  what happens to amino acids in the hypothermic kidney
Fever |  Claim |  The Hunger Games are not based on a novel. |  when did bag raiders album come out in australia
|  |  Pharmacology is a science. |  when did angela lien win her first olympic medal
SCIDOCS |  Paper Title |  Multi-task Domain Adaptation for Sequence Tagging |  what is the use of blockchain in software architecture
|  |  HF outphasing transmitter using class-E power amplifiers |  what is the relationship between test anxiety and mathematics performance
Signal-1M |  News Headline |  Inside Trevor Noah’s final test run for the new ‘Daily Show’ |  how many food items at the state fair
|  |  Fancy Cooking |  where is book week held in the uk
Touché-2020 |  Controversial Questions |  Is golf a sport? |  who is the democrat in the 2020 presidential debate
|  |  Is drinking milk healthy for humans? |  what kind of music can you debate on wikipedia
Arguana |  Argument |  Small businesses need advertisements to make their products known. If there wasn’t advertising then small businesses would have no chance at all to make their product well known. Adverts can actually level the playing field - if you have a good new product, and market it in a clever way then it doesn’t matter how small your company is, you can still make consumers interested…. |  who benefits from the mass media and advertising

Table 6: Examples of test questions in some corpora and generated synthetic
questions.

When examining the ranking outputs on MS MARCO, we find that top ranked
predictions from HYRR is more semantically and lexical relevant to the query,
while BM25RR and DERR sometimes return partially related predictions. For
example, query “what mlb team play in” which asks for general information
about major league baseball(mlb) shown in Table 5, HYRR returns the correct
predictions, while DERR returns predictions about individual team in mlb, and
BM25RR returns less relevant information about mlb.

We pick SciFact from BEIR as an example for zero-shot setting. The results are
shown in bottom part of Table 4, and we observe the similar trends on other
datasets in BEIR. Similarly, HYRR improves both nDCG@10 and Recall@100 over
all three retrievers. Specifically, it outperforms BM25RR and DERR not only
when we apply it to the matched Hybrid retriever but also the other two
retrievers. This believe is the evidence that the robustness of the training
data for the reranker is carried over to the robustness of the reranker
itself.

In addition, to understand the benefit to use hybrid retriever for training
data generation, we conduct another ablation experiment. We mix the training
data generated from the BM25 and the dual encoder model in 1:1 ratio and train
a reranker. We evaluate on MS MARCO and the model achieves 0.417 in MRR@10 and
0.480 in nDCG@10 when reranking BM25 top 1000. When comparing with the results
in row 1 from Table 4, we can see that our approach significantly outperforms
the approach that simply applying training data ensemble.

###  5.4 Discussion

Robustness against Initial Checkpoints. While when reranking the BM25 outputs
in the zero-shot settings, our best reranking model is the fine-tuned model on
a supervised reranking model (NQRR), it is worth to note that the zero-shot
reranking model also achieves strong performance and outperforms NQRR on many
datasets as shown in Table 3. This shows that our proposed training framework
is useful in many practical situation when supervised data is not available.

Robustness against Model Sizes. Regarding the model size, recent work shows
that increasing model size results in large gains in zero-shot performance for
reranking models. A 10% improvement in average nDCG@10 over all datasets in
BEIR can be seen when the model size increases from T5-small to T5-3B Rosa et
al. (2022). From Table 2 and 3, we can see that our proposed model is smaller,
but more effective than some larger models.

Robustness against Retrievers. We also show that despite the common wisdom
that the best reranking model will be one trained on the same retriever as
will be used at inference time, a model trained with hybrid retrieval output
is strong for both BM25 and dual encoder retrievers. Reranking BM25 resulted
in superior performance despite the mismatch between the retriever used for
generating training data and that used for retrieval. The ablation experiment
also confirms this observation.

Robustness against Query Generation. In addition, we analyze the synthetically
generated questions and show that our proposed model generalizes to different
domains and different query types. In Table 6, we show examples of queries
from the test set and the synthetically generated questions. As we can see,
when synthetic questions are similar to test queries, our model performs very
well even though the target domain is very different from the domain where the
query generator is trained. For example, questions in FiQA-2018 are from
investment topic and questions in BioASQ are scientific questions in
biomedical domain. Both retriever and reranker performs very well on those two
datasets. Interestingly, we find that our model also performs well in some
cases where test queries and synthetic questions are of different type. For
example, Fever asks to retrieve passages to verify given claims. The test
queries are claims rather than questions. A similar pattern exists for SCIDOCS
where the queries are paper titles. Our retrieval and ranking models learn the
relevance between queries and passages despite the test queries not being
questions.

However, when the synthetic question is very different from the test queries,
our model fails. For example, Signal-1M asks to retrieve relevant tweets given
news headlines. Another example is Touché-2020, where the task is to retrieve
relevant arguments given a question on a controversial topic. Also in Arguana,
the task is to retrieve counterargument to an argument, and queries are long
text passages. The nuance of the task itself and whether it requires term-
based or semantic abstractions to match queries to passages appears to be
where our QGen approach is ineffective.

##  6 Conclusion

We proposed a generic training framework for rerankers based on a hybrid
retriever. While the hybrid retriever is composed of term-based and neural
models, the reranker is a neural cross-attention model which learns from
negatives examples generated by the hybrid retriever. The proposed approach is
robust and outperforms several strong baselines on MS MARCO passage ranking
task and BEIR benchmark dataset, which demonstrates that it is practical and
generalized. We observe that a model trained with robust training instances
(in this case, from the hybrid retriever) produces a reranker that outperforms
matched-training rerankers for term-based or neural retrievers.

## References

  * Alberti et al. (2019) Chris Alberti, Daniel Andor, Emily Pitler, Jacob Devlin, and Michael Collins. 2019\.  Synthetic qa corpora generation with roundtrip consistency.  Florence, Italy.
  * Bonifacio et al. (2022) Luiz Bonifacio, Hugo Abonizio, Marzieh Fadaee, and Rodrigo Nogueira. 2022.  Inpars: Data augmentation for information retrieval using large language models.
  * Bruch et al. (2019) Sebastian Bruch, Xuanhui Wang, Michael Bendersky, and Marc Najork. 2019.  An analysis of the softmax cross entropy loss for learning-to-rank with binary relevance.  In _Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval_ , ICTIR ’19, page 75–78, New York, NY, USA. Association for Computing Machinery.
  * Chen et al. (2022) Tao Chen, Mingyang Zhang, Jing Lu, Michael Bendersky, and Marc Najork. 2022.  Out-of-domain semantics to the rescue! zero-shot hybrid retrieval models.  In _Advances in Information Retrieval - 44th European Conference on IR Research, ECIR 2022, Stavanger, Norway, April 10-14, 2022, Proceedings, Part I_ , volume 13185 of _Lecture Notes in Computer Science_ , pages 95–110. Springer.
  * Craswell et al. (2022) Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Jimmy Lin. 2022.  Overview of the trec 2021 deep learning track.  In _Text REtrieval Conference (TREC)_. TREC.
  * Dai et al. (2022) Zhuyun Dai, Vincent Y. Zhao, Ji Ma, Yi Luan, Jianmo Ni, Jing Lu, Anton Bakalov, Kelvin Guu, Keith B. Hall, and Ming-Wei Chang. 2022.  Promptagator: Few-shot dense retrieval from 8 examples.
  * Ekstrand et al. (2022) Michael D. Ekstrand, Graham McDonald, Amifa Raj, and Isaac Johnson. 2022.  Overview of the trec 2021 fair ranking track.  In _The Thirtieth Text REtrieval Conference (TREC 2021) Proceedings_.
  * Gao et al. (2021) Luyu Gao, Zhuyun Dai, and Jamie Callan. 2021.  Rethink training of bert rerankers in multi-stage retrieval pipeline.  In _Advances in Information Retrieval: 43rd European Conference on IR Research, ECIR 2021, Virtual Event, March 28 – April 1, 2021, Proceedings, Part II_ , page 280–286, Berlin, Heidelberg. Springer-Verlag.
  * Henderson et al. (2017) Matthew Henderson, Rami Al-Rfou, Brian Strope, Yun hsuan Sung, László Lukács, Ruiqi Guo, Sanjiv Kumar, Balint Miklos, and Ray Kurzweil. 2017.  Efficient natural language response suggestion for smart reply.  _ArXiv e-prints_.
  * Izacard et al. (2021) Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2021.  Unsupervised dense information retrieval with contrastive learning.
  * Jagerman et al. (2022) Rolf Jagerman, Xuanhui Wang, Honglei Zhuang, Zhen Qin, Mike Bendersky, and Marc Najork. 2022.  Rax: Composable learning-to-rank using jax.  In _KDD 2022_.
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020.  Dense passage retrieval for open-domain question answering.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6769–6781.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019.  Natural questions: a benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:453–466.
  * Lewis et al. (2021) Patrick Lewis, Yuxiang Wu, Linqing Liu, Pasquale Minervini, Heinrich Küttler, Aleksandra Piktus, Pontus Stenetorp, and Sebastian Riedel. 2021.  PAQ: 65 million probably-asked questions and what you can do with them.  _Transactions of the Association for Computational Linguistics_ , 9:1098–1115.
  * Ma et al. (2021) Ji Ma, Ivan Korotkov, Yinfei Yang, Keith B. Hall, and Ryan T. McDonald. 2021.  Zero-shot neural passage retrieval via domain-targeted synthetic question generation.  In _EACL_ , pages 1075–1088.
  * Muennighoff (2022) Niklas Muennighoff. 2022.  Sgpt: Gpt sentence embeddings for semantic search.  _arXiv preprint arXiv:2202.08904_.
  * Nguyen et al. (2016) Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016.  MS MARCO: A human generated machine reading comprehension dataset.  In _Proceedings of the Workshop on Cognitive Computation: Integrating neural and symbolic approaches 2016 co-located with the 30th Annual Conference on Neural Information Processing Systems_.
  * Ni et al. (2022) Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernández Ábrego, Ji Ma, Vincent Y. Zhao, Yi Luan, Keith B. Hall, Ming-Wei Chang, and Yinfei Yang. 2022\.  Large dual encoders are generalizable retrievers.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
  * Nogueira and Cho (2019) Rodrigo Nogueira and Kyunghyun Cho. 2019.  Passage re-ranking with bert.
  * Nogueira et al. (2020) Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. 2020.  Document ranking with a pretrained sequence-to-sequence model.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 708–718, Online. Association for Computational Linguistics.
  * Nogueira et al. (2019) Rodrigo Nogueira, Wei Yang, Kyunghyun Cho, and Jimmy Lin. 2019.  Multi-stage document ranking with bert.
  * Pradeep et al. (2021a) Ronak Pradeep, Rodrigo Nogueira, and Jimmy Lin. 2021a.  The expando-mono-duo design pattern for text ranking with pretrained sequence-to-sequence models.
  * Pradeep et al. (2021b) Ronak Pradeep, Rodrigo Nogueira, and Jimmy Lin. 2021b.  The expando-mono-duo design pattern for text ranking with pretrained sequence-to-sequence models.
  * Qu et al. (2021) Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. 2021.  RocketQA: An optimized training approach to dense passage retrieval for open-domain question answering.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5835–5847.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67.
  * Ren et al. (2021) Ruiyang Ren, Yingqi Qu, Jing Liu, Wayne Xin Zhao, QiaoQiao She, Hua Wu, Haifeng Wang, and Ji-Rong Wen. 2021.  RocketQAv2: A joint training method for dense passage retrieval and passage re-ranking.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 2825–2835, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Robertson and Zaragoza (2009) Stephen Robertson and Hugo Zaragoza. 2009.  The probabilistic relevance framework: Bm25 and beyond.  _Found. Trends Inf. Retr._ , 3(4):333–389.
  * Rosa et al. (2022) Guilherme Moraes Rosa, Luiz Bonifacio, Vitor Jeronymo, Hugo Abonizio, Marzieh Fadaee, Roberto Lotufo, and Rodrigo Nogueira. 2022.  No parameter left behind: How distillation and model size affect zero-shot retrieval.
  * Sachan et al. (2022) Devendra Singh Sachan, Mike Lewis, Mandar Joshi, Armen Aghajanyan, Wen-tau Yih, Joelle Pineau, and Luke Zettlemoyer. 2022.  Improving passage retrieval with zero-shot question generation.  _arXiv preprint arXiv:2204.07496_.
  * Sanh et al. (2022) Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. 2022.  Multitask prompted training enables zero-shot task generalization.  In _International Conference on Learning Representations_.
  * Thakur et al. (2021) Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021.  BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in neural information processing systems_ , pages 5998–6008. Curran Associates, Inc.
  * Wang et al. (2020) Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020.  Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers.  volume 33, pages 5776–5788.
  * Yang et al. (2017) Peilin Yang, Hui Fang, and Jimmy Lin. 2017.  Anserini: Enabling the use of lucene for information retrieval research.  In _Proceedings of the 40th international ACM SIGIR conference on research and development in information retrieval_ , pages 1253–1256.
  * Zhang et al. (2022) Yanzhao Zhang, Dingkun Long, Guangwei Xu, and Pengjun Xie. 2022.  Hlatr: Enhance multi-stage text retrieval with hybrid list aware transformer reranking.
  * Zhuang et al. (2022) Honglei Zhuang, Zhen Qin, Rolf Jagerman, Kai Hui, Ji Ma, Jing Lu, Jianmo Ni, Xuanhui Wang, and Michael Bendersky. 2022.  Rankt5: Fine-tuning t5 for text ranking with ranking losses.

##  Appendix A Loss function for T5 dual encoder

We train the duel encoder model using an in-batch sampled softmax loss:

$\ell=\frac{e^{\text{sim}(q_{i},\hat{p}_{i})/\tau}}{\sum_{j\in\mathcal{B}}{e^{\text{sim}(q_{i},\hat{p}_{j})/\tau}}}$

where $q_{i}$ is the question encoding and $\hat{p}_{i}$ is the encoding of
gold passage. sim is cosine similarity, $\mathcal{B}$ is a mini-batch of
examples, and $\tau$ is the softmax temperature.