# Retrieval for Extremely Long Queries and Documents with RPRS: a Highly
Efficient and Effective Transformer-based Re-Ranker

Arian Askari  0000-0003-4712-832X<EMAIL_ADDRESS>Leiden
UniversityThe Netherlands ,  Suzan Verberne  0000-0002-9609-9505
<EMAIL_ADDRESS>Leiden UniversityThe Netherlands ,  Amin
Abolghasemi  0009-0002-3725-7312<EMAIL_ADDRESS>Leiden
UniversityThe Netherlands ,  Wessel Kraaij  0000-0001-7797-619X
<EMAIL_ADDRESS>Leiden UniversityThe Netherlands and  Gabriella
Pasi  0000-0002-6080-8170<EMAIL_ADDRESS>University of Milano-
BicoccaItaly

(2022)

###### Abstract.

Retrieval with extremely long queries and documents is a well-known and
challenging task in information retrieval and is commonly known as Query-by-
Document (QBD) retrieval. Specifically designed Transformer models that can
handle long input sequences have not shown high effectiveness in QBD tasks in
previous work. We propose a Re-Ranker based on the novel Proportional
Relevance Score (RPRS) to compute the relevance score between a query and the
top-k candidate documents. Our extensive evaluation shows RPRS obtains
significantly better results than the state-of-the-art models on five
different datasets. Furthermore, RPRS is highly efficient since all documents
can be pre-processed, embedded, and indexed before query time which gives our
re-ranker the advantage of having a complexity of $O(N)$ where $N$ is the
total number of sentences in the query and candidate documents. Furthermore,
our method solves the problem of the low-resource training in QBD retrieval
tasks as it does not need large amounts of training data, and has only three
parameters with a limited range that can be optimized with a grid search even
if a small amount of labeled data is available. Our detailed analysis shows
that RPRS benefits from covering the full length of candidate documents and
queries.

Query-by-document retrieval · Sentence-BERT based ranking, Neural information
retrieval

††copyright: acmcopyright††journalyear: 2022††doi: XXXXXXX.XXXXXXX††ccs:
Information systems Novelty in information retrieval

##  1\. Introduction

Query-by-document (QBD) retrieval is a task in which a seed document acts as a
query – instead of a few keywords – with the aim of finding similar (relevant)
documents from a document collection (Yang et al., 2009, 2018; Le et al.,
2021). Examples of QBD tasks are professional, domain-specific retrieval tasks
such as legal case retrieval (Yoshioka, 2021; Tran et al., 2020; Askari and
Verberne, 2021; Shao et al., 2020), patent prior art retrieval (Piroi et al.,
2011; Piroi and Hanbury, 2019; Fujii et al., 2007), and scientific literature
retrieval (Cohan et al., 2020; Mysore et al., 2021b). In the literature, three
other terms are used to refer to this type of task: Query-by-Example (QBE)
retrieval, document similarity ranking, and document-to-document similarity
ranking.

Transformer-based ranking models (Vaswani et al., 2017), such as BERT-based
rankers (Nogueira and Cho, 2019; Yan et al., 2019) have yielded improvements
in many IR tasks. However, the time and memory complexity of the self-
attention mechanism in these architectures is $O(L^{2})$ for a sequence of
length $L$ (Kitaev et al., 2020). For that reason, architectures based on BERT
have an input length limitation of 512 tokens. This causes challenges in QBD
tasks where we have long queries and documents. For instance, the average
length of queries and documents in the legal case retrieval task of COLIEE
2021 (Askari and Verberne, 2021) is more than $5k$ words. Variants of
Transformers that aim to cover long sequences such as LongFormer (Beltagy et
al., 2020) and Big Bird (Zaheer et al., 2020) have not shown high
effectiveness, which could be due to either their scattered attention
mechanism (Sekulić et al., 2020) or to the limited number of training
instances for QBD tasks (Askari and Verberne, 2021). To overcome these
limitations, in this paper we propose an effective and efficient Transformer-
based re-ranker for QBD retrieval that covers the whole length of the query
and candidate documents without memory limitations. We focus on re-ranking in
a two-stage retrieval pipeline following prior studies that address retrieval
tasks in a multi-stage retrieval pipeline (Nogueira and Cho, 2019; Padigela et
al., 2019; Abolghasemi et al., 2022b; Chen et al., 2022).

In domain-specific tasks, there is often the need for expensive professional
searchers as annotators in order to create QBD test collections, e.g., lawyer
annotators for case law retrieval tasks. This makes the annotation process for
QBD tasks expensive and, as a result, low-resource training sets are a
significant issue in these tasks (Askari and Verberne, 2021). One such example
is the training set of the COLIEE 2021 dataset for case law retrieval that
consists of only $3,297$ relevant documents for $900$ queries as training
instances, which is very few compared to general web search datasets, e.g. MS
MARCO (Nguyen et al., 2016) has approximately $161$ times more training
instances ($532,761$ queries). This limitation is overcome by our proposed
method, RPRS (a Re-ranker based on Proportional Relevance Score computation),
which does not need large amounts of training data because it has three
parameters with a limited range that can be – but do not necessarily need to
be – optimized using a grid search if a small amount of labelled data is
available. Compared to BM25, our proposed method has only one more parameter,
which we optimize over 10 different possible values. The two other parameters’
ranges are equal to those of BM25.

Given a query document and a set of candidate documents, we split the query
and document text into sentences and use Sentence-BERT (SBERT) bi-encoders
(Reimers and Gurevych, 2019) to embed the individual sentences. Then, a
relevance score is computed by the RPRS relevance model we propose in this
paper, which is based on the similarity of individual sentences between a
query and a document. The intuition behind the proposed method is based on the
following assumption: a candidate document from the ranked list of documents
in response to a query is more likely to be relevant if it contains a
relatively high number of sentences that are similar to query sentences
compared to other documents’ sentences in the ranked list. Since RPRS has been
designed based on this assumption, we conduct an experimental investigation to
assess the quality of our proposed method by addressing the following research
questions:

RQ1: What is the effectiveness of RPRS compared to the State-of-the-art models
for QBD retrieval?

To answer this question we evaluate the effectiveness of RPRS on legal case
retrieval (i.e. given a legal case, find the related cases) using the COLIEE
2021 dataset (Rabelo et al., 2022). As RPRS uses a SBERT model in order to
embed query and document sentences, we first run a series of experiments to
find the SBERT model with which RPRS achieves the highest effectiveness.
Additionally, we investigate two different unsupervised and self-supervised
domain adaption approaches on the top-two most effective SBERT models without
domain adaption. We conclude that RPRS achieves higher effectiveness with
domain-specific Transformers in the SBERT architecture when adapting them to
the domain by using TSDA (Wang et al., 2021a). We continue the experiments
based on this finding. We also compare RPRS to two prior models: (1) the
state-of-the-art model on the COLIEE dataset: MTFT-BERT (Abolghasemi et al.,
2022b), and (2) Self-Supervised Document Similarity Ranking (SDR) (Ginzburg et
al., 2021) which also uses bi-encoders for sentence-level representation to
compute relevance scores and is the state-of-the-art on two Wikipedia document
similarity datasets.

For relevance estimation with RPRS, we split both the query and document
content into sentences. However, there are other ways for splitting query and
document. Therefore, we investigate if sentences are the most appropriate and
informative textual units for RPRS by addressing:

RQ2: How effective is RPRS with shorter or longer text units instead of
sentences?

Next, for analyzing the cross-data generalizability of the proposed method we
address the following question:

RQ3: What is the effectiveness of RPRS with parameters that were tuned on a
different dataset in the same domain?

To address this question, we have evaluated the proposed method on the
_Caselaw_ dataset (Locke et al., 2017) with the parameters that were tuned on
the COLIEE dataset without doing domain adaption on the collection for the
Transformer model. In other words, our goal is to analyze how much the three
optimized parameters of RPRS on the COLIEE dataset are transferable to another
dataset in the same (i.e., legal) domain. Next, we assess the generalizability
of our method by addressing:

RQ4: To what extent is RPRS effective and generalizable across different
domains with different type of documents?

In this regard, we evaluate our method on two different domains using datasets
of patents and Wikipedia webpages. For patents, we use CLEF-IP 2011 (Piroi et
al., 2011), a patent prior art retrieval dataset, and for Wikipedia we assess
the effectiveness of our method on WWG and WWA datasets, which are two
datasets for similarity ranking between Wikipedia pages in the wine and video
game domains. We show that the proposed method achieves higher ranking
effectiveness over the state-of-the-art in all five datasets (COLIEE, Caselaw,
CLEF-IP 2011, WWG, WWA).111The implementation and models will be available
after publication.

In summary, our contributions are as follows:

  1. (1)

We propose an effective and highly efficient re-ranker (RPRS) for QBD tasks
that covers the full text of query and candidate documents without any length
limitation. The efficiency of our method is justified by using bi-encoders and
its effectiveness is evaluated on five datasets.

  2. (2)

RPRS is suitable for QBD retrieval datasets with low-resource training data
since it has only three parameters with limited range that can be optimized
using a grid search even if a relatively low amount of labelled data is
available.

  3. (3)

We show how the use of various SBERT models and adapting them on domain
specific datasets affects the effectiveness of RPRS. This results in three
SBERT models in the legal, patent, and Wikipedia domains, which we will make
publicly available on Huggingface.

  4. (4)

Our proposed model outperforms the state-of-the-art models for each of the
five benchmarks, including SDR (Ginzburg et al., 2021), which was also
proposed as a sentence-based ranker for QBD tasks.

  5. (5)

An ablation study shows that Document Proportion – the proportion of the
document that is relevant to the query – is the most important component in
RPRS. In addition, we found the crucial role of covering the full length of
queries and documents for QBD tasks by studying the effect of feeding RPRS
with truncated queries and documents.

The structure of the paper is as follows: after a discussion of related work
in Section 2, we describe the proposed method in section 3 and the details of
the BM25 and SDR baselines in section 4. The experiments and implementation
details are covered in Section 5. The results are examined and the research
questions are addressed in Section 6. In section 7, we investigate and discuss
our proposed method in more-depth. Finally, the conclusion is described in
Section 8.

##  2\. Related Work

In the following, we first introduce Query-by Document (QBD) retrieval tasks.
Then, we provide an overview on prior methods for sentence embedding and pre-
training. Finally, we give an overview of previous works on sentence-level
retrieval methods.

###  2.1. QBD Tasks

Legal case retrieval. In countries with _common law_ systems, finding
supporting precedents to a new case, is vital for a lawyer to fulfill their
responsibilities to the court. However, with the large amount of digital legal
records – the number of filings in the U.S. district courts for total cases
and criminal defendants was and is 544,460 in
2020222https://www.uscourts.gov/statistics-reports/judicial-business-2020 – it
takes a significant amount of time for legal professionals to scan for
specific cases and retrieve the relevant sections manually. Studies have shown
that attorneys spend approximately 15 hours in a week seeking case law
(Lastres, 2015). This workload necessitates the need for information retrieval
(IR) systems specifically designed for the legal domain. The goal of these
systems is to assist lawyers in their duties by exploiting AI and traditional
information retrieval methods. One particular legal IR task is case law
retrieval.

The majority of the work on case law retrieval takes place in COLIEE, the
Competition on Legal Information Extraction and Entailment (Rabelo et al.,
2020, 2022). Askari and Verberne (2021) combine lexical and neural ranking
models for legal case retrieval. They optimize BM25 and obtain state-of-the-
art results with lexical models. However, recently, on top of the optimized
BM25, Abolghasemi et al. (2022b) present multi-task learning as a re-ranker
for QBD retrieval and set the new state-of-the-art. The limitation of the
method by (Abolghasemi et al., 2022b) is that the input is limited by the BERT
architecture (512 tokens) (Devlin et al., 2018). Therefore, it cannot cover
the full-text of long documents in QBD datasets like COLIEE.

Patent prior art retrieval. Patents serve as stand-ins for various domains
including technological, economic, and even social activities. The
Intellectual Property (IP) system encourages the disclosure of innovative
technology and ideas by granting inventors exclusive monopoly rights over the
economic value of their inventions. Therefore, according to Piroi et al.,
(Piroi et al., 2011) patents have a considerable impact on the market value of
companies. With the number of patent applications filed each year continuing
to rise, the need for effective and efficient solutions for handling such huge
amounts of information grows more essential. There are various patent analysis
tasks. In this work, we focus on patent prior art retrieval, the aim of which
is to find patent documents in the target collection that may invalidate a
specific patent application (Piroi et al., 2011; Piroi and Hanbury, 2019).

Patent retrieval is a challenging task: the documents are lengthy, the
language is formal, legal, and technical with long sentences (Verberne et al.,
2010). Shalaby et al. (Shalaby and Zadrozny, 2019) provide a detailed review
of research on patent retrieval that shows that the most successful methods on
the CLEF-IP (Piroi et al., 2011) benchmarks are traditional lexical-based
methods (Piroi et al., 2013; Verberne and D’hondt, 2009), sometimes extended
with syntactic information (D’hondt et al., 2011). There are more recent
studies that focus on patent retrieval in passage-level assessment (Hofstätter
et al., 2019; Althammer et al., 2021). However, recent studies do not address
document-level patent retrieval due to the limitation of Transformer-based
methods on taking into account the full length of the documents.

Document similarity for Wikipedia-based datasets. Estimating the similarity
within Wikipedia pages is useful for many applications such as clustering,
categorization, finding relevant web pages, etc. Ginzburg et al. (2021)
propose two new datasets on Wikipedia annotated by experts. The documents come
from the wine and video game domains; we refer to the collections as (1)
Wikipedia video games (WVG) and (2) Wikipedia wine articles (WWA). For both
datasets, the task is finding relevant Wikipedia pages given a seed page.

###  2.2. Sentence embedding

Sentence embedding is a well-studied topic and it is suitable for measuring
sentence similarity, clustering, information retrieval via semantic search,
etc. BERT-based sentence embedding methods employ Transformer models to
effectively and efficiently embed sentences. The challenge is that BERT’s
architecture makes it inappropriate for semantic similarity search in its
original configuration since it requires both sentences to be concatenated
into the network and applies multiple attention layers between all tokens of
both sentences, which has a significant computational overhead. For instance,
to find the most similar pair in a collection of 10,000 sentences BERT needs
roughly 50 million inference calculations, which takes around 65 hours
(Reimers and Gurevych, 2019).

Humeau et al. (2019) introduce poly-encoders to tackle the run-time overhead
of the BERT cross-encoder and present a method to calculate a score employing
attention between the context vectors and pre-computed candidate embeddings.
However, Poly-encoders have the disadvantage that their score function is not
symmetric and that their computing cost is excessively high and require
$O(n^{2})$ score calculations. Reimers and Gurevych (2019) therefore proposed
Sentence-BERT (SBERT) as a variant of the pre-trained BERT model that uses
siamese and triplet network architectures to produce semantically meaningful
embeddings that can be compared using cosine similarity. SBERT decreases the
time it takes to find the most similar pair from $65$ hours to around $5$
seconds while keeping BERT’s performance (Reimers and Gurevych, 2019). In this
work, we utilize SBERT as our embedding model, making the efficiency of RPRS
much higher than cross-encoder based re-rankers. We exploit a selection of
SBERT models that are tuned on different datasets.

###  2.3. Unsupervised sentence embeddings training

Previous works on unsupervised sentence embeddings learning have achieved
promising results on semantic textual similarity tasks by combining pre-
trained Transformers with various training objectives. Carlsson et al. (2020)
propose Contrastive Tension (CT) that views identical and different sentences
as positive and negative examples respectively and trains two independent
encoders. BERT-flow (Li et al., 2020) debiases the embedding distribution
towards Gaussian to train the model. SimCSE (Gao et al., 2021) uses
contrastive learning (Hadsell et al., 2006; Chen et al., 2020) to classify
identical sentences with different dropout masks as positive instances.

Wang et al. (2021a) propose an unsupervised state-of-the-art method called
TSDAE that is based on Transformers and denoising auto-encoders that encode
damaged sentences into fixed-sized vectors and require the decoder to
reconstruct the original sentences from these sentence embeddings (Wang et
al., 2021a). Wang et al. (2021b) propose Generative Pseudo Labeling (GPL),
which combines a query generator with pseudo labeling from a cross-encoder.
However, the GPL methodology is not suitable for QBD tasks as it relies on
cross-encoders that are limited both in length and efficiency. We evaluate the
effect of two state-of-the-art pre-training methods in section 6.

###  2.4. Sentence-level Retrieval

Addressing relevance of candidate documents by leveraging sentence-level
evidence has been studied for long documents retrieval in the past years
(Yilmaz et al., 2019a; Akkalyoncu Yilmaz et al., 2019; Yilmaz et al., 2019b).
However, there is no work on sentence-based relevance score computation using
Transformer-based models (i.e, SBERT (Reimers and Gurevych, 2019)) on QBD
retrieval tasks similar to our approach. Yilmaz et al. (2019a) apply inference
at the sentence level for each document and aggregate the sentence-level
inference by learning a weight for each top-scoring sentence in each candidate
document. Zhang et al. (2017) observe that the ‘best’ sentence or paragraph in
a document gives a decent proxy for document relevance, which was the
inspiration for Yilmaz et al. (2019a) as well. We also use this intuition in
our approach to QBD Retrieval, where both the query and the documents are long
texts. Mysore et al. (2021a) propose a scientific document similarity model
based on sentence-level similarity that leverages co-citation sentences as a
source of document similarity.

Recently, Ginzburg et al. (2021) proposed an unsupervised ranker called Self-
Supervised Document Similarity Ranking (SDR) that computes the final relevance
score by computing two sentence- and paragraph-level matrices. They evaluate
SDR’s effectiveness on two new datasets annotated by human experts. We
replicate SDR (Ginzburg et al., 2021) as the most recent and comparable
methodology to RPRS because: (1) Although SDR’s mechanism is dissimilar to
RPRS fundamentally, it also is a sentence-level relevance scoring model
designed for QBD tasks that covers the full length of both queries and
candidate document texts using sentence embeddings, and (2) Similar to RPRS,
SDR has a complexity of $O(N)$ where $N$ is count of sentences – due to the
utilization of bi-encoder sentence embeddings. Therefore, besides comparing
RPRS with the state-of-the-art model on each dataset, we compare its
effectiveness to SDR. SDR’s architecture makes it suitable for both full-
ranking and re-ranking settings.

Figure 1. The workflow of computing $R_{n}$ given a query document $q$ and set
of top-$k$ candidate documents. $i$ and $j$ refer to the last sentence of each
query and document.

##  3\. Proposed Method: RPRS

As mentioned, we assume that a candidate document $d$ is likely to be relevant
to a query document $q$ if a large proportion of $d$ is similar to $q$, and a
large proportion of $q$ is similar to $d$. In the following, we define our
concepts and proposed methodology.

###  3.1. Definitions

In Table 1 we show the legend of the symbols we employ in order to formally
introduce our methodology. In our sentence-based relevance model, we first
retrieve the set of documents $TK_{q}$ for $q$ with an initial ranker. Given
$q$, we compute the cosine similarity of each $\overrightarrow{q_{s}}$ with
each $\overrightarrow{d_{s}}$ , i.e. we compute the similarity of each query
sentence with each sentence in $S_{TK_{q}}$. The result $r(q_{s},S_{TK_{q}})$
is a ranked list of sentences from $S_{TK_{q}}$ for $q_{s}$. The set
$r_{n}(q_{s},S_{TK_{q}})$ contains the top-$n$ sentences from
$r(q_{s},S_{TK_{q}})$, i.e. the document sentences most similar to $q_{s}$.
From now on, we call a $d_{s}$ ‘most similar’ to $q_{s}$ if $d_{s}$ is a
member of $r_{n}(q_{s},S_{TK_{q}})$, meaning that it is among the top-n
sentences from the ranked list for sentences from the top-k candidate
documents. This indicates that the document sentence $d_{s}$ is placed among
the top-$n$ highest most similar document sentences to the query sentence
$q_{s}$.

The set $R_{n}(S_{q},S_{TK_{q}})$ contains all $r_{n}(q_{s},S_{TK_{q}})$ for
all query sentences. Figure 1 shows the workflow of computing $R_{n}$ step-by-
step. Based on our assumption, $d$ and $q$ are likely to be relevant if a
large proportion of $S_{d}$ occurs in a large proportion of each $r_{n}$s from
$R_{n}$.

Table 1. Meaning of symbols Symbol | Meaning
---|---
q and d | query and candidate document
$q_{s}$ and $d_{s}$ | a sentence of $q$ and $d$ respectively
$\overrightarrow{q_{s}}$ and $\overrightarrow{d_{s}}$ | vector representation (sentence embedding) of $q_{s}$ and $d_{s}$
$S_{d}$ and $S_{q}$ | the sets of all sentences of $d$ and $q$.
$TK_{q}$ | set of top-$k$ candidate documents retrieved for $q$
$S_{TK_{q}}$ | set of all sentences from $TK_{q}$
$r(q_{s},S_{TK_{q}})$ | a ranked list of $S_{TK_{q}}$ for $q_{s}$
$r_{n}(q_{s},S_{TK_{q}})$ | top-$n$ sentences from $r(q_{s},S_{TK_{q}})$
$R_{n}(S_{q},S_{TK_{q}})$ | contains all $r_{n}(q_{s},S_{TK_{q}})$ for all sentences of $S_{q}$

###  3.2. Re-ranker based on Proportional Relevance Score (RPRS)

We formally define the proportional relevance score for $RPRS$ as:

(1)
$\small\begin{array}[]{c}PRS(q,d,S_{TK_{q}},n)=QP(S_{q},S_{d},S_{TK_{q}},n)\times
DP(S_{d},S_{q},S_{TK_{q}},n)\end{array}$

Where $PRS$ is the Proportional Relevance Score (PRS), $QP$ is Query
Proportion, and $DP$ is Document Proportion. Given $q$, we compute $PRS$ for
$d$ based on the $R_{n}$ and $S_{TK_{q}}$. The parameter $n$ controls the
number of top-$n$ similar sentences per query sentence in $r_{n}$ and $R_{n}$.
In the following, we define two functions: $q_{R_{n}}(S_{q},S_{d},S_{TK_{q}})$
and $d_{R_{n}}(S_{d},S_{q},S_{TK_{q}})$ that we use in the later equations for
computing $QP$ and $DP$:

(2) $q_{R_{n}}(S_{q},S_{d},S_{TK_{q}})=\sum_{q_{s}}^{S_{q}}min(1,|S_{d}\cap
r_{n}(q_{s},S_{TK_{q}})|)$

Where $|x|$ denotes the cardinality of set $x$, and the $min$ function returns
1 if at least one of the sentences of $d$ ($S_{d}$’s sentences) is in
$r_{n}(q_{s},S_{TK_{q}})$, zero otherwise. Here, we do not take into account
the repetition of $S_{d}$’s sentences in each $r_{n}$ for a $q_{s}$ because of
the $min$ function. However, we incorporate that in a controllable way in the
extended variation ($RPRSw/freq$) in section 3.3 by defining parameter $k1$.
The function $q_{R_{n}}$ counts for how many sentences of $q$, at least one
sentence of the candidate document sentences ($S_{d}$) occurs at least one
time in the set of $q_{s}$’s $r_{n}$. Next, we define $d_{R_{n}}$ which is the
main component for computing $DP$ as:

(3)
$d_{R_{n}}(S_{d},S_{q},S_{TK_{q}})={{\sum_{d_{s}}^{S_{d}}}}min(1,\sum_{r_{n}}^{R_{n}}|\\{d_{s}\\}\cap
r_{n}|)$

Where $\\{d_{s}\\}$ denotes a singleton, i.e. a set with only one sentence of
$S_{d}$. For a query document $q$ and a candidate document $d$, the function
$d_{R_{n}}$ iterates over all sentences of $S_{d}$ and counts how many
sentences of $d$ occur at least one time in $R_{n}$. Given the parameter $n$,
it is possible that more than one sentence of $d$ occurs in $r_{n}$ for a
query sentence. We now define $QP$ and $DP$ based on equations (2) and (3):

(4)
$QP(S_{q},S_{d},S_{TK_{q}},n)=\frac{q_{R_{n}}(S_{q},S_{d},S_{TK_{q}})}{\text{count
of $q$'s sentences}}$ (5)
$DP(S_{d},S_{q},S_{TK_{q}},n)=\frac{d_{R_{n}}(S_{d},S_{q},S_{TK_{q}})}{\text{count
of $d$'s sentences}}$

It is noteworthy that the denominator of $QP$ (count of $q^{\prime}$s
sentences) has the same value for all candidate documents, and thus it could
be ignored for ranking; however, we keep it as it makes relevance scores
comparable for score analysis.

Intuitively, the relevance score of $d$ should increase by having more of its
sentences in $R_{n}$ for query $q$. The candidate document $d$ receives the
highest relevance score if all $r_{n}$s’ sentences in $R_{n}$ are from $d$. On
the other hand, $d$ receives the lowest relevance score if none of its
sentences are in $r_{n}$s’ sentences in $R_{n}$.

A specific challenge in QBD tasks is that a document can be very long (more
than $10$k words) and it may contain several topics. For a long query document
$q$, this can cause an irrelevant candidate document $d$ positioned on top
because only one topic of $d$ is very similar to one topic of $q$. Similarly,
a long irrelevant candidate document could be ranked on top because one of its
topics is very similar to only one topic of query document. We address this
problem in our method by integrating the length of the query and document into
$QP$ and $DP$.

###  3.3. Taking into account frequency (RPRS w/freq)

In the previous section, by using the minimum function in equations 2 and 3,
we only accounted for the occurrence of _at least one_ of the candidate
documents’ sentences in $r_{n}$ and $R_{n}$. In other words, we did not
consider the frequency of the occurrences in our definition for functions
$q_{R_{n}}$ and $d_{R_{n}}$. To empower our model, we define $Fq_{R_{n}}$ and
$Fd_{R_{n}}$ as the modified versions of $q_{R_{n}}$ and $d_{R_{n}}$ that take
into account the frequency:

(6) $Fq_{R_{n}}(S_{q},S_{d},S_{TK_{q}})=\sum_{q_{s}}^{S_{q}}\frac{|S_{d}\cap
r_{n}(q_{s},S_{TK_{q}})|}{|S_{d}\cap
r_{n}(q_{s},S_{TK_{q}})|+k1((1-b)+\frac{b\cdot dl}{avgdl})}$ (7)
$Fd_{R_{n}}(S_{d},S_{q},S_{TK_{q}})=\sum_{d_{s}}^{S_{d}}\frac{\sum_{r_{n}}^{R_{n}}|\\{d_{s}\\}\cap
r_{n}|}{\sum_{r_{n}}^{R_{n}}|\\{d_{s}\\}\cap r_{n}|+k1((1-b)+\frac{b\cdot
dl}{avgdl})}$

Here, inspired by the BM25 Okapi schema, we take the frequency into account in
a controllable way. To this aim, we introduce a modified version of RPRS,
which we denote by RPRS w/freq, which relies on two parameters: $k1$, the
frequency saturation parameter, for controlling the effect of frequency of
occurrence of a candidate document’s sentence in $R_{n}$, and $b$, the
document length normalization parameter, for controlling the effect of the
candidate document length. Similar to the BM25 formula, $dl$ refers to
document length and $avgdl$ refers to average length of documents. According
to the frequency saturation parameter ($k1$) of the proposed method, which
works similarly to BM25’s ‘term saturation’ mechanism, the occurrence of
multiple sentences of a candidate document in only one $r_{n}$ of $R_{n}$ has
a lower impact on the relevance score than the occurrence of multiple
sentences of a candidate document each of which occurs only once in several
different $r_{n}$s of $R_{n}$. We parameterize the degree of normalizing the
relevance according to the document length (i.e., number of sentences of a
document) with the parameter $b$, which works similarly to BM25’s length
normalization parameter (Robertson and Walker, 1994; Lipani et al., 2015). If
$b=0$, the relevance score is not normalized by the document length at all
because the right side of the denominator in equation 6 and 7 will be:
$k1((1-b)+\frac{b\cdot dl}{avgdl})=((1-0)+\frac{0\cdot
dl}{avgdl})=k1((1)+0)=k1$. Therefore, only $k1$ will be kept in the
denominator and the whole denominator will be $|S_{d}\cap
r_{n}(q_{s},S_{TK_{q}})|+k1$ and $\sum_{r_{n}}^{R_{n}}|\\{d_{s}\\}\cap
r_{n}|+k1$ for equation 6 and 7 respectively. As $b$ increases from $0$
towards $1$, the impact of document length – compared to the average length of
documents: $dl/adl$ – in normalizing the relevance score will be higher.
Consequently, $b=1$ means the relevance score will be fully normalized based
on the document length because the right side of the denominator in equation 6
and 7 will be: $k1((1-b)+\frac{b\cdot dl}{avgdl})=((1-1)+\frac{1\cdot
dl}{avgdl})=k1((0)+\frac{1\cdot dl}{avgdl})=k1\cdot\frac{dl}{avgdl}$.
Therefore, the document length will be divided by the average length of
documents and will be multiplied to $k1$. As a result, the denominator will be
fully normalized based on the ratio of document length to the average length
of all documents in the denominator, and the whole denominator will be
$|S_{d}\cap r_{n}(q_{s},S_{TK_{q}})|+k1\cdot\frac{dl}{avgdl}$ and
$\sum_{r_{n}}^{R_{n}}|\\{d_{s}\\}\cap r_{n}|+k1\cdot\frac{dl}{avgdl}$ for
equations 6 and 7 respectively.

In summary, the final proposed method has three parameters: $k1$ and $b$ as
described above, and $n$ that controls the number of top-$n$ sentences in
$r_{n}$ per query sentence. All parameters can be tuned on the training set.
Furthermore, the parameters can be used with default values or with values
that are obtained by tuning the method on another dataset. With _RPRS_ and
_RPRS w/freq_ , we only need to compute the cosine similarity between
embeddings of query sentences and the document sentences. Our proposed method
is efficient because all documents can be pre-processed, embedded and indexed
before query time. At query time, producing the embedding for query sentences
using SentenceBERT is highly efficient (Reimers and Gurevych, 2019). This
gives either _RPRS_ or _RPRS w/freq_ the advantage of having a complexity of
$O(N)$ where $N$ is the total number of sentences in the query and candidate
documents compared to re-rankers based on Cross-encoders with $O(N^{2})$.
Furthermore, calculating cosine similarity is a simple operation, thus
resulting in a fast inference time.

Figure 2. An example for the intuition of $RPRS\,w/freq$ according to a query
$(q)$ and two candidate documents $(d1,d2)$. A sentence of the query and a
document having the same colour means they are most similar to each other.
Both candidate documents have the same length and for each of them all of
their sentences are at least most similar to one sentence of query. In this
example, $d_{2}$ is relevant to more sentences of the query document than
$d_{1}$. Hence, intuitively, $d_{2}$ should be ranked higher.

###  3.4. An example of relevance computation by _RPRS w/freq_

Figure 2 shows an example for explaining the intuition of the proposed method
and impact of saturating frequency in a real example. While both candidate
documents are equal in length, the main difference between them is for $d2$
each sentence is most similar to a different sentence of query while for
$d_{1}$ all of its sentences are most similar to only one sentence of query.
As a result, according to the saturation mechanism that has been explained
above, we expect $RPRSw/freq$ to score $d2$ higher than $d_{1}$. In order to
compute the relevance score for $RPRSw/freq$, we first start with computing
$Fq_{R_{n}}$ (equation 6) for candidate document $d_{1}$ according to the
query. For simplicity, $b$ and $avgdl$ are excluded from the computation. In
other words, $b$ is set to $0$. The value of the frequency saturation
parameter parameter $k1$ is set to $2$ in this example. For $d_{1}$:

$Fq_{R_{n}}=\frac{5}{5+2}+\frac{0}{0+2}+\frac{0}{0+2}+\frac{0}{0+2}+\frac{0}{0+2}+\frac{0}{0+2}\approx
0.71$

Where each fraction represents the relevance score between one sentence of the
query and all sentences of $d_{1}$, i.e., the first fraction refers to the
first sentence of $q$ and last fraction refers to the last sentence of $q$. In
the fist fraction ($\frac{5}{5+2}$), the value of $k1$ is $2$, and ‘$5$’ in
the numerator and denominator is the result of computing equation 6 for the
first sentence of the query. In other words, it is the answer to: “Sentence 1
of $q$ is most similar to how many sentences of $d_{1}$?”, which is 5
sentences. Since all of the sentences of $d_{1}$ are only most similar to the
first sentence of the query, and the remaining sentences of the query
sentences are most similar to other documents, there is a zero in the
numerator and denominator of the rest fractions. The number of fractions is
six, which refers to the number of $q$ sentences. Next, we compute
$Fd_{R_{n}}$ (equation 7) between $d_{1}$ and $q$:

$Fd_{R_{n}}=\frac{1}{1+2}+\frac{1}{1+2}+\frac{1}{1+2}+\frac{1}{1+2}+\frac{1}{1+2}\approx
1.67$

Where the fractions are represented in the same order of $d_{1}$’s sentences.
The ‘$1$’ in the numerator and denominator of first fraction ($\frac{1}{1+2}$)
is the answer to: “Sentence $1$ of $d_{1}$ is most similar to how many
sentences of $q$?”, which is 1 sentence. The count of fractions is five, which
refers to the count of $d_{1}$ sentences and $2$ is the value of $k1$
parameter. Having $Fq_{R_{n}}$ and $Fd_{R_{n}}$ computed between $q$ and
$d_{1}$, now we compute $QP$ (equation 4) and $DP$ (equation 5):

$QP=\frac{Fq_{R_{n}}}{|q|}=\frac{0.71}{6}\approx 0.12$
$DP=\frac{Fd_{R_{n}}}{|d|}=\frac{1.67}{5}\approx 0.33$

Where $|q|$ and $|d|$ refer to count of sentences of $q$ and $d_{1}$. Finally,
we compute $RPRS\,w/freq$ for $d_{1}$ according to q:

$RPRS\,w/freq=QP\times DP=0.12\times 0.33\approx 0.04$

Therefore, the final relevance score for $d_{1}$ is $0.04$. Next, similarly we
compute $RPRS\,w/freq$ for $d2$ according to the $q$:

$Fq_{R_{n}}=\frac{1}{1+2}+\frac{0}{0+2}+\frac{1}{1+2}+\frac{1}{1+2}+\frac{1}{1+2}+\frac{1}{1+2}\approx
1.67$

$Fd_{R_{n}}=\frac{1}{1+2}+\frac{1}{1+2}+\frac{1}{1+2}+\frac{1}{1+2}+\frac{1}{1+2}\approx
1.67$

$QP=\frac{Fq_{R_{n}}}{|q|}=\frac{1.67}{6}\approx 0.28$

$DP=\frac{Fd_{R_{n}}}{|d|}=\frac{1.67}{5}\approx 0.33$

$RPRS\,w/freq=QP\times DP=0.28\times 0.33=0.09$

As expected, the final relevance score for $d2$ is ($0.09$) higher than
$d_{1}$ ($0.04$). This shows the saturation mechanism aligns well with the
intuition _RPRS w/freq_ is based on.

##  4\. Baseline retrieval models

In this section we introduce BM25 and SDR that are the two main baselines in
our experiments.

###  4.1. BM25

Lexical retrievers estimate the relevance of a document to a query based on
word overlap (Robertson et al., 2009). Many lexical methods, including vector
space models, Okapi BM25, and query likelihood, have been developed in
previous decades. We use BM25 because of its popularity as first-stage ranker
in current systems and its strong effectiveness on QBD tasks (Rosa et al.,
2021). Based on the statistics of the words that overlap between the query and
the document, BM25 calculates a score for the pair:

(8) $s_{lex}(q,d)=BM25(q,d)=\sum_{t\in q\cap
d}{rsj_{t}.\frac{tf_{t,d}}{tf_{t,d}+k_{1}\\{(1-b)+b\frac{|d|}{l}\\}}}$

where $t$ is a term, $tf_{t,d}$ is the frequency of $t$ in document $d$,
$rsj_{t}$ is the Robertson-Spärck Jones weight (Robertson and Walker, 1994) of
$t$, and $l$ is the average document length. $k_{1}$ and $b$ are parameters.

###  4.2. SDR (Ginzburg et al., 2021)

We replicate SDR (Ginzburg et al., 2021) as the most recent and comparable
methodology to RPRS because: (1) Although SDR’s mechanism is dissimilar to
RPRS fundamentally, it also is a sentence-level relevance scoring model
designed for QBD tasks that cover the full length of both queries and
candidate document texts using sentence embeddings, and (2) Similar to RPRS,
SDR has a complexity of $O(N)$ where $N$ is count of sentences – due to the
utilization of bi-encoder sentence embeddings instead of cross-encoders – for
computing relevance scores in contrast to other techniques with $O(N^{2})$
(Devlin et al., 2018; Barkan et al., 2020). Therefore, besides comparing RPRS
with the state-of-the-art model on each dataset, we compare its effectiveness
with SDR. SDR’s architecture makes it suitable for both full-ranking and re-
ranking settings; as a result, we evaluate it in both configurations and only
report the result of the setup (ranker or re-ranker) in which it performs
best. In the following, we introduce SDR’s pre-training and inference
methodology.

####  4.2.1. Self-supervised pre-training

Given a collection of documents $D$, SDR samples sentence pairs from the same
paragraph of a given document (intra-samples) and sentence pairs from
different paragraphs taken from different documents (inter samples) with equal
probability (i.e., $0.5$ for each type of sampling). It tokenizes sentences
and aggregates them into batches, and it randomly masks them in a similar way
to the RoBERTa pre-training paradigm. The authors use the Roberta model for
their implementation (Liu et al., 2019). The pretraining objective of SDR
comprises a dual-term loss: (1) a standard MLM loss adopted from (Devlin et
al., 2018) that allows the model to specialize in the domain of the given
collection of documents (Gururangan et al., 2020), and (2) a contrastive loss
(Hadsell et al., 2006) aims to minimize the distance between the
representations of sentence-pairs from the same paragraph (intra-samples),
while maximizing the distance between the representations of sentence-pairs
from different paragraphs (inter-samples).

####  4.2.2. Inference

SDR computes the relevance score between a query $q$ and a candidate document
$d$ by computing a two-stage hierarchical similarity score: (1) SDR creates a
sentence similarity matrix $M$ for all possible pairs of query paragraphs and
candidate document paragraphs. Each cell in $M$ represents the cosine
similarity between a sentence from paragraph $i$ of the query and a sentence
from paragraph $j$ of the candidate document. Specifically, the rows of $M$
correspond to sentences from the query paragraph, and the columns correspond
to sentences from the candidate document paragraph. Therefore, a cell
$M_{a,b}^{i,j}$ represents the cosine similarity between sentence $a$ from
paragraph $i$ of the query and sentence $b$ from paragraph $j$ of the
candidate document; (2) Next, a paragraph similarity matrix $P$ is created for
the candidate document $d$ based on all pairs of paragraphs (query paragraph
vs document paragraph). Each cell in $P$ represents the similarity between a
paragraph from the query and a paragraph from the candidate document and is
computed using the maximum cosine similarity between sentences in the two
paragraphs one from the query and one from the candidate document. The
motivation for the creation of matrix $P$ is that similar paragraph pairs
should incorporate similar sentences that are more likely to correlate under
the cosine metric.

Finally, $P$ is normalized globally ($NRM(P)$), based on other candidate
documents’ paragraph similarity matrices. Based on the $NRM(P)$ generated for
a candidate document, the total similarity score $S$ is obtained by computing
an average of the highest cosine similarity scores in each row of $P$. The
motivation for $S$ is that the most correlated paragraph pair only contributes
to the total similarity score. In contrast to RPRS, SDR does not take into
account the length of candidate document because the denominator in its
formula for computing matrix $P$ is count of sentences in a query’s paragraph
and for total score $S$ is the count of paragraphs in the query. However, this
issue has been handled in our re-ranker by integrating the length of the query
and document into $QP$ and $DP$ (See section 3.2). Moreover, if one paragraph
of a candidate document be the most similar paragraph to all paragraph of
query, SDR does not penalize this repetition while we take that into account
with $k1$ parameter in _RPRS w/freq_ that controls frequency saturation (See
section 3.3). In addition, the cosine score is used directly into the
approach, while we only consider that score for ranking sentences in $R_{n}$
(See section 3.2 and Figure 1).

##  5\. Experiments

In this section, we first describe the datasets, the pre-trained Transformers
that we use, and our implementation details for the retrieval models. Finally,
we provide information about parameter tuning and pre-processing of the
dataset.

###  5.1. Datasets

We evaluate our models on three QBD retrieval tasks: legal case retrieval,
patent prior art retrieval, and document similarity ranking for Wikipedia
pages.

##### COLIEE’21.

We first use the COLIEE’21 dataset to evaluate the effectiveness of the
proposed method (PRS) on legal case retrieval. There are $650$ query documents
in the train set and $250$ in the test set (Rabelo et al., 2022), with 4,415
documents as candidate documents in both sets. The candidate documents’
average length is $5,226$ (tokenized by SparkNLP (Kocaman and Talby, 2021),
see below), with outliers reaching $80,322$ words.

##### Caselaw.

To investigate the generalizability of our model, _RPRS w/freq_ , we tuned it
on the COLIEE’21 dataset and evaluate the tuned model on a different dataset:
the Caselaw dataset (Locke et al., 2017). Therefore, we use the Caselaw
dataset as a test set to assess how well the tuned _RPRS w/freq_ model can
generalize to new and unseen data. The Caselaw dataset contains $100$ query
documents, $2,645$ relevance assessments, and $63,916$ candidate documents. In
Caselaw, the candidate documents’ average length is $2,344$, with outliers up
to $124,092$ words.

##### CLEF-IP 2011.

For patent prior art retrieval (Piroi et al., 2011), we experiment on the
CLEF-IP 2011 dataset that contains $300$ and $3,973$ query documents in the
train set and test set respectively, both of which have document-level
relevance assessments. The count of candidate documents per query is about $3$
Million. In this work, we select the English subset of CLEF-IP 2011 that
contains about $900,000$ candidate documents per query, and $100$ and $1,324$
query documents in the train set and test set respectively. We concatenate
title, abstract, description, and claims as the whole patent document. The
candidate documents’ average length in CLEF-IP is $10,001$ words, with
outliers up to $407,308$ words.

##### Wikipedia.

For the Wikipedia datasets, we use the Wikipedia video games (WVG) and the
Wikipedia wine articles (WWA) datasets (Ginzburg et al., 2021) that contain
$21,935$ and $1,635$ candidate documents and $90$ and $92$ query documents
respectively. The WWG collection consists of articles reviewing video games
from all genres and consoles and the WWA collection consists of a mixture of
articles discussing different types of wine categories, brands, wineries,
grape varieties, and more. The documents’ average length is $1061$ and $966$,
with outliers up to $23,048$ and $13,081$ words for the WVG and WWA datasets
respectively.

###  5.2. Baselines

BM25 has previously been shown to be a strong baseline for QBD retrieval (Rosa
et al., 2021), and it holds the state-of-the-art among all lexical models on
COLIEE (Askari and Verberne, 2021). We implement BM25 as the initial ranker
using Elasticsearch on all five datasets. In addition to BM25 on the complete
query document text, we employ BM25 on the top-$10$ percent of query document
terms that are extracted and scored using Kullback-Leibler divergence for
Informativeness (KLI) (Verberne et al., 2016) following prior work (Locke et
al., 2017; Askari and Verberne, 2021). As KLI has been shown to be an
effective approach for making shorter queries for BM25 (Locke et al., 2017),
we employ KLI on documents of all three datasets and refer to that in the
result tables as ‘BM25 + KLI’.

Moreover, we replicate the SDR (Ginzburg et al., 2021) ranker as $SDR_{inf}$
where ‘inf’ refers to inference. $SDR_{inf}$ is a comparable sentence-level
baseline to _RPRS w/freq_. It should be noted that wherever SDR is mentioned
in the tables, it is referring to the best-performing SDR variant, either as a
re-ranker after the best first-stage ranker, or as a full ranker. We
experiment with Roberta besides the other SBERT’s models as SDR’s authors use
the Roberta model for their implementation (Liu et al., 2019).

For COLIEE 2021, we compare the proposed method to the lexical state-of-the-
art model, which is $BM25_{optimised}$ with optimized parameters ($b=1,k=2.8$)
(Askari and Verberne, 2021). Additionally, we do comparisons with two re-
rankers: the BERT re-ranker (Nogueira and Cho, 2019), and the multi-task fine-
tuned BERT re-ranker (MTFT-BERT), which is the neural state-of-the-art on
COLIEE (Abolghasemi et al., 2022b). For Caselaw (Locke et al., 2017), we re-
use the lexical and neural models that we fine-tuned on COLIEE to analyze the
generalizability of them and the proposed method. The state-of-the-art method
for Caselaw is _K_ (Locke et al., 2017).333We do not report result with
manually created boolean queries for fair comparison. We found that _K_ ’s run
file is the most effective initial ranker and thus we used that as our initial
ranker on Caselaw (Locke et al., 2017).

For patent retrieval, there is no Transformer-based method baseline, which
could be due to the fact that average length of documents in the patent
dataset is around $10,001$ and thus cannot be handled by Transformer models
straightforwardly. Therefore, we compare our result with the best two methods
for the English language in the CLEF IP 2011 competition based on Figure 3
from Piroi et al. (2011): Ch.2 and Hy.5. We found that Ch.2 is the most
effective method on CLEF-IP 2011, and thus we used that as our initial ranker.

For the Wikipedia datasets, we compare our result with $SDR_{inf}$ which
proposed these datasets recently and is the state-of-the-art model for them.

###  5.3. Pre-trained Sentence BERT models

Reimers and Gurevych (2019) have published a set of SBERT embedding models
that they trained and evaluated extensively with respect to their
effectiveness for semantic textual similarity (Sentence Embeddings) and
Semantic Search tasks on 14 and 5 different datasets
respectively.444https://www.sbert.net/docs/pretrained_models.html We exploit
the following top-$4$ ranked embedding models555Please note that these models
are top-ranked at the time of submission and the SBERT list can be updated
later. that are different in terms of training data or architecture:

  * •

all-mpnet-base-v2 666https://huggingface.co/sentence-transformers/all-mpnet-
base-v2, all-distilroberta-v1 777https://huggingface.co/sentence-
transformers/all-distilroberta-v1, and all-MiniLM-L12-v2
888https://huggingface.co/sentence-transformers/all-MiniLM-L12-v2 are SBERT
models trained based on _mpnet-base_ (Song et al., 2020), _distilbert-base-
cased_ (Sanh et al., 2019) and _MiniLM-L12-H384-uncased_ (Wang et al., 2020)
models respectively on more than $1$ billion sentence pairs as general purpose
models for sentence similarity;

  * •

multi-qa-mpnet-base-dot-v1 999https://huggingface.co/sentence-
transformers/multi-qa-mpnet-base-dot-v1 is a _mpnet-base_ model (Song et al.,
2020) trained on $215M$ question-answer pairs from various sources and
domains, including StackExchange, Yahoo Answers, Google and Bing search
queries and many more as a model for question answering and IR tasks.

Besides the above mentioned SBERT models, we utilize the following domain
specific and general well-known variants of BERT and Roberta models in the
architecture of SBERT to investigate their effectiveness, considering that
they have the efficiency of the SBERT’s architecture:

  * •

BERT base uncased, BERT large uncased, Roberta base uncased, and Roberta large
uncased are pre-trained on a large corpus of English raw data (Devlin et al.,
2018; Liu et al., 2019);

  * •

Legal BERT base uncased (Chalkidis et al., 2020) is a light-weight model of
BERT-BASE (33% the size of BERT-BASE) pre-trained from scratch on $12$ GB of
diverse English legal text of several types (e.g., legislation, court cases,
contracts);

  * •

Patent BERT (Srebrovic and Yonamine, 2020) is trained by Google on $100M+$
patents (not just US patents) including abstract, claims, description based on
BERT large uncased architecture.

We only report the results of the trained SBERT models as they obtained higher
than the general well-known variants of BERT models in the architecture of
SBERT in all datasets except for the Wikipedia datasets (WWG and WWA). For
these, we report the Roberta large model in the architecture of SBERT, which
achieved the best result.

###  5.4. Implementation details

For SBERT, we first experiment with all the models that are introduced in
section 5.3 and then adapting the two best models for each dataset on the
domain using the methods proposed by Wang et al. (2021a) (TSDAE) and Ginzburg
et al. (2021) (SDR) for domain adaptation without labeled data.101010The
resulting models on the patent, legal and Wikipedia domain will be publicly
available. The SDR pre-training method is a dual-term loss objective that is
composed of a standard MLM loss adopted from (Devlin et al., 2018) – which
allows the model to specialize in the domain of the given collection
(Gururangan et al., 2020) – and a contrastive loss (Hadsell et al., 2006). For
SDR pre-training, we tokenize sentences, aggregate them into batches, and
randomly mask them in a similar way to the RoBERTa pre-training paradigm. For
TSDAE pre-training, as suggested by (Wang et al., 2021a), we use $10,000$
sentences for each domain which is only $2\%$ of the COLIEE’21, $1\%$ of the
Wikipedia (WWG and WWA) datasets, and less than $0.1\%$ of the CLEF-IP 2011
dataset. PyTorch (Paszke et al., 2019), HuggingFace (Wolf et al., 2019), and
Sentence BERT (Reimers and Gurevych, 2019) are used to implement all of our
models.

We use $BM25_{optimised}$ (Askari and Verberne, 2021) as our initial ranker
for COLIEE, _K_ for Caselaw (Locke et al., 2017), and _Ch.2_ for CLEF-IP 2011
(Piroi et al., 2011). To determine the optimal re-ranking depth, we increase
the depth of the initial rank result on the validation set from $15$ to $100$
in steps of $5$. We found $50$ to be optimal for COLIEE, $100$ for WWG and
WWA, and $20$ for the CLEF-IP 2011 dataset. As we use Caselaw as test set
only, we employ COLIEE’s re-ranking depth for Caselaw.

###  5.5. Parameter tuning

For tuning parameters we experiment with values from $1$ to $10$ for parameter
$n$: $n=[1,2,3,4,5,6,7,8,9,10]$. For the $b$ and $k1$ parameters of _RPRS
w/freq_ , we used the same range as when tuning them for BM25
($b=[0.0,0.1,0.2,\cdots,1],k1=[0.0,0.2,0.4,\cdots,3.0]$). We found
$(b=1,k1=2.8,n=4)$ for COLIEE’21, $(b=0.9,k1=3.0,n=4)$ for WWG and WWA, and
$(b=0.8,k1=2.4,n=5)$ for CLEF-IP.

###  5.6. Pre-processing

We investigated sentence segmentation using three libraries since legal case
retrieval and patent retrieval are challenging domain-specific tasks with long
legal sentences (Sanchez, 2019; Verberne et al., 2010). We empirically found
that SparkNLP (Kocaman and Talby, 2021) is segmenting the legal text into
sentences with a higher quality than NLTK (Bird et al., 2009) and Stanza (Qi
et al., 2020). We split long sentences into sequences of $25$ words for COLIEE
and $30$ words for CLEF-IP (average length of sentences in both collections).
We then detected French sentences in the COLIEE data using the method by
(Danilak, 2014) and translated them to English using the Google Translate API.

##  6\. Results

We employ a paired t-test between the proposed method, i.e., _RPRS w/freq_ ,
and the state-of-the-art model for each dataset. In the following, we first
address the choice of SBERT model for the proposed method, _RPRS w/freq_.
Next, we analyze the effectiveness of the proposed method on the five domain-
specific QBD datasets and discuss the results per domain. In summary, we found
that _RPRS w/freq_ outperforms the state-of-the-art models significantly for
all official metrics in all five datasets.

Table 2. Results of the Proposed Method, _RPRS w/freq_ , on COLIEE’21 using different Transformer Models Embedding. Top SBERT models are top-$4$ trained and extensively evaluated models by SBERT and are publicly available. Legal SBERT is the Legal BERT (Chalkidis et al., 2020) that is loaded into the SBERT (Reimers and Gurevych, 2019) architecture. Domain adaption has been done using Self-Supervised, SDR, and Unsupervised, TSDAE, methods on all-MiniLM-L12-v2 and Legal SBERT models as the models show top-two highest effectiveness without Domain Adaption. | Model Name | Precision | Recall | F1
---|---|---|---|---
Initial Ranker |  |  |
a | BM25optimized \+ KLI | 0.1700 | 0.2536 | 0.2035
Top SBERT models |  |  |
b | all-mpnet-base-v2 | 0.1760 | 0.2478 | 0.2058
c | multi-qa-mpnet-base-dot-v1 | 0.1840 | 0.2625 | 0.2163
d | all-distilroberta-v1 | 0.1900 | 0.2632 | 0.2206
e | all-MiniLM-L12-v2 | 0.1920 | 0.2745 | 0.2259
Domain-Specific |  |  |
f | Legal SBERT | 0.1880 | 0.2720 | 0.2223
Domain Adaptation |  |  |
g | all-MiniLM-L12-v2-SDR | 0.1817 | 0.2594 | 0.2137
h | all-MiniLM-L12-v2-TSDAE | 0.1840 | 0.2584 | 0.2149
i | Legal SBERT-SDR | 0.1890 | 0.2762 | 0.2244
j | Legal SBERT-TSDAE | 0.1960 | 0.2891 | 0.2336

###  6.1. Choice of SBERT model for _RPRS w/freq_

In order to address RQ1, we first run a set of experiments with different
SBERT models to find with which SBERT model the _RPRS w/freq_ achieve higher
effectiveness. Table 2 shows that our re-ranker achieves a higher
effectiveness than the strong initial ranker (line $a$) (Askari and Verberne,
2021) using any sentence embedding models (line $b-j$). This shows the
effectiveness of our re-ranker is more dependent on the proposed method rather
than sentence embedding models, while our re-ranker effectiveness could be
improved with a sentence embedding model that captures the legal context more
accurate. Morover, the table shows that out of the four SBERT models (line
$b-e$), _all-MininLM-L12-v2_ , achieves highest effectiveness. Interestingly,
this is the SBERT model with the lowest number of parameters ($33M$, compared
to $82M$, $110M$, and $110M$ for the other three). This could indicate that
between SBERT models which have shown their effectiveness in other (general)
domains, the chance of generalizing to a new domain-specific dataset for a
SBERT model that has lowest parameters might be higher than others. This
finding is in line with the fact that, in the context of traditional bias-
variance trade-offs, neural models with fewer parameters are more likely to
generalize to the other domains (Zhong et al., 2021; Zhou et al., 2021). The
higher effectiveness for _multi-qa-mpnet-base-dot-v1_ (line $c$) compared to
_all-mpnet-base-v2_ (line $b$) while they both have same number of parameters
and same base architecture could be due to the fact that _multi-qa-mpnet-base-
dot-v1_ was trained on (question, pair) instances, and as a result, it is
suitable for semantic search task and aligns with our re-ranking task while
all-mpnet-base-v2 is trained on various tasks.

Table 3. Results for COLIEE’21.$\dagger$ and * indicate a statistically significant improvement over State-of-the-art (MTFT-BERT, line g) and BM25optimized (line c) respectively, according to a paired-test (p¡0.005) with Bonferroni correction for multiple testing. The winning team in the COLIEE 2021 competition for legal case retrieval is TLIR (Ma et al., 2021). The initial ranker for all re-rankers (BERT, MTFT (Askari and Verberne, 2021), SDR, and RPRS ) is BM25optimized+KLI. Legal SBERT-TSDAE is used to embed sentences for $SDR_{inf}$, RPRS, and _RPRS w/freq_. |  | Model | Precision | Recall | F1
---|---|---|---|---|---
|  Probabilistic lexical matching baselines |  |  |
| a | BM25 | 0.0770 | 0.1959 | 0.1113
| b | BM25 + KLI | 0.0983 | 0.1980 | 0.1313
| c | BM25optimized + KLI | 0.1700 | 0.2536 | 0.2035
| d | TLIR | 0.1533 | 0.2556 | 0.1917
|  Cross-encoders |  |  |
| e | BERT | 0.1340 | 0.2263 | 0.1683
| f | Legal BERT | 0.1440 | 0.2463 | 0.1817
| g | MTFT-BERT (Abolghasemi et al., 2022b) (Previous state-of-the-art) | 0.1744 | 0.2999 | 0.2205
|  Sentence-based baseline |  |  |
| h | $SDR_{inf}$ | .1470 | 0.2063 | 0.1716
|  Proposed methods |  |  |
| i | RPRS without frequency | 0.1890 | 0.2799 | 0.2256
| j | RPRS w/freq | 0.1960$\dagger$* | 0.2891* | 0.2336$\dagger$*

The second highest effectiveness is achieved by _Legal SBERT_ , which is the
domain-specific Legal BERT model (Chalkidis et al., 2020) that is loaded into
the SBERT architecture (Reimers and Gurevych, 2019) and has $36M$ parameters
(more than MiniLM and less than our three other SBERT models). The
effectiveness of our proposed method with the usage of Legal SBERT model could
be due the fact it is pre-trained on the legal documents. However, with domain
adaptation on COLIEE corpus using TSDAE method (Wang et al., 2021a), i.e.,
Legal SBERT-TSDAE, _RPRS w/freq_ achieves the highest effectiveness (row $j$).
We found the same pattern for SDR and RPRS without frequency (section 3.2),
i.e., they achieve highest effectiveness with Legal-BERT-TSDAE model
embeddings, but we only report it for _RPRS w/freq_ which is our complete
proposed method.

Furthermore, an observation by comparing rows ($g,i$) and ($h,j$) of Table 2,
is that the domain adaption with TSDA results in higher effectiveness of our
re-ranker compared to SDR domain adaption method which is a contrastive-
learning domain adaption method. We suppose the higher effectiveness of our
re-ranker with domain adaption by TSDAE compared to SDR could be due to the
fact that TSDAE removes words from the input randomly and as the result forces
the network to produce robust embeddings. In contrast, the input sentences for
a contrastive-learning approach such as SDR are not modified, resulting in
less stable embeddings. This finding was confirmed previously by Wang et al.
(Wang et al., 2021a).

Additionally, by comparing rows $e$ with $g$ and $h$, we can see that domain
adaption on all-MiniLM-L12-v2 results in drops in the effectiveness of our re-
ranker. This observation is in line with the finding in prior work (Wang et
al., 2021a) which shows that domain adaption with TSDAE after supervised
training results in a drop in the effectiveness. It is noteworthy to mention
that all-MiniLM-L12-v2 is trained on 1 billion samples in a supervised manner.
A better strategy in this case, would be first doing domain adaption as a pre-
training step with TSDAE and then doing supervised training with 1 billion
samples as was suggested by the authors of TSDAE (Wang et al., 2021a).
111111Also reported on https://github.com/UKPLab/sentence-
transformers/issues/1372#issuecomment-1028525300 (visited February 27, 2023)
However, re-training MiniLM with 1 billion examples was not possible for us
due to computational limitations.

Therefore, based on our empirical findings, for the next experiments on each
domain, we pick the domain-specific Transformer model and then adapt it into
the the target domain, e.g., legal, using TSDAE. We argue that doing this
domain adaption is realistic for a real-world application because it is
trained on the document collection that are provided for training, and does
not need or use any supervision from labelled data.

###  6.2. Effectiveness on the COLIEE dataset (RQ1)

Taking into account the finding from previous section, we exploit Legal SBERT-
TSDAE in Table 3 to embed sentences for _RPRS_ , _RPRS w/freq_ , and SDR. Row
$i$ of Table 3 show that even without taking into account frequency, _RPRS_
achieves higher effectiveness than the initial ranker ($BM25_{optimized}+KLI$,
line $c$) and the state-of-the-art re-ranker (MTFT-BERT, line $g$) on
precision and the official metric ($F1$). This is while _RPRS_ has only one
parameter, $n$. Row $j$ of the table shows that $RPRSw/freq$ achieve highest
effectiveness and significantly better results over the state-of-the-art
(MTFT-BERT(Abolghasemi et al., 2022b), line $g$) on precision and the official
metric ($F1$) which indicates the importance and role of adding the document
length normalization parameter $b$, and frequency saturation parameter $k1$ in
increasing the effectiveness of the proposed method. We emphasize that the
official metric for COLIEE is $F1$ and the COLIEE organizers only report $F1$
in their official reports for case law retrieval task (Rabelo et al., 2022).
We assume achieving lower recall by our re-ranker, _RPRS w/freq_ (line $j$),
compared to MTFT-BERT (Abolghasemi et al., 2022b) (line $g$) could be due to
the inverse relationship between recall and precision which is a common
observation based on experimental evidence (Buckland and Gey, 1994). Cleverdon
(1972) provides an in-depth explanation of why precision and recall have often
an inverse relationshi. $SDR_{inf}$ (line $h$) achieves lower performance than
$RPRSw/freq$ and $RPRS$ results (line $i-j$).

Figure 3. Effectiveness of RPRS w/freq using fixed-length units instead of
sentences on COLIEE 2021 (segment size in tokens).

###  6.3. Effect of using other units than sentences (RQ2)

We investigate the effect of using embeddings of different textual units than
sentences in _RPRS w/freq_ to analyze if “sentence” is the most appropriate
unit for our re-ranker. To do so, we split the documents in segments of length
$l$ with $l\in\\{16,32,64,128,256,512\\}$. Figure 3 shows that sentences are
better units than sequences with pre-defined fixed-length, also compared to
sequences with fixed-length $16$ and $32$ which are similar in length to the
average and median sentence length ($29.4$ and $24.8$ tokens). This confirms
the relevance of sentences as units for retrieval, one of the premises of
RPRS.

Table 4. Results for CaseLaw. $\dagger$ indicate a statistically significant improvement over the state-of-the-art (K (Locke et al., 2017), line c) according to a paired-test ($p<0.05$) with Bonferroni correction for multiple testing. We pick the optimized values of the proposed method parameters from COLIEE dataset, and use them to analyze the proposed method cross-data generalizability on Caselaw dataset. Legal SBERT-TSDAE is used to embed sentences for $SDR_{inf}$, RPRS, and _RPRS w/freq_. | Model | P@1 | R@1 | MAP@5 | NDCG@5 | MRR
---|---|---|---|---|---|---
Probabilistic lexical matching baselines |  |  |  |  |
a | BM25 | 0.500 | 0.025 | 0.227 | 0.358 | 0.618
b | BM25 + KLI | 0.660 | 0.115 | 0.232 | 0.398 | 0.726
c | K (Locke et al., 2017) (Previous State-of-the-art) | 0.730 | 0.119 | 0.307 | 0.473 | 0.803
Cross-encoders |  |  |  |  |
d | BERT | 0.320 | 0.064 | 0.118 | 0.201 | 0.438
e | Legal BERT | 0.330 | 0.065 | 0.128 | 0.217 | 0.449
f | MTFT-BERT | 0.360 | 0.064 | 0.160 | 0.252 | 0.481
Sentence-based baseline |  |  |  |  |
g | $SDR_{inf}$ | 0.500 | 0.070 | 0.171 | 0.320 | 0.585
Proposed methods |  |  |  |  |
h | RPRS without frequency | 0.730 | 0.128 | 0.314 | 0.489 | 0.807
i | RPRS w/freq | 0.780$\dagger$ | 0.138$\dagger$ | 0.321$\dagger$ | 0.496$\dagger$ | 0.837$\dagger$

###  6.4. Effectiveness without parameter optimization on target dataset(RQ3)

To address this question, we evaluate the proposed method on the _Caselaw_
dataset (Locke et al., 2017) with the parameters that were tuned on the COLIEE
dataset without doing domain adaption on the collection for the Transformer
model. In other words, our goal is to analyze how much the three optimized
parameters of RPRS on the COLIEE dataset are transferable to another dataset
in the same (i.e., legal) domain. Table 4 shows that the effectiveness of our
re-ranker is higher than all models including the state-of-the-art model (_K_
, line $c$) for all evaluation metrics with (line $h$) or without (line $i$)
taking into account frequency. This supports the cross-data generalizability
of our re-ranker, i.e., optimizied parameters of our proposed method work
effectively for another dataset in the same domain, as we re-use the tuned
$RPRSw/freq$ on COLIEE for the Caselaw dataset without optimizing its
parameters on the Caselaw dataset. $SDR_{inf}$ (line $g$) achieves lower
performance than $RPRSw/freq$ and $RPRS$ results (line $h-i$).

###  6.5. Generalizability of porposed method (RQ4)

Patent domain (CLEF-IP 2011 dataset). Table 5 shows that our re-ranker
outperform the state-of-the-art model on the CLEF-IP 2011 for all evaluation
metrics with or without taking into account frequency (lines $f$ and $g$). Row
$f$ of Table 5 show that even without taking into account frequency _RPRS_
achieves higher effectiveness than the initial ranker (Ch.2, line $d$), the
previous state-of-the-art on CLEF-IP’11. Patent SBERT-TSDAE is used to embed
sentences for $SDR_{inf}$, RPRS, and _RPRS w/freq_ , which is Patent BERT
(Srebrovic and Yonamine, 2020) model that is loaded into into the SBERT
(Reimers and Gurevych, 2019) architecture and adapted into the domain using
TSDAE method (Wang et al., 2021a). $SDR_{inf}$ (line $e$) shows competitive
results compared to BM25 (line $a$ and $b$), however, it obtained much lower
results than $RPRSw/freq$.

Table 5. Results for Patent Retrieval. $\dagger$ indicate the statistically
significant improvement over best team on the CLEF-IP11 data (Ch.2 (Piroi et
al., 2011)) according to a paired-test (p¡0.001) with Bonferroni correction
for multiple testing respectively. Patent SBERT-TSDAE is used to embed
sentences for $SDR_{inf}$, RPRS, and _RPRS w/freq_.

|  Model | P@5 | P@10 | R@5 | R@10 | MAP@5 | MAP@10 | MRR
---|---|---|---|---|---|---|---|---
Probabilistic lexical matching baselines |  |  |  |  |  |  |
a | BM25 | 0.084 | 0.066 | 0.010 | 0.147 | 0.053 | 0.065 | 0.187
b | BM25 + KLI | 0.095 | 0.071 | 0.011 | 0.167 | 0.074 | 0.087 | 0.221
Top-two teams in English CLEF-IP’11 |  |  |  |  |  |  |
c | Hy.5 | 0.057 | 0.041 | 0.071 | 0.103 | 0.045 | 0.051 | 0.162
d | Ch.2 (Piroi et al., 2011) (Previous state-of-the-art) | 0.122 | 0.089 | 0.150 | 0.216 | 0.102 | 0.118 | 0.296
Sentence-based baseline |  |  |  |  |  |  |
e |  $SDR_{inf}$ \+ Ch.2 | 0.105 | 0.077 | 0.120 | 0.175 | 0.078 | 0.090 | 0.251
Proposed methods |  |  |  |  |  |  |
f | RPRS + Ch.2 | 0.131 | 0.092 | 0.160 | 0.221 | 0.113 | 0.129 | 0.319
g | RPRS w/freq + Ch.2 | 0.132$\dagger$ | 0.093$\dagger$ | 0.167$\dagger$ | 0.229$\dagger$ | 0.116$\dagger$ | 0.132$\dagger$ | 0.332$\dagger$

Wikipeda domain (Wine and Video games datasets). Table 6 shows that
$SDR_{inf}$ (line $c$ and $d$) obtains competitive results to $BM25+KLI$ (line
$b$) on SDR’s own datasets. However, as we achieved higher recall with
BM25+KLI, we use it as the initial ranker for our re-ranker. Results show the
generalizability and effectiveness of $RPRSw/freq$ and $RPRS$ on a very
different domain (Wikipedia) for two datasets as they could outperform the
state-of-the-art model on the WWG and WWA datasets for all evaluation metrics
with or without taking into account frequency (lines $e$ and $f$). It is
noteworthy to mention that we report all of the official metrics of WWG and
WWA datasets. We refer to _Hit Ratio $@k$_ in the original paper (Ginzburg et
al., 2021) as recall in this table as their definition for the _Hit Ratio_ is
equal to recall in Information Retrieval. We compute Mean Percentile Rank
($MPR$) using the original implementation by SDR’s authors (Ginzburg et al.,
2021).121212https://github.com/microsoft/SDR/blob/main/models/reco/wiki_recos_eval/eval_metrics.py

Table 6. Results for the video games (left), and wines (right) datasets from
Ginzburg et al. (2021). R referse to Roberta large in this table. $\dagger$
and * indicate a statistically significant improvement over previous State-of-
the-art (SDR, line b) and BM25+KLI (line d) respectively, according to a
paired-test ($p<0.005$) with Bonferroni correction for multiple testing.

| Model | Embedding |  Video games (WWG) |  | Wines (WWA)
---|---|---|---|---|---
| recall@10 | recall@100 | MPR | MRR |  | recall@10 | recall@100 | MPR | MRR
Probabilistic lexical matching baselines
a | BM25 | - | 0.2200 | 0.4887 | 0.8606 | 0.5681 |  | 0.1704 | 0.5041 | 0.8164 | 0.4562
b | BM25 + KLI | - | 0.2425 | 0.5499 | 0.8748 | 0.6150 |  | 0.1732 | 0.6332 | 0.8191 | 0.4759
Previous state-of-the-art
c | $SDR_{inf}$ | R SDR | 0.2360 | 0.5400 | 0.9740 | 0.6400 |  | 0.1700 | 0.5900 | 0.8930 | 0.5090
d | $SDR_{inf}$ | R TSDAE | 0.2384 | 0.5425 | 0.9761 | 0.6433 |  | 0.1712 | 0.5980 | 0.8940 | 0.5100
Proposed Methods
e | RPRS | R TSDAE | 0.2580$\dagger$* | 0.5499 | 0.9768 | 0.6501$\dagger$* |  | 0.1892$\dagger$* | 0.6332 | 0.8955 | 0.5210$\dagger$*
f | RPRS w/freq | R TSDAE |  0.2663$\dagger$* | 0.5499 | 0.9774 |  0.6676$\dagger$* |  |  0.2015$\dagger$* | 0.6332 | 0.8980 |  0.5360$\dagger$*

##  7\. Further analysis and discussion

In this section, we further analyze our results, starting with an ablation
study on the components of proposed methods, followed by the effect of
document length, document coverage, the parameter space, and a more detailed
comparison to the SDR baseline.

Table 7. Ablation study results on the COLIEE 2021 legal case retrieval task. |  | P | R | F1
---|---|---|---|---
| Full method (RPRS w/freq) | 0.1960 | 0.2891 | 0.2336
a | No QP (Eq.4) | 0.1670 | 0.2421 | 0.1976
b | No DP (Eq.5) | 0.1410 | 0.2180 | 0.1712
c | No b and k1 params | 0.1860 | 0.2691 | 0.2199
| RPRS without freq (Eq.1) | 0.1890 | 0.2799 | 0.2256
d | No min function in PRS | 0.1798 | 0.2644 | 0.2140

###  7.1. Ablation Study on RPRS w/freq components

The contribution of each module in RPRS w/freq is assessed by an ablation
study in Table LABEL:tab:ablation. The role of Query Proportion (QP), Document
Proportion (DP), and the parameters ($b$ and $k1$) has been evaluated in rows
$a$ (No QP), $b$ (No DP), $c$ (no b and k1). For the sake of clarity, we re-
write the modified eqution according to each row. For row $a$ (No QP), we
remove the equation 4 from _RPRS w/freq_ formula:

$RPRS\,w/freq=DP$

Similarly for row $b$ (No DP):

$RPRS\,w/freq=QP$

To do the ablation study of row $c$ (no b and k1), we modify equation 6,
$Fq_{R_{n}}$, and equation 7, $Fd_{R_{n}}$, as follow:

$Fq_{R_{n}}(S_{q},S_{d},S_{TK_{q}})=\frac{\sum_{q_{s}}^{S_{q}}|S_{d}\cap
r_{n}(q_{s},S_{TK_{q}})|}{\sum_{q_{s}}^{S_{q}}|S_{d}\cap
r_{n}(q_{s},S_{TK_{q}})|}$

$Fd_{R_{n}}(S_{d},S_{q},S_{TK_{q}})=\frac{\sum_{d_{s}}^{S_{d}}\sum_{r_{n}}^{R_{n}}|\\{d_{s}\\}\cap
r_{n}|}{\sum_{d_{s}}^{S_{d}}\sum_{r_{n}}^{R_{n}}|\\{d_{s}\\}\cap r_{n}|}$

The first and second highest drop in quality of ranking is caused by removing
$DP$ and $QP$, which shows the crucial role of both $QP$ and $DP$ in our re-
ranker. This aligns with the assumption for designing our re-ranker: a
candidate document and query are likely relevant if a large proportion of
their textual content is similar to each other. We further analyse the effect
of the min function in $RPRS$ for the equations of $q_{R_{n}}$ (Eq 3) and
$d_{R_{n}}$ (Eq 2), in row $d$. To do so, we modify equation 2, $q_{R_{n}}$,
and 3, $q_{R_{n}}$, as follows:

$q_{R_{n}}(S_{q},S_{d},S_{TK_{q}})=\sum_{q_{s}}^{S_{q}}|S_{d}\cap
r_{n}(q_{s},S_{TK_{q}})|$
$d_{R_{n}}(S_{d},S_{q},S_{TK_{q}})=\sum_{d_{s}}^{S_{d}}\sum_{r_{n}}^{R_{n}}|\\{d_{s}\\}\cap
r_{n}|$

In general, Table LABEL:tab:ablation shows that the full method with all
components including QP, DP, and $b$ and $k1$ parameters obtains highest
effectiveness and supports the necessity of each component. Morover, it ranks
the importance and impact of each component in the effectiveness of the
method. The importance of the $min$ function in $RPRS$ supports the intuition
that if the repetition of sentences of a candidate document in $R_{n}$ is not
controlled, the quality of the RPRS drops. This supports the $min$ function
for _RPRS_ and motivate proposing _RPRS w/freq_ to take into account the
repetitions in an intelligent way by adding $k1$ and $b$.

Figure 4. The probability of relevance and probability of retrieval on COLIEE
2021. In comparison to the probability of relevance, longer documents have a
disproportionately smaller probability of being retrieved by MTFT-BERT while
_RPRS w/freq_ is not biased against retrieving long documents.

###  7.2. Effect of document length in comparison to MTFT-BERT

We plot Figure 4 in order to analyze the effect of document length on the
effectiveness of the proposed method, and compare it with the MTFT-BERT model
on COLIEE21. This comparison is justified due to the fact that while in MTFT-
BERT the document length is bounded by the maximum input length of BERT, it is
the current state-of-the-art model on COLIEE’21. Therefore, we select the most
effective available model on this dataset for comparison in our analysis. In
Figure 4, probability of relevance $P(relevant|length(doc))$ indicates the
chance of having a relevant document with a specific length among all
documents, and probability of retrieval $P(ret|length(doc))$ shows the chance
of retrieving a relevant document with a specific length among all retrieved
documents, within top-$k$ ranks. To compute the probability of retrieval, we
set $k=5$ because the average number of relevant documents per query in the
collection is five. We analyze the probability of retrieval and probability of
relevance from relatively shorter documents – with $1$,$000$ words – towards
longer documents with $10$,$000$ words. An ideal ranker does not lose
effectiveness, probability of retrieval in Figure 4, by the increment in the
length of documents. As shown in Figure 4, the _RPRS w/freq_ model not only
does not loose effectiveness with increasing the document length, but also
retrieves the longer relevant documents easier and gets closer to the
probability of relevance $P(relevant|length(doc))$ for them. This could be due
to the fact that longer documents provide more information, and the _RPRS
w/freq_ model can take use that information effectively for ranking. This is
while MTFT-BERT does not improve by increasing document length and receive
similar effectiveness even with the more information that exist in a longer
document. This indicates that one of the reasons for gaining a higher
effectiveness of F1 score, which is the official metric of COLIEE dataset, by
the proposed method compared to the other models could be because of the fact
that _RPRS w/freq_ could capture the full information in the lengthy documents
and retrieves them more effective than other models such as MTFT-BERT.

###  7.3. Effect of covering the full length of queries and documents

We analyze if taking into account the full length of queries and documents is
an advantage for the proposed method in Figure 5. We argue that an ideal
semantic re-ranker should achieve higher effectiveness by receiving the full
length of queries and documents rather than truncated text as the input.
Therefore, we analyze the power of our re-ranker when the length of queries
and documents increases: we experiment with different maximum lengths for the
input. Given $l\in\\{256,$ $512,$ $1024,$ $2048,$ $4096,$ $8092,$ $16384\\}$
as the maximum input length, we select the first $l$ tokens of the query and
documents as their representation. Moreover, our analysis gives an in-depth
insight about the robustness of the proposed method and assess whether it
collapse when presented with long queries and documents or not. As Figure 5
shows, our re-ranker takes advantage of seeing the whole content of query and
documents in terms of effectiveness as it has the highest effectiveness when
we feed it with the full length and the lowest effectiveness with only $256$
tokens as the input. The figure also shows the largest leap is between 1024
and 4096 tokens, which is approaching the average document length in COLIEE
($5226$). This confirms that $RPRSw/freq$ takes advantage of the full document
length in estimating relevance.

Figure 5. Effectiveness of RPRS w/freq over varying truncation length for
queries and documents on COLIEE 2021 where truncation size is the maximum
length input.

###  7.4. Parameter sensitivity for $RPRS\,w/freq$

In addition to the tuned parameters that were found and explained in Section
5.5, we analyze the sensitivity of the proposed parameters in other ranges in
the following. Figure 6 shows the effects of changes in parameters $n,b$ and
$k1$ on the overall performance of our method on the COLIEE 2021 dataset. In
each diagram, the value of each point represents the F1 score according to
different values of the parameters. The following observations can be made
from Figure 6:

(a) n=1

(b) n=5

(c) n=10

Figure 6. Sensitivity analysis of our proposed method, $RPRSw/freq$, to
changes in parameters n, b and k1 on the COLIEE 2021 dataset.

  * •

A comparison between different values of $n$ – independent from $b$ and $k1$ –
indicates that the proposed method has the lowest effectiveness by reducing
the value of $n$ to $1$. The effectiveness of the model increases as $n$ is
set to $5$, and as we set $n=10$, we see that model effectiveness declines
while still being superior to $n$ set to $1$. We argue that this demonstrates
the existence of a trade off in the value of $n$, which determines the number
of the most similar sentences per query sentence in set $r_{n}$. If $n$ is set
to $1$, the selection of most similar sentences per query sentence would be
very strict, and the effectiveness would decrease. This could be because that
there may be other sentences from documents that are similar enough to qualify
as a most similar sentence to a query sentence — in addition to the top-$1$
most similar sentence — but with $n=1$, they will not be taken into account in
the computation of relevance score. On the other hand, if $n$ is set to $10$,
some sentences that are not similar enough to qualify as a most similar
sentence to a query sentence may appear in $r_{n}$, and as the result the
model obtain less effectiveness compared to $n=5$ because it would be less
strict in creation of $r_{n}$s for query sentences.

  * •

If the $k1$ parameter is set to $0$ for any value of the $b$ parameter, the
$f1$ score depends only on the value of the $n$ parameter. This is due to the
fact that $k1$ multiplies to the $b$ parameter, and if $k1$ is set to zero,
the relevance score is independent of changes in the $b$ parameter’s value.
Taking this into consideration, we could reduce the search space for parameter
optimization by eliminating the necessity of grid search for $b$ values when
$k1$ is set to zero.

  * •

By increasing the value of the $n$ parameter the proposed method becomes more
sensitive to the document length normalization parameter $b$, and frequency
saturation parameter $k1$. There are larger disparities in the $f1$ scores
when the $k1$ or $b$ parameter are changed for $n=10$ than for $n=5$, and for
$n=5$ than for $n=1$. This indicates where the method is less strict in
qualifying a sentence as a most similar sentence to a query sentence, the
impact of term saturation parameter ($k1$) and document length normalization
($b$) are higher.

  * •

The proposed method performs better in terms of effectiveness without doing
length normalization ($b=0$) when $n$ is set to $1$, and, as the result, the
selection of the most similar sentences per query sentence is very strict, and
it performs better with length normalization ($b=0.5$ and $b=1.0$) when $n$ is
less strict ($n=5$ and $n=10$). This could probably shows that the negative
effects on effectiveness by less strictly selecting the most similar sentences
per query sentence – by increasing the value of $n$ parameter – could might be
controlled with $b$ parameter.

In general, the parameter sensitivity analysis reveals that while optimizing
RPRS parameters increases effectiveness, it can still result in increased
effectiveness even without optimization by initializing each of the three
parameters with the value from middle of their range, e.g., $n=5$, $k1=1.5$,
$b=0.5$ that obtains $22.75$ f1 score which is still better than current
state-of-the-art on the dataset (MTFT-BERT(Abolghasemi et al., 2022b)) while
not being the ideal result by our _RPRS w/freq_ re-ranker.

###  7.5. Further analysis of RPRS compared to SDR

We compared RPRS to SDR because it has some similarities: it also uses
sentence-level relevance scoring based on sentence embeddings and it uses a
bi-encoder architecture, like RPRS. There are fundamentally significance
differences, however: SDR creates a sentence-level similarity matrix $M$ for
each pair of query and document paragraphs and based on $M$ matrices, creates
a paragraph similarity matrix $P$ for a candidate document. Each cell of $P$
contains the similarity between a pair of paragraphs from query and a
candidate document. The matrix $P$ is then normalized to ($NRM(P)$) and the
total score $S$ is computed based on $NRM(P)$. The denominator in the formula
for computing the matrix $P$ is the number of sentences in a query paragraph
and the denominator for the total score $S$ is the number of paragraphs in the
query. An important consequence is that the length of the candidate documents
is not taken into account, while RPRS explicitly does so. Moreover, if one
paragraph of a candidate document is the most similar paragraph to all
paragraphs of the query, SDR does not penalize this repetition, while we take
that into account with the $k1$ parameter in _RPRS w/freq_ , which controls
frequency saturation. In addition, it is noteworthy to mention that SDR uses
the cosine similarity directly in their approach while we only consider that
score for ranking sentences in $R_{n}$.

Table 8. Pearson Product-Moment Correlation coefficients between document length and relevance score. |  | Pearson Product-Moment Correlation coefficient
---|---|---
| Model | COLIEE 2021 | Caselaw | CLEF-IP | Video Games | Wines datasets
| SDR | $0.5768$ | $0.5901$ | $0.612$ | $0.542$ | $0.5366$
| RPRS w/freq | $-0.0565$ | $-0.0401$ | $0.0105$ | $-0.0322$ | $0.0109$

Figure 7. Distribution of word counts per each document for all of the five
datasets: Patent retrieval (CLEF-IP), Case law retrieval (COLIEE 2021 and
CASELAW), and Wikipedia (WWG and WWA).

Losada et al. (Losada et al., 2008) indicate that an ideal retrieval model
should not be tuned to favour longer documents and the relevance scores it
produces should not be correlated with document length. They argue that the
previous empirical evidence supporting the scope hypothesis131313The scope
hypothesis in Information Retrieval states that a relationship exists between
document length and relevance (Robertson and Walker, 1994). is over-
exaggerated and inaccurate due to the incompleteness of modern collections.
This is even more important in QBD retrieval as the length of relevant
documents could vary from short to very long. Therefore, the retrieval model
should not be biased to the document length. Consequently, in order to study
the sensitivity of our proposed method to document length, and compare it with
SDR, we analyze the correlation between candidate document length and the
relevance score produced by _RPRS w/freq_ and SDR in Table 8. We find that the
Pearson Product-Moment Correlation coefficients for _RPRS w/freq_ on all five
datasets are close to zero while for _SDR_ these correlations are much higher,
above $0.5$, for all datasets. This indicates the strong correlation of SDR
relevance score with the document length and on the other hand the robustness
of _RPRS w/freq_ with respect to document length. In other words, _RPRS
w/freq_ is effective while it is not biased to the length of the document.

In addition to the correlation analysis, we further study the statistics over
word counts of documents in all datasets to find out if there is a specific
statistical characteristic in the Video games and Wines datasets that makes
SDR stronger on those datasets compared to the three datasets in the legal and
patent domain. Figure 7 shows the document length distribution over the
datasets that indicates higher variance among Patent and Legal datasets
compared to Wikipedia datasets (WVG and WWA). The boxes bound the $25$th to
$75$th percentiles, top whisker cover data within $1.5\times$ the inter-
quartile range (IQR), and outliers are removed. Moreover, the standard
deviation of documents length for the Video Games and Wines datasets are 675
abd $721$ respectively. On the other hand, for COLIEE 2021, Caselaw, and CLEF-
IP, the standard deviation of documents length are $6933$, $4937$, and
$11402$. Thus, the documents in SDR’s Wikipedia datasets have less variance
compared to the three legal and patent datasets and are shorten than COLIEE
and Patent datasets. As a result, we conclude that the lower effectiveness of
SDR on the legal and patent datasets is likely caused by SDR not being robust
against the length of documents and suffers from the high standard deviation,
and longer documents in the legal and patent datasets in comparison to its own
datasets. Our experimental results in Section 6.5 indicate that our method
_RPRS w/freq_ also outperforms SDR on their datasets, so the robustness of
_RPRS w/freq_ is not only beneficial for extremely lengthy documents but also
in other domains.

##  8\. Conclusion and Future work

In this paper, we proposed methods for effectively exploiting sentence-level
representations produced by the highly efficient bi-encoder architecture for
Query-by-Document (QBD) re-ranking. We proposed a novel model using SBERT
representations, with a frequency-based extension inspired by BM25’s ‘term
saturation’ mechanism and the incorporation of document length normalization
into the relevance score computation. Our experiments on five datasets show
that our model _RPRS w/freq_ takes advantage of the long queries and documents
that are common in QBD retrieval. While our _RPRS w/freq_ model is
unsupervised with only three tunable parameters, it is more effective than
state-of-the-art supervised neural and lexical models. In addition, it is
highly efficient for retrieval tasks with long documents and long queries
because the operation in order to compute relevance score by _RPRS w/freq_ are
based on: (1) a bi-encoder, SBERT, which is about $46,800$ times faster than
the common cross-encoder BERT architecture; (2) the pre-processing, embedding,
and indexing of document sentences could be done before the query time; (3)
the only calculation based on the embeddings are cosine similarity and sorting
that are simple and efficient operations.

We show the effectiveness of RPRS w/freq on five datasets with low-resource
training data, which indicates its suitability for QBD retrieval tasks in
which the training data is very limited compared to general web search due to
the high cost of dataset creation for these tasks. Therefore, we attain high
efficiency and effectiveness while being optimized on low-recourse training
data with _RPRS w/freq_.

While we outperform the state-of-the-art models on each dataset with our
proposed method, the effectiveness results show the difficult nature of the
tasks. One reason for that is the low effectiveness of the first stage
retrieval model limits the re-ranker performance. As one direction of
improvement for future work, we aim to focus on first-stage retrieval by
designing a modification to our proposed method which is suitable for first-
stage retrieval tasks. The parameter sensitivity analysis reveals that while
optimizing RPRS parameters increases effectiveness, it can still result in
increased effectiveness even without optimization by initializing each of the
three parameters with a value from the middle of their range.

For further future improvement on our model, we might be inspired by two other
variants of BM25 to make the proposed method parameter-free with dynamically
computed parameters. For the frequency saturation parameter, $k1$, a variant
of BM25 called ‘BM25 adaptive’ (Lv and Zhai, 2011) which dynamically computes
the $k1$ parameter could be taken into account as inspiration. Additionally,
for the document length normalization parameter, $b$, Lipani et al. (Lipani et
al., 2015) propose a length normalization method that removes the need for a
$b$ parameter in BM25 which could inspire us to dynamically compute the $b$
parameter of _RPRS w/freq_ in future. Considering the intuition of both
approaches, working on dynamically initializing the $n$ parameter could make
our method completely parameter-free which is an interesting direction for
future work. We argue because a sensitivity analysis reveals a pattern that
indicates robust and greater effectiveness relative to the baseline for the
proposed method with different parameter values, finding a nearly optimal
parameter value dynamically might likely be possible.

We believe the proposed method is suitable for other tasks that deal with long
documents. In addition, we suggest analyzing the effectiveness of our proposed
method on other IR tasks in which we do not have extremely long documents, but
the query and candidate document do consist of multiple sentences (Cohan et
al., 2020; Abolghasemi et al., 2022a). We hope that our work will spark more
research interest in retrieval for extremely long queries and open up
possibilities for highly efficient long-document retrieval.

## References

  * (1)
  * Abolghasemi et al. (2022a) Amin Abolghasemi, Arian Askari, and Suzan Verberne. 2022a.  On the Interpolation of Contextualized Term-Based Ranking with BM25 for Query-by-Example Retrieval. In _Proceedings of the 2022 ACM SIGIR International Conference on Theory of Information Retrieval_ (Madrid, Spain) _(ICTIR ’22)_. Association for Computing Machinery, New York, NY, USA, 161–170.  https://doi.org/10.1145/3539813.3545133
  * Abolghasemi et al. (2022b) Amin Abolghasemi, Suzan Verberne, and Leif Azzopardi. 2022b.  Improving BERT-based Query-by-Document Retrieval with Multi-Task Optimization. In _Advances in Information Retrieval, 44th European Conference on IR Research, ECIR 2022_.  https://arxiv.org/abs/2202.00373
  * Akkalyoncu Yilmaz et al. (2019) Zeynep Akkalyoncu Yilmaz, Wei Yang, Haotian Zhang, and Jimmy Lin. 2019.  Cross-Domain Modeling of Sentence-Level Evidence for Document Retrieval. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. Association for Computational Linguistics, Hong Kong, China, 3490–3496.  https://doi.org/10.18653/v1/D19-1352
  * Althammer et al. (2021) Sophia Althammer, Sebastian Hofstätter, and Allan Hanbury. 2021.  Cross-domain Retrieval in the Legal and Patent Domains: a Reproducibility Study. In _European Conference on Information Retrieval_. Springer, 3–17.
  * Askari and Verberne (2021) A Askari and S Verberne. 2021.  Combining lexical and neural retrieval with longformer-based summarization for effective case law retrieva. In _Proceedings of the second international conference on design of experimental search & information REtrieval systems_. CEUR, 162–170.
  * Barkan et al. (2020) Oren Barkan, Noam Razin, Itzik Malkiel, Ori Katz, Avi Caciularu, and Noam Koenigstein. 2020\.  Scalable attentive sentence pair modeling via distilled sentence embedding. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 34. 3235–3242.
  * Beltagy et al. (2020) Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020\.  Longformer: The long-document transformer.  _arXiv preprint arXiv:2004.05150_ (2020).
  * Bird et al. (2009) Steven Bird, Ewan Klein, and Edward Loper. 2009.  _Natural language processing with Python: analyzing text with the natural language toolkit_.  ” O’Reilly Media, Inc.”.
  * Buckland and Gey (1994) Michael Buckland and Fredric Gey. 1994.  The relationship between recall and precision.  _Journal of the American society for information science_ 45, 1 (1994), 12–19.
  * Carlsson et al. (2020) Fredrik Carlsson, Amaru Cuba Gyllensten, Evangelia Gogoulou, Erik Ylipää Hellqvist, and Magnus Sahlgren. 2020.  Semantic re-tuning with contrastive tension. In _International Conference on Learning Representations_.
  * Chalkidis et al. (2020) Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. 2020.  LEGAL-BERT: The muppets straight out of law school.  _arXiv preprint arXiv:2010.02559_ (2020).
  * Chen et al. (2020) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020.  A simple framework for contrastive learning of visual representations. In _International conference on machine learning_. PMLR, 1597–1607.
  * Chen et al. (2022) Xiaoyang Chen, Kai Hui, Ben He, Xianpei Han, Le Sun, and Zheng Ye. 2022.  Incorporating Ranking Context for End-to-End BERT Re-ranking. In _European Conference on Information Retrieval_. Springer, 111–127.
  * Cleverdon (1972) Cyril W Cleverdon. 1972\.  On the inverse relationship of recall and precision.  _Journal of documentation_ 28, 3 (1972), 195–201.
  * Cohan et al. (2020) Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel S Weld. 2020.  Specter: Document-level representation learning using citation-informed transformers.  _arXiv preprint arXiv:2004.07180_ (2020).
  * Danilak (2014) M Danilak. 2014\.  langdetect: Language detection library ported from Google’s language detection.  _See https://pypi. python. org/pypi/langdetect/(accessed 19 January 2015)_ (2014).
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ (2018).
  * D’hondt et al. (2011) Eva D’hondt, Suzan Verberne, Wouter Alink, and Roberto Cornacchia. 2011.  Combining Document Representations for Prior-art Retrieval.. In _CLEF (Notebook Papers/Labs/Workshop)_.
  * Fujii et al. (2007) Atsushi Fujii, Makoto Iwayama, and Noriko Kando. 2007\.  Overview of the Patent Retrieval Task at the NTCIR-6 Workshop.. In _NTCIR_.
  * Gao et al. (2021) Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021.  Simcse: Simple contrastive learning of sentence embeddings.  _arXiv preprint arXiv:2104.08821_ (2021).
  * Ginzburg et al. (2021) Dvir Ginzburg, Itzik Malkiel, Oren Barkan, Avi Caciularu, and Noam Koenigstein. 2021.  Self-Supervised Document Similarity Ranking via Contextualized Language Models and Hierarchical Inference. In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_. Association for Computational Linguistics, Online, 3088–3098.  https://doi.org/10.18653/v1/2021.findings-acl.272
  * Gururangan et al. (2020) Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. 2020.  Don’t stop pretraining: adapt language models to domains and tasks.  _arXiv preprint arXiv:2004.10964_ (2020).
  * Hadsell et al. (2006) Raia Hadsell, Sumit Chopra, and Yann LeCun. 2006\.  Dimensionality reduction by learning an invariant mapping. In _2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06)_ , Vol. 2. IEEE, 1735–1742.
  * Hofstätter et al. (2019) Sebastian Hofstätter, Navid Rekabsaz, Mihai Lupu, Carsten Eickhoff, and Allan Hanbury. 2019\.  Enriching word embeddings for patent retrieval with global context. In _European Conference on Information Retrieval_. Springer, 810–818.
  * Humeau et al. (2019) Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. 2019.  Poly-encoders: Transformer architectures and pre-training strategies for fast and accurate multi-sentence scoring.  _arXiv preprint arXiv:1905.01969_ (2019).
  * Kitaev et al. (2020) Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. 2020\.  Reformer: The efficient transformer.  _arXiv preprint arXiv:2001.04451_ (2020).
  * Kocaman and Talby (2021) Veysel Kocaman and David Talby. 2021.  Spark NLP: Natural language understanding at scale.  _Software Impacts_ (2021), 100058\.  https://doi.org/10.1016/j.simpa.2021.100058
  * Lastres (2015) Steven A Lastres. 2015\.  Rebooting legal research in a digital age.
  * Le et al. (2021) Nhat XT Le, Moloud Shahbazi, Abdulaziz Almaslukh, and Vagelis Hristidis. 2021.  Query by documents on top of a search interface.  _Information Systems_ 101 (2021), 101793.
  * Li et al. (2020) Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li. 2020\.  On the sentence embeddings from pre-trained language models.  _arXiv preprint arXiv:2011.05864_ (2020).
  * Lipani et al. (2015) Aldo Lipani, Mihai Lupu, Allan Hanbury, and Akiko Aizawa. 2015\.  Verboseness fission for bm25 document length normalization. In _Proceedings of the 2015 International Conference on the Theory of Information Retrieval_. 385–388.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ (2019).
  * Locke et al. (2017) Daniel Locke, Guido Zuccon, and Harrisen Scells. 2017\.  Automatic query generation from legal texts for case law retrieval. In _Asia Information Retrieval Symposium_. Springer, 181–193.
  * Losada et al. (2008) David E Losada, Leif Azzopardi, and Mark Baillie. 2008\.  Revisiting the relationship between document length and relevance. In _Proceedings of the 17th ACM conference on Information and knowledge management_. 419–428.
  * Lv and Zhai (2011) Yuanhua Lv and ChengXiang Zhai. 2011.  Adaptive term frequency normalization for BM25. In _Proceedings of the 20th ACM international conference on Information and knowledge management_. 1985–1988.
  * Ma et al. (2021) Yixiao Ma, Yunqiu Shao, Bulou Liu, Yiqun Liu, Min Zhang, and Shaoping Ma. 2021\.  Retrieving Legal Cases from a Large-scale Candidate Corpus. In _Proceedings of the eighth International Competition on Legal Information Extraction/Entailment, COLIEE2021 (2021)_.
  * Mysore et al. (2021a) Sheshera Mysore, Arman Cohan, and Tom Hope. 2021a.  Multi-Vector Models with Textual Guidance for Fine-Grained Scientific Document Similarity.  _arXiv preprint arXiv:2111.08366_ (2021).
  * Mysore et al. (2021b) Sheshera Mysore, Tim O’Gorman, Andrew McCallum, and Hamed Zamani. 2021b.  CSFCube–A Test Collection of Computer Science Research Articles for Faceted Query by Example.  _arXiv preprint arXiv:2103.12906_ (2021).
  * Nguyen et al. (2016) Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016.  MS MARCO: A human generated machine reading comprehension dataset.  _choice_ 2640 (2016), 660.
  * Nogueira and Cho (2019) Rodrigo Nogueira and Kyunghyun Cho. 2019.  Passage Re-ranking with BERT.  _arXiv preprint arXiv:1901.04085_ (2019).
  * Padigela et al. (2019) Harshith Padigela, Hamed Zamani, and W Bruce Croft. 2019\.  Investigating the successes and failures of BERT for passage re-ranking.  _arXiv preprint arXiv:1905.01758_ (2019).
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019\.  Pytorch: An imperative style, high-performance deep learning library.  _Advances in neural information processing systems_ 32 (2019), 8026–8037.
  * Piroi and Hanbury (2019) Florina Piroi and Allan Hanbury. 2019.  Multilingual Patent Text Retrieval Evaluation: CLEF–IP.  In _Information Retrieval Evaluation in a Changing World_. Springer, 365–387.
  * Piroi et al. (2013) Florina Piroi, Mihai Lupu, and Allan Hanbury. 2013\.  Overview of clef-ip 2013 lab. In _International Conference of the Cross-Language Evaluation Forum for European Languages_. Springer, 232–249.
  * Piroi et al. (2011) Florina Piroi, Mihai Lupu, Allan Hanbury, and Veronika Zenz. 2011.  CLEF-IP 2011: Retrieval in the Intellectual Property Domain.. In _CLEF (notebook papers/labs/workshop)_. Citeseer.
  * Qi et al. (2020) Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020.  Stanza: A Python Natural Language Processing Toolkit for Many Human Languages. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_.
  * Rabelo et al. (2022) Juliano Rabelo, Randy Goebel, Mi-Young Kim, Yoshinobu Kano, Masaharu Yoshioka, and Ken Satoh. 2022\.  Overview and discussion of the competition on legal information extraction/entailment (COLIEE) 2021.  _The Review of Socionetwork Strategies_ 16, 1 (2022), 111–133.
  * Rabelo et al. (2020) Juliano Rabelo, Mi-Young Kim, Randy Goebel, Masaharu Yoshioka, Yoshinobu Kano, and Ken Satoh. 2020\.  COLIEE 2020: Methods for Legal Document Retrieval and Entailment.  https://sites.ualberta.ca/~rabelo/COLIEE2021/COLIEE_2020_summary.pdf
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.  _arXiv preprint arXiv:1908.10084_ (2019).
  * Robertson et al. (2009) Stephen Robertson, Hugo Zaragoza, et al. 2009\.  The probabilistic relevance framework: BM25 and beyond.  _Foundations and Trends® in Information Retrieval_ 3, 4 (2009), 333–389.
  * Robertson and Walker (1994) Stephen E Robertson and Steve Walker. 1994.  Some simple effective approximations to the 2-poisson model for probabilistic weighted retrieval. In _SIGIR’94_. Springer, 232–241.
  * Rosa et al. (2021) Guilherme Moraes Rosa, Ruan Chaves Rodrigues, Roberto Lotufo, and Rodrigo Nogueira. 2021.  Yes, BM25 is a Strong Baseline for Legal Case Retrieval.  _arXiv preprint arXiv:2105.05686_ (2021).
  * Sanchez (2019) George Sanchez. 2019\.  Sentence Boundary Detection in Legal Text. In _Proceedings of the Natural Legal Language Processing Workshop 2019_. Association for Computational Linguistics, Minneapolis, Minnesota, 31–38.  https://doi.org/10.18653/v1/W19-2204
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter.  _arXiv preprint arXiv:1910.01108_ (2019).
  * Sekulić et al. (2020) Ivan Sekulić, Amir Soleimani, Mohammad Aliannejadi, and Fabio Crestani. 2020.  Longformer for MS MARCO Document Re-ranking Task.  _arXiv preprint arXiv:2009.09392_ (2020).
  * Shalaby and Zadrozny (2019) Walid Shalaby and Wlodek Zadrozny. 2019.  Patent retrieval: a literature review.  _Knowledge and Information Systems_ 61, 2 (2019), 631–660.
  * Shao et al. (2020) Yunqiu Shao, Jiaxin Mao, Yiqun Liu, Weizhi Ma, Ken Satoh, Min Zhang, and Shaoping Ma. 2020.  BERT-PLI: Modeling Paragraph-Level Interactions for Legal Case Retrieval. In _Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20_. 3501–3507.
  * Song et al. (2020) Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2020.  Mpnet: Masked and permuted pre-training for language understanding.  _Advances in Neural Information Processing Systems_ 33 (2020), 16857–16867.
  * Srebrovic and Yonamine (2020) Rob Srebrovic and Jay Yonamine. 2020.  _Leveraging the BERT Algorithm for Patents with TensorFlow and Big Query_.  Technical Report. Google (2020).  https://services.google.com/fh/files/blogs/bert_for_patents_white_paper.pdf
  * Tran et al. (2020) Vu Tran, Minh Le Nguyen, Satoshi Tojo, and Ken Satoh. 2020\.  Encoded summarization: summarizing documents into continuous vector space for legal case retrieval.  _Artificial Intelligence and Law_ 28, 4 (2020), 441–467.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need. In _Advances in neural information processing systems_. 5998–6008.
  * Verberne et al. (2010) Suzan Verberne, EKL D’hondt, NHJ Oostdijk, and Cornelis HA Koster. 2010.  Quantifying the challenges in parsing patent claims. In _Proceedings of the 1st International Workshop on Advances in Patent Information Retrieval (AsPIRe’10), March 28, 2010, Milton Keynes._
  * Verberne and D’hondt (2009) Suzan Verberne and Eva D’hondt. 2009.  Prior art retrieval using the claims section as a bag of words. In _Workshop of the Cross-Language Evaluation Forum for European Languages_. Springer, 497–501.
  * Verberne et al. (2016) Suzan Verberne, Maya Sappelli, Djoerd Hiemstra, and Wessel Kraaij. 2016.  Evaluation and analysis of term scoring methods for term extraction.  _Information Retrieval Journal_ 19, 5 (2016), 510–545.
  * Wang et al. (2021a) Kexin Wang, Nils Reimers, and Iryna Gurevych. 2021a.  TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learning.  _arXiv preprint arXiv:2104.06979_ (2021).
  * Wang et al. (2021b) Kexin Wang, Nandan Thakur, Nils Reimers, and Iryna Gurevych. 2021b.  GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval.  _arXiv preprint arXiv:2112.07577_ (2021).
  * Wang et al. (2020) Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020.  Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers.  _Advances in Neural Information Processing Systems_ 33 (2020), 5776–5788.
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019\.  Huggingface’s transformers: State-of-the-art natural language processing.  _arXiv preprint arXiv:1910.03771_ (2019).
  * Yan et al. (2019) Ming Yan, Chenliang Li, Chen Wu, Bin Bi, Wei Wang, Jiangnan Xia, and Luo Si. 2019\.  IDST at TREC 2019 Deep Learning Track: Deep Cascade Ranking with Generation-based Document Expansion and Pre-trained Language Modeling.. In _TREC_.
  * Yang et al. (2018) Eugene Yang, David D Lewis, Ophir Frieder, David A Grossman, and Roman Yurchak. 2018.  Retrieval and Richness when Querying by Document.. In _DESIRES_. 68–75.
  * Yang et al. (2009) Yin Yang, Nilesh Bansal, Wisam Dakka, Panagiotis Ipeirotis, Nick Koudas, and Dimitris Papadias. 2009\.  Query by document. In _Proceedings of the Second ACM International Conference on Web Search and Data Mining_. 34–43.
  * Yilmaz et al. (2019a) Zeynep Akkalyoncu Yilmaz, Shengjin Wang, Wei Yang, Haotian Zhang, and Jimmy Lin. 2019a.  Applying BERT to document retrieval with birch. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations_. 19–24.
  * Yilmaz et al. (2019b) Zeynep Akkalyoncu Yilmaz, Wei Yang, Haotian Zhang, and Jimmy Lin. 2019b.  Cross-domain modeling of sentence-level evidence for document retrieval. In _Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP)_. 3490–3496.
  * Yoshioka (2021) Masaharu Juliano Yoshioka. 2021\.  COLIEE 2020: Methods for Legal Document Retrieval and Entailment. In _New Frontiers in Artificial Intelligence: JSAI-isAI 2020 Workshops, JURISIN, LENLS 2020 Workshops, Virtual Event, November 15-17, 2020, Revised Selected Papers_ , Vol. 12758. Springer Nature, 196.
  * Zaheer et al. (2020) Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020\.  Big Bird: Transformers for Longer Sequences.. In _NeurIPS_.
  * Zhang et al. (2017) Haotian Zhang, Mustafa Abualsaud, Nimesh Ghelani, Angshuman Ghosh, Mark D Smucker, Gordon V Cormack, and Maura R Grossman. 2017.  UWaterlooMDS at the TREC 2017 Common Core Track.. In _TREC_.
  * Zhong et al. (2021) Ruiqi Zhong, Dhruba Ghosh, Dan Klein, and Jacob Steinhardt. 2021.  Are larger pretrained language models uniformly better? comparing performance at the instance level.  _arXiv preprint arXiv:2105.06020_ (2021).
  * Zhou et al. (2021) Helong Zhou, Liangchen Song, Jiajie Chen, Ye Zhou, Guoli Wang, Junsong Yuan, and Qian Zhang. 2021.  Rethinking soft labels for knowledge distillation: A bias-variance tradeoff perspective.  _arXiv preprint arXiv:2102.00650_ (2021).