11institutetext: TU Wien, Austria
11email<EMAIL_ADDRESS>
# Cross-domain Retrieval in the Legal and Patent Domains: a Reproducibility
Study

Sophia Althammer  Sebastian Hofstätter  Allan Hanbury

###### Abstract

Domain specific search has always been a challenging information retrieval
task due to several challenges such as the domain specific language, the
unique task setting, as well as the lack of accessible queries and
corresponding relevance judgements. In the last years, pretrained language
models – such as BERT – revolutionized web and news search. Naturally, the
community aims to adapt these advancements to cross-domain transfer of
retrieval models for domain specific search. In the context of legal document
retrieval, Shao et al. propose the BERT-PLI framework by modeling the
Paragraph-Level Interactions with the language model BERT. In this paper we
reproduce the original experiments, we clarify pre-processing steps and add
missing scripts for framework steps, however we are not able to reproduce the
evaluation results. Contrary to the original paper, we demonstrate that the
domain specific paragraph-level modelling does not appear to help the
performance of the BERT-PLI model compared to paragraph-level modelling with
the original BERT. In addition to our legal search reproducibility study, we
investigate BERT-PLI for document retrieval in the patent domain. We find that
the BERT-PLI model does not yet achieve performance improvements for patent
document retrieval compared to the BM25 baseline. Furthermore, we evaluate the
BERT-PLI model for cross-domain retrieval between the legal and patent domain
on individual components, both on a paragraph and document-level. We find that
the transfer of the BERT-PLI model on the paragraph-level leads to comparable
results between both domains as well as first promising results for the cross-
domain transfer on the document-level. For reproducibility and transparency as
well as to benefit the community we make our source code and the trained
models publicly available.

###### Keywords:

Information Retrieval Domain Specific Search Reproducibility Legal Search
Patent Search Cross-domain Retrieval

##  1 Introduction

Bringing the substantial effectiveness gains from contextualized language
retrieval models from web and news search to other domains is paramount to the
equitable use of machine learning models in Information Retrieval (IR). The
promise of these pre-trained models is a cross-domain transfer with limited
in-domain training data. Thus we investigate in this paper the document
retrieval on two specific language domains, the legal and the patent domain,
and study the transferability of the retrieval models between both domains.

In case law systems the precedent cases are a key source for lawyers,
therefore it is essential for the lawyers’ work to retrieve prior cases which
support the query case. Similarly in the patent domain, patent examiners
review patent applications and search for prior art, in order to determine
what contribution the invention makes over the prior art. The recent advances
in language modelling have shown that contextualized language models enhance
the performance of information retrieval models in the web and news domain
compared to traditional ad-hoc retrieval models [10, 11]. However for legal
and patent retrieval we have a different task setting as the documents contain
longer text with a mean of 11,100 words per document. In document retrieval
every passage may be relevant, therefore in a high-recall setting such as ours
it is crucial for the retrieval model to take the whole document into account.
This is a challenge for contextualized language retrieval models, which are
only capable of computing short passages with a length up to $512$ tokens [7,
25, 26].

Recently, Shao et al. [20] aimed to bring the gains of language modelling to
legal document retrieval and tackle the challenge of long documents by
proposing BERT-PLI, a multi-stage framework which models Paragraph-Level
Interactions of queries and candidates with multiple paragraphs using BERT
[6]. The document-level relevance of each query and candidate pair is
predicted based on paragraph-level interaction of the query and candidate
paragraphs which are aggregated with a recurrent neural network (LSTM or GRU).
The BERT-PLI model is trained in two stages: first, BERT is trained on a
paragraph entailment task, and second the recurrent aggregation component is
trained on a binary classification task.

In this paper we reproduce the results for the legal retrieval task. We found
shortcomings in the description of the data pre-processing and evaluation
methods, after a discussion with the authors of the original paper we could
clarify how the evaluation results are achieved. As the published code is
missing crucial parts, we re-implement the pre-processing, the first stage
BERT fine-tuning as well as the retrieval with BM25 in the second stage and
the overall evaluation. Furthermore we analyze the ablation study of the
original paper and answer the following research question:

  * RQ1

Does fine-tuning BERT on domain specific paragraphs improve the retrieval
performance for document retrieval?

The original paper finds a $7-9\%$ performance improvement of the BERT-PLI
model for legal retrieval, when fine-tuning BERT on the legal paragraphs.
Contrary to the original paper, we find that the paragraph-level modelling
with BERT, fine-tuned on the domain specific paragraph-level modelling, does
not appear to help the BERT-PLI model’s performance on legal document
retrieval. In line with that, we also demonstrate that the patent specific
paragraph-level modelling harms the performance of the BERT-PLI model also for
the patent retrieval task and remains a promising opportunity.

In order to analyze the proposed BERT-PLI model for another document retrieval
task with long documents, we investigate following research question:

  * RQ2

To what extent is a BERT-PLI model, which is trained on patent retrieval,
beneficial for document retrieval in the patent domain?

We find that the patent domain BERT-PLI model is outperformed by the BM25
baseline for the patent retrieval task. This shows that the document retrieval
with BERT is not yet beneficial for the patent retrieval and stays a promising
opportunity.

As the legal and patent documents come from similar language domains, it
becomes an interesting question to what extent we can transfer the domain
specific retrieval models from one to the other domain. Especially because of
the restricted accessbility of domain specific, labelled retrieval data there
is the need for studying cross-domain transfer of document retrieval models.

  * RQ3

To what extent is cross-domain transfer on paragraph- and document-level of
the domain specific BERT-PLI model between legal and patent domain possible?

We show that the transfer of the domain specific paragraph-level interaction
modelling is possible between the legal and patent domain with similar
performance of the retrieval model. Furthermore we find on the document-level
transfer that the zero-shot application of a patent domain specific BERT-PLI
model for the legal retrieval task achieves a lower performance than the BM25
baseline. Showing first promising results, the cross-domain transfer of
retrieval models stays an open and exciting research direction. Our main
contributions are:

  * $\bullet$

We reproduce the experiments of Shao et al. [20] and investigate shortcomings
in the data pre-processing and model methods. Contrary to the paper we find
that domain specific paragraph-level modelling does not appear to help the
performance of the BERT-PLI model for legal document retrieval

  * $\bullet$

We train a domain specific BERT-PLI model for the patent domain and
demonstrate that it does not yet outperform the BM25 baseline

  * $\bullet$

We analyze the cross-domain transfer of the BERT-PLI model between the legal
and patent domain with first promising results

  * $\bullet$

In order to make our results available for reproduction and to benefit the
community, we publish the source code and trained models at:
https://github.com/sophiaalthammer/bert-pli

##  2 Methods

###  2.1 Task description

Document retrieval in the legal and patent domain are specialized IR tasks
with the particularity that query and candidates are long documents which use
domain specific language.
In legal document retrieval, the relevant documents are defined as the
previous cases which should be noticed for solving the query case [17], in
other words which support or contradict the query document [20]. The legal
documents consist of long text containing the factual description of a case.
Relevance in the patent domain is defined for the prior art search task [15],
i.e. it is the task to find documents in the corpus that are related to the
new invention or describe the same invention. The patent documents consist of
a title, an abstract, claims and a description as well as metadata like the
authors or topical classifications. As we investigate retrieval and
classification based on the textual information, we will only consider the
textual data of the patent documents.

###  2.2 BERT-PLI architecture overview

Figure 1: BERT-PLI Multistage architecture

As the BERT model advanced the state-of-the-art in natural language processing
and information retrieval, but has the restriction that it can only model the
relation between short paragraphs, Shao et al. [20] propose a multi-stage
framework model using BERT for the retrieval of long documents which is
illustrated in Figure 1. The training is separated into two stages. In stage
1, BERT is fine-tuned on a relevance prediction task on a paragraph-level.
BERT takes the concatenated query and document paragraph as input and is then
fine-tuned on predicting the relevance of the candidate paragraph to the query
paragraph given the output vector of the special [CLS] token of BERT.
Therefore this output vector is trained to be a relevance representation on a
paragraph-level of the two concatenated input paragraphs.
This fine-tuned BERT model is used in stage 2, where the full document
retrieval with paragraph-level interaction modelling takes place. For a query
document $q$ the top $K$ candidates are retrieved from a corpus using BM25
[18], and the query document as well as the top $K$ candidates are split into
paragraphs. Then for each candidate $i\in{1,..,K}$ the first $N$ paragraphs of
the query document and the first $M$ paragraphs of the candidate are
concatenated and their relevance representation is calculated with the BERT
model from stage 1. This yields an interaction matrix between the query and
candidate paragraphs. An additional Maxpooling layer captures the strongest
matching signals per query paragraph and yields a document-level relevance
representation of the query and the candidate. This document-level relevance
representation is used to train an RNN model with a succeeding attention and
fully-connected forward layer which we will refer to as Attention RNN. This
Attention RNN yields the binary prediction of the relevance for the query and
candidate document.

###  2.3 Cross-domain evaluation approach

Figure 2: Cross-domain evaluation approach

In the first stage of the BERT-PLI framework the BERT model learns to model
the paragraph-level interaction. For the two different domains we fine-tune
the BERT model on a paragraph-level relevance prediction task, which yields
the paragraph-level interaction LawBERT model for the legal and the PatentBERT
model for the patent domain. In order to analyze the influence of the domain
specific paragraph-level modelling, we compare the document retrieval models
trained with the paragraph-level modelling of LawBERT or PatentBERT to
document retrieval models trained on the paragraph-level modelling of the
original BERT model. The paragraph-level modelling with the original BERT
model is denoted with $\textbf{BERT}_{\textbf{ORG}}$ as in Figure 2. Based on
these paragraph-level interaction representations we train an AttentionRNN on
the legal as well as on the patent document-level retrieval task, which we
denote with LawRNN or PatentRNN respectively. In order to isolate the impact
of the different modelling of the paragraph-level interactions from LawBERT
and PatentBERT, we additionally train an AttentionRNN on the patent document
retrieval task given their LawBERT relevance representations and vice versa.
We evaluate the resulting models on the legal or the patent test document
retrieval set, namely LawDocTest or PatentDocTest. This process is visualized
in Figure 2 and yields six evaluation results R1-6 for each test set. For
example for LawDocTest, R3 is the in-domain evaluation result, whereas the
other results denote cross domain evaluations. For LawDocTest the results R1,
R3 and R5 are all from LawRNN document retrieval, but the LawRNNs differ in
the paragraph-level relevance representation they are trained with. Therefore
comparison of the results R1, R3 and R5 on LawDocTest shows the
transferability of the paragraph-level modelling between the legal and patent
domains and the difference of domain-specific paragraph-level modelling to the
non-domain specific modelling. Furthermore to analyze the cross-domain
transfer on the document-level, we compare the evaluation results of
LawDocTest and PatentDocTest of R1 and R2, R3 and R4 as well as R5 and R6.
This comparison shows the cross-domain transferability on the document-level
as the LawRNN and PatentRNN share the same paragraph-level relevance
representations, which they are trained on.

##  3 Experiments

###  3.1 Datasets

####  3.1.1 Legal retrieval dataset

Like Shao et al. [20], we use the legal retrieval collections from the COLIEE
evaluation campaign 2019 [17], which consist of a paragraph-level and a
document-level retrieval task. Both retrieval collections are based on cases
from the Canadian case law system and are written in English. The paragraph-
level task (COLIEE 2019 Task 2) involves the identification of a paragraph
which entails the given query paragraph [17]. For this task the COLIEE
evaluation campaign provides training and test queries with relevance
judgements which we will refer to as LawParaTrain and LawParaTest. In the
document-retrieval task (COLIEE 2019 Task 1) it is asked to find supporting
cases from a provided set of candidate documents, which support the decision
of the query document. As in the original paper we take $20\%$ of the queries
of the training set as validation set, denoted with LawDocVal. We will refer
to the training and test datasets for the document retrieval as LawDocTrain
and LawDocTest.

####  3.1.2 Patent retrieval dataset

For the patent retrieval queries and relevance judgments we use the datasets
from the CLEF-IP evaluation campaign [14] as they provide a patent corpus and
training and test collections for patent retrieval tasks on the paragraph- and
document-level. The tasks contain English, French and German queries, we only
consider the English queries and candidates. For the paragraph-level training
and test collection we choose the provided queries and relevance judgements
from the passage retrieval task starting from claims of the CLEF-IP 2013 [14]
where the participants are asked to find passages from patent documents which
are relevant to a given set of claims. We refer to these datasets as
PatentParaTrain and PatentParaTest. As the document-level training and test
collection we choose the queries and relevance judgements from the prior art
candidate search from the CLEF-IP evaluation campaign 2011 [15] and refer to
them as PatentDocTrain and PatentDocTest. As in the original paper, we take
$20\%$ of the training set as validation set, denoted with PatentDocVal. Both
patent retrieval tasks retrieve paragraphs and documents from the patent
corpus which consists of 3.5 million patent documents filed at the European
Patent Office (EPO) or at the World Intellectual Property Office (WIPO).
The dataset statistics can be found in Table 1.

Table 1: Statistics of the training and test set for the paragraph the document-level retrieval task | LawPara | LawDoc | PatentPara | PatentDoc
---|---|---|---|---
| Train | Test | Train | Test | Train | Test | Train | Test
# of queries | 181 | 41 | 285 | 61 | 44 | 42 | 351 | 100
avg # of candidates | 32.12 | 32.19 | 200 | 200 | 3.5M | 3.5M | 3.5M | 3.5M
avg # relevant candidates | 1.12 | 1.02 | 5.21 | 5.41 | 43.52 | 76.3 | 3.27 | 2.85

###  3.2 Experiment setting

####  3.2.1 Stage 1: BERT fine-tuning

In the first stage we fine-tune the BERT model111checkpoint from
https://github.com/google-research/bert on the paragraph-level relevance
classification for either the legal domain or the patent domain to attain
LawBERT and PatentBERT. As there was no code open-sourced for fine-tuning
BERT, we use the HuggingFace transformers
library222https://github.com/huggingface/transformers and add the BERT fine-
tuning script to the published code.
For LawBERT we use the LawParaTrain as training and LawParaTest as test
queries and relevance judgements. In order to use the queries and relevance
judgements for a binary classification task, we consider the paragraph pairs
of the query and one relevant candidate as positive samples. It was not stated
clearly in the original paper how the paragraph pairs of negative samples are
constructed, therefore we investigate this data pre-processing decision. We
find that taking all paragraph pairs constructed of the query and a non-
relevant paragraph from the paragraph candidates as negatives, yields
comparable results for fine-tuning the BERT model on the legal domain as in
the original paper. This negative sampling approach results in $3\%$ positive
and $97\%$ negative samples in the training set. The queries and paragraph
candidates have less than $100$ words on average and are truncated
symmetrically if they exceed the maximum input length of $512$ tokens of BERT.
For the training batch size we do a grid search and find that the F1-score of
LawParaTest is the highest with a batch size of $2$ ($65.1\%$ F1-Score)
instead of $1$ ($63.4\%$ F1-Score) after fine-tuning BERT for $3$ epochs on
LawParaTrain, contrary to the original paper: they report the highest F1-score
of $65.2\%$ without reporting the batch size. As stated in the original code,
we assumed they used the batch size of $1$, due to our comparison we use a
batch size of $2$ instead of $1$. After a remark of the original authors it
turns out the original implementation was done with a batch size of $16$. For
the learning rate we also do a grid search and find that the learning rate of
$1e-5$ is optimal as in the original paper. As in the original paper, we fine-
tune for $3$ epochs and we do the final fine-tuning of the LawBERT model on
the merged training and test set. This is permissible as we train and evaluate
the BERT-PLI model on LawDocTrain and LawDocTest, the LawParaTrain and
LawParaTest sets are only used for fine-tuning LawBERT.
For the PatentBERT fine-tuning we use the PatentParaTrain as training and
PatentParaTest as test set. We construct the negative paragraph pairs by
sampling randomly paragraphs (which are not the relevant paragraph) from the
documents which contain a relevant paragraph to a query paragraph. Here we
sample randomly $5$ times the number of positive paragraphs as negatives, as
otherwise the share of positive pairs is below $1\%$ and in order to have a
similar ratio as for the legal domain. We do a grid search for the training
batch size and learning rate and find that a batch size of $2$ with a learning
rate of $2e-5$ yields the highest F1-score of $19.0\%$. We fine-tune
PatentBERT solely on PatentParaTrain as it is common practice to hold out the
test set.

####  3.2.2 Stage 2: Document retrieval

In stage 2 the first step is to retrieve relevant documents from the given set
of candidates (in the legal domain) or from the whole corpus (in the patent
domain). As it was not clearly stated in the original paper nor was there code
published, how to employ the BM25 algorithm [18] for this first step, we re-
implement this step and use the BM25 algorithm [18] with $k_{1}=0.9$ and
$b=0.4$ implemented in the Pyserini
toolkit333https://github.com/castorini/pyserini. Furthermore we do a grid
search for the input length to the BM25 algorithm and find that the top $K=50$
retrieval with input length of $250$ leads to similar recall scores as the
original paper for the LawDocTrain set ($93.22\%$) and the LawDocTest
($92.23\%$). Here we only consider recall scores as in the original paper, as
the focus of the first step BM25 retrieval is to retrieve all relevant cases
for re-ranking for the training and test set.
For patent document retrieval, the task is to retrieve relevant documents from
the patent corpus with $3.5$ million documents. As in the patent document
retrieval task only $3.27$ relevant patent documents per query document are
contained and as the recall does not significantly increase when taking $K=50$
candidates, we choose the top $K=20$ from the BM25 retrieval, in order to have
a similar ratio of positive and negative pairs as in the legal document
retrieval for training the AttentionRNN. Here we find that the BM25 algorithm
with the document input length of $250$ reaches the highest recall score of
$9.42\%$ on PatentDocTrain compared to other document input lengths. Due to
the low recall score of the retrieved documents on PatentDocTrain we add the
relevant documents from the relevance judgements to PatentDocTrain and sample
randomly non-relevant documents from the BM25 candidates for the training
dataset, so that we have in total $20$ candidates. For PatentDocTest we
retrieve the top $50$ candidates as in the original implementation where we
reach a recall of $10.66\%$, but we do not add the relevant candidates after
the BM25 retrieval step. In order to reproduce the experiments for modelling
the paragraph-level interaction and training the Attention RNN, we use the
open-sourced repository444https://github.com/ThuYShao/BERT-PLI-IJCAI2020 of
the original paper. As in the original paper we set the number of paragraphs
of the query $N=54$ and the number of paragraphs of the candidate $M=40$ for
legal and patent retrieval. The query and candidate documents are split up in
paragraphs of $256$ tokens. We model the paragraph-level interactions of
LawDocTrain, LawDocTest, PatentDocTrain and PatentDocTest using LawBERT or
PatentBERT or $\text{BERT}_{\text{ORG}}$. With these paragraph-level
representations of each query and its candidate document we train an
AttentionRNN network with either an LSTM [9] or a GRU network [3] as RNN on
classifying the relevance between the query and candidate document. The
AttentionRNN trained on the LawDocTrain is denoted with LawRNN, on
PatentDocTrain it is denoted with PatentRNN. For training the AttentionRNN we
use the same hyperparameter as in the original implementation, except for the
PatentBERT LawRNN configuration, where we find that the learning rate of
$1e-4$ is better suited, when evaluated on the LawDocVal set.

##  4 Evaluation and Analysis

###  4.1 In-domain evaluation for legal document retrieval (RQ1)

Shao et al. [20] evaluate their models using the binary classification metrics
precision, recall and F1-Score on the whole test set. Furthermore they compare
their model performance to the two best runs from the COLIEE 2019 denoted by
the team names JNLP [22] and ILPS [19]. As it was not clearly stated in the
original paper, we assume that Shao et al. [20] evaluate the BERT-PLI models
on the whole LawDocTest set with all $200$ given candidates per query. With
the first retrieval step, the top $50$ query candidate pairs are retrieved for
binary classification, therefore we assume the lower $150$ candidates
classified as irrelevant. As in [20], we use a cutoff value of $5$ for the
evaluation of ranking algorithms like BM25, this means the top $5$ retrieved
documents are classified as relevant, whereas the remaining $195$ are
considered irrelevant.
As Shao et al. [20] evaluate in their published code the top $50$ candidates,
we investigate the overall evaluation of our reproduced BERT-PLI models for
all $200$ candidates with the precision, recall and F1-score using the
SciKitlearn classification report 555https://scikit-
learn.org/stable/modules/generated/sklearn.metrics.classification_report.html.
The results can be found in Table 2, we test the statistical significance
compared to the BM25 baseline with the Student’s paired, independent t-test
[21, 23]. Comparing the evaluation results stated in the original paper and
our evaluation results, we find that our reproduced BERT-PLI LawBERT LSTM and
GRU model reach similar values. On the effect of domain specific paragraph-
level modelling on the legal case retrieval task (RQ1), the original paper
reports a $7-9\%$ performance improvement for legal retrieval with the BERT-
PLI model, when BERT is fine-tuned on the legal paragraph-level modelling
compared to the original BERT. Contrary to that, we find that the domain
specific paragraph-level modelling does not appear to help the performance of
the legal case retrieval. Our reproduced $\text{BERT}_{\text{ORG}}$ LawRNN GRU
model outperforms all other BERT-PLI models except on the recall, however this
shows that contrary to the findings in the original paper, the domain specific
paragraph-level modelling does not always improve the performance of the BERT-
PLI model.

Table 2: Precision, Recall and F1-Score comparison of Shao et al. [20] and our reproduction, BM25 cutoff value of 5 as in [20], JNLP [22] and ILPS [19] denote the best two runs of the COLIEE 2019, † indicates statistically significant difference to BM25, $\alpha=0.05$ Team/Model | Precision | Recall | F1-Score
---|---|---|---
JNLP [22] | 0.6000 | 0.5545 | 0.5764
ILPS [19] | 0.68 | 0.43 | 0.53
$\text{BERT}_{\text{ORG}}$ LawRNN LSTM [20] | 0.5278 | 0.4606 | 0.4919
$\text{BERT}_{\text{ORG}}$ LawRNN GRU [20] | 0.4958 | 0.5364 | 0.5153
LawBERT LawRNN LSTM [20] | 0.5931 | 0.5697 | 0.5812
LawBERT LawRNN GRU [20] | 0.6026 | 0.5697 | 0.5857
Reproduction |  |  |
BM25 (cutoff at 5) | 0.5114 | 0.5360 | 0.5234
Repr $\text{BERT}_{\text{ORG}}$ LawRNN LSTM | $0.7053^{\dagger}$ | $0.5017^{\dagger}$ | $0.5863^{\dagger}$
Repr $\text{BERT}_{\text{ORG}}$ LawRNN GRU | $\textbf{0.8972}^{\dagger}$ | $0.4501^{\dagger}$ | $\textbf{0.5995}^{\dagger}$
Repr LawBERT LawRNN LSTM | $0.8620^{\dagger}$ | $0.4295^{\dagger}$ | $0.5733^{\dagger}$
Repr LawBERT LawRNN GRU | $0.3826^{\dagger}$ | $\textbf{0.6838}^{\dagger}$ | $0.4907^{\dagger}$

###  4.2 In-domain evaluation for patent document retrieval (RQ2)

In order to investigate the applicability of the BERT-PLI model for
information retrieval in the patent domain, we evaluate the PatentBERT
PatentRNN models trained on PatentDocTrain. The results can be found in Table
3, now we analyze the in-domain evaluation for the PatentBERT PatentRNN models
on PatentDocTest. This shows that the in-domain, patent BERT-PLI model is not
beneficial for patent document retrieval, as it is outperformed by the BM25
baseline on all metrics. We reason this could be due to the number of
considered query and candidate paragraphs ($N$ and $M$), which is fit to the
legal retrieval but not to the patent retrieval and could be unsuitable for
patent retrieval as PatentDocTrain and PatentDocTest contain on average more
paragraphs than LawDocTrain and LawDocTest. This demonstrates that the
document retrieval with contextualized language models for the patent domain
is not yet beneficial and needs to be taken under further investigation. In
line with the findings regarding RQ1 for the legal document retrieval, we find
that the paragraph-level modelling with the PatentBERT model impairs the
performance of the document retrieval compared to the paragraph-level
modelling with $\text{BERT}_{\text{ORG}}$. This shows that the domain specific
paragraph-level modelling is not always beneficial for BERT-PLI for the legal
and patent document retrieval.

Table 3: In-domain and cross-domain evaluation on the legal and patent document retrieval test set, in-domain evaluation for LawBERT LawRNN models on LawDocTest and PatentBERT PatentRNN on PatentDocTest, R1-6 denote the result numbers from Figure 2, † indicates statistically significant difference to BM25, $\alpha=0.05$ |  | LawDocTest | PatentDocTest
---|---|---|---
Model |  | Prec | Rec | F1 | Prec | Rec | F1
In-domain |  |  |  |  |  |  |
BM25 (cutoff at 5) |  | 0.5114 | 0.5360 | 0.5234 | 0.0500 | 0.3968 | 0.0888
LawBERT LawRNN (R3) | LSTM | $\textbf{0.8620}^{\dagger}$ | $0.4295^{\dagger}$ | $\textbf{0.5733}^{\dagger}$ | $0.0207^{\dagger}$ | $0.4761^{\dagger}$ | $0.0398^{\dagger}$
GRU | $0.3826^{\dagger}$ | $\textbf{0.6838}^{\dagger}$ | $0.4907^{\dagger}$ | $0.0181^{\dagger}$ | $0.4444^{\dagger}$ | $0.0349^{\dagger}$
PatentBERT PatentRNN (R6) | LSTM | $0.7500^{\dagger}$ | $0.2268^{\dagger}$ | $0.3482^{\dagger}$ | $0.0365^{\dagger}$ | $0.1904^{\dagger}$ | $0.0613^{\dagger}$
GRU | $0.1153^{\dagger}$ | $0.0412^{\dagger}$ | $0.0607^{\dagger}$ | $0.0416^{\dagger}$ | $0.1904^{\dagger}$ | $0.0683^{\dagger}$
Cross-domain |  |  |  |  |  |  |
LawBERT PatentRNN (R4) | LSTM | $0.1103^{\dagger}$ | $0.5292^{\dagger}$ | $0.1826^{\dagger}$ | $0.0277^{\dagger}$ | $0.1587^{\dagger}$ | $0.0472^{\dagger}$
GRU | $0.0961^{\dagger}$ | $0.2749^{\dagger}$ | $0.1424^{\dagger}$ | $0.0246^{\dagger}$ | $0.1904^{\dagger}$ | $0.0436^{\dagger}$
PatentBERT LawRNN (R5) | LSTM | $0.8000^{\dagger}$ | $0.4673^{\dagger}$ | $0.5900^{\dagger}$ | $0.0188^{\dagger}$ | $0.3650^{\dagger}$ | $0.0357^{\dagger}$
GRU | $0.5460^{\dagger}$ | $0.5704^{\dagger}$ | $0.5579^{\dagger}$ | $0.0233^{\dagger}$ | $0.5555^{\dagger}$ | $0.0448^{\dagger}$
BERTOrg PatentRNN (R2) | LSTM | $0.0000^{\dagger}$ | $0.0000^{\dagger}$ | $0.0000^{\dagger}$ | $0.0602^{\dagger}$ | $0.0793^{\dagger}$ | $0.0684^{\dagger}$
GRU | $0.0000^{\dagger}$ | $0.0000^{\dagger}$ | $0.0000^{\dagger}$ | $\textbf{0.0769}^{\dagger}$ | $0.0952^{\dagger}$ | $0.0851^{\dagger}$
BERTOrg LawRNN (R1) | LSTM | $0.7053^{\dagger}$ | $0.5017^{\dagger}$ | $0.5863^{\dagger}$ | $0.0160^{\dagger}$ | $\textbf{0.8095}^{\dagger}$ | $0.0314^{\dagger}$
GRU | $\textbf{0.8972}^{\dagger}$ | $0.4501^{\dagger}$ | $\textbf{0.5995}^{\dagger}$ | $0.0199^{\dagger}$ | $0.4285^{\dagger}$ | $0.0381^{\dagger}$

###  4.3 Cross-domain evaluation (RQ3)

In order to analyze the cross-domain retrieval between the legal and patent
domain, we evaluate each model on LawDocTest and PatentDocTest set as
illustrated in Figure 2 and compare for each test set the performance of the
different models in order to gain insights about the transferability of the
models between the legal and patent retrieval task and on the paragraph as
well as on the document-level.
Analyzing the cross-domain transfer on the paragraph-level for LawDocTest, we
see in Table 3 that the performance is similar for the LawRNNs when modelling
the paragraph-level interaction with PatentBERT instead of LawBERT. An
interesting result is the performance of the PatentBERT PatentRNN LSTM model,
which was not trained on modelling legal paragraph-interactions nor legal
document retrieval, but performs well on LawDocTest, however it does not
outperform the domain independent BM25 baseline. On the document-level we see
that the PatentRNN models have on average a $40\%$ lower F1-Score than the
LawRNN models with the same paragraph-level modelling, although we see a
positive effect of modelling the paragraph-level interactions with
$\text{BERT}_{\text{ORG}}$ instead of LawBERT or PatentBERT.
For the cross-domain evaluation on PatentDocTest, we find that each BERT-PLI
model is outperformed by the BM25 baseline, except for the precision of the
$\text{BERT}_{\text{ORG}}$ PatentRNN models and the recall of the
$\text{BERT}_{\text{ORG}}$ LawRNN models. On the document-level transfer we
see a consistent performance improvement of the PatentRNN models compared to
the LawRNN models independent of the paragraph-level modelling, which leads to
the conclusion that the domain specific training for patent document retrieval
is beneficial here. On a paragraph-level transfer we can see a similar
performance of the LawRNN models, independent of the paragraph-level
modelling. For the PatentRNN models we find that the paragraph-level modelling
with $\text{BERT}_{\text{ORG}}$ outperforms the modelling with PatentBERT and
LawBERT.

##  5 Related Work

There are numerous evaluation campaigns for patent [15] and legal retrieval
[2, 8, 17] with the goal to create and provide queries and relevance
judgements for domain-specific retrieval and with this promote research in
legal and patent IR. For legal retrieval, Cormack et al. [5] evaluate
continuous, simple active and passive learning models in the TREC legal
evaluation campaign [8] and propose an autonomous active learning framework
[4]. In the COLIEE evaluation campaign, Rossi et al. [19] combine text
summarization and a generalized language model to predict pairwise relevance
for the legal case retrieval task, whereas Tran et al. [22] apply a
summarization method and the extraction of lexical features. In the patent
retrieval evaluation campaign CLEF-IP [14], Piroi et al. [16] report different
approaches using the probabilistic BM25 model [18] as well as SVM-classifier
trained on pretrained word-level representations.
As the language model BERT [6] advanced the state-of-the-art in language
modeling, there are numerous approaches to apply BERT to IR tasks [12, 25] and
for cross-domain IR for web and news search [1] as well as for biomedical
search [13, 24].

##  6 Conclusion and Future Work

We reproduced the BERT-PLI model of Shao et al. [20] for the legal document
retrieval task of the COLIEE evaluation campaign 2019 [17]. We have addressed
shortcomings of the description of the data pre-processing and the second
stage retrieval, which we investigated and for which we complemented the
published code. Contrary to the original paper, we find that modelling the
paragraph-level interactions with a BERT model fine-tuned on the domain does
not appear to help the performance of the BERT-PLI model for document
retrieval compared to modelling the paragraph-level interactions with the
original BERT model. Furthermore we have analyzed the applicability of the
BERT-PLI model for document retrieval in the patent domain, but we find that
the BERT-PLI model does not yet improve the patent document retrieval compared
to the BM25 baseline. We reason that the optimal number of query and candidate
paragraphs to be considered for the interaction modelling could be a decisive
hyperparameter to take into account. However bringing the gains from
contextualized language model to patent document retrieval stays an open
problem. We have investigated to what extend the BERT-PLI model is
transferable between the legal and patent domain on the paragraph and
document-level by evaluating the cross-domain retrieval of the BERT-PLI model.
We show that the cross-domain transfer on the paragraph-level yields
comparable performance between the legal and the patent domain. Furthermore
the comparison on the document-level transfer shows first promising results
when applying the BERT-PLI model trained on the patent domain to the legal
domain. How to bring the benefits of contextualized language models to domain-
specific search and how to transfer retrieval models across different domains
remain open and exciting questions.

## Acknowledgements

This work was supported by the EU Horizon 2020 ITN/ETN on Domain Specific
Systems for Information Extraction and Retrieval (H2020-EU.1.3.1., ID:
860721).

## References

  * [1] Akkalyoncu Yilmaz, Z., Yang, W., Zhang, H., Lin, J.: Cross-domain modeling of sentence-level evidence for document retrieval. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). pp. 3490–3496. Association for Computational Linguistics, Hong Kong, China (Nov 2019). https://doi.org/10.18653/v1/D19-1352, https://www.aclweb.org/anthology/D19-1352
  * [2] Bhattacharya, P., Ghosh, K., Ghosh, S., Pal, A., Mehta, P., Bhattacharya, A., Majumder, P.: Fire 2019 aila track: Artificial intelligence for legal assistance. In: Proceedings of the 11th Forum for Information Retrieval Evaluation. p. 4–6. FIRE ’19, Association for Computing Machinery, New York, NY, USA (2019). https://doi.org/10.1145/3368567.3368587, https://doi.org/10.1145/3368567.3368587
  * [3] Cho, K., van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., Bengio, Y.: Learning phrase representations using RNN encoder–decoder for statistical machine translation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). pp. 1724–1734. Association for Computational Linguistics, Doha, Qatar (Oct 2014). https://doi.org/10.3115/v1/D14-1179, https://www.aclweb.org/anthology/D14-1179
  * [4] Cormack, G., Grossman, M.: Autonomy and reliability of continuous active learning for technology-assisted review (04 2015)
  * [5] Cormack, G.V., Grossman, M.R.: Evaluation of machine-learning protocols for technology-assisted review in electronic discovery. In: Proceedings of the 37th International ACM SIGIR Conference on Research and Development in Information Retrieval. p. 153–162. SIGIR ’14, Association for Computing Machinery, New York, NY, USA (2014). https://doi.org/10.1145/2600428.2609601, https://doi.org/10.1145/2600428.2609601
  * [6] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). pp. 4171–4186. Association for Computational Linguistics, Minneapolis, Minnesota (Jun 2019). https://doi.org/10.18653/v1/N19-1423, https://www.aclweb.org/anthology/N19-1423
  * [7] Gao, L., Dai, Z., Callan, J.: Modularized transfomer-based ranking framework. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (2020)
  * [8] Hedin, B., Zaresefat, S., Baron, J., Oard, D.: Overview of the trec 2009 legal track. In: The Eighteenth Text REtrieval Conference (TREC 2009) Proceedings (01 2009)
  * [9] Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (Nov 1997). https://doi.org/10.1162/neco.1997.9.8.1735, https://doi.org/10.1162/neco.1997.9.8.1735
  * [10] Hofstätter, S., Hanbury, A.: Let’s measure run time! Extending the IR replicability infrastructure to include performance aspects. In: Proc. of OSIRRC (2019)
  * [11] Hofstätter, S., Zlabinger, M., Hanbury, A.: Interpretable & Time-Budget-Constrained Contextualization for Re-Ranking. In: Proc. of ECAI (2020)
  * [12] Lee, K., Chang, M.W., Toutanova, K.: Latent retrieval for weakly supervised open domain question answering. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. pp. 6086–6096. Association for Computational Linguistics, Florence, Italy (Jul 2019). https://doi.org/10.18653/v1/P19-1612, https://www.aclweb.org/anthology/P19-1612
  * [13] MacAvaney, S., Cohan, A., Goharian, N.: SLEDGE-Z: A zero-shot baseline for COVID-19 literature search. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pp. 4171–4179. Association for Computational Linguistics, Online (Nov 2020). https://doi.org/10.18653/v1/2020.emnlp-main.341, https://www.aclweb.org/anthology/2020.emnlp-main.341
  * [14] Piroi, F., Lupu, M., Hanbury, A.: Overview of CLEF-IP 2013 lab. In: Forner, P., Müller, H., Paredes, R., Rosso, P., Stein, B. (eds.) Information Access Evaluation. Multilinguality, Multimodality, and Visualization. pp. 232–249. Springer Berlin Heidelberg, Berlin, Heidelberg (2013)
  * [15] Piroi, F., Lupu, M., Hanbury, A., Zenz, V.: CLEF-IP 2011: Retrieval in the intellectual property domain (01 2011)
  * [16] Piroi, F., Tait, J.: CLEF-IP 2010: Retrieval experiments in the intellectual property domain (2010)
  * [17] Rabelo, J., Kim, M.Y., Goebel, R., Yoshioka, M., Kano, Y., Satoh, K.: A summary of the COLIEE 2019 competition. In: Sakamoto, M., Okazaki, N., Mineshima, K., Satoh, K. (eds.) New Frontiers in Artificial Intelligence. pp. 34–49. Springer International Publishing, Cham (2020)
  * [18] Robertson, S., Zaragoza, H.: The probabilistic relevance framework: Bm25 and beyond. Found. Trends Inf. Retr. 3(4), 333–389 (Apr 2009). https://doi.org/10.1561/1500000019, https://doi.org/10.1561/1500000019
  * [19] Rossi, J., Kanoulas, E.: Legal information retrieval with generalized language models (2019)
  * [20] Shao, Y., Mao, J., Liu, Y., Ma, W., Satoh, K., Zhang, M., Ma, S.: BERT-PLI: Modeling paragraph-level interactions for legal case retrieval. In: Bessiere, C. (ed.) Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20. pp. 3501–3507. International Joint Conferences on Artificial Intelligence Organization (7 2020), main track
  * [21] Smucker, M.D., Allan, J., Carterette, B.: A comparison of statistical significance tests for information retrieval evaluation. In: Proceedings of the Sixteenth ACM Conference on Conference on Information and Knowledge Management. p. 623–632. CIKM ’07, Association for Computing Machinery, New York, NY, USA (2007). https://doi.org/10.1145/1321440.1321528, https://doi.org/10.1145/1321440.1321528
  * [22] Tran, V., Nguyen, M.L., Satoh, K.: Building legal case retrieval systems with lexical matching and summarization using a pre-trained phrase scoring model. In: Proceedings of the Seventeenth International Conference on Artificial Intelligence and Law. p. 275–282. ICAIL ’19, Association for Computing Machinery, New York, NY, USA (2019). https://doi.org/10.1145/3322640.3326740, https://doi.org/10.1145/3322640.3326740
  * [23] Urbano, J., Lima, H., Hanjalic, A.: Statistical significance testing in information retrieval: An empirical analysis of type i, type ii and type iii errors. In: Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. p. 505–514. SIGIR’19, Association for Computing Machinery, New York, NY, USA (2019). https://doi.org/10.1145/3331184.3331259, https://doi.org/10.1145/3331184.3331259
  * [24] Xiong, C., Liu, Z., Sun, S., Dai, Z., Zhang, K., Yu, S., Liu, Z., Poon, H., Gao, J., Bennett, P.: Cmt in trec-covid round 2: Mitigating the generalization gaps from web to special domain search. In: ArXiv preprint (2020)
  * [25] Yang, W., Xie, Y., Lin, A., Li, X., Tan, L., Xiong, K., Li, M., Lin, J.: End-to-end open-domain question answering with BERTserini. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations). pp. 72–77. Association for Computational Linguistics, Minneapolis, Minnesota (Jun 2019). https://doi.org/10.18653/v1/N19-4013, https://www.aclweb.org/anthology/N19-4013
  * [26] Zhang, Y., Nie, P., Geng, X., Ramamurthy, A., Song, L., Jiang, D.: Dc-bert: Decoupling question and document for efficient contextual encoding (2020)