# Advancing continual lifelong learning in neural information retrieval:
definition, dataset, framework, and empirical evaluation

Jingrui Hou<EMAIL_ADDRESS>Georgina Cosma<EMAIL_ADDRESS>Axel Finke
<EMAIL_ADDRESS>
###### Abstract

Continual learning refers to the capability of a machine learning model to
learn and adapt to new information, without compromising its performance on
previously learned tasks. Although several studies have investigated continual
learning methods for neural information retrieval (NIR) tasks, a well-defined
task definition is still lacking, and it is unclear how typical learning
strategies perform in this context. To address this challenge, a systematic
task definition of continual NIR is presented, along with a multiple-topic
dataset that simulates continuous information retrieval. A comprehensive
continual neural information retrieval framework consisting of typical
retrieval models and continual learning strategies is then proposed. Empirical
evaluations illustrate that the proposed framework can successfully prevent
catastrophic forgetting in neural information retrieval and enhance
performance on previously learned tasks. The results also indicate that
embedding-based retrieval models experience a decline in their continual
learning performance as the topic shift distance and dataset volume of new
tasks increase. In contrast, pretraining-based models do not show any such
correlation. Adopting suitable learning strategies can mitigate the effects of
topic shift and data augmentation in continual neural information retrieval.

###### keywords:

neural information retrieval , continual learning , catastrophic forgetting ,
topic shift , data augmentation

††journal: Information Sciences

[aff1]organization=Department of Computer Science, School of Science,
Loughborough University,addressline=Epinal Way, city=Loughborough,
postcode=LE11 3TU, state=Leicestershire, country=UK
[aff2]organization=Department of Mathematical Sciences, School of Science,
Loughborough University,addressline=Epinal Way, city=Loughborough,
postcode=LE11 3TU, state=Leicestershire, country=UK

##  1 Introduction

Information retrieval, a fundamental area of research within natural language
processing, focuses on identifying and extracting relevant information from a
collection of documents [1]. Information retrieval technologies have led to
the development of various real-world applications, including search engines,
question-answering systems, and e-commerce recommender platforms [2]. With the
advancement of deep learning, neural network-based information retrieval
approaches known as neural information retrieval (NIR) [1] have exhibited
superior results. Earlier NIR models were primarily based on word embedding
methods, while more recent NIR models [3] have utilised pretrained language
models (PLMs) [4] for improved performance.

Most deep learning models are trained using a fixed dataset, known as the
isolated learning paradigm [5], and NIR is no exception. However, in real-
world information retrieval applications and systems, new information is
constantly arriving. Taking a document retrieval system as an example, the
internal neural ranking model trained on old information may struggle to
accurately sort and recommend the latest content, as new articles are
constantly injected into the system over time, potentially leading to
suboptimal user experiences due to the presence of different topics in the new
articles. One common approach to addressing this issue is by retraining a new
model each time novel data emerge. However, this practice can be considered
impractical due to its low efficiency and high resource consumption [6]. As a
consequence, the continual learning paradigm strives to facilitate the
seamless integration of new data streams into neural networks as they become
available [7, 8, 9]. The main challenge in continual learning is to avoid
catastrophic forgetting, i.e. the problem that deep learning models tend to
inadvertently forget previously acquired knowledge when learning on new data
[10].

Most existing continual learning strategies are designed for classification
tasks [11]. There has been limited focus on continual learning in information
retrieval [12, 13, 14], and it is not yet clear how existing learning
strategies perform in this field of information retrieval. Hence, the primary
objective of this study is to bridge this research gap. Moreover, the
performance of continual learning can be affected by two important factors:
data volume and topic shift. Data volume, i.e. the size of the new dataset,
can lead to varying degrees of catastrophic forgetting as the amount of data
increases [15]. On the other hand, topic shift, i.e. dissimilarity between new
tasks and previously learned tasks, can also impact continual learning
performance [16]. However, the effect of these two factors on continual
learning in information retrieval tasks has not been investigated. Therefore,
this study aims to fill this gap.

Considering the underutilization of continual learning strategies in
information retrieval tasks and a lack of understanding regarding the impact
of data volume and topic shift on continual learning performance within this
domain, this study proposes multiple research objectives. The primary
objective is to design a framework that incorporates typical continual
learning strategies into multiple NIR methods. The second objective involves
determining the most effective continual learning strategies tailored to
different NIR models. Lastly, this study investigates how variations in data
volume and topic shift influence continual NIR performance.

However, it is important to note that, as of the present time, a well-defined
mathematical task definition and a benchmark dataset for NIR in a continual
learning context are still absent. This absence poses a challenge to achieving
the aforementioned research objectives effectively. In this paper, significant
groundwork is laid out, which includes a precise task definition, the
establishment of a benchmark dataset, and the development of a comprehensive
continual NIR framework. These foundational elements serve as prerequisites
for the empirical evaluation of the proposed research objectives. In summary,
the key contributions of this paper are as follows:

  1. 1.

A precise definition of the continual learning paradigm in the context of NIR
tasks, addressing the lack of a standardized definition tailored to continual
learning in this domain.

  2. 2.

A new dataset, called Topic-MSMARCO, specifically designed for evaluating
continual NIR tasks. Topic-MSMARCO includes multiple thematic NIR tasks and
predefined task similarity, filling the gap in the availability of evaluation
benchmarks tailored to continual learning in NIR.

  3. 3.

A novel framework named Continual Learning Framework for Neural Information
Retrieval (CLNIR). By allowing for the pairing of different models and
strategies, CLNIR offers a flexible and customizable approach to continual
NIR, generating a wide array of continual learning methods. This framework
provides researchers with a comprehensive baseline for further exploration.

  4. 4.

An empirical evaluation of the performance of different continual learning
strategies for various NIR models, as well as an investigation into the impact
of topic shift distance and data volume on continual NIR performance.

The remainder of the paper is organized as follows: Section 2 provides an
overview of existing continual learning strategies and their attempted
application in the field of NIR, thus laying the foundations for the proposed
research. Section 3 outlines the task definition and the evaluation metrics
used in this study. Section 4 presents the proposed CLNIR and its
architecture. Section 5 details the proposed Topic-MSMARCO dataset and the
experiments on evaluating the two research objectives. Lastly, Section 6
concludes this study and provides suggestions for future work.

##  2 Related work

This section provides an overview of existing research on continual learning
strategies and their application to information retrieval tasks.

###  2.1 Continual learning strategies

Continual learning can be classified into three categories: regularization-
based, replay-based, and parameter isolation methods [11].

Regularization-based strategies primarily employ penalty terms in loss
functions to control parameter updates. A well-known example of this approach
is Elastic Weight Consolidation (EWC) [17], which slows down learning on
certain neural parameter weights based on their importance to previous tasks.
Liu et al. [18] enhanced EWC with a reparameterization method utilizing the
Fisher information matrix of the network parameters. Another algorithm in this
category is Synaptic Intelligence (SI) [19], which computes per-synapse
consolidation strength in an online fashion over the entire learning
trajectory in parameter space. Moreover, Chaudhry et al. [20] further combined
EWC and SI, and the fused method demonstrated superior continual learning
performance. To prevent important knowledge related to previous tasks from
being overwritten, the Memory Aware Synapses (MAS) strategy [21] stores
important information in a separate memory during training on new tasks and
uses a regularization term to encourage the model to learn new information
without disrupting the important information stored in the memory module.
Additionally, Mazur et al. [22] introduced a regularization strategy utilizing
the Cramer-Wold distance between target layer distributions to represent both
current and past information. This approach effectively addresses the problem
of increasing memory consumption commonly encountered in similar methods. In
order to diminish the upper bound of forgetting in regularization-based
methods, Kong et al. [23] investigated the geometric properties of the
approximation loss of prior tasks concerning the maximum eigenvalue. For
pretrained NLP tasks, Zhang et al. [24] introduced the lifelong language
method with adaptive uncertainty regularization, enabling a single BERT model
to dynamically adapt to various tasks without performance degradation.

Replay-based approaches typically store samples from old tasks and retrain
them in new tasks. One example of this approach is the incremental classifier
and representation learning [25], which uses a herding-based step for
prioritized exemplar selection and only requires a small number of exemplars
per class. Another replay-based method is Gradient Episodic Memory (GEM) [26],
which uses a subset of observed examples to minimize negative backward
transfer. Zhuang et al. [27] have proposed a continual learning technique that
selects samples by combining various criteria such as distance to prototype,
intra-class cluster variation, and classifier loss. Another type of replay is
experience replay. One typical method is continual learning with experience
and replay (CLEAR) [28], which employs on-policy learning on fresh experiences
to adapt rapidly to new tasks while utilizing off-policy learning with
behavioural cloning on replay experience to maintain and modestly enhance
performance on past tasks. Li et al. [29] further proposed the adaptive
experience replay (AdaER), which enhances existing experience replay by
introducing a novel contextually-cued recall strategy for the replay stage and
entropy-balanced reservoir sampling for the update stage.

Parameter isolation methods allocate dynamic network parameters to incremental
data or tasks. One example of this approach is the progressive neural network
[30], which prevents catastrophic forgetting by instantiating a new neural
network for each task being solved. Another parameter isolation method is the
expert gate [31] which addresses continual learning issues by adding a new
expert model when a new task arrives or knowledge is transferred from previous
models. Furthermore, in dynamic graph learning, the parameter isolation graph
neural network, introduced by Zhang et al. [32], is designed for continual
learning on dynamic graphs, overcoming tradeoffs through parameter isolation
and expansion.

###  2.2 Attempts of continual learning in information retrieval

A limited number of studies have explored the application of continual
learning in information retrieval tasks. Wang et al. [33] and Song et al. [34]
demonstrated the impact of continual learning methods on multi-modal embedding
spaces in cross-modal retrieval tasks. In the text modality, Lovón-Melgarejo
et al. [12] investigated the catastrophic forgetting problem in neural ranking
models for information retrieval and their experiments revealed that the
effectiveness of neural ranking comes at the cost of forgetting. However,
transformer-based models provided a good balance between effectiveness and
memory retention; and EWC [17] effectively mitigated catastrophic forgetting
while maintaining a good trade-off between multiple tasks. Gerald and Soulier
[13] built a sequential information retrieval dataset using the MSMARCO
dataset [35] with sentence embedding and evaluated two pretraining-based NIR
models: Vanilla BERT [4] and MonoT5 [36]. Gerald and Soulier [13] also found
that catastrophic forgetting does exist in information retrieval, but to a
lesser extent compared to some classification tasks. Furthermore, in the
general retrieval scenario, Chen et al. [14] proposed the continual-learner
for generative retrieval (CLEVER). CLEVER introduces Incremental Product
Quantization for encoding new documents and a memory-augmented learning
mechanism to retain previous knowledge. However, this method only considers
the emergence of new documents.

Several studies have explored continual learning in NIR from different
perspectives. Nevertheless, a precise definition of continual NIR and
strategies compatible with multiple NIR methods is still lacking. This study
aims to fill this research gap.

##  3 Continual NIR task definition and evaluation metrics

This section introduces the task definition of continual learning in NIR,
followed by a description of the metrics used to evaluate the task.

###  3.1 Task definition

A continual neural information retrieval task is a specific type of continual
learning task that involves a model that is able to retrieve relevant
information from a continuously growing set of queries and documents.

  * 1.

Let $T\in\mathbb{N}$ be the total number of tasks.

  * 2.

For each task $t\in\\{1,\dotsc,T\\}$, let $Q_{t}$ be a set of queries and let
$D_{t}$ be a set of documents.

  * 3.

For each task $t\in\\{1,\dotsc,T\\}$, let $S_{t}$ be a (training) dataset for
NIR, i.e., for some finite number of samples $n_{t}$,
$S_{t}=\\{(x_{t}^{i},y_{t}^{i})\\}_{i=1}^{n_{t}}$, where

    * (a)

$x_{t}^{i}\coloneqq(q_{t}^{i},d_{t}^{i})\in Q_{t}\times D_{t}$ is the $i$th
query–document pair for the $t$th task;

    * (b)

$y_{t}^{i}\in\mathbb{R}$ is the ground-truth relevance score associated with
this query–document pair.

Relevance scores can be either a discrete value (“relevant” or “irrelevant”)
or a continuous value (degree of relevance, e.g. between 0 and 1).

  * 4.

A (training) dataset for continual NIR is then given by
$\mathcal{S}\coloneqq\\{S_{1},\dotsc,S_{T}\\}$.

The continual neural information retrieval task requires parameters $\theta$
of a neural information retrieval model $\mathcal{R}(\theta)$ to be learned
from the sequence of training sets in $\mathcal{S}$, one at a time. Starting
with some initial parameters $\theta_{0}$. When a new training set $S_{t}$ is
fed into $\mathcal{R}_{t-1}\coloneqq\mathcal{R}(\theta_{t-1})$, the parameter
$\theta_{t-1}$ is updated to $\theta_{t}$ by learning the samples in $S_{t}$,
yielding the new model $\mathcal{R}_{t}\coloneqq\mathcal{R}(\theta_{t})$.

###  3.2 Performance evaluation

Let
$\widetilde{\mathcal{S}}\coloneqq\\{\widetilde{S}_{1},\dotsc,\widetilde{S}_{T}\\}$
be a (sequential) test data set (for NIR). That is, for each
$t\in\\{1,\dotsc,T\\}$,
$\widetilde{S}_{t}=\\{(\tilde{x}_{t}^{i},\tilde{y}_{t}^{i})\\}_{i=1}^{\tilde{n}_{t}}$,
again contains a finite number of tuples of query–document pairs
$\tilde{x}_{t}^{i}=(\tilde{q}_{t}^{i},\tilde{d}_{t}^{i})\in Q_{t}\times D_{t}$
and associated (ground-truth) relevance scores
$\tilde{y}_{t}^{i}\in\mathbb{R}$.

The model performance may then be evaluated as follows. When training on the
$t$th training set $S_{t}$ is completed, the updated model $\mathcal{R}_{t}$
is applied to the query–document pairs
$\tilde{x}_{s}^{i}=(\tilde{q}_{s}^{i},\tilde{d}_{s}^{i})$ in the $s$th test
set, for all $s\in\\{1,\dotsc,T\\},$ to yield a predicted relevance score,
$\mathcal{R}_{t}(\tilde{x}_{s}^{i})$.

Finally, classical information retrieval metrics such as mean reciprocal
recall or mean average precision then compute performance scores $P_{t,s}$
based on the true relevance scores $\tilde{y}_{s}^{i}$ and the predicted
scores $\smash{\mathcal{R}_{t}(\tilde{x}_{s}^{i})}$. The choice of evaluation
metric is flexible and depends on the specific retrieval scenario. Some
possible choices are discussed later.

Figure 1 provides an example of a continual neural information retrieval task.
It illustrates how the model adapts to a continuously growing set of queries
and documents and continually records its performance on multiple test sets
over time.

Figure 1: An example diagram of continual neural information retrieval with
three tasks ($T=3$). The neural information retrieval model is initially
trained on the training set of task 1 and then tested on all three test sets
to generate $P_{1,1}$, $P_{1,2}$, and $P_{1,3}$. After training on the set for
task 2, $P_{2,1}$, $P_{2,2}$, and $P_{2,3}$ will be generated. Finally, upon
completion of task 3, the model will produce $P_{3,1}$, $P_{3,2}$, and
$P_{3,3}$.

###  3.3 Evaluation metrics

The average evaluation performance of
$\mathcal{R}_{T}=\mathcal{R}(\theta_{T})$ (the retrieval model after learning
all tasks) is

$P_{\mathrm{final}}\coloneqq\frac{1}{T}\sum_{t=1}^{T}P_{T,t}.$ (1)

To further explore the fine-grained continual learning performance, this study
introduces backward transfer (BWT) and forward transfer (FWT) [26] into the
information retrieval scenario.

The backward transfer score, $\mathit{BWT}$, measures the influence that
learning a task has, on average, on the performance of previous tasks:

$\mathit{BWT}\coloneqq\frac{2}{T(T-1)}\sum_{t=2}^{T}\sum_{s=1}^{t}(P_{t,s}-P_{s,s}).$
(2)

A negative score, $\mathit{BWT}<0$, indicates the average performance
reduction of the previous test sets on newly trained tasks. Conversely,
$\mathit{BWT}\geq 0$ indicates, on average, an absence of catastrophic
forgetting.

The forward transfer score, $\mathit{FWT}$, measures the influence that
learning a task has on the performance of future tasks:

$\mathit{FWT}\coloneqq\frac{2}{T(T-1)}\sum_{t=1}^{T-1}\sum_{s=t+1}^{T}P_{t,s}.$
(3)

A large value of $\mathit{FWT}$ indicates that, on average, the model performs
well on test sets of unlearned tasks; a small value of $\mathit{FWT}$
indicates that the model is not able to effectively transfer knowledge from
previous tasks to new ones.

##  4 Proposed CLNIR framework architecture

This section introduces the proposed Continual Learning Framework for Neural
Information Retrieval (CLNIR). Inspired by a prior continual learning
framework for classification tasks [37], the CLNIR framework allows for
flexibility by decoupling the selection and implementation of the NIR model
from the choice of learning strategy. The architecture of the proposed CLNIR
framework is depicted in Figure 2. It consists of a learning strategy pool, a
neural ranking model pool, and an agent model that performs NIR tasks.

Figure 2: The CLNIR framework architecture.The NIR models and learning
strategies mentioned will be explained in Sections 4.2 and 4.3 respectively.

###  4.1 Agent model

Algorithm 1 shows the workflow of the CLNIR framework. Line 1 initialises the
agent model of the CLNIR framework with a designated NIR model (ranker
$\mathcal{R}$), learning strategy ($\mathcal{L}$) and a loss function
($\ell$). The neural parameters within $\mathcal{R}$ are the optimization
target of $\ell$. When new training samples are fed into CLNIR, $\mathcal{R}$
first learns the new data, similar to non-incremental learning tasks (Line 5).
The CLNIR framework allows for two types of learning strategies:

  * 1.

For regularization-based strategies ($\mathcal{L}\in\mathit{regularization}$),
Line 7 lets the agent model generate a new regularization item, $\Omega_{t}$,
which measures the importance of the different components of $\theta_{t}$.
Line 8 then updates the loss function to account for the modified
regularization term. A number of suitable regularization-based strategies will
be described in Section 4.3.1.

  * 2.

For replay-based strategies ($\mathcal{L}\in\mathit{replay}$), Line 10 first
initializes an empty memory set, $\mathbf{M}$, when processing the first task.
Line 11 then updates the memory set from the current training set using
certain replay rules. For certain replay-based strategies, the loss function
or the training set for the next task then need to be updated based on
$\mathbf{M}$ (Lines 12–13). A number of suitable replay-based strategies will
be described in Section 4.3.2.

After the training process, Line 14 updates the agent model; Lines 15–18 then
send all test sets into the updated agent model to generate performance
scores.

input : continual NIR agent model $\mathcal{A}$;

neural ranker $\mathcal{R}$ with initial parameter $\theta_{0}$;

learning strategy $\mathcal{L}$;

loss function $\ell$;

sequential NIR (training and test) datasets $\mathcal{S}$ and
$\widetilde{\mathcal{S}}$ with $T$ tasks;

1

output : $\mathit{BWT}$; $\mathit{FWT}$; $P_{\mathrm{final}}$;

2 initialize agent model:
$\mathcal{A}_{0}\leftarrow\mathcal{A}(\mathcal{R}(\theta_{0}),\mathcal{L})$;

3 initialize $U$ as an empty list;

4 initialize $V$ as an empty list;

5 for  _$t\leftarrow 1$ to $T$_ do

6 $\theta_{t}\leftarrow$ Train(_$\mathcal{A}_{t-1}$ , $S_{t}$_);

7 if  _$\mathcal{L}\in\mathit{regularization}$_ then

8

9 $\Omega_{t}\leftarrow$
CalculateParameterImportance(_$S_{t},\theta_{t},\Omega_{t-1},\mathcal{L}$_);

10 $\ell\leftarrow$ UpdateLossFunction(_$\ell$ , $\Omega_{t}$_);

11

12 if  _$\mathcal{L}\in\mathit{replay}$_ then

13 if  _$t=0$_ then initialize memory set $\mathbf{M}$ an empty list;

14 $\mathbf{M}\leftarrow$ UpdateTaskMemory(_$S_{t}$ , $\mathbf{M}$_);

15 $\ell\leftarrow$ UpdateLossFunction(_$\ell$ , $\mathbf{M}$_);

16 $S_{t+1}\leftarrow$ UpdateNextTrainingData(_$S_{t}$ , $\mathbf{M}$_);

17

18 update agent model:
$\mathcal{A}_{t}\leftarrow\mathcal{A}(\mathcal{R}(\theta_{t}),\mathcal{L})$;

19 for  _$s\leftarrow 1$ to $T$_ do

20 $P_{t,s}\leftarrow$ Test(_$\mathcal{A}_{t}$ , $\widetilde{S}_{s}$_);

21 if  _$s\leq t$_ then append $P_{t,s}$ to $U$;

22 else append $P_{t,s}$ to $V$;

23

24

Algorithm 1 Continual neural information retrieval

###  4.2 NIR model pool

The proposed CLNIR framework consists of three state-of-the-art embedding-
based methods and two pretraining-based methods. However, researchers who
adopt the proposed framework can include other models.

####  4.2.1 Embedding based NIR models

DRMM. Deep Relevance Matching Models (DRMMs) [38] are interaction-based NIR
models. Their architecture is depicted in Figure 3(a). Given a query text $q$
and a candidate document $d$, the DRMM model tokenizes the texts into fixed-
length tokens $\mathbf{w}_{q}$ and $\mathbf{w}_{d}$, and generates
corresponding masks $\mathbf{m}_{q}$ and $\mathbf{m}_{d}$. The tokens are then
input into an embedding layer, and the resulting embedding matrices are masked
by $\mathbf{m}_{q}$ and $\mathbf{m}_{d}$. The produced matrices for $q$ and
$d$ can be represented as $\mathbf{E}_{q}\in\mathbb{R}^{\mathit{l}(q)\times
n}$ and $\mathbf{E}_{d}\in\mathbb{R}^{\mathit{l}(d)\times n}$, respectively,
where $\mathit{l}(q)$ and $\mathit{l}(d)$ represent the lengths of the query
and document, respectively, and $n$ is the embedding dimension. The query
representation is obtained by dividing $\mathbf{E}_{q}$ by its matrix norm,
while the document representation is obtained from $\mathbf{E}_{d}$ in the
same manner. Both representation matrices have the same dimensions as their
corresponding embedding matrices.

The relevance between query representation and document representation is
determined based on the cosine matrix similarity and tensor histogram. The
cosine matrix
$M_{\mathrm{cos}}\in\mathbb{R}^{\mathit{l}(q)\times\mathit{l}(d)}$ is obtained
by taking the product of query representation and the transpose of document
representation. Then, each row in $M_{\mathrm{cos}}$ is sorted into equal
width ($b$) by a histogram function, resulting in a transformed matrix
$M_{\mathrm{his}}\in\mathbb{R}^{\mathit{l}(q)\times b}$, where $b$ is the
number of bins. Next, a series of linear and hyperbolic tangent (Tanh) layers
are used to compress $M_{\mathrm{his}}$ and $\mathbf{E}_{q}$ into a relevance
score vector $M_{\mathrm{rel}}\in\mathbb{R}^{\mathit{l}(q)}$ and a query gate
vector $M_{\mathrm{gate}}\in\mathbb{R}^{\mathit{l}(q)}$, respectively.
Finally, the relevance score is obtained by summing the element-wise product
of $M_{\mathrm{rel}}$ and $M_{\mathrm{gate}}$.

KNRM. The architecture of Kernel-based Neural Ranking Models (KNRMs) [39] is
shown in Figure 3(b). Similar to DRMM, their input includes tokens
$\mathbf{w}_{q}$ and $\mathbf{w}_{d}$, and masks $\mathbf{m}_{q}$ and
$\mathbf{m}_{d}$. Then tokens $\mathbf{w}_{q}$ and $\mathbf{w}_{d}$ are sent
into an embedding layer to acquire query and document embeddings
$\mathbf{E}_{q}$ and $\mathbf{E}_{d}$, which have the same shape as embedding
matrices of DRMM. After normalization, $\mathbf{E}_{q}$ and
$\mathbf{E}_{d}^{\mkern-1.5mu\mathsf{T}}\in\mathbb{R}^{n\times\mathit{l}(d)}$
are multiplied to an interaction matrix
$M_{\mathrm{inter}}\in\mathbb{R}^{\mathit{l}(q)\times\mathit{l}(d)}$.

The $k$-kernel pooling function is then applied to $M_{\mathrm{inter}}$ to
create a kernel pooling tensor
$M_{\mathrm{pool}}\in\mathbb{R}^{\mathit{l}(q)\times\mathit{l}(d)\times k}$,
where $k$ is the kernel size. The $M_{\mathrm{pool}}$ is multiplied with
$\mathbf{m}_{d}$ to block unnecessary information, and then reduced to two
dimensional $M^{\prime}_{\mathrm{pool}}\in\mathbb{R}^{\mathit{l}(q)\times k}$
by summing the elements in the second dimension. The tensor
$M^{\prime}_{\mathrm{pool}}$ is then multiplied with $\mathbf{m}_{q}$ to block
further unnecessary information and reduced to one dimensional
$M^{\prime\prime}_{\mathrm{pool}}\in\mathbb{R}^{k}$ by summing the elements in
the first dimension. Finally, $M^{\prime\prime}_{\mathrm{pool}}$ is processed
by a feed-forward and sigmoid layer to produce a final relevance score.

DUET. The Duet [40] architecture consists of both a local model and a
distributed model. The local model uses a local representation to match the
query and document, while the distributed model uses learned distributed
representations for the same purpose. The architecture of Duet is shown in
Figure 3(c). In contrast to DRMM and KNRM, Duet includes an additional term
frequency-inverse document frequency (TF-IDF) module to initialize a local
distribution model
$M_{\mathrm{local}}\in\mathbb{R}^{\mathit{l}(q)\times\mathit{l}(d)}$. Each
element $w_{i}^{j}\in M_{\mathrm{local}}$ represents the TF-IDF score for the
$i$th term in $q$ and the $j$th word in $d$ if the two terms are the same. The
TF-IDF matrix is then flattened to one dimension, a vector
$M^{\prime}_{\mathrm{local}}\in\mathbb{R}^{n}$ for the local representation.

In parallel, the embeddings of the query
$\mathbf{E}_{q}\in\mathbb{R}^{\mathit{l}(q)\times n}$ and document
$\mathbf{E}_{d}\in\mathbb{R}^{\mathit{l}(d)\times n}$ undergo one-dimensional
convolution, max-pooling, and other tensor operations to derive the query
representation $M_{q}\in\mathbb{R}^{n}$ and document representation
$M_{d}\in\mathbb{R}^{n\times c}$, respectively, where $c$ represents the
output channel dimension. The product of $M_{q}$ and $M_{d}$ is then added to
$M^{\prime}_{\mathrm{local}}$ to generate a relevance score.

(a) DRMM

(b) KNRM

(c) DUET

Figure 3: Architectures of word embedding based NIR models.

####  4.2.2 Pretraining based NIR models

BERTdot. BERT with Dot Production (BERTdot) [41] is a direct application of
the BERT model [4]. Its architecture is shown in Figure 4(a). A BERT tokenizer
processes the query text $q$ and document text $d$ to produce tokens and
masks, which are then input into the BERT encoding layer to generate text
representation vectors. BERTdot uses the pooled vectors for query and document
representation, denoted as $\mathbf{B}_{q}\in\mathbb{R}^{n}$ and
$\mathbf{B}_{d}\in\mathbb{R}^{n}$, where $n$ is typically set to 768, the
default hidden size of BERT. To calculate the dot product between these
representations, they are expanded to $\mathbf{B}_{q}\in\mathbb{R}^{1\times
n}$ and $\mathbf{B}_{d}\in\mathbb{R}^{n\times 1}$. The resulting output vector
is then condensed to a scalar to represent the relevance score.

ColBERT. The Contextualized Late Interaction over BERT (ColBERT) model [42]
encodes queries and documents separately and calculates the query-document
similarity using a late interaction architecture, as shown in Figure 4(b). In
contrast to BERTdot, the query and document representations,
$\mathbf{B}_{q}\in\mathbb{R}^{\mathit{l}(q)\times n}$ and
$\mathbf{B}_{d}\in\mathbb{R}^{\mathit{l}(d)\times n}$, respectively, are two-
dimensional tensors encoded from the BERT architecture. The query
representation and the transposed document representation are multiplied to
obtain an interaction matrix
$M_{\mathrm{inter}}\in\mathbb{R}^{\mathit{l}(q)\times\mathit{l}(d)}$. The
maximum value in the last dimension of $M_{\mathrm{inter}}$ is then taken to
represent the query term similarity score
$M_{q}\in\mathbb{R}^{\mathit{l}(q)}$. Finally, all the similarity scores are
summed to generate the final relevance score.

(a) BERTdot

(b) ColBERT

Figure 4: Architectures of pre-trained NIR models.

###  4.3 Learning strategies pool

In a real-world retrieval scenario, the amount of emerging data is vast
(potentially infinite), making it unrealistic to employ parameter isolation
strategies that demand a separate parameter space for every new dataset. While
replay-based methods also face this challenge, it is possible to establish a
fixed memory space of adequate size. Regularization-based strategies, on the
other hand, are generally not influenced by the number of tasks, therefore,
the learning strategy pool considered in this study encompasses both the
widely used regularization-based and replay-based methods.

####  4.3.1 Regularization strategies for NIR

Let $\mathbf{x}=\langle\mathbf{q},\mathbf{pos},\mathbf{neg}\rangle$ be a batch
of pairwise ranking samples for the current model input, where $\mathbf{q}$
represents a batch of queries, $\mathbf{pos}$ and $\mathbf{neg}$ represent
either two document batches relevant and irrelevant to $\mathbf{q}$,
respectively, or two documents where the former is more relevant to
$\mathbf{q}$ than the latter. Regularization-based strategies aim to control
parameter updates with a penalty item in the loss function. The resulting
penalized loss function is:

$\ell(\mathbf{x};\theta)=\ell^{\prime}(\mathbf{x};\theta)+\lambda\sum_{i}(\theta^{i}-\vartheta^{i})^{2}\times\Omega^{i},$
(4)

where $\lambda>0$ is the importance coefficient; $\vartheta=(\vartheta^{i})$
is the current value of the parameter $\theta=(\theta^{i})$;
$\Omega=(\Omega^{i})$ is a regularization item obtained from the previous task
to measure the importance of the $i$th parameter in $\theta$. Then, to train
$x$ pairwise, a margin ranking loss is used for loss calculation. That is,
$\ell^{\prime}(\mathbf{x};\theta)$ is the margin ranking loss function,
defined as:

$\displaystyle\ell^{\prime}(\mathbf{x};\theta)=\max\\{0,-y(\mathcal{R}(\mathbf{q},\mathbf{pos};\theta)-\mathcal{R}(\mathbf{q},\mathbf{neg};\theta))+\mathit{margin}\\}.$
(5)

Here, $\mathcal{R}(\mathbf{q},\mathbf{pos};\theta)$ calculates the relevance
scores between $\mathbf{q}$ and $\mathbf{pos}$, while
$\mathcal{R}(\mathbf{q},\mathbf{neg};\theta)$ calculates the scores between
$\mathbf{q}$ and $\mathbf{neg}$; $\mathit{margin}$ is a parameter to control
the minimal difference between two ranking scores; and $y$ is a constant
value. The difference between various regularization strategies lies in the
addition of distinct regularization items in the loss function. The following
are some methods used to implement these regularization items.

L2. The L2 strategy is a direct application of L2-regularization. The
parameter importance is set to a constant to keep every parameter equal in
importance:

$\Omega^{i}=1.$ (6)

EWC and EWCol. The Elastic Weight Consolidation (EWC) [17] strategy calculates
the parameter importance based on the parameter gradient with the Fisher
information matrix. After all samples in a training set $S_{t}$ have been
learned, a subset $S_{t}^{*}\subseteq S_{t}$ of $K$ training samples, called
Fisher samples, is chosen to build a so-called Fisher information matrix. Then
$S_{t}^{*}$ can be organized pairwise and the accumulated gradients generated
from pairwise samples are used as parameter importance. The $k$th Fisher
sample is denoted as $x_{k}^{*}=\langle
q_{k}^{*},pos_{k}^{*},neg_{k}^{*}\rangle$. The average square of the gradients
of each parameter is used as the regularization item for the next task. That
is, if $\vartheta$ is the current parameter, set the regularisation item to

$\Omega^{i}=\frac{1}{K}\sum_{k=1}^{K}\mathbf{g}^{i}(x_{k}^{*};\vartheta)^{2},$
(7)

where

$\mathbf{g}^{i}(x;\vartheta)=\frac{\partial\ell^{\prime}(x;\vartheta)}{\partial\vartheta^{i}}.$
(8)

For Elastic Weight Consolidation in the Online Paradigm (EWCol), the
previously accumulated importance factors will be added to the new ones:

$\Omega^{i}\leftarrow\Omega^{i}+\frac{1}{K}\sum_{k=1}^{K}\mathbf{g}^{i}(x_{k};\vartheta)^{2}.$
(9)

SI. The Synaptic Intelligence (SI) [19] strategy uses an importance measure to
reflect the past credits for improvements of the regularized loss function for
each parameter. Before training on a new batch $\mathbf{x}_{k}$, given the
current parameter value $\vartheta_{k-1}$, with the convention that
$\vartheta_{0}=\vartheta$ is the parameter learned at the end of the previous
task and $\vartheta_{K}=\theta$ is the new parameter at the end of the current
task. Then SI updates the parameter from $\vartheta_{k-1}$ to $\vartheta_{k}$
by learning $\mathbf{x}_{k}$ with a regularized loss function. An importance
measure is used to accumulate the importance of each parameter, which is
calculated as the product of the gradient and the distance between the two
parameters:

$\omega^{i}=-\sum_{k}\mathbf{g}^{i}(x_{k};\vartheta_{k-1})\times(\vartheta_{k}^{i}-\vartheta_{k-1}^{i}).$
(10)

The regularization item is then updated as

$\Omega^{i}\leftarrow\Omega^{i}+\frac{\omega^{i}}{(\theta^{i}-\vartheta^{i})^{2}+\xi},$
(11)

where $\xi$ is a damping factor to avoid denominators becoming zero. It should
be noted that $\vartheta=\vartheta_{0}$ and $\theta=\vartheta_{K}$ are the
parameters at the end of the previous and current tasks, respectively.

MAS. The Memory Aware Synapses (MAS) [21] computes the importance of each
parameter in the network by measuring the sensitivity of the predicted output
function to changes in that parameter. After completing the training process
for the previous task, the resulting parameter $\theta$ is obtained. Before
starting the next training process, the MAS method reuses all the pairwise
training samples and accumulates the gradients of the square difference of
relevance scores in each pair with respect to $\theta$. For the $k$th batch,
$\mathbf{x}_{k}=\langle\mathbf{q}_{k},\mathbf{pos}_{k},\mathbf{neg}_{k}\rangle$
in the current task, the process can be denoted as follows:

$\Omega^{i}\leftarrow\Omega^{i}+\frac{1}{K}\sum_{k=1}^{K}\biggl{\lvert}\frac{\partial(\mathcal{R}(\mathbf{q}_{k},\mathbf{pos}_{k};\theta)-\mathcal{R}(\mathbf{q}_{k},\mathbf{neg}_{k};\theta))^{2}}{\partial\theta^{i}}\biggr{\rvert}.$
(12)

####  4.3.2 Replay strategies for NIR

NR. The Naive Rehearsal (NR) is a direct replay method. Before training a new
task, previously accumulated samples will be added to the new training set.
The new training set, $\mathbf{W}_{t}$ is a union of the current training set
and the previous memory sets:

$\displaystyle\mathbf{W}_{t}\coloneqq
S_{t}\cup\mathbf{M},\quad\text{with}\quad\mathbf{M}\coloneqq\bigcup_{s=1}^{t}\mathbf{M}_{s}^{t-1},$
(13)

where $\mathbf{M}_{s}^{t}$ represents a set of samples from the $s$th task
that are used for training in the $t$th task.

To ensure that the total memory size,
$\\#\bigcup_{s=1}^{t}\mathbf{M}_{s}^{t-1}$ remains consistent across different
tasks, samples are regularly removed so that
$\mathbf{M}_{s}^{1}\supseteq\mathbf{M}_{s}^{2}\supseteq\dotsb\supseteq\mathbf{M}_{s}^{t-1}$.

GEM. The Gradient Episodic Memory (GEM) [26] strategy still uses a memory set
to alleviate catastrophic forgetting. Unlike NR, the memory set in GEM is used
for the update of the gradient for new task samples. Before training a new
batch $\mathbf{x}_{k}$, GEM calculates the gradients of all memory samples
with respect to the current parameter value $\vartheta_{k-1}$ in a set-wise
manner.

Let
$\mathbf{G}_{\mathbf{M}}=(\mathbf{g}_{\mathbf{M}}^{1},\mathbf{g}_{\mathbf{M}}^{2},\dotsc,\mathbf{g}_{\mathbf{M}}^{t-1})^{\mkern-1.5mu\mathsf{T}}$,
where $\mathbf{g}_{\mathbf{M}}^{s}$ is the gradient of the $s$th memory set.
Then $\mathbf{x}_{k}$ will be learned and the corresponding parameter is
updated to $\vartheta_{k}$, and $\mathbf{g}_{k}$ represents the new gradient.

The angle between $\mathbf{g}_{k}$ and all memory gradients will be used as a
constraint to control parameter update, shown as:

$\mathbf{g}_{k}^{\mkern-1.5mu\mathsf{T}}\mathbf{g}_{\mathbf{M}}^{s}\geq
0,\text{ for all }s\textless t.$

If all constraints are satisfied, the $\mathbf{g}_{k}$ will continue being
used as the gradient for the next sample. However, if constraints are
violated, a gradient projection will be used to alleviate forgetting. The new
gradient $\widetilde{\mathbf{g}_{k}}$ replaces $\mathbf{g}_{k}$:

$\widetilde{\mathbf{g}_{k}}=\mathbf{g}_{\mathbf{M}}^{\mkern-1.5mu\mathsf{T}}\times
v^{*}+\mathbf{g}_{k},$ (14)

where $v^{*}$ is the vector of the result of the dual quadratic program:

$\displaystyle\underset{v}{\text{minimize}}$
$\displaystyle\frac{1}{2}v^{\mkern-1.5mu\mathsf{T}}\mathbf{g}_{\mathbf{M}}\mathbf{g}_{\mathbf{M}}^{\mkern-1.5mu\mathsf{T}}v+\mathbf{g}_{k}^{\mkern-1.5mu\mathsf{T}}\mathbf{g}_{\mathbf{M}}^{\mkern-1.5mu\mathsf{T}}v$
subject to $\displaystyle v\geq 0.$

##  5 Experiments

In the absence of a benchmark dataset tailored for evaluating continual
learning in neural information retrieval, this section is initiated with
dataset preparation, followed by the description of the experiment setup.
Subsequently, the results of the evaluation of continual learning performance
are presented, alongside an examination of the impact of data volume and topic
shift. Finally, the key findings are summarized.

###  5.1 Topic-MSMARCO: a new dataset for continual NIR

To assess the performance of methods for continual NIR, a NIR dataset with
multiple subsets that simulates the emergence of real-world data is necessary.
MSMARCO [35], one of the most important and largest benchmark datasets for
NIR, was selected to derive several subsets based on the topic distribution.
The ‘Passage Ranking’ dataset of MSMARCO is an information retrieval dataset
comprising real-world questions and answers from a search engine.

A recent study [43] concluded that using a combination of word2vector and
clustering models results in improved clustering outcomes. This methodology
was applied in the study. In this study, only query texts were utilized to
derive the topic distribution. The word2vector model was employed to train
word embeddings on the cleaned query corpus. By incorporating the embedded
query terms, a sequence of clustering experiments using the KMeans algorithm
was executed, encompassing diverse cluster numbers. The optimal cluster
configuration was then manually selected based on perplexity evaluations.
Subsequently, any noisy clusters were expunged from consideration, and
clusters exhibiting analogous thematic content were amalgamated to enhance
coherence and interpretability. The created dataset is referred to as Topic-
MSMARCO. As shown in Table 1, it encompasses six tasks (t0 to t5), each with a
distinct topic. The pairwise topic distances (see Figure 5) are defined as the
squared distances between the respective cluster centres.

Figure 5: Pairwise task distances in Topic-MSMARCO. A larger value denotes a
larger semantic distance. Table 1: Metadata and representative query terms of
Topic-MSMARCO.

Topic index | Number of queries | Theme | Representative query terms |
---|---|---|---|---
t0 | 64,719 | IT |  | upload, emails, gmail, ios, icloud, cisco, photo,
---
powerpoint, audio, instagram
t1 | 40,501 | Furnishing |  | doors, fiberglass, cement, drywall, fireplace,
---
siding, stair, dryer, porch, ceramic
t2 | 51,002 | Food |  | broccoli, cabbage, canned, hamburger, fried,
---
carrots, greens, peas, dried, tuna,
t3 | 47,646 | Health |  | painful, sleeping, vomiting, ache, weak, coughing,
---
canine, uterus, lip, dysfunction, thrush, chronic,
osteoarthritis, cell, mitochondria, subcutaneous,
narrow, forehead,
t4 | 12,204 | Tourism |  | tampa, nashville, philadelphia, mall, downtown, rio,
---
denver, veniceniagara, cape, hills metro, zoo,
city, yellowstone
t5 | 46,326 | Finance |  | employers, qualified, orders, settlement, fraud,
---
resident, reimbursement, charges, pays, residency,
commission, officers

###  5.2 Experimental setup

Single NIR task evaluation. As described in the last paragraph in Section 3.1,
different NIR tasks require different evaluation metrics. In this work, the
performance score on a single test set is calculated by mean reciprocal rank
(MRR), as it is an officially recommended metric from the MSMARCO
team111https://microsoft.github.io/msmarco/. If $Q$ is the set of queries (for
which the test set includes relevance scores), the MRR is defined as:

$\mathit{MRR}=\frac{1}{\\#Q}\sum_{q\in Q}\frac{1}{rank_{q}},$ (15)

where $\mathrm{rank}_{q}$ is the rank of the first relevant item for query
$q$; $\\#Q$ is the number of queries in $Q$.

Hardware and implementation. The hardware environment for the experiments was
an Nvidia RTX 3080 GPU workstation with an Intel I7-10 700K CPU and 64GB of
RAM. All neural network models in CLNIR were implemented using
PyTorch222https://pytorch.org/. The word embedding vectors used were GloVe
[44], and 300-dimensional word vectors were used as the initial weight for the
embedding layer of DRMM, KNRM, and DUET. The PLM used for ColBERT and BERTdot
is described in Wolf et al. [45].

###  5.3 Evaluating CLNIR using the topic-MSMARCO dataset

The first experiment aims to compare the continual learning performance of
various combinations of neural retrieval models and learning strategies within
the CLNIR framework proposed in Section 4.

To eliminate the potential impact of inconsistent sample sizes in each task,
$10\,000$ queries were randomly selected as the training set and $2\,000$
queries were randomly selected as the test set for each task. The CLNIR
framework was used to pair neural ranking models with learning strategies,
resulting in 35 continual retrieval methods. Five fine-tuning methods (without
a learning strategy) were considered as baselines.

####  5.3.1 Average final performance

Table 2 shows the final performance score for the final average performance,
$P_{\mathrm{final}}$, for each possible combination of NIR model and learning
strategy under the CLNIR framework as well as for the baseline methods.

Table 2: Final performance scores $P_{\mathrm{final}}$ defined in Section 3.3 ($\pm$ standard error). Best performing strategies for each model are highlighted in bold. Strategy | Model
---|---
DRMM | KNRM | DUET | BERTdot | ColBERT
Baseline | 0.074±0.002 | 0.173±0.003 | 0.129±0.002 | 0.237±0.003 | 0.322±0.003
L2 | 0.086±0.002 | 0.185±0.003 | 0.147±0.002 | 0.240±0.003 | 0.331±0.003
EWC | 0.082±0.002 | 0.185±0.003 | 0.221±0.003 | 0.227±0.003 | 0.300±0.003
EWCol | 0.085±0.002 | 0.185±0.003 | 0.222±0.003 | 0.210±0.003 | 0.295±0.003
SI | 0.080±0.002 | 0.185±0.003 | 0.223±0.003 | 0.228±0.003 | 0.299±0.003
MAS | 0.080±0.002 | 0.183±0.003 | 0.181±0.003 | 0.244±0.003 | 0.325±0.003
NR | 0.081±0.002 | 0.174±0.003 | 0.186±0.003 | 0.265±0.003 | 0.346±0.003
GEM | 0.061±0.002 | 0.134±0.002 | 0.144±0.002 | 0.241±0.003 | 0.312±0.003

DRMM group. Despite the lower MRR scores compared with other NIR models, some
continual learning strategies lead to performance improvement. Without
employing any learning strategy, the final performance score is relatively
low. However, with the L2 and EWCol, it increases noticeably. The EWC, MAS,
NR, and SI strategies also have a positive effect on the final performance.
However, the GEM strategy does not offer any performance improvement in the
DRMM group, leading to a lower final performance compared to the baseline.

KNRM group. The L2, EWC, EWCol, SI, and MAS strategies yield higher scores,
while the NR strategy performs similarly to the baseline. The sole strategy
that adversely affects the average final performance is GEM, which is even
lower than the baseline.

DUET group. Continual learning strategies exhibit the most significant
improvement in the DUET group. Notably, this group is the only one where all
pair combinations of NIR models with various learning strategies outperform
the baseline method. Particularly, the average final performance of EWC,
EWCol, and SI represents the largest improvement among all NIR groups.

BERTdot group. Both replay strategies (NR, GEM) outperform the baseline
method. The highest score in this group is achieved by NR, which is 0.028
higher than the baseline. It is worth noting that among all the
regularization-based strategies, only MAS and L2 show a slight improvement
over the baseline score, while the two EWC strategies and SI result in lower
scores compared to the baseline.

ColBERT group. In this experiment, ColBERT outperforms the other models. All
strategies except for EWCol yield an average final performance score higher
than 0.3. Similar to the BERTdot group, NR achieves the highest score. Both
EWC strategies and GEM demonstrate inferior performance compared to the
baseline, while L2 slightly improves upon the baseline.

####  5.3.2 Backward-transfer performance

BWT performance of baseline methods. As demonstrated in Table 3, all the
baseline methods experienced catastrophic forgetting as all their BWT scores
are negative. The BWT scores of the DUET and BERTdot models are a few
hundredths below zero, indicating that they are more susceptible to forgetting
without learning strategies, compared to the remaining NIR models whose BWT
scores are a few thousandths under zero.

Impact of learning strategies on BWT. As shown in Table 3, when combined with
learning strategies, all NIR models achieved nonnegative BWT scores,
indicating that NIR models can perform better on BWT with learning strategies.
The DUET model achieved the highest BWT scores with the EWC, EWCol and SI
strategies, at around 0.02. ColBERT showed a higher degree of adaptability to
diverse learning strategies, with only one of its BWT scores less than zero
when combined with learning strategies.

Learning strategies comparison on BWT. With regards to the learning
strategies, according to Table 3, the SI strategy is observed to achieve the
most positive BWT scores, outperforming the other strategies. Both the MAS and
NR strategies have two positive BWT scores on two pretraining-based models. In
contrast, the L2 and two EWC strategies yield only one positive score each.
Notably, the GEM strategy stands out as the sole strategy unable to overcome
catastrophic forgetting.

Table 3: BWT scores $\mathit{BWT}$ defined in Section 3.3 ($\pm$ standard error). Best performing strategies for each model are highlighted in bold. Strategy | Model
---|---
DRMM | KNRM | DUET | BERTdot | ColBERT
Baseline | -0.009±0.001 | -0.007±0.001 | -0.037±0.002 | -0.012±0.002 | -0.002±0.002
L2 | -0.003±0.001 | 0.000±0.000 | -0.020±0.002 | -0.010±0.002 | 0.004±0.002
EWC | -0.001±0.001 | 0.000±0.000 | 0.018±0.001 | 0.000±0.000 | 0.000±0.000
EWCol | 0.000±0.000 | 0.000±0.000 | 0.019±0.001 | -0.013±0.001 | 0.000±0.000
SI | 0.000±0.000 | 0.000±0.000 | 0.017±0.001 | 0.001±0.000 | 0.001±0.000
MAS | -0.007±0.001 | -0.002±0.001 | -0.006±0.002 | 0.001±0.000 | 0.003±0.002
NR | -0.004±0.001 | -0.007±0.001 | -0.004±0.002 | 0.008±0.002 | 0.008±0.002
GEM | -0.013±0.001 | -0.028±0.001 | -0.032±0.002 | -0.010±0.002 | -0.006±0.002

####  5.3.3 Forward-transfer performance

Impact of learning strategies on NIR models. Table 4 indicates that the FWT
scores of the NIR groups that pretraining-based NIR models outperform the
embedding-based ones, consistent with their average final performance. ColBERT
paired with the NR strategy achieves the highest FWT score. Similarly, the
combination of BERTdot and NR also demonstrates notable performance within the
BERTdot group. In the DUET group, the EWCol strategy exhibits the most
effective FWT performance, followed closely by EWC and SI. Within the KNRM and
DRMM groups, all strategies except for GEM either maintain parity with or
slightly enhance the FWT performance compared to the baseline strategy.

Comparison of FWT between baselines and learning strategies. According to
Table 4, among all learning strategies, GEM stands out as the sole strategy
that consistently reduces baseline FWT scores to varying degrees in both the
DRMM and KNRM groups. Notably, within the groups featuring pretraining-based
NIR models, several learning strategies exhibit a negative impact on FWT
performance. Specifically, in the BERTdot group, two strategies (EWC and SI)
yield lower FWT scores compared to the baseline, while in the ColBERT group,
the NR strategy stands out as the only one surpassing the baseline method.

Table 4: FWT scores $\mathit{FWT}$ defined in Section 3.3 ($\pm$ standard error). Best performing strategies for each model are highlighted in bold. Strategy | Model
---|---
DRMM | KNRM | DUET | BERTdot | ColBERT
Baseline | 0.070±0.001 | 0.160±0.002 | 0.142±0.002 | 0.217±0.002 | 0.292±0.002
L2 | 0.074±0.001 | 0.164±0.002 | 0.147±0.002 | 0.216±0.002 | 0.289±0.002
EWC | 0.073±0.001 | 0.164±0.002 | 0.165±0.002 | 0.206±0.002 | 0.275±0.002
EWCol | 0.074±0.001 | 0.164±0.002 | 0.167±0.002 | 0.222±0.002 | 0.271±0.002
SI | 0.070±0.001 | 0.164±0.002 | 0.165±0.002 | 0.206±0.002 | 0.274±0.002
MAS | 0.072±0.001 | 0.162±0.002 | 0.154±0.002 | 0.223±0.002 | 0.289±0.002
NR | 0.074±0.001 | 0.160±0.002 | 0.153±0.002 | 0.226±0.002 | 0.300±0.002
GEM | 0.063±0.001 | 0.156±0.002 | 0.151±0.002 | 0.218±0.002 | 0.287±0.002

####  5.3.4 Overall comparison

Most effective learning strategies for evaluating Topic-MSMARCO. From the
previous comparisons of the final average performance, BWT and FWT, it can be
observed that all five NIR models benefit from learning strategies. NIR models
combined with the appropriate learning strategies always outperform the
baseline methods in any evaluation metric. Notably, the two pretraining-based
NIR models exhibit significant performance enhancement when paired with the NR
strategy, showcasing the most effective performance across all three metrics
within their respective groups. However, the optimal learning strategies for
different embedding-based NIR models vary. Detailed insights into the most
effective learning strategies for different NIR models and evaluation metrics
are provided in Table 5.

Table 5: Most effective learning strategies for different NIR models on
different evaluation metrics.

Metric | Model
---|---
DRMM | KNRM | DUET | BERTdot | ColBERT
$P_{\mathrm{final}}$ |  L2/EWCol |  L2/EWC/EWCol/SI/MAS |  EWC/EWCol/SI | NR | NR
$\mathit{BWT}$ |  EWCol/SI |  L2/EWC/EWCol/SI |  EWC/EWCol/SI | NR | NR
$\mathit{FWT}$ |  L2/EWC/EWCol/NR |  L2/EWC/EWCol/SI/MAS |  EWC/EWCol/SI | NR | NR

To provide further insight into the effectiveness of learning strategies, the
performance scores of three metrics on the five NIR models are depicted in
Figure 6. For ease of comparison, the combination of NIR models and their best
corresponding learning strategies will be referred to as “CLNIR best”.

(a) Average final MRR

(b) Forward transfer ability

(c) Backward transfer ability

Figure 6: Performance comparison between baseline and CLNIR best.

###  5.4 Relationship between data volume/topic shift and continual NIR

This discusses the relationship between the performance of continual
information retrieval on the one hand and topic shift and data volume on the
other.

####  5.4.1 Topic shift

Aim and methodology of the experiment. The experiment aims to explore the
impact of topic shift distances on the performance of NIR tasks in continual
learning. The methodology is as follows:

  * 1.

Topic t1 (“furnishing”) was selected as the starting point because it has the
smallest mean standard deviation among the other topics, resulting in the most
uniform topic distances according to Figure 5.

  * 2.

The remaining topics (t0, t2–t5) were learned in parallel as the second
topics. Therefore, five sub-experiments were conducted, each with two topics.

  * 3.

The NIR models used for the experiment were KNRM and ColBERT, as their
baseline methods performed the best among embedding-based and pretraining-
based models, respectively.

  * 4.

The continual learning strategy used was the SI strategy, as it achieved the
most non-negative FWT scores in Table 3.

Analysing the topic shift in the KNRM group. According to Figure 7(a), the
baseline KNRM model without a learning strategy shows that as the topic
distance increases, the MRR score decreases. The Pearson correlation
coefficient between topic distances and MRR scores is -0.953. However, when
combined with the SI strategy, the decrease in performance is less severe
because the performance curve of the SI strategy has a slower decline rate
compared to the baseline. This implies that the SI strategy can effectively
mitigate the issue of catastrophic forgetting while learning a new task,
especially when the topic distance is greater than 4.9. In such cases, the MRR
scores generated with the SI strategy are both more stable and higher than
those of the baseline methods.

(a) Topic shift phenomenon of KNRM model

(b) Topic shift phenomenon of ColBERT model

Figure 7: Impact of topic shift on continual learning performance. The
“benchmark” refers to the results obtained prior to learning the second task;
the “baseline” represents the results of NIR models without implementing any
learning strategies; and “SI” refers to the results of models that have been
combined with SI strategy.

Analysing the topic shift in the ColBERT group. Figure 7(b) demonstrates that
ColBERT produced different results compared to the KNRM group. The MRR
performance curve for the baseline ColBERT displays fluctuations as the topic
distance increases, resulting in a general decline in MRR scores. On the other
hand, the MRR curve for the ColBERT combined with SI shows a narrower range of
fluctuation, however, the MRR scores are lower than those of the baseline
ColBERT. This suggests that ColBERT does not consistently show a decrease in
MRR scores with increasing topic distance. Furthermore, when combined with the
SI strategy, the majority of the MRR scores are lower than the baseline.

####  5.4.2 Data volume

Aim and methodology of the experiment. This experiment aims to investigate the
relationship between data volume and performance. The methodology for
conducting this experiment is described below:

  * 1.

KNRM and ColBERT were chosen as the tested NIR models and the SI strategy was
employed. The reasons for these choices are as previously stated.

  * 2.

The initial task was selected to be “tourism” (t4) and the subsequent task was
“food” (t2), as these two topics were deemed to have the greatest disparity
among all topic pairs.

  * 3.

The influence of varying training sample sizes on model performance was
assessed by setting the sample size for the second task in a flexible manner.
Considering that the sample size for t2 is approximately five times larger
than that of t4, the sample size for the second task was established to range
from 0.1 to 1 times, incrementing by 0.1 at each interval, and then from 1 to
5 times with an interval of 1.

Analysing the data volume in the KNRM group. The relationship between data
volume and the performance of KNRM methods is demonstrated in Figure 8(a). It
has been observed that in the baseline KNRM method, there exists a negative
correlation between data volume and continual learning performance, where the
data volume of the new task is one time larger. This correlation is measured
using the Pearson correlation coefficient of -0.987. On the other hand, the
SI-combined KNRM method exhibits more stable results, indicating that the SI
strategy effectively reduces the impact of data volume. It should be noted
that both KNRM methods exhibit an upward trend when the sample size of the
subsequent task is smaller than the initial task, which suggests that
embedding-based models can still benefit from small-scale new tasks even when
there is a significant topic distance.

Analysing the data volume in the ColBERT group. The results for the ColBERT
group are depicted in Figure 8(b). It is shown that the SI-combined ColBERT
method provides more stable performance compared to the baseline ColBERT
method, but it failed to achieve higher scores. Both methods have local high
points when the sample size of the subsequent task is one time smaller, which
is similar to the findings in the KNRM group. However, the baseline ColBERT
method demonstrates a weaker correlation with data volume, with a Pearson
correlation coefficient of 0.490. This is further evidenced by the fluctuating
performance curve.

(a) Results of the KNRM group

(b) Results of the ColBERT group.

Figure 8: Impact of data volume on continual learning performance

###  5.5 Key findings

Three experiments were conducted to evaluate the performance of the proposed
CLNIR framework and reveal the relationships between continual learning
performance, topic shift, and data volume.

Findings about the continual learning performance of CLNIR. The findings of
the first experiment showed that pretraining-based NIR models outperformed
embedding-based models in terms of average final performance and forward
transfer performance. The forward transfer performance rank order was similar
to that of the final average performance. For pretraining-based NIR models,
the NR strategy was found to be the most effective, while the best strategy
for embedding-based NIR models varied. However, in some instances, the GEM-
combined NIR models performed worse than the baseline approaches. These
results demonstrate the effectiveness of the proposed CLNIR framework in
mitigating the catastrophic forgetting phenomenon and improving continual
learning performance in NIR tasks.

Findings about topic shift and data augmentation. In the second and third
experiments, it was observed that both topic shift distance and data
augmentation have a negative correlation with continual learning performance
when using the embedding-based baseline method. As the topic shift distance or
data volume increases, the continual learning performance tends to decrease.
However, the use of a learning strategy such as SI can mitigate this trend.
This pattern is also evident in the pretraining-based baseline methods, where
MRR scores fluctuate with the increment of topic shift distance or data
volume, but models combined with SI are consistently more stable than those
baseline models. While learning strategies can address the issue of
catastrophic forgetting caused by topic shifts or data augmentation, they may
also diminish the performance improvements caused by a new task.

##  6 Conclusion

This paper presents a task definition and novel dataset for continual neural
information retrieval tasks and introduces a framework called CLNIR for
addressing the lack of continual learning ability in current NIR models. The
CLNIR includes several typical NIR models and learning strategies. Experiments
were conducted to evaluate the performance of different pair combinations of
NIR models and learning strategies in the framework. The results show that the
best learning strategies for different NIR models vary, but NIR models
combined with a suitable learning strategy can always outperform the baseline
methods where no learning strategy is used. Additionally, the impact of topic
shift and data augmentation has also been investigated through experiments. It
has been found that learning strategies can mitigate the impact of topic shift
and data augmentation, leading to more stable performance. In conclusion, the
proposed CLNIR framework can effectively deal with the catastrophic forgetting
problem in NIR. The proposed framework can even improve the performance of old
tasks by learning new tasks.

The primary limitations of this study include the scope of continual learning
strategies employed. Specifically, the investigation was restricted to seven
strategies based predominantly on regularization and replay mechanisms,
excluding other approaches such as optimization-based and architecture-based
strategies [9]. Moreover, the six thematic tasks utilized in this research all
originate from MS MARCO, a question-answering retrieval dataset. This singular
data source may not fully represent the performance capabilities of the
framework across diverse new information retrieval tasks, such as recommender
systems. Lastly, due to computational constraints, a shared bi-encoder was
used for both queries and documents within the experiments. Using other NIR
architectures, such as cross-encoder or phrase-level representation [3], may
lead to subtly different experimental observations.

Future research directions can address the current study’s limitations and
expand the utility of the CLNIR framework in several ways:

  * 1.

Development of novel continual learning strategies. The continual learning
strategies in the CLNIR framework are adapted from other domains. This study
analyzed their performance on different NIR models and concluded that no
single strategy outperforms others across all models. Therefore, future
efforts could concentrate on crafting innovative strategies custom-designed to
tackle the unique challenges of information retrieval.

  * 2.

Expansion to multimodal information retrieval. With the advent of large-scale
multimodal models [46], the capabilities for multimodal information retrieval
have been significantly enhanced. The current framework, which operates solely
on text modality, is poised for expansion into multimodal information
retrieval. Future studies can further explore this potential, assessing how
the CLNIR framework can incorporate and leverage various modalities such as
images, videos, and audio for more comprehensive retrieval tasks.

## Data availability

The data described in this paper was obtained from the public repository
available at https://microsoft.github.io/msmarco/. To reorganize the dataset
and reproduce the experiments, please refer to the paper’s GitHub repository
located at https://github.com/JingruiHou/CLNIR.

## Acknowledgments

Jingrui Hou was supported by funding from CSC (China Scholarship Council,
No.202208060371) and Loughborough University.

## References

  * Ceri et al. [2013] S. Ceri, A. Bozzon, M. Brambilla, E. Della Valle, P. Fraternali, S. Quarteroni, An Introduction to Information Retrieval, Springer, Berlin, Heidelberg, 2013, pp. 3–11. doi:10.1007/978-3-642-39314-3_1.
  * Guo et al. [2020] J. Guo, Y. Fan, L. Pang, L. Yang, Q. Ai, H. Zamani, C. Wu, W. B. Croft, X. Cheng,  A deep look into neural ranking models for information retrieval,  Inf. Process. Manage. 57 (2020) 102067. doi:10.1016/j.ipm.2019.102067.
  * Zhao et al. [2024] W. X. Zhao, J. Liu, R. Ren, J.-R. Wen,  Dense text retrieval based on pretrained language models: A survey,  ACM Trans. Inf. Syst. 42 (2024). doi:10.1145/3637870.
  * Devlin et al. [2019] J. Devlin, M. Chang, K. Lee, K. Toutanova,  BERT: pre-training of deep bidirectional transformers for language understanding,  in: J. Burstein, C. Doran, T. Solorio (Eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Association for Computational Linguistics, 2019, pp. 4171–4186. doi:10.18653/v1/n19-1423.
  * Liu [2017] B. Liu,  Lifelong machine learning: a paradigm for continuous learning,  Front. Comput. Sci. 11 (2017) 359–361. doi:10.1007/s11704-016-6903-6.
  * Harun et al. [2023] M. Y. Harun, J. Gallardo, T. L. Hayes, C. Kanan,  How efficient are today’s continual learning algorithms?,  in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 2430–2435. doi:10.1109/CVPRW59228.2023.00241.
  * Pratama and Wang [2019] M. Pratama, D. Wang,  Deep stacked stochastic configuration networks for lifelong learning of non-stationary data streams,  Inf. Sci. 495 (2019) 150–174. doi:10.1016/j.ins.2019.04.055.
  * Biesialska et al. [2020] M. Biesialska, K. Biesialska, M. R. Costa-jussà,  Continual lifelong learning in natural language processing: A survey,  in: Proceedings of the 28th International Conference on Computational Linguistics, International Committee on Computational Linguistics, Barcelona, Spain (Online), 2020, pp. 6523–6541. doi:10.18653/v1/2020.coling-main.574.
  * Wang et al. [2024] L. Wang, X. Zhang, H. Su, J. Zhu,  A comprehensive survey of continual learning: Theory, method and application,  IEEE Trans. Pattern Anal. Mach. Intell. (2024) 1–20. doi:10.1109/TPAMI.2024.3367329.
  * French [1999] R. M. French,  Catastrophic forgetting in connectionist networks,  TRENDS COGN. SCI. 3 (1999) 128–135. doi:10.1016/S1364-6613(99)01294-2.
  * Lange et al. [2022] M. D. Lange, R. Aljundi, M. Masana, S. Parisot, X. Jia, A. Leonardis, G. G. Slabaugh, T. Tuytelaars,  A continual learning survey: Defying forgetting in classification tasks,  IEEE Trans. Pattern Anal. Mach. Intell. 44 (2022) 3366–3385. doi:10.1109/TPAMI.2021.3057446.
  * Lovón-Melgarejo et al. [2021] J. Lovón-Melgarejo, L. Soulier, K. Pinel-Sauvagnat, L. Tamine,  Studying catastrophic forgetting in neural ranking models,  in: D. Hiemstra, M.-F. Moens, J. Mothe, R. Perego, M. Potthast, F. Sebastiani (Eds.), Advances in Information Retrieval, Springer International Publishing, Cham, 2021, pp. 375–390. doi:10.1007/978-3-030-72113-8_25.
  * Gerald and Soulier [2022] T. Gerald, L. Soulier,  Continual learning of long topic sequences in neural information retrieval,  in: European Conference on Information Retrieval, Springer, 2022, pp. 244–259. doi:10.1007/978-3-030-99736-6_17.
  * Chen et al. [2023] J. Chen, R. Zhang, J. Guo, M. de Rijke, W. Chen, Y. Fan, X. Cheng,  Continual learning for generative retrieval over dynamic corpora,  in: Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, CIKM ’23, Association for Computing Machinery, New York, NY, USA, 2023, p. 306–315. doi:10.1145/3583780.3614821.
  * Karakida and Akaho [2021] R. Karakida, S. Akaho,  Learning curves for continual learning in neural networks: Self-knowledge transfer and forgetting,  in: International Conference on Learning Representations, 2021. doi:10.48550/arXiv.2112.01653.
  * Lee et al. [2021] S. Lee, S. Goldt, A. Saxe,  Continual learning in the teacher-student setup: Impact of task similarity,  in: M. Meila, T. Zhang (Eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, PMLR, 2021, pp. 6109–6119. doi:10.48550/arXiv.2107.04384.
  * Kirkpatrick et al. [2017] J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, D. Hassabis, C. Clopath, D. Kumaran, R. Hadsell,  Overcoming catastrophic forgetting in neural networks,  Proc. Natl. Acad. Sci. U. S. A. 114 (2017) 3521–3526. doi:10.1073/pnas.1611835114.
  * Liu et al. [2018] X. Liu, M. Masana, L. Herranz, J. Van de Weijer, A. M. López, A. D. Bagdanov,  Rotate your networks: Better weight consolidation and less catastrophic forgetting,  in: 2018 24th International Conference on Pattern Recognition, 2018, pp. 2262–2268. doi:10.1109/ICPR.2018.8545895.
  * Zenke et al. [2017] F. Zenke, B. Poole, S. Ganguli,  Continual learning through synaptic intelligence,  in: Proceedings of the 34th International Conference on Machine Learning – Volume 70, ICML’17, JMLR.org, 2017, p. 3987–3995. doi:10.48550/arXiv.1703.04200.
  * Chaudhry et al. [2018] A. Chaudhry, P. K. Dokania, T. Ajanthan, P. H. S. Torr,  Riemannian walk for incremental learning: Understanding forgetting and intransigence,  in: V. Ferrari, M. Hebert, C. Sminchisescu, Y. Weiss (Eds.), Proceedings of the European conference on computer vision, Springer International Publishing, Cham, 2018, pp. 556–572. doi:10.1007/978-3-030-01252-6_33.
  * Aljundi et al. [2018] R. Aljundi, F. Babiloni, M. Elhoseiny, M. Rohrbach, T. Tuytelaars,  Memory aware synapses: Learning what (not) to forget,  in: V. Ferrari, M. Hebert, C. Sminchisescu, Y. Weiss (Eds.), Proceedings of the European conference on computer vision, Springer International Publishing, Cham, 2018, pp. 144–161. doi:10.1007/978-3-030-01219-9_9.
  * Mazur et al. [2022] M. Mazur, Ł. Pustelnik, S. Knop, P. Pagacz, P. Spurek,  Target layer regularization for continual learning using Cramer-Wold distance,  Inf. Sci. 609 (2022) 1369–1380. doi:10.1016/j.ins.2022.07.085.
  * Kong et al. [2023] Y. Kong, L. Liu, H. Chen, J. Kacprzyk, D. Tao,  Overcoming catastrophic forgetting in continual learning by exploring eigenvalues of Hessian matrix,  IEEE Trans. Neural Netw. Learn. Syst. (2023) 1–15. doi:10.1109/TNNLS.2023.3292359.
  * Zhang et al. [2023] L. Zhang, S. Wang, F. Yuan, B. Geng, M. Yang,  Lifelong language learning with adaptive uncertainty regularization,  Inf. Sci. 622 (2023) 794–807. doi:10.1016/j.ins.2022.11.141.
  * Rebuffi et al. [2017] S.-A. Rebuffi, A. Kolesnikov, G. Sperl, C. H. Lampert,  iCaRL: Incremental classifier and representation learning,  in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 2001–2010. doi:10.1109/CVPR.2017.587.
  * Lopez-Paz and Ranzato [2017] D. Lopez-Paz, M. Ranzato,  Gradient episodic memory for continual learning,  in: Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, Curran Associates Inc., Red Hook, NY, USA, 2017, p. 6470–6479.
  * Zhuang et al. [2022] C. Zhuang, S. Huang, G. Cheng, J. Ning,  Multi-criteria selection of rehearsal samples for continual learning,  Pattern Recognit. 132 (2022) 108907. doi:10.1016/j.patcog.2022.108907.
  * Rolnick et al. [2019] D. Rolnick, A. Ahuja, J. Schwarz, T. Lillicrap, G. Wayne,  Experience replay for continual learning,  Adv. Neural. Inf. Process. Syst. 32 (2019). doi:10.48550/arXiv.1811.11682.
  * Li et al. [2024] X. Li, B. Tang, H. Li,  AdaER: An adaptive experience replay approach for continual lifelong learning,  Neurocomputing 572 (2024) 127204. doi:10.1016/j.neucom.2023.127204.
  * Rusu et al. [2016] A. A. Rusu, N. C. Rabinowitz, G. Desjardins, H. Soyer, J. Kirkpatrick, K. Kavukcuoglu, R. Pascanu, R. Hadsell,  Progressive neural networks,  CoRR abs/1606.04671 (2016). arXiv:1606.04671.
  * Aljundi et al. [2017] R. Aljundi, P. Chakravarty, T. Tuytelaars,  Expert gate: Lifelong learning with a network of experts,  in: 2017 IEEE Conference on Computer Vision and Pattern Recognition,, IEEE Computer Society, 2017, pp. 7120–7129. doi:10.1109/CVPR.2017.753.
  * Zhang et al. [2023] P. Zhang, Y. Yan, C. Li, S. Wang, X. Xie, G. Song, S. Kim,  Continual learning on dynamic graphs via parameter isolation,  in: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’23, Association for Computing Machinery, New York, NY, USA, 2023, p. 601–611. doi:10.1145/3539618.3591652.
  * Wang et al. [2021] K. Wang, L. Herranz, J. van de Weijer,  Continual learning in cross-modal retrieval,  in: IEEE Conference on Computer Vision and Pattern Recognition Workshops, Computer Vision Foundation / IEEE, 2021, pp. 3628–3638. doi:10.1109/CVPRW53098.2021.00402.
  * Song et al. [2023] G. Song, X. Tan, M. Yang,  Deep continual hashing with gradient-aware memory for cross-modal retrieval,  Pattern Recognit. 137 (2023) 109276. doi:10.1016/j.patcog.2022.109276.
  * Nguyen et al. [2016] T. Nguyen, M. Rosenberg, X. Song, J. Gao, S. Tiwary, R. Majumder, L. Deng,  MS MARCO: A human generated machine reading comprehension dataset,  in: T. R. Besold, A. Bordes, A. S. d’Avila Garcez, G. Wayne (Eds.), Proceedings of the Workshop on Cognitive Computation: Integrating neural and symbolic approaches 2016 co-located with the 30th Annual Conference on Neural Information Processing Systems, volume 1773 of CEUR Workshop Proceedings, CEUR-WS.org, 2016. doi:10.48550/arXiv.1611.09268.
  * Nogueira et al. [2020] R. Nogueira, Z. Jiang, R. Pradeep, J. Lin,  Document ranking with a pretrained sequence-to-sequence model,  in: T. Cohn, Y. He, Y. Liu (Eds.), Findings of the Association for Computational Linguistics: EMNLP 2020, Association for Computational Linguistics, Online, 2020, pp. 708–718. doi:10.18653/v1/2020.findings-emnlp.63.
  * Hsu et al. [2018] Y. Hsu, Y. Liu, Z. Kira,  Re-evaluating continual learning scenarios: A categorization and case for strong baselines,  CoRR abs/1810.12488 (2018). arXiv:1810.12488.
  * Guo et al. [2016] J. Guo, Y. Fan, Q. Ai, W. B. Croft,  A deep relevance matching model for Ad-Hoc retrieval,  in: Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, CIKM ’16, Association for Computing Machinery, New York, NY, USA, 2016, p. 55–64. doi:10.1145/2983323.2983769.
  * Xiong et al. [2017] C. Xiong, Z. Dai, J. Callan, Z. Liu, R. Power,  End-to-end neural Ad-Hoc ranking with kernel pooling,  in: Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’17, Association for Computing Machinery, New York, NY, USA, 2017, p. 55–64. doi:10.1145/3077136.3080809.
  * Mitra et al. [2017] B. Mitra, F. Diaz, N. Craswell,  Learning to match using local and distributed representations of text for web search,  in: Proceedings of the 26th International Conference on World Wide Web, WWW ’17, International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE, 2017, p. 1291–1299. doi:10.1145/3038912.3052579.
  * Hofstätter et al. [2021] S. Hofstätter, S.-C. Lin, J.-H. Yang, J. Lin, A. Hanbury,  Efficiently teaching an effective dense retriever with balanced topic aware sampling,  in: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’21, Association for Computing Machinery, New York, NY, USA, 2021, p. 113–122. doi:10.1145/3404835.3462891.
  * Khattab and Zaharia [2020] O. Khattab, M. Zaharia,  ColBERT: Efficient and effective passage search via contextualized late interaction over BERT,  in: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’20, Association for Computing Machinery, New York, NY, USA, 2020, p. 39–48. doi:10.1145/3397271.3401075.
  * Hong et al. [2021] Y. Hong, H. Xie, G. Bhumbra, I. Brilakis,  Comparing natural language processing methods to cluster construction schedules,  J. Constr. Eng. Manage. 147 (2021) 04021136. doi:10.1061/(ASCE)CO.1943-7862.0002165.
  * Pennington et al. [2014] J. Pennington, R. Socher, C. Manning,  GloVe: Global vectors for word representation,  in: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Doha, Qatar, 2014, pp. 1532–1543. doi:10.3115/v1/D14-1162.
  * Wolf et al. [2020] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. Le Scao, S. Gugger, M. Drame, Q. Lhoest, A. Rush,  Transformers: State-of-the-art natural language processing,  in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Association for Computational Linguistics, Online, 2020, pp. 38–45. doi:10.18653/v1/2020.emnlp-demos.6.
  * Wu et al. [2023] J. Wu, W. Gan, Z. Chen, S. Wan, P. S. Yu,  Multimodal large language models: A survey,  in: 2023 IEEE International Conference on Big Data, 2023, pp. 2247–2256. doi:10.1109/BigData59044.2023.10386743.