# Recent Advances of Foundation Language Models-based Continual Learning: A
Survey

Yutao Yang  ,  Jie Zhou<EMAIL_ADDRESS>,  Xuanwen Ding  ,  Tianyu Huai
,  Shunyu Liu  ,  Qin Chen  ,  Liang He  and  Yuan Xie  School of Computer
Science and Technology, East China Normal UniversityShanghaiChina

(2018; 20 February 2007; 12 March 2009; 5 June 2009)

###### Abstract.

Recently, foundation language models (LMs) have marked significant
achievements in the domains of natural language processing (NLP) and computer
vision (CV). Unlike traditional neural network models, foundation LMs obtain a
great ability for transfer learning by acquiring rich commonsense knowledge
through pre-training on extensive unsupervised datasets with a vast number of
parameters. However, they still can not emulate human-like continuous learning
due to catastrophic forgetting. Consequently, various continual learning
(CL)-based methodologies have been developed to refine LMs, enabling them to
adapt to new tasks without forgetting previous knowledge. However, a
systematic taxonomy of existing approaches and a comparison of their
performance are still lacking, which is the gap that our survey aims to fill.
We delve into a comprehensive review, summarization, and classification of the
existing literature on CL-based approaches applied to foundation language
models, such as pre-trained language models (PLMs), large language models
(LLMs) and vision-language models (VLMs). We divide these studies into offline
CL and online CL, which consist of traditional methods, parameter-efficient-
based methods, instruction tuning-based methods and continual pre-training
methods. Offline CL encompasses domain-incremental learning, task-incremental
learning, and class-incremental learning, while online CL is subdivided into
hard task boundary and blurry task boundary settings. Additionally, we outline
the typical datasets and metrics employed in CL research and provide a
detailed analysis of the challenges and future work for LMs-based continual
learning.

Continual Learning, Foundation Language Models, Pre-trained Language Models,
Large Language Models, Vision-Language Models, Survey

††copyright: acmlicensed††journalyear: 2018††doi: XXXXXXX.XXXXXXX††conference:
Make sure to enter the correct conference title from your rights confirmation
emai; June 03–05, 2018; Woodstock, NY††isbn: 978-1-4503-XXXX-X/18/06††ccs:
Computing methodologies Natural language generation††ccs: Computing
methodologies Scene understanding††ccs: Computing methodologies Cognitive
robotics††ccs: Computing methodologies Cognitive science††ccs: Computing
methodologies Intelligent agents

##  1\. Introduction

Recent advancements in foundation language models (LMs) have set new
benchmarks in both natural language processing (NLP) (Zhao et al., 2023; Min
et al., 2023; Zhou et al., 2023a) and computer vision (CV) (Wang et al.,
2023b). Foundation LMs encompass three primary categories: Pre-trained
Language Models (PLMs) (Min et al., 2023), Large Language Models (LLMs) (Zhao
et al., 2023), and Vision-Language Models (VLMs) (Du et al., 2022). PLMs such
as BERT (Kenton and Toutanova, 2019), RoBERTa (Liu et al., 2019a), and BART
(Lewis et al., 2020) focus on text-based tasks and are crucial for
understanding and generating language by leveraging tasks like masked language
modeling during pre-training. LLMs, including models like GPT-4 (Achiam et
al., 2023) and LLaMA (Touvron et al., 2023), extend the capabilities of PLMs
by increasing the scale of model architecture and training data, thus
enhancing their generality and adaptability across a broader range of tasks.
VLMs, represented by VisualBERT (Li et al., 2019), CLIP (Radford et al.,
2021), LLaVA (Liu et al., 2024) and DALL-E (Ramesh et al., 2021), integrate
text and image modalities to enable complicated interactions between visual
and textual information. The underlying paradigm of these models involves pre-
training on extensive, often unlabeled datasets to capture rich semantic
information, which is subsequently fine-tuned for specific tasks or domains.
This methodology not only boosts performance across various applications but
also significantly enhances the models’ flexibility and task adaptability.

Figure 1. Comparison between traditional CL and Foundation language models
(LMs)-Based CL.

However, these foundation models often demonstrate limitations in dynamic
environments with a sequence of tasks, primarily due to their fixed parameters
once training is completed. These models generally lack the capability to
integrate new data or concepts without undergoing a retraining process. A
significant challenge associated with training on a sequence of tasks is
“catastrophic forgetting” (Kirkpatrick et al., 2017), a phenomenon where a
model loses previously acquired knowledge upon learning new information. This
is in stark contrast to human learning processes, which are inherently
continuous and adaptive. Despite the successes of multi-task learning (MTL)
and transfer learning (TL) in certain applications, they have limitations in
real-world scenarios. MTL necessitates having all tasks and their data
available upfront, which poses a challenge when launching a new service as the
model must be retrained with all the data. Furthermore, TL is typically done
with only two tasks, i.e., the source and the target, rendering it impractical
for real-world online platforms with multiple target tasks. To address these
challenges, it is crucial for models to process and learn the continuously
expanding and diversifying datasets. This requires mechanisms that allow
models to adapt to new linguistic phenomena and trends without compromising
the accuracy and sensitivity towards historical data.

Consequently, continual learning (CL) (Wang et al., 2024b; Van de Ven et al.,
2022), also referred to as lifelong learning (Parisi et al., 2019) or
incremental learning (Zhou et al., 2023b), is a crucial area in artificial
intelligence that seeks to develop systems capable of continuously updating
themselves and acquiring new knowledge, without forgetting previously learned
information, similar to human learning (De Lange et al., 2021). This paradigm
is particularly relevant in the context of foundation language models (LMs),
which are challenged by specific issues such as catastrophic forgetting (CF)
and cross-task knowledge transfer (KT). Catastrophic forgetting represents a
significant challenge, where a model tends to lose previously acquired
knowledge upon learning new information. To address this, language models must
maintain a robust grasp of past language data while adapting to new linguistic
trends. Furthermore, cross-task knowledge transfer is essential for enhancing
the continual learning process. Effective KT not only accelerates the learning
curve for new tasks (forward transfer) but also enhances the model’s
performance on prior tasks via the feedback of new knowledge (backward
transfer).

forked edges, for tree= grow=east, reversed=true, anchor=base west, parent
anchor=east, child anchor=west, base=left, font=, rectangle, draw=hidden-
black, rounded corners, align=left, minimum width=4em, edge+=darkgray, line
width=1pt, s sep=3pt, inner xsep=2pt, inner ysep=3pt, line width=0.8pt,
ver/.style=rotate=90, child anchor=north, parent anchor=south, anchor=center,
, where level=1text width=7.4em,font=,, where level=2text width=8.4em,font=,,
where level=3text width=10.0em,font=,, where level=4text width=10.5em,font=,,
[ Foundation LMs-based CL, ver [ Offline Continual
Learning (§4) [ Domain-Incremental
Learning (§4.1) [ PLMs-based DIL (§4.1.1) [ LFPT5 (Qin and Joty, 2022b), B-CL
(Ke and Xu, 2021), ELLE (Qin et al., 2022), AdapterCL (Madotto et al., 2020),
RMR_ DSE (Li et al., 2022a), DEMIX (Geng et al., 2021) , CLASSIC (Ke et al.,
2021b),
CPT (Ke et al., 2022), C-PT (Zhu et al., 2022), CL-KD (Castellucci et al.,
2021), PlugLM (Cheng et al., 2023b), Pretr (Cossu et al., 2022), AEWC (Lee,
2017), Continual DAP-training (Ke et al., 2023) , leaf, text width=44.6em ] ]
[ LLM-based DIL (§4.1.2) [ COPR (Zhang et al., 2023b), LAMOL (Sun et al.,
2020a), RVAE_LAMOL (Wang et al., 2022a), Adapt-Retrieve-Revise (Zhang et al.,
2023c), Lifelong-MoE (Chen et al., 2023),
CKL (Jang et al., 2022b), DACP (Xie et al., 2023), CPPO (Zhang et al., 2024),
EcomGPT-CT (Ma et al., 2023) , leaf, text width=44.6em ] ] [ VLMs-based DIL
(§4.1.3) [ S-Prompt (Wang et al., 2022b), Medical AI (Yi et al., 2023), VQACL
(Zhang et al., 2023d) , leaf, text width=44.6em ] ] ] [ Task-Incremental
Learning (§4.2) [ PLMs-based TIL (§4.2.1) [ PP (Razdaibiedina et al., 2023),
CTR (Ke et al., 2021a), MeLL (Wang et al., 2021a), LINC (Liu and Mazumder,
2021), ERDA (Qin and Joty, 2022a), PCLL (Zhao et al., 2022), CLIF (Jin et al.,
2021a), ConTinTin (Yin et al., 2022),
HMI (Maekawa et al., 2023), Adaptive Compositional Modules (Zhang et al.,
2022b), DYNAINST (Mok et al., 2023), Conure (Yuan et al., 2021), TERACON (Kim
et al., 2023),
ERNIE 2.0 (Sun et al., 2020b), RecyclableTuning (Qin et al., 2023) , leaf,
text width=44.6em ] ] [ LLMs-based TIL (§4.2.2) [ Conpet (Song et al., 2023),
InstructAlign (Cahyawijaya et al., 2023), Continual-T0 (Scialom et al.,
2022a), DynaMind (Du et al., 2023), ELM (Jang et al., 2023), O-LoRA (Wang et
al., 2023a), JARe (PENG et al., 2024) , leaf, text width=44.6em ] ] [ VLMs-
based TIL (§4.2.3) [ Medical AI (Yi et al., 2023), CTP (Zhu et al., 2023),
ZSCL (Zheng et al., 2023), MoE-Adapters4CL (Yu et al., 2024), TRIPLET (Qian et
al., 2023) , leaf, text width=44.6em ] ] ] [ Class-Incremental
Learning (§4.3) [ PLMs-based CIL (§4.3.1) [ EPI (Wang et al., 2023c), IDBR
(Huang et al., 2021), PAGeR (Varshney et al., 2022), ENTAILMENT (Xia et al.,
2021), ExtendNER (Monaikul et al., 2021), PLE (Li et al., 2022b), DE&E (Wójcik
et al., 2023),
SRC (Liu et al., 2019b) , leaf, text width=44.6em ] ] [ VLMs-based CIL
(§4.3.2) [ MoE-Adapters4CL (Yu et al., 2024), VLM-PL (Kim et al., 2024),
Adaptation-CLIP (Liu et al., 2023a), PROOF (Zhou et al., 2023c), LGCL (Khan et
al., 2023), ZSCL (Zheng et al., 2023),
CLAP (Jha et al., 2024), GMM (Cao et al., 2024) , leaf, text width=44.6em ] ]
] ] [ Online Continual
Learning (§5) [ Hard Task
Boundary (§5.1) [ PLMs-based HTB (§5.1.1) [ MBPA++ (de Masson D’Autume et al.,
2019), Meta-MBPA++ (Wang et al., 2020), OML-ER (Holla et al., 2020), TPEM
(Geng et al., 2021), CID (Liu et al., 2021b), ProgModel (Shen et al., 2019) ,
leaf, text width=44.6em ] ] ] [ Blurry Task
Boundary (§5.2) [ PLMs-based BTB (§5.2.1) [ MBPA++ (de Masson D’Autume et al.,
2019), Meta-MBPA++ (Wang et al., 2020), OML-ER (Holla et al., 2020), TPEM
(Geng et al., 2021), CID (Liu et al., 2021b) , leaf, text width=44.6em ] ] [
VLMs-based BTB (§5.2.2) [ CBA (Wang et al., 2023d), MVP (Moon et al., 2023),
DKR (Cui et al., 2024) , leaf, text width=44.6em ] ] ] ] ]

Figure 2. Taxonomy of foundation language models for continual learning.

Recent advancements in continual learning methodologies have substantially
enhanced the adaptability and knowledge retention capabilities of foundational
language models (LMs). These developments are crucial for addressing complex
challenges previously observed in CL. Researchers have formulated innovative
strategies to mitigate these challenges, thereby enabling LMs to maintain high
performance across a variety of tasks while continually integrating new
knowledge (Mehta et al., 2021; Cossu et al., 2022; Lee et al., 2023). Notable
successes have been documented in diverse downstream tasks, such as aspect-
based sentiment analysis, where continual learning enables dynamic adaptation
to evolving aspects and sentiments (Ke et al., 2021b). Similarly, in dialogue
generation, the novel technologies assist models in refining and expanding
their conversational capabilities through ongoing interactions (Scialom et
al., 2022b). In text classification, continual learning facilitates the
incorporation of new categories and adjustments to shifts in text
distributions without the need for complete retraining (Razdaibiedina et al.,
2023). Moreover, in the realm of visual question answering, continual learning
is essential for updating the models’ capabilities to process and respond to
new types of visual content and queries (Zhang et al., 2023d; Qian et al.,
2023). The aforementioned works underscore the potential of continual learning
to significantly boost the performance of foundation LMs.

In the domain of continual learning, there has been a significant paradigm
shift from traditional methodologies to those that integrate foundation LMs
(See Figure 1). First, foundation LMs demonstrate enhanced generalization and
transfer learning abilities across diverse tasks owing to their broad pre-
training on large-scale datasets. The model has specialized transfer
capability to quickly adapt to downstream tasks with only a few samples.
Consequently, it is crucial to mitigate the degradation of both the zero-shot
transfer and history task abilities in LMs while facilitating the acquisition
of new skills. Second, due to the substantial number of parameters in
foundation LMs, it is crucial to employ parameter-efficient techniques (Han et
al., 2024), such as prompt tuning (Liu et al., 2022) and adapters (Mundra et
al., 2024), to update parameters without comprehensive retraining. Third, the
foundation LMs possess the capability to follow instructions through
instructional learning (Dong et al., 2022; Ouyang et al., 2022), enabling more
dynamic and context-aware interactions.

This review systematically categorizes these strategies and technologies into
two core areas: offline continual learning and online continual learning
(Figure 2). We first give detailed definitions and scenarios to format the
setting of offline and online CL, where offline CL includes domain-
incremental, task-incremental and class-incremental CL, and online CL includes
hard task boundary and blurry task boundary. These learning strategies are
further subdivided into methods based on Pre-trained Language Models (PLMs),
Large Language Models (LLMs), and Vision-Language Models (VLMs). Then, we
summarize the related papers about traditional methods, continual pre-training
methods, parameter-efficient tuning methods and instruction-based methods.
Finally, we static the main datasets from various perspectives and review the
key metrics to evaluate the forgetting and transferring of the models.

The main contributions of this survey paper can be summarized as follows.

  * •

We thoroughly review the existing literature on foundation LMs-based CL
approaches, which integrate foundation LMs with CL to learn new knowledge
without retraining the models. It is quite different from traditional CL since
foundation LMs have great abilities of transfer learning, zero-shot and
instruction following with huge parameters.

  * •

We give the definitions of different settings and categorize these studies
into various classes to better understand the development of this domain. In
addition to the traditional methods like replay, regularization and parameter-
isolation-based algorithms, we also summarize the works about continual pre-
training methods, parameter-efficient tuning methods and instruction tuning-
based methods.

  * •

We provide the characters of existing datasets for CL and present the main
metrics to evaluate the performance of preventing forgetting and knowledge
transfer.

  * •

We discuss the most challenging problems of foundation LMs-based CL and point
out promising future research directions in this field.

The paper is organized as follows. In Section 2, we review the mainly related
surveys about continual learning. Then, we introduce the base settings and
learning modes of continual learning in Section 3, including the definitions
and scenarios of CL. Furthermore, we present the related studies about offline
continual learning, which can be divided into domain-incremental learning,
task-incremental learning and class-incremental learning in Section 4. In
Section 5, we focus on online continual learning, including hard task boundary
and blurry task boundary settings. The typical datasets and metrics are
provided in Section 6 and 7. Finally, we analyze the challenge and further
work in Section 8 and give the conclusion in Section 9.

##  2\. Related Surveys

###  2.1. Continual Learning

Early examinations of Continual Learning (CL) have provided broad coverage, as
observed in surveys such as Parisi et al. (Parisi et al., 2019). Recently,
Wang et al. (Wang et al., 2024b) conduct a comprehensive survey that
categorizes five key strategies in CL: regularization-based, replay-based,
optimization-based, representation-based, and architecture-based approaches.
This survey reflects an effort to organize and understand the diverse
methodologies employed in the field. Notably, there is a growing focus on
class-incremental setting (Zhou et al., 2023b; Masana et al., 2022; Belouadah
et al., 2021) and replay-based approaches (Hayes et al., 2021), reflecting the
increasing granularity of research interests within the CL domain.

###  2.2. Continual Learning for Computer Vision

In the realm of computer vision, De et al. (De Lange et al., 2021) address the
pressing challenge of continual learning, specifically in the task-incremental
setting, where tasks arrive sequentially with clear boundaries. They introduce
a novel stability-plasticity trade-off framework tailored for continual
learners and undertake a comprehensive experimental analysis, comparing the
efficacy of 11 methodologies across three benchmarks. Qu et al. (Qu et al.,
2021) present a comprehensive examination of continual learning, highlighting
its vital role in the accumulation of knowledge from sequential data streams.
This research investigates a range of approaches, encompassing regularization,
knowledge distillation, memory-based methods, and more. These approaches are
systematically categorized by their characteristics and applications in
computer vision. Moreover, Mai et al. (Mai et al., 2022) focus on the realm of
online continual learning within image classification, addressing catastrophic
forgetting. This study evaluates the efficacy of state-of-the-art
methodologies across diverse memory and data configurations. Masana et al.
(Masana et al., 2022) present a comprehensive performance evaluation of class-
incremental methods applied to image classification. The experiments span a
range of scenarios, incorporating large-scale datasets and varied network
architectures. Belouadah et al. (Belouadah et al., 2021) pay more attention to
class-incremental learning algorithms for visual tasks. This study defines the
essential properties of incremental learning algorithms, offers a unified
formalization of the class-incremental learning problem, and provides an
evaluation framework for detailed analysis.

###  2.3. Continual Learning for NLP

Biesialska et al. (Biesialska et al., 2020) address the challenge of continual
learning within Natural Language Processing (NLP), wherein conventional
architectures struggle to accommodate new tasks without compromising
previously acquired knowledge. The research presents an extensive review of
methods, including rehearsal, regularization, and architectural approaches,
all designed to mitigate the aforementioned challenge. In a similar vein, Ke
et al. (Ke and Liu, 2022) offer a focused survey on continual learning within
the NLP domain, providing a comprehensive examination of various continual
learning settings, methodologies, and challenges. This work presents an in-
depth analysis of state-of-the-art approaches and extends original CL settings
to be more general and up-to-date. Additionally, it emphasizes the
significance of knowledge transfer within NLP and the challenges posed by
inter-task class separation.

###  2.4. Continual Learning for Other Domains

Recent surveys, surveys like (Zhang and Kim, 2023; Shaheen et al., 2022;
Lesort et al., 2020) investigate the advancements in incremental learning for
neural recommendation systems and continual learning (CL) in robotics,
respectively. Zhang et al. (Zhang and Kim, 2023) make a notable contribution
to narrow the gap between academic research and industrial applications
through the introduced Incremental Update Recommendation Systems (IURS). They
highlight the imperative for real-time updates with streaming data and focus
on the distinctive challenges posed by IURS in contrast to traditional Batch
Update Recommendation Systems (BURS) and offer a thorough examination of
existing literature and evaluation methodologies in this domain. Shaheen et
al. (Shaheen et al., 2022) provide a comprehensive overview of contemporary
approaches for CL within real-world contexts. Their analysis focuses on
learning algorithms that efficiently handle large sequential datasets within
computational and memory constraints. The survey also explores challenges in
applying CL to autonomous systems, comparing methods across metrics such as
computational efficiency, memory utilization, and network complexity. In the
field of robotics, it is essential for agents to adapt and interact with their
environment using a continuous stream of observations. Thus, Lesort et al.
(Lesort et al., 2020) explore CL within this domain, defining CL as a paradigm
where both data distribution and learning objectives evolve dynamically. They
emphasize the challenges in evaluating CL algorithms in robotic applications
and introduce a novel framework alongside metrics tailored to effectively
present and assess CL methodologies.

This paper centers on the crucial advancements in CL as applied to
foundational language models, which have obtained significant success in the
fields of NLP and multimodal. We categorize existing works into offline and
online CL based on PLMs, LLMs, and VLMs.

Figure 3. The setting of different offline continual learning tasks, including
task-incremental learning, class-incremental learning and domain-incremental
learning. The samples with different classes (domains) are marked with various
shapes (colors). Figure 4. The setting of different online continual learning
tasks, including hard task boundary arriving and blurry task boundary
arriving. The samples with different classes (domains) are marked with various
shapes (colors).

##  3\. Settings and Learning Modes of CL

###  3.1. Basic Formulation

Continual learning is an advanced method in machine learning. Within this
framework, the model is sequentially trained across a diverse array of tasks
denoted as $t$ within the set $T=\\{1,2,...,N\\}$, where each task $t$ is
associated with its individual dataset
$X_{t}=\\{(x^{(t)}_{i},y^{(t)}_{i})\\}^{|X_{t}|}_{i=1}$. Here, $x^{(t)}_{i}$
represents an individual training example, and $y^{(t)}_{i}$ denotes the
corresponding class label for task $t$, while $|X_{t}|$ indicates the total
number of samples in task $t$. However, the data distributions between any two
tasks $t$ and $t^{\prime}$ are distinct ($p(X_{t})\neq p(X_{t^{\prime}})$ for
all $t\neq t^{\prime}$). This distinction presents a fundamental challenge in
managing the diversity of data distributions across multiple tasks. This setup
necessitates the model to learn new knowledge while retaining past
information.

Continual learning encompasses two principal paradigms: offline and online
continual learning. These paradigms define how data arrives and how the model
updates its knowledge over time.

  * •

Offline Continual Learning: This setting involves learning across a series of
tasks, with each task fully presented before handling the next task. For each
task $t$, the model trains on the entire dataset $D_{t}$ through multiple
epochs. The model progresses to task $t+1$ only upon achieving the desired
proficiency on task $t$.

  * •

Online Continual Learning: This setting operates within a dynamic framework
wherein the model learns knowledge from a stream of data points or mini-
batches presented sequentially. Additionally, the model lacks access to the
entire dataset for a given task. This setting closely mirrors real-world
scenarios characterized by continuous data flow, compelling the model to adapt
in real time.

###  3.2. Typical Scenarios

####  3.2.1. Offline Continual Learning

Offline CL (Figure 3) comprises three principal scenarios, each distinguished
by distinct characteristics: Domain-Incremental Learning, Task-Incremental
Learning, and Class-Incremental Learning.

  * •

Domain-Incremental Learning (DIL): The model aims to process diverse data
distributions. Specifically, in DIL, while the data distributions $p(X_{t})$
in task $t$ and $p(X_{t^{\prime}})$ in task $t^{\prime}$ are different, their
task types and class labels remain consistent. The task identities (task IDs)
are not required.

  * •

Task-Incremental Learning (TIL): The model is designed to handle a series of
tasks, each with unique objectives. The classes within these tasks may or may
not be disjoint. The boundaries of each task are clear, and task IDs are
provided during both the training and testing phases.

  * •

Class-Incremental Learning (CIL): The model is designed to continually learn
new class information while retaining knowledge of previously learned classes.
For tasks $t$ and $t^{\prime}$, while they might share the same task type
(such as classification), their class sets $C_{t}$ and $C_{t^{\prime}}$ are
distinct. Moreover, the task IDs are only available during training.

In summary, Domain-Incremental Learning concentrates on adapting the model to
the shifts in input data distributions while maintaining consistency in tasks
and classes. Task-Incremental Learning necessitates the model’s ability to
learn and retain task-specific knowledge over successive tasks. On the other
hand, Class-Incremental Learning highlights the gradual integration of new
classes into the model’s recognition capabilities without compromising
knowledge of previously learned classes.

####  3.2.2. Online Continual Learning

In online continual learning (See Figure 4), existing research is categorized
into two configurations based on the arrival pattern of tasks: ”Hard Task
Boundary” and ”Blurry Task Boundary”:

  * •

Hard Task Boundary: The arrival of tasks follows a strictly structured and
sequential process. Data from the preceding task is completely processed
before transitioning to the next task, ensuring no overlap of data between
tasks.

  * •

Blurry Task Boundary: The distinction between tasks is less clear, similar to
real-world scenarios. Data from different tasks are intermixed, making it
difficult to pinpoint when one task ends and another begins.

In both setups, the main challenge lies in achieving the balance of learning
new data while preserving previously gained knowledge, often termed as
catastrophic forgetting. Numerous approaches, such as experience replay (Qin
et al., 2022; Sun et al., 2020a), elastic weight consolidation (EWC)
(Kirkpatrick et al., 2017), and progressive neural networks (Ke and Xu, 2021;
Ke et al., 2021a), have emerged to address this issue. Each method comes with
its unique strengths and weaknesses upon the task arrival configuration.

##  4\. Offline Continual Learning

###  4.1. Domain-Incremental Learning

####  4.1.1. PLMs-based DIL

##### Traditional Methods.

Continual Learning methodologies are frequently employed in the context of
Pre-trained Language Models (PLMs), encompassing approaches such as replay-
based, regularization-based, and parameter-isolation-based algorithms. Li et
al. (Li et al., 2022a) present a regularization-centric framework for Lifelong
Learning (LLL) termed RMR-DSE, specifically tailored for sequential operation
across multiple domains. Unlike conventional strategies requiring incremental
memory allocation, RMR-DSE employs a recall optimization mechanism, driven by
regularization, to selectively retain important parameters from preceding
tasks. Furthermore, it incorporates a domain drift estimation algorithm to
address embedding space shifts. Castellucci et al. (Castellucci et al., 2021)
propose a Knowledge Distillation-based Continual Learning (CL-KD) approach
with a Teacher-Student framework. When the student model is trained on a new
language, the teacher model also transfers its knowledge of supported
languages to the student model. Lee et al. (Lee, 2017) introduce a domain-
agnostic framework that employs a two-phase training process, initially using
synthetic data to learn general conversational patterns and subsequently using
human-computer dialogs in customer support. Moreover, the Adaptive Elastic
Weight Consolidation algorithm is applied to adjust the loss function,
ensuring the balance between acquiring new knowledge and retaining previously
learned information.

Gururangan et al. (Gururangan et al., 2022) develop a parameter-isolation-
based method, named DEMIX layers, which consists of a collection of experts.
The dynamic addition or removal of these experts can enhance the model’s
ability to adapt to new domains while maintaining robust performance in
previously established ones. PlugLM (Cheng et al., 2023b) is a pre-training
model equipped with a differentiable plug-in memory (DPM) designed for domain-
adaptive continual training. The core concept behind this approach is to
separate the knowledge storage from model parameters using an adaptable key-
value memory structure. It enables the explicit retrieval and utilization of
knowledge stored within the DPM.

(a) CLASSIC

(b) Lifelong-MoE

(c) S-Prompts

Figure 5. Frameworks in DIL: CLASSIC (PLM-based) (Ke et al., 2021b), Lifelong-
MoE (LLM-based) (Chen et al., 2023), S-Prompts (VLM-based) (Wang et al.,
2022b).

##### Continual Pre-training Methods.

Continual domain-adaptive pre-training (DAP-training) (Ke et al., 2023) is
based on two main ideas: (1) the general knowledge in the LM and the knowledge
gained from prior domains are crucial to mitigating CF and enhancing cross-
task knowledge transfer. This is achieved through soft-masking units based on
their importance, and (2) the model is designed to develop complementary
representations of both the current domain and prior domains, thereby
facilitating the integration of knowledge. The key novelty of Continual DAP-
training is a soft-masking mechanism that directly controls the update to the
LM. Cossu et al. (Cossu et al., 2022) formalize and explore the dynamics of
continual pre-training (Pretr) scenarios across language and vision domains.
In this framework, models undergo continuous pre-training on a sequential
stream of data before subsequent fine-tuning for various downstream tasks.

##### Parameter-Efficient Tuning Methods.

Due to the huge parameters of LMs, parameter-efficient tuning methods like
adaptors (Houlsby et al., 2019; Poth et al., 2023) and p-tuning (Liu et al.,
2022) are used for domain-incremental CL (Madotto et al., 2020; Ke and Xu,
2021; Ke et al., 2021b; Zhu et al., 2022).

The adapter architecture incorporates a skip-connection to minimize the number
of parameters. A notable exemplar of this approach is AdapterCL (Madotto et
al., 2020), which employs residual adapters tailored specifically for task-
oriented dialogue systems. This framework, comprising 37 domains, is
structured to facilitate continual learning across four important aspects:
intent recognition, state tracking, natural language generation, and end-to-
end processing. In a related vein, Ke et al. (Ke and Xu, 2021) introduce B-CL
model to tackle critical challenges in CL for Aspect-Based Sentiment
Classification. B-CL integrates continual learning adapters within capsule
network architectures. Aiming at mitigating catastrophic forgetting during
fine-tuning, the CLASSIC model (Ke et al., 2021b) presents an innovative
solution by deploying adapters to tap into BERT’s capabilities (Figure 5(a)).
A novel contrastive continual learning strategy is used to facilitate the
transfer of knowledge across tasks and distill insights from previous tasks to
subsequent ones. It also effectively eliminates the necessity for task
identifiers during testing. Furthermore, Continual PostTraining (CPT) (Ke et
al., 2022) introduces two continual learning plug-in modules, termed CL-
plugins, embedded within each transformer layer of RoBERTa.

Prompt tuning (Liu et al., 2022), or P-tuning, introduces trainable continuous
prompts into the sequence of input word embeddings, while the language model
remains frozen. To tackle the challenge of CL under limited labeled data, Qin
et al. (Qin and Joty, 2022b) propose a Lifelong Few-shot Language Learning
framework (LFPT5). In this framework, prompt tuning, replay and regularization
strategies are leveraged. When presented with a new task, the model generates
pseudo-labeled samples representative of prior domains. The training process
then incorporates these pseudo-labeled samples alongside new task-specific
data. Additionally, the KL divergence loss is employed to maintain label
consistency between the previous and the current model. Furthermore, Zhu et
al. (Zhu et al., 2022) introduced Continual Prompt Tuning (C-PT) as a
methodology to address the challenges of continual learning within dialogue
systems. C-PT facilitates knowledge transfer between tasks through continual
prompt initialization, query fusion, memory replay, and a memory-guided
technique.

##### Instruction Tuning-based Methods.

Instruction tuning-based methods involve transforming a given task into
natural language instructions. Qin et al. (Qin et al., 2022) propose ELLE, a
novel approach aimed at effectively incorporating continuously expanding
streaming data into pre-trained language models (PLMs). It consists of two
fundamental components: (1) function-preserved model expansion, which enhances
knowledge acquisition efficiency by changing the width and depth of an
existing PLM, and (2) pre-trained domain prompts, which significantly enhance
the adaptation for downstream tasks by effectively segregating the diverse
knowledge acquired during pre-training phases.

####  4.1.2. LLMs-based DIL

##### Traditional Methods.

In many practical scenarios, retraining Language Models (LMs) is challenging
due to resource constraints and data privacy concerns. Zhang et al. (Zhang et
al., 2024) introduce Continual Proximal Policy Optimization (CPPO) to address
this issue. CPPO integrates sample-wise weighting into the Proximal Policy
Optimization (PPO) algorithm, effectively balancing policy learning and
knowledge retention. Zhang et al. (Zhang et al., 2023b) propose Continual
Optimal Policy Regularization (COPR), which calculates the optimal policy
distribution without the partition function and uses the previous optimal
policy to regularize the current policy. Sun et al. (Sun et al., 2020a)
introduce LAMOL, which generates pseudo-samples from previous tasks while
training on a new task. It effectively mitigates knowledge loss without
requiring additional memory or computational resources. Building on this
framework, Wang et al. (Wang et al., 2022a) developed RVAE_LAMOL, which
integrates a residual variational autoencoder (RVAE) to encode input data into
a unified semantic space, thereby enhancing task representation. This model
also incorporates an identity task to enhance the model’s discriminative
ability for task identification. To enhance training efficacy, the Alternate
Lag Training (ALT) is devised to segment the training process into multiple
phases.

To reduce hallucinations in specialized domains such as the Chinese legal
domain, Zhang et al. (Zhang et al., 2023c) propose a novel domain adaptation
framework, named Adapt-Retrieve-Revise (ARR). It consists of three steps:
adapting a 7-billion-parameter language model for initial responses,
retrieving corroborative evidence from an external knowledge base, and
integrating these to refine the final response with GPT-4. Gogoulou et al.
(Gogoulou et al., 2023) study the pros and cons of updating a language model
when new data comes from new languages – the case of continual learning under
language shift. They feed various languages into the model to examine the
impact of pre-training sequence and linguistic characteristics on both forward
and backward transfer effects across three distinct model sizes. A new
continual learning (CL) problem, named Continual Knowledge Learning (CKL), is
introduced by Jang et al. (Jang et al., 2022b). To assess CKL approaches, the
authors establish a benchmark and metric measuring knowledge retention,
updating, and acquisition.

##### Continual Pre-training Methods

LLMs have demonstrated remarkable proficiency in tackling open-domain tasks.
However, their application in specific domains faces notable challenges,
encompassing the lack of domain-specific knowledge, limited capacity to
utilize such knowledge, and inadequate adaptation to domain-specific data
formats. To address these issues, researchers have explored a novel approach
known as continual pre-training, aiming to adapt LLMs to specific domains (Xie
et al., 2023; Ma et al., 2023; Cheng et al., 2023a). Among these studies,
Cheng et al. (Cheng et al., 2023a) draw inspiration from human learning
patterns to develop a novel method that transforms raw corpora into reading
comprehension texts. Furthermore, they discover that while training directly
on raw data enhances the model’s domain knowledge, it significantly hurts the
question-answering capability of the model.

Domain-adaptive Continual Pre-training (DACP) uses a large domain corpus,
leading to high costs. To reduce these, Xie et al. (Xie et al., 2023) propose
two strategies: Efficient Task-Similar Domain-Adaptive Continual Pre-training
(ETS-DACP) and Efficient Task-Agnostic Domain-Adaptive Continual Pre-training
(ETA-DACP). ETS-DACP is tailored to improve performance on specific tasks,
building task-specific foundational LLMs. Conversely, ETA-DACP selects the
most informative samples across the domain. Given the high cost of training
LLMs from scratch and limited annotated data in certain domains, Ma et al. (Ma
et al., 2023) propose a novel model called EcomGPT-CT. It employs a fusion
strategy to exploit semi-structured E-commerce data. Moreover, multiple tasks
are designed to assess LLMs’ few-shot in-context learning ability and zero-
shot performance after fine-tuning.

##### Parameter-Efficient Tuning Methods.

Chen et al. (Chen et al., 2023) present an innovative Lifelong Learning
framework, termed Lifelong-MoE, which leverages a Mixture-of-Experts (MoE)
architecture (Figure 5(b)). This architecture enhances the model’s capacity by
incorporating new experts, where previously trained experts and gating
mechanisms are frozen.

####  4.1.3. VLMs-based DIL

Vision-language models (VLMs) have demonstrated superiority in domain-
incremental learning contexts. Yi et al. (Yi et al., 2023) integrate VLMS with
continual learning methodologies to develop a general-purpose medical AI.
Moreover, their study highlights the significance of data-efficient adaptation
algorithms in minimizing the necessity for extensive labeling when
transitioning to new domains or tasks. Furthermore, the prompt text is
utilized to master the pre-trained knowledge embedded within VLMs. Aiming to
independently learn prompts across disparate domains by using pre-trained
VLMs, S-Prompt (Wang et al., 2022b) is devised (Figure 5(c)). This method
encompasses techniques for acquiring image prompts and introduces an
innovative methodology for language-image prompt acquisition. Prompt learning
is conducted separately, utilizing a unified cross-entropy loss function
during training. During inference, a K-NN (k-nearest neighbors) technique is
employed to discern the domain.

In the domain of visual question answering, Zhang et al. (Zhang et al., 2023d)
introduce VQACL, a novel framework designed to effectively integrate data from
both visual and linguistic modalities. This integration is achieved through a
dual-level task sequence that enhances the model’s performance on complex
multimodal tasks. Central to VQACL is a compositionality test that evaluates
the model’s ability to generalize new skill and concept combinations. The
framework also incorporates a novel representation learning strategy that
differentiates between sample-specific (SS) and sample-invariant (SI)
features. SS features capture distinctive attributes of individual inputs,
enhancing output uniqueness, while SI features, derived from category
prototypes, ensure essential characteristics are retained.

(a) HMI

(b) DynaMind

(c) TRIPLET

Figure 6. Frameworks in TIL: HMI (PLM-based) (Maekawa et al., 2023), DynaMind
(LLM-based) (Du et al., 2023), TRIPLET (VLM-based) (Qian et al., 2023).

###  4.2. Task-Incremental Learning

####  4.2.1. PLMs-based TIL

##### Traditional Methods.

Drawing inspiration from neurobiological mechanisms, Maekawa et al. (Maekawa
et al., 2023) present an inventive approach known as Hippocampal Memory
Indexing (HMI) to augment the generative replay technique. HMI leverages
hippocampal memory indexing to integrate compressed representations of prior
training instances, facilitating selective guidance for the generation of
training samples. This methodological refinement contributes to heightened
specificity, balance, and overall quality of the replayed samples. In tackling
the Continual Few-Shot Relation Learning (CFRL) challenge, Qin et al. (Qin and
Joty, 2022a) propose ERDA as a solution, drawing upon replay- and
regularization-based techniques. The ERDA framework integrates embedding space
regularization and data augmentation strategies to effectively confront the
task of acquiring new relational patterns from scarce labeled instances while
mitigating the risk of catastrophic forgetting. Wang et al. (Wang et al.,
2019b) introduced a memory-based approach to continual learning, termed
Episodic Memory Replay (EMR). This method leverages working memory by
selectively replaying stored samples during each iteration of learning new
tasks, thereby facilitating the integration of new knowledge while preserving
previously acquired information.

Conure (Yuan et al., 2021) is a framework that effectively manages multiple
tasks by leveraging the redundancy of parameters in deep user representation
models. Initially, it prunes less critical parameters to make room for new,
task-specific ones. Subsequently, it incorporates these new parameters while
retaining key parameters from prior tasks, facilitating positive transfer
learning. To prevent the loss of previously acquired knowledge, it maintains
these essential parameters in a fixed state. Notably, TERACON (Kim et al.,
2023) utilizes task-specific soft masks to isolate parameters, which not only
targets parameter updates during training but also clarifies the relationships
between tasks. This method includes a novel knowledge retention module that
utilizes pseudo-labeling to mitigate the well-known problem of catastrophic
forgetting.

Ke et al. (Ke et al., 2021a) introduced a model known as CTR, which employs
innovative techniques such as CL-plugin and task masking to tackle the issue
of catastrophic forgetting and to facilitate knowledge transfer across tasks.
These strategies are particularly effective when utilized in conjunction with
pre-trained language models, such as BERT, enhancing their adaptability and
efficacy. In the specific context of User Intent Classification (UIC) within
large-scale industrial applications, Wang et al. (Wang et al., 2021a)
introduce a novel methodology, MeLL, which utilizes a BERT-based text encoder
to generate robust, dynamically updated text representations for continual
learning. MeLL combines global and local memory networks to preserve prototype
representations across tasks, acting as a meta-learner that rapidly adapts to
new challenges. It employs a Least Recently Used (LRU) policy for efficient
global memory management and minimizes parameter growth.

Recent advancements in conversational AI have concentrated on mitigating the
limitations of traditional chatbots, which are dependent on static knowledge
bases and extensive manual data annotation. The Lifelong INteractive learning
in Conversation (LINC) methodology represents a significant stride in this
direction, embodying a dynamic learning framework that mimics human cognitive
processes during interactions (Liu and Mazumder, 2021; Mazumder and Liu, 2024;
Liu, 2020). Structured around an Interaction Module, a Task Learner, and a
Knowledge Store, LINC enables chatbots to dynamically integrate and utilize
knowledge within conversational contexts (Figure LABEL:fig:LINC). This
approach not only facilitates real-time information extraction and learning
from user interactions but also addresses challenges such as managing
erroneous inputs, refining conversational skills, and maintaining user
engagement, thereby enhancing the chatbots’ linguistic and interactive
capabilities.

##### Continual Pre-training Methods.

Continual pre-training represents a paradigm wherein pre-trained language
models (PLMs) are progressively enhanced through the assimilation of new
knowledge from expanding datasets. Sun et al. (Sun et al., 2020b) introduced a
framework called ERNIE 2.0, which incrementally constructs pre-training tasks,
enabling the acquisition of complex lexical, syntactic, and semantic nuances
embedded within the training corpora. This model deviates from traditional
fixed-task training, instead employing continual multi-task learning to refine
its capabilities continually. Further advancing the field of continual pre-
training, RecyclableTuning (Qin et al., 2023) introduces a novel concept of
recyclable tuning that features two distinct strategies: initialization-based
and distillation-based methods. The former uses fine-tuned weights of existing
PLMs as the basis for further enhancements, capitalizing on the established
parametric relationships. Conversely, the distillation-based approach
harnesses outdated weights to maintain knowledge continuity and efficiency in
successor models.

##### Parameter-Efficient Tuning Methods.

Expounding upon the crucial need for more efficacious knowledge integration,
Zhang et al. (Zhang et al., 2022b) propose a pioneering strategy that entails
the integration of Adaptive Compositional Modules alongside a replay
mechanism. These modules are designed to dynamically adjust to new tasks and
are supplemented by pseudo-experience replay, significantly enhancing
knowledge transfer. This framework is distinguished by its adaptive
integration of modules within transformer architectures, skillfully
orchestrating the interactions between existing and new modules to address
emerging tasks. Additionally, the implementation of pseudo-experience replay
promotes efficient knowledge transfer across these modules. Concurrently, Jin
et al. (Jin et al., 2021a) introduce the Continual Learning of Few-Shot
Learners (CLIF) challenge, wherein a model accumulates knowledge continuously
across a series of NLP tasks. Their investigation delves into the impact of
continual learning algorithms on generalization capabilities and advances a
novel approach for generating regularized adapters.

##### Instruction Tuning-based Methods.

Zhao et al. (Zhao et al., 2022) propose the Prompt Conditioned VAE for
Lifelong Learning (PCLL) specifically designed for task-oriented dialogue
(ToD) systems. PCLL employs a conditional variational autoencoder influenced
by natural language prompts to generate high-quality pseudo samples,
effectively capturing task-specific distributions. Additionally, a
distillation process is integrated to refine past knowledge by reducing noise
within pseudo samples. Razdaibiedina et al. (Razdaibiedina et al., 2023)
introduce Progressive Prompts (PP), a novel approach to continual learning in
language models. PP addresses catastrophic forgetting without resorting to
data replay or an excessive proliferation of task-specific parameters. This
method involves acquiring a fresh soft prompt for each task, gradually
appending it to previously learned prompts while keeping the base model
unchanged. The ConTinTin paradigm (Yin et al., 2022) develops a computational
framework for sequentially mastering a series of new tasks guided by explicit
textual instructions. It synthesizes projected outcomes for new tasks based on
instructions, while facilitating knowledge transfer from prior tasks (forward-
transfer) and maintaining proficiency in previous tasks (backward-transfer).

In the realm of lifelong in-context instruction learning aimed at enhancing
the target PLM’s instance- and task-level generalization performance as it
observes more tasks, DYNAINST is devised by Mok et al. (Mok et al., 2023).
DYNAINST integrates the principles of parameter regularization and experience
replay. The regularization technique employed by DYNAINST is tailored to
foster broad local minima within the target PLM. In order to devise a memory-
and computation-efficient experience replay mechanism, they introduce Dynamic
Instruction Replay, comprising Dynamic Instance Selection (DIS) and Dynamic
Task Selection (DTS). DIS and DTS dynamically determine the selection of
instances and tasks to be stored and replayed, respectively.

####  4.2.2. LLMs-based TIL

Recent attention has focused on the convergence of large language models
(LLMs) with continual learning methodologies, exemplified by significant
contributions such as those by Wang et al. (Wang et al., 2024a) and Peng et
al. (PENG et al., 2024). Benefiting from vast corpora and advanced hardware
infrastructure, LLMs showcase remarkable capabilities in language
comprehension and generation. However, challenges arise in scenarios involving
sequential tasks, where LLMs often exhibit a decline in performance known as
catastrophic forgetting.

##### Traditional Methods.

DynaMind, introduced by Du et al. (Du et al., 2023), emerges as a pioneering
framework that intricately incorporates memory mechanisms and modular
operators, enhancing the precision of LLM outputs. Comprising three essential
components, DynaMind includes a memory module dedicated to storing and
updating acquired knowledge, a modular operator for processing input data, and
a continual learning module responsible for dynamically adjusting LLM
parameters in response to new knowledge. Furthermore, Luo et al. (Luo et al.,
2023) conduct a thorough investigation into catastrophic forgetting (CF) in
Large Language Models (LLMs) during continual fine-tuning. Their experiments
across various domains, including domain knowledge, reasoning, and reading
comprehension, reveal the prevalence of CF in LLMs ranging from 1b to 7b
scale, with severity increasing with model size. Comparative analysis between
decoder-only BLOOMZ and encoder-decoder mT0 indicates that BLOOMZ exhibits
less forgetting. Additionally, LLMs demonstrate the ability to mitigate
language bias during continual fine-tuning. Contrasting ALPACA against LLAMA,
the study highlights ALPACA’s superiority in preserving knowledge and
capacity, suggesting that general instruction tuning aids in mitigating CF
during subsequent fine-tuning phases. This research provides valuable insights
into CF dynamics in LLMs, offering strategies for knowledge retention and bias
mitigation.

Wang et al. (Wang et al., 2024a) present TRACE, an innovative benchmark
meticulously designed to evaluate continual learning capabilities in LLMs.
Comprising eight distinct datasets spanning challenging tasks, including
domain-specific challenges, multilingual capabilities, code generation, and
mathematical reasoning, TRACE serves as a comprehensive evaluation platform.
The authors rigorously examine the effectiveness of conventional Continual
Learning (CL) methods when applied to LLMs within the TRACE framework. Peng et
al. (PENG et al., 2024) propose the Joint Adaptive ReParameterization (JARe)
framework, enhanced with Dynamic Task-related Knowledge Retrieval (DTKR), to
facilitate adaptive adjustment of language models tailored to specific
downstream tasks. This innovative approach leverages task distribution within
the vector space, aiming to streamline and optimize the continual learning
process seamlessly.

##### Parameter-Efficient Tuning Methods.

Large language models (LLMs) encounter several substantial challenges that
limit their practical applications. These include high computational
requirements, significant memory demands, and a tendency toward catastrophic
forgetting. Such limitations highlight the need for ongoing research into more
efficient and robust approaches to training and deploying these models.
Continual Parameter-Efficient Tuning (ConPET) (Song et al., 2023) is designed
for the continuous adaptation of LLMs across diverse tasks, leveraging
parameter-efficient tuning (PET) strategies to enhance both efficiency and
performance. ConPET encompasses two primary modes: Static ConPET and Dynamic
ConPET. Static ConPET adapts techniques previously utilized in smaller models
for LLMs, thus minimizing tuning costs and reducing the risk of overfitting.
Conversely, Dynamic ConPET enhances scalability by employing distinct PET
modules for varying tasks, supplemented by a sophisticated selector mechanism.

Moreover, the ELM strategy (Jang et al., 2023) involves initially training a
compact expert adapter on the LLM for each specific task, followed by
deploying a retrieval method to select the most appropriate expert LLM for
each new task. Furthermore, Wang et al. (Wang et al., 2023a) have proposed
orthogonal low-rank adaptation (O-LoRA), a straightforward yet efficacious
method for facilitating continual learning in language models. O-LoRA
mitigates catastrophic forgetting during task acquisition by employing
distinct low-rank vector subspaces maintained orthogonally to minimize task
interference. This method highlights the potential of orthogonal subspace
techniques in improving the adaptability of language models to new tasks
without compromising previously acquired knowledge.

##### Instruction Tuning-based Methods.

Scialom et al. (Scialom et al., 2022a) introduced Continual-T0, an innovative
framework aimed at exploring the capabilities of large language models (LLMs)
through continual learning, incorporating rehearsal techniques. A central
aspect of this approach is the employment of instruction tuning, a key
strategy designed to enhance the adaptability and effectiveness of LLMs when
encountering novel tasks. Leveraging self-supervised pre-training,
Continual-T0 demonstrates exceptional proficiency in mastering new language
generation tasks while maintaining high performance across a diverse range of
70 previously encountered datasets. Despite the demonstrated proficiency of
LLMs in adhering to instructions, their ability to generalize across
underrepresented languages remains suboptimal. In response, InstructAlign
(Cahyawijaya et al., 2023) is proposed to address this challenge by aligning
newly introduced languages with those previously learned, which possess
abundant linguistic resources, thereby mitigating instances of catastrophic
forgetting. The core novelty of this approach lies in its advancement of
language adaptation methodologies for instruction-tuned LLMs, with particular
emphasis on integrating underrepresented languages.

####  4.2.3. VLMs-based TIL

The long-term sustainability of pre-trained visual-language models (VLMs) is
increasingly under scrutiny due to their dependence on continually expanding
datasets. Although these models demonstrate robust performance across a
diverse range of downstream tasks, the incessant growth of real-world data
poses substantial challenges to the sustainability of traditional offline
training methodologies.

##### Traditional Methods.

CTP (Zhu et al., 2023) employs topology preservation and momentum contrast to
maintain consistent relationships within sample mini-batches across tasks,
thereby preserving the distribution of prior embeddings. CTP also introduces
the P9D dataset, comprising over one million image-text pairs across nine
domains, aimed at visual language continuous pre-training (VLCP). Zheng et al.
(Zheng et al., 2023) address the issue of zero-shot transfer degradation in
visual language models by introducing the Zero-Shot Continual Learning (ZSCL)
method. This novel approach utilizes a label-free dataset to facilitate
distillation in the feature space, coupled with the application of weight
regularization within the parameter space. Furthermore, they introduce a new
benchmark, the Multi-Domain Task Incremental Learning (MTIL), designed to
evaluate incremental learning strategies across various domains, thereby
enhancing the assessment of such methods. Moreover, ZSCL has also been adapted
for use in the CIL setting, further broadening its applicability

##### Instruction Tuning-based Methods.

By decoupling prompts and prompt interaction strategies, TRIPLET (Qian et al.,
2023) effectively captures complex interactions between modalities. This
includes specific designs for visual, textual, and fused prompts, as well as
how to interact between different tasks through these prompts and retain
crucial information, thereby reducing catastrophic forgetting. Decoupled
prompts are designed to separate prompts in terms of multi-modality, layer-
wise, and complementary, with each type of prompt containing learnable
parameters intended to capture modality-specific knowledge from pre-trained
visual-language models and training data. The prompt interaction strategies
consist of three main components: the Query-and-Match Strategy, the Modality-
Interaction Strategy, and the Task-Interaction Strategy. These components work
together to enhance the model’s adaptability to different tasks and its memory
for old tasks.

Moreover, COIN (Chen et al., 2024) introduces a new continuous instruction
tuning benchmark designed to evaluate the performance of Multimodal Large
Language Models (MLLMs) in the sequential instruction fine-tuning paradigm.
CoIN includes 10 commonly used data sets covering 8 task categories, ensuring
the diversity of instructions and tasks. In addition, the trained model is
evaluated from two aspects: instruction following and general knowledge, which
respectively evaluate the consistency with human intention and the
preservation of knowledge for reasoning. CoIN converts commonly used visual-
linguistic datasets into instruction fine-tuning data formats by using
different templates to explore the behavior of MLLMs in continuous instruction
fine-tuning. This method takes into account the diversity between different
tasks and attempts to enhance the model’s adaptability to new and old tasks
through diversified instruction templates. To alleviate the catastrophic
forgetting problem of MLLMs, CoIN introduces the MoELoRA method. This method
reduces forgetting by using different experts to learn knowledge on different
tasks and using gate functions to regulate the output of these experts.

##### Parameter-Efficient Methods.

MoE-Adapters4CL (Yu et al., 2024) introduces a parameter-efficient continual
learning method to mitigate long-term forgetting in incremental learning of
visual-language models. Their approach involves dynamically extending the pre-
trained CLIP model to accommodate new tasks by integrating a Mixture-of-
Experts (MoE) adapter. Specifically, MoE consists of several LoRA adapter
experts and routers, where the router calculates gating weights and uses the
$TopK$ function to select the $k$ most relevant experts for learning the
current task. To maintain the zero-shot recognition capabilities of the
visual-language model, a Distribution Discriminative Automatic Selector (DDAS)
is further introduced, which can automatically route in-distribution and out-
of-distribution inputs to the MoE adapters and the original CLIP,
respectively. Furthermore, the MoE-Adapters4CL framework has also been adapted
for use in the CIL setting.

###  4.3. Class-Incremental Learning

####  4.3.1. PLMs-based CIL

##### Traditional Methods.

The study presented in (Monaikul et al., 2021) introduces ExtendNER, a novel
framework for continual learning in Named Entity Recognition (NER) that
obviates the need for extensive re-annotation. This framework employs a
knowledge distillation (KD) technique, wherein a pre-existing named entity
recognition (NER) model, termed the ”teacher”, imparts knowledge to a newly
developed model, termed the ”student”. The student model, designed to identify
new entity types, progressively learns by emulating the teacher model’s
responses to a new dataset. This method allows the student to acquire the
ability to recognize new entities while retaining knowledge of previously
identified ones. Additionally, Liu et al. (Liu et al., 2019b) propose an
innovative method for learning distributed representations of sentences. This
method initiates with the configuration of sentence encoders using features
independent of any specific corpus, followed by iterative updates through
Boolean operations on conceptor matrices. This technique ensures that the
encoders maintain their performance on existing datasets while adapting
effectively to new data.

(a) IDBR

(b) PLE

(c) Adaptation-CLIP

Figure 7. Frameworks in CIL: IDBR (PLM-based) (Huang et al., 2021), PLE (PLM-
based) (Li et al., 2022b), Adaptation-CLIP (VLM-based) (Liu et al., 2023a).

Huang et al. (Huang et al., 2021) introduced an innovative methodology known
as Information Disentanglement-based Regularization (IDBR) to address the
enduring challenges associated with continual text classification. This method
effectively disentangles the hidden spaces of text into task-generic and task-
specific representations, employing distinct regularization strategies to
enhance knowledge retention and facilitate generalization. Furthermore, the
integration of two auxiliary tasks, namely next sentence prediction and task-
id prediction, serves to augment the learning process by reinforcing the
separation between generic and specific representational spaces.

##### Instruction Tuning-based Methods.

Introduced by Varshney et al. (Varshney et al., 2022), Prompt Augmented
Generative Replay (PAGeR) is a method that enables continual learning in
intent detection without retaining past data. PAGeR leverages pre-trained
language models to generate intent-specific utterances specific to new intents
while preserving accuracy on existing ones. Unlike exemplar replay, which
stores specific examples, PAGeR is structured to selectively maintain relevant
contexts for each specified intent, which are then employed as generation
prompts. This approach combines utterance generation and classification into
one model, enhancing knowledge transfer and optimization.

##### Parameter-Efficient Tuning Methods.

In the domain of task-oriented dialogue systems, Continual Few-shot Intent
Detection (CFID) focuses on recognizing new intents with few examples. Li et
al. (Li et al., 2022b) propose a Prefix-guided Lightweight Encoder (PLE) to
address this by using a parameter-efficient tuning method that combines a
Continual Adapter module with a frozen Pre-trained Language Model (PLM) and a
Prefix-guided Attention mechanism to reduce forgetting. To further mitigate
forgetting, the Pseudo Samples Replay (PSR) strategy reinforces prior
knowledge by replaying crucial samples from previous tasks. The Teacher
Knowledge Transfer (TKT) strategy uses distillation to transfer task-specific
knowledge to maintain performance on new tasks. Additionally, the Dynamic
Weighting Replay (DWR) strategy dynamically adjusts weights of previous tasks
to balance new knowledge acquisition with the revision of old tasks,
navigating the variability and potential negative impacts of prior tasks

The Efficient Parameter Isolation (EPI) method, introduced in Wang et al.
(Wang et al., 2023c), assigns unique subsets of private parameters to each
task alongside a shared pre-trained model. This approach ensures precise
parameter retrieval and has been shown to outperform non-caching methods in
continual language learning benchmarks, while remaining competitive with
caching methods. Furthermore, EPI employs random static masking to reduce
storage requirements, increasing its viability in resource-constrained
environments.

In complex system environments, efficient and adaptable machine learning
architectures are crucial, especially for classification tasks with
sequentially presented data. Wójcik et al. (Wójcik et al., 2023) devise a
novel architecture, Domain and Expertise Ensemble (DE&E), comprising a feature
extractor, classifier, and gating mechanism. The feature extractor employs a
multi-layer neural network to convert input data into embeddings, while the
classifier, a mixture of binary class-specific experts, leverages a gating
mechanism to select the appropriate expert for the current input dynamically.
This Mixture of Experts-based method promotes incremental learning by training
experts with class-specific samples and combines their outputs during testing
to derive the final classification.

####  4.3.2. VLMs-based CIL

##### Traditional Methods.

Kim et al. (Kim et al., 2024) introduce VLM-PL, a novel approach for class
incremental object detection that incorporates new object classes into a
detection model without forgetting old ones. Utilizing a visual-language model
(VLM), this method enhances the pseudo-labeling process to improve the
accuracy and performance of object detection in continual learning settings.
VLM-PL starts with a pre-trained detector to generate initial pseudo-labels,
which are then validated through a visual-language evaluation using a
specially designed hint template. Accurate pseudo-labels are retained and
combined with ground truth labels to train the model, ensuring it remains
proficient in recognizing both new and previously learned categories. Cao et
al. (Cao et al., 2024) introduce a framework for a Generative Multi-modal
Model (GMM) that leverages large language models for class-incremental
learning. This innovative approach entails the generation of labels for images
by employing an adapted generative model. Following the production of detailed
textual descriptions, a text encoder is utilized to extract salient features
from these descriptions. These extracted features are subsequently aligned
with existing labels to ascertain the most fitting label for classification
predictions.

PROOF (Zhou et al., 2023c) develops a method to enhance model memory retention
when adapting to downstream tasks. This method involves a projection technique
that maps pre-trained features into a new feature space designed to preserve
prior knowledge. Additionally, to effectively utilize cross-modal information,
PROOF introduces a fusion module that employs an attention mechanism. This
module adjusts both visual and textual features simultaneously, enabling the
capture of semantic information with enhanced expressive power. Recently, Jha
et al. (Jha et al., 2024) present a new approach for adapting pre-trained
vision-language models like CLIP to new tasks without forgetting previous
knowledge. It employs a Variational Inference framework to probabilistically
model the distribution of visual-guided text features, enhancing fine-tuning
reliability by accounting for uncertainties in visual-textual interactions.
Key to CLAP is the visual-guided attention (VGA) module, which aligns text and
image features to prevent catastrophic forgetting. Additionally, CLAP includes
lightweight, task-specific inference modules that learn unique stochastic
factors for each task, allowing continuous adaptation and knowledge retention.

##### Parameter-Efficient Tuning Methods.

Liu et al. (Liu et al., 2023a) introduce Adaptation-CLIP, which employs three
strategies for CLIP’s continual learning: linear adapter, self-attention
adapter, and prompt tuning. The first two strategies add a linear layer and a
self-attention mechanism, respectively, after the image encoder while freezing
the remaining architecture. The third, prompt tuning, integrates trained
prompts into the text encoder to enhance task comprehension and splices these
with prior prompts to maintain continuity. To prevent catastrophic forgetting,
a parameter retention strategy preserves significantly altered parameters from
$M_{t-1}$ to $M_{t}$, ensuring stability and effective continual learning.

##### Instruction Tuning-based Methods.

Khan et al. (Khan et al., 2023) introduce two notable advancements: an
enhanced prompt pool key query mechanism and category-level language guidance.
The key query mechanism uses CLS features to improve prompt selection,
featuring key replacement with a fixed CLS tag and dynamic mapping to task-
level language representations, thereby enhancing accuracy and robustness
across different tasks. Meanwhile, category-level language guidance is
implemented in the vision transformer to better align output features with
category-specific language representations, significantly improving task
handling and category differentiation, leading to improved model performance.

##  5\. Online Continual Learning

###  5.1. Hard Task Boundary

####  5.1.1. PLMs-based HTB

The Hard Task Boundary (HTB) setting has been developed to enable continuous
knowledge acquisition by learning models from a dynamically changing stream of
textual data, without the need for dataset identifiers. For example, Shen et
al. (Shen et al., 2019) have implemented HTB in slot filling, Michieli et al.
(Michieli et al., 2023) have utilized it in audio classification, and Vander
et al. (Vander Eeckt et al., 2023) have explored its use in automatic speech
recognition.

##### Traditional Methods.

Continual learning (CL) methodologies, particularly pertinent to online
scenarios, encompass a variety of approaches. These include parameter-
isolation-based methods (de Masson D’Autume et al., 2019; Wang et al., 2020),
replay-based methods (Holla et al., 2020) and regularization-based methods
(Vander Eeckt et al., 2023; Liu et al., 2021b). MBPA++ (de Masson D’Autume et
al., 2019) introduces a framework for lifelong language learning, enabling a
pre-trained model to learn continually from textual examples without requiring
labeled datasets. It employs an episodic memory system with sparse experience
replay and local adaptation techniques to prevent catastrophic forgetting.
Extending this framework, Meta-MBPA++ (Wang et al., 2020) integrates three
core lifelong learning principles, enhancing performance in text
classification and question-answering tasks while using only 1% of the typical
memory usage. Liu et al. (Liu et al., 2021b) introduce a regularization-based
strategy, CID, for lifelong intent detection. This method uses cosine
normalization, hierarchical knowledge distillation, and inter-class margin
loss to tackle the challenges of data imbalances in the lifelong intent
detection (LID) task, aiming to mitigate the negative impacts associated with
these imbalances.

(a) MBPA++

(b) CBA

Figure 8. Frameworks in Online Continual Learning: MBPA++ (PLM-based HTB/BTB)
(de Masson D’Autume et al., 2019), CBA (VLM-base BTB) (Wang et al., 2023d).

###  5.2. Blurry Task Boundary

####  5.2.1. PLMs-based BTB

MBPA++ (de Masson D’Autume et al., 2019) and Meta-MBPA++ (Wang et al., 2020)
can also exemplify models capable of adapting to environments with indistinct
task boundaries. TPEM (Geng et al., 2021) employs a tripartite approach within
an encoder-decoder framework, consisting of pruning, expanding, and masking
techniques. Pruning helps preserve essential information from previous tasks,
expansion increases the model’s capacity to accommodate new tasks, and masking
reduces interference from previous tasks’ weights, thus enhancing learning
efficiency. Originally, online meta-learning (OML) (Javed and White, 2019) and
a neuromodulatory meta-learning algorithm (ANML) (Beaulieu et al., 2020) were
intended to continuously learn new sequences of tasks during the testing
phase. Holla et al. (Holla et al., 2020) adapt these algorithms for a
conventional continual learning context, where the focus is on evaluating
performance on previously encountered tasks. The enhanced versions, named OML-
ER and ANML-ER, incorporate an episodic memory module designed for experience
replay.

####  5.2.2. VLMs-based BTB

##### Traditional Methods.

Cui et al. (Cui et al., 2024) propose the Dynamic Knowledge Rectification
(DKR) framework, designed to mitigate the propagation of incorrect information
in foundation LMs. The DKR framework operates by initially leveraging an
existing model to identify and exclude obsolete or erroneous knowledge when
confronted with new data. Subsequently, a rectification process is employed to
amend these inaccuracies while ensuring the preservation of valid data
associations. This process is especially vital when integrating new data, as
it prevents the perpetuation of outdated or incorrect information. In cases
where data is inaccessible through existing models, DKR utilizes paired
ground-truth labels to support the continuous evolution of knowledge bases,
thereby enhancing the model’s accuracy.

##### Parameter-Efficient Tuning Methods.

Wang et al. (Wang et al., 2023d) introduce the Continual Bias Adaptor (CBA)
(Figure 8(b)), a novel method designed to enhance the efficacy of online CL by
mitigating catastrophic forgetting. The CBA method dynamically modifies the
classifier’s network to adapt to significant shifts in data distribution
observed during training, thereby preserving knowledge from earlier tasks.
Notably, the CBA module can be deactivated during testing phases, eliminating
additional computational burdens or memory demands. This feature highlights
the practicality and applicability of the CBA in real-world scenarios.

##### Instruction Tuning-based Methods.

In the BTB scenario, the Mask and Visual Prompt tuning (MVP) method, as
detailed by Moon et al. (Moon et al., 2023), addresses challenges like intra-
and inter-task forgetting and class imbalance effectively. MVP features
instance-wise logit masking to prevent irrelevant information retention,
contrastive visual prompt tuning loss to ensure consistent prompt selection,
gradient similarity-based focal loss to focus on overlooked samples, and
adaptive feature scaling to balance the integration of new knowledge with
existing data retention.

##### Online Learning for CV

Recent advancements in class-incremental online learning predominantly address
computer vision tasks, encapsulating various methodologies such as
regularization-based (Yin et al., 2021; Gu et al., 2022; Guo et al., 2022;
Bang et al., 2022), replay-based (Aljundi et al., 2019; Shim et al., 2021;
Caccia et al., 2021; Lin et al., 2023; Zhang et al., 2022a), distillation-
based (He et al., 2020; Fini et al., 2020; Koh et al., 2022), and gradient-
based approaches (Jin et al., 2021b; Guo et al., 2023; Chen et al., 2020; Yoon
et al., 2021; Ororbia et al., 2022). Among these, Koh et al. (Koh et al.,
2021) introduced the Class-Incremental Blurry (CLIB) model, which
distinguishes itself through its task-free nature, adaptability to class
increments, and prompt response to inference queries, showing superior
performance over traditional continual learning methods. In a related vein,
Gunasekara et al. (Gunasekara et al., 2023) explore the Online Streaming
Continual Learning (OSCL), a hybrid of Stream Learning (SL) and Online
Continual Learning (OCL), which integrates aspects of both domains.
Furthermore, issues of shortcut learning and bias in online continual learning
have been tackled by Wei et al. (Wei et al., 2023) and Chrysalis et al.
(Chrysakis and Moens, 2023). Additionally, Semola et al. (Semola et al., 2022)
propose Continual-Learning-as-a-Service (CLaaS), a service model that
leverages continual learning to monitor shifts in data distribution and update
models efficiently. This array of developments highlights the dynamic
capabilities of online continual learning frameworks to effectively address
complex challenges across a variety of computer vision tasks.

Table 1. The statistics information of the existing CL datasets. #D/T/C means the number of domains/tasks/classes, respectively. Datasets | #Train | #Val | #Test | #Total | CL Settings |  NLP Problems | Language | #D/T/C
---|---|---|---|---|---|---|---|---
Offline
Progressive Prompts (Razdaibiedina et al., 2023) | - | - | - | - | TIL |  Sentiment analysis, topic classification, boolean QA, QA, paraphrase detection, word sense disambiguation, natural language inference | English | 15 tasks
MeLL (Wang et al., 2021a) | 1,430,880 | 173,781 | 118,240 | 1,722,901 | TIL |  Intent classification | English | 1184 tasks
Continual-T0 (Scialom et al., 2022b) | 800,000 | - | 33,382 | 833,382 | TIL |  Text Simplification, Headline Generation with Constraint, Haiku Generation, Covid QA, Inquisitive Question Generation, Empathetic Dialogue Generation, Explanation Generation, Twitter Stylometry | English | 8 tasks
COPR (Zhang et al., 2023b) | - | - | - | - | TIL |  QA tasks, summary task, positive file review generation task | English | 3 tasks
Adaptive Compositional Modules (Zhang et al., 2022b) | 50,725 | - | 27,944 | 78,669 | TIL |  Natural language generation, SQL query generation, Summarization and Task-oriented dialogue | English | 8 tasks
CODETASKCL (Yadav et al., 2023) | 181,000 | 9,700 | 10,000 | 200,700 | TIL |  Code generation, code translation, code summarization, and code refinement | Hybrid | 4 tasks
Lifelong Simple Questions (Wang et al., 2019b) | - | - | - | - | TIL |  single-relation questions | English | 20 tasks
Lifelong FewRel (Wang et al., 2019b) | - | - | - | - | TIL |  few-shot relation detection | English | 10 tasks
InstrDialog (Zhang et al., 2023a) | 9,500 | 950 | 1,900 | 12,350 | TIL |  Dialogue state tracking, dialogue generation, intent identification | English | 19 tasks
InstrDialog++ (Zhang et al., 2023a) | 3,800 | 1,900 | 3,800 | 9,500 | TIL |  Dialogue Generation, Intent Identification, Dialogue State Tracking, Style Transfer, Sentence Ordering, Word Semantics, Text Categorization, Pos Tagging, Fill in The Blank, Program Execution, Question Generation, Misc, Coherence Classification, Question Answering, Summarization, Commonsense Classification, Wrong Candidate Generation and Toxic Language Detection | English | 38 tasks
ConTinTin (Yin et al., 2022) | - | - | - | - | TIL |  Question generation tasks (QG), answer generation tasks (AG), classification tasks (CF), incorrect answer generation tasks (IAG), minimal modification tasks (MM) and verification tasks (VF) | English | 61 tasks
Tencent TL (Yuan et al., 2021) | - | - | - | - | TIL |  personalized recommendations and profile predictions | English | 6 tasks
Movielens (Yuan et al., 2021) | - | - | - | - | TIL |  personalized recommendations and profile predictions | English | 3 tasks
NAVER Shopping (Kim et al., 2023) | - | - | - | - | TIL |  search query prediction tasks | English | 6 tasks
TRACE (Wang et al., 2024a) | 40,000 | - | 16,000 | 56,000 | TIL |  Domain-specific task, Multi-lingual task, Code completion task, Mathematical reasoning task | Hybrid | 8 tasks
ABSC (Ke and Xu, 2021) | 3,452 | 150 | 1,120 | 4,722 | DIL |  Aspect-based sentiment classification | English | 19 Domains
DecaNLP (Sun et al., 2020a) | 169,824 | - | 32,116 | 201,940 | DIL |  Question answering, semantic parsing, sentiment analysis, semantic role labeling, and goal-oriented dialogue | English | 5 domains
Foundational text classification (Sun et al., 2020a) | 115,000 | - | 7,600 | 122,600 | DIL |  News classification, sentiment analysis, Wikipedia article classification, and question-and-answer categorization | English | 5 domains
RVAE_LAMOL (Wang et al., 2022a) | 15,870 | - | 5,668 | 21,538 | DIL |  Oriented dialogue of the restaurant reservation task, semantic role labeling, sentiment classification | English | 3 domains
COPR (Zhang et al., 2023b) | - | - | - | - | DIL |  - | English | 18 domains
SGD (Zhu et al., 2022) | 38,745 | 5,210 | 11,349 | 40,287 | DIL |  Dialogue state tracking | English | 19 Domains
CPT (Ke et al., 2022) | 3,121,926 | - | - | 3,121,926 | DIL |  Domain-adaptive pre-training task | English | 4 Domains
CKL(Jang et al., 2022b) | - | - | - | 30,372 | DIL |  Domain-adaptive pre-training task | English | 3 Domains
ELLE (Qin et al., 2022) | - | - | - | - | DIL |  Domain-adaptive pre-training task | English | 5 Domains
Domain-incremental Paper Stream (Jin et al., 2022) | - | - | - | - | DIL |  Relation extraction and named entity recognition | English | 4 domains
Chronologically-ordered Tweet Stream (Jin et al., 2022) | - | - | - | - | DIL |  multi-label hashtag prediction and single-label emoji prediction | English | 4 domains
AdapterCL (Madotto et al., 2020) | 31,426 | 4,043 | 4,818 | 40,287 | DIL |  Intent classification, Dialogue State Tracking (DST), Natural Language Generation (NLG), end-to-end (E2E) modeling | English | 37 Domains
DE&E (Wójcik et al., 2023) | 28,982 | - | 12,089 | 41,071 | CIL |  Text classification | English | 3 tasks
EPI (Wang et al., 2023c) | 12,840 | 3,524 | 6,917 | 23,281 | CIL |  Text classification, topic classification | English | 13 Classes
PAGeR (Varshney et al., 2022) | 59,754 | 7,115 | 15,304 | 82,173 | CIL |  Intent classification | English | 355 Classes
PLE (Li et al., 2022b) | 4,669 | 4,650 | 31,642 | 40,961 | CIL |  Intent classification | English | 477 Classes
CoNLL-03 (Monaikul et al., 2021) | 23,326 | 5,902 | 5,613 | 34,841 | CIL |  Named Entity Recognition | English | 4 Classes
OntoNotes (Monaikul et al., 2021) | 107,169 | 16,815 | 10,464 | 134,448 | CIL |  Named Entity Recognition | English | 6 Classes
Online
Foundational text classification (de Masson D’Autume et al., 2019) | 115,000 | - | 7,600 | 122,600 | Hard and Blurry |  News classification, sentiment analysis, Wikipedia article classification, and question-and-answer categorization | English | 5 tasks
MBPA++ (de Masson D’Autume et al., 2019) | 881,000 | 35,000 | 38,000 | 954,000 | Hard and Blurry |  News classification, sentiment analysis, Wikipedia article classification, questions and answers categorization, question answering | English | 9 tasks
Lifelong FewRel (Holla et al., 2020) | - | - | - | - | Hard and Blurry |  few-shot relation detection | English | 10 tasks
Firehose (Hu et al., 2022) | - | - | - | 110,000,000 | Blurry |  Personalized online language learning | English | 1 tasks
TemporalWiki (Jang et al., 2022a) | - | - | - | - | - |  - | English | -

##  6\. Datasets

###  6.1. Offline Datasets for NLP

####  6.1.1. Datasets for Classification.

##### Text Classification.

The most typical task for continual learning is text classification. The
foundational text classification benchmark encompasses five text
classification datasets introduced by (Zhang et al., 2015), including AG News,
Amazon Reviews, Yelp Reviews, DBpedia, and Yahoo Answers (Sun et al., 2020a).
Particularly, the AG News dataset has 4 classes for news classification; the
Amazon and Yelp dataset has 5 classes for sentiment analysis; the DBpedia
dataset has 14 classes for Wikipedia text classification; and the Yahoo
dataset has 10 classes for Q&A classification. The text classification
benchmark includes 115,000 training and 7,600 test examples for each task,
holding out 500 samples per class from the training set for validation.
Building upon this, Razdaibiedina et al. (Razdaibiedina et al., 2023)
developed a novel continual learning (CL) benchmark. This benchmark not only
utilizes the foundational text classification benchmark but also integrates
additional datasets from the GLUE benchmark (Wang et al., 2018), SuperGLUE
benchmark (Wang et al., 2019a), and the IMDB dataset (Maas et al., 2011).
Specifically, the GLUE benchmark datasets included are MNLI, QQP, RTE, and
SST2, focusing on tasks like natural language inference, paraphrase detection,
and sentiment analysis. Similarly, the SuperGLUE datasets—WiC, CB, COPA,
MultiRC, and BoolQ—encompass tasks ranging from word sense disambiguation to
question answering. DE&E (Wójcik et al., 2023) uses three common text
classification data sets with different characteristics-News-groups, BBC News,
and Consumer Finance Complaints2. Such datasets can be used to evaluate the
models on tasks with different difficulty levels.

The datasets introduced in (Wang et al., 2023c) further are categorized into
two groups based on the domain relevance between tasks: far-domain and near-
domain. The far-domain group comprises two text classification tasks, which
are foundational benchmarks (Zhang et al., 2015) divided into topic
classification (AG News, Yahoo Answers, DBpedia) and sentiment classification
(Yelp, Amazon Reviews). In contrast, the near-domain group uses the Web of
Science (WOS) (Kowsari et al., 2017) and 20 Newsgroups (Lang, 1995), which are
restructured according to their high inter-task relevance. The WOS dataset
comprises seven parent classes, each with five closely related sub-classes,
while the 20 Newsgroups dataset, containing six news topics, is reorganized
into four tasks to maximize inter-task correlation.

##### Intent Classification.

Some studies focus on intent classification tasks, where the classes are quite
different in various domains or scenarios. In the realm of intent
classification and detection, several datasets have been specifically designed
to advance the field by addressing different challenges and providing diverse
environments for model training and evaluation. The dataset, as introduced in
PAGeR (Varshney et al., 2022), aims to tackle the lifelong intent detection
problem by combining three public intent classification datasets (CLINC150
(Larson et al., 2019), HWU64 (Liu et al., 2021a), BANKING77 (Casanueva et al.,
2020)), one text classification dataset (Stackoverflow S20 (Xu et al., 2017)),
and two public multidomain dialog intent detection datasets (SGD (Rastogi et
al., 2020), MWOZ (Budzianowski et al., 2018)). Moreover, FewRel (Han et al.,
2018) is also incorporated to tackle the lifelong relation extraction problem.
This integration is intended to simulate real-world applications by
encompassing a broad spectrum of domains and query distributions, thereby
facilitating the development of more robust and versatile intent detection
systems.

Conversely, the dataset compiled in PLE (Li et al., 2022b) consolidates nine
well-regarded intent detection datasets, including CLINC150 (Larson et al.,
2019) and HWU64 (Liu et al., 2021a), among others, arranged in a fixed random
sequence to form a standardized benchmark. This dataset emphasizes the
importance of consistency and comparability in performance evaluations across
different intent detection models, providing a platform for assessing and
enhancing various methodologies. The dataset described by MeLL (Wang et al.,
2021a) specifically addresses intent detection within two distinct contexts:
task-oriented dialogues (TaskDialog-EUIC) and real-world e-commerce
interactions (Hotline-EUIC). TaskDialog-EUIC integrates data from Snips
(Coucke et al., 2018), TOP semantic parsing (Gupta et al., 2018), and
Facebook’s Multilingual Task Oriented Dataset (Schuster et al., 2018) into 90
tasks with overlapping label sets, amounting to over ten thousand samples.
Hotline-EUIC is derived from an e-commerce dialogue system (Li et al., 2017)
and the hotline audios are transcribed to text by a high-accuracy industrial
Automatic Speech Recognition (ASR) system.

##### Fine-grained Sentiment Analysis.

Ke et al. (Ke and Xu, 2021) developed a task incremental learning dataset for
aspect-based sentiment classification (ABSC). This dataset aggregates reviews
from four distinct sources, thereby enhancing its diversity and applicability
across multiple domains. The sources include the L5Domains dataset by Hu et
al. (Hu and Liu, 2004), which features consumer reviews for five different
products; the Liu3Domains dataset by Liu (Liu et al., 2015), comprising
reviews pertaining to three products; the Ding9Domains dataset by Ding et al.
(Ding et al., 2008), which includes reviews of nine varied products; and the
SemEval14 dataset, which is focused on reviews of two specific
products—laptops and restaurants.

####  6.1.2. Datasets for Generation.

In the rapidly advancing field of machine learning, diverse datasets function
as crucial benchmarks for exploring various dimensions of language and code
generation. These datasets address both universal and task-specific
challenges, enabling a comprehensive evaluation of model capabilities. A
particularly significant dataset highlighted in the work by Continual-T0
(Scialom et al., 2022b) focuses on English language generation tasks,
including text simplification and empathetic dialogue generation, among others
(Camburu et al., 2018; Bin Tareaf, 2017). The design of this dataset maintains
uniformity in size, facilitating effective comparative analyses of performance
across distinct tasks by ensuring a consistent volume of data for training. In
a subsequent study, Luo et al. (Luo et al., 2023) conduct an analysis of
catastrophic forgetting on Bloomz (Scao et al., 2022) using Continual T0
datasets.

The dataset, introduced in LAMOL (Sun et al., 2020a), integrates elements from
both DecaNLP (McCann et al., [n. d.]) and the foundational text classification
benchmark (Zhang et al., 2015). This dataset encompasses five distinct NLP
tasks originally sourced from DecaNLP: question answering, semantic parsing,
sentiment analysis, semantic role labeling, and goal-oriented dialogue. For
the purposes of this dataset, all tasks, whether derived from DecaNLP or the
foundational text classification benchmark, are restructured into a uniform
format, conceptualized under the framework of a question answering task.
Moreover, the dataset devised in RVAE_LAMOL (Wang et al., 2022a), employs
three tasks from DecaNLP: the English Wizard of Oz (WOZ) for goal-oriented
dialogue, QA-SRL for semantic role labeling in a SQuAD-style format, and SST,
which is a binary version of the Stanford Sentiment Treebank categorizing
sentiments as positive or negative. These tasks are specifically treated as
sequence generation tasks.

The dataset introduced in COPR (Zhang et al., 2023b) represents a pioneering
effort in applying both Task Incremental Learning (TIL) and Domain Incremental
Learning (DIL) within the context of benchmarks that utilize existing human
preferences. Specifically, the TIL framework in this dataset mandates that the
model sequentially acquires knowledge from three distinct tasks. These include
the question-answering task utilizing the HH-RLHF dataset (Bai et al., 2022),
the summarization task based on the Reddit TL, DR dataset with human feedback
(Volske et al., 2017), and the positive film review generation task using the
IMDB dataset (Maas et al., 2011). Meanwhile, the DIL framework requires the
model to adapt to three distinct segments from the SHP dataset, as described
by Ethayarajh et al. (Ethayarajh et al., 2022).

The dataset described in Adaptive Compositional Modules (Zhang et al., 2022b)
explores sequence generation and categorizes tasks into ”similar” and
”dissimilar” groups based on their characteristics. Tasks classified as
similar, including E2ENLG (Novikova et al., 2017) and four domains
(restaurant, hotel, TV, laptop) from RNNLG (Wen et al., 2015), demonstrate
shared patterns and are tested across four sequence orders, comprising a total
of five tasks. In contrast, dissimilar tasks such as WikiSQL (SQL query
generation) (Zhong et al., 2017), CNN/DailyMail (news article summarization)
(See et al., 2017), and MultiWOZ (semantic state sequence generation)
(Budzianowski et al., 2018) exhibit significant distributional shifts from
previously encountered tasks. The CODETASKCL dataset, explored by Yadav et al.
(Yadav et al., 2023), encompasses a diverse array of code-centric tasks,
including code generation (Iyer et al., 2018), summarization (Husain et al.,
2019), translation (Lu et al., 2021), and refinement (Tufano et al., 2019)
across various programming languages. This dataset significantly enhances the
breadth of language processing applications within technical fields.

####  6.1.3. Datasets for Information Extraction.

In the realm of natural language processing (NLP), various datasets are
tailored to specific aspects of the task under continual learning paradigms.
The dataset introduced in ExtendNER (Monaikul et al., 2021), exemplifies a
continual learning approach to Named Entity Recognition (NER). This dataset
amalgamates the CoNLL-03 English NER (Sang and De Meulder, 2003) and OntoNotes
(Hovy et al., 2006), covering a broad spectrum of entity types and sources.
This hybrid dataset is structured to challenge the adaptability and
generalization capabilities of NER systems across varied contexts. Unlike the
static nature of text in NER tasks, the Schema-Guided Dialog (SGD) (Rastogi et
al., 2020) dataset, utilized in C-PT (Zhu et al., 2022), serves the Dialog
State Tracking aspect of IE, which involves maintaining the context of a
dialog over time. The SGD dataset features 44 services across 19 domains, each
treated as a separate task, and is designed to evaluate models on their
ability to manage and extract information across conversational turns. Lastly,
the lifelong SimpleQuestions and lifelong FewRel datasets, devised in (Wang et
al., 2019b) is crafted for the task of relation extraction. It merges elements
from the SimpleQuestions (Bordes et al., 2015) and FewRel (Han et al., 2018)
to form a lifelong learning benchmark that confronts the challenges of
relation detection in a few-shot context.

####  6.1.4. Datasets for Continual Pre-training.

In the realm of continual pre-training for large language models (LMs), the
development and utilization of specialized benchmarks play a pivotal role in
evaluating and enhancing the effectiveness of continual learning systems. The
dataset, introduced in CPT (Ke et al., 2022), primarily focuses on the
continual post-training of LMs across a series of domain-specific, unlabeled
datasets. It provides a rigorous test environment by using diverse corpora
such as Yelp Restaurant Reviews (Xu et al., 2019), AI and ACL Papers (Lo et
al., 2020), and AGNews articles (Zhang et al., 2015). Its main objective is to
gauge how well an LM can incrementally integrate domain-specific knowledge
without forgetting previously learned information, thereby enhancing its few-
shot learning capabilities in these domains. Contrary to the datasets employed
in CPT (Ke et al., 2022), which evaluate domain-specific adaptability and
incremental learning, the CKL benchmark (Jang et al., 2022b) is meticulously
designed to measure the LM’s ability to retain timeless knowledge, update
obsolete information, and acquire new knowledge. It comprises subsets like
INVARIANTLAMA, UPDATEDLAMA, and NEWLAMA, which are crafted to probe specific
types of knowledge that an LM may encounter in its learning trajectory.

Whereas the aforementioned two datasets assess more controlled dimensions of
knowledge integration and retention, the dataset introduced in ELLE (Qin et
al., 2022) focuses on the dynamic scenario of accumulating streaming data from
diverse sources in a lifelong learning context. This dataset mirrors the real-
world challenge of a language model (LM) that must continuously adapt to new
data inflows from multiple domains, including BOOKCORPUS (WB) (Zhu et al.,
2015), NEWS ARTICLES (NS) (Zellers et al., 2019), AMAZON REVIEWS (REV) (He and
McAuley, 2016), BIOMEDICAL PAPERS (BIO) (Lo et al., 2020) and COMPUTER SCIENCE
PAPERS (CS) (Lo et al., 2020). The benchmark evaluates the LM’s capacity to
effectively integrate new information from these varied sources over time,
highlighting the essential need for LMs to evolve in response to continual
data growth and shifts in data distribution. Jin et al. (Jin et al., 2022)
construct data streams to represent two prevalent types of domain shifts
observed in practical scenarios. The first, a Domain-incremental Paper Stream,
simulates the sequential evolution of research areas within academic papers,
encompassing diverse disciplines such as biomedical and computer science. The
second, a Chronologically-ordered Tweet Stream, models the temporal
progression of tweets over time.

####  6.1.5. Datasets for Hybrid Tasks.

An increasing number of datasets are adopting a hybrid task approach that
integrates multiple learning paradigms and task types, aimed at testing and
enhancing the adaptability of models. A notable example is the dataset
introduced in AdapterCL (Madotto et al., 2020), which is tailored for task-
oriented dialogue systems. This dataset incorporates four task-oriented
datasets: TaskMaster 2019 (TM19) (Byrne et al., 2019), TaskMaster 2020 (TM20)
(Byrne et al., 2019), Schema Guided Dialogue (SGD) (Rastogi et al., 2020), and
MultiWoZ (Budzianowski et al., 2018). These datasets have been pre-processed
to form a curriculum encompassing 37 domains, structured under four continual
learning settings: INTENT classification, Dialogue State Tracking (DST),
Natural Language Generation (NLG), and end-to-end (E2E) modeling.

Continual Instruction Tuning Benchmark (CITB) (Zhang et al., 2023a) extends
the concept of continual learning by focusing on instruction-based NLP tasks.
Built on the comprehensive SuperNI (Wang et al., 2022c) dataset, it includes
over 1,600 tasks across diverse NLP categories. CITB differentiates itself by
formulating two distinct streams—InstrDialog and InstrDialog++—to examine how
models integrate and retain new dialogue-oriented and varied NLP tasks under
continual learning settings. This benchmark suite not only tests task
retention and adaptability but also explores how instruction tuning can be
optimized for a continual learning framework. The ConTinTin (Yin et al., 2022)
is an adaptation of the NATURAL-INSTRUCTIONS dataset, specifically
restructured to facilitate a continual learning framework. This adaptation
involves decomposing the original crowdsourcing instructions into smaller,
distinct sub-tasks to create a new dataset. Additionally, the new dataset
incorporates a novel experimental design where tasks are selected randomly to
create diverse sequences, enabling the evaluation of a model’s adaptability to
novel instructions without prior exposure.

The dataset, used in Conure (Yuan et al., 2021), consists of Tencent TL (TTL)
(Yuan et al., 2020) and Movielens (ML). The TTL dataset is designed to address
three item recommendation tasks and three user profiling tasks, whereas the ML
dataset exclusively focuses on three item recommendation tasks. Both datasets
have been pre-processed to facilitate a continual learning framework,
simulating environments where models must adapt to evolving data streams.
Furthermore, Kim et al. (Kim et al., 2023) introduced the proprietary NAVER
Shopping dataset, which builds upon the previously mentioned datasets. The
NAVER Shopping dataset features six tasks: two for search query prediction,
two for purchased item category prediction, and two for user profiling, all
designed to meet real-world industry requirements.

Finally, the TRACE dataset, introduced by Wang et al. (Wang et al., 2024a), is
specifically designed to bridge the existing gap in the evaluation of large
language models (LLMs) within the continual learning framework, encompassing a
wide range of complex and specialized tasks. Distinguished from other
datasets, TRACE targets domain-specific tasks that are multilingual and
technical, including code completion and mathematical reasoning. This
diversity presents a unique set of challenges that span both specialized and
broad dimensions. Moreover, TRACE rigorously assesses the models’ ability to
sustain performance across tasks that demand different knowledge bases and
cognitive skills. This evaluation highlights the essential need for
adaptability in LLMs trained to operate under continual learning conditions,
underscoring their potential in dynamic real-world applications.

###  6.2. Online Datasets for NLP

####  6.2.1. Datasets for Classification.

The foundational text classification benchmark, as introduced by Zhang et al.
(Zhang et al., 2015) has traditionally been applied in offline continual
learning settings. Recent advancements have adapted this benchmark for online
continual learning, notably in studies such as MBPA++ (de Masson D’Autume et
al., 2019) and OML-ER (Holla et al., 2020)

####  6.2.2. Datasets for Generation.

The dataset, used in MBPA++ (de Masson D’Autume et al., 2019), comprises three
distinct question-answering collections: SQuAD 1.1 (Rajpurkar et al., 2016),
TriviaQA (Joshi et al., 2017), and QuAC (Choi et al., 2018). SQuAD 1.1 is a
reading comprehension dataset based on Wikipedia articles, designed to assess
the ability to derive answers from structured text. TriviaQA consists of
question-answer pairs developed by trivia enthusiasts, accompanied by
corroborative evidence sourced from both the web and Wikipedia, testing the
model’s capability to handle diverse information sources. QuAC adopts a
dialog-style format in which a student queries about information in a
Wikipedia article and a teacher responds using text directly from the article,
challenging the model’s interactive response generation.

####  6.2.3. Datasets for Information Extraction.

The lifelong relation extraction benchmark, used in OML-ER (Holla et al.,
2020), is structured by Wang et al. (Wang et al., 2019b) based on FewRel.
Unlike the original application by Wang et al., the benchmark in OML-ER is
adapted for online continuous learning scenarios.

####  6.2.4. Datasets for Other Tasks.

Hu et al. (Hu et al., 2022) compile the Firehose dataset, consisting of 110
million tweets from over 920,000 users between January 2013 and September
2019. This dataset is split into FIREHOSE 10M and FIREHOSE 100M. TemporalWiki
(Jang et al., 2022a) addresses temporal misalignment by serving as a lifelong
benchmark that trains and evaluates LMs using consecutive snapshots of
Wikipedia and Wikidata. This methodology assists in assessing an LM’s capacity
to both retain previously acquired knowledge and assimilate new information
over time.

###  6.3. Offline CL Datasets for Multi-modal Tasks

The P9D dataset (Zhu et al., 2023) consists of over one million image-text
pairs from e-commerce data, organized into nine industry sector-based training
tasks. It includes 1,014,599 training pairs, 2,846 for cross-modal retrieval
tests, and 4,615 query pairs with 46,855 gallery pairs for multi-modal
retrieval. Qian et al. (Qian et al., 2023) introduce two novel benchmarks for
continual learning, namely CL-TDIUC and CL-VQA2.0, which are derived from the
TDIUC (Kafle and Kanan, 2017) and VQA2.0 (Goyal et al., 2017), respectively.
These benchmarks are categorized into three scenarios: the Continual Vision
Scenario, which deals with new visual scenes; the Continual Language Scenario,
focusing on new questions in existing scenes; and the Continual Vision-
Language Scenario, addressing changes in both questions and visuals. DKR (Cui
et al., 2024) comprises five benchmark datasets: MS-COCO Caption (MS-COCO)
(Lin et al., 2014), Flickr30K (Young et al., 2014), IAPR TC-12 (Grubinger et
al., 2006), ECommerce-T2I (EC) (Yang et al., 2021), and RSICD (Lu et al.,
2017). Furthermore, two experimental scenarios are established. The first
scenario involves a sequential processing of the datasets, specifically MS-
COCO, Flickr30K, IAPR TC-12, EC, and RSICD, in that order. The second
scenario, which builds on the approach proposed by Ni et al. (Ni et al.,
2023), partitions the EC dataset into five sub-datasets for the training
phase. The model’s performance is subsequently tested on the Flickr30K, MS-
COCO, and EC datasets.

###  6.4. Offline and Online Datasets for Other Tasks

In this paper, we provide a comprehensive review of recent studies in natural
language processing (NLP) and multi-modal tasks, with a particular focus on
the use of PLMs, LLMs and VLMs. Additionally, we introduce a range of offline
continual learning (CL) datasets employed across various applications,
including automatic speech recognition (Vander Eeckt et al., 2023; Della
Libera et al., 2023), autonomous driving (Verwimp et al., 2023), disease
classification (Derakhshani et al., 2022), reinforcement learning (Wołczyk et
al., 2021), graph data (Ko et al., 2022), and computer vision (Faber et al.,
2023; Lin et al., 2021). Furthermore, some online CL benchmarks are also
proposed for computer version tasks, such as continual visual learning (Cai et
al., 2021), continual object detection (Wang et al., 2021b).

Figure 9. Illustration of calculating metrics.

##  7\. Metrics

In this section, we review the principal metrics commonly used to evaluate
continual learning. These metrics can be categorized into three main types:
(1) overall performance, which assesses the algorithm’s effectiveness across
all tasks; (2) memory stability, which measures the extent to which an
algorithm retains previously acquired knowledge; and (3) learning plasticity,
which evaluates the algorithm’s capacity to acquire new skills or knowledge.
Each of these metrics provides insights into different aspects of the
algorithm’s performance in a continual learning context.

To begin, we establish the notation (Figure 9) used throughout the learning
and evaluation phases of the model. Once the model completes a learning task,
denoted as $T_{i}$, it evaluates its performance on a test set that
encompasses all $N$ tasks, where $N$ is the total number of tasks in the set
$T$. This evaluation is represented by a matrix $R\in\mathbb{R}^{N\times N}$,
wherein each element $R_{i,j}$ indicates the model’s test classification
accuracy on task $T_{j}$ after training on task $T_{i}$.

###  7.1. Overall Performance.

The metric termed “Last” (Lopez-Paz and Ranzato, 2017; Zheng et al., 2023)
evaluates the overall performance of a continual learning (CL) method upon the
completion of all tasks. Specifically, it computes the average score from the
last row in the performance matrix $R$.

(1) $Last=\frac{1}{N}\sum_{i=1}^{N}R_{N,i}$

Also, Zheng et al. (Zheng et al., 2023) devise the “Avg” score metric, which
computes the mean accuracy across all datasets and timestamps.

(2)
$Avg=\frac{1}{N}\sum_{i=1}^{N}\left(\frac{1}{N}\sum_{j=1}^{N}R_{i,j}\right)$

In the seminal works of Rebuffi et al. (Rebuffi et al., 2017) and Douillard et
al. (Douillard et al., 2020), the concept of Average Incremental Accuracy
(AIA) is introduced. This metric is specifically designed to quantify the
historical performance across different tasks. It calculates the average
performance for each task by considering the lower triangular portion of the
matrix $R$, effectively capturing the evolving competence of the system as new
tasks are learned.

(3)
$AIA=\frac{1}{N}\sum_{i=1}^{N}\left(\frac{1}{i}\sum_{j=1}^{i}R_{i,j}\right)$

The metric, termed Transfer, is derived by computing the average of the
performance values for tasks that are represented in the upper-right triangle
of matrix $R$. This approach uniformly weights each dataset by averaging their
performance across different tasks, thereby assessing the preservation of
zero-shot transfer capabilities. Prior to commencing learning on task $T_{i}$,
no fine-tuning is performed on tasks that precede $T_{i}$.

(4)
$Transfer=\frac{1}{N-1}\sum_{i=2}^{N}\left(\frac{1}{i-1}\sum_{j=1}^{i-1}R_{j,i}\right)$

Moreover, Chaudhry et al. (Chaudhry et al., 2019) devise a metric known as
Learning Curve Area (LCA), which quantifies the speed of learning in a model.
Qin et al. (Qin et al., 2022) propose two metrics designed to evaluate pre-
trained language models (PLMs) based on their performance within learned
domains: Average Perplexity ($AP$) and Average Increased Perplexity
($AP^{+}$).

###  7.2. Memory Stability.

Memory stability is typically evaluated by backward transfer (BWT) (Lopez-Paz
and Ranzato, 2017) and forgetting measure (FM) (Chaudhry et al., 2018).

Backward Transfer (BWT) emerges as a pivotal concept extensively documented in
the literature, notably by Lopez et al. (Lopez-Paz and Ranzato, 2017) and Wu
et al. (Wu et al., 2023). BWT measures the performance degradation on
previously mastered tasks after the model is trained on new tasks. This
performance degradation phenomenon is often referred to as “forgetting”.

(5) $BWT=\frac{1}{N-1}\sum_{i=1}^{N-1}R_{N,i}-R_{i,i}$

Additionally, Chaudhry et al. (Chaudhry et al., 2018) introduce the Forgetting
Measure (FM), a metric designed to quantify the extent of forgetting a model
experiences for a specific task. A lower FM indicates better retention of
previous tasks. Davari et al. (Davari et al., 2022) propose a method named
linear probes (LP) to assess representation forgetting. This approach measures
the effectiveness of learned representations via an optimal linear classifier
trained on the frozen activations of a base network. Representation forgetting
is quantified by evaluating the change in Language Processing (LP) performance
before and after the introduction of a new task. Kemker et al. (Kemker et al.,
2018) introduce three metrics, where $\Omega_{\text{base}}$ assesses retention
of initial learning, $\Omega_{\text{new}}$ measures recall of new tasks, and
$\Omega_{\text{all}}$ evaluates overall proficiency in maintaining old
knowledge and acquiring new information. Additionally, researchers (Koh et
al., 2022) devise a novel metric, termed the Knowledge Loss Ratio (KLR),
quantifies knowledge degradation using principles from information theory.

###  7.3. Learning Plasticity.

Evaluating learning plasticity can be effectively accomplished through two key
metrics: forward transfer (FWT) (Lopez-Paz and Ranzato, 2017) and
intransigence measure (IM) (Chaudhry et al., 2018).

Forward Transfer (FWT) (Lopez-Paz and Ranzato, 2017) assesses the beneficial
effects on the performance of subsequent tasks following a model’s training on
prior tasks.

(6) $FWT=\frac{1}{N-1}\sum_{i=2}^{N}R_{i-1,i}-R_{0,i}$

where $R_{0,i}$ denotes the performance metric associated with training on
task $i$ independently. Higher values of FWT indicate superior model
performance. It is important to note that discussing backward transfer for the
initial task is not applicable, as there are no preceding tasks to influence
its performance.

Intransigence measure (IM), as defined by Chaudhry et al. (Chaudhry et al.,
2018), quantifies a model’s inability to learn new tasks. This measure is
calculated by comparing the performance difference of a task when trained
jointly with other tasks versus when trained in a continual learning setting.
Moreover, Koh et al. (Koh et al., 2022) introduce novel metrics, known as
Knowledge Gain Ratio (KGR), which quantifies the capacity to acquire new
knowledge through the calculation of knowledge gain.

###  7.4. Metrics for Continual Pre-training.

CKL (Jang et al., 2022b) introduces a novel metric, named FUAR (FORGOTTEN /
(UPDATED + ACQUIRED) RATIO), which quantitatively measures the efficiency of
each CKL method. It calculates the number of instances of time-invariant
knowledge that a model forgets in order to learn or update one instance of new
knowledge. When FUAR is equal to 1.0, it signifies an equilibrium where one
time-invariant knowledge instance is forgotten on average to obtain a new or
updated knowledge instance.

###  7.5. Online CL-Specific Metrics.

Near-future accuracy (NFA) (Al Kader Hammoud et al., 2023) is introduced as a
novel evaluation metric for OCL problem. Unlike traditional evaluation methods
that assess models on immediately subsequent samples, NFA evaluates models on
samples slightly further into the future, using a minimal shift $S$. Such
operation can mitigate label correlation effects, which can adversely impact
the accuracy of model adaptability assessments. The smallest shift $S$ is
selected to ensure that the test sample aligns closely with the distribution
of recently observed training data. Yogatama et al. (Yogatama et al., 2019)
proposed a novel online codelength, inspired by prequential encoding (Blier
and Ollivier, 2018), to quantify how quickly an existing model can adapt to a
new task.

##  8\. Challenges and Further Work

##### Autonomous Continual Learning.

Most existing studies in the domain of continual learning assume static
datasets with known distributions in a relatively closed environment.
Moreover, these studies mainly focus on simple tasks (e.g., text
classification, sentiment analysis and intent classification) with clear
labels. These assumptions do not hold in real-world applications, where
environments continually evolve and introduce novel stimuli. A key challenge
is to develop continual learning models that operate effectively in complex,
noisy environments where clear labels are not always available, and task
domains frequently change. Liu et al. (Liu et al., 2023b) recently proposed
the SOLA framework to address these limitations by facilitating autonomous
adaptation in AI systems. Despite this progress, significant challenges remain
in enabling these systems to independently adjust to new, dynamic environments
without ongoing human oversight. Future research should focus on developing
algorithms capable of autonomously detecting and adapting to shifts in data
distribution, thereby improving the applicability of AI in dynamic real-world
scenarios.

##### Learning Knowledge from Conversation.

Traditional AI systems are typically trained on static data sets, which
starkly contrasts with human conversational learning that dynamically updates
knowledge through interaction (Liu and Mazumder, 2021). The challenge for AI
lies in transitioning from static data learning to more dynamic,
conversational engagements. The future direction in this area could involve
the development of models that mimic human conversational learning processes,
capable of context adaptation, new concept inference, and dynamic knowledge
application within ongoing interactions.

##### Multi-modal Continual Learning.

Continual learning research has predominantly concentrated on natural language
processing tasks such as sentiment analysis and text classification. Recent
studies have begun exploring basic multi-modal tasks, such as text-to-image
retrieval, text-image classification, and visual question answering. The
integration of diverse data types—textual, visual, and auditory—poses a
substantial challenge. Future studies should expand to more complex multi-
modal datasets and strive to devise methodologies that effectively synthesize
these varied modalities, thereby enhancing the model’s capability to maintain
continuous learning across different sensory inputs.

##### Privacy Protection in Continual Learning.

Privacy protection in continual learning systems poses a significant
challenge, particularly as these systems are designed to continuously update
and refine their models based on incoming data streams. Unlike traditional
static machine learning models, continual learning systems frequently access
and process sensitive data across different contexts and time periods, raising
substantial concerns about data confidentiality and user privacy. Effective
privacy-preserving mechanisms must be integrated into the architecture of
these systems to ensure that they do not inadvertently expose or misuse
personal data. Techniques such as differential privacy (Dwork, 2006),
federated learning (Zhang et al., 2021), and secure multi-party computation
(Goldreich, 1998) offer promising solutions by allowing models to learn from
decentralized data sources without needing to access the actual data directly.
Future research in continual learning should not only focus on enhancing
learning efficiency and adaptability but also prioritize the development of
robust frameworks that safeguard user privacy across all phases of data
handling and model updating.

##### Robust Continual Learning.

The existing studies mainly focus on designing a continual learning model to
improve the performance of forgetting and transferring with various metrics
while the robustness of continual learning systems is not well studied. It is
critical, especially in applications where safety and reliability are
paramount. The main challenges include evaluating the robustness of these
systems against adversarial attacks or when faced with drastically changing
environments. Future research could focus on developing evaluation metrics for
robustness in continual learning and designing systems that maintain
performance reliability over time despite environmental changes.

##### Large-Scale and High-Quality Datasets and Benchmarks.

As discussed in Section 6, most of the datasets are constructed by merging the
existing datasets. This often results in datasets that lack diversity and
real-world complexity, which hampers the development of robust and adaptable
continual learning models. The creation of large-scale, high-quality datasets
that accurately reflect real-world complexities represents a critical
challenge. Moving forward, the development of such datasets and benchmarks
will be essential not only for assessing the efficacy of continual learning
algorithms but also for pushing the limits of what these algorithms can
achieve in practical settings.

##  9\. Conclusions

This survey provides an in-depth exploration of continual learning (CL)
methodologies tailored for foundation language models (LMs), such as pre-
trained language models (PLMs), large language models (LLMs), and vision-
language models (VLMs). By integrating the dynamic adaptability of CL with the
robust foundational capabilities of LMs, this field promises to significantly
advance the state of artificial intelligence. We categorize existing research
into offline and online continual learning paradigms, offering a clear
distinction between the settings and methodologies used within these
frameworks. Offline CL is discussed in terms of domain-incremental, task-
incremental, and class-incremental learning. Meanwhile, online CL is analyzed
with a focus on the delineation between hard and blurry task boundaries,
providing insights into how these approaches handle real-time data streams.
Our review of the literature not only clarifies the current landscape of CL
approaches for foundation LMs but also emphasizes the innovative integration
of continual pre-training, parameter-efficient tuning, and instruction tuning
methods that are specifically designed to leverage the vast capabilities of
foundation LMs. Furthermore, we highlight the main characteristics of datasets
used in this domain and the metrics that effectively measure both the
mitigation of catastrophic forgetting and the enhancement of knowledge
transfer. This work hopes to inspire further research that will ultimately
lead to more robust, efficient, and intelligent systems capable of lifelong
learning.

##

  * (1)
  * Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023\.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_ (2023).
  * Al Kader Hammoud et al. (2023) Hasan Abed Al Kader Hammoud, Ameya Prabhu, Ser-Nam Lim, Philip HS Torr, Adel Bibi, and Bernard Ghanem. 2023.  Rapid Adaptation in Online Continual Learning: Are We Evaluating It Right?. In _ICCV_. 18852–18861.
  * Aljundi et al. (2019) Rahaf Aljundi, Eugene Belilovsky, Tinne Tuytelaars, Laurent Charlin, Massimo Caccia, Min Lin, and Lucas Page-Caccia. 2019.  Online Continual Learning with Maximal Interfered Retrieval. In _NeurIPS_ , Vol. 32.
  * Bai et al. (2022) Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022\.  Training a helpful and harmless assistant with reinforcement learning from human feedback.  _arXiv preprint arXiv:2204.05862_ (2022).
  * Bang et al. (2022) Jihwan Bang, Hyunseo Koh, Seulki Park, Hwanjun Song, Jung-Woo Ha, and Jonghyun Choi. 2022.  Online continual learning on a contaminated data stream with blurry task boundaries. In _CVPR_. 9275–9284.
  * Beaulieu et al. (2020) Shawn Beaulieu, Lapo Frati, Thomas Miconi, Joel Lehman, Kenneth O Stanley, Jeff Clune, and Nick Cheney. 2020.  Learning to continually learn.  _arXiv preprint arXiv:2002.09571_ (2020).
  * Belouadah et al. (2021) Eden Belouadah, Adrian Popescu, and Ioannis Kanellos. 2021.  A comprehensive study of class incremental learning algorithms for visual tasks.  _Neural Networks_ 135 (2021), 38–54.
  * Biesialska et al. (2020) Magdalena Biesialska, Katarzyna Biesialska, and Marta R Costa-jussà. 2020.  Continual Lifelong Learning in Natural Language Processing: A Survey. In _COLING_. 6523–6541.
  * Bin Tareaf (2017) Raad Bin Tareaf. 2017.  Tweets Dataset - Top 20 most followed users in Twitter social platform.
  * Blier and Ollivier (2018) Léonard Blier and Yann Ollivier. 2018.  The description length of deep learning models.  _NeurIPS_ 31 (2018).
  * Bordes et al. (2015) Antoine Bordes, Nicolas Usunier, Sumit Chopra, and Jason Weston. 2015.  Large-scale simple question answering with memory networks.  _arXiv preprint arXiv:1506.02075_ (2015).
  * Budzianowski et al. (2018) Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. 2018.  MultiWOZ–a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling.  _arXiv preprint arXiv:1810.00278_ (2018).
  * Byrne et al. (2019) Bill Byrne, Karthik Krishnamoorthi, Chinnadhurai Sankar, Arvind Neelakantan, Daniel Duckworth, Semih Yavuz, Ben Goodrich, Amit Dubey, Andy Cedilnik, and Kyu-Young Kim. 2019.  Taskmaster-1: Toward a realistic and diverse dialog dataset.  _arXiv preprint arXiv:1909.05358_ (2019).
  * Caccia et al. (2021) Lucas Caccia, Rahaf Aljundi, Nader Asadi, Tinne Tuytelaars, Joelle Pineau, and Eugene Belilovsky. 2021.  New insights on reducing abrupt representation change in online continual learning.  _arXiv preprint arXiv:2104.05025_ (2021).
  * Cahyawijaya et al. (2023) Samuel Cahyawijaya, Holy Lovenia, Tiezheng Yu, Willy Chung, and Pascale Fung. 2023.  InstructAlign: High-and-Low Resource Language Alignment via Continual Crosslingual Instruction Tuning. In _the First Workshop in South East Asian Language Processing_. 55–78.
  * Cai et al. (2021) Zhipeng Cai, Ozan Sener, and Vladlen Koltun. 2021.  Online continual learning with natural distribution shifts: An empirical study with visual data. In _ICCV_. 8281–8290.
  * Camburu et al. (2018) Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. 2018.  e-snli: Natural language inference with natural language explanations.  _NeurIPS_ 31 (2018).
  * Cao et al. (2024) Xusheng Cao, Haori Lu, Linlan Huang, Xialei Liu, and Ming-Ming Cheng. 2024.  Generative Multi-modal Models are Good Class-Incremental Learners.  _arXiv preprint arXiv:2403.18383_ (2024).
  * Casanueva et al. (2020) Iñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. 2020.  Efficient intent detection with dual sentence encoders.  _arXiv preprint arXiv:2003.04807_ (2020).
  * Castellucci et al. (2021) Giuseppe Castellucci, Simone Filice, Danilo Croce, and Roberto Basili. 2021.  Learning to Solve NLP Tasks in an Incremental Number of Languages. In _ACL_. 837–847.
  * Chaudhry et al. (2018) Arslan Chaudhry, Puneet K Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. 2018.  Riemannian walk for incremental learning: Understanding forgetting and intransigence. In _ECCV_. 532–547.
  * Chaudhry et al. (2019) Arslan Chaudhry, Marc’Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. 2019.  Efficient Lifelong Learning with A-GEM. In _ICLR_.
  * Chen et al. (2024) Cheng Chen, Junchen Zhu, Xu Luo, Hengtao Shen, Lianli Gao, and Jingkuan Song. 2024.  CoIN: A Benchmark of Continual Instruction tuNing for Multimodel Large Language Model.  _arXiv preprint arXiv:2403.08350_ (2024).
  * Chen et al. (2020) Hung-Jen Chen, An-Chieh Cheng, Da-Cheng Juan, Wei Wei, and Min Sun. 2020.  Mitigating forgetting in online continual learning via instance-aware parameterization.  _NeurIPS_ 33 (2020), 17466–17477.
  * Chen et al. (2023) Wuyang Chen, Yanqi Zhou, Nan Du, Yanping Huang, James Laudon, Zhifeng Chen, and Claire Cui. 2023.  Lifelong language pretraining with distribution-specialized experts. In _ICML_. PMLR, 5383–5395.
  * Cheng et al. (2023a) Daixuan Cheng, Shaohan Huang, and Furu Wei. 2023a.  Adapting large language models via reading comprehension.  _arXiv preprint arXiv:2309.09530_ (2023).
  * Cheng et al. (2023b) Xin Cheng, Yankai Lin, Xiuying Chen, Dongyan Zhao, and Rui Yan. 2023b.  Decouple knowledge from paramters for plug-and-play language modeling. In _ACL_. 14288–14308.
  * Choi et al. (2018) Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018.  QuAC: Question Answering in Context. In _EMNLP_. 2174–2184.
  * Chrysakis and Moens (2023) Aristotelis Chrysakis and Marie-Francine Moens. 2023.  Online bias correction for task-free continual learning.  _ICLR_ (2023).
  * Cossu et al. (2022) Andrea Cossu, Tinne Tuytelaars, Antonio Carta, Lucia Passaro, Vincenzo Lomonaco, and Davide Bacciu. 2022.  Continual pre-training mitigates forgetting in language and vision.  _arXiv preprint arXiv:2205.09357_ (2022).
  * Coucke et al. (2018) Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Caltagirone, Thibaut Lavril, et al. 2018\.  Snips voice platform: an embedded spoken language understanding system for private-by-design voice interfaces.  _arXiv preprint arXiv:1805.10190_ (2018).
  * Cui et al. (2024) Zhenyu Cui, Yuxin Peng, Xun Wang, Manyu Zhu, and Jiahuan Zhou. 2024.  Continual Vision-Language Retrieval via Dynamic Knowledge Rectification. In _AAAI_ , Vol. 38. 11704–11712.
  * Davari et al. (2022) MohammadReza Davari, Nader Asadi, Sudhir Mudur, Rahaf Aljundi, and Eugene Belilovsky. 2022.  Probing representation forgetting in supervised and unsupervised continual learning. In _CVPR_. 16712–16721.
  * De Lange et al. (2021) Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. 2021.  A continual learning survey: Defying forgetting in classification tasks.  _TPAMI_ 44, 7 (2021), 3366–3385.
  * de Masson D’Autume et al. (2019) Cyprien de Masson D’Autume, Sebastian Ruder, Lingpeng Kong, and Dani Yogatama. 2019.  Episodic memory in lifelong language learning.  _NeurIPS_ 32 (2019).
  * Della Libera et al. (2023) Luca Della Libera, Pooneh Mousavi, Salah Zaiem, Cem Subakan, and Mirco Ravanelli. 2023.  CL-MASR: A Continual Learning Benchmark for Multilingual ASR.  _arXiv preprint arXiv:2310.16931_ (2023).
  * Derakhshani et al. (2022) Mohammad Mahdi Derakhshani, Ivona Najdenkoska, Tom van Sonsbeek, Xiantong Zhen, Dwarikanath Mahapatra, Marcel Worring, and Cees GM Snoek. 2022.  Lifelonger: A benchmark for continual disease classification. In _International Conference on Medical Image Computing and Computer-Assisted Intervention_. 314–324.
  * Ding et al. (2008) Xiaowen Ding, Bing Liu, and Philip S Yu. 2008.  A holistic lexicon-based approach to opinion mining. In _WSDM_. 231–240.