# Towards Automated Metamorphic Test Identification for Ocean System Models

1st J Hiremath, Dilip  MarDATA – Helmholtz School for Marine Data Science
GEOMAR Helmholtz Centre for Ocean Research Kiel
& Kiel University
Kiel, Germany
0000-0001-7095-6583  2nd Claus, Martin  Ocean Circulation and Climate Dynamics
Department
GEOMAR Helmholtz Centre for Ocean Research Kiel
& Kiel University
Kiel, Germany
0000-0002-7525-5134  3rd Hasselbring, Wilhelm  Software Engineering Group,
Computer Science Department
Kiel University
Kiel, Germany
0000-0001-6625-4335  4th Rath, Willi  Ocean Circulation and Climate Dynamics
Department
GEOMAR Helmholtz Centre for Ocean Research Kiel
Kiel, Germany
0000-0003-1951-8494

###### Abstract

Metamorphic testing seeks to verify software in the absence of test oracles.
Our application domain is ocean system modeling, where test oracles rarely
exist, but where symmetries of the simulated physical systems are known. The
input data set is large owing to the requirements of the application domain.

This paper presents work in progress for the automated generation of
metamorphic test scenarios using machine learning. We extended our previously
proposed method [1] to identify metamorphic relations with reduced
computational complexity. Initially, we represent metamorphic relations as
identity maps. We construct a cost function that minimizes for identifying a
metamorphic relation orthogonal to previously found metamorphic relations and
penalize for the identity map. A machine learning algorithm is used to
identify all possible metamorphic relations minimizing the defined cost
function. We propose applying dimensionality reduction techniques to identify
attributes in the input which have high variance among the identified
metamorphic relations. We apply mutation on these selected attributes to
identify distinct metamorphic relations with reduced computational complexity.
For experimental evaluation, we subject the two implementations of an ocean-
modeling application to the proposed method to present the use of metamorphic
relations to test the two implementations of this application.

###### Index Terms:

Metamorphic testing, Ocean System Models testing, Oracle problem, Metamorphic
relation, Test case generation, Software testing.

##  I Introduction

Software quality assurance forms an important part of the software development
cycle. Its importance is accepted throughout the software engineering
community. Verification and validation of an application are essential steps
in software testing. The increasing complexity of present-day software and the
use of machine learning techniques presents new challenges to test software
sufficiently. However, machine learning techniques are used in research and in
developing solutions for real-world applications [2, 3]. Besides this, the
nature of research software is exploratory. The output for such software is
usually unknown and cost-intensive to compute. The unknown output of the
application is known as the test oracle problem in software engineering.

There have been many approaches to test software that pose the test oracle
problem. Metamorphic testing (MT) is the most prominent approach among these,
as seen by the increase in number of papers published and increased research
efforts in recent times. MT was first introduced by [4]. It has seen an
increase adoption in research and real-world applications. Comprehensive
surveys of metamorphic testing may be found in [5, 6, 7, 8, 9, 10].

Metamorphic testing is based on the idea that most of the time it is easier to
predict relations between outputs of a program, than understanding its input-
output behavior [11]. The central element of metamorphic testing is the
metamorphic relation (MR). An MR is a necessary property of the target
application in relation to multiple inputs and their corresponding outputs
[12].

Identifying metamorphic relations is labor-, time-, and cost-intensive.
Experienced domain experts require a thorough understanding of the application
and the desired behavior of MRs to identify metamorphic test scenarios.
Automating this process would reduce the cost and increase the probability of
widespread adoption. Using the available automated test case generation
methods, we can develop test cases once the MRs are available as shown by Hui
et al. [13]. We can harness testing frameworks to automate application testing
using the generated test cases and test suites.

MT is not without its limitations. A major limit is that MT by itself cannot
prove that the output of the application is correct [8]. It is also
challenging to quantify the effectiveness of MT. Since the basis of MT relies
on comparing multiple successive outputs with respect to each other and the
morphed inputs. If the initial outputs themselves were wrong and the
successive outputs were within an expected variation satisfying the MR, MT
would not detect the error in the software under test. The other limitation of
the MT approach is that for comprehensive testing the number of MRs required
is large and thus leads to an extensive test suite. MT finds faults in the
application but does not point to the function with the bug, thus debugging
the software under test is challenging. One method is to follow the changes to
input parameters for failed metamorphic tests, find the parameters with high
covariance, and investigate their path for debugging.

Section II explains the motivation behind our research. In Section III we
introducing an example application, before formulating the problem to be
solved in Section IV. In Section V, we explain our proposed method and sketch
the solution employing machine learning. Section VI of the paper highlights
the challenges and an outlook to future work. Section VII provides the details
of the resources available for this paper.

##  II Motivation

Our application domain are ocean system models. Currently, there are no papers
based on MT focused on ocean modeling, to the best of our knowledge. However,
there are a few papers on applying MT to geographic information systems [7,
13] Ocean system modeling software is often based on legacy code that has over
the years undergone multiple development cycles leading to complex intertwined
code. They rarely employ a standard systematic testing approach to verify the
software. The challenges to this have been explained by [14], [15], and [16].
The focus is on assimilating functions that work rather than developing the
software adhering to the principles of software engineering. Verification is
based on the modeler’s undocumented plausibility check to see if the output
lies within the output range, the researcher is expecting. Though researchers
accept software testing as a value-adding step in developing software, they
rarely identify or maintain separate test cases to verify their functions and
software.

Some recent developments of ocean simulation software are collaborative, open-
source, and in newer programming languages. This has led to refactoring some
existing components and modular addition of new functions to the existing
codebase. Testing efforts are focused on regression tests as part of the
continuous integration framework in most collaborative open-source scientific
software development [17]. A goal should be to follow the FAIR principles not
only for research data but also for research software [18].

The increasing use of machine learning techniques to compute intermediate
input data points demands non-traditional software techniques to ensure the
correctness of the software in use. There is a substantial gap between
adopting artificial intelligence in software development and testing
capabilities for valid implementations of artificial intelligence
methodologies. Testing these systems has proven to be challenging. Though
there is an increase in the solution’s effectiveness by adopting these
techniques, the lack of methods to verify and the inability to explain the
improvement in effectiveness has resulted in cautious adoption of these
software development techniques. With the increasing assimilation of
artificial Intelligence methodologies as a part of software engineering, the
need for better testing approaches for validating software posing test oracle
problems has increased.

From our current knowledge, there does not exist an established standard
testing procedure for verifying the developed scientific software. Most of the
verification efforts are based on unit tests written, managed, and executed by
contributors at their discretion. A structured quality assurance framework
would help with the adoption of the testing capabilities across the research
facility and lead to more reliable software development. One of the major
hurdles to adopting MT is the cost associated with identifying metamorphic
relations. The central idea of this short paper is to present an approach to
automate identifying metamorphic tests. This will increase the widespread
application of MT as an efficient verification technique for the software that
poses test oracle problems.

The future of Software Engineering will revolve around harnessing artificial
intelligence techniques for solving real-world complex and interdependent
problems. Automated identification of metamorphic tests will lead to improved
adoption of MT-based test suit development for automated program repair as
shown by Jiang et al. [19].

##  III Example Application

We use a simple but realistic ocean-modeling application to show our approach
[20]. We can access the application on Binder [21]. The example application is
to calculate a time series of Kinetic energy of the surface ocean for randomly
generated data for a grid of ($10\times 20$) for $30$ temporal resolutions

$e(t,y,x)=\frac{1}{2}\frac{\int{\rm d}y{\rm
d}x(u(t,y,x)^{2}+v(t,y,x)^{2})}{\int{\rm d}y{\rm d}x}$ (1)

where $e$ represents the time series of Kinetic energy, $t$ the time, $y$ and
$x$ are spatial coordinates, and horizontal surface-velocities $u$ and $v$ are
calculated from the sea-level $\eta$ using

$(u,v)(t,y,x)=\frac{G}{F}\left(-\frac{\partial}{\partial
y},\frac{\partial}{\partial x}\right)\eta(t,y,x)$ (2)

where $G$ represents the gravitational acceleration (in $meters/second^{2}$)
and $F$ the Coriolis parameter (in $1/second$) at $30$° North.

The two functions of Equation (1) are coded in the sample application, one of
which is not respecting cyclic boundary conditions. To numerically implement
Equation (1), we discretize the integral in (1) and the partial derivatives in
(2) by applying the method of finite differences and link it back to Equation
(3). Thus, the input data $\vec{x}$ contains all discretized values of sea
level $\eta$, of all coordinates $t$, $y$, $x$, and the physical constants $G$
and $F$. A metamorphic transformation $g(x)$ could change all or some of the
atomic data points in $\vec{x}$ such that all $e(g(\vec{x}))=e(\vec{x})$ for
all discrete values of time $t$.

Figure 1: Time series outputs of Kinetic energy of surface ocean calculated
using the function not respecting the cyclic boundary conditions for (x,y)
coordinates for multiple applied MRs.

Fig.1 shows the time series output for multiple applied MRs calculated using
the function in which the cyclic boundary condition is not implemented. These
MRs were manually identified based on the symmetries of the energy equation.
The bottom most plot is that of the original input data. The applied MRs are
multiplying $g$ and $h$ by the same constant, Keeping $G/F$ as constant,
scaling $\eta,x$ and $y$ by a constant, transposing $x$ and $y$, reversing the
direction of $x$, reversing the direction $y$, transposing the $x$ coordinates
cyclically and transposing the $y$ coordinates cyclically. We observe that for
applied initial MRs the time series outputs are in accordance with the applied
MRs. Time series out puts for transposing the $x$ coordinates cyclically and
transposing the $y$ coordinates cyclically differs from the rest as
highlighted in Fig. 1 by red arrows.

##  IV Problem Formulation

For the given application domain, we may define the function under test
$f:X\to Y$, where both $X$ and $Y$ are finite dimensional normed vector
spaces. Then a metamorphic relation $(g,h)$ satisfies

$f(g(\vec{x}))=h(f(\vec{x}))$ (3)

where $\vec{x}\in X$, $g:X\to X$ is the input mapping from source input to
follow-up (morphed) input, and $h:Y\to Y$ is the output mapping from the
source output to the follow-up output, that is the output associated with the
morphed input.

Throughout this paper, we limit $h$ to be the identity map

$f(g(\vec{x}))=f(\vec{x})$ (4)

and we constrain $g$ to be an affine transformation

$\displaystyle g(\vec{x})=\Gamma\cdot\vec{x}+\vec{\beta}$ (5)

where $\Gamma$ is a matrix associated with an endormophism of $X$, and
$\vec{\beta}\in X$ is an offset. Note that the limitation of $g$ being an
affine transformation is motivated by retaining the ability of capturing
physical properties related to symmetries of the governing equations, such as
invariance under translation or rotation, while greatly reducing the soultion
space.

##  V Proposed Method

There may exist an infinite number of possible $g$ satisfying (4) but one
always is the identity map. The task is then to find new $g$, or at least
approximations to $g$, which differ from all $g$ known so far. For this we use
an iterative approach. We represent the set of identified metamorphic
relations associated with $f$ as

$G^{f}_{n-1}=\\{g_{0},g_{1},\ldots,g_{n-1}\\}$ (6)

We seek to find a new possible $g_{n}$ by minimizing a cost function that
rewards a $g_{n}$ which minimizes the distance between source output and
follow-up output $|f(g_{n}(\vec{x}))-f(\vec{x})|$ and penalizes if $g_{n}$ is
already known. Throughout the text, we take the norm $\left|\cdot\right|$ to
be the Euclidean distance. The cost function for a candidate $g_{n}$ and a
given $G^{f}_{n-1}$ is defined by

$J(g_{n},G^{f}_{n-1})=\int_{X}\frac{|f(g_{n}(\vec{x}))-f(\vec{x})|}{\epsilon+\prod_{g\in
G^{f}_{n-1}}\left|g_{n}(\vec{x})-g(\vec{x})\right|^{2}}dx$ (7)

where $g_{n}$ represents a potential morphing to $\vec{x}$ such that it
satisfies (4) and $\epsilon$ is a small machine precision constant to avoid
division by zero when $g_{n}$ is identical to a previously identified $g\in
G^{f}_{n-1}$. Note that in the denomiator we write the product of squared
differences to ensure $\lim_{g_{n}\to g}J(g_{n},G^{f}_{n-1})=\infty$ which
heavily penalizes $g_{n}$ for being close to an element of $G^{f}_{n-1}$.

We apply a modified algorithm loosely based on the genetic algorithm with a
Monte Carlo optimization to find a $g_{n}$ that minimizes the cost function
(7). We start the iteration with $G^{f}_{0}=\\{\mathrm{id}_{X}\\}$ containing
only the identity map on $X$. Initially, we assign random values to
$\\{\Gamma,\vec{\beta}\\}$ and in each step a mutation
$\\{\delta\Gamma,\vec{\delta\beta}\\}$ is proposed. An acceptance threshold
$p$ is set which determines at what rate a mutation that would increase the
cost function is still not rejected. If the cost after the application of the
mutation is smaller than before the mutation, the applied mutation is always
accepted. The process is repeated to move towards a global minimum. After
converging to a minimum, we update the solution set
$G^{f}_{n}=G^{f}_{n-1}\cap\\{g_{n}\\}$. Then a new $g_{n+1}$ is created with
randomly assigned $\\{\Gamma,\vec{\beta}\\}$ and the process is restarted,
this time minimizing $J(g_{n+1},G^{f}_{n})$.

With this method we are able to identify multiple metamorphic transformations
which satisfy equation (4).

##  VI Challenges and Future Work

We identify multiple MRs using the approach proposed in Section V. We suspect
the identified MRs are a chain of multiple elementary constituent MRs. Hence,
it will be challenging to explain identified MRs in terms of physical
symmetries that hold in the real oceans. We propose dimensionality reduction
techniques to decompose the identified chained MRs. The decomposed MRs will be
closer to the constraints of the Physical Ocean and easier to interpret. Since
the resulting MRs will be distinct after decomposing, we expect test cases
constructed with these MRs to be effective in uncovering defects. They will
also increase the test coverage of the application.

Since we assume input and output mapping relations, MRs corresponding to
functional states of the software are not identified by the method proposed in
Section V.

Further work is to relax the restriction to $h$ in (3) being the identity map.
Next steps will explore other optimization methods for the cost function
constructed in SectionV including deep learning frameworks.

##  VII Data Availability

The sample application presented in Section III and the data generated are
available in [20]. The published artifact contains a link to an interactive
session where the example application can be executed.

## Acknowledgments

The first author is funded through the Helmholtz School for Marine Data
Science (MarDATA), Grant No. HIDSS-0005. We have developed the approach in
collaboration of the Computer Science Department at Kiel University and the
Ocean Circulation and Climate Dynamics Department at the GEOMAR Helmholtz
Centre for Ocean Research Kiel.

## References

  * [1] Dilip J. Hiremath, Martin Claus, Wilhelm Hasselbring and Willi Rath  “Automated identification of metamorphic test scenarios for an ocean-modeling application”  In _Proceedings - 2020 IEEE International Conference on Artificial Intelligence Testing, AITest 2020_ Institute of ElectricalElectronics Engineers Inc., 2020, pp. 62–63  DOI: 10.1109/AITEST49225.2020.00016
  * [2] Markus Reichstein et al.  “Deep learning and process understanding for data-driven Earth system science”  In _Nature_ 566.7743 Springer US, 2019, pp. 195–204  DOI: 10.1038/s41586-019-0912-1
  * [3] Arne Johanson et al.  “Modeling Polyp Activity of Paragorgia Arborea Using Supervised Learning”  In _Ecological Informatics_ 39, 2017, pp. 109–118  DOI: 10.1016/j.ecoinf.2017.02.007
  * [4] TY Chen, SC Cheung and SM Yiu  “Metamorphic testing: a new approach for generating next test cases” arXiv:2002.12543, 2020
  * [5] Jie M. Zhang, Mark Harman, Lei Ma and Yang Liu  “Machine Learning Testing: Survey, Landscapes and Horizons”  In _IEEE Transactions on Software Engineering_ , 2020  DOI: 10.1109/tse.2019.2962027
  * [6] Sergio Segura, Dave Towey, Zhi Quan Zhou and Tsong Yueh Chen  “Metamorphic Testing: Testing the Untestable”  In _IEEE Software_ 37.3 IEEE Computer Society, 2020, pp. 46–53  DOI: 10.1109/MS.2018.2875968
  * [7] Xuanyi Lin et al.  “Exploratory Metamorphic Testing for Scientific Software”  In _Computing in Science and Engineering_ 22.2 IEEE Computer Society, 2020, pp. 78–87  DOI: 10.1109/MCSE.2018.2880577
  * [8] Sergio Segura, Dave Towey, Zhi Quan Zhou and T. Y. Chen  “An extended abstract of ”metamorphic testing: Testing the untestable””  In _Proceedings - International Computer Software and Applications Conference_ 1 IEEE, 2019, pp. 209–210  DOI: 10.1109/COMPSAC.2019.00037
  * [9] Sergio Segura and Zhi Quan Zhou  “Metamorphic testing 20 years later: A hands-on introduction”  In _Proceedings - International Conference on Software Engineering_ IEEE Computer Society, 2018, pp. 538–539  DOI: 10.1145/3183440.3183468
  * [10] Sergio Segura, Gordon Fraser, Ana B. Sanchez and Antonio Ruiz-Cortes  “A Survey on Metamorphic Testing”  In _IEEE Transactions on Software Engineering_ , 2016  DOI: 10.1109/TSE.2016.2532875
  * [11] Prashanta Saha  “Improving The Effectiveness of Automatically Generated Test”  In _Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops_ New York, NY, USA: ACM, 2020, pp. 418–419  DOI: 10.1145/3387940.3392253
  * [12] Tsong Yueh Chen et al.  “Metamorphic testing: A review of challenges and opportunities”  In _ACM Computing Surveys_ 51.1, 2018  DOI: 10.1145/3143561
  * [13] Zhan Wei Hui, Song Huang, Caslon Chua and Tsong Yueh Chen  “Semiautomated metamorphic testing approach for geographic information systems: An empirical study”  In _IEEE Transactions on Reliability_ 69.2 IEEE, 2020, pp. 657–673  DOI: 10.1109/TR.2019.2931561
  * [14] Arne Johanson and Wilhelm Hasselbring  “Software Engineering for Computational Science: Past, Present, Future”  In _Computing in Science and Engineering_ , 2018, pp. 1–31  DOI: 10.1109/MCSE.2018.108162940
  * [15] Upulee Kanewala and James M. Bieman  “Testing scientific software: A systematic literature review”  In _Information and Software Technology_ 56.10 Elsevier B.V., 2014, pp. 1219–1232  DOI: 10.1016/j.infsof.2014.05.006
  * [16] Diane F Kelly and Rebecca Sanders  “The Challenge of Testing Scientific Software”  In _3rd Annual Conference of the Association for Software Testing: Beyond the Boundaries: Interdisciplinary Approaches to Software Testing (CAST’08)_ , 2008, pp. 30–36  URL: http://www.cast2008.org
  * [17] Wilhelm Hasselbring et al.  “Open Source Research Software”  In _Computer_ 53.8, 2020  DOI: 10.1109/MC.2020.2998235
  * [18] Wilhelm Hasselbring et al.  “From FAIR research data toward FAIR and open research software”  In _it - Information Technology_ 62.1, 2020, pp. 39–47  DOI: 10.1515/itit-2019-0040
  * [19] Mingyue Jiang, Tsong Yueh Chen, Zhi Quan Zhou and Zuohua Ding  “Input Test Suites for Program Repair: A Novel Construction Method Based on Metamorphic Relations”  In _IEEE Transactions on Reliability_ , 2020, pp. 1–19  DOI: 10.1109/TR.24
  * [20] Willi Rath  “Artifacts for: Automated generation of metamorphic test scenarios for an ocean-modeling application”, 2019  DOI: 10.5281/zenodo.3566954
  * [21] Project Jupyter et al.  “Binder 2.0 - Reproducible, Interactive, Sharable Environments for Science at Scale.”  In _PROC_ , 2018, pp. 113


# MoRAL: MoE Augmented LoRA for LLMs’ Lifelong Learning

Shu Yang∗,1,2,3, Muhammad Asif Ali∗,1,2, Cheng-Long Wang1,2, Lijie Hu†,1,2,4,
and Di Wang†,1,2,4
1Provable Responsible AI and Data Analytics (PRADA) Lab
2King Abdullah University of Science and Technology
3University of Macau 4SDAIA-KAUST AI
∗Equal Contribution †Corresponding Author

###### Abstract

Adapting large language models (LLMs) to new domains/tasks and enabling them
to be efficient lifelong learners is a pivotal challenge. In this paper, we
propose MoRAL, i.e., Mixture-of-Experts augmented Low Rank Adaptation for
Lifelong learning. MoRAL combines the multi-tasking abilities of MoE with the
fine-tuning abilities of LoRA for effective life-long learning of LLMs. In
contrast to the conventional approaches that use factual triplets as inputs
MoRAL relies on simple question-answer pairs, which is a more practical and
effective strategy for robust and efficient learning. Owing to new data
settings, we introduce a new evaluation benchmark namely: Life Long Learning
of LLM (5L-bench) encompassing a newly curated dataset of question-answer
pairs, and a set of evaluation metrics for rigorous evaluation of MoRAL in
open-book and closed-book settings. Experimental evaluation shows (i) LLMs
learn fast in open-book settings with up to 30.15% improvement in "RA" for
Phi-2-2.7B compared to closed-book (for models fine-tuned with MoRAL); (ii)
MoRAL shows higher performance improvement for models with a greater number of
parameters; (iii) MoRAL is robust to catastrophic forgetting offering better
knowledge retention compared to baselines.

##  1 Introduction

Large language models (LLMs) trained using massive computational clusters and
expansive datasets, have demonstrated impressive proficiency in natural
language processing Zhao et al. (2023); Kaddour et al. (2023). These models
excel in a variety of downstream tasks, such as machine translation Zhu et al.
(2023a); Xu et al. (2024a), grammatical error correction Fang et al. (2023);
Wu et al. (2023a) etc. The success of LLM arises from the powerful knowledge
processing and compression capabilities Zhu et al. (2023b); Huang et al.
(2023); Delétang et al. (2023), which allow LLMs to construct information in a
way that is somehow similar to humans, and even complete never-before-seen
tasks Grosse et al. (2023); Kirk et al. (2024).

However, a significant challenge for LLMs is their restricted adaptability to
the latest available data/information, which restraints them to generate
responses about recent events thus leading to information gaps. This entails
hallucination, a phenomenon when LLM tries to generate plausible but incorrect
answers about unknown facts Rawte et al. (2023). An example in this regard is
shown in Figure 1, where ChatGPT-4 is unable to correctly answer a question
about Mistral 8x7B, a model recently released in Dec 2023. Addressing issues
such as outdated training data Zhang et al. (2023d), hallucination Zhang et
al. (2023c), and factual inaccuracies in LLMs Wang et al. (2023a) is not only
costly but also vulnerable to risks like model collapse Shumailov et al.
(2023) and catastrophic forgetting Luo et al. (2023). Adapting these models to
specific domains/tasks further intensifies these challenges Ling et al.
(2023).

Figure 1: An example illustration, ChatGPT-4 is unable to provide accurate
information about events that occurred after April 2023.

It is important to make LLMs efficient lifelong learners. Recently, there have
been multiple different attempts to propose lifelong learning methods for
knowledge updating Wu et al. (2023b); Zhang et al. (2024a) and skill
acquisition Ling et al. (2023); Zhang et al. (2023b); Lewis et al. (2021), a
comprehensive overview of these existing strategies is provided in Appendix
(Table 3). However, existing approaches pose the following limitations: (i)
These methods rely on sentences curated from fact triplets as the model’s
inputs, which is not practically feasible, as it is hard to organize all
available information as structured units, e.g., a set of triplets; (ii) The
majority of the existing approaches are vulnerable to catastrophic forgetting;
(iii) These approaches either focus on "open-book" or "closed-book" settings
(Section 3.1), with none of them providing an in-depth analysis of both
approaches at the same time. This calls the need for practical/easily
adaptable data curation methodologies and accordingly better modeling
strategies for the life-long learning of LLMs.

To address these challenges, in this paper, we propose Mixture-of-Experts
augmented Low Rank Adaptation for Lifelong learning (MoRAL). MoRAL simply
relies on question-answer pairs for life-long learning. Our key observation
is: this architecture simultaneously exploits the multi-task modeling
capability of the MoE structure and the parameter-efficient features of LoRA
to achieve efficient lifelong learning. In order to test the effectiveness of
MoRAL for different LLMs, we also introduce an evaluation benchmark, i.e.,
Life-Long Learning of LLMs (5L-bench), encompassing a newly proposed dataset
(question-answer pairs directly captured from the unstructured text rather
than fact triplets) and novel evaluation metrics for performance comparison.
We summarize the major contributions of this paper as follows:

  1. 1.

We propose MoRAL, an effective strategy that combines the benefits of MoE
along with LoRA as an effective and efficient strategy for lifelong learning
of LLMs.

  2. 2.

We introduce a new evaluation benchmark, i.e., 5L-bench, tailored to
evaluating the lifelong learning abilities of LLMs using casual question-
answer pairs from unstructured text rather than fact triplets.

  3. 3.

We perform a rigorous evaluation of MoRAL under both "open-book" and "closed-
book" settings. We delve into the interplay of these two methodologies,
looking for insights, respective strengths, and limitations of MoRAL.

##  2 Related Works

Continual Learning. Continual learning (CL) aims to learn new skills and
knowledge without forgetting previous knowledge, also known as catastrophic
forgetting Kirkpatrick et al. (2017); Kaushik et al. (2021). Maltoni and
Lomonaco (2019) delineated three principal strategies in CL: architectural
Rusu et al. (2022); Lomonaco and Maltoni (2017), regularization Zenke et al.
(2017), and rehearsal Hayes et al. (2019). They also conducted a thorough
analysis of these strategies in sequentially learning incremental tasks.
Lesort et al. (2019); Wang et al. (2023c) provided a comprehensive summary of
lifelong learning from the perspective of autonomous agents. They emphasized
that agents must adopt continuous methodologies for adaptation Sprechmann et
al. (2018), catastrophic forgetting, data distribution shifts Gepperth and
Karaoguz (2016). Also, in our experiments, we not only focus on the model’s
ability to learn new domain knowledge but also avoid catastrophic forgetting.

Lifelong Training of LLMs. Continual learning offers a practical solution for
adapting to novel data distributions Gururangan et al. (2020); Xiong et al.
(2023). However, this approach is vulnerable to overfitting. To mitigate this,
Chen et al. (2023) introduced the Lifelong-MoE, an extensible architecture to
allow pre-training over diverse data distributions. Other than that, fine-
tuning pre-trained foundation models also serve as an effective strategy for
downstream task adaptation Zhou et al. (2023); Raffel et al. (2023). Amongst
them, the parameter efficient variants include LoRA Hu et al. (2021), Prompt
Tuning Lester et al. (2021), etc. These methods optimize task-specific
objectives by fine-tuning only a small set of parameters Mangrulkar et al.
(2022); Hu et al. (2023); Yu et al. (2023); Ling et al. (2023). Motivated by
these, in this paper we combine the multi-tasking modeling capability of the
MoE structure and the parameter-efficient features of LoRA for an efficient
lifelong training method.

Model Editing. Model editing methods are used to make targeted, cost-effective
fixes to edit the information contained in the LLMs Hartvigsen et al. (2023).
Existing model editing solutions support targeted operations, i.e., knowledge
insertion, modification, and erasure Hase et al. (2023); Wen et al. (2023);
Wang et al. (2023d). These methods may be categorized into: (i) meta-learning
methods, which use external networks to predict gradients, e.g., MEND Mitchell
et al. (2022) and (ii) locate-then-edit methods, which directly identify and
update the target parameters, e.g., ROME Meng et al. (2023a), MEMIT Meng et
al. (2023b), etc., refer to Zhang et al. (2024a) for a recent survey. In-
context learning methods resort to external knowledge Zheng et al. (2023), and
memory-based information retrieval Lewis et al. (2021); Gao et al. (2024) to
directly edit the model’s knowledge Ovadia et al. (2024); Pawelczyk et al.
(2023).

Key limitations of the existing methods is their reliance on factual triplets
data, which creates challenges in data preparation and fully evaluating the
effectiveness of the model performance Wu et al. (2023b), some challenges that
are addressed by MoRAL.

##  3 Preliminaries

Notations: In this paper, we use $x$ to represent the input and $y$ as the
output of the MoRAL architecture. For the 5L-evaluation benchmark, we use $q$
to represent a query, $C$ to represent the context. $C_{r}$ represents the
context fragments relevant to the query $q$, $R_{o}$ represents the open-book
response, $R_{c}$ represents the close-book response, and $G_{t}$ is the
ground truth.

###  3.1 "Open/Closed" book and Cross setting

"Open-book" and "closed-book" are two different strategies for querying LLMs.
The major differences between these strategies are as follows:

(a) Open-book. This strategy assumes that LLMs may refer to external data
sources for inference. The external data sources may include but are not
limited to databases, knowledge graphs, unstructured text, examples, etc.

(b) Closed-book. This strategy treats LLM as a data storage bucket that
answers solely based on the knowledge gained during model training AlKhamissi
et al. (2022).

(c) Cross-Setting. The two settings ("open-book" and "close-book") are
interconnected. For this, we establish a criteria to investigate how
enhancements in the model’s closed-book capabilities simultaneously influence
open-book setting. Likewise, there are some metrics equally important for both
scenarios, e.g., fluency of the response. To quantify this, we use "cross-
setting" that evaluates all responses equally across different settings and
computes the average scores.

Figure 2: Example illustration of difference between the input data for
conventional approaches and MoRAL.

###  3.2 Fact Triples vs Question-Answer Pairs

We illustrate the key differences between the input data format used by the
conventional approaches and MoRAL in Figure 2. For illustration, we want to
update the knowledge of the model from {"The president of United states is
_Donald Trump_ "} to "_Joe Biden_ ". The conventional methods will extract the
relevant information triplet (president, Joe Biden, United States) from the
raw documents, which will be later used to formulate a sentence. Whereas our
method (5L-bench) reformulates this information as question-answer pairs. We
argue the latter approach is a more feasible and practical solution, as it is
not possible to convert all available information as a set of triplets,
leading to information loss.

##  4 MoRAL for Lifelong LLMs

As we mentioned, we aim to develop a lifelong learning method to keep LLMs up
to date with the latest available knowledge and information. Unlike previous
works relying on sentences directly curated using fact triplets, we use casual
question-answer pairs directly captured from the unstructured text as the
input (explained in Section 3.2). For the lifelong learning strategy, we aim
to combine the multi-task learning abilities of the MoE with the fine-tuning
abilities of LoRA for effective learning. Specifically, we propose MoRAL
(i.e., Mixture-of-Experts augmented Low Rank Adaptation for Lifelong
learning). MoRAL uses a divide-and-conquer strategy. It incorporates the
benefits of using multiple experts along multiple different low-rank intrinsic
knowledge dimensions with the hope of performing the end task in a
performance-enhanced fashion.

The underlying motivation is that within the foundation LLMs, the
knowledge/information resides along multiple different intrinsic/salient
dimension, similar to subspaces Ali et al. (2019); Hu et al. (2021), and we
may have multiple different localized/specialized experts to learn and/or
override the prior information/knowledge contained by the LLM. We summarize
the workflow of MoRAL as follows:

Figure 3: MoRAL architecture for life-long learning of LLMs. We use $n$
experts. FFN in the figure represents Feed-Foward Network.

  * •

Introduce low-rank matrices to decompose the weight matrices corresponding to
the pre-trained LLMs.

  * •

Use these low-rank matrices as experts to be used on top of the pre-trained
model.

  * •

Allow conditional computation over multiple experts using a gating mechanism,
also known as a router network.

For MoRAL, we configured eight LoRA expert modules, adopting a top-k routing
strategy analogous to Jiang et al. (2024). Figure 3 presents an illustration
of the MoRAL structure. The computational steps for the router network and
inference stages are explained as follows:

#### (a) Router Network.

Assume there are $n$ localized experts, we use router network to compute the
proportional score contribution of each expert. The router network is defined
as:

$\vspace{-0.7ex}G(x)_{i}=\text{softmax}(W_{g}^{T}x)\vspace{-0.7ex}$ (1)

where $W_{g}\in\mathbf{R}^{d_{m}\times n}$ represents the trainable weights of
the router network with $d_{m}$ as the input dimension and $n$ as the number
of experts.

#### (b) MoRAL Output.

The final output of the MoRAL architecture is computed as:

$\vspace{-1.1ex}y=\sum_{i=1}^{n}s_{i}\cdot E_{i}(x)\vspace{-0.7ex}$ (2)

where, $s_{i}=G(x)_{i}$ is the gating score for the $i^{th}$ expert, and
$E_{i}(x)$ is the output from the expert for the input $x$.

##  5 5L-Bench (Evaluation Benchmark)

For the performance evaluation of MoRAL, we propose a new benchmark (i.e.,
5L-bench). It encompasses: (1) A new curated dataset namely: Arxiv, to test
the ability of MoRAL to adapt to new data domains. (2) A pre-existing dataset,
i.e., HotpotQA Yang et al. (2018) used to test the ability of MoRAL to
restrain knowledge by not allowing the model to forget old knowledge. (3)
Newly proposed evaluation metrics to rigorously evaluate the performance of
MoRAL under open-book, closed-book and cross-settings.

###  5.1 Arxiv Data Curation

Our data curation pipeline is shown in the upper-half of Figure 4. It aims to
curate a set of question-answer pairs from unstructured text, and it is
explained as follows.

Firstly, we acquire unlabeled raw documents from Arxiv and split them into
information chunks $C$. Then, we employ GPT-3.5-turbo-16k to generate the
corresponding questions $q$ for each chunk $c\in C$ Li et al. (2023).
Following this, we use GPT-4 to generate the ground truth $G_{t}$, creating
standard answers based on the questions and their associated information.

Data leakage is a key challenge when it comes to evaluating LLMs on vast
datasets Li et al. (2024). To prevent the model from having prior exposure to
the data we intend to use for model training, we use the latest papers, i.e.,
from December 2023 Arxiv, as our data source. To ensure precise document
segmentation and facilitate data generation in a format conducive to our
analysis, we utilize the method of recursively splitting by
character111https://python.langchain.com/docs/modules/data_connection/document_transformers/recursive_text_splitter.
Additionally, we leverage prompts detailed in Appendix B.5 to guide the model
to output data in the desired format.

The data curation process generates a quintuple dataset denoted as: {$q$:
query, $C$: context, $C_{r}$: retrieved contexts, $R_{o}$: open-book response,
$R_{c}$: closed-book response, $G_{t}$: ground truth}. Note that in our
configuration, each query $q$ is uniquely paired with a context $c\in C$. The
retrieved context set $C_{r}$ encompasses fragments from the context $c$ that
exhibit relevance to the query. This relevance is computed by the cosine
similarity between the embeddings of $q$ and each context in $C$, exceeding a
predefined threshold $\theta$.

$C_{r}=\\{c\in C\mid\cos(\text{EMB}(q),\text{EMB}(c))>\theta\\},$ (3)

where $\text{EMB}(\cdot)$ denotes the text embeddings, and $\cos(\xx,\yy)$
denotes the cosine similarity between vectors $\xx$ and $\yy$. Note, this is a
widely adopted method for semantic-based information retrieval Reimers and
Gurevych (2020, 2019).

Figure 4: Overview of the 5L-Bench data curation and evaluation pipeline.

###  5.2 Evaluation Metrics

5L-bench uses different evaluation metrics to test MoRAL under open-book,
closed-book and cross settings. Details are as follows:

#### (a) Open-book Settings.

For the open-book settings, we aim to explore the ability of LLMs to utilize
external information within the context window Xu et al. (2024b). A key
concern is to investigate if the model leverages the additional knowledge for
reasoning or merely for replication. For this, we design an evaluation
criterion sub-divided into four distinct scenarios:

  * •

Context Faithfulness (Faith). When the model is provided with "golden
context", that is, $C_{r}=\\{c\\}$, our criterion focuses more on the
consistency between LLM and external information to see whether the final
answer conflicts with the given context.

  * •

Irrelevant Context Filtering (Filter). When the model uses external
information that encompasses $c$ along with other unrelated contexts, i.e.,
$c\in C_{r}$ and $|C_{r}|>1$, our criterion prioritizes how well LLM avoids
answering irrelevant or context.

  * •

Refusal Rate (RR). When LLM is presented with external data entirely unrelated
to the question, i.e., $c\notin C_{r}$, we assess the ability of LLM to refuse
to answer the query.

  * •

For the cases with $C_{r}=\emptyset$, we will employ the same metrics used in
the closed-book setting.

#### (b) Closed-book Settings.

For the closed-book settings, we focus on learning objectives, i.e., we test
the new knowledge or capabilities for LLMs. For this, we use Recall Accuracy
(RA) as the evaluation metric Derczynski (2016); Es et al. (2023). The
computation of RA is illustrated in Appendix B.1.

#### (c) Cross Settings.

For cross-settings, we evaluate: (i) the compliance of model’s response with
the given query, i.e., how well the model answers a given question also
referred to as Query Relevance (QR) by Es et al. (2023); and (ii) the model’s
linguistic modeling capabilities, assessed via the fluency (FL) of the model’s
responses. Details on "QR" and "FL" are provided in the Appendix B.1.

##  6 Experimentation

###  6.1 Experimental Setup

Datasets. For performance evaluation, we use newly curated data (Arxiv) and an
open-source data HotpotQA Yang et al. (2018). The Arxiv dataset comprises
seven domains with a diverse range of topics from mathematics to artificial
intelligence. We use this dataset as the target data for learning new
knowledge. We split this data into 80% and 20% for training and test sets,
respectively. The HotpotQA dataset encompasses 1,500 rows. We use this data as
the hold-out data for testing knowledge-retaining ability. The statistics of
the data are given in Appendix B.4 (Table 5).

Experimental Settings. In order to train MoRAL, we utilize the Adam optimizer
Kingma and Ba (2014) with a learning rate of 0.0001. The batch size is set to
16, and the model is trained for 2 epochs. As shown in Figure 3, we apply
MoRAL to the frozen FFN layers. We use the number of experts (n = 8), and top
$k$ = 2. For Equation 3, we use $\theta$ = 0.87. All experiments are performed
using Pytorch and Nvidia A100 80G GPU.

Large Language Models. For experimental evaluation, we use multiple different
open-source and closed-source LLMs. Specifically, we use basic large language
models TinyLlama-1.1B Zhang et al. (2024b),
Phi-2-2.7B222https://huggingface.co/microsoft/phi-2, Llama2-7B Touvron et al.
(2023), and state-of-the-art (SOTA) closed-source LLMs including
GPT-3.5-turbo-16k333https://platform.openai.com/docs/models/gpt-3-5-turbo,
Gemini-pro Google (2023), and Claude-2.1 Anthropic (2023). Details about these
models are in the Appendix B.2.

Baselines. We use multiple parameter-efficient fine-tuning approaches as
baselines, namely: (a) LoRA Hu et al. (2021), (b) IA3 Liu et al. (2022), and
(c) LLaMA-Adapter Zhang et al. (2023a). It is notable that our model is not
comparable with the existing knowledge-editing and life-long learning
baselines, e.g., MELO by Yu et al. (2023), MEND by Mitchell et al. (2022)
etc., as these models rely on factual triplets as the model input, which makes
them different from our work. Details about the baseline approaches are in the
Appendix B.3.

Evaluation Workflow. Our evaluation is structured around three primary
settings, i.e., open-book, closed-book and cross configurations (see Section
3.1). In the closed-book setting, the model generates responses solely based
on its internal knowledge following the given instructions. For the open-book
setting, we employ the bge-large-en-v1.5 model Xiao et al. (2023) for
embedding generation, coupled with chroma444https://www.trychroma.com/ as the
vector database to store the embeddings of text blocks $c\in C$. It allows us
to identify text blocks with cosine similarity scores against the query
exceeding a predefined threshold $\theta$. These blocks, denoted as $C_{r}$,
are then inserted into the model’s context window, guiding the model to
evaluate the relevance between questions and answers. This process enables the
model to autonomously refine, filter, and if necessary decline to respond
based on the context’s relevancy and the instructions provided. The response
output of the model is finally assessed to measure the disparity between the
generated answer and the ground truth using the evaluation metrics explained
in Section 5.2. To mitigate the risk of bias arising from the use of a single
model Zeng et al. (2023); Hada et al. (2023) in our evaluation, we employ
GPT-4-1106-preview and GLM-4 555https://zhipuai.cn/devday as evaluators. We
use the average scores of these evaluators as the final assessment metric.

###  6.2 Experimental Results

Models | Open-book | Closed-book | Cross-setting
---|---|---|---
Faith.↑ | Filter.↑ | RR.↑ | RA↑ | RA↑ | QR↑ | FL.↑
TinyLlama-1.1B-Chat | 0.65 | 0.40 | 0.24 | 0.86 | 0.60 | 0.82 | 0.95
TinyLlama-1.1B-Chat+IA3 | 0.54 | 0.38 | 0.25 | 0.82 | 0.64 | 0.82 | 0.90
TinyLlama-1.1B-Chat+LLaMA-Adapter | 0.66 | 0.36 | 0.29 | 0.74 | 0.67 | 0.89 | 0.91
TinyLlama-1.1B-Chat+LoRA | 0.69 | 0.43 | 0.32 | 0.89 | 0.82 | 0.85 | 0.90
TinyLlama-1.1B-Chat+MoRAL | 0.63 | 0.58 | 0.28 | 0.91 | 0.77 | 0.90 | 0.93
Phi-2-2.7B | 0.54 | 0.31 | 0.33 | 0.73 | 0.41 | 0.88 | 0.89
Phi-2-2.7B+IA3 | 0.55 | 0.28 | 0.28 | 0.62 | 0.40 | 0.80 | 0.83
Phi-2-2.7B+LLaMA-Adapter | 0.59 | 0.30 | 0.35 | 0.69 | 0.48 | 0.84 | 0.85
Phi-2-2.7B+LoRA | 0.47 | 0.35 | 0.39 | 0.77 | 0.66 | 0.80 | 0.84
Phi-2-2.7B+MoRAL | 0.59 | 0.46 | 0.37 | 0.82 | 0.63 | 0.86 | 0.88
Llama-2-7B-chat-hf | 0.62 | 0.54 | 0.40 | 0.82 | 0.47 | 0.80 | 0.92
Llama-2-7B-chat-hf+IA3 | 0.67 | 0.50 | 0.43 | 0.77 | 0.50 | 0.75 | 0.86
Llama-2-7B-chat-hf+LLaMA-Adapter | 0.61 | 0.52 | 0.37 | 0.67 | 0.54 | 0.77 | 0.89
Llama-2-7B-chat-hf+LoRA | 0.65 | 0.50 | 0.48 | 0.83 | 0.72 | 0.83 | 0.87
Llama-2-7B-chat-hf+MoRAL | 0.71 | 0.61 | 0.51 | 0.90 | 0.79 | 0.92 | 0.90
GPT-3.5-turbo-16k | 0.80 | 0.64 | 0.75 | 0.92 | 0.73 | 0.92 | 0.97
Gemini-pro666https://ai.google.dev/models/gemini | 0.83 | 0.82 | 0.79 | 0.83 | 0.80 | 0.95 | 0.95
Claude-2 | 0.92 | 0.87 | 0.90 | 0.96 | 0.77 | 0.98 | 0.95

Table 1: MoRAL performance comparison against different LLMs using Arxiv data.
All evaluation metrics range from 0 to 1, with higher scores indicating better
performance. We boldface the best-performing scores.

#### LLMs Learn Fast in "Open-book".

Table 1 shows the results of MoRAL on the Arxiv dataset, compared against the
baseline models. We use the notation "+(strategy)" to specify the
corresponding fine-tuning strategy employed by the LLM.

For the open-source LLMs without any fine-tuning, we observe that exposing the
large model solely to the relevant context within the contextual window for
inference significantly enhances its performance. This is evident by an
increased Recall Accuracy (RA) score for TinyLlama-1.1B, i.e., 0.86 in open-
book settings, compared to 0.6 in the closed-book setting. Likewise, the
performance of Phi-2-2.7B and Llama-2-7B improves significantly, i.e., 0.73
and 0.82 in open-book compared to 0.41 and 0.47 in closed-book respectively.

A similar trend is observed for the closed-source LLMs, with GPT-3.5-turbo,
Gemini-pro, and Claude-2 improving the "RA" by 26.0%, 3.7%, and 24.6% for the
open-book settings compared to the closed-book settings. Despite fine-tuning,
closed-source LLMs continue to outperform open-source smaller models,
particularly in terms of metrics "Faith", "Filter", and "RR". This superiority
could stem from the large models’ effective human-alignment strategies Ouyang
et al. (2022), which enhance their contextual understanding and adherence to
instructions. This suggests that the real disparity between small open-source
and proprietary large models may lie deeper in their ability to model the
comprehension of language and tasks Brown et al. (2020); Sun et al. (2024)
rather than their capacity to generate responses aligned with standard
answers.

Overall results showcase the immense potential for integrating dynamic
information retrieval methods in LLMs’ context for enhanced performance. These
results strongly correlate with earlier studies by Balaguer et al. (2024) and
Zheng et al. (2023) that emphasize the significance of retrieval-augmented
generation for large models Gao et al. (2024).

#### MoRAL vs Baselines.

Comparing the results of MoRAL against the baseline models, we observe for
"RA" metric, MoRAL consistently outperforms the baseline models in the open-
book settings with very few exceptions. For instance, compared to the pre-
trained models, MoRAL improves the "RA" score for TinyLlama-1.1B, Phi-2-2.7B
and Llama-2-7B by 5.81%, 12.32% and 9.75% respectively in open-book settings.
Whereas, for the closed-book settings, TinyLlama-1.1B and Phi-2-2.7B models
fine-tuned using LoRA exhibit slightly better or comparable "RA" scores
compared to MoRAL, except for Llama-2-7B which performs best when fine-tuned
using MoRAL.

Comparing the results for other open-book metrics, i.e., "Faith", "Filter",
and "RR", we observe: (i) For the metric "Faith", LLMs trained using MoRAL
results a higher score except for TinyLlama-1.1B where LoRA performs slightly
better; (ii) for the metric "Filter" MoRAL consistently outperforms all
baseline models by a significant margin; (iii) for "RR" results of MoRAL are
comparable with the baseline models. These results strongly portray the
immense potential of MoRAL when employed in the open-book settings.

For the cross-settings, we observe that MoRAL results in higher "QR" with very
relatively low distortion in the fluency (FL) compared to baselines. This
decrease in the "FL" after instruction fine-tuning is likely due to the
prevalence of scientific descriptions and mathematical formulas in our data,
which reduced the model’s general language modeling capability Ji et al.
(2023). Notably, the decline in fluency was less pronounced for the models
fine-tuned using MoRAL.

Figure 5: Performance comparison of MoRAL vs LoRA for large models with
varying number of model parameters, best viewed in colors. These results are
computed using the Arxiv dataset.

We also observe, as the scale of the model increases, the model’s capabilities
in filtering and analyzing information from context increases significantly.
This is also evident by a relative higher scores for the metrics: "Faith",
"Filter", and "RR" for the large models fine-tuned using MoRAL vs models with
relatively lower parameters. This is also illustrated in Figure 5, where the
dark green region shows that MoRAL yields a higher improvement in the "RA"
score for Phi-2-2.7B and Llama-2-7B compared to that of TinyLlama-1.1B. It
also shows that relative improvement in performance for MoRAL is higher
compared to LoRA as a baseline.

To summarize, these results show MoRAL presents a promising direction for
effective and efficient learning of LLMs. This ascertains our hypothesis that
the multi-tasking ability of a mixture of experts when coupled with LoRA
significantly augments the contextual learning ability of the end model, also
shown previously on multiple different tasks Zoph et al. (2022); Xue et al.
(2024).

###  6.3 Further Discussions

In this section, we perform an in-depth analysis in attempts to understand the
life-long learning of LLMs from multiple different perspectives.

Models | Open-book | Closed-book | Cross-setting
---|---|---|---
Faith.↑ | Filter.↑ | RR.↑ | RA↑ | RA↑ | QR↑ | FL.↑
TinyLlama-1.1B-Chat-v1.0 | 0.67 | 0.41 | 0.20 | 0.89 | 0.72 | 0.90 | 0.95
TinyLlama-1.1B-Chat+IA3 | 0.60 | 0.33 | 0.21 | 0.73 | 0.64 | 0.80 | 0.89
TinyLlama-1.1B-Chat+LLaMA-Adapter | 0.61 | 0.39 | 0.20 | 0.75 | 0.64 | 0.81 | 0.88
TinyLlama-1.1B-Chat+LoRA | 0.68 | 0.38 | 0.17 | 0.87 | 0.65 | 0.83 | 0.91
TinyLlama-1.1B-Chat+MoRAL | 0.67 | 0.43 | 0.21 | 0.89 | 0.70 | 0.87 | 0.96
Phi-2-2.7B | 0.55 | 0.33 | 0.42 | 0.76 | 0.68 | 0.93 | 0.92
Phi-2-2.7B+IA3 | 0.46 | 0.33 | 0.40 | 0.73 | 0.60 | 0.88 | 0.90
Phi-2-2.7B+LLaMA-Adapter | 0.49 | 0.30 | 0.23 | 0.70 | 0.55 | 0.90 | 0.89
Phi-2-2.7B+LoRA | 0.49 | 0.27 | 0.31 | 0.71 | 0.64 | 0.80 | 0.90
Phi-2-2.7B+MoRAL | 0.53 | 0.30 | 0.37 | 0.74 | 0.65 | 0.86 | 0.93
Llama-2-7B-chat-hf | 0.72 | 0.62 | 0.51 | 0.90 | 0.75 | 0.95 | 0.96
Llama-2-7B-chat-hf+IA3 | 0.63 | 0.50 | 0.45 | 0.86 | 0.71 | 0.88 | 0.92
Llama-2-7B-chat-hf+LLaMA-Adapter | 0.60 | 0.48 | 0.41 | 0.80 | 0.68 | 0.89 | 0.86
Llama-2-7B-chat-hf+LoRA | 0.65 | 0.47 | 0.43 | 0.85 | 0.71 | 0.89 | 0.91
Llama-2-7B-chat-hf+MoRAL | 0.69 | 0.58 | 0.48 | 0.89 | 0.71 | 0.90 | 0.95

Table 2: MoRAL performance for different LLMs using HotpotQA-fullwiki dataset.
All evaluation metrics range from 0 to 1, with higher scores indicating better
performance. We boldface the best performing scores.

More Data or More Parameters? We first aim to answer the question: "In terms
of data and model parameters, what is required to make the end-model a better
lifelong learner?"

Surprisingly, we observe among the pre-trained LLMs (w/o fine-tuning),
TinyLlama-1.1B with only 1.1B parameters shows the best "RA" performance
compared to other baselines, i.e., RA is 0.60 and 0.86 in closed-book and
open-book settings respectively (Table 1). It showcases the potential of
"small" language models trained on vast datasets. This finding is also aligned
with a recent work, where a relatively small model, i.e., MiniCPM Hu et al.
(2024) with only 2B parameters is able to outperform 13B models on
UltraEval777https://github.com/OpenBMB/UltraEval.

However, a model with fewer parameters, i.e., TinyLlama-1.1B, exhibits
significantly lower scores for the open-book metrics ("Faith", "Filter", and
"RR"), compared to larger counterparts, with "RR" showing the most substantial
disparity—TinyLlama-1.1B’s baseline score is 0.24, compared to 0.4 for
Llama-2-7B. This shows that larger models are more adept at declining
questions beyond their comprehension scope . It also speaks of larger models’
enhanced in-context learning ability Wei et al. (2023), enabling them to
better filter and summarize information.

Learning New without Forgetting Old.

Figure 6: MoRAL performance comparisons for "RA". The left half of the Figure
reports results on Arxiv data from Table 1. The right half of the Figure
reports results on HotpotQA data from Table 2.

Adjusting parameters in large models risks catastrophic forgetting, a critical
challenge in lifelong learning that emphasizes the need for adapting to new
domains/tasks without losing prior knowledge New et al. (2022); Luo et al.
(2023). To test the knowledge retention ability of MoRAL compared against
baselines, we use the HotpotQA dataset as a holdout test set to re-evaluate
models fine-tuned using the Arxiv dataset. Corresponding results in Table 2
show that the baseline models yield a lower score for the metrics: "Faith",
"Filter", "RR", etc. MoRAL on the other hand, results in minimal loss for
these metrics while at the same time it shows proficiency in instruction
compliance and language fluency (FL).

Correlating the results for Table 1 and Table 2, we observe that although
baseline fine-tuning approaches significantly boost "RA" scores for new target
domains, however, they yield a decline in "RA" score for the holdout tests.
MoRAL on the other hand exhibits better resistance to catastrophic forgetting
by exhibiting a relatively stable performance. This is also illustrated in
Figure 6. The left half of the Figure shows MoRAL augments the knowledge
retention ability compared to baseline while learning new knowledge. The right
half of the Figure shows that for the HotpotQA data, where the LoRA baseline
yields lower "RA" scores, whereas MoRAL fights back to uplift the "RA" score.
Overall knowledge retention ability of MoRAL is more pronounced for the open-
book scenarios.

Note, for these experiments, we observe higher initial scores for the HotpotQA
dataset for the pre-trained base models, possibly because: (i) The HotpotQA
dataset and its Wikipedia sources were part of LLMs’ training data; (ii) The
training datasets include knowledge from 2018, aligning with the view that
models are biased to answer questions from this period due to temporal
information encoded in their parameters Nylund et al. (2023).

##  7 Conclusions

In this paper, we make following contributions: (i) we propose MoRAL for
efficient and effective life-long learning of LLMs; (ii) we propose an
evaluation benchmark (5L-bench) to evaluate the performance of MoRAL compared
against the baseline models. In the future, we plan to explore larger-scale
models and more efficient hybrid structures, such as Mixture of Vectors (MoV)
by Zadouri et al. (2023).

##  8 Limitations

#### Surface Learning vs. Deep Understanding.

Although, this paper shows that fine-tuned models are able to achieve
significant improvements in both open-book and closed-book settings. Still,
this work did not evaluate if the models only superficially learn to produce
answers that conform more closely to standard responses without truly
familiarizing themselves with the knowledge and concepts within the training
data.

#### Reliability of LLMs as Evaluators.

In our work, both GPT-4 and GLM-4 are employed as evaluators to mitigate the
bias that may arise from relying on a single model for assessmentHada et al.
(2023). Although large language models are extensively being used in
evaluating language tasks, demonstrating higher consistency compared to human
evaluators. Yet, employing more robust models to assess less advanced
counterparts essentially guides the models towards alignment with the
evaluator’s characteristics Lin and Chen (2023). This alignment could
potentially limit the models’ capacity to align with human understanding,
thereby constraining their performance upper bounds.

## References

  * Aghajanyan et al. (2021) Armen Aghajanyan, Anchit Gupta, Akshat Shrivastava, Xilun Chen, Luke Zettlemoyer, and Sonal Gupta. 2021.  Muppet: Massive multi-task representations with pre-finetuning.
  * Ali et al. (2019) Muhammad Asif Ali, Yifang Sun, Xiaoling Zhou, Wei Wang, and Xiang Zhao. 2019.  Antonym-synonym classification based on new sub-space embeddings.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33(01), pages 6204–6211.
  * AlKhamissi et al. (2022) Badr AlKhamissi, Millicent Li, Asli Celikyilmaz, Mona Diab, and Marjan Ghazvininejad. 2022.  A review on language models as knowledge bases.
  * Anthropic (2023) Anthropic. 2023.  Model card and evaluations for claude models.
  * Balaguer et al. (2024) Angels Balaguer, Vinamra Benara, Renato Luiz de Freitas Cunha, Roberto de M. Estevão Filho, Todd Hendry, Daniel Holstein, Jennifer Marsman, Nick Mecklenburg, Sara Malvar, Leonardo O. Nunes, Rafael Padilha, Morris Sharp, Bruno Silva, Swati Sharma, Vijay Aski, and Ranveer Chandra. 2024.  Rag vs fine-tuning: Pipelines, tradeoffs, and a case study on agriculture.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.
  * Cao et al. (2021) Nicola De Cao, Wilker Aziz, and Ivan Titov. 2021.  Editing factual knowledge in language models.
  * Chaudhary (2023) Sahil Chaudhary. 2023.  Code alpaca: An instruction-following llama model for code generation.  https://github.com/sahil280114/codealpaca.
  * Chen et al. (2023) Wuyang Chen, Yanqi Zhou, Nan Du, Yanping Huang, James Laudon, Zhifeng Chen, and Claire Cu. 2023.  Lifelong language pretraining with distribution-specialized experts.
  * Cui et al. (2023) Jiaxi Cui, Zongjian Li, Yang Yan, Bohua Chen, and Li Yuan. 2023.  Chatlaw: Open-source legal large language model with integrated external knowledge bases.
  * Delétang et al. (2023) Grégoire Delétang, Anian Ruoss, Paul-Ambroise Duquenne, Elliot Catt, Tim Genewein, Christopher Mattern, Jordi Grau-Moya, Li Kevin Wenliang, Matthew Aitchison, Laurent Orseau, Marcus Hutter, and Joel Veness. 2023.  Language modeling is compression.
  * Derczynski (2016) Leon Derczynski. 2016.  Complementarity, F-score, and NLP evaluation.  In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16)_ , pages 261–266, Portorož, Slovenia. European Language Resources Association (ELRA).
  * Dhingra et al. (2022) Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W. Cohen. 2022.  Time-aware language models as temporal knowledge bases.  _Transactions of the Association for Computational Linguistics_ , 10:257–273.
  * Ding et al. (2023) Ning Ding, Yulin Chen, Bokai Xu, Shengding Hu, Yujia Qin, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. 2023.  Ultrachat: A large-scale auto-generated multi-round dialogue data.  https://github.com/thunlp/ultrachat.
  * Es et al. (2023) Shahul Es, Jithin James, Luis Espinosa-Anke, and Steven Schockaert. 2023.  Ragas: Automated evaluation of retrieval augmented generation.
  * Fang et al. (2023) Tao Fang, Shu Yang, Kaixin Lan, Derek F. Wong, Jinpeng Hu, Lidia S. Chao, and Yue Zhang. 2023.  Is chatgpt a highly fluent grammatical error correction system? a comprehensive evaluation.
  * Gao et al. (2024) Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Qianyu Guo, Meng Wang, and Haofen Wang. 2024.  Retrieval-augmented generation for large language models: A survey.
  * Gepperth and Karaoguz (2016) Alexander Gepperth and Cem Karaoguz. 2016.  A bio-inspired incremental learning architecture for applied perceptual problems.  _Cognitive Computation_ , 8(5):924–934.
  * Google (2023) Google. 2023.  Gemini: A family of highly capable multimodal models.
  * Grosse et al. (2023) Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, Evan Hubinger, Kamilė Lukošiūtė, Karina Nguyen, Nicholas Joseph, Sam McCandlish, Jared Kaplan, and Samuel R. Bowman. 2023.  Studying large language model generalization with influence functions.
  * Gunasekar et al. (2023) Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi Li. 2023.  Textbooks are all you need.
  * Gururangan et al. (2021) Suchin Gururangan, Mike Lewis, Ari Holtzman, Noah A. Smith, and Luke Zettlemoyer. 2021.  Demix layers: Disentangling domains for modular language modeling.
  * Gururangan et al. (2020) Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020.  Don’t stop pretraining: Adapt language models to domains and tasks.
  * Hada et al. (2023) Rishav Hada, Varun Gumma, Adrian de Wynter, Harshita Diddee, Mohamed Ahmed, Monojit Choudhury, Kalika Bali, and Sunayana Sitaram. 2023.  Are large language model-based evaluators the solution to scaling up multilingual evaluation?
  * Hartvigsen et al. (2023) Thomas Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim, and Marzyeh Ghassemi. 2023.  Aging with grace: Lifelong model editing with discrete key-value adaptors.  In _Advances in Neural Information Processing Systems_.
  * Hase et al. (2023) Peter Hase, Mohit Bansal, Been Kim, and Asma Ghandeharioun. 2023.  Does localization inform editing? surprising differences in causality-based localization vs. knowledge editing in language models.
  * Hayes et al. (2019) Tyler L. Hayes, Nathan D. Cahill, and Christopher Kanan. 2019.  Memory efficient experience replay for streaming learning.
  * Hu et al. (2021) Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021.  Lora: Low-rank adaptation of large language models.
  * Hu et al. (2024) Shengding Hu, Yuge Tu, Xu Han, Ganqu Cui, Chaoqun He, Weilin Zhao, Xiang Long, Zhi Zheng, Yewei Fang, Kaihuo Zhang, Yuxiang Huang, Zhenning Dai, Baitao Gong, Chongyi Wang, Yuan Yao, Jie Zhou, Jie Cai, Xinrong Zhang, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2024.  Minicpm: Unveiling the potential of end-side large language models.
  * Hu et al. (2023) Zhiqiang Hu, Yihuai Lan, Lei Wang, Wanyu Xu, Ee-Peng Lim, Roy Ka-Wei Lee, Lidong Bing, and Soujanya Poria. 2023.  Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models.  _arXiv preprint arXiv:2304.01933_.
  * Huang et al. (2023) Cynthia Huang, Yuqing Xie, Zhiying Jiang, Jimmy Lin, and Ming Li. 2023.  Approximating human-like few-shot learning with gpt-based compression.
  * Jelinek et al. (1977) Fred Jelinek, Robert L Mercer, Lalit R Bahl, and James K Baker. 1977.  Perplexity—a measure of the difficulty of speech recognition tasks.  _The Journal of the Acoustical Society of America_ , 62(S1):S63–S63.
  * Ji et al. (2023) Yunjie Ji, Yong Deng, Yan Gong, Yiping Penga, Qiang Niu, Lei Zhang, Baochang Ma, and Xiangang Li. 2023.  Exploring the impact of instruction data scaling on large language models: An empirical study on real-world use cases.  _arXiv preprint arXiv:2303.14742_.
  * Jiang et al. (2024) Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2024.  Mixtral of experts.
  * Kaddour et al. (2023) Jean Kaddour, Joshua Harris, Maximilian Mozes, Herbie Bradley, Roberta Raileanu, and Robert McHardy. 2023.  Challenges and applications of large language models.
  * Kaushik et al. (2021) Prakhar Kaushik, Alex Gain, Adam Kortylewski, and Alan Yuille. 2021.  Understanding catastrophic forgetting and remembering in continual learning with optimal relevance mapping.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_.
  * Kirk et al. (2024) Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. 2024.  Understanding the effects of rlhf on llm generalisation and diversity.
  * Kirkpatrick et al. (2017) James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. 2017.  Overcoming catastrophic forgetting in neural networks.  _Proceedings of the National Academy of Sciences_ , 114(13):3521–3526.
  * Kudithipudi et al. (2022) Dhireesha Kudithipudi, Mario Aguilar-Simon, Jonathan Babb, Maxim Bazhenov, Douglas Blackiston, Josh Bongard, Andrew P. Brna, Suraj Chakravarthi Raja, Nick Cheney, Jeff Clune, Sandeep Madireddy, and Angel Yanguas-Gil. 2022.  Biological underpinnings for lifelong learning machines.  _Nature Machine Intelligence_ , 4(3).
  * Lesort et al. (2019) Timothée Lesort, Vincenzo Lomonaco, Andrei Stoian, Davide Maltoni, David Filliat, and Natalia Díaz-Rodríguez. 2019.  Continual learning for robotics: Definition, framework, learning strategies, opportunities and challenges.
  * Lester et al. (2021) Brian Lester, Rami Al-Rfou, and Noah Constant. 2021.  The power of scale for parameter-efficient prompt tuning.
  * Lewis et al. (2021) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2021.  Retrieval-augmented generation for knowledge-intensive nlp tasks.
  * Li et al. (2023) Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2023.  Camel: Communicative agents for "mind" exploration of large language model society.  In _Thirty-seventh Conference on Neural Information Processing Systems_.
  * Li et al. (2024) Yucheng Li, Frank Guerin, and Chenghua Lin. 2024.  An open source data contamination report for large language models.
  * Lin and Chen (2023) Yen-Ting Lin and Yun-Nung Chen. 2023.  Llm-eval: Unified multi-dimensional automatic evaluation for open-domain conversations with large language models.
  * Ling et al. (2023) Chen Ling, Xujiang Zhao, Jiaying Lu, Chengyuan Deng, Can Zheng, Junxiang Wang, Tanmoy Chowdhury, Yun Li, Hejie Cui, Xuchao Zhang, Tianjiao Zhao, Amit Panalkar, Wei Cheng, Haoyu Wang, Yanchi Liu, Zhengzhang Chen, Haifeng Chen, Chris White, Quanquan Gu, Jian Pei, and Liang Zhao. 2023.  Domain specialization as the key to make large language models disruptive: A comprehensive survey.
  * Liu et al. (2022) Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. 2022.  Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning.
  * Liu et al. (2020) Yaoyao Liu, Yuting Su, An-An Liu, Bernt Schiele, and Qianru Sun. 2020.  Mnemonics training: Multi-class incremental learning without forgetting.  In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE.
  * Lomonaco and Maltoni (2017) Vincenzo Lomonaco and Davide Maltoni. 2017.  Core50: a new dataset and benchmark for continuous object recognition.
  * Luo et al. (2023) Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. 2023.  An empirical study of catastrophic forgetting in large language models during continual fine-tuning.
  * Lyu et al. (2024) Yuanjie Lyu, Zhiyu Li, Simin Niu, Feiyu Xiong, Bo Tang, Wenjin Wang, Hao Wu, Huanyong Liu, Tong Xu, and Enhong Chen. 2024.  Crud-rag: A comprehensive chinese benchmark for retrieval-augmented generation of large language models.  _arXiv preprint arXiv:2401.17043_.
  * Maltoni and Lomonaco (2019) Davide Maltoni and Vincenzo Lomonaco. 2019.  Continuous learning in single-incremental-task scenarios.
  * Mangrulkar et al. (2022) Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. 2022.  Peft: State-of-the-art parameter-efficient fine-tuning methods.  https://github.com/huggingface/peft.
  * Meng et al. (2023a) Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2023a.  Locating and editing factual associations in gpt.
  * Meng et al. (2023b) Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. 2023b.  Mass-editing memory in a transformer.
  * Mitchell et al. (2022) Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D. Manning. 2022.  Fast model editing at scale.
  * Mukherjee et al. (2021) Subhabrata Mukherjee, Xiaodong Liu, Guoqing Zheng, Saghar Hosseini, Hao Cheng, Greg Yang, Christopher Meek, Ahmed Hassan Awadallah, and Jianfeng Gao. 2021.  Clues: Few-shot learning evaluation in natural language understanding.
  * New et al. (2022) Alexander New, Megan Baker, Eric Nguyen, and Gautam Vallabha. 2022.  Lifelong learning metrics.
  * Nylund et al. (2023) Kai Nylund, Suchin Gururangan, and Noah A. Smith. 2023.  Time is encoded in the weights of finetuned language models.
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022.  Training language models to follow instructions with human feedback.
  * Ovadia et al. (2024) Oded Ovadia, Menachem Brief, Moshik Mishaeli, and Oren Elisha. 2024.  Fine-tuning or retrieval? comparing knowledge injection in llms.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, July 6-12, 2002, Philadelphia, PA, USA_ , pages 311–318. ACL.
  * Pawelczyk et al. (2023) Martin Pawelczyk, Seth Neel, and Himabindu Lakkaraju. 2023.  In-context unlearning: Language models as few shot unlearners.
  * Raffel et al. (2023) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2023.  Exploring the limits of transfer learning with a unified text-to-text transformer.
  * Rawte et al. (2023) Vipula Rawte, Swagata Chakraborty, Agnibh Pathak, Anubhav Sarkar, SM Tonmoy, Aman Chadha, Amit P Sheth, and Amitava Das. 2023.  The troubling emergence of hallucination in large language models–an extensive definition, quantification, and prescriptive remediations.  _arXiv preprint arXiv:2310.04988_.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.
  * Reimers and Gurevych (2020) Nils Reimers and Iryna Gurevych. 2020.  Making monolingual sentence embeddings multilingual using knowledge distillation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.
  * Rusu et al. (2022) Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. 2022.  Progressive neural networks.
  * Shumailov et al. (2023) Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Anderson. 2023.  The curse of recursion: Training on generated data makes models forget.
  * Sprechmann et al. (2018) Pablo Sprechmann, Siddhant M. Jayakumar, Jack W. Rae, Alexander Pritzel, Adrià Puigdomènech Badia, Benigno Uria, Oriol Vinyals, Demis Hassabis, Razvan Pascanu, and Charles Blundell. 2018.  Memory-based parameter adaptation.
  * Sun et al. (2024) Jiankai Sun, Chuanyang Zheng, Enze Xie, Zhengying Liu, Ruihang Chu, Jianing Qiu, Jiaqi Xu, Mingyu Ding, Hongyang Li, Mengzhe Geng, Yue Wu, Wenhai Wang, Junsong Chen, Zhangyue Yin, Xiaozhe Ren, Jie Fu, Junxian He, Wu Yuan, Qi Liu, Xihui Liu, Yu Li, Hao Dong, Yu Cheng, Ming Zhang, Pheng Ann Heng, Jifeng Dai, Ping Luo, Jingdong Wang, Ji-Rong Wen, Xipeng Qiu, Yike Guo, Hui Xiong, Qun Liu, and Zhenguo Li. 2024.  A survey of reasoning with foundation models.
  * Thrun and Mitchell (1995) Sebastian Thrun and Tom M Mitchell. 1995.  Lifelong robot learning.  _Robotics and autonomous systems_ , 15(1-2):25–46.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023.  Llama: Open and efficient foundation language models.
  * Wang et al. (2023a) Cunxiang Wang, Xiaoze Liu, Yuanhao Yue, Xiangru Tang, Tianhang Zhang, Cheng Jiayang, Yunzhi Yao, Wenyang Gao, Xuming Hu, Zehan Qi, Yidong Wang, Linyi Yang, Jindong Wang, Xing Xie, Zheng Zhang, and Yue Zhang. 2023a.  Survey on factuality in large language models: Knowledge, retrieval and domain-specificity.
  * Wang et al. (2023b) Haochun Wang, Sendong Zhao, Zewen Qiang, Zijian Li, Nuwa Xi, Yanrui Du, MuZhen Cai, Haoqiang Guo, Yuhan Chen, Haoming Xu, Bing Qin, and Ting Liu. 2023b.  Knowledge-tuning large language models with structured medical knowledge bases for reliable response generation in chinese.
  * Wang et al. (2023c) Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. 2023c.  A comprehensive survey of continual learning: Theory, method and application.
  * Wang et al. (2023d) Song Wang, Yaochen Zhu, Haochen Liu, Zaiyi Zheng, Chen Chen, and Jundong Li. 2023d.  Knowledge editing for large language models: A survey.
  * Wei et al. (2023) Jerry W. Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, and Tengyu Ma. 2023.  Larger language models do in-context learning differently.  _CoRR_ , abs/2303.03846.
  * Wen et al. (2023) Cheng Wen, Xianghui Sun, Shuaijiang Zhao, Xiaoquan Fang, Liangyu Chen, and Wei Zou. 2023.  Chathome: Development and evaluation of a domain-specific language model for home renovation.
  * Wu et al. (2023a) Haoran Wu, Wenxuan Wang, Yuxuan Wan, Wenxiang Jiao, and Michael Lyu. 2023a.  Chatgpt or grammarly? evaluating chatgpt on grammatical error correction benchmark.
  * Wu et al. (2023b) Suhang Wu, Minlong Peng, Yue Chen, Jinsong Su, and Mingming Sun. 2023b.  Eva-kellm: A new benchmark for evaluating knowledge editing of llms.
  * Xiao et al. (2023) Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. 2023.  C-pack: Packaged resources to advance general chinese embedding.
  * Xiong et al. (2023) Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, Sinong Wang, and Hao Ma. 2023.  Effective long-context scaling of foundation models.
  * Xu et al. (2024a) Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. 2024a.  Contrastive preference optimization: Pushing the boundaries of llm performance in machine translation.
  * Xu et al. (2024b) Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. 2024b.  Retrieval meets long context large language models.
  * Xue et al. (2024) Fuzhao Xue, Zian Zheng, Yao Fu, Jinjie Ni, Zangwei Zheng, Wangchunshu Zhou, and Yang You. 2024.  Openmoe: An early effort on open mixture-of-experts language models.
  * Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018.  Hotpotqa: A dataset for diverse, explainable multi-hop question answering.
  * Yao et al. (2023a) Yuanshun Yao, Xiaojun Xu, and Yang Liu. 2023a.  Large language model unlearning.
  * Yao et al. (2023b) Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang. 2023b.  Editing large language models: Problems, methods, and opportunities.
  * Yu et al. (2023) Lang Yu, Qin Chen, Jie Zhou, and Liang He. 2023.  Melo: Enhancing model editing with neuron-indexed dynamic lora.
  * Yue et al. (2023) Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. 2023.  Mammoth: Building math generalist models through hybrid instruction tuning.  _arXiv preprint arXiv:2309.05653_.
  * Zadouri et al. (2023) Ted Zadouri, Ahmet Üstün, Arash Ahmadian, Beyza Ermiş, Acyr Locatelli, and Sara Hooker. 2023.  Pushing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning.
  * Zeng et al. (2023) Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. 2023.  Evaluating large language models at evaluating instruction following.
  * Zenke et al. (2017) Friedemann Zenke, Ben Poole, and Surya Ganguli. 2017.  Continual learning through synaptic intelligence.
  * Zhang et al. (2024a) Ningyu Zhang, Yunzhi Yao, Bozhong Tian, Peng Wang, Shumin Deng, Mengru Wang, Zekun Xi, Shengyu Mao, Jintian Zhang, Yuansheng Ni, Siyuan Cheng, Ziwen Xu, Xin Xu, Jia-Chen Gu, Yong Jiang, Pengjun Xie, Fei Huang, Lei Liang, Zhiqiang Zhang, Xiaowei Zhu, Jun Zhou, and Huajun Chen. 2024a.  A comprehensive study of knowledge editing for large language models.
  * Zhang et al. (2024b) Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. 2024b.  Tinyllama: An open-source small language model.
  * Zhang et al. (2023a) Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. 2023a.  Llama-adapter: Efficient fine-tuning of language models with zero-init attention.
  * Zhang et al. (2023b) Yue Zhang, Leyang Cui, Deng Cai, Xinting Huang, Tao Fang, and Wei Bi. 2023b.  Multi-task instruction tuning of llama for specific scenarios: A preliminary study on writing assistance.
  * Zhang et al. (2023c) Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, Longyue Wang, Anh Tuan Luu, Wei Bi, Freda Shi, and Shuming Shi. 2023c.  Siren’s song in the ai ocean: A survey on hallucination in large language models.
  * Zhang et al. (2023d) Zihan Zhang, Meng Fang, Ling Chen, Mohammad-Reza Namazi-Rad, and Jun Wang. 2023d.  How do large language models capture the ever-changing world knowledge? a review of recent advances.
  * Zhao et al. (2023) Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. 2023.  A survey of large language models.
  * Zheng et al. (2023) Ce Zheng, Lei Li, Qingxiu Dong, Yuxuan Fan, Zhiyong Wu, Jingjing Xu, and Baobao Chang. 2023.  Can we edit factual knowledge by in-context learning?
  * Zhou et al. (2023) Ce Zhou, Qian Li, Chen Li, Jun Yu, Yixin Liu, Guangjing Wang, Kai Zhang, Cheng Ji, Qiben Yan, Lifang He, Hao Peng, Jianxin Li, Jia Wu, Ziwei Liu, Pengtao Xie, Caiming Xiong, Jian Pei, Philip S. Yu, and Lichao Sun. 2023.  A comprehensive survey on pretrained foundation models: A history from bert to chatgpt.
  * Zhu et al. (2023a) Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. 2023a.  Multilingual machine translation with large language models: Empirical results and analysis.
  * Zhu et al. (2023b) Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. 2023b.  A survey on model compression for large language models.
  * Zoph et al. (2022) Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. 2022.  St-moe: Designing stable and transferable sparse expert models.

##  Appendix A Existing Challenges in Lifelong Learning

Lifelong learning, initially conceptualized by Thrun and Mitchell (1995),
refers to a paradigm where a model leverages its previously acquired knowledge
to enhance subsequent learning Thrun and Mitchell (1995). The primary features
of lifelong learning include knowledge transfer, adaptation to new
environments, and overcoming catastrophic forgetting Kudithipudi et al.
(2022); New et al. (2022). With the advent of LLMs, the distinction between
knowledge and skills is increasingly ambiguous. The definition of lifelong
learning for these models still lacks clarity. We summarize the existing
lifelong learning methods along with their corresponding evaluation metrics in
Table 3. Briefly, these existing approaches can be divided into two
categories: one is to train the model to "remember" new knowledge and skills
(closed-book); the other is to put additional information into the model’s
context window so that the model can "see it" and make a response (open-book).

We observe a notable challenge for lifelong learning of LLMs is the diversity
in data formats, such as factual triplets Cao et al. (2021); Mitchell et al.
(2022), supervised input-output pairs Chaudhary (2023); Yue et al. (2023), and
information chunks Lewis et al. (2021). Such a vast diversity of data formats
complicates data preparation and reuse. There is a dire need to use simple
data preparation strategies along with robust lifelong learning methods. Also,
in practice, we usually use a combination of methods to adapt LLMs to new
domains and tasks Wang et al. (2023b); Cui et al. (2023) which makes it
difficult to make evaluations with traditional evaluation pipelines that are
isolated from each other.

Methodologies |  Scenarios |  Elements
---|---|---
Continual Pre-training |  Out-Distribution Adaptation (for better downstream tasks performance) |  PPL Jelinek et al. (1977); Forget R. Liu et al. (2020); MF1; Acc Gururangan et al. (2021); Aghajanyan et al. (2021)
Knowledge Editing |  Knowledge Insertion |  Reliability; Generalization Zhang et al. (2024a); Portability; Locality Yao et al. (2023b); FluencyMeng et al. (2023a); Cross-lingual Evaluation (CKEE) Wu et al. (2023b)
Knowledge Modification
Knowledge Erasure
Fine-tuning |  Downstream Tasks |  F1 (political affiliation classification); ROUGE-L (news summarization) Dhingra et al. (2022); BLEU (Machine Translation) Papineni et al. (2002);CLUES Mukherjee et al. (2021)
Model Unlearning |  Knowledge Erasure |  UnlearningSuccess (Generic tasks) Pawelczyk et al. (2023); Unlearning Harmfulness (Security and Alignment) Yao et al. (2023a)
RAG |  Create,Read,Update and Delete (CRUD) Lyu et al. (2024) |  ROUGE, BLEU, bertScore, RAGQuestEval Lyu et al. (2024), Ragas Es et al. (2023)
In-context Learning |  Downstream Tasks |  F1 (political affiliation classification); ROUGE-L (news summarization) Dhingra et al. (2022); BLEU (Machine Translation) Papineni et al. (2002)

Table 3: Different methodologies and evaluation metrics for lifelong learning.

##  Appendix B Details of Experiments

###  B.1 Evaluation Metrics

#### (a) QR

(Query Relevance): QR measures how well the response is aligned with the input
query/question ($q$). For the computation of QR, we use the same settings as
that of Es et al. (2023). We use context $c$ and response $R$ in order to
generate the question $Q(R,c)$. Later, we compute the similarity between the
generated question and query $q$. This score is computed as:

$\text{QR}=\frac{1}{n}\sum_{i=1}^{n}\text{sim}(q,Q(a,c))$ (4)

where sim is the cosine similarity of the corresponding embedding vectors.

#### (b) FL

(Fluency): FL measures if the text generated is well-written and grammatical.
Fang et al. (2023); Wu et al. (2023a) have shown the remarkable capabilities
of large language models in assessing sentence fluency and grammatical
accuracy, highlighting their superiority over conventional approaches. In our
experimental framework, we employ GPT-4 and GLM-4 for FL evaluation. The
prompts utilized in our study are delineated in Table 10. The final FL score
is computed as:

$\text{FL}=\frac{1}{n}\sum_{i=1}^{n}\text{mean}(\text{GPT-4},\text{GLM-4})$
(5)

#### (c) RA

(Recall Accuracy): In order to compute RA, we first compute: TP (True
Positives), FP (False Positives), and FN (False Positives). Then RA is
computed as:

$\text{RA}=\frac{F1\times w_{0}+\cos(\text{EMB}(a),\text{EMB}(G_{t}))\times
w_{1}}{w_{0}+w_{1}}$ (6)

where the $F1$ score is computed as:
$\text{TP}/(\text{TP}+0.5\times(\text{FP}+\text{FN}))$. In above equation,
$\text{EMB}(y)$ and $\text{EMB}(G_{t})$ represent the embeddings of the
model’s output and the ground truth. The weights $w_{0}$ and $w_{1}$ are used
to balance the F1 score and the cosine similarity of the embeddings.

###  B.2 Large Language Models

#### (a) TinyLlama-1.1B-Chat-v1.0.

TinyLlama-1.1B, a relatively smaller model compared to Llama, was pre-trained
on 3 trillion tokens Zhang et al. (2024b) and fine-tuned on the Ultrachat Ding
et al. (2023) dataset.

#### (b) Phi-2-2.7B.

Phi-2-2.7B888https://huggingface.co/microsoft/phi-2, a model with 2.7 billion
parameters, was trained on a dataset comprising 1.4 trillion tokens, including
a substantial number of textbooks Gunasekar et al. (2023).

#### (c) Llama2-7b-chat.

Llama2-7b-chat is an open-source model pre-trained on 2.0 trillion tokens,
fine-tuned on publicly available instruction datasets, as well as over one
million new human-annotated examples Touvron et al. (2023).

#### (d) SOTA Closed-source LLMs.

Among the closed-source LLMs, we compare MoRAL against state-of-the-art (SOTA)
models, including: GPT-3.5-turbo-16k Brown et al. (2020), Gemini-pro Google
(2023), and Claude-2.1 Anthropic (2023). These models are accessed via API
calls.

Symbols | Meaning
---|---
$q$ | the query
$C$ | the context
$C_{r}$ | the context fragments relevant to the query $q$
$R_{o}$ | the open-book response
$R_{c}$ | the close-book response
$G_{t}$ | ground truth

Table 4: Notations.

###  B.3 Baselines

#### (a) LoRA.

LoRA uses a set of trainable rank decomposition matrices for the Transformer
layers fine-tuning phase Hu et al. (2021). In our case, we use LoRA adaptors
for the attention layer, i.e., for the query ($q$) and key ($k$) matrices to
enable efficient learning.

#### (b) IA3.

IA3 re-calibrates internal activations by suppressing and amplifying them,
thus injecting adapters through the modulation of internal activations. These
learned vectors are integrated into the attention and feed-forward modules of
typical Transformer-based architectures Liu et al. (2022). In our case, IA3
weights are added to the outputs of the key and value layers, as well as the
input to the second feed-forward layer in each Transformer block.

#### (c) LLaMA-Adapter.

The Llama-Adapter is designed to adapt the Llama model for instruction
following tasks. To avoid introducing noise into the tokens, the adapter
employs zero-init attention Zhang et al. (2023a). Additionally, the adapter
incorporates a learnable gating factor, also initialized to zero, which allows
for the gradual introduction of information to the model during training.

###  B.4 Data Statistics

The statistics of the dataset is shown in Table 5.

Domain | Size
---|---
Arxiv-Math | 1,518 rows
Arxiv-Astro-ph | 1,811 rows
Arxiv-Gr-qc | 1,749 rows
Arxiv-Q-bio | 1,749 rows
Arxiv-Q-fin | 2,513 rows
Arxiv-Statistics | 2,208 rows
Arxiv-EESS | 1,442 rows
Arxiv-Ai | 2,001 rows
HotpotQA-fullwiki | 1,500 rows

Table 5: Dataset distribution of different datasets, i.e., Arxiv and HotpotQA.

###  B.5 Prompts

In this section, we present a detailed overview of the prompts used for data
generation. Table 6 demonstrates the prompts designed for generating queries
from various data sources. Following this, Table 7 describes the prompts used
for generating ground truth data used for evaluating model performance. Table
8 and Table 9 cover the methodologies for prompt generation in open-book and
closed-book settings, respectively.

    You are a University Professor creating
    a test for advanced students. For each
    context, create a question that is
    specific to the context. Avoid creating
    generic or general questions.
    Context: {context}
    Question: a question about the context.
    Format the output as *JSON* with the
    following key:
    "question" \n\n

---
Table 6: The prompt template for generating the query from existing context.

    You are a University Professor creating
    a test for advanced students. For each
    question and context, create a standard
    answer.
    Context: {context}
    Question: {question}

    Format the output as *JSON* with the f
    ollowing keys:
    “ground truth” \n\n

---
Table 7: The prompt template for generating the ground truth.

    Please answer the question based only
    on the following context. If you cannot
    answer the question with the context,
    please respond with ’I don’t know’:
        ### CONTEXT
        {context}
        ### QUESTION
        Question: {question}
        \n
    Please give your answer below:

---
Table 8: The prompt template for open-book setting.

    Please answer the following question.
        ### QUESTION
        Question: {question}
        \n
    Give your answer below:

---
Table 9: The prompt template for closed-book setting.

    Evaluate the fluency of a given piece
    of text on a scale from 0 to 1, where
    0 represents very poor fluency with
    numerous grammatical errors and awkward
    phrasing, and 1 represents excellent
    fluency with smooth, natural language
    and no grammatical mistakes. Consider
    aspects such as grammar, syntax, c
    oherence, and the natural flow of
    ideas. Please provide a clear rating
    and a brief justification for your
    assessment, highlighting specific
    examples from the text that influenced
    your rating. Your evaluation should
    be flexible enough to accommodate
    a variety of texts while maintaining
    a focus on fluency and coherence.
    \n
    ##Text
    Text: {response}
    \n
    Give your score below:

---
Table 10: The prompt template for FL evaluation.