# Knowledge Editing for Large Language Models: A Survey

Song Wang  University of VirginiaCharlottesvilleVirginiaUSA<EMAIL_ADDRESS>,  Yaochen Zhu  University of VirginiaCharlottesvilleVirginiaUSA
<EMAIL_ADDRESS>,  Haochen Liu  University of
VirginiaCharlottesvilleVirginiaUSA<EMAIL_ADDRESS>,  Zaiyi Zheng
University of VirginiaCharlottesvilleVirginiaUSA<EMAIL_ADDRESS>,  Chen
Chen  University of VirginiaCharlottesvilleVirginiaUSA<EMAIL_ADDRESS>and
Jundong Li  University of VirginiaCharlottesvilleVirginiaUSA
<EMAIL_ADDRESS>
###### Abstract.

Large language models (LLMs) have recently transformed both the academic and
industrial landscapes due to their remarkable capacity to understand, analyze,
and generate texts based on their vast knowledge and reasoning ability.
Nevertheless, one major drawback of LLMs is their substantial computational
cost for pre-training due to their unprecedented amounts of parameters. The
disadvantage is exacerbated when new knowledge frequently needs to be
introduced into the pre-trained model. Therefore, it is imperative to develop
effective and efficient techniques to update pre-trained LLMs. Traditional
methods encode new knowledge in pre-trained LLMs through direct fine-tuning.
However, naively re-training LLMs can be computationally intensive and risks
degenerating valuable pre-trained knowledge irrelevant to the update in the
model. Recently, Knowledge-based Model Editing (KME) has attracted increasing
attention, which aims to precisely modify the LLMs to incorporate specific
knowledge, without negatively influencing other irrelevant knowledge. In this
survey, we aim to provide a comprehensive and in-depth overview of recent
advances in the field of KME. We first introduce a general formulation of KME
to encompass different KME strategies. Afterward, we provide an innovative
taxonomy of KME techniques based on how the new knowledge is introduced into
pre-trained LLMs, and investigate existing KME strategies while analyzing key
insights, advantages, and limitations of methods from each category. Moreover,
representative metrics, datasets, and applications of KME are introduced
accordingly. Finally, we provide an in-depth analysis regarding the
practicality and remaining challenges of KME and suggest promising research
directions for further advancement in this field.

Model Editing, Knowledge Update, Fine-tuning, Large Language Models

††price: 15.00††isbn: 978-1-4503-XXXX-X/18/06††ccs: Computing methodologies
Natural language processing

##  1\. Introduction

Recently, large language models (LLMs) have become a heated topic that
revolutionizes both academia and industry (OpenAI, 2023; Brown et al., 2020;
Touvron et al., 2023; Zhao et al., 2023). With the substantial factual
knowledge and reasoning ability gained from pre-training on large corpora,
LLMs have exhibited an unprecedented understanding of textual information,
which are able to analyze and generate texts akin to human experts.
Nevertheless, one main drawback of LLMs is the extremely high computational
overhead of the training process due to the large amounts of parameters. This
is exacerbated by the continuous evolvement of the world where the requirement
of updating pre-trained LLMs to rectify obsolete information or incorporate
new knowledge to maintain their relevancy is constantly emerging (Zhong et
al., 2023). For example, as in Fig. 1, an outdated LLM cannot precisely
describe the latest achievements of Lionel Messi, which requires explicit
injection of new knowledge to generate the correct answers.

One feasible while straightforward strategy for updating pre-trained LLMs is
through naive fine-tuning (Dubois et al., 2023; Taori et al., 2023; Wei et
al., 2021; Chung et al., 2022), where parameters of pre-trained LLMs are
directly optimized to encode new knowledge from new data (Zhao et al., 2023;
Azamfirei et al., 2023; Peng et al., 2023a; Menick et al., 2022). For example,
various instruction-tuning-based methods are proposed to fine-tune the pre-
trained LLMs on the newly collected corpus in a supervised learning manner
(Wang et al., 2022a; Peng et al., 2023b; Wang et al., 2023c; Min et al.,
2023). Although such fine-tuning techniques are widely used and capable of
injecting new knowledge into LLMs, they are known for the following
disadvantages: (1) Even with several parameter-efficient strategies proposed
to improve the efficiency (Liu et al., 2022; Zaken et al., 2022; Wang et al.,
2022b), fine-tuning LLMs may still require intensive computational resources
(Mitchell et al., 2022b; Zheng et al., 2023; Meng et al., 2022). (2) The fine-
tuned model could overfit the new data, especially when the dataset utilized
for fine-tuning is small in scale (De Cao et al., 2021; Meng et al., 2023;
Mitchell et al., 2022a). (3) More importantly, fine-tuning LLMs alters the
pre-trained weights with no constraints, which risks losing valuable existing
knowledge in LLMs (Madaan et al., 2022; Huang et al., 2023; Dong et al.,
2022a). These challenges limit the practicality of fine-tuning techniques in
updating LLMs with new knowledge.

Figure 1. An intuitive example of KME for efficient knowledge update of pre-
trained LLMs.

To address the drawbacks of naive fine-tuning for updating LLMs, more
attention has been devoted to Knowledge-based Model Editing (KME), also known
as Knowledge Editing. In general, KME aims to precisely modify the behavior of
pre-trained LLMs to update specific knowledge without negatively influencing
other pre-trained knowledge irrelevant to the updates (Yao et al., 2023; Wang
et al., 2023b; Pinter and Elhadad, 2023). In KME, the update of a specific
piece of knowledge in LLMs is typically formulated as an edit, such as
rectifying the answer to “Who is the president of the USA?” from “Trump” to
“Biden”. Regarding a specific edit, KME strategies typically modify the model
output by either introducing an auxiliary network (or set of parameters) into
the pre-trained model (Li et al., 2023b; Zhong et al., 2023; Hartvigsen et
al., 2022) or updating the (partial) parameters to store the new knowledge (Ha
et al., 2016; Dai et al., 2022; Li et al., 2023a; Gupta et al., 2023). Through
these strategies, the KME techniques can store new knowledge in memories or
locate it in model parameters for updating, thereby precisely injecting the
knowledge into the model. In addition, certain methods also introduce explicit
loss to contain the updating process, such that the edited model maintains
consistent behaviors on unmodified knowledge. With these advantages, KME
techniques can provide an efficient and effective way to constantly update
LLMs with novel knowledge without explicit model re-training.

Although KME shares certain similarities with fine-tuning strategies, it poses
unique advantages in updating LLMs, which are worthy of deeper investigations.
Particularly, both KME and model fine-tuning seek to update pre-trained LLMs
by injecting new knowledge. However, aside from this shared objective, KME
focuses more on two crucial properties that cannot be easily addressed by
fine-tuning. (1) Locality requires that the edited model does not
unintentionally influence the output of other irrelevant inputs with distinct
semantics. For example, when the edit regarding the president of the USA is
updated, the edited model should not alter its knowledge about the prime
minister of the UK. The practicality of KME methods largely relies on their
ability to maintain the outputs for unrelated inputs, which serves as a major
difference between KME and fine-tuning (Qin et al., 2022). (2) Generality
represents whether the edited model can generalize to a broader range of
relevant inputs regarding the edited knowledge. Specifically, it indicates the
model’s capability to present consistent behavior on inputs that share
semantic similarities. For example, when the model is edited regarding the
president, the answer to the query of the president’s spouse should also
change accordingly. In practice, it is important for KME methods to ensure
that the edited model can adapt well to these related input texts. In summary,
due to these two unique objectives, KME remains a challenging task that
requires specific strategies for satisfactory effectiveness.

Differences between this survey and existing ones. Several surveys have been
conducted to examine various aspects of (large) language models (Zhao et al.,
2023; Thirunavukarasu et al., 2023; Fan et al., 2023; Kalyan et al., 2022;
Chang et al., 2023; Kasneci et al., 2023). Nevertheless, there is still a
dearth of thorough investigations that comprehensively encompass the existing
literature and the continuous progress in the field of LLM editing. For
example, recent works (Wang et al., 2023c; Min et al., 2023) have discussed
the fine-tuning strategies to incorporate new knowledge in pre-trained LLMs
with more data samples. However, the distinctiveness of KME, i.e., locality
and generality, is not adequately discussed, which will be thoroughly analyzed
in this survey. Two additional surveys (Fei et al., 2021; Hu et al., 2023)
review knowledge-enhanced language models. However, their main focus is on
leveraging external knowledge to enhance the performance of the pre-trained
LLMs, without addressing the editing task based on specific knowledge. To the
best of our knowledge, the most related paper to our survey is (Yao et al.,
2023), which provides a brief overview of KME and concisely discusses the
advantages of KME methods and their challenges. Nevertheless, the
investigation lacks a thorough examination of more details of KME, e.g.,
categorizations, datasets, and applications. Another recent work (Wang et al.,
2023b) proposes a framework for KME that unifies several representative
methods. This work focuses on the implementation of KME techniques, with less
emphasis on the technical details of different strategies. More recently, a
work (Pinter and Elhadad, 2023) discusses the limitations of KME methods
regarding the faithfulness of edited models, while it is relatively short and
lacks a more comprehensive introduction to all existing methods. Considering
the rapid advancement of KME techniques, we believe it is imperative to review
details of all representative KME methods, summarizing the commonalities while
discussing the uniqueness of each method, and discuss open challenges and
prospective directions in the domain of KME, which will facilitate the further
advancement of the field.

Contributions of this survey. This survey provides a comprehensive and in-
depth analysis of techniques, challenges, and opportunities associated with
the editing of pre-trained LLMs. We first provide an overview of KME tasks
along with an innovative formulation. Particularly, we formulate the general
KME task as a constrained optimization problem, which simultaneously
incorporates the goals of accuracy, locality, and generality. We then classify
the existing KME strategies into three main categories, i.e., external
memorization, global optimzation, and local modification. Importantly, we
demonstrate that methods in each category can be formulated as a specialized
constrained optimization problem, where the characteristics are theoretically
summarized based on the general formulation. In addition, we provide valuable
insights into the effectiveness and feasibility of methods in each category,
which can assist practitioners in selecting the most suitable KME method
tailored to a specific task. Our analysis regarding the strengths and
weaknesses of KME methods also serves as a catalyst for ongoing progress
within the KME research community. In summary, our key contributions can be
summarized into three folds as follows:

  * •

Novel Categorization. We introduce a comprehensive and structured
categorization framework to systematically summarize the existing works for
LLM editing. Specifically, based on how the new knowledge is introduced into
pre-trained LLMs, our categorization encompasses three distinct categories:
external memorization, global optimization, and local modification, where
their commonality and differences are thoroughly discussed in this survey.

  * •

In-Depth Analysis. We formulate the task of LLM editing as a constrained
optimization problem, where methods from each category can be viewed as a
special case with refined constraints. Furthermore, we emphasize the primary
insights, advantages, and limitations of each category. Within this context,
we delve deep into representative methods from each category and
systematically analyze their interconnections.

  * •

Future Directions. We analyze the practicality of existing KME techniques
regarding a variety of datasets and applications. We also comprehensively
discuss the challenges of the existing KME techniques and suggest promising
research directions for future exploration.

The remainder of this paper is organized as follows. Section 2 introduces the
background knowledge for LLM editing. Section 3 provides a general formulation
of the KME task, which can fit into various application scenarios. Section 4
provides a comprehensive summary of evaluation metrics for KME strategies,
which is crucial for a fair comparison of various methods. Before delving into
the specific methods, we provide a comprehensive categorization of existing
methods into three classes in Section 5.1, where their relationship and
differences are thoroughly discussed. Then we introduce the methods from the
three categories in detail, where the advantages and limitations of each
category are summarized. Section 6 introduces the prevalently used public
datasets. Section 7 provides a thorough introduction to various realistic
tasks that can benefit from KME techniques. Section 8 discusses the potential
challenges of KME that have not been addressed by existing techniques. This
section also provides several potential directions that can inspire future
research. Lastly, we conclude this survey in Section 9.

##  2\. Background

In this section, we provide an overview of the editing strategies for machine
learning models and the basics of large language models (LLMs) as background
knowledge to facilitate the understanding of technical details in KME. In this
survey, we use bold uppercase letters (e.g., $\mathbf{K}$ and $\mathbf{V}$) to
represent matrices, use lowercase bold letters (e.g., $\mathbf{k}$ and
$\mathbf{v}$) to represent vectors, and use calligraphic uppercase letters
(e.g., $\mathcal{X}$ and $\mathcal{Y}$) to represent sets. We summarize the
primary notations used in this survey in Table 1 for the convenience of
understanding.

###  2.1. Editing of Machine Learning Models

Machine learning models (He et al., 2016; Gemmeke et al., 2017; Kenton and
Toutanova, 2019) pre-trained on large datasets frequently serve as foundation
models for various tasks in the real-world (Deng et al., 2009; Schuhmann et
al., 2021). In practical scenarios, there is often a need to modify these pre-
trained models to enhance the performance for specific downstream tasks
(Zhuang et al., 2020; Wortsman et al., 2022; Muennighoff et al., 2022; Chung
et al., 2022; Chiang and Lee, 2023), reduce biases or undesirable behaviors
(Ribeiro and Lundberg, 2022; Ganguli et al., 2022; Perez et al., 2022; Murty
et al., 2022), tailor models to align more closely with human preferences
(Glaese et al., 2022; Kasirzadeh and Gabriel, 2023; Liu et al., 2023), or
incorporate novel information (Zhu et al., 2020; Mitchell et al., 2022a; Yao
et al., 2023).

Model Editing is a special type of model modification strategy where the
modification should be as precise as possible. Nevertheless, it should
accurately modify the pre-trained model to encode specific knowledge while
maximally preserving the existing knowledge, without affecting their behavior
on unrelated inputs (Ilharco et al., 2023). First explored in the computer
vision field, Bau et al. (2020) investigate the potential of editing
generative adversarial networks (GAN) (Goodfellow et al., 2020) by viewing an
intermediate layer as a linear memory, which can be manipulated to incorporate
novel content. Afterward, Editable Training (Sinitsin et al., 2020) is
proposed to encourage fast editing of the trained model in a model-agnostic
manner. The goal is to change the model predictions on a subset of inputs
corresponding to misclassified objects, without altering the results for other
inputs. In (Santurkar et al., 2021), the authors propose a method that allows
for the modification of a classifier’s behavior by editing its decision rules,
which can be used to correct errors or reduce biases in model predictions. In
the field of natural language processing, several works (Dai et al., 2022;
Mitchell et al., 2022b) have been proposed to perform editing regarding
textual information. Specifically, Zhu et al. (2020) propose a constrained
fine-tuning loss to explicitly modify specific factual knowledge in
transformer-based models (Vaswani et al., 2017). More recent works (Geva et
al., 2021, 2022) discover that the MLP layers in transformers actually act as
key–value memories, thereby enabling the editing of specific knowledge within
the corresponding layers.

Table 1.  Important notations used in this survey.

Notations | Detailed Descriptions
---|---
$x$ | Input (prompt) to LLMs
$y$ | Output of LLMs
$(x,y)$ | Input-output pair
$t=(s,r,o)$ | Original knowledge triple (before editing)
$s$ | Subject in a knowledge triple
$r$ | Relation in a knowledge triple
$o$ | Object in a knowledge triple
$t^{*}=(s,r,o^{*})$ | Target knowledge triple (after editing)
$e=(s,r,o\rightarrow o^{*})$ | Edit descriptor
$\mathcal{X}_{e}$ | In-scope input space
$\mathcal{Y}_{e}$ | Original output space (before editing)
$\mathcal{Y}_{e}^{*}$ | Target output space (after editing)
$\mathcal{E}=\\{e_{i}\\}$ | Set of edits
$\mathcal{O}_{e}$ | Out-scope input space
$\mathbf{q}^{(l)}_{i}$ | Query vector for the $i$-th attention head of the $l$-th attention module in Transformer
$\mathbf{k}^{(l)}_{i}$ | Key vector for the $i$-th attention head of the $l$-th attention module in Transformer
$\mathbf{v}^{(l)}_{i}$ | Value vector for the $i$-th attention head of the $l$-th attention module in Transformer
$\mathbf{W}^{(l)}_{1}$, $\mathbf{W}^{(l)}_{2}$ | Weights of the fully connected layers of the $l$-th attention module in Transformer
$\mathbf{h}^{(l)}$ | Output from the $l$-th self-attention module in Transformer
$\|$ | Vector concatenation

###  2.2. Large Language Models

####  2.2.1. Transformers.

Transformers lie in the core of large language models (LLMs) (Vaswani et al.,
2017; Devlin et al., 2018; Reimers and Gurevych, 2019). The fully-fledged
transformer possesses an encoder-decoder architecture initially designed for
the neural machine translation (NMT) task (Stahlberg, 2020). Nowadays,
transformers have found wide applications in most fields of the NLP community,
beyond their original purpose. Generally, a transformer network is constructed
from multiple stacks of the self-attention module with residual connections,
which is pivotal for capturing contextual information from textual sequences.
The self-attention module is composed of a self-attention layer (SelfAtt) and
a point-wise feed-forward neural network layer (FFN) formulated as follows:

(1)
$\displaystyle\mathbf{h}^{A,(l-1)}_{i}=\operatorname{SelfAtt}_{i}\left(\mathbf{h}^{(l-1)}_{i}\right)=\operatorname{Softmax}\left(\mathbf{q}^{(l)}_{i}\left(\mathbf{k}^{(l)}_{i}\right)^{\top}\right)\mathbf{v}_{i}^{(l)},$
$\displaystyle\mathbf{h}^{F,(l-1)}=\operatorname{FFN}\left(\mathbf{h}^{(l-1)}\right)=\operatorname{GELU}\left(\mathbf{h}^{(l-1)}\mathbf{W}^{(l)}_{1}\right)\mathbf{W}^{(l)}_{2},\mathbf{h}^{(0)}=\mathbf{x},$
$\displaystyle\mathbf{h}^{(l)}=\mathbf{h}^{A,(l-1)}+\mathbf{h}^{F,(l-1)}=\big{\|}_{i}\operatorname{SelfAtt}_{i}\left(\mathbf{h}^{(l-1)}_{i}\right)+\operatorname{FFN}\left(\mathbf{h}^{(l-1)}\right),$

where $\mathbf{q}^{(l)}_{i}$, $\mathbf{k}^{(l)}_{i}$, and
$\mathbf{v}^{(l)}_{i}$ represent the sequences of query, key, and value
vectors for the $i$-th attention head of the $l$-th attention module,
respectively. GELU is an activation function. They are calculated from
$\mathbf{h}^{(l-1)}_{i}$, the $i$-th slice of the outputs from the $(l-1)$-th
self-attention module (i.e., $\mathbf{h}^{(l-1)}$), and $\mathbf{x}$ denotes
the input sequence of token embeddings. $\|$ represents vector concatenation.
Normalizing factors in the self-attention layer are omitted for simplicity.

Generally, multi-head self-attention directs the model to attend to different
parts of the sequence to predict the next token. Specifically, the prediction
is based on different types of relationships and dependencies within the
textual data, where the output $\mathbf{h}^{A,(l-1)}_{i}$ is a weighted sum of
the value vector of other tokens. In contrast, FFN adds new information
$\mathbf{h}^{F,(l-1)}_{i}$ to the weighted sum of the embeddings of the
attended tokens based on the information stored in the weights of the fully
connected layers, i.e., $\mathbf{W}^{(l)}_{1}$ and $\mathbf{W}^{(l)}_{2}$. The
final layer outputs of the transformer, i.e., $\mathbf{h}^{(L)}$, can be used
in various downstream NLP tasks. For token-level tasks (e.g., part-of-speech
tagging (Chiche and Yitagesu, 2022)), the entire hidden representation
sequence $\mathbf{h}^{(L)}$ can be utilized to predict the target sequence.
For the sequence-level tasks (e.g., sentiment analysis (Wankhade et al.,
2022)), the hidden representation of the last token, i.e.,
$\mathbf{h}^{(L)}_{-1}$, can be considered as a summary of the sequence and
thus used for the predictions.

####  2.2.2. Large Language Models (LLMs).

Transformers with billions of parameters trained on large corpora have
demonstrated emergent ability, showcasing an unprecedented understanding of
factual and commonsense knowledge (Zhao et al., 2023). Consequently, these
models are referred to as large language models (LLMs) to indicate their
drastic distinction from traditional small-scale language models
(Thirunavukarasu et al., 2023; Fan et al., 2023). Generally, based on the
specific parts of the transformer utilized for language modeling, existing
LLMs can be categorized into three classes: encoder-only LLMs, such as BERT
(Bidirectional Encoder Representations from Transformers) (Kenton and
Toutanova, 2019), encoder-decoder-based LLMs such as T5 (Text-To-Text Transfer
Transformer) (Raffel et al., 2020), and decoder-only models (also the most
common structures in LLMs) such as different versions of GPT (Generative Pre-
trained Transformer) (Radford et al., 2018) and LLaMA (Large Language Model
Meta AI) (Touvron et al., 2023).

####  2.2.3. Training Strategies.

Typically, LLMs with encoder-only structures are trained based on masked
language modeling (MLM) (Devlin et al., 2018), which randomly masks out tokens
in the corpus and predicts the missing tokens by attending to tokens from both
sides. Therefore, token embeddings can be better learned by considering
context from a broader scope. On the other hand, LLMs based on encoder-decoder
and decoder-only structures utilize the causal language modeling (CLM)
strategy for pre-training, which autoregressively predicts the next token by
only attending to previous tokens. Therefore, they are more powerful at
generating new content based on pretexts.

The training of generative LLMs is mainly based on two stages, i.e., the pre-
training stage and the fine-tuning stage. In the first stage, LLMs are trained
by language modeling (i.e., next/masked token prediction) on large corpus such
as Wikipedia (Vrandečić and Krötzsch, 2014) with commonsense and factual
knowledge, ArXiv papers (Beltagy et al., 2019) with scientific/math knowledge,
codes from GitHub (Chen et al., 2021) with coding knowledge, etc. Then during
the fine-tuning stage, exemplar prompt-answer pairs (e.g., question and
answers, demands and codes, description, and formula) are provided to the LLMs
such that they are able to answer questions based on the prompt instead of
simply completing the sentence. In addition, RLHF (reinforcement learning from
human feedback) (Lambert et al., 2022) is usually introduced to further fine-
tune the model by learning from human ranks of different generated answers
without the explicit requirement of ground truth pairs. After training, the
LLM can be viewed as a database, where prompts formed as natural language can
be used as queries to obtain knowledge encoded in the weights of transformers
(Chang et al., 2023; Kasneci et al., 2023).

##  3\. Problem Formulation

In this section, we provide a formal definition for the knowledge-based model
editing (KME) task for pre-trained LLMs, where a general formulation of the
KME objective is formulated to encompass specific KME strategies. The task of
KME for LLMs can be broadly defined as the process of precisely modifying the
behavior of pre-trained LLMs, such that new knowledge can be incorporated to
maintain the currentness and relevancy of LLMs can be maintained, without
negatively influencing other pre-trained knowledge irrelevant to the edits. To
provide a clear formulation, we present the definitions of different terms
used in KME, where the overall process is illustrated in Fig. 2.

Editing Target. In this survey, we represent the knowledge encoded or required
to be injected into the pre-trained LLMs as a knowledge triple $t=(s,r,o)$,
where $s$ is the subject (e.g., president of the USA), $r$ is the relation
(e.g., is), and $o$ is the object (e.g., Biden). From the perspective of
knowledge triple, KME for LLMs aims to modify the original knowledge triple
$t=(s,r,o)$ encoded in the pre-trained weights of the model into the target
knowledge triple $t^{*}=(s,r,o^{*})$, where $o^{*}$ is the target object
different from $o$. In this manner, we can define an edit as a tuple
$e=(t,t^{*})=(s,r,o\rightarrow o^{*})$, which denotes the update of the
obsolete old knowledge $t$ into the new knowledge $t^{*}$.

Figure 2. The formulation of the KME objective.

Input and Output Space. Given a pair of subject $s$ and relation $r$, to query
LLMs to obtain the object $o$, $(s,r)$ needs to be transformed into natural
language, which we denoted as $x$. $x$ is also referred to as the prompt in
this survey. The LLM output $y$ is also textual and can be converted back to
an object $o$ as the query result. In this manner, $(x,y)$ can be considered
as the natural language input-output pair associated with the knowledge triple
$t=(s,r,o)$. For example, the prompt $x$ transformed from $s$ and $r$ can be
“The president of the USA is”, and $y$ is the model output “Joe Biden”. Note
that due to the diversity of natural language, multiple $(x,y)$ pairs can be
associated with the same knowledge triple $t$. We denote the set of textual
inputs that can be associated with subject $s$ and relation $r$ in an edit $e$
as $\mathcal{X}_{e}=I(s,r)$, referred to as in-scope input space. Similarly,
we define the set of textual outputs that can be associated with the object
$o$ in the same edit $e$ as $\mathcal{Y}_{e}^{*}=O^{*}(s,r,o^{*})$ (i.e.,
target output space), and the original textual output space as
$\mathcal{Y}_{e}=O(s,r,o)$ (i.e., original output space). Given an edit $e$,
the aim of KME is to modify the behavior of language models from
$\mathcal{Y}_{e}$ to $\mathcal{Y}_{e}^{*}$, regarding the input in
$\mathcal{X}_{e}$. To accommodate the scenarios where multiple edits are
performed, we can define the union of $\mathcal{X}_{e}$ over a set of edits
$\mathcal{E}=\\{e_{1},e_{2},\dotsc\\}$ as
$\mathcal{X}_{\mathcal{E}}=\bigcup_{e\in\mathcal{E}}\mathcal{X}_{e}$.
Similarly, we can define
$\mathcal{Y}_{\mathcal{E}}=\bigcup_{e\in\mathcal{E}}\mathcal{Y}_{e}$ and
$\mathcal{Y}^{*}_{\mathcal{E}}=\bigcup_{e\in\mathcal{E}}\mathcal{Y}^{*}_{e}$.

Formulation. We denote the pre-trained LLM with parameter $\phi$ as
$f:\mathcal{X}\rightarrow\mathcal{Y}$ and the edited model with updated
parameter $\phi^{*}$ as $f^{*}:\mathcal{X}\rightarrow\mathcal{Y}^{*}$. The
objective of knowledge-based model editing is to precisely update the pre-
trained LLM $f$ into $f^{*}$ according to edits in the edit set $\mathcal{E}$
such that for each edit $e$ and for each $y\in\mathcal{Y}_{e}$, the changes to
the input-output pairs irrelevant to the edits is minimized. The problem of
KME can be formulated as follows:

###### Definition 0.

The objective for KME on a series of edits $\mathcal{E}$ is represented as
follows:

(2)
$\displaystyle\min\mathbb{E}_{e\in\mathcal{E}}\mathbb{E}_{x,y^{*}\in\mathcal{X}_{e},\mathcal{Y}^{*}_{e}}\mathcal{L}(f^{*}(x),y^{*}),\text{where}\
\ f^{*}=M(f;\mathcal{E}),$ $\displaystyle\;\text{s.t.}\;f^{*}(x)=f(x),\ \
\forall x\in\mathcal{X}\setminus\mathcal{X}_{\mathcal{E}},$

where $\mathcal{L}$ is a loss that measures the discrepancy between the LM
output $f^{*}(x)$ and $y^{*}$ from the desirable response set
$\mathcal{Y}^{*}_{e}$. $M(f;\mathcal{E})$ denotes the modification applied to
$f$ based on the desirable edits $\mathcal{E}$.

From the above definition, we can summarize two crucial perspectives regarding
the objective of KME: (1) Generality, which requires that the correct answers
in the target output space $\mathcal{Y}^{*}_{e}$ can be achieved, provided
prompts in the in-scope input space, i.e., $\mathcal{X}_{e}$, where the target
knowledge triple $t^{*}\in e$ can be updated into the pre-trained model; (2)
Locality, which requires the consistency of model output regarding unrelated
input, i.e., $\mathcal{X}\setminus\mathcal{X}_{\mathcal{E}}$, where valuable
pre-trained knowledge can be maximally preserved after the editing. Here, we
note that locality is especially important for editing LLMs, as the knowledge
that needs to be updated often occupies only a small fraction of all knowledge
encompassed by the pre-trained model. In other words, the output of an edited
model regarding most input prompts should remain consistent with the output
before editing.

##  4\. Evaluation Metrics

Before introducing the taxonomy of KME and the exemplar methods in detail, in
this section, we first discuss various metrics commonly used to evaluate the
effectiveness of different KME strategies from varied perspectives. We
summarize the metrics to facilitate the understanding in terms of the
properties and advantages of different methods.

###  4.1. Accuracy

Accuracy is the most straightforward measure for the effectiveness of KME
techniques, which is defined as the success rate of editing in terms of a
specific set of pre-defined input-output pairs
$\mathcal{E}=\\{x_{i},y^{*}_{i}\\}$ associated with all the edited knowledge.
Accuracy can be easily defined to evaluate the performance of KME on
classification tasks, e.g., fact checking (Mitchell et al., 2022b; Petroni et
al., 2021), where the answers $y$ are categorical. If we define the prompt and
ground truth related to an edit $e$ as $x_{e}$ and $y^{*}_{e}$, respectively,
the metric of the accuracy of an edited model $f^{*}$ can be formulated as
follows:

(3)
$\textbf{Acc}(f^{*};\mathcal{E})=\mathbb{E}_{e\in\mathcal{E}}\mathbbm{1}\\{f^{*}(x_{e})=y^{*}_{e}\\}.$

Since accuracy is defined on a deterministic set of prompts-answer pairs
$\mathcal{E}=\\{x_{i},y^{*}_{i}\\}$, it provides a fair comparison between KME
methods. Nevertheless, it is challenging to evaluate the practicality of KME
methods with accuracy, as there is no consensus on how to design the
$\mathcal{E}$, especially when the task needs to output a long sequence such
as question answering or text generation (Meng et al., 2022; Dong et al.,
2022a; Meng et al., 2023).

###  4.2. Locality

One crucial property for the KME strategies is locality, which measures the
capability of the edited model $f^{*}$ to preserve the pre-trained knowledge
in $f$ irrelevant to the editions $\mathcal{E}$. We note that in most cases of
KME, the number of required edits makes for an extremely small fraction of the
entire knowledge learned and preserved in the pre-trained LLMs. Consequently,
the locality measurement is of great importance in assessing the capability of
edited models to preserve unrelated knowledge. Given an edit $e$, an edited
model $f^{*}$, and the original pre-trained model $f$, the locality of $f^{*}$
can be defined as the expectation of unmatched agreement between the edited
model and unedited model for out-scope inputs, which can be defined as
follows:

(4)
$\textbf{Loc}(f^{*},f;e)=\mathbb{E}_{x\notin\mathcal{X}_{e}}\mathbbm{1}\\{f^{*}(x)\neq
f(x)\\}.$

We can also consider the locality regarding the entire edit set $\mathcal{E}$,
which can be defined as follows:

(5)
$\textbf{Loc}(f^{*},f;\mathcal{E})=\mathbb{E}_{x\notin\mathcal{X}_{\mathcal{E}}}\mathbbm{1}\\{f^{*}(x)\neq
f(x)\\},\ \ \text{where}\ \
\mathcal{X}_{\mathcal{E}}=\bigcup_{e\in\mathcal{E}}\mathcal{X}_{e}.$

Although the above metric measures the overall locality of $f^{*}$ based on
all inputs that are not in $\mathcal{X}_{\mathcal{E}}$, it is difficult to
compute in realistic scenarios, as the entire input space can be excessively
large or even infinite. Therefore, existing methods generally resort to
alternative solutions that pre-define the specific range of out-scope inputs
to calculate the expected difference in Eqs. (4) and (5). For example, in
SERAC (Mitchell et al., 2022b), the authors generate hard out-scope examples
from the dataset zsRE (Levy et al., 2017) by selectively sampling from
training inputs with high semantic similarity with the edit input, based on
embeddings obtained from a pre-trained semantic embedding model. If we denote
the out-scope input space related to the input $\mathcal{X}_{e}$ as
$\mathcal{O}_{e}$, we can similarly define the feasible out-scope input space
for multiple edits as
$\mathcal{O}_{\mathcal{E}}=\bigcup_{e\in\mathcal{E}}\mathcal{O}_{e}$. In this
manner, we define a specific metric of locality of $f^{*}$ regarding
$\mathcal{E}$ as follows:

(6)
$\textbf{Loc}(f^{*},f;\mathcal{O}_{e})=\mathbb{E}_{x\in\mathcal{O}_{e}}\mathbbm{1}\\{f^{*}(x)\neq
f(x)\\},$ (7)
$\textbf{Loc}(f^{*},f;\mathcal{O}_{\mathcal{E}})=\mathbb{E}_{x\in\mathcal{O}_{\mathcal{E}}}\mathbbm{1}\\{f^{*}(x)\neq
f(x)\\},\ \ \text{where}\ \
\mathcal{O}_{\mathcal{E}}=\bigcup_{e\in\mathcal{E}}\mathcal{O}_{e}.$

###  4.3. Generality

Aside from locality, another crucial property to measure the effectiveness of
KME is generality, which indicates the capability of the edited model $f^{*}$
to generalize the updated knowledge to correctly answer prompts from other in-
scope inputs not appearing in the training set that convey similar or related
meanings. Ensuring the generality of edited models prevents the edited model
from overfitting to a particular input. Specifically, in the scenarios of
knowledge-based model editing, the inherent diversity of natural language
determines that various in-scope inputs $x$ can correspond to a specific
knowledge triple $t$ (Wang et al., 2023b). These semantically equivalent
inputs can involve differences in aspects such as syntax, morphology, genre,
or even language. Existing works mostly pre-define a specific in-scope input
space of each edit via different strategies. For example, in the CounterFact
dataset proposed in ROME (Meng et al., 2022), the authors utilize prompts that
involve distinct yet semantically related subjects as the in-scope input. In
general, the generality of an edited model $f^{*}$ is defined as the
expectation of exact-match agreement between the output of the edited model
and true labels for in-scope inputs, which can be defined on either an edit
$e$ or the edit set $\mathcal{E}$ as follows:

(8)
$\textbf{Gen}(f^{*};e)=\mathbb{E}_{x\in\mathcal{X}_{{e}}}\mathbbm{1}\\{f^{*}(x)\in\mathcal{Y}_{e}^{*}\\},$
(9)
$\textbf{Gen}(f^{*};\mathcal{E})=\mathbb{E}_{x\in\mathcal{X}_{\mathcal{E}}}\mathbbm{1}\\{f^{*}(x)\in\mathcal{Y}_{e}^{*}\\},\
\ \text{where}\ \
\mathcal{X}_{\mathcal{E}}=\bigcup_{e\in\mathcal{E}}\mathcal{X}_{e}.$

###  4.4. Retainability

Retainability characterizes the ability of KME techniques to preserve the
desired properties of edited models after multiple consecutive edits. In the
presence of ever-evolving information, practitioners may need to frequently
update a conversational model (i.e., sequential editing). Such a KME setting
requires that the model not forget previous edits after each new modification.
It is essential to distinguish retainability from scalability, which evaluates
the model’s ability to handle a vast number of edits. In contrast,
retainability assesses the consistent performance of the model after each
individual edit, presenting a more challenging objective to achieve. Recently,
T-Patcher (Huang et al., 2023) first explores the sequential setting of KME
and observes that many existing approaches significantly fall short in terms
of retainability. In SLAG (Hase et al., 2023), the authors also discover a
significant drop in editing performance when multiple beliefs are updated
continuously. To thoroughly assess the retainability of an edited language
model $f^{*}$, we define it as follows:

(10)
$\displaystyle\mathbf{Ret}(M;\mathcal{E})=\frac{1}{|\mathcal{E}|-1}\sum\limits_{i=1}^{|\mathcal{E}|-1}\mathbf{Acc}(M(f;\\{e_{1},e_{2},\dotsc,e_{i+1}\\}))-\mathbf{Acc}(M(f;\\{e_{1},e_{2},\dotsc,e_{i}\\}))$

where $\mathbf{Acc}$ is the accuracy measurement, $|\mathcal{E}|$ is the
number of edits in the edit set, and $M$ denotes the editing strategy that
modifies the pre-trained model $f$ into $f^{*}$ with $i/i+1$ consecutive edits
$\\{e_{1},e_{2},\dotsc,e_{i},(e_{i+1})\\}$. The retainability metric aims to
quantify the effect of applying consecutive edits to a model and measures how
the performance will change the editing strategy $M$, where a higher
retainability means that after each edit, the less the change in the overall
performance of the edited model $f^{*}$ is required.

###  4.5. Scalability

The scalability of an editing strategy refers to its capability to incorporate
a large number of edits simultaneously. Recently, several works have emerged
that can inject multiple new knowledge into specific parameters of pre-trained
LLMs. For instance, SERAC (Mitchell et al., 2022b) can perform a maximum of 75
edits. In addition, MEMIT (Meng et al., 2023) is proposed to enable thousands
of edits without significant influence on editing accuracy. When there is a
need to edit a model with a vast number of edits concurrently, simply
employing the current knowledge-based model editing techniques in a sequential
manner is proven ineffective in achieving such scalability (Yao et al., 2023).
To effectively evaluate the scalability of edited language models, we define
the scalability of an edited model as follows:

(11)
$\mathbf{Sca}(M;\mathcal{E})=\mathbb{E}_{e\in\mathcal{E}}\mathbf{Acc}(M(f;e))-\mathbf{Acc}(M(f;\mathcal{E})),$

where $\mathbf{Acc}(M(f;\mathcal{E}))$ denotes the accuracy of the edited
model after conducting all edits in $\mathcal{E}$, whereas
$\mathbf{Acc}(M(f;e))$ is the accuracy of only performing the edit $e$.
$\mathbf{Sca}$ demonstrates the model performance and practicality in the
presence of multiple edits. Nevertheless, we note that baseline value
$\mathbf{Acc}(M(f;\\{e\\}))$ is also important in evaluating the scalability
of various models. This is because, with higher accuracy for each $e$, the
retainment of such performance after multiple edits is more difficult.
Therefore, we further define the relative version of Eq. (11) as follows:

(12)
$\mathbf{Sca}_{rel}(M;\mathcal{E})=\left(\mathbb{E}_{e\in\mathcal{E}}\mathbf{Acc}(M(f;\\{e\\}))-\mathbf{Acc}(M(f;\mathcal{E}))\right)/\mathbb{E}_{e\in\mathcal{E}}\mathbf{Acc}(M(f;\\{e\\})).$

The introduced scalability measurement further considers the magnitude of the
original accuracy to provide a fairer evaluation.

Figure 3. The detailed categorization of knowledge-based model editing (KME)
techniques for LLMs. Table 2.  Comparisons among different categories of
methods across the four metrics.

Category | Strategy | Locality | Generality | Retainability | Scalability | References
---|---|---|---|---|---|---
External Memorization | Memory-based | ✔ |  |  | ✔ | (Mitchell et al., 2022b; Murty et al., 2022; Madaan et al., 2022; Li et al., 2023b; Zheng et al., 2023; Zhong et al., 2023)
Extension-based | ✔ |  | ✔ | ✔ | (Dong et al., 2022a; Huang et al., 2023; Hartvigsen et al., 2022)
Global Optimization | Meta-learning |  | ✔ |  | ✔ | (Sinitsin et al., 2020; Ha et al., 2016; De Cao et al., 2021; Hase et al., 2021; Mitchell et al., 2022a)
Subspace Fine-tuning |  | ✔ | ✔ | ✔ | (Aghajanyan et al., 2021; Qin et al., 2022)
Local Modification | Gradient-based | ✔ | ✔ | ✔ |  | (Dai et al., 2022; Meng et al., 2022, 2023; Li et al., 2023a)
Prompt-based | ✔ | ✔ |  |  | (Gupta et al., 2023; Meng et al., 2022)

##  5\. Methodologies

In this section, we introduce existing knowledge-based model editing (KME)
strategies in detail. We first provide an innovative taxonomy of existing KME
strategies based on how and where the new knowledge is injected into the pre-
trained LLMs, where the advantages and drawbacks are thoroughly discussed. We
then introduce various methods from each category, with an emphasis on
analyzing the technical details, insights, shortcomings, and their
relationships.

###  5.1. Categorization of KME Methods

Faced with the rapid deprecation of old information and the emergence of new
knowledge, various KME methodologies have been proposed to update the pre-
trained LLMs to maintain their updatedness and relevancy. KME ensures that new
knowledge can be efficiently incorporated into the pre-trained LLMs without
negatively influencing the pre-trained knowledge irrelevant to the edit. In
this survey, we categorize existing KME methods into three main classes as
follows:

  * •

External Memorization-based methods leverage an external memory to store the
new knowledge for editing without modifying the pre-trained weights, where the
pre-trained knowledge can be fully preserved in the LLM weights. By storing
new knowledge with external parameters, the memory-based strategies enable
precise representation of new knowledge with good scalability, as the memory
is easily extensible to incorporate new knowledge.

  * •

Global Optimization-based methods seek to achieve generalizable incorporation
of the new knowledge into pre-trained LLMs via optimization with the guidance
of new knowledge, where tailored strategies are introduced to limit the
influence of other pre-trained knowledge, distinguishing it from naive fine-
tuning. Nevertheless, these methods may fall short in editing efficiency when
applied to LLMs due to the large number of parameters to be optimized.

  * •

Local Modification-based methods aim to locate the related parameters of
specific knowledge in LLMs and update it accordingly to incorporate the new
knowledge relevant to the edit. The main advantage of local modification is
the possibility of only updating a small fraction of model parameters, thereby
providing considerable memory efficiency compared to memorization-based
methods and computational efficiency compared to global optimization.

The above categorization is based on where (e.g., external parameters or
internal weights) and how (e.g., via optimization or direct incorporation) new
information is introduced into the LLM. Specifically, methods from each
category exhibit different strengths and weaknesses regarding the four crucial
evaluation metrics introduced in Sec. 4. For example, external memorization
prevails in scenarios that require massive editing while the computational
resources are limited, as the size of the memory is controllable to fit into
different requirements. On the other hand, global optimization is advantageous
when practitioners focus more on the generality of edited knowledge, as the
optimization can promote the learning of relevant knowledge (Aghajanyan et
al., 2021). The taxonomy is visually illustrated in Fig 3, and the specific
characteristics of all methods are summarized in Table 2.

Figure 4. The illustration of three categories of KME methods: External
Memorization, Global Optimization, and Local Modification.

###  5.2. External Memorization

####  5.2.1. Overview

The editing approach via external memorization aims to modify the current
model $f_{\phi}$ (with parameter $\phi$) via introducing external memory
represented by additional trainable parameters $\omega$ that encodes the new
knowledge, resulting in an edited LLM model $f^{*}_{\phi,\omega}$. The
rationale behind the external memorization strategy is that storing new
knowledge in additional parameters is intuitive and straightforward to edit
the pre-trained LLMs with good scalability, as the parameter size can be
expanded to store more knowledge. In addition, the influence on the pre-
trained knowledge can be minimized as this strategy does not alter the
original parameters $\phi$. Based on the general formulation of LLM editing in
Eq. (3.1), the objective of LLM editing via external memorization can be
formulated as follows:

(13)
$\displaystyle\min\mathbb{E}_{e\in\mathcal{E}}\mathbb{E}_{x,y^{*}\in\mathcal{X}_{e},\mathcal{Y}^{*}_{e}}\mathcal{L}(f^{*}_{\phi,\omega}(x),y^{*}),\text{where}\
\ f^{*}_{\phi,\omega}=M(f_{\phi},\omega;\mathcal{E}),$
$\displaystyle\;\text{s.t.}\;f^{*}_{\phi,\omega}(x)=f_{\phi}(x),\ \ \forall
x\in\mathcal{X}\setminus\mathcal{X}_{\mathcal{E}},$

where $f_{\phi}$ denotes the LLM before editing with the pre-trained parameter
$\phi$, and $f^{*}_{\phi,\omega}$ denotes the edited LLM with $\phi$ and
additional parameter $\omega$ as the external memorization. Moreover, based on
whether the introduced parameters are directly incorporated into the model
process or not, external memorization strategies can be divided into two
categories, i.e., memory-based methods and extension-based methods.

####  5.2.2. Memory-based Strategies

In memory-based strategies, the external memory, outside the intrinsic
architecture of the pre-trained LLM, functions as a repository to store edited
knowledge. The LLM can access and update this memory as required during
inference.

One exemplar work is SERAC (Mitchell et al., 2022b), which stores the edited
samples $x,y^{*}\in\mathcal{X}_{e},\mathcal{Y}^{*}_{e}$ in a cache without
performing modifications on the original model. When presented with a new
prompt $x^{\prime}$, SERAC uses a scope classifier to determine whether the
prompt falls within the scope of any cached instances. If yes, the desirable
output $y^{\prime}$ associated with the new prompt $x^{\prime}$ is predicted
via a counterfactual model $f_{c}$ which utilizes the most relevant edit
example as follows:

(14)
$f^{*}_{\phi,\omega}(x)=\left\\{\begin{array}[]{ll}f_{\phi}(x),&\text{if}\ x\
\text{is not in scope of any edit},\\\
f_{c}(x,\mathcal{E}),&\text{otherwise}.\\\ \end{array}\right.$

SERAC is a gradient-free approach to KME without relying on gradients of the
target label $y^{*}$ w.r.t. the pre-trained model parameters. In addition to
using memory as an external repository, the desirable edits can also be stored
in the form of human feedback. For example, Language Patch (Murty et al.,
2022) performs editing by integrating patches in natural language and
MemPrompt (Madaan et al., 2022) involves human feedback prompts to address the
issue of lacking commonsense knowledge regarding a particular task. An
integral feature of the Language Patch (Murty et al., 2022) framework is its
ability to empower practitioners with the capability to create, edit, or
remove patches without necessitating frequent model re-training. This trait
not only streamlines the development process but also enhances the
adaptability and versatility of the edited model. To enable the automatic
correction in memory, MemPrompt (Madaan et al., 2022) equips the language
model with a memory bank containing corrective feedback to rectify
misunderstandings. Specifically, MemPrompt leverages question-specific
historical feedback to refine responses on novel and unencountered instances
through prompt adjustments.

In KAFT (Li et al., 2023b), controllability is achieved through the
utilization of counterfactual data augmentations. In this approach, the entity
representing the answer within the context is substituted with an alternative
but still plausible entity. This substitution is intentionally designed to
introduce a conflict with the genuine ground truth, thereby enhancing the
controllability and robustness of LLMs with respect to their working memory.
The aim is to ensure that LLMs remain responsive to pertinent contextual
information while filtering out noisy or irrelevant data.

In addition to relying on parameter-based memory, recent works also leverage
prompting techniques of LLMs, e.g., in-context learning (Dong et al., 2022b)
and chain-of-thought prompting (Wei et al., 2022), to promote editing
performance of external memorization. Specifically, IKE (Zheng et al., 2023)
introduces novel factual information into a pre-trained LLM via in-context
learning, where a set of $k$ demonstrations, i.e.,
$\omega=\\{x_{i},y^{*}_{i}\\}_{i=1}^{k}$, is selected as the reference points.
These demonstrations will alter the prediction of a target factual detail when
the input is influenced by an edit. Particularly, IKE guarantees a balance
between generality and locality via storing factual knowledge as prompts. The
process can be formulated as follows:

(15) $f^{*}_{\phi,\omega}(x)=f_{\phi}(\omega\|x),\ \text{where}\
\omega=\\{x_{i},y^{*}_{i}\\}_{i=1}^{k}.$

Here $\|$ denotes the concatenation of the reference points in $\omega$ and
the input $x$, which follows an in-context learning manner. Note that in this
process, the framework first transforms all new facts into natural language to
input them into LLMs. Although IKE effectively edits factual knowledge via in-
context learning, it cannot solve more complex questions that involve multiple
relations. To deal with this, MQuAKE (Zhong et al., 2023) first explores the
evaluation of the editing effectiveness in language models regarding multi-hop
knowledge. For example, when editing knowledge about the president of a
country, the query regarding the president’s children should change
accordingly. Specifically, MQuAKE aims to assess whether edited models can
correctly answer questions that represent such chains of facts. The authors
propose to enable editing based on multi-hop knowledge by breaking down each
question into iterative subquestions, such that the base model generates a
provisional answer. Subsequently, each subquestion is used to retrieve the
most pertinent fact from the edited fact memory. The language model then
assesses whether the retrieved fact contradicts the provisional answer and
adjusts the prediction accordingly.

####  5.2.3. Extension-based Strategies

Extension-based strategies utilize supplementary parameters to assimilate
modified or additional information into the original language model. These
supplementary parameters are designed to represent the newly introduced
knowledge or necessary adjustments tailored for specific tasks or domains.
Different from memory-based methods, by incorporating new parameters into the
language model, extension-based approaches can effectively leverage and expand
the model’s functionality.

Extension-based methods can be implemented through various means, and one
representative way is to modify the Feed-forward Neural Network (FFN) output.
For example, CALINET (Dong et al., 2022a) uses the output from sub-models
fine-tuned specifically on factual texts to refine the original FFN output
produced by the base model. Another technique T-Patcher (Huang et al., 2023)
introduces a limited number of trainable neurons, referred to as “patches,” in
the final FFN layer to alter the model’s behavior while retaining all original
parameters to avoid reducing the model’s overall performance. Generally, these
methods that refine the structure of FFN can be formulated as follows:

(16)
$\operatorname{FFN}(\textbf{h})=\operatorname{GELU}\left(\textbf{h}\mathbf{W}_{1}\right)\mathbf{W}_{2}+\operatorname{GELU}\left(\mathbf{h}\cdot\mathbf{k}_{p}+b_{p}\right)\cdot\mathbf{v}_{p},$

where $\mathbf{k}_{p}$ is the patch key, $\mathbf{v}_{p}$ is the patch value,
and $b_{p}$ is the patch bias scalar. The introduced patches are flexible in
size and can be accurately activated to edit specific knowledge without
affecting other model parameters.

Alternatively, a different technique involves integrating an adapter into a
specific layer of a pre-trained model. This adapter consists of a discrete
dictionary comprising keys and values, where each key represents a cached
activation generated by the preceding layer and each corresponding value
decodes into the desired model output. This dictionary is systematically
updated over time. In line with this concept, GRACE (Hartvigsen et al., 2022)
introduces an adapter that enables judicious decisions regarding the
utilization of the dictionary for a given input, accomplished via the
implementation of a deferral mechanism. It is crucial to achieve a balance
between the advantages of preserving the original model’s integrity and the
practical considerations associated with storage space when implementing this
approach.

####  5.2.4. Summary

The External Memorization editing methodology operates by preserving the
parameters within the original model while modifying specific output results
through external comparisons and interventions. One notable advantage of this
approach is its minimal perturbation of the original model, thereby ensuring
the consistency of unedited knowledge. It allows for precise adjustments
without necessitating a complete overhaul of the model’s architecture.
However, it is imperative to acknowledge a trade-off inherent in this
methodology. Its efficacy is contingent upon the storage and invocation of the
edited knowledge, a factor that leads to concerns regarding storage capacity.
Depending on the scale of knowledge to be edited, this approach may entail
substantial storage requisites. Therefore, cautiously seeking a balance
between the advantages of preserving the original model’s integrity and the
practical considerations of storage capacity becomes a pivotal concern when
employing this particular approach.

###  5.3. Global Optimization

####  5.3.1. Overview

Different from external memorization-based methods that introduce new
parameters to assist the editing of pre-trained LLMs, there also exist
branches of works that do not rely on external parameters or memory.
Concretely, the Global Optimization strategy aims to inject knowledge into
LLMs by updating all parameters, i.e., $\phi$ in Eq. (13). Through fine-tuning
model parameters with specific constraints to ensure the preservation of
knowledge irrelevant to the target knowledge $t^{*}$, the LLMs are endowed
with the ability to absorb new information without altering unedited
knowledge. Generally, the goal of global optimization methods can be
formulated as follows:

(17)
$\displaystyle\min\mathbb{E}_{e\in\mathcal{E}}\mathbb{E}_{x,y^{*}\in\mathcal{X}_{e},\mathcal{Y}^{*}_{e}}\mathcal{L}(f_{\phi^{*}}(x),y^{*}),\
\text{where}\ \ f_{\phi^{*}}=M(f_{\phi};\mathcal{E}),$
$\displaystyle\;\text{s.t.}\;f_{\phi^{*}}(x)=f_{\phi}(x),\ \ \forall
x\in\mathcal{X}\setminus\mathcal{X}_{\mathcal{E}},$

where $f_{\phi}$ denotes the LLM before editing with the pre-trained parameter
$\phi$, and $f_{\phi^{*}}$ denotes the edited LLM with updated parameter
$\phi^{*}$. Generally, these methods focus more on the precision and
generality of desirable knowledge, as the fine-tuning process ensures that the
LLMs achieve satisfactory results regarding the edits and relevant knowledge.
Nevertheless, as fine-tuning affects all parameters, they cannot easily
preserve the locality of edited models, i.e., maintaining consistent output
for unedited knowledge (Yao et al., 2023). In practice, directly applying
fine-tuning strategies typically exhibits suboptimal performance on KME due to
overfitting concerns (Wang et al., 2023b; Meng et al., 2023). Furthermore,
fine-tuning large language models is also time-consuming and lacks scalability
for multiple edits. Therefore, it is imperative to devise specific constraints
for global optimization to tackle these challenges in KME.

A simple and straightforward idea is to fine-tune the model while preventing
it from updating on non-target knowledge in
$\\{\mathcal{X}\setminus\mathcal{X}_{\mathcal{E}},\mathcal{Y}\setminus\mathcal{Y}_{\mathcal{E}}\\}$,
thereby transforming the objective in Eq. (17) into a constrained optimization
problem:

(18)
$\displaystyle\min\mathbb{E}_{e\in\mathcal{E}}\mathbb{E}_{x,y^{*}\in\mathcal{X}_{e},\mathcal{Y}^{*}_{e}}\mathcal{L}(f_{\phi^{*}}(x),y^{*}),\
\text{where}\ \ f_{\phi^{*}}=M(f_{\phi};\mathcal{E}),$
$\displaystyle\;\text{s.t.}\;\
\|\mathcal{L}(f_{\phi^{*}}(x),y)-\mathcal{L}(f_{\phi}(x),y)\|\leq\delta,\forall
x,y\in\mathcal{X}\setminus\mathcal{X}_{\mathcal{E}},\mathcal{Y}\setminus\mathcal{Y}_{\mathcal{E}},$

where $\phi$, $\phi^{*}$ are the parameters before and after updating,
respectively. $\delta$ is a scalar hyper-parameter to restrict the difference
between losses of $f_{\phi^{*}}$ and $f_{\phi}$. The constraint in Eq. (18)
restricts the change of the edited model on unmodified knowledge. Zhu et al.
(Zhu et al., 2020) first propose an approximate optimization constraint that
is easier for implementation and computation:

(19)
$\displaystyle\min\mathbb{E}_{e\in\mathcal{E}}\mathbb{E}_{x,y^{*}\in\mathcal{X}_{e},\mathcal{Y}^{*}_{e}}\mathcal{L}(f_{\phi^{*}}(x),y^{*}),\
\text{where}\ \ f_{\phi^{*}}=M(f_{\phi};\mathcal{E}),$
$\displaystyle\;\text{s.t.}\;\ \|\phi^{*}-\phi\|\leq\delta.$

The updates are simply regularized by restricting the norm of parameters
before and after updating. Despite direct fine-tuning, more recent works
(Aghajanyan et al., 2021; Qin et al., 2022) also introduce meta-learning and
subspace fine-tuning methods to address the issue of overfitting and improve
scalability.

####  5.3.2. Meta-learning Strategies

Meta-learning (Finn et al., 2017) addresses the issue of overfitting by
training a model that can quickly adapt to new tasks. By exposing the model to
a variety of tasks during training, meta-learning improves the model’s ability
to generalize from limited data and reduces the risk of overfitting individual
tasks. In the scenario of KME, the optimal model parameters $\phi^{*}$ should
minimize the expected loss over a variety of meta-tasks (Ravi and Larochelle,
2016):

(20)
$\phi^{*}=\operatorname*{argmin}_{\phi}\mathbb{E}_{D\sim\mathcal{D}}[\mathcal{L}_{\phi}({D})],$

where $\mathcal{D}$ corresponds to the sample set for each meta-task $D$.
Moreover, each meta task ${D}$ contains multiple $(x^{*},y^{*})$ pairs for
editing. In practice, such methods often introduce additional objective
functions or networks to regulate parameter updates. As a typical meta-
learning method for KME, Editable Training (Sinitsin et al., 2020) focuses on
effectively rectifying errors within models while preserving their performance
on other data instances. Following a model-agnostic training manner, the
authors introduce additional constraints to restrict parameter updates in a
different way. Specifically, the loss function is separated into
$\mathcal{L}_{base}$ (task-specific objective function), $\mathcal{L}_{edit}$
(computed on the edit set $\mathcal{X}_{\mathcal{E}}$), and
$\mathcal{L}_{local}$ (computed on samples in
$\mathcal{X}\setminus\mathcal{X}_{\mathcal{E}}$). Moreover, the models are
updated in a meta-learning manner, where $k$ steps of gradient descent would
be applied for parameters before computing the objective function.

While traditional meta-learning techniques have demonstrated remarkable
efficacy in tasks such as machine translation (Gu et al., 2018), they exhibit
instability when applied to KME, primarily due to the necessity of fine-
tuning. A potential solution to address this challenge is presented in Hyper-
Network  (Ha et al., 2016). The Hyper-Network is a smaller network that
generates the weights for a larger network, referred to as the main network.
Specifically, the Hyper-Network takes inputs that contain information about
the structure of the weights and generates the weights for layers in the main
network. With the generated weights, the main network learns to map input data
to desired output targets. The updating process for the main network, denoted
as $\phi$, can be defined as follows:

(21) $\displaystyle\phi^{*}$ $\displaystyle=\phi+\Delta\phi,$
$\displaystyle\Delta\phi$
$\displaystyle=\text{H}(\nabla_{\phi}\mathcal{L}(f_{\phi}(x),y^{*})),\
\text{where}\
x,y^{*}\in\mathcal{X}_{\mathcal{E}},\mathcal{Y}^{*}_{\mathcal{E}},$

where $\text{H}(\cdot)$ corresponds to the hyper-network. $\Delta\phi$ is the
weight deviation calculated by the hyper-network. According to a recent work
(von Oswald et al., 2022), task-specific Hyper-Networks (i.e., networks that
generate target model weights based on task attributes) are effective in
mitigating catastrophic forgetting issues. Therefore, such methods are
suitable for the setting of KME, which requires the preservation of unedited
knowledge.

Recently, researchers have proposed to adopt hyper-networks for parameter
updates in KME. Among them, KE (De Cao et al., 2021) proposes to edit
knowledge and rectify erroneous or unexpected predictions without expensive
fine-tuning. Specifically, it trains a hyper-network via constrained
optimization to modify facts without affecting pre-trained knowledge
irrelevant to the edit. The trained hypernetwork is then used to predict the
weight update at the inference time. Based on KE, SLAG (Hase et al., 2023)
further appends metrics for two types of input texts: (1) Inputs that are not
in the desired edit set $\mathcal{X}_{\mathcal{E}}$ but logically related to
$\mathcal{E}$; (2) Inputs that share a formal resemblance to edited knowledge,
but do not lead to changes in the prediction outcomes. However, hyper-networks
are generally not capable of updating large language models due to the massive
parameter size. To tackle this challenge, MEND (Mitchell et al., 2022a) adopts
a mechanism referred to as gradient decomposition. In particular, MEND uses
small auxiliary editing networks to transform the gradient obtained by
standard fine-tuning. With a low-rank decomposition of the gradients, MEND
achieves the transformation via a tractable parameterization of it. The
authors evaluate MEND on language models including T5 (Raffel et al., 2020),
GPT (Radford et al., 2018), BERT (Devlin et al., 2018), and BART (Lewis et
al., 2020). The results demonstrate that MEND effectively edits the behavior
of models with more than ten billion parameters.

####  5.3.3. Subspace Fine-tuning Strategies

Subspace fine-tuning strategies aim to fine-tune a model within a smaller and
constrained subspace of parameters (Li et al., 2018), instead of its full
parameter space. These methods can comprehensively reduce the computational
complexity and the number of updated parameters while improving the model
performance. A recent work (Ding et al., 2022) has shown that such subspace
fine-tuning methods can help decrease the generalization gap between training
and inference. Hence, due to the generality and efficiency of these methods,
they are widely adopted for knowledge-based model editing.

Denote $d_{\phi}$ as the dimension of all the model parameters $\phi$.
Generally, given a set of downstream tasks
$\mathcal{T}=\\{t_{1},...,t_{n}\\}$, the generalization gap between these
tasks could be located in a smaller parameter subspace $S_{\mathcal{T}}$ of
dimension $d_{s}$. Thus, subspace fine-tuning could be formulated as (Li et
al., 2018):

(22) $\phi^{*(d_{\phi})}=\phi^{(d_{\phi})}+\mathbf{P}\phi^{\prime(d_{s})},$

where $\mathbf{P}\in\mathbb{R}^{d_{\phi}\times d_{s}}$ is the dimension
expansion matrix, and $\phi^{\prime(d_{s})}$ is the vectorized parameter in
the subspace $S_{\mathcal{T}}$. While Eq. (22) does not offer direct
guidelines for establishing $\mathbf{P}$ or $\phi^{\prime(d)}$, numerous
methods have been introduced to address this challenge in implementation. In
the context of KME, the task set $\mathcal{T}$ is a partition of the entire
dataset.

Recently, SAID (Aghajanyan et al., 2021) explores the effectiveness of
language model fine-tuning through the lens of intrinsic dimensionality. The
framework employs the Fastfood transform (Le et al., 2013) as the expansion
matrix $\mathbf{P}$. Instead of opting for a uniform value throughout the
language model, the authors take into account the layer-wise architecture of
transformers and exchange $m$ parameters within $\phi^{\prime}$ for scaling
purposes. Consequently, the optimization strategy introduced is defined as
follows:

(23)
$\phi^{*(d_{\phi})}_{i}=\phi^{(d_{\phi})}_{i}+\lambda_{i}\mathbf{P}\phi^{{}^{\prime}(d_{s}-m)}_{i},$

where $i$ represents the layer index, and $m$ is the number of layers in the
model. SAID reallocates $m$ parameters from the subspace parameter
$\phi^{\prime(d_{s})}$, thereby enabling layer-wise scaling through the
jointly learned parameter $\lambda$. This adjustment empowers the SAID method
to allocate a greater capacity of $\phi^{{}^{\prime}(d_{s}-m)}$ towards
particular layers that contain more pertinent information for the specific
knowledge. The authors also connect intrinsic dimensionality with
generalization and show that pre-trained models with lower intrinsic
dimensions across various tasks achieve higher evaluation accuracy and lower
relative generalization gaps. Nevertheless, SAID requires extensive parameter
tuning, making it difficult to converge during training. To deal with this
challenge, IPT (Qin et al., 2022) proposes to find a common low-dimensional
intrinsic task subspace through a soft prompt after prompt tuning. The authors
conduct experiments to examine the universality of the subspace and discover
that it can be reparameterized as optimizing only a few free parameters.
Generally, IPT is effective for various few-shot NLP tasks, including
knowledge-based model editing. However, it is also noteworthy that soft
prompts are difficult to optimize, and the performance changes non-
monotonically in trainable parameters (Bakker et al., 2022).

####  5.3.4. Summary

KME methods based on global optimization typically apply specific fine-tuning
restrictions to regularize parameter updates. This is to prevent overfitting
and ensure the model’s performance on the unedited knowledge. One crucial
advantage of this strategy is its generality regarding the relevant knowledge,
i.e., in-scope inputs $\mathcal{X}_{e}$ of edit $e$. As the global
optimization affects all parameters in a language model, the relevant
knowledge in it will also be edited, thereby generalizing to such knowledge.
Furthermore, global optimization methods are mostly model-agnostic, which
means they can be applied to other editing methods. Nevertheless, such
possibilities are less explored in the context of KME. In terms of the
drawbacks, global optimization methods are suboptimal in maintaining the
locality of edited models, as the optimization can easily influence unedited
knowledge. Hence, it is crucial to achieve a balance between generality and
locality when optimizing language models with specific constraints.

###  5.4. Local Modification

####  5.4.1. Overview

To tackle the challenge of fine-tuning methods with respect to locality,
extensive research has been conducted on the local modification strategy for
KME tasks (Mitchell et al., 2022b). These techniques originate from the
concept of identifying and modifying specific relevant weights in a pre-
trained model to achieve desirable outputs. The primary objective is to first
locate the weights $\phi_{k}$ that store the knowledge in a pre-trained model
$f_{\phi}$ regarding the input $x$. Afterward, by adjusting these weights, it
becomes possible to generate the correct output $y^{*}$ from the same input
$x$ without re-training or fine-tuning the whole model. Recently, researchers
have generalized the local modification strategy to LLMs, where the efficiency
of information updates for pre-trained LLMs can be substantially improved.
Generally, the goal of the local modification strategy of KME can be
formulated as a constrained optimization problem with refined constraints as
follows:

(24)
$\displaystyle\min_{\phi^{*}_{k}}\mathbb{E}_{e\in\mathcal{E}}\mathbb{E}_{x,y^{*}\in\mathcal{X}_{e},\mathcal{Y}^{*}_{e}}\mathcal{L}(f^{*}_{\widebar{\phi}_{k},\phi_{k}^{*}}(x),y^{*}),$
$\displaystyle\;\text{s.t.}\;f^{*}_{\widebar{\phi}_{k},\phi_{k}^{*}}(x)=f(x),\
\ \forall x\in\mathcal{X}\setminus\mathcal{X}_{\mathcal{E}},$
$\displaystyle\text{where}\ \ \phi_{k}=L(f_{\phi},\mathcal{E}),\
\widebar{\phi}_{k}=\phi\setminus\phi_{k},\
f^{*}_{\widebar{\phi}_{k},\phi^{*}_{k}}=M(f_{\phi},\mathcal{E}).$

Here $\phi^{*}$ denotes the edited weights related to the new knowledge, and
$\widebar{\phi}_{k}$ denotes the unedited weights. In Eq. (24), the local
modification strategy for KME is naturally broken down into two steps: (1) The
locating step, denoted by function $L$, locates the relevant weights
$\phi_{k}$ in pre-trained model $f_{\phi}$ that store the obsolete information
regarding the query $x$, (2) The editing step, denoted by function $M$, edits
the located weights $\phi_{k}$ into new weights $\phi_{k}^{*}$ such that the
correct answer $y^{*}$ given the query $x$ can be generated by the model with
$\phi_{k}^{*}$. By only updating a small fraction of model weights, the
editing step avoids negatively influencing other irrelevant information,
(i.e., $x\in\mathcal{X}\setminus\mathcal{X}_{\mathcal{E}}$).

In the following subsections, we first introduce the concept of knowledge
neuron in LLMs, which are specific neurons that store factual knowledge and
can be activated to generate the desirable answer based on a certain query
$x$. Then we discuss two local modification strategies for KME: (1) the
gradient-based strategies, which identify and edit knowledge neurons based on
the model outputs; (2) the prompt-based strategies, which locate knowledge
neurons based on the input prompts.

Knowledge Neurons. LLMs pre-trained on large corpora can be viewed as
databases that store factual and common-sense knowledge in the pre-trained
model weights (Gupta et al., 2023). To update such knowledge by locally
modifying the weights in the pre-trained LLMs, it is imperative to identify
which weights store such information, i.e., locating the knowledge neurons.
This can be challenging due to the complex transformer architecture of LLMs
(Bakker et al., 2022).

As described in Section 2.2.1, the transformer structure of LLMs consists of
two primary types of layers, i.e., (1) the self-attention layer and (2) the
point-wise feed-forward (FFN) layer, which is implemented as a two-layer
multi-layer perceptron (MLP). Particularly, given a prompt $x$, the self-
attention layers of the LLMs use the query vector of the last token and the
key vectors of the previous tokens to calculate a weighted sum of their value
vectors. Therefore, given the input $x$, these layers provide information
about which previous tokens we should consider when generating the answer.
Here we provide a simplified example for illustration. To answer the question
“Who is the current president of the USA?”, the self-attention layer indicates
that the model should attend to words “president” and “USA”, i.e.,
$\textbf{v}_{president}$, $\textbf{v}_{USA}$, to determine the answer. This
provides us with a start-up embedding $\textbf{h}^{start}$ to generate the
answer token, which is the weighted sum of the values of the two attended
words, i.e., $w_{1}\textbf{v}_{president}+w_{2}\textbf{v}_{USA}$. However, the
information regarding the current president of the USA is not provided. In
contrast, recent works (Geva et al., 2021, 2022; Meng et al., 2022, 2023)
claim that the residual added to $\textbf{h}^{start}$ by the outputs of FNN
layers, i.e.,
$\textbf{h}^{next}=\textbf{h}^{start}+\operatorname{FFN}(\textbf{h}^{start})$,
injects the information “Biden” to $\textbf{h}^{start}$ and leads to the
generation of correct answers. Therefore, neurons in the FFN can be viewed as
the knowledge neurons that store the factual knowledge. The role of FFN in
storing knowledge can be theoretically analyzed by revisiting their
formulation in Eq. (1), which we rewrite as follows:

(25) $\displaystyle\operatorname{SelfAtt}_{i}(\textbf{x})$
$\displaystyle=\operatorname{Softmax}\left(\textbf{q}_{i}\textbf{k}_{i}^{\top}\right)\textbf{v}_{i}$
$\displaystyle\operatorname{FFN}(\textbf{h})$
$\displaystyle=\operatorname{GELU}\left(\textbf{h}\textbf{W}_{1}\right)\textbf{W}_{2}.$

Specifically, comparing the above two equations, we observe that the input h
to the FFN acts similarly to the query q to the SelfAtt. Moreover, the weights
of the first layer $\mathbf{W}_{1}$ can be viewed as the key $\mathbf{v}$,
where $\operatorname{GELU}\left(\textbf{h}\textbf{W}_{1}\right)$ can be viewed
as calculating an unnormalized attention score over the row vectors of
$\textbf{W}_{2}$. Finally, the weights of the second layer $\textbf{W}_{2}$
can be viewed as the value (or the memory) that stores the knowledge, which
can be retrieved according to the unnormalized weights calculated by the first
layer.

####  5.4.2. Gradient-based Strategies

Based on the knowledge neuron view of the FFN layer weights in pre-trained
LLMs, various gradient-based methods are proposed to locate and edit the pre-
trained LLMs. Generally, they perform editing in a top-down manner, utilizing
the supervision signal provided by the correct answer $y^{*}$. As an exemplar
work, KD (Dai et al., 2022) proposes to change each weight $w^{(l)}_{i}$
(i.e., the $i$-th weight in the $l$-th layer of FFN) from 0 to the pre-trained
value $\hat{w}^{(l)}_{i}$. Then it calculates the cumulative change in the
probability of predicting the output $y^{*}$ with input $x$, where the weights
with a high cumulative probability are considered relevant for knowledge
regarding $y^{*}$. In contrast to locating and editing an individual weight
${w}^{(l)}_{i}$ in the FFN layer, ROME (Meng et al., 2022) proposes to update
a whole FFN layer to encode the new knowledge $y^{*}$. Specifically, they view
the second layer weights $\textbf{W}_{2}$ in the FFN layer in Eq. (25) as a
linear associative memory (Kohonen, 1972; Anderson, 1972) in the form of
$\textbf{K}\textbf{W}_{2}=\textbf{V}$, where the keys K and values V
associated with $\textbf{W}_{2}$ can be directly calculated with Moore-Penrose
pseudoinverse. With such a view of $\textbf{W}_{2}$ in the FFN layer, the
optimization objective of updating it into $\hat{\textbf{W}}_{2}$ to encode
new knowledge in the edit $e=(s,r,o\rightarrow o^{*})$ can be formulated as
follows:

(26) $\min\|\textbf{K}\hat{\textbf{W}}_{2}-\textbf{V}\|\ \text{s.t.}\
\hat{\textbf{W}}\textbf{k}^{*}=\textbf{h}^{*}.$

Here $\textbf{k}^{*}$, which should encode the information of the subject $s$,
is calculated by sampling multiple $x\sim\mathcal{X}_{e}$ and taking the
average of the output of the first dense layer of the FFN. The target
activation $\textbf{h}^{*}$ is calculated via optimizing the probability of
outputting the correct answers $y^{*}\in\mathcal{Y}_{e}$ of the pre-trained
LLM via the subsequent layers. Then, an efficient rank-one update is conducted
on the weights $\textbf{W}_{2}$ according to Eq. (26), such that after the
update, the edited FFN layer can output the correct hidden representation
$\textbf{h}^{*}$ conducive to the generation of the right answer $y^{*}$ from
$\textbf{k}^{*}$. Recently, MEMIT (Meng et al., 2023) proposes to further
generalize the above editing strategy of the FFN layers of pre-trained LLMs to
the mass editing of different knowledge. Particularly, with $u$ new edits
$\\{e_{1},e_{2},...e_{u}\\}$ that are required to be updated in the weights
$\textbf{W}_{2}$, the mass editing problem can be formulated as the following
optimization problem:

(27)
$\min\left(\sum_{i=1}^{n}\left\|\textbf{k}_{i}\hat{\textbf{W}}_{2}-\textbf{v}_{i}\right\|^{2}+\sum_{i=n+1}^{n+u}\left\|\textbf{k}^{*}_{i}\hat{\textbf{W}}_{2}-\textbf{v}^{*}_{i}\right\|^{2}\right).$

where $\textbf{k}_{i}$, $\textbf{v}_{i}$ are the original key, value pairs
associated with the weights $\textbf{W}_{2}$ (i.e., row vectors in matrices K,
V in Eq. (26)), whereas $\textbf{k}_{i}^{*}$, $\textbf{v}^{*}_{i}$ are the
updated key, value pairs calculated from the $i$-th edit $e_{i}$ as with Eq.
(26). In addition, since multiple edits are required, the update is shared
among different MLP layers, which is conducted in a top-down manner to prevent
the potential issue of editing layers that could affect the ones that have
already been edited. The residual for each edit is spread evenly over the
range of the critical FFN layer. The strategy of residual attribution has
recently been improved by PMET (Li et al., 2023a), which adopts a square root
strategy to spread residuals to bottom FFN layers such that more precise
information can be conveyed to critical layers.

####  5.4.3. Prompt-based Strategies

Tailored to characteristics of LLMs that provide answer $y^{*}$ based on the
prompt $x$, the operation of locating and editing knowledge neurons can also
be conducted in a bottom-up manner, which aims to change the prompt to detect
neurons to be edited. Specifically, by masking out the key information and
observing the difference of activations in the intermediate layers of the LLM,
the weights that store the information regarding the query $x$ can be located
and updated to store the new information $y^{*}$. For example, ROME (Meng et
al., 2022) proposes a corruption-and-restore based strategy to identify
relevant layers (or their hidden output variables h) that store the
information based on the prompt $x$. It first randomly masks the hidden
representations of the key vectors $\mathbf{k}$ (as described in Eq. (1)) of
the tokens in the prompts from a certain intermediate layer of the pre-trained
LLM. Then it calculates the reduced probability of predicting $y$ (i.e., the
obsolete outputs) as the causal mediation effects of $x$ on $y$ mediated by h.
Consequently, the weights in layers with large mediated effects are viewed as
knowledge neurons that store the information of $y$.
$\text{{MEMIT}}_{\text{{CSK}}}$ (Gupta et al., 2023) extends the above
corruption-based strategy to editing common sense knowledge. The authors argue
that, different from the factual knowledge that can be directly retrieved by
the subject $s$, the object $o$ and relation $r$ also matter for commonsense
knowledge. Therefore, three types of corruption and edit locations, i.e.,
subject, verb, and object, are thoroughly analyzed, where the performance of
editing commonsense knowledge can be improved.

####  5.4.4. Summary

In this part, we introduce the local modification strategy for pre-trained
LLMs for efficient updates of new information without adding new weights or
optimizing the whole network. We start by analyzing the pivotal role of the
point-wise feedforward layers, i.e., the FFNs, to store the factual
information in pre-trained LLMs, with the knowledge neurons associated with
the FFN layer thoroughly analyzed. We then discuss the gradient-based and
prompt-based strategies that conduct local modification of LLMs, which achieve
the modification in a top-down manner based on the output $y$ and in a bottom-
up manner with the input prompt $x$, respectively. Nevertheless, the
scalability and retainability of local modification methods lack further
improvements, as the performance might deteriorate with more edits performed
(Meng et al., 2023).

Table 3.  Statistics of the KME datasets.

Dataset | Type | # Train | # Test | Input Format | Output Format | Used in
---|---|---|---|---|---|---
zsRE | Relational | 244,173 | 244,173 | Factual Statement | Object | (De Cao et al., 2021; Mitchell et al., 2022a; Meng et al., 2022; Mitchell et al., 2022b; Huang et al., 2023; Hartvigsen et al., 2022; Meng et al., 2023; Lee et al., 2022)
WikiGen | Generation | N/A | 68k | Wiki Passage | Continuation | (Mitchell et al., 2022a)
T-REx-100/-1000 | Relational | N/A | 100/1,000 | Factual Statement | Object | (Dong et al., 2022a; Li et al., 2023b)
CounterFact | Relational | N/A | 21,919 | Factual Question | Object | (Meng et al., 2022; Zheng et al., 2023; Meng et al., 2023)
ParaRel | Relational | N/A | 253,448 | Factual Question | Object | (Dai et al., 2022)
NQ-SituatedQA | QA | N/A | 67.3k | User Query | Answer | (Lee et al., 2022; Dai et al., 2023)
MQuAKE-CF/-T | Relational | N/A | 9,218/1,825 | Multi-hop Question | Object | (Zhong et al., 2023)
FEVER | Fact Checking | 104,966 | 10,444 | Fact Description | Binary Label | (De Cao et al., 2021; Mitchell et al., 2022a; Huang et al., 2023)
ConvSent | Sentimental | 287,802 | 15,989 | Topic Opinion | Sentiment | (Mitchell et al., 2022b)
Bias in Bio | Biographical | 5,000 | 5,000 | Biographical Sentence | Occupation | (Hernandez et al., 2023)
VitaminC-FC | Fact Checking | 370,653 | 55,197 | Fact Description | Binary Label | (Mitchell et al., 2022b)

##  6\. Datasets

Recently, multiple datasets have been established to facilitate the evaluation
of KME methods. In this section, we summarize the commonly-used datasets in
Table 3 to benefit future KME research. Specifically, these datasets can be
divided into two groups: textual-output datasets (i.e., generation tasks) and
categorical-output datasets (i.e., classification tasks). The datasets are
obtained from a variety of sources, including knowledge graphs, Wikipedia
pages, crowd-sourced responses, etc., which are adapted by researchers to fit
into the KME setting.

###  6.1. Generation Datasets

For generation datasets, the target is in the form of textual content that is
required to be generated by LLMs. Serving as pivotal resources to evaluate KME
methods, most generation datasets are based on relational knowledge and used
for assessing the ability of editing techniques to inject new factual
knowledge. This is because relational datasets preserve more definitive
answers for each input and thus are more convenient and precise for
evaluation. Specifically, these datasets are generally curated from the
corresponding relational datasets to encompass diverse relational contexts,
ranging from question-answer pairs to intricate multi-hop queries. Therefore,
the most prevalent output format is an object to be predicted.

In this subsection, we present the most representative generation datasets,
shedding light on their unique attributes, the nature of their content, and
the specific challenges they present for evaluating KME methods on factual
knowledge as follows:

  * •

zsRE (Levy et al., 2017): zsRE is one of the most prevalent Question Answering
(QA) datasets extended and adopted by (De Cao et al., 2021; Mitchell et al.,
2022a) for KME evaluation. zsRE is suitable for evaluating KME due to its
annotations of human-generated question paraphrases, which allow researchers
to assess the model resilience to semantically equivalent inputs. In zsRE,
each relation is associated with a set of crowd-sourced template questions,
such as "What is Albert Einstein’s alma mater?". Each data entry cites a
Wikipedia sentence that serves as the factual basis or provenance. The dataset
also contains negative examples, which are generated by pairing a valid
question with a random sentence.

  * •

WikiGen (Mitchell et al., 2022a): Firstly proposed in MEND (Mitchell et al.,
2022a), WikiGen consists of approximately 68k question-answer pairs, with a
similar size to zsRE. Here, each question corresponds to a sentence randomly
sampled from Wikitext-103, and each answer is a 10-token sample obtained from
a pre-trained distilGPT-2 model (Ma, 2021). It is noteworthy that greedy
10-token prediction of the base model only aligns with edit targets for less
than 1% of samples.

  * •

T-REx-100 & T-REx-1000 (Elsahar et al., 2018): First used in CALINET (Dong et
al., 2022a), the authors adopt the classic relational dataset T-REx (Elsahar
et al., 2018) for evaluating model editors by extracting factual triplets of
varying sizes (100 and 1,000). Particularly, for each triplet, the authors
insert the head and tail entities into the template in LAMA (Petroni et al.,
2019) based on the relation they share, which results in two datasets with 100
and 1,000 facts, respectively, for the purpose of false knowledge detection.
It should be noted that each fact in these datasets is represented by several
paraphrased sentences.

  * •

CounterFact (Meng et al., 2022): CounterFact is established to distinguish
superficial alterations in the word selections and significant, generalized
modifications in its foundational factual knowledge. Proposed in ROME (Meng et
al., 2022), each entry in CounterFact originates from a related record in
ParaRel (Elazar et al., 2021), containing a knowledge triple and meticulously
crafted prompt templates. It is important to note that all subjects,
relations, and objects in this tuple are recognized entities in Wikidata
(Vrandečić and Krötzsch, 2014).

  * •

ParaRel (Elazar et al., 2021): ParaRel is an expert-curated dataset that
comprises diverse prompt templates for 38 relations, sourced from the T-REx
dataset (Elsahar et al., 2018). Firstly used in KN (Dai et al., 2022), the
authors insert the head entity into each relational fact and set the tail
entity as a blank for prediction. To ensure a rich variety in templates,
relations with less than four prompt templates are excluded, resulting in 34
relations in total. Each of these relations, on average, preserves 8.63
distinct prompt templates, leading to a total of 253,448 knowledge-revealing
prompts for 27,738 relational facts.

  * •

NQ-SituatedQA (Kwiatkowski et al., 2019): NQ (Natural Questions) is a
comprehensive question-answering dataset originating from user searches. In
PPA (Lee et al., 2022), the authors utilize NQ as the source knowledge while
excluding any outdated information as identified by SituatedQA (Zhang and
Choi, 2021) to create a novel dataset NQ-SituatedQA. SituatedQA is a dataset
containing questions within a subset of NQ that are dependent on specific time
and location. The authors then incorporate the time-dependent QA pairs from
this subset, annotated using the 2021 Wikipedia (Vrandečić and Krötzsch, 2014)
dump.

  * •

MQuAKE (Zhong et al., 2023): MQuAKE is constructed from Wikidata (Vrandečić
and Krötzsch, 2014) for evaluating the effectiveness of KME methods on multi-
hop questions. In particular, it is designed to assess whether the edited
models can correctly answer questions generated by chains of facts in plain
text. MQuAKE consists of two datasets. (1) MQuAKE-CF is a diagnostic dataset,
specifically crafted to evaluate KME methods in the context of counterfactual
edits. (2) MQuAKE-T focuses on temporal-based knowledge updates and is aimed
at assessing the effectiveness of KME techniques in updating outdated
information with contemporary factual data.

###  6.2. Classification Datasets

Classification datasets are also widely adopted to evaluate the effectiveness
of KME. These datasets consist of prompt-target pairs, where the target is a
discrete label instead of a textual sentence. In the context of KME, these
labels help ascertain the alignment of model performance with desired edits,
whether they are sentiment-based, fact-checking, or entaiKMEnt-related. The
advantages of classification datasets also involve their preciseness in
evaluation without the need to define the specific output space. In this
section, we summarize notable classification datasets that have been tailored
and leveraged for assessing KME techniques as follows:

  * •

FEVER (Thorne et al., 2018): FEVER is a fact-checking dataset originally
processed in KILT (Petroni et al., 2021) for verifying factual knowledge in
the form of binary classification. It necessitates the retrieval of sentence-
level evidence to determine whether a claim is supported or refuted, and is
widely used for evaluating the performance of KME. Specifically, FEVER
excludes claims labeled as lacking sufficient information, as they typically
do not provide any evidence to evaluate the claim.

  * •

ConvSent (Mitchell et al., 2022b): Firstly processed in SERAC (Mitchell et
al., 2022b), ConvSent is used to evaluate the capability of an editor to
modify a dialog agent’s sentiment about a particular topic without influencing
its responses to other topics. ConvSent is obtained from a list of 15,000 non-
numeric entities from zsRE (Levy et al., 2017; De Cao et al., 2021), combined
with 989 noun phrases from GPT-3 (Brown et al., 2020) for 15,989 topics.
Particularly, for each entity, there are ten positive and ten negative
sentiment completions, which can be noisy, from the BlenderBot model with 3B
parameters (Roller et al., 2021). The refined sentiment labels are achieved by
a sentiment classifier (Heitmann, 2020) pre-trained on RoBERTa (Liu et al.,
2019).

  * •

Bias in Bios (De-Arteaga et al., 2019): Bias in Bios is a dataset originally
proposed for fairness-related machine learning, containing approximately 397k
short professional biographies of online individuals, which are not relatively
famous. Each biographical sentence is assigned an associated occupation label
for the described person. To adopt this dataset for evaluating the performance
of KME methods, the authors of REMEDI (Hernandez et al., 2023) extract a
single sentence, modify it to display only the person’s first name, and then
query the language model with the prompt that follows the structure: "Person
has the occupation of…". Then they evaluate the relative probabilities of the
language model assigned to 28 potential occupations, where the language model
is considered to be correct if the ground-truth occupation is ranked top-1.

  * •

VitaminC-FC (Schuster et al., 2021): Firstly processed by SERAC (Mitchell et
al., 2022b), VitaminC-FC is constructed based on a fact-checking dataset,
VitaminC (Schuster et al., 2021). Particularly, VitaminC consists of more than
400,000 evidence-claim pairs, each of which is assigned a binary label to
indicate whether the evidence entails the claim. The dataset was gathered from
over 100,000 Wikipedia revisions that modify an underlying fact, along with
additional synthetic ones. In SERAC, the authors convert VitaminC into a KME
dataset by using the evidence as the edit descriptor and using claims from the
same Wiki pages accordingly as in-scope samples.

Table 4.  Examples of different downstream applications of KME: Question
Answering (QA), Fact Checking (FC), and Natural Language Generation (NLG).

Task | Edit Descriptor $e$ |  In-scope Input $x\sim\mathcal{X}_{e}$ |  Original Output $y\sim\mathcal{Y}_{e}$ |  Target Output $y\sim\mathcal{Y}_{e}^{*}$
---|---|---|---|---
QA | (Kazakhstan, Captital, | What is the capital of | Astana | Nur-Sultan
| Astana$\rightarrow$Nur-Sultan) | Kazakhstan? |  |
FC | (Marathon, Record, | Kipchoge holds the men’s | True | False
| Kipchoge$\rightarrow$Kiptum) | marathon world record. |  |
NLG | (Jordan Poole, Play In, | Provide a short introduction | Jordan Poole entered | In 2023, Jordan Poole transitioned
Warriors$\rightarrow$Wizards) | to Jordan Poole, describing | the Warriors’ rotation | from the Warriors to the Wizards,
| his current position. | recently. | remarking a significant change.

##  7\. Applications

KME can benefit multiple downstream applications with the ability to precisely
and efficiently inject knowledge into pretrained LLMs. In the following, we
will introduce several key applications of KME techniques in realistic
scenarios, where intuitive examples are provided in Table 4.

###  7.1. Question Answering

####  7.1.1. Background.

Question Answering (QA) is a core NLP task that aims to comprehend queries
posed by users in natural language and provide answers based on the encoded
knowledge in the pre-trained language model (Shin et al., 2020). Traditional
models for QA are generally fixed in their knowledge, capturing only the
information available at the training time of (Petroni et al., 2019; Jiang et
al., 2020). However, in our dynamic world, new information is generated
incessantly, which necessitates the constant update of QA models (Talmor et
al., 2018). Fortunately, KME methods enable the modification of QA models to
cater to specific questions without disrupting responses to other unrelated
inputs. Therefore, with KME strategies, the QA model can be efficiently
updated on the run, where the currentness of the model can be guaranteed.
Consequently, language model editing techniques have found broad applications
across a myriad of QA contexts with potentially distinct requirements (Lee et
al., 2022).

####  7.1.2. Existing Works.

The QA task encompasses various aspects, such as conversational QA,
definition-based QA, and notably, relation-based QA (Pandya and Bhatt, 2021).
Relation-based QA is primarily adopted as an evaluation benchmark as it
necessitates the retrieval of precise real-world facts in response to queries.
This particular emphasis on specific information retrieval renders relation-
based QA especially conducive to the benefits of KME techniques. For example,
PPA (Lee et al., 2022) introduces an innovative task of CuQA (Continuously-
updated QA), which intentionally emphasizes recurrent, substantial edits for
language models to constantly update them with new information. An important
aspect of the CuQA task is to ensure that the existing pre-trained knowledge
remains unaltered with the integration of new knowledge. Therefore, this
property is one important evaluation to assess model editing in CuQA tasks. In
MQuAKE (Zhong et al., 2023), the authors innovatively propose a multi-hop QA
task that involves answering questions generated by chains of facts in plain
text. Specifically, the task requires edited models to infer implicit
relations that can be several hops away from the objects in the edit. For
example, when a language model is modified regarding the president of the USA,
an ideal model should also authentically alter answers to “Who is the son of
the president of the USA”, which is a two-hop relation. Such a task is
significantly more challenging as it necessitates the model to alter its
reasoning results in addition to the original edit. Nevertheless, the proposed
method MeLLo in MQuAKE still exhibits outstanding performance on this
difficult task, demonstrating the potential of KME in generalizing edited
knowledge to multi-hop relations.

###  7.2. Fact Checking

####  7.2.1. Background.

Fact-checking (FC) is a pivotal task in journalism, information verification,
and combating misinformation that aims to scrutinize and affirm the
authenticity of claims, statements, or information in news articles, social
media, and other media content (Schuster et al., 2021). In a world overwhelmed
with ever-emerging information, fact-checking facilitates the trustworthiness
in the sharing of distributed information, promotes information transparency,
and aids individuals in making well-informed decisions (Thorne et al., 2018).
However, it is crucial to constantly update fact-checking models. For
instance, during the COVID-19 pandemic, initial understandings and guidelines
about the virus evolved as researchers gathered more data (Shahi et al.,
2021). A fact-checking model that cannot adapt to these rapidly changing facts
would quickly become outdated and potentially spread misinformation, thereby
requiring the application of language model editing. By integrating KME
techniques into fact-checking models to consistently update them with the
latest information and facts, it becomes possible to ensure the currentness,
trustworthiness, and accuracy of the model despite the persistent evolution of
information.

####  7.2.2. Existing Works.

Recently, several works have proposed to apply KME techniques in fact-checking
models. In (Zhu et al., 2020), the authors first explore the potential of
modifying specific factual knowledge within the transformer backbone of the
fact-checking model while ensuring that overall model performance remains
intact on facts irrelevant to the editing purpose. Particularly, they identify
the critical components within the transformer backbones conducive to
effective knowledge modifications. In SERAC (Mitchell et al., 2022b), the
authors propose to use evidence gathered from Wikipedia as edit descriptors to
update potentially outdated knowledge in the model. The proposed method
exhibits significant performance improvements over baselines and can be
generalized to other in-scope inputs collected from the same Wikipedia page.

###  7.3. Natural Language Generation

####  7.3.1. Background.

KME techniques are also promising to ensure the relevancy of the Natural
Language Generation (NLG) task, which aims to generate coherent and
contextually relevant content based on provided instructions (REITER and DALE,
1997). Considering the rapid evolution of the global information landscape, it
is essential for NLG models to remain up-to-date and ensure the accuracy of
generated text while avoiding potentially false statements that may mislead
the users.

####  7.3.2. Existing Works.

In practice, several works have been proposed to introduce KME in natural
language generation tasks. For instance, FRUIT (au2 et al., 2022) proposes to
update outdated Wikipedia articles according to the collection of new
information about the article’s subject. Based on the T5 model (Raffel et al.,
2020), the authors utilize a compressed output format to eliminate the
necessity of generating the entire update from scratch and promote thoughtful
content structuring, which effectively handles the challenge of incoherence.
In MEND (Mitchell et al., 2022a), the authors apply their proposed method in
the Wikitext generation task, where the edited model is required to produce
credible 10-token extensions based on a provided Wikitext prefix (Ma, 2021).
With modification on multi-layer token-wise activations and gradients, the
edited model presents higher coherence on the NLG task, which demonstrates the
effectiveness of KME in generating target texts with richer information than
QA or FC.

##  8\. Discussion

Despite the continual progress of works on KME, several critical aspects have
been inadequately addressed by existing studies. Delving deeper into these
challenges could offer researchers fresh insights and pave the way for the
further advancement of the field. Consequently, in this section, we outline
the pressing challenges that await solutions in KME.

###  8.1. Challenges

####  8.1.1. Trade-off between Locality and Generality.

In KME, it is crucial to balance two objectives, locality and generality (as
defined in Sec. 4), such that a higher edit success rate can be achieved with
minimal negative influence on other knowledge irrelevant to the edits. When
editing a language model, a potential trade-off might emerge between these two
desirable properties. As demonstrated in (Yao et al., 2023), local
modification methods, such as MEMIT (Meng et al., 2023) and ROME (Meng et al.,
2022) generally preserve a higher level of locality, as they locate precise
locations of target knowledge to conduct the edition, which does not largely
affect the unrelated weights. In addition, T-Patcher (Huang et al., 2023)
points out that increasing the size of memory increases locality while
decreasing the generality. These observations underscore the intricate balance
between locality and generality. However, it remains challenging to tackle the
trade-off problem and achieve a balance between these two desirable properties
of KME methods.

####  8.1.2. Theoretical Analysis.

While many current KME studies focus on developing effective methods to
enhance the editing performance regarding various desirable properties, there
exists a notable gap between the practical application and the comparatively
less discovered theoretical analysis. Recently, in (Tanno et al., 2022), the
authors provide theoretical support for the justification of identifying
harmful training examples and editing the model by erasing the information
from a Bayesian view. LEACE (Belrose et al., 2023) introduces an analytical
framework that offers a theoretical perspective for the task of erasing target
concept information from every layer in language models. However, to the best
of our knowledge, there still does not exist any comprehensive theoretical
analysis regarding the KME problem that involves novel knowledge. We hope that
future research will enrich the theoretical discourse that can deliver
profound insights into the substantial foundations of KME methods.

####  8.1.3. Editing at Scale.

Another crucial property that hinders the practical application of KME is
scalability — the ability of editing strategy to effectively perform a large
number of edits simultaneously (Mitchell et al., 2022a). For example,
conversational systems (Zheng et al., 2023) are expected to be constantly
updated to incorporate an enormous number of global events and the information
originating from them. However, as the number of applied edits increases, the
coherence of language models is severely jeopardized, as multiple edits might
contradict a broader spectrum of pre-existing knowledge in the models (Wang et
al., 2023b). This can lead to decreased editing performance in both locality
and generality metrics (Mitchell et al., 2022b). Although external
memorization methods can alleviate such problems with a larger size of
memories of additional parameters, they are still vulnerable if thousands of
edits are required (Meng et al., 2022). Moreover, simply adapting single-edit
techniques for a multi-edit environment by merely applying them sequentially
has been demonstrated to be proven suboptimal (Meng et al., 2023). Therefore,
the unique and intricate challenge of coherence renders editing at scale a
formidable task.

###  8.2. Future Directions

Despite the recent achievements in the development of KME strategies for
effective and efficient updating of new knowledge into LLMs, KME research is
still in its emerging stage. Several promising directions could be pursued to
further advance this field. Accordingly, we identify five inspiring and
important open problems worthy of exploration in the future as follows:

Optimization-Free Editing. Recently, prompt engineering has become a prevalent
solution for modifying the behaviors of pre-trained LLMs in a human-preferable
manner without the requirement of parameter update (Dong et al., 2022b). For
example, in-context learning provides task descriptions and/or demonstrations
in the form of plain text to promote the model performance (Brown et al.,
2020), which makes it a potentially more efficient and practical strategy for
language models. We note that IKE (Zheng et al., 2023) proposes a novel
framework that relies on demonstration contexts for KME without parameter
updating, which explicitly formats the demonstrations that can guide the
language model to copy, update, and retain the prediction of different
prompts. However, such a strategy is difficult to scale and usually has
unsatisfactory retention. Therefore, it remains a crucial while challenging
task to develop optimization-free KME methods.

Auto-Discovery of Editing Targets. Current KME methods mainly rely on human
expertise to identify and incorporate desirable knowledge into pre-trained
LLMs. This approach is inherently labor-intensive and can incur significant
costs, especially considering the vast and rapidly expanding new information
needed to be integrated into language models. A promising future direction
lies in the automation of the edits, which aims to identify, evaluate, and
prioritize new knowledge that needs to be integrated from raw resources such
as websites and social media. Through this strategy, the application of KME
can be streamlined, rendering it more practical and adaptable in real-world
scenarios. A straightforward solution would be crawling new knowledge and
transforming it into a knowledge base, querying LLMs for each knowledge
triple, and editing the wrong answer. However, such a strategy still lacks
efficiency. Therefore, it remains a crucial task to discover editing knowledge
from various resources without human effort.

Continual Editing. Current methodologies on LLM editing primarily consider
one-step offline editing (De Cao et al., 2021; au2 et al., 2022); however,
such an approach is not aligned with real-world applications where models
might continually encounter novel knowledge to be injected. For example, an
online question-answering (QA) model may continually encounter reports of
incorrect answers from end users, where the editing needs to be conducted on
the run (Huang et al., 2023). Therefore, an optimal KME technique should be
capable of instantaneously and continuously rectifying emergent issues. We
note that continual editing of pre-trained LLMs presents a unique challenge:
preventing the edited models from forgetting or contradicting previous edits.
Despite the inherent complexities, the persistent demand for continual editing
in real-world applications underscores the importance of developing KME
techniques to navigate this challenge effectively.

Robust Editing. An important direction for the advancement of KME lies in
enhancing its robustness. In an era where misinformation spreads rapidly, it
is urgent that edited models not only retain their accuracy but also resist
adversarial attacks and misinformation (Ganguli et al., 2022). Here, we should
note that the concept of robustness extends beyond just maintaining factual
accuracy; it involves fortifying the model against potentially adversarial
external perturbations (Perez et al., 2022). For example, if KME is
maliciously applied to inject harmful knowledge into language models, the
edited models can be easily transformed into tools for misinformation (Taori
et al., 2023). Therefore, to prevent such cases, it is crucial for KME
techniques to develop capabilities that can identify and counteract such
unwanted inputs, thereby enhancing their resilience against adversarial
actions. In practice, as the trend leans towards open-sourcing large language
models, it becomes ever more crucial to safeguard against potential
manipulations that can turn these models harmful.

Editable Fairness. With the wide application of large language models (LLMs)
to support decisions, the emphasis on fairness has grown significantly (Wang
et al., 2023a), which requires LLMs to fairly treat people with diverse
background (Abid et al., 2021). However, LLMs trained on large datasets
inevitably incorporate certain biases during this pre-training phase (Dong et
al., 2019). Fortunately, the precision and efficiency of KME techniques offer
a promising solution to mitigate such biases and promote fairness in pre-
trained LLMs. For instance, in a model designed to classify biographical
sentences with occupation (De-Arteaga et al., 2019), KME can be used to inject
nuanced knowledge about a particular profession, guiding the model towards a
more equitable understanding of individuals associated with that profession
(Hernandez et al., 2023). However, this remains a complex challenge, as
fairness often entails considering disparate groups of individuals rather than
specific people. This broader focus makes knowledge injection via KME a non-
trivial task. Despite these difficulties, the enhancement of fairness in
language models is paramount, and KME techniques present a promising avenue to
achieve this goal.

##  9\. Conclusions

In this survey, we present a comprehensive and in-depth review of knowledge-
based model editing (KME) techniques for precise and efficient updating of new
knowledge in pre-trained LLMs. We first formulate the KME problem as a
constrained optimization objective that simultaneously ensures the accuracy
and retention of editing, which is general to encompass different KME
strategies. We then provide an overview of the evaluation metrics for KME,
which sheds light on the desirable attributes of edited models. Subsequently,
we propose a structured taxonomy framework to systematically categorize
existing KME techniques. Within each category, we outline the central
challenges, elaborate on the representative methods, and discuss their
strengths and weaknesses. Furthermore, we summarize the datasets widely
utilized to assess KME techniques, highlighting that certain techniques demand
specific dataset structures for training or evaluation. To inspire researchers
to devise more practical implementations, we also spotlight the real-world
applications of KME techniques. Finally, we identify several potential
challenges for future research and provide insightful directions that are
conducive to further advancement of the field.

###### Acknowledgements.

This work is supported by the National Science Foundation under grants
(IIS-2006844, IIS-2144209, IIS-2223769, CNS2154962, and BCS-2228534), the
Commonwealth Cyber Initiative awards (VV-1Q23-007, HV-2Q23-003, and
VV-1Q24-011), the JP Morgan Chase Faculty Research Award, the Cisco Faculty
Research Award, the Jefferson Lab subcontract, and the UVA 4-VA collaborative
research grant.

## References

  * (1)
  * Abid et al. (2021) Abubakar Abid, Maheen Farooqi, and James Zou. 2021\.  Persistent anti-muslim bias in large language models. In _Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society_. 298–306.
  * Aghajanyan et al. (2021) Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. 2021\.  Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning. In _ACL_.
  * Anderson (1972) James A Anderson. 1972\.  A simple neural network generating an interactive memory.  _Mathematical biosciences_ 14, 3-4 (1972), 197–220.
  * au2 et al. (2022) Robert L. Logan IV au2, Alexandre Passos, Sameer Singh, and Ming-Wei Chang. 2022.  FRUIT: Faithfully Reflecting Updated Information in Text.  arXiv:2112.08634 [cs.CL]
  * Azamfirei et al. (2023) Razvan Azamfirei, Sapna R Kudchadkar, and James Fackler. 2023\.  Large language models and the perils of their hallucinations.  _Critical Care_ 27, 1 (2023), 1–2.
  * Bakker et al. (2022) Michiel Bakker, Martin Chadwick, Hannah Sheahan, Michael Tessler, Lucy Campbell-Gillingham, Jan Balaguer, Nat McAleese, Amelia Glaese, John Aslanides, Matt Botvinick, et al. 2022\.  Fine-tuning language models to find agreement among humans with diverse preferences. In _NeurIPS_.
  * Bau et al. (2020) David Bau, Steven Liu, Tongzhou Wang, Jun-Yan Zhu, and Antonio Torralba. 2020.  Rewriting a deep generative model. In _ECCV_.
  * Belrose et al. (2023) Nora Belrose, David Schneider-Joseph, Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman. 2023.  LEACE: Perfect linear concept erasure in closed form.  _arXiv preprint arXiv:2306.03819_ (2023).
  * Beltagy et al. (2019) Iz Beltagy, Kyle Lo, and Arman Cohan. 2019.  SciBERT: A Pretrained Language Model for Scientific Text. In _EMNLP-IJCNLP_.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020\.  Language models are few-shot learners. In _NeurIPS_ , Vol. 33. 1877–1901.
  * Chang et al. (2023) Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Kaijie Zhu, Hao Chen, Linyi Yang, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2023\.  A survey on evaluation of large language models.  _arXiv preprint arXiv:2307.03109_ (2023).
  * Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021\.  Evaluating large language models trained on code.  _arXiv preprint arXiv:2107.03374_ (2021).
  * Chiang and Lee (2023) Cheng-Han Chiang and Hung-yi Lee. 2023.  Can Large Language Models Be an Alternative to Human Evaluations?  _arXiv preprint arXiv:2305.01937_ (2023).
  * Chiche and Yitagesu (2022) Alebachew Chiche and Betselot Yitagesu. 2022.  Part of speech tagging: a systematic review of deep learning and machine learning approaches.  _Journal of Big Data_ 9, 1 (2022), 1–25.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022\.  Scaling instruction-finetuned language models.  _arXiv preprint arXiv:2210.11416_ (2022).
  * Dai et al. (2022) Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. 2022\.  Knowledge Neurons in Pretrained Transformers. In _ACL_. 8493–8502.
  * Dai et al. (2023) Damai Dai, Wenbin Jiang, Qingxiu Dong, Yajuan Lyu, and Zhifang Sui. 2023.  Neural knowledge bank for pretrained transformers. In _CCF International Conference on Natural Language Processing and Chinese Computing_. Springer, 772–783.
  * De-Arteaga et al. (2019) Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. 2019.  Bias in bios: A case study of semantic representation bias in a high-stakes setting. In _FAcct_.
  * De Cao et al. (2021) Nicola De Cao, Wilker Aziz, and Ivan Titov. 2021.  Editing Factual Knowledge in Language Models. In _EMNLP_.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009\.  Imagenet: A large-scale hierarchical image database. In _CVPR_. Ieee, 248–255.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ (2018).
  * Ding et al. (2022) Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, Jing Yi, Weilin Zhao, Xiaozhi Wang, Zhiyuan Liu, Hai-Tao Zheng, Jianfei Chen, Yang Liu, Jie Tang, Juanzi Li, and Maosong Sun. 2022\.  Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Models.  arXiv:2203.06904 [cs.CL]
  * Dong et al. (2019) Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019.  Unified language model pre-training for natural language understanding and generation.  _NeurIPS_ (2019).
  * Dong et al. (2022a) Qingxiu Dong, Damai Dai, Yifan Song, Jingjing Xu, Zhifang Sui, and Lei Li. 2022a.  Calibrating Factual Knowledge in Pretrained Language Models. In _EMNLP_.
  * Dong et al. (2022b) Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022b.  A survey for in-context learning.  _arXiv preprint arXiv:2301.00234_ (2022).
  * Dubois et al. (2023) Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023.  Alpacafarm: A simulation framework for methods that learn from human feedback.  _arXiv preprint arXiv:2305.14387_ (2023).
  * Elazar et al. (2021) Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. 2021\.  Measuring and improving consistency in pretrained language models.  _Transactions of the Association for Computational Linguistics_ (2021).
  * Elsahar et al. (2018) Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Frederique Laforest, and Elena Simperl. 2018\.  T-rex: A large scale alignment of natural language with knowledge base triples. In _LREC_.
  * Fan et al. (2023) Wenqi Fan, Zihuai Zhao, Jiatong Li, Yunqing Liu, Xiaowei Mei, Yiqi Wang, Jiliang Tang, and Qing Li. 2023\.  Recommender systems in the era of large language models (llms).  _arXiv preprint arXiv:2307.02046_ (2023).
  * Fei et al. (2021) Hao Fei, Yafeng Ren, Yue Zhang, Donghong Ji, and Xiaohui Liang. 2021.  Enriching contextualized language model from knowledge graph for biomedical information extraction.  _Briefings in bioinformatics_ 22, 3 (2021), bbaa110.
  * Finn et al. (2017) Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017\.  Model-agnostic meta-learning for fast adaptation of deep networks. In _ICML_. PMLR, 1126–1135.
  * Ganguli et al. (2022) Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. 2022\.  Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned.  _arXiv preprint arXiv:2209.07858_ (2022).
  * Gemmeke et al. (2017) Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. 2017\.  Audio set: An ontology and human-labeled dataset for audio events. In _ICASSP_. IEEE, 776–780.
  * Geva et al. (2022) Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg. 2022\.  Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space. In _EMNLP_.
  * Geva et al. (2021) Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021\.  Transformer Feed-Forward Layers Are Key-Value Memories. In _EMNLP_. 5484–5495.
  * Glaese et al. (2022) Amelia Glaese, Nat McAleese, Maja Trębacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. 2022\.  Improving alignment of dialogue agents via targeted human judgements.  _arXiv preprint arXiv:2209.14375_ (2022).
  * Goodfellow et al. (2020) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2020\.  Generative adversarial networks.  _Commun. ACM_ (2020).
  * Gu et al. (2018) Jiatao Gu, Yong Wang, Yun Chen, Victor OK Li, and Kyunghyun Cho. 2018.  Meta-Learning for Low-Resource Neural Machine Translation. In _EMNLP_. 3622–3631.
  * Gupta et al. (2023) Anshita Gupta, Debanjan Mondal, Akshay Krishna Sheshadri, Wenlong Zhao, Xiang Lorraine Li, Sarah Wiegreffe, and Niket Tandon. 2023\.  Editing Commonsense Knowledge in GPT.  _arXiv preprint arXiv:2305.14956_ (2023).
  * Ha et al. (2016) David Ha, Andrew Dai, and Quoc V. Le. 2016.  HyperNetworks.  arXiv:1609.09106 [cs.LG]
  * Hartvigsen et al. (2022) Thomas Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim, and Marzyeh Ghassemi. 2022\.  Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors. In _NeurIPS Workshop on Robustness in Sequence Modeling_.
  * Hase et al. (2021) Peter Hase, Mona Diab, Asli Celikyilmaz, Xian Li, Zornitsa Kozareva, Veselin Stoyanov, Mohit Bansal, and Srinivasan Iyer. 2021\.  Do language models have beliefs? methods for detecting, updating, and visualizing model beliefs.  _arXiv preprint arXiv:2111.13654_ (2021).
  * Hase et al. (2023) Peter Hase, Mona Diab, Asli Celikyilmaz, Xian Li, Zornitsa Kozareva, Veselin Stoyanov, Mohit Bansal, and Srinivasan Iyer. 2023\.  Methods for measuring, updating, and visualizing factual beliefs in language models. In _EACL_.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016\.  Deep residual learning for image recognition. In _CVPR_. 770–778.
  * Heitmann (2020) Mark Heitmann. 2020\.  More than a feeling: Benchmarks for sentiment analysis accuracy.  In _More than a Feeling: Benchmarks for Sentiment Analysis Accuracy: Heitmann, Mark_.
  * Hernandez et al. (2023) Evan Hernandez, Belinda Z Li, and Jacob Andreas. 2023.  Measuring and manipulating knowledge representations in language models.  _arXiv preprint arXiv:2304.00740_ (2023).
  * Hu et al. (2023) Linmei Hu, Zeyi Liu, Ziwang Zhao, Lei Hou, Liqiang Nie, and Juanzi Li. 2023\.  A Survey of Knowledge Enhanced Pre-Trained Language Models.  _IEEE TKDE_ (2023).
  * Huang et al. (2023) Zeyu Huang, Yikang Shen, Xiaofeng Zhang, Jie Zhou, Wenge Rong, and Zhang Xiong. 2023\.  Transformer-Patcher: One Mistake Worth One Neuron. In _ICLR_.
  * Ilharco et al. (2023) Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. 2023.  Editing models with task arithmetic. In _ICLR_.
  * Jiang et al. (2020) Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. 2020\.  How can we know what language models know?  _Transactions of the Association for Computational Linguistics_ (2020).
  * Kalyan et al. (2022) Katikapalli Subramanyam Kalyan, Ajit Rajasekharan, and Sivanesan Sangeetha. 2022.  AMMU: a survey of transformer-based biomedical pretrained language models.  _Journal of Biomedical Informatics_ 126 (2022), 103982.
  * Kasirzadeh and Gabriel (2023) Atoosa Kasirzadeh and Iason Gabriel. 2023.  In conversation with Artificial Intelligence: aligning language models with human values.  _Philosophy & Technology_ (2023).
  * Kasneci et al. (2023) Enkelejda Kasneci, Kathrin Seßler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, et al. 2023\.  ChatGPT for good? On opportunities and challenges of large language models for education.  _Learning and Individual Differences_ 103 (2023), 102274.
  * Kenton and Toutanova (2019) Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _NAACL_.
  * Kohonen (1972) Teuvo Kohonen. 1972\.  Correlation matrix memories.  _IEEE Trans. Comput._ 100, 4 (1972), 353–359.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019\.  Natural questions: a benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ (2019).
  * Lambert et al. (2022) Nathan Lambert, Louis Castricato, Leandro von Werra, and Alex Havrilla. 2022.  Illustrating Reinforcement Learning from Human Feedback (RLHF).  _Hugging Face Blog_ (2022).  https://huggingface.co/blog/rlhf.
  * Le et al. (2013) Quoc Le, Tamás Sarlós, Alex Smola, et al. 2013\.  Fastfood-approximating kernel expansions in loglinear time. In _ICML_ , Vol. 85. 8.
  * Lee et al. (2022) Kyungjae Lee, Wookje Han, Seung won Hwang, Hwaran Lee, Joonsuk Park, and Sang-Woo Lee. 2022\.  Plug-and-Play Adaptation for Continuously-updated QA.  arXiv:2204.12785 [cs.CL]
  * Levy et al. (2017) Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. 2017\.  Zero-Shot Relation Extraction via Reading Comprehension. In _CoNLL 2017_.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. In _ACL_.
  * Li et al. (2018) Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. 2018.  Measuring the Intrinsic Dimension of Objective Landscapes.  arXiv:1804.08838 [cs.LG]
  * Li et al. (2023b) Daliang Li, Ankit Singh Rawat, Manzil Zaheer, Xin Wang, Michal Lukasik, Andreas Veit, Felix X. Yu, and Sanjiv Kumar. 2023b.  Large Language Models with Controllable Working Memory. In _ACL_.
  * Li et al. (2023a) Xiaopeng Li, Shasha Li, Shezheng Song, Jing Yang, Jun Ma, and Jie Yu. 2023a.  PMET: Precise Model Editing in a Transformer.  _arXiv preprint arXiv:2308.08742_ (2023).
  * Liu et al. (2023) Hao Liu, Carmelo Sferrazza, and Pieter Abbeel. 2023\.  Chain of hindsight aligns language models with feedback.  _arXiv preprint arXiv:2302.02676_ 3 (2023).
  * Liu et al. (2022) Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. 2022.  Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. In _NeurIPS_.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ (2019).
  * Ma (2021) Yuxuan Ma. 2021\.  distilgpt2-finetuned-wikitext2.  https://huggingface.co/MYX4567/distilgpt2-finetuned-wikitext2
  * Madaan et al. (2022) Aman Madaan, Niket Tandon, Peter Clark, and Yiming Yang. 2022\.  Memory-assisted prompt editing to improve GPT-3 after deployment.  _CoRR_ (2022).
  * Meng et al. (2022) Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022\.  Locating and editing factual associations in GPT. In _NeurIPS_ , Vol. 35. 17359–17372.
  * Meng et al. (2023) Kevin Meng, Arnab Sen Sharma, Alex J Andonian, Yonatan Belinkov, and David Bau. 2023\.  Mass-Editing Memory in a Transformer. In _The Eleventh International Conference on Learning Representations_.
  * Menick et al. (2022) Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young, Lucy Campbell-Gillingham, Geoffrey Irving, et al. 2022\.  Teaching language models to support answers with verified quotes.  _arXiv preprint arXiv:2203.11147_ (2022).
  * Min et al. (2023) Bonan Min, Hayley Ross, Elior Sulem, Amir Pouran Ben Veyseh, Thien Huu Nguyen, Oscar Sainz, Eneko Agirre, Ilana Heintz, and Dan Roth. 2023.  Recent advances in natural language processing via large pre-trained language models: A survey.  _Comput. Surveys_ 56, 2 (2023), 1–40.
  * Mitchell et al. (2022a) Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. 2022a.  Fast Model Editing at Scale. In _ICLR_.
  * Mitchell et al. (2022b) Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D. Manning, and Chelsea Finn. 2022b.  Memory-Based Model Editing at Scale. In _ICML_.
  * Muennighoff et al. (2022) Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al. 2022\.  Crosslingual generalization through multitask finetuning.  _arXiv preprint arXiv:2211.01786_ (2022).
  * Murty et al. (2022) Shikhar Murty, Christopher D. Manning, Scott M. Lundberg, and Marco Túlio Ribeiro. 2022\.  Fixing Model Bugs with Natural Language Patches. In _EMNLP_.
  * OpenAI (2023) OpenAI. 2023.  GPT-4 Technical Report.  arXiv:2303.08774 [cs.CL]
  * Pandya and Bhatt (2021) Hariom A. Pandya and Brijesh S. Bhatt. 2021.  Question Answering Survey: Directions, Challenges, Datasets, Evaluation Matrices.  arXiv:2112.03572 [cs.CL]
  * Peng et al. (2023a) Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, et al. 2023a.  Check your facts and try again: Improving large language models with external knowledge and automated feedback.  _arXiv preprint arXiv:2302.12813_ (2023).
  * Peng et al. (2023b) Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. 2023b.  Instruction tuning with GPT-4.  _arXiv preprint arXiv:2304.03277_ (2023).
  * Perez et al. (2022) Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. 2022.  Red Teaming Language Models with Language Models. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_. 3419–3448.