# Rethinking Machine Unlearning for
Large Language Models

Sijia Liu1,2 Yuanshun Yao3 Jinghan Jia1 Stephen Casper4 Nathalie Baracaldo2
Peter Hase5
Xiaojun Xu3 Yuguang Yao1 Hang Li3 Kush R. Varshney2 Mohit Bansal5 Sanmi
Koyejo6 Yang Liu3,7
1Computer Science & Engineering Dept., Michigan State University, USA
2IBM Research, USA
3ByteDance Research, USA
4Computer Science and Artificial Intelligence Laboratory, MIT, USA
5Computer Science Dept., University of North Carolina at Chapel Hill, USA
6Computer Science Dept., Stanford University, USA
7Computer Science & Engineering Dept., University of California, Santa Cruz,
USA Correspondence to: Sijia Liu (liusiji5@msu.edu), Yang Liu
(yangliu@ucsc.edu).

###### Abstract

We explore machine unlearning (MU) in the domain of large language models
(LLMs), referred to as LLM unlearning. This initiative aims to eliminate
undesirable data influence (e.g., sensitive or illegal information) and the
associated model capabilities, while maintaining the integrity of essential
knowledge generation and not affecting causally unrelated information. We
envision LLM unlearning becoming a pivotal element in the life-cycle
management of LLMs, potentially standing as an essential foundation for
developing generative AI that is not only safe, secure, and trustworthy, but
also resource-efficient without the need of full retraining. We navigate the
unlearning landscape in LLMs from conceptual formulation, methodologies,
metrics, and applications. In particular, we highlight the often-overlooked
aspects of existing LLM unlearning research, e.g., unlearning scope, data-
model interaction, and multifaceted efficacy assessment. We also draw
connections between LLM unlearning and related areas such as model editing,
influence functions, model explanation, adversarial training, and
reinforcement learning. Furthermore, we outline an effective assessment
framework for LLM unlearning and explore its applications in copyright and
privacy safeguards and sociotechnical harm reduction.

##  1 Introduction

Large language models (LLMs) have shown exceptional proficiency in generating
text that closely resembles human-authored content. However, their ability to
memorize extensive corpora may also lead to ethical and security concerns.
These include societal biases and stereotyping (Bender et al., 2021; Motoki et
al., 2023; Kotek et al., 2023), the generation of sensitive, private, harmful,
or illegal content (Nasr et al., 2023; Wen et al., 2023; Karamolegkou et al.,
2023; Patil et al., 2024), ease of jailbreaking (Wei et al., 2023; Zou et al.,
2023; Liu et al., 2023a), and possible malicious use in developing
cyberattacks or bioweapons (Barrett et al., 2023; Hendrycks et al., 2023).
These concerns emphasize the need to adeptly and efficiently tailor pre-
trained LLMs to suit diverse safety contexts while meeting specific
requirements of users and sectors.

With the costly and prolonged training periods of LLMs, retraining these
models to eliminate undesirable data effects is often impractical. Machine
unlearning (MU) has emerged as an alternative to remove the influence of
undesirable data and associated model capabilities from the pre-trained models
(Cao & Yang, 2015; Bourtoule et al., 2021; Nguyen et al., 2022; Si et al.,
2023; Zhang et al., 2023a; Eldan & Russinovich, 2023; Yao et al., 2023). For
example, MU is used as a strategy to prevent the generation of copyrighted
material from the Harry Potter series in (Eldan & Russinovich, 2023). In the
context of classification tasks, MU has been extensively studied by Ginart et
al. (2019); Neel et al. (2021); Ullah et al. (2021); Sekhari et al. (2021);
Golatkar et al. (2020); Jia et al. (2023). Yet, its application and
understanding in LLMs remains limited, where models are typically used for
generative tasks such as summarization, sentence completion, paraphrasing, and
question answering. Therefore, this paper specifically concentrates on
exploring the MU problems in LLMs, termed ‘LLM Unlearning’.

As data-model scales continue to grow, the emergence of LLM unlearning
introduces new challenges and complexities, as will be elaborated on in Sec.
2. For example, current research efforts (Lu et al., 2022; Jang et al., 2022;
Ilharco et al., 2022; Eldan & Russinovich, 2023; Wu et al., 2023b; Yu et al.,
2023; Zhang et al., 2023c; Yao et al., 2023) suffer from a lack of
standardized corpora and evaluation methods for LLM unlearning, often facing
issues related to unlearning precision and efficiency. We also refer readers
to Sec. 3 for a summary of existing LLM unlearning tasks.

Figure 1: Demonstration of how MU can be incorporated into LLM development
cycle. The landscape of LLM unlearning will be mainly navigated from
applications (‘why’), methods (‘where’ and ‘how’), and evaluations.

Although preliminary surveys of LLM unlearning have been provided by Si et al.
(2023) and Zhang et al. (2023a), this paper is, to the best of our knowledge,
the first to offer a thorough and in-depth analysis of LLM unlearning. Our
contributions are summarized below. Fig. 1 provides an overview of the LLM
unlearning landscape that we explore.

(1) Surveying: We conduct an in-depth review of the foundational concepts and
principles of LLM unlearning, delving into the problem formulation, categories
of unlearning methods, evaluation approaches, and practical applications.

(2) Uncovering: We bring to light previously overlooked dimensions of LLM
unlearning, e.g., emphasizing the significance of precisely defining the
unlearning scope, elucidating the interplay between data and model
interactions, and exploring the adversarial assessment of unlearning efficacy.

(3) Connecting: We establish connections between LLM unlearning and other
relevant problems and domains, providing a comparative analysis with related
topics such as model editing, influence function, and adversarial learning.

(4) Forecasting: We offer insights into the future of LLM unlearning by
identifying novel opportunities & challenges.

This paper is positioned to reassess the challenge of LLM unlearning, refining
its scope across various dimensions: conceptual formulation (Sec. 3), methods
(Sec. 4), assessment (Sec. 5), and applications (Sec. 6); see the schematic
overview in Fig. 1. We delve into each dimension through surveying,
uncovering, connecting, and forecasting. We conclude that unlearning will be a
valuable tool for making LLMs more trustworthy, but making more progress on
this will require updating the unlearning paradigm. We aspire for this work to
pave the way for developing LLM unlearning, illuminating its opportunities,
challenges, and untapped potential.

##  2 Preliminaries and Related Work

LLM unlearning has garnered attention for addressing trustworthiness concerns
such as toxicity (Lu et al., 2022), copyright and privacy (Jang et al., 2022;
Eldan & Russinovich, 2023; Wu et al., 2023b), fairness (Yu et al., 2023),
model editing (Ilharco et al., 2022; Zhang et al., 2023c), hallucination (Yao
et al., 2023), and sensitive knowledge (Barrett et al., 2023; Hendrycks et
al., 2023). In this section, we present a succinct overview of MU, tracing its
journey from traditional ML models to the emerging challenges in LLMs.

### MU for non-LLMs.

The study of MU can be traced back to non-LLMs in response to data protection
regulations such as ‘the right to be forgotten’ (Cao & Yang, 2015; Hoofnagle
et al., 2019; Bourtoule et al., 2021; Nguyen et al., 2022). Due to its
capability of assessing data influence on model performance, the landscape of
MU has expanded to encompass diverse domains, such as image classification
(Ginart et al., 2019; Golatkar et al., 2020; Neel et al., 2021; Ullah et al.,
2021; Sekhari et al., 2021), text-to-image generation (Gandikota et al., 2023;
Zhang et al., 2023b; Kumari et al., 2023; Fan et al., 2024), federated
learning (Liu et al., 2020; Wang et al., 2022; Che et al., 2023; Liu et al.,
2023b; Halimi et al., 2022), and graph neural networks (Chen et al., 2022;
Chien et al., 2022; Wu et al., 2023a).

In the literature, ‘exact’ unlearning, which involves retraining the model
from scratch after removing specific training data points, is often considered
the gold standard. However, this approach comes with significant computational
demands and requires access to the entire training set (Thudi et al., 2022).
To address these challenges, many research efforts have shifted towards the
development of scalable and effective approximate unlearning methods (Golatkar
et al., 2020; Warnecke et al., 2021; Becker & Liebig, 2022; Thudi et al.,
2022; Jia et al., 2023; Chen et al., 2023a). In addition, probabilistic
methods with certain provable removal guarantees have been explored, often
leveraging the concept of differential privacy (Ginart et al., 2019; Guo et
al., 2019; Neel et al., 2021; Ullah et al., 2021; Sekhari et al., 2021).

### Challenges of MU for LLMs.

LLM unlearning introduces new challenges and complexities.

First, LLMs are trained on massive amounts of data, which can unintentionally
introduce biases and the memorization of personal and confidential
information. Accordingly, it becomes challenging to precisely define and
localize the ‘unlearning targets’, such as the subset of the training set or a
knowledge concept that needs to be removed. Therefore, current studies on LLM
unlearning (Lu et al., 2022; Jang et al., 2022; Ilharco et al., 2022; Eldan &
Russinovich, 2023; Wu et al., 2023b; Yu et al., 2023; Zhang et al., 2023c; Yao
et al., 2023) are typically context and task-dependent. There is a lack of
standardized corpora for LLM unlearning.

Second, the growing size of LLMs and the rise of black-box access to LLM-as-a-
service present challenges in developing scalable and adaptable MU techniques
to LLMs (Bucknall & Trager, 2023; Casper et al., 2024). This also affects
performance evaluation, given the absence of retraining as a benchmark. To
address these challenges, previous studies have proposed approaches like in-
context unlearning (Pawelczyk et al., 2023) and fictitious unlearning (Maini
et al., 2024), where the former enables unlearning on black-box models, and
the latter provides alternatives to retraining.

Third, the scope of unlearning is often underspecified for LLMs. This issue is
similar to challenges faced in model editing (Mitchell et al., 2022). For
instance, effective unlearning should ensure that LLMs delete knowledge of the
targeted data within the predefined scope while simultaneously maintaining its
utility for data outside of this scope.

Fourth, despite the potential of LLM unlearning in diverse applications, there
is a notable absence of comprehensive and reliable evaluation. For example,
recent studies (Shi et al., 2023; Patil et al., 2024) have demonstrated that
sensitive information can be reverse-engineered from an edited model, even if
efforts were made to delete this information. This highlights the need for
thorough and adversarial evaluations and the design of more mechanistic
methods to guarantee the authenticity of unlearning.

##  3 Unpacking LLM Unlearning

In this section, we formalize the LLM unlearning problem and delve into its
configuration. We will outline the inherent facts and complexities in its
formulation.

### Problem statement.

In light of the literature on unlearning (Bourtoule et al., 2021; Jia et al.,
2023; Kurmanji et al., 2023), and its progression in LLMs (Pawelczyk et al.,
2023; Yao et al., 2023; Ishibashi & Shimodaira, 2023; Maini et al., 2024), we
define the problem of LLM unlearning below.

(LLM unlearning) How can we efficiently and effectively eliminate the
influence of specific ‘unlearning targets’ and remove associated model
capabilities while preserving model performance for non-targets?

We dissect the above statement from the perspectives: (1) unlearning targets,
(2) influence erasure, (3) unlearning effectiveness, and (4) efficiency. See
Table 1 for a summary of existing LLM unlearning studies based on (1)-(4).

Table 1: A summary of existing LLM unlearning problems through unlearning
targets, influence erasure, effectiveness, and efficiency. An asterisk (∗)
indicates the incapability of evaluating unlearning for LLMs due to the
impracticality of retraining these models.

|  Related work
---
|  Unlearning targets/tasks
---
|  Influence erasure methods
---
|  Effectiveness:
---
(I) In-scope evaluation for unlearning efficacy
(O) Out-of-scope evaluation for model utility
| Efficiency
---
(Lu et al., 2022) |  | Reducing toxic content,
---
avoiding undesirable sentiments,
and preventing repeated text generation
| Reward-reinforced model fine-tuning
---
|  (I) Toxic prompts, specific sentiments,
---
& repetitive sentences
(O) Unlearning target-irrelevant prompts
| N/A
---
(Jang et al., 2022) |  | Degenerating private information,
---
w/ unlearning response irrelevant to this info
| Gradient ascent-based fine-tuning
---
|  (I) Prompts from training data extraction
---
(O) Natural language understanding tasks
| Runtime cost
---
(Kumar et al., 2022) |  | Text de-classification, w/ unlearning
---
response close to that of retraining⋆
| Sharded, isolated, sliced, and
---
aggregated (SISA) training via adapter
|  (I) No evaluation for unlearning efficacy
---
(O) Test set
| Runtime cost
---
Memory cost
| (Ilharco et al., 2022)
---
(Zhang et al., 2023c)
| Degenerating toxic content
---
| Task vector-based parameter-efficient
---
fine-tuning via LoRA
|  (I) Prompts leading to toxic generation
---
(O) Perplexity on other datasets
| N/A
---
(Wang et al., 2023) |  | Text de-classification/de-generation,
---
unlearning specific words in translation,
w/ response close to that of retraining∗
| KL-divergence-based fine-tuning
---
|  (I) Training subset
---
(O) Test set
| Runtime cost
---
(Yu et al., 2023) |  | Unlearning gender and profession bias,
---
with de-biased unlearning response
| Weight importance-informed
---
& relabeling-based fine-tuning
|  (I) Biased prompts
---
(O) No evaluation for model utility
| N/A
---
(Pawelczyk et al., 2023) |  | Text de-classification, w/ unlearning
---
response close to that of retraining∗
| In-context learning
---
|  (I) Training subset
---
(O) Retain & test sets
| Black-box access
---
(Eldan & Russinovich, 2023) |  | Degenerating Harry Potter-related
---
book content, w/ unlearning response
irrelevant to Harry Potter
| Relabeling-based fine-tuning
---
|  (I) Questions and their rephrased/hard versions
---
about Harry Potter
(O) NLU tasks
| N/A
---
(Ishibashi & Shimodaira, 2023) |  | Unlearning knowledge from QA dataset,
---
with refusal response (e.g., ‘I don’t know’)
| Relabeling-based fine-tuning
---
|  (I) Adversarial and original questions
---
about forgotten knowledge
(O) Other QA prompts
| N/A
---
(Chen & Yang, 2023) |  | Text de-classification and de-generation,
---
with response close to that of retraining∗
| KL divergence-based parameter-
---
efficient fine-tuning via adapter
|  (I) Training subset
---
(O) Retain & test sets
| Runtime cost
---
(Wu et al., 2023b) |  | Degenerating private information,
---
w/ unlearning response irrelevant to this info
| Importance-based neuron editing
---
|  (I) Memorized private data points
---
(O) Test set
| Runtime cost
---
(Yao et al., 2023) |  | Degenerating harmful prompts,
---
degenerating Harry Potter-related
book content, and reducing hallucination
| Integration of gradient ascent,
---
random labeling,
& KL divergence-based fine-tuning
|  (I) Prompts related to unlearning targets
---
(O) NLU tasks
| Runtime cost
---
(Maini et al., 2024) |  | Unlearning biographical knowledge
---
about fictitious authors.
| Fine-tuning with various objectives
---
|  (I) Q&A about the unlearning authors
---
(O) Q&A about other authors and general facts
| Runtime cost
---
(Patil et al., 2024) |  | Degenerating sensitive information
---
using factual information as a testbed
| Model editing techniques and
---
constrained finetuning
|  (I) Prompts for unlearned factual knowledge
---
(O) Prompts for unrelated factual knowledge
| White-box v.
---
black-box access

$\bullet$ Unlearning targets: Unlearning tasks may take on various forms and
are closely related to the unlearning objectives. For instance, one could
focus on data influence removal, while the other could emphasize model
capability removal. Although these two aspects are intertwined, the former is
often crucial for intellectual property (IP) protection, while the latter is
more practical for AI alignment and safety. The literature identifies
unlearning targets as specific data points, which could involve content
containing harmful, unethical, or illegal language (Jang et al., 2022; Wu et
al., 2023b). They have also been represented by higher-level unlearned
knowledge, expressed through an unwanted text prompt or concept (Lu et al.,
2022; Yao et al., 2023; Eldan & Russinovich, 2023). For example, Eldan &
Russinovich (2023) defined the unlearning target as ‘Harry Potter’-related
content, with the objective to avoid generating such content irrespective of
where the content was learned: from the copyrighted material, blog posts, or
news articles.

$\bullet$ Influence erasure: Erasing the influence of unlearning targets and
associated model capabilities requires a joint examination of both data and
model influences rather than studies in isolation. Specifically, it is
important to scrutinize the contributions of data sources to undesired model
outputs, as well as the roles played by individual components within a model
in generating these undesirable outcomes. This dual examination allows us to
gain a more comprehensive understanding of the mechanisms driving these
outputs, thereby facilitating the development of unlearning strategies to
prevent them effectively. The objective of achieving complete influence
erasure also implies the importance of robustness and generalization in
unlearned behavior. When evaluating LLM unlearning, especially when using
approximate methods shown in Table 1, a rigorous criterion is needed. Patil et
al. (2024) underscore this viewpoint by demonstrating that sensitive
information can persist in model weights after editing or unlearning, and by
Maini et al. (2024) and Shi et al. (2023) who expose the limitations of
conventional unlearning methods.

$\bullet$ Unlearning effectiveness: The effectiveness of LLM unlearning
extends beyond merely diminishing the specific data influence or model
capability. A crucial aspect of effectiveness is the concept of unlearning
scope, as inspired by the editing scope (Mitchell et al., 2022). This has
implications for the development of metrics to evaluate the effectiveness of
unlearning and the design of effective unlearning methods. The unlearning
scope defines the success of influence erasure by its capacity to accurately
modify the model’s behavior for in-scope examples, which are either directly
or indirectly related to unlearning targets, such as paraphrased text prompts
to be degenerated (Zhong et al., 2023; Patil et al., 2024). At the same time,
it must also ensure the generation consistency for out-of-scope examples,
necessitating the preservation of model generation capabilities post-
unlearning. Differentiating between in-scope and out-of-scope examples for
unlearning is its own unsolved problem, as it requires determining when facts
logically imply one another (Hase et al., 2023b; Cohen et al., 2023).

$\bullet$ Unlearning efficiency & feasibility: Current research efforts
focused on the development of rapid unlearning methods for pre-trained LLMs
due to the significant re-training cost (Jang et al., 2022; Eldan &
Russinovich, 2023; Yao et al., 2023). However, beyond computational
efficiency, LLMs present additional efficiency challenges. These include the
complexity and, at times, the infeasibility of pinpointing and attributing
training data points designated for unlearning. Additionally, there is the
challenge of executing unlearning in the context of black-box LLMs (Pawelczyk
et al., 2023), where interactions with models are constrained to input-output
queries.

### Distinguishing LLM unlearning from conventional MU.

According to the above dimensions, LLM unlearning involves a broader range of
targets, which are often context-dependent and less clearly defined. Moreover,
the effectiveness of LLM unlearning is not limited to forgetting the influence
of specific data points but also includes defining a broader unlearning scope
for model capability removal. Furthermore, there is a critical need to devise
more mechanistic methods that guarantee effective and robust unlearning, while
also enhancing their practicality and feasibility.

### Mathematical modeling.

Building upon the high-level formulation of LLM unlearning problems presented
earlier, we next provide mathematical modeling details and discuss the
associated design choices. To facilitate comprehension, we provide a commonly-
used formulation of LLM unlearning problems below. While this may not be the
sole or optimal approach to LLM unlearning, it incorporates several key
elements that are essential to the problem setup.

$\displaystyle\begin{array}[]{l}\displaystyle\min_{\bm{\theta}}\,\,\underbrace{\mathbb{E}_{(x,y_{\mathrm{f}})\in\mathcal{D}_{\mathrm{f}}}[\ell(y_{\mathrm{f}}|x;\bm{\theta})]}_{\text{Forget}}+\lambda\underbrace{\mathbb{E}_{(x,y)\in\mathcal{D}_{\mathrm{r}}}[\ell(y|x;\bm{\theta})]}_{\text{Retain}}\end{array}$
(2)

where $\ell(y|x;\bm{\theta})$ denotes the prediction loss of using
$\bm{\theta}$ given the input $x$ with respect to the response $y$,
$\mathcal{D}_{\mathrm{f}}$ and $\mathcal{D}_{\mathrm{r}}$ refer to ‘forget’
and ‘retain’ sets which will be explained later, $y_{\mathrm{f}}$ denotes the
desired model response post-unlearning, and $\lambda\geq 0$ is a
regularization parameter to balance ‘forget’ and ‘retain’ (e.g., $\lambda=0$
if retain set is not given a priori).

In the dataset setup of LLM unlearning, we typically assume access to a forget
set ($\mathcal{D}_{\mathrm{f}}$), the influence of which should be eliminated
in LLM generation. For instance, $\mathcal{D}_{\mathrm{f}}$ might consist of a
collection of harmful or toxic prompt-response pairs designated for
degeneration (Yao et al., 2023). Moreover, if the original training set is
available, then $\mathcal{D}_{\mathrm{f}}$ can be composed of a subset of
training data points related to the unlearning target. Alternatively, it can
be generated using synthesized data points based on a higher-level unlearned
knowledge concept, or it can be derived from a set of extracted training data
points reverse-engineered from the given LLM itself. In practice, the forget
set $\mathcal{D}_{\mathrm{f}}$ is not required to belong precisely to the
LLM’s training corpus. Given the diversity and extensive size of the model’s
training data, the content we aim to unlearn is more likely to represent a
general concept. Thus, LLM unlearning needs to not only unlearn specific
training samples but also generalize to similar samples that share common
characteristics.

Besides the forget set $\mathcal{D}_{\mathrm{f}}$, sometimes there is a need
for a retain set ($\mathcal{D}_{\mathrm{r}}$), which contains samples that are
not subject to unlearning and used to preserve the utility of the unlearned
model. Through the lens of the unlearning scope we discussed earlier, the
forget set ($\mathcal{D}_{\mathrm{f}}$) provides in-scope examples earmarked
for unlearning, while the retain set ($\mathcal{D}_{\mathrm{r}}$) involves
examples out of the unlearning scope. Some recent studies have also attempted
to develop LLM unlearning approaches that operate independently of access to
forget and/or retain sets (Pawelczyk et al., 2023; Li et al., 2023b).

We next introduce the model and optimization setups for LLM unlearning.
Unlearning is performed at the post-model training phase. As shown in (2), a
common unlearning objective is to efficiently update the original pre-trained
model so that the updated model can unlearn on $\mathcal{D}_{\mathrm{f}}$
while retaining its generation capability on $\mathcal{D}_{\mathrm{r}}$. In
addition, another design element is unlearning response ($y_{\mathrm{f}}$),
referred to as the response of an unlearned model to in-scope examples. For
example, in the stateful LLM unlearning method aimed at erasing information
related to ‘Who’s Harry Potter?’ (Eldan & Russinovich, 2023), the unlearning
response is based on word replacements using generic translations, like
substituting ‘Quidditch’ with ‘Skyball’, as part of the unlearning process.
However, this type of approach may inadvertently blur the line between LLM
hallucination and legitimate responses, highlighting the need for improvements
in unlearning response design. Another choice is empty response (Wu et al.,
2023b; Patil et al., 2024), given by the reject ‘I don’t know’ (Patil et al.,
2024) or the customized response by ‘masking’ the unlearning information using
specialized tokens such as ‘$\blacksquare$’ in (Wu et al., 2023b). However, we
need to ensure that the empty response targets only examples within the
unlearning scope. Otherwise, frequent rejections may occur, potentially
diminishing the user experience with LLMs.

##  4 LLM Unlearning Methods Revisited: Prior Art and Overlooked Principles

In this section, we begin with a review of existing MU (machine unlearning)
methodologies adapted for LLM unlearning. Following that, we delve into the
design factors that are often overlooked in the literature, including the
intertwined data-model influence, the relationship with model editing, and
adversarial training.

### Review of existing unlearning principles.

Existing LLM unlearning methods can be broadly categorized into two groups:
model-based and input-based. Model-based methods involve modifying the weights
and/or architecture components of LLMs to achieve the unlearning objective
(Jang et al., 2022; Lu et al., 2022; Yao et al., 2023; Yu et al., 2023; Chen &
Yang, 2023; Zhang et al., 2023c; Hase et al., 2023a; Wu et al., 2023b;
Rafailov et al., 2023), e.g., following the mathematical formulation in Sec.
3. Input-based methods design input instructions (Madaan et al., 2022; Zheng
et al., 2023; Pawelczyk et al., 2023), such as in-context examples or prompts,
to guide the original LLM (without parameter updating) towards the unlearning
objective.

In the literature, the predominant research emphasis lies on model-based
methods as shown in Tab. 1. We review some of the most representative
approaches below.

$\bullet$ Gradient ascent and its variants (Jang et al., 2022; Yao et al.,
2023; Yu et al., 2023): Gradient ascent (GA) stands as one of the most
straightforward unlearning methods, updating the model parameters by
maximizing the likelihood of mis-prediction for the samples within the forget
set $\mathcal{D}_{\mathrm{f}}$ (Jang et al., 2022). However, it is worth
noting that GA alone can be sensitive to the choice of hyperparameters during
optimization, such as the number of ascent steps and the learning rate (Jia et
al., 2023; Fan et al., 2024). This has given rise to improved variants of GA.
For example, Yao et al. (2023) incorporate random labeling to augment the
unlearning objective and ensure utility preservation on the retain set
$\mathcal{D}_{\mathrm{r}}$.

Another variant of GA is to transform it into a gradient descent approach,
which aims to minimize the likelihood of predictions on relabeled forgetting
data (Yu et al., 2023). This gradient descent-based fine-tuning, over
relabeled or randomly labeled forgetting data, is also employed in (Eldan &
Russinovich, 2023), where generic translations are used to replace the
unlearned texts. GA and its variants often involve fine-tuning pre-trained
LLMs for unlearning purposes. To enhance efficiency, parameter-efficient fine-
tuning (PEFT) techniques could be employed. Chen & Yang (2023) fine-tune an
adapter over the unlearning objective that acts as an unlearning layer within
the LLM. Zhang et al. (2023c) use LoRA to create task vectors and accomplish
unlearning by negating tasks under these task vectors.

$\bullet$ Localization-informed unlearning (Meng et al., 2022; Yu et al.,
2023; Wu et al., 2023b): The pursuit of parameter efficiency is also in line
with the objective of identifying and localizing a subset of model units
(e.g., layers, weights, or neurons) that are essential for the unlearning
task. For example, the process of localization can be accomplished through
representation denoising, also known as causal tracing, in (Meng et al.,
2022), focusing on the unit of model layers. Previous work shows that it is
important to delete information about unlearning targets wherever it is
represented in models in order to protect against attacks (Patil et al.,
2024). In addition, gradient-based saliency (Yu et al., 2023) is employed to
identify the crucial weights that need to be fine-tuned to achieve the
unlearning objective. In (Wu et al., 2023b), neurons that respond to
unlearning targets are identified within the feed-forward network and
subsequently selected for knowledge unlearning. In the context of vision
models, unlearning can also benefit from localizing weights salient to
unlearning, as demonstrated by Jia et al. (2023) and Fan et al. (2024).
Furthermore, the concept of localization-informed unlearning resonates with
the future modular machine learning solution development (Menik & Ramaswamy,
2023). This modularity allows the emerging foundation models to be partitioned
into manageable subparts, facilitating easier maintenance and independent
updates for each component.

$\bullet$ Influence function-based methods: While the influence function (Koh
& Liang, 2017; Bae et al., 2022) is a standard approach to assess the effect
of data removal on model performance (Izzo et al., 2021; Warnecke et al.,
2021), it is not commonly employed in the context of LLM unlearning for two
main reasons: the computational complexity involved in inverting the Hessian
matrix, and the reduced accuracy resulting from the use of approximations in
influence function derivation (Jia et al., 2023). However, the potential of
influence functions in LLM unlearning may be underestimated, given that
scalability issue has been improved by Grosse et al. (2023), and approximation
errors can be mitigated by focusing on localized weights that are salient to
unlearning, as described in the previous category.

$\bullet$ Other model-based methods: Other types of LLM unlearning methods
fall outside the categories mentioned above. For example, Jang et al. (2022);
Chen & Yang (2023) show that sequential unlearning outperforms batch
unlearning However, as indicated by Gu et al. (2024), sequential editing of
LLMs may compromise their general capabilities. Therefore, the study of
sequential unlearning in LLMs continues to be an unresolved issue.

$\bullet$ Input-based vs. model-based: Input-based strategies (Madaan et al.,
2022; Zheng et al., 2023; Pawelczyk et al., 2023) have also shown promise in
addressing the challenges posed by the restricted access to black-box LLMs and
achieving parameter efficiency of LLM unlearning. Here the learnable
parameters are given by input prompts rather than model weights/architecture
components. However, we posit that input-based methods may not necessarily
yield genuinely unlearned models, leading to weaker unlearning strategies
compared to model-based methods because modifying the inputs of LLMs alone may
not be sufficient to completely erase the influence of unlearning targets
(Toyer et al., 2023). This assertion is also supported by the existence of
hard or even adversarial in-scope examples associated with unlearning targets
and the challenge to remove their influence in LLMs (Zhong et al., 2023; Patil
et al., 2024).

### Exploring data-model interactions.

A key objective of unlearning is to eliminate the influence of the forgotten
data points/knowledge on the model’s performance. However, this process is not
studied in isolation: It is closely connected to exploring the influence of
model weights or architecture components. Unlearning requires a sense of
locality, which involves addressing the specific unlearning target and its
associated unlearning scope. Consequently, exploring model influence helps
identify the specific, localized areas of the model that are relevant to this
locality. This is further reinforced by the surveyed weight localization
techniques (Meng et al., 2022; Yu et al., 2023; Wu et al., 2023b; Patil et
al., 2024). Thus, model influence and data influence are intertwined in LLM
unlearning, and a comprehensive understanding of the former can streamline the
process of handling data influence.

### Relationship with model editing.

Model editing, closely related to LLM unlearning, focuses on the local
alteration of pre-trained models’ behavior to introduce new knowledge or
rectify undesirable behaviors. First, the objective of editing could align
with that of unlearning when editing is introduced to erase information.
Second, like unlearning scope, editing scope (Mitchell et al., 2022; Hase et
al., 2023b; Cohen et al., 2023) is crucial to ensure that editing is executed
without compromising the generative capabilities of the model outside the
defined scope. Third, both model editing and unlearning can be approached
using the ‘locate first, then edit/unlearn’ principle. Localization in the
context of model editing has also been applied to various elements, including
neurons (Dai et al., 2021), network layers (Meng et al., 2022; Gupta et al.,
2023), and feed-forward components of LLMs (Geva et al., 2020; Li et al.,
2023c).

Despite the aforementioned connections, there are clear distinctions between
LLM unlearning and editing. First, the unlearning response is sometimes
unknown compared to the editing response. The specificity of an incorrect or
improper unlearning response could be seen as a form of LLM hallucination
after unlearning. Second, although unlearning and model editing may share some
common algorithmic foundations, the former does not create new answer
mappings. Rather, its central aim is the comprehensive elimination of the
influence attributed to a specific knowledge or concept within a pre-trained
LLM. Third, we can differentiate model editing from unlearning from the
perspective of ‘working memory’. It is known in (Li et al., 2022) that working
memory in LLMs is maintained through neuron activations rather than weight-
based long-term memory. Thus, existing memory-based model editing techniques
(Li et al., 2022; Mitchell et al., 2022; Madaan et al., 2022; Zheng et al.,
2023) focus on updating short-term working memory instead of altering the
long-term memory encapsulated in the model’s weights. Yet, we posit that
unlearning requires more mechanistic approaches that facilitate ‘deep’
modifications to pre-trained LLMs.

### Adversarial training for robust unlearning.

An increasing body of research highlights the weaknesses of existing
unlearning methods (Shi et al., 2023; Maini et al., 2024), particularly in
their vulnerability to test-time adversaries attempting to jailbreak unlearned
models. This issue has been explored by Patil et al. (2024) for LLMs and Zhang
et al. (2023d) for diffusion models, and inspires us to integrate adversarial
training (Madry et al., 2017) into the unlearning process, resulting in what
we term adversarial unlearning. However, this approach has received relatively
little attention thus far. While adversarial unlearning increases training
costs, it also presents new opportunities. For instance, localization-informed
unlearning can significantly reduce the computation expenses associated with
adversarial unlearning by focusing on a small portion of model units for
updating. Furthermore, advanced adversarial training techniques such as fast
adversarial training (Shafahi et al., 2019; Wong et al., 2020; Zhang et al.,
2022) and generalized adversarial training (Zhu et al., 2019; Kumari et al.,
2019; Zhang et al., 2022; Robey et al., 2023) have the potential to enhance
the scalability of adversarial unlearning while preserving its effectiveness.

### Reinforcement learning.

The mainstream technique for aligning LLMs with human values is RLHF
(reinforcement learning from human feedback) and its variants (Christiano et
al., 2017; Ouyang et al., 2022; Bai et al., 2022; Yuan et al., 2023; Lee et
al., 2023; Rafailov et al., 2023; Casper et al., 2023). However, RLHF is
resource-intense: (1) it requires human inputs that are expensive to collect,
and (2) it is computationally costly (i.e., the standard three-stage aligning
procedure). LLM unlearning arises as an alternative aligning method, where
collecting negative (i.e., low-quality and harmful) samples is much easier
through user reporting or (internal) red teaming than positive (i.e., high-
quality and helpful) samples which often require hiring humans. Furthermore,
we can envision a reinforced unlearning paradigm with a properly defined
reward function for the unlearned tasks. In a similar vein, Rafailov et al.
(2023); Lu et al. (2022) utilize a reward model to assist in unlearning.

##  5 Assessment

In this section, we discuss key considerations for designing an effective
evaluation pipeline for LLM unlearning.

### Tasks and benchmarks.

Datasets related to harmful content degeneration, personal identification
information removal, and copyrighted information prevention serve as suitable
benchmarks for evaluating the effectiveness of LLM unlearning. Some notable
examples of these datasets include: The Enron dataset, which comprises
employee emails publicly disclosed during Enron’s legal investigation by the
Federal Energy Regulatory Commission (Wu et al., 2023b), the Training Data
Extraction Challenge dataset used in (Jang et al., 2022), the Harry Potter
book series dataset (Eldan & Russinovich, 2023; Shi et al., 2023), the
toxicity generation dataset (Lu et al., 2022; Gehman et al., 2020), and the
TOFU dataset for fictitious unlearning (Maini et al., 2024).

### Evaluation of unlearning effectiveness.

We examine efficacy from three perspectives: comparison with retraining (i.e.,
the gold standard of unlearning), ‘hard’ in-scope evaluation or robustness,
and training data detection.

$\bullet$ LLM unlearning vs. retraining. In classic unlearning paradigms
(Golatkar et al., 2020; Thudi et al., 2022; Jia et al., 2023; Fan et al.,
2024), retraining a model from scratch after removing the forgotten data from
the original training set is regarded as exact unlearning. However, the
scalability challenges of retraining LLMs make it difficult to establish a
performance upper bound for evaluating LLM unlearning. A recent solution by
Maini et al. (2024) is to incorporate fictitious data (synthetic author
profiles) into the model training paradigm. Since the injected set never
appeared in the original pretraining set, LLM fine-tuning can simulate the
retraining process over the newly-introduced set. Despite the progress with
regard to the specialized TOFU dataset (Maini et al., 2024), there is still a
pressing need to advance current evaluation metrics and pipelines to
accurately assess the gap between (approximate) LLM unlearning methods and
exact unlearning.

$\bullet$ ‘Hard’ in-scope evaluation or robustness. As demonstrated in Sec. 3,
unlearning is generally context and task-dependent, contingent upon an
unlearning scope. Another effectiveness metric of LLM unlearning is to ensure
forgetting concerning in-scope unlearned examples, even for those ‘hard’ ones
that fall within the unlearning scope but may not be directly associated with
the unlearning targets. The assessment of ‘hard’ in-scope examples can be
achieved by techniques such as paraphrasing what LLMs intend to forget or
creating multi-hop questions (Zhong et al., 2023). Evaluating ‘hard’ in-scope
examples aligns seamlessly with the underlying principles of ‘worst-case’ or
‘adversarial’ evaluation methods for unlearning, resembling LLM jailbreaking
attacks (Zhang et al., 2023d; Yong et al., 2023; Patil et al., 2024). For
instance, it is shown by Yong et al. (2023) that unlearning a scope using an
English-only example would not guarantee a similar unlearned outcome when
translated into other languages. It is also crucial to evaluate the robustness
of unlearned LLMs after fine-tuning. Recent studies have revealed that fine-
tuning LLMs can sometimes lead to the re-emergence of behaviors that were not
anticipated (Yang et al., 2023; Qi et al., 2023; Lermen et al., 2023; Yong et
al., 2023).

$\bullet$ Training data detection or membership inference. Membership
inference attacks (MIA) (Shokri et al., 2017), designed to detect if a data
point is part of a victim model’s training set, serve as a crucial data
privacy-unveiled metric for evaluating machine unlearning methods (Thudi et
al., 2022; Jia et al., 2023). This metric gains even more significance in the
context of LLM unlearning, particularly when retraining is not an option. This
concept is also connected to training data memorization (Carlini et al.,
2022), as well as training data extraction attacks (Nasr et al., 2023) in
LLMs. In the realm of LLM unlearning, privacy-related evaluation metrics have
been explored and considered in various studies (Shi et al., 2023; Wu et al.,
2023b; Jang et al., 2022; Pawelczyk et al., 2023; Maini et al., 2024).

### Utility preservation.

Another crucial metric is to ensure the retained generation capabilities of
unlearned LLMs on standard language modeling tasks that fall outside the
unlearning scope. For example, evaluation on natural language understanding
tasks (Jang et al., 2022; Eldan & Russinovich, 2023; Yao et al., 2023) and
perplexity (Ilharco et al., 2022; Zhang et al., 2023c) has been considered in
the literature. In line with evaluating the effectiveness of LLM unlearning on
‘hard’ in-scope examples, it is equally crucial to assess utility preservation
using ‘hard’ out-of-scope examples, achieved by e.g., using data
transformations. Lastly, we note that it can be difficult to determine the
exact scope for some unlearning target (Hase et al., 2023b; Cohen et al.,
2023), so part of the challenge here is deciding which generation capabilities
should be retained in the first place.

### Efficiency.

Computation cost has been a predominant efficiency metric when evaluating LLM
unlearning methods, as demonstrated in Table 1. In addition to that, efforts
have been made to extend LLM unlearning to black-box models, without access to
model parameters, as demonstrated by Pawelczyk et al. (2023). Furthermore,
memory efficiency could also serve as a crucial efficiency metric. The
distinction from parameter efficiency is that current parameter-efficient
fine-tuning methods still impose substantial memory costs for storing LLMs and
for executing back-propagation (Malladi et al., 2023). Thus, a future research
direction is to explore memory-efficient fine-tuning methods for LLM
unlearning.

##  6 Applications of LLM Unlearning

In this section, we discuss two application categories facilitated by LLM
unlearning: the first focused on data influence and the second on model
capabilities.

### Copyright and privacy protection.

One application of unlearning involves legal and ethical considerations around
the fair use of training data. Algorithmic disgorgement is the term applied in
law and policy for the requirement put on a company by a regulator, such as
the Federal Trade Commission (FTC) in the United States, to completely destroy
a model that was trained on data without legal consent (Li, 2022; Goland,
2023; Belkadi & Jasserand, 2023; Achille et al., 2023). The most famous case
to-date is the FTC calling for the destruction of a weight loss application by
WW International, whose underlying model contained illegal health information
from children. Unlearning presents a viable alternative to complete
disgorgement by removing the effect of the illegal data.

Also, the tension between data owners (e.g., authors) and LLM service
providers is escalating, leading to legislation such as legal disputes
involving OpenAI, Meta, and New York Times (Small, 2023; Grynbaum & Mac,
2023). This trend is likely to persist due to increasing societal concerns
about AI data usage. The need for copyright-protected content removal aligns
with the capabilities of LLM unlearning. However, it is often challenging to
pinpoint the exact sources of training data that need to be deleted, giving
rise to the issue of data attribution (Li et al., 2023a). For example, the
leakage related to the ‘Harry Potter’ series (Eldan & Russinovich, 2023) can
have multiple possible causes, e.g., the books were used in the LLM’s training
data, the training data containing online discussions related to the series,
or the LLM using retrieval-augmented generation (Gao et al., 2023) which might
lead to leakage from the search results.

Similar to deleting copyrighted information from the training data, another
scenario is preventing LLMs from leaking user privacy, especially personal
identification information (PII). This concern is closely related to LLM
memorization and training data extraction (Carlini et al., 2019, 2021, 2022;
Jang et al., 2022; Nasr et al., 2023).

### Sociotechnical harm reduction.

Another application of LLM unlearning is alignment (Ouyang et al., 2022),
aimed at aligning LLMs with human instructions and making sure generated text
conforms to human values. Unlearning can be used to forget harmful behaviors
such as the production of toxic, discriminatory, illegal, or morally
undesirable outputs (Shevlane et al., 2023), e.g., instructions to build CBRN
(chemical, biological, radiological, and nuclear) weapons. Unlearning, as a
safety alignment tool, can happen at the different stages of LLM development,
e.g., before, during, or after alignment. Current research has focused on the
‘pre-alignment’ stage (Yao et al., 2023), there may be untapped opportunities
in the others.

Hallucinations, which involve the generation of false or inaccurate content
that may appear plausible, are a significant challenge in LLMs. Previous
research has demonstrated that unlearning can reduce LLM hallucinations by
targeting and unlearning factually incorrect responses given specific
questions (Yao et al., 2023). Since hallucination is likely to be caused by
multiple sources, the possible usage is to unlearn factually incorrect data
that serve as the source of commonly shared hallucinations or misconceptions.

LLMs are known to generate biased decisions and outputs (Perez et al., 2022;
Tamkin et al., 2023; Cui et al., 2023). In the vision domain, unlearning has
proven to be an effective tool for reducing discrimination to enable fair
decision-making (Sattigeri et al., 2022; Chen et al., 2023b). In the language
domain, unlearning has been applied to mitigate gender-profession bias (Yu et
al., 2023) and many other fairness issues (Sattigeri et al., 2022; Oesterling
et al., 2023; Kadhe et al., 2023). However, more opportunities exist, such as
unlearning stereotypes in training data.

LLMs are also known to be vulnerable to jailbreaking attacks (Wei et al.,
2023; Qi et al., 2023; Huang et al., 2023) (i.e., adversarially crafted
prompts that lead the LLM to generate undesired outputs) as well as
poisoning/backdoor attacks (Rando & Tramèr, 2023; Carlini et al., 2023).
Unlearning can be a natural solution for both types of attacks given the
existing success of unlearning as a defense against adversarial attacks in
other domains (Wang et al., 2019; Li et al., 2021; Liu et al., 2022; Jia et
al., 2023).

##  7 Discussion and Conclusion

This position paper rethinks the paradigm of unlearning for modern LLMs to
uncover its under-explored aspects. To achieve this, we dissect LLM unlearning
into four essential aspects: formulation, methodologies, evaluation metrics,
and applications. We show that there are considerable challenges in both
foundational research and practical, use case-driven research. These include:
(Generality) A desired solution for LLM unlearning should take into account
the generality of the unlearning target and dataset choice, accommodate
various model setups including both white-box and black-box scenarios, and
consider the specifics of the unlearning method. (Authenticity) LLM unlearning
should focus on effectively removing both data influence and specific model
capabilities, in order to authenticate unlearning across a range of evaluation
methods, particularly in adversarial contexts. (Precision) LLM unlearning
should precisely define the scope of unlearning, while ensuring the
preservation of general language modeling performance outside this unlearning
scope.

By examining the current state of the art, we gain insights for the future
development of LLM unlearning. For example, localization-informed unlearning
shows promise with possible dual advantages of efficiency and efficacy.
Effective unlearning requires careful consideration of data-model influences
and adversaries. Despite similarities between LLM unlearning and model
editing, they differ in their formulation and methodological design.
Furthermore, insights gained from the study of LLM unlearning could catalyze
technological advancements in other types of foundation models, e.g., large
vision-language models.

##  8 Broader Impacts

The broader impact of our work, particularly its ethical implications and
societal consequences, revolves around the responsible use of data and
maintaining the integrity of large language models (LLMs) and other
foundational AI models. Our research aims to explore machine unlearning
methods to enhance safety and security, uphold user privacy, reduce biases,
and guarantee the generation of trustworthy information by AI systems. The
methodologies and frameworks we explored have the potential to significantly
influence ethical AI practices in the future. We recognize, however, that the
societal implications of progress in machine unlearning are intricate and
diverse. This necessitates ongoing dialogue and scrutiny, especially in
relation to the dynamic field of AI ethics and governance. As machine
unlearning finds application in practical scenarios, it is essential to
thoughtfully address any ethical dilemmas and societal ramifications that may
arise. This underscores our dedication to advancing AI technology in a manner
that is both responsible and mindful of its broader impacts.

##  9 Acknowledgements

S. Liu, J. Jia, and Yuguang Yao were supported by the National Science
Foundation (NSF) Robust Intelligence (RI) Core Program Award IIS-2207052 and
the Cisco Research Faculty Award. P. Hase and M. Bansal were supported by NSF-
CAREER Award 1846185, NSF-AI Engage Institute DRL-2112635, DARPA MCS Grant
N66001-19-2-4031, and Google PhD fellowship.

## References

  * Achille et al. (2023) Achille, A., Kearns, M., Klingenberg, C., and Soatto, S.  Ai model disgorgement: Methods and choices.  _arXiv preprint arXiv:2304.03545_ , 2023.
  * Bae et al. (2022) Bae, J., Ng, N., Lo, A., Ghassemi, M., and Grosse, R. B.  If influence functions are the answer, then what is the question?  _Advances in Neural Information Processing Systems_ , 35:17953–17967, 2022.
  * Bai et al. (2022) Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., et al.  Constitutional ai: Harmlessness from ai feedback.  _arXiv preprint arXiv:2212.08073_ , 2022.
  * Barrett et al. (2023) Barrett, C., Boyd, B., Bursztein, E., Carlini, N., Chen, B., Choi, J., Chowdhury, A. R., Christodorescu, M., Datta, A., Feizi, S., et al.  Identifying and mitigating the security risks of generative ai.  _Foundations and Trends® in Privacy and Security_ , 6(1):1–52, 2023.
  * Becker & Liebig (2022) Becker, A. and Liebig, T.  Evaluating machine unlearning via epistemic uncertainty.  _arXiv preprint arXiv:2208.10836_ , 2022.
  * Belkadi & Jasserand (2023) Belkadi, L. and Jasserand, C.  From algorithmic destruction to algorithmic imprint: Generative ai and privacy risks linked to potential traces of personal data in trained models.  In _ICML Workshop on Generative AI + Law_ , 2023.
  * Bender et al. (2021) Bender, E. M., Gebru, T., McMillan-Major, A., and Shmitchell, S.  On the dangers of stochastic parrots: Can language models be too big?  In _Proceedings of the 2021 ACM conference on fairness, accountability, and transparency_ , pp. 610–623, 2021.
  * Bourtoule et al. (2021) Bourtoule, L., Chandrasekaran, V., Choquette-Choo, C. A., Jia, H., Travers, A., Zhang, B., Lie, D., and Papernot, N.  Machine unlearning.  In _2021 IEEE Symposium on Security and Privacy (SP)_ , pp. 141–159. IEEE, 2021.
  * Bucknall & Trager (2023) Bucknall, B. S. and Trager, R. F.  Structured access for third-party research on frontier ai models: Investigating researchers’model access requirements.  2023\.
  * Cao & Yang (2015) Cao, Y. and Yang, J.  Towards making systems forget with machine unlearning.  In _2015 IEEE symposium on security and privacy_ , pp. 463–480. IEEE, 2015.
  * Carlini et al. (2019) Carlini, N., Liu, C., Erlingsson, Ú., Kos, J., and Song, D.  The secret sharer: Evaluating and testing unintended memorization in neural networks.  In _28th USENIX Security Symposium (USENIX Security 19)_ , pp. 267–284, 2019.
  * Carlini et al. (2021) Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., et al.  Extracting training data from large language models.  In _30th USENIX Security Symposium (USENIX Security 21)_ , pp. 2633–2650, 2021.
  * Carlini et al. (2022) Carlini, N., Ippolito, D., Jagielski, M., Lee, K., Tramer, F., and Zhang, C.  Quantifying memorization across neural language models.  _arXiv preprint arXiv:2202.07646_ , 2022.
  * Carlini et al. (2023) Carlini, N., Jagielski, M., Choquette-Choo, C. A., Paleka, D., Pearce, W., Anderson, H., Terzis, A., Thomas, K., and Tramèr, F.  Poisoning web-scale training datasets is practical.  _arXiv preprint arXiv:2302.10149_ , 2023.
  * Casper et al. (2023) Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., Freedman, R., Korbak, T., Lindner, D., Freire, P., et al.  Open problems and fundamental limitations of reinforcement learning from human feedback.  _arXiv preprint arXiv:2307.15217_ , 2023.
  * Casper et al. (2024) Casper, S., Ezell, C., Siegmann, C., Kolt, N., Curtis, T. L., Bucknall, B., Haupt, A., Wei, K., Scheurer, J., Hobbhahn, M., Sharkey, L., Krishna, S., Hagen, M. V., Alberti, S., Chan, A., Sun, Q., Gerovitch, M., Bau, D., Tegmark, M., Krueger, D., and Hadfield-Menell, D.  Black-box access is insufficient for rigorous ai audits, 2024.
  * Che et al. (2023) Che, T., Zhou, Y., Zhang, Z., Lyu, L., Liu, J., Yan, D., Dou, D., and Huan, J.  Fast federated machine unlearning with nonlinear functional theory.  In _International conference on machine learning_ , pp. 4241–4268. PMLR, 2023.
  * Chen & Yang (2023) Chen, J. and Yang, D.  Unlearn what you want to forget: Efficient unlearning for llms.  _arXiv preprint arXiv:2310.20150_ , 2023.
  * Chen et al. (2022) Chen, M., Zhang, Z., Wang, T., Backes, M., Humbert, M., and Zhang, Y.  Graph unlearning.  In _Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security_ , pp. 499–513, 2022.
  * Chen et al. (2023a) Chen, M., Gao, W., Liu, G., Peng, K., and Wang, C.  Boundary unlearning: Rapid forgetting of deep networks via shifting the decision boundary.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 7766–7775, 2023a.
  * Chen et al. (2023b) Chen, R., Yang, J., Xiong, H., Bai, J., Hu, T., Hao, J., FENG, Y., Zhou, J. T., Wu, J., and Liu, Z.  Fast model debias with machine unlearning.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023b.
  * Chien et al. (2022) Chien, E., Pan, C., and Milenkovic, O.  Efficient model updates for approximate unlearning of graph-structured data.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Christiano et al. (2017) Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D.  Deep reinforcement learning from human preferences.  _Advances in neural information processing systems_ , 30, 2017.
  * Cohen et al. (2023) Cohen, R., Biran, E., Yoran, O., Globerson, A., and Geva, M.  Evaluating the ripple effects of knowledge editing in language models.  _arXiv preprint arXiv:2307.12976_ , 2023.
  * Cui et al. (2023) Cui, C., Zhou, Y., Yang, X., Wu, S., Zhang, L., Zou, J., and Yao, H.  Holistic analysis of hallucination in gpt-4v (ision): Bias and interference challenges.  _arXiv preprint arXiv:2311.03287_ , 2023.
  * Dai et al. (2021) Dai, D., Dong, L., Hao, Y., Sui, Z., Chang, B., and Wei, F.  Knowledge neurons in pretrained transformers.  _arXiv preprint arXiv:2104.08696_ , 2021.
  * Eldan & Russinovich (2023) Eldan, R. and Russinovich, M.  Who’s harry potter? approximate unlearning in llms, 2023.
  * Fan et al. (2024) Fan, C., Liu, J., Zhang, Y., Wei, D., Wong, E., and Liu, S.  Salun: Empowering machine unlearning via gradient-based weight saliency in both image classification and generation.  In _International Conference on Learning Representations_ , 2024.
  * Gandikota et al. (2023) Gandikota, R., Materzynska, J., Fiotto-Kaufman, J., and Bau, D.  Erasing concepts from diffusion models.  _arXiv preprint arXiv:2303.07345_ , 2023.
  * Gao et al. (2023) Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., and Wang, H.  Retrieval-augmented generation for large language models: A survey.  _arXiv preprint arXiv:2312.10997_ , 2023.
  * Gehman et al. (2020) Gehman, S., Gururangan, S., Sap, M., Choi, Y., and Smith, N. A.  Realtoxicityprompts: Evaluating neural toxic degeneration in language models.  _arXiv preprint arXiv:2009.11462_ , 2020.
  * Geva et al. (2020) Geva, M., Schuster, R., Berant, J., and Levy, O.  Transformer feed-forward layers are key-value memories.  _arXiv preprint arXiv:2012.14913_ , 2020.
  * Ginart et al. (2019) Ginart, A., Guan, M., Valiant, G., and Zou, J. Y.  Making ai forget you: Data deletion in machine learning.  _Advances in neural information processing systems_ , 32, 2019.
  * Goland (2023) Goland, J. A.  Algorithmic disgorgement: Destruction of artificial intelligence models as the ftc’s newest enforcement tool for bad data.  _Richmond Journal of Law and Technology_ , 29(2), 2023.
  * Golatkar et al. (2020) Golatkar, A., Achille, A., and Soatto, S.  Eternal sunshine of the spotless net: Selective forgetting in deep networks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 9304–9312, 2020.
  * Grosse et al. (2023) Grosse, R., Bae, J., Anil, C., Elhage, N., Tamkin, A., Tajdini, A., Steiner, B., Li, D., Durmus, E., Perez, E., et al.  Studying large language model generalization with influence functions.  _arXiv preprint arXiv:2308.03296_ , 2023.
  * Grynbaum & Mac (2023) Grynbaum, M. and Mac, R.  The times sues openai and microsoft over a.i. use of copyrighted work.  The New York Times, 2023.  URL https://www.nytimes.com/2023/12/27/business/media/new-york-times-open-ai-microsoft-lawsuit.html.  Accessed: 2024-01-16.
  * Gu et al. (2024) Gu, J.-C., Xu, H.-X., Ma, J.-Y., Lu, P., Ling, Z.-H., Chang, K.-W., and Peng, N.  Model editing can hurt general abilities of large language models.  _arXiv preprint arXiv:2401.04700_ , 2024.
  * Guo et al. (2019) Guo, C., Goldstein, T., Hannun, A., and Van Der Maaten, L.  Certified data removal from machine learning models.  _arXiv preprint arXiv:1911.03030_ , 2019.
  * Gupta et al. (2023) Gupta, A., Mondal, D., Sheshadri, A., Zhao, W., Li, X., Wiegreffe, S., and Tandon, N.  Editing common sense in transformers.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pp. 8214–8232, 2023.
  * Halimi et al. (2022) Halimi, A., Kadhe, S., Rawat, A., and Baracaldo, N.  Federated unlearning: How to efficiently erase a client in fl?  _arXiv preprint arXiv:2207.05521_ , 2022.
  * Hase et al. (2023a) Hase, P., Bansal, M., Kim, B., and Ghandeharioun, A.  Does localization inform editing? surprising differences in causality-based localization vs. knowledge editing in language models.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023a.
  * Hase et al. (2023b) Hase, P., Diab, M., Celikyilmaz, A., Li, X., Kozareva, Z., Stoyanov, V., Bansal, M., and Iyer, S.  Do language models have beliefs? methods for detecting, updating, and visualizing model beliefs.  _EACL_ , 2023b.
  * Hendrycks et al. (2023) Hendrycks, D., Mazeika, M., and Woodside, T.  An overview of catastrophic ai risks.  _arXiv preprint arXiv:2306.12001_ , 2023.
  * Hoofnagle et al. (2019) Hoofnagle, C. J., van der Sloot, B., and Borgesius, F. Z.  The european union general data protection regulation: what it is and what it means.  _Information & Communications Technology Law_, 28(1):65–98, 2019.
  * Huang et al. (2023) Huang, Y., Gupta, S., Xia, M., Li, K., and Chen, D.  Catastrophic jailbreak of open-source llms via exploiting generation.  _arXiv preprint arXiv:2310.06987_ , 2023.
  * Ilharco et al. (2022) Ilharco, G., Ribeiro, M. T., Wortsman, M., Gururangan, S., Schmidt, L., Hajishirzi, H., and Farhadi, A.  Editing models with task arithmetic.  _arXiv preprint arXiv:2212.04089_ , 2022.
  * Ishibashi & Shimodaira (2023) Ishibashi, Y. and Shimodaira, H.  Knowledge sanitization of large language models.  _arXiv preprint arXiv:2309.11852_ , 2023.
  * Izzo et al. (2021) Izzo, Z., Smart, M. A., Chaudhuri, K., and Zou, J.  Approximate data deletion from machine learning models.  In _International Conference on Artificial Intelligence and Statistics_ , pp. 2008–2016. PMLR, 2021.
  * Jang et al. (2022) Jang, J., Yoon, D., Yang, S., Cha, S., Lee, M., Logeswaran, L., and Seo, M.  Knowledge unlearning for mitigating privacy risks in language models.  _arXiv preprint arXiv:2210.01504_ , 2022.
  * Jia et al. (2023) Jia, J., Liu, J., Ram, P., Yao, Y., Liu, G., Liu, Y., Sharma, P., and Liu, S.  Model sparsity can simplify machine unlearning.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023.
  * Kadhe et al. (2023) Kadhe, S. R., Halimi, A., Rawat, A., and Baracaldo, N.  Fairsisa: Ensemble post-processing to improve fairness of unlearning in llms.  _arXiv preprint arXiv:2312.07420_ , 2023.
  * Karamolegkou et al. (2023) Karamolegkou, A., Li, J., Zhou, L., and Søgaard, A.  Copyright violations and large language models.  _arXiv preprint arXiv:2310.13771_ , 2023.
  * Koh & Liang (2017) Koh, P. W. and Liang, P.  Understanding black-box predictions via influence functions.  In _International conference on machine learning_ , pp. 1885–1894. PMLR, 2017.
  * Kotek et al. (2023) Kotek, H., Dockum, R., and Sun, D.  Gender bias and stereotypes in large language models.  In _Proceedings of The ACM Collective Intelligence Conference_ , pp. 12–24, 2023.
  * Kumar et al. (2022) Kumar, V. B., Gangadharaiah, R., and Roth, D.  Privacy adhering machine un-learning in nlp.  _arXiv preprint arXiv:2212.09573_ , 2022.
  * Kumari et al. (2019) Kumari, N., Singh, M., Sinha, A., Machiraju, H., Krishnamurthy, B., and Balasubramanian, V. N.  Harnessing the vulnerability of latent layers in adversarially trained models.  In _Proceedings of the 28th International Joint Conference on Artificial Intelligence_ , pp. 2779–2785, 2019.
  * Kumari et al. (2023) Kumari, N., Zhang, B., Wang, S.-Y., Shechtman, E., Zhang, R., and Zhu, J.-Y.  Ablating concepts in text-to-image diffusion models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 22691–22702, 2023.
  * Kurmanji et al. (2023) Kurmanji, M., Triantafillou, P., and Triantafillou, E.  Towards unbounded machine unlearning.  _arXiv preprint arXiv:2302.09880_ , 2023.
  * Lee et al. (2023) Lee, H., Phatale, S., Mansoor, H., Lu, K., Mesnard, T., Bishop, C., Carbune, V., and Rastogi, A.  Rlaif: Scaling reinforcement learning from human feedback with ai feedback.  _arXiv preprint arXiv:2309.00267_ , 2023.
  * Lermen et al. (2023) Lermen, S., Rogers-Smith, C., and Ladish, J.  Lora fine-tuning efficiently undoes safety training in llama 2-chat 70b.  _arXiv preprint arXiv:2310.20624_ , 2023.
  * Li et al. (2022) Li, D., Rawat, A. S., Zaheer, M., Wang, X., Lukasik, M., Veit, A., Yu, F., and Kumar, S.  Large language models with controllable working memory.  _arXiv preprint arXiv:2211.05110_ , 2022.
  * Li et al. (2023a) Li, D., Sun, Z., Hu, X., Liu, Z., Chen, Z., Hu, B., Wu, A., and Zhang, M.  A survey of large language models attribution.  _arXiv preprint arXiv:2311.03731_ , 2023a.
  * Li et al. (2023b) Li, M., Davies, X., and Nadeau, M.  Circuit breaking: Removing model behaviors with targeted ablation.  _arXiv preprint arXiv:2309.05973_ , 2023b.
  * Li (2022) Li, T. C.  Algorithmic destruction.  _SMU Law Rev._ , 75:479, 2022.
  * Li et al. (2023c) Li, X., Li, S., Song, S., Yang, J., Ma, J., and Yu, J.  Pmet: Precise model editing in a transformer.  _arXiv preprint arXiv:2308.08742_ , 2023c.
  * Li et al. (2021) Li, Y., Lyu, X., Koren, N., Lyu, L., Li, B., and Ma, X.  Anti-backdoor learning: Training clean models on poisoned data.  _Advances in Neural Information Processing Systems_ , 34:14900–14912, 2021.
  * Liu et al. (2020) Liu, G., Ma, X., Yang, Y., Wang, C., and Liu, J.  Federated unlearning.  _arXiv preprint arXiv:2012.13891_ , 2020.
  * Liu et al. (2022) Liu, Y., Fan, M., Chen, C., Liu, X., Ma, Z., Wang, L., and Ma, J.  Backdoor defense with machine unlearning.  In _IEEE INFOCOM 2022-IEEE Conference on Computer Communications_ , pp. 280–289. IEEE, 2022.
  * Liu et al. (2023a) Liu, Y., Deng, G., Xu, Z., Li, Y., Zheng, Y., Zhang, Y., Zhao, L., Zhang, T., and Liu, Y.  Jailbreaking chatgpt via prompt engineering: An empirical study.  _arXiv preprint arXiv:2305.13860_ , 2023a.
  * Liu et al. (2023b) Liu, Z., Jiang, Y., Shen, J., Peng, M., Lam, K.-Y., and Yuan, X.  A survey on federated unlearning: Challenges, methods, and future directions.  _arXiv preprint arXiv:2310.20448_ , 2023b.
  * Lu et al. (2022) Lu, X., Welleck, S., Hessel, J., Jiang, L., Qin, L., West, P., Ammanabrolu, P., and Choi, Y.  Quark: Controllable text generation with reinforced unlearning.  _Advances in neural information processing systems_ , 35:27591–27609, 2022.
  * Madaan et al. (2022) Madaan, A., Tandon, N., Clark, P., and Yang, Y.  Memory-assisted prompt editing to improve gpt-3 after deployment.  _arXiv preprint arXiv:2201.06009_ , 2022.
  * Madry et al. (2017) Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A.  Towards deep learning models resistant to adversarial attacks.  _arXiv preprint arXiv:1706.06083_ , 2017.
  * Maini et al. (2024) Maini, P., Feng, Z., Schwarzschild, A., Lipton, Z. C., and Kolter, J. Z.  Tofu: A task of fictitious unlearning for llms, 2024.
  * Malladi et al. (2023) Malladi, S., Gao, T., Nichani, E., Damian, A., Lee, J. D., Chen, D., and Arora, S.  Fine-tuning language models with just forward passes.  _arXiv preprint arXiv:2305.17333_ , 2023.
  * Meng et al. (2022) Meng, K., Bau, D., Andonian, A., and Belinkov, Y.  Locating and editing factual associations in gpt.  _Advances in Neural Information Processing Systems_ , 35:17359–17372, 2022.
  * Menik & Ramaswamy (2023) Menik, S. and Ramaswamy, L.  Towards modular machine learning solution development: Benefits and trade-offs.  _arXiv preprint arXiv:2301.09753_ , 2023.
  * Mitchell et al. (2022) Mitchell, E., Lin, C., Bosselut, A., Manning, C. D., and Finn, C.  Memory-based model editing at scale.  In _International Conference on Machine Learning_ , pp. 15817–15831. PMLR, 2022.
  * Motoki et al. (2023) Motoki, F., Pinho Neto, V., and Rodrigues, V.  More human than human: Measuring chatgpt political bias.  _Available at SSRN 4372349_ , 2023.
  * Nasr et al. (2023) Nasr, M., Carlini, N., Hayase, J., Jagielski, M., Cooper, A. F., Ippolito, D., Choquette-Choo, C. A., Wallace, E., Tramèr, F., and Lee, K.  Scalable extraction of training data from (production) language models.  _arXiv preprint arXiv:2311.17035_ , 2023.
  * Neel et al. (2021) Neel, S., Roth, A., and Sharifi-Malvajerdi, S.  Descent-to-delete: Gradient-based methods for machine unlearning.  In _Algorithmic Learning Theory_ , pp. 931–962. PMLR, 2021.
  * Nguyen et al. (2022) Nguyen, T. T., Huynh, T. T., Nguyen, P. L., Liew, A. W.-C., Yin, H., and Nguyen, Q. V. H.  A survey of machine unlearning.  _arXiv preprint arXiv:2209.02299_ , 2022.
  * Oesterling et al. (2023) Oesterling, A., Ma, J., Calmon, F. P., and Lakkaraju, H.  Fair machine unlearning: Data removal while mitigating disparities.  _arXiv preprint arXiv:2307.14754_ , 2023.
  * Ouyang et al. (2022) Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744, 2022.
  * Patil et al. (2024) Patil, V., Hase, P., and Bansal, M.  Can sensitive information be deleted from llms? objectives for defending against extraction attacks.  _ICLR_ , 2024.
  * Pawelczyk et al. (2023) Pawelczyk, M., Neel, S., and Lakkaraju, H.  In-context unlearning: Language models as few shot unlearners.  _arXiv preprint arXiv:2310.07579_ , 2023.
  * Perez et al. (2022) Perez, E., Ringer, S., Lukošiūtė, K., Nguyen, K., Chen, E., Heiner, S., Pettit, C., Olsson, C., Kundu, S., Kadavath, S., et al.  Discovering language model behaviors with model-written evaluations.  _arXiv preprint arXiv:2212.09251_ , 2022.
  * Qi et al. (2023) Qi, X., Zeng, Y., Xie, T., Chen, P.-Y., Jia, R., Mittal, P., and Henderson, P.  Fine-tuning aligned language models compromises safety, even when users do not intend to!  _arXiv preprint arXiv:2310.03693_ , 2023.
  * Rafailov et al. (2023) Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., and Finn, C.  Direct preference optimization: Your language model is secretly a reward model.  _arXiv preprint arXiv:2305.18290_ , 2023.
  * Rando & Tramèr (2023) Rando, J. and Tramèr, F.  Universal jailbreak backdoors from poisoned human feedback.  _arXiv preprint arXiv:2311.14455_ , 2023.
  * Robey et al. (2023) Robey, A., Latorre, F., Pappas, G. J., Hassani, H., and Cevher, V.  Adversarial training should be cast as a non-zero-sum game.  _arXiv preprint arXiv:2306.11035_ , 2023.
  * Sattigeri et al. (2022) Sattigeri, P., Ghosh, S., Padhi, I., Dognin, P., and Varshney, K. R.  Fair infinitesimal jackknife: Mitigating the influence of biased training data points without refitting.  _Advances in Neural Information Processing Systems_ , 35:35894–35906, 2022.
  * Sekhari et al. (2021) Sekhari, A., Acharya, J., Kamath, G., and Suresh, A. T.  Remember what you want to forget: Algorithms for machine unlearning.  _Advances in Neural Information Processing Systems_ , 34:18075–18086, 2021.
  * Shafahi et al. (2019) Shafahi, A., Najibi, M., Ghiasi, M. A., Xu, Z., Dickerson, J., Studer, C., Davis, L. S., Taylor, G., and Goldstein, T.  Adversarial training for free!  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Shevlane et al. (2023) Shevlane, T., Farquhar, S., Garfinkel, B., Phuong, M., Whittlestone, J., Leung, J., Kokotajlo, D., Marchal, N., Anderljung, M., Kolt, N., et al.  Model evaluation for extreme risks.  _arXiv preprint arXiv:2305.15324_ , 2023.
  * Shi et al. (2023) Shi, W., Ajith, A., Xia, M., Huang, Y., Liu, D., Blevins, T., Chen, D., and Zettlemoyer, L.  Detecting pretraining data from large language models.  _arXiv preprint arXiv:2310.16789_ , 2023.
  * Shokri et al. (2017) Shokri, R., Stronati, M., Song, C., and Shmatikov, V.  Membership inference attacks against machine learning models.  In _2017 IEEE symposium on security and privacy (SP)_ , pp. 3–18. IEEE, 2017.
  * Si et al. (2023) Si, N., Zhang, H., Chang, H., Zhang, W., Qu, D., and Zhang, W.  Knowledge unlearning for llms: Tasks, methods, and challenges.  _arXiv preprint arXiv:2311.15766_ , 2023.
  * Small (2023) Small, Z.  Sarah silverman sues openai and meta over copyright infringement.  The New York Times, 2023.  URL https://www.nytimes.com/2023/07/10/arts/sarah-silverman-lawsuit-openai-meta.html.  Accessed: 2024-01-16.
  * Tamkin et al. (2023) Tamkin, A., Askell, A., Lovitt, L., Durmus, E., Joseph, N., Kravec, S., Nguyen, K., Kaplan, J., and Ganguli, D.  Evaluating and mitigating discrimination in language model decisions.  _arXiv preprint arXiv:2312.03689_ , 2023.
  * Thudi et al. (2022) Thudi, A., Deza, G., Chandrasekaran, V., and Papernot, N.  Unrolling sgd: Understanding factors influencing machine unlearning.  In _2022 IEEE 7th European Symposium on Security and Privacy (EuroS &P)_, pp. 303–319. IEEE, 2022.
  * Toyer et al. (2023) Toyer, S., Watkins, O., Mendes, E. A., Svegliato, J., Bailey, L., Wang, T., Ong, I., Elmaaroufi, K., Abbeel, P., Darrell, T., et al.  Tensor trust: Interpretable prompt injection attacks from an online game.  _arXiv preprint arXiv:2311.01011_ , 2023.
  * Ullah et al. (2021) Ullah, E., Mai, T., Rao, A., Rossi, R. A., and Arora, R.  Machine unlearning via algorithmic stability.  In _Conference on Learning Theory_ , pp. 4126–4142. PMLR, 2021.
  * Wang et al. (2019) Wang, B., Yao, Y., Shan, S., Li, H., Viswanath, B., Zheng, H., and Zhao, B. Y.  Neural cleanse: Identifying and mitigating backdoor attacks in neural networks.  In _2019 IEEE Symposium on Security and Privacy (SP)_ , pp. 707–723. IEEE, 2019.
  * Wang et al. (2022) Wang, J., Guo, S., Xie, X., and Qi, H.  Federated unlearning via class-discriminative pruning.  In _Proceedings of the ACM Web Conference 2022_ , pp. 622–632, 2022.
  * Wang et al. (2023) Wang, L., Chen, T., Yuan, W., Zeng, X., Wong, K.-F., and Yin, H.  Kga: A general machine unlearning framework based on knowledge gap alignment.  _arXiv preprint arXiv:2305.06535_ , 2023.
  * Warnecke et al. (2021) Warnecke, A., Pirch, L., Wressnegger, C., and Rieck, K.  Machine unlearning of features and labels.  _arXiv preprint arXiv:2108.11577_ , 2021.
  * Wei et al. (2023) Wei, A., Haghtalab, N., and Steinhardt, J.  Jailbroken: How does llm safety training fail?  _arXiv preprint arXiv:2307.02483_ , 2023.
  * Wen et al. (2023) Wen, J., Ke, P., Sun, H., Zhang, Z., Li, C., Bai, J., and Huang, M.  Unveiling the implicit toxicity in large language models.  In _The 2023 Conference on Empirical Methods in Natural Language Processing_ , 2023.
  * Wong et al. (2020) Wong, E., Rice, L., and Kolter, J. Z.  Fast is better than free: Revisiting adversarial training.  _arXiv preprint arXiv:2001.03994_ , 2020.
  * Wu et al. (2023a) Wu, K., Shen, J., Ning, Y., Wang, T., and Wang, W. H.  Certified edge unlearning for graph neural networks.  In _Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_ , pp. 2606–2617, 2023a.
  * Wu et al. (2023b) Wu, X., Li, J., Xu, M., Dong, W., Wu, S., Bian, C., and Xiong, D.  Depn: Detecting and editing privacy neurons in pretrained language models.  _arXiv preprint arXiv:2310.20138_ , 2023b.
  * Yang et al. (2023) Yang, X., Wang, X., Zhang, Q., Petzold, L., Wang, W. Y., Zhao, X., and Lin, D.  Shadow alignment: The ease of subverting safely-aligned language models.  _arXiv preprint arXiv:2310.02949_ , 2023.
  * Yao et al. (2023) Yao, Y., Xu, X., and Liu, Y.  Large language model unlearning.  _arXiv preprint arXiv:2310.10683_ , 2023.
  * Yong et al. (2023) Yong, Z.-X., Menghini, C., and Bach, S. H.  Low-resource languages jailbreak gpt-4, 2023.
  * Yu et al. (2023) Yu, C., Jeoung, S., Kasi, A., Yu, P., and Ji, H.  Unlearning bias in language models by partitioning gradients.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pp. 6032–6048, 2023.
  * Yuan et al. (2023) Yuan, Z., Yuan, H., Tan, C., Wang, W., Huang, S., and Huang, F.  Rrhf: Rank responses to align language models with human feedback without tears.  _arXiv preprint arXiv:2304.05302_ , 2023.
  * Zhang et al. (2023a) Zhang, D., Finckenberg-Broman, P., Hoang, T., Pan, S., Xing, Z., Staples, M., and Xu, X.  Right to be forgotten in the era of large language models: Implications, challenges, and solutions.  _arXiv preprint arXiv:2307.03941_ , 2023a.
  * Zhang et al. (2023b) Zhang, E., Wang, K., Xu, X., Wang, Z., and Shi, H.  Forget-me-not: Learning to forget in text-to-image diffusion models.  _arXiv preprint arXiv:2303.17591_ , 2023b.
  * Zhang et al. (2023c) Zhang, J., Chen, S., Liu, J., and He, J.  Composing parameter-efficient modules with arithmetic operations.  _arXiv preprint arXiv:2306.14870_ , 2023c.
  * Zhang et al. (2022) Zhang, Y., Zhang, G., Khanduri, P., Hong, M., Chang, S., and Liu, S.  Revisiting and advancing fast adversarial training through the lens of bi-level optimization.  In _International Conference on Machine Learning_ , pp. 26693–26712. PMLR, 2022.
  * Zhang et al. (2023d) Zhang, Y., Jia, J., Chen, X., Chen, A., Zhang, Y., Liu, J., Ding, K., and Liu, S.  To generate or not? safety-driven unlearned diffusion models are still easy to generate unsafe images… for now.  _arXiv preprint arXiv:2310.11868_ , 2023d.
  * Zheng et al. (2023) Zheng, C., Li, L., Dong, Q., Fan, Y., Wu, Z., Xu, J., and Chang, B.  Can we edit factual knowledge by in-context learning?  _arXiv preprint arXiv:2305.12740_ , 2023.
  * Zhong et al. (2023) Zhong, Z., Wu, Z., Manning, C. D., Potts, C., and Chen, D.  Mquake: Assessing knowledge editing in language models via multi-hop questions.  _arXiv preprint arXiv:2305.14795_ , 2023.
  * Zhu et al. (2019) Zhu, C., Cheng, Y., Gan, Z., Sun, S., Goldstein, T., and Liu, J.  Freelb: Enhanced adversarial training for natural language understanding.  _arXiv preprint arXiv:1909.11764_ , 2019.
  * Zou et al. (2023) Zou, A., Wang, Z., Kolter, J. Z., and Fredrikson, M.  Universal and transferable adversarial attacks on aligned language models.  _arXiv preprint arXiv:2307.15043_ , 2023.