# Large Language Model Unlearning via Embedding-Corrupted Prompts

Chris Yuhao Liu Yaxuan Wang Jeffrey Flanigan Yang Liu22footnotemark: 2

University of California, Santa Cruz
<EMAIL_ADDRESS>Corresponding author:
<EMAIL_ADDRESS>advising.

###### Abstract

Large language models (LLMs) have advanced to encompass extensive knowledge
across diverse domains. Yet controlling what a large language model should not
know is important for ensuring alignment and thus safe use. However,
accurately and efficiently unlearning knowledge from an LLM remains
challenging due to the potential collateral damage caused by the fuzzy
boundary between retention and forgetting, and the large computational
requirements for optimization across state-of-the-art models with hundreds of
billions of parameters. In this work, we present Embedding-COrrupted (ECO)
Prompts, a lightweight unlearning framework for large language models to
address both the challenges of knowledge entanglement and unlearning
efficiency. Instead of relying on the LLM itself to unlearn, we enforce an
unlearned state during inference by employing a prompt classifier to identify
and safeguard prompts to forget. We learn corruptions added to prompt
embeddings via zeroth order optimization toward the unlearning objective
offline and corrupt prompts flagged by the classifier during inference. We
find that these embedding-corrupted prompts not only lead to desirable outputs
that satisfy the unlearning objective but also closely approximate the output
from a model that has never been trained on the data intended for forgetting.
Through extensive experiments on unlearning, we demonstrate the superiority of
our method in achieving promising unlearning at nearly zero side effects in
general domains and domains closely related to the unlearned ones.
Additionally, we highlight the scalability of our method to 100 LLMs, ranging
from 0.5B to 236B parameters, incurring no additional cost as the number of
parameters increases.

Figure 1: Using embedding-corrupted prompts to maintain an unlearned state on
the LLM subject to unlearning We first employ a classifier to identify whether
the incoming prompt falls within the scope of the unlearning target. We
construct embedding-corrupted prompts by selectively corrupting dimensions
within the tokens’ embeddings. The corruption parameter is learned offline via
zeroth order optimization. An unlearned state is imposed during inference and
does not require any updates to the original model’s weights.

##  1 Introduction

The use of large language models (LLMs), trained on extensive text corpora [2,
42, 6, 124, 61, 8, 137], has increasingly become standard in daily life since
the arrival of ChatGPT [98]. Despite the benefits LLMs offer, they pose
potential risks across a range of domains, such as copyright infringement [62,
46, 69], dissemination of hazardous knowledge [70, 50, 37, 109], and privacy
violations [119, 90, 95]. Adherence to the General Data Protection Regulation
(GDPR) [35], which requires the removal of users’ data post-training. Machine
unlearning has emerged as a new paradigm [18, 96] and has been widely studied
for classification models and tasks in recent years [123, 76, 66, 36].
However, unlearning in the context of LLMs remains largely underexplored,
presenting unique challenges and risks that extend beyond privacy concerns due
to the infeasibility of retraining from scratch [18, 19], the ease for anyone
to access powerful models, and the substantial capabilities of these models
across various tasks [77, 85].

Various machine unlearning methods have been proposed specifically for LLMs to
address the above challenges. A major line of approaches focuses on parameter
fine-tuning [57] based on a modified loss, usually by unlearning on the forget
data and learning from the retained data to preserve utility [130, 136, 22,
135, 70, 142, 59], which require only a small number of weight updates
compared to retraining from scratch. Other approaches include model editing
techniques [55, 132, 13, 141, 53, 97, 80], unlearning via in-context examples
in the prompt [101, 94], and guarding the prompts themselves [122]. Although
effective, some approaches have been shown to impair a model’s general
capabilities [47, 85]. This is due to knowledge entanglement [87, 85] caused
by either the fuzzy boundary between retention and forgetting objectives
(e.g., forgetting a single person without affecting other related ones).
Additionally, most prior work targets unlearning at the million- to billion-
parameter scale through gradient-based optimization [136, 53, 34, 135, 87,
142, 70, 59], making the cost of unlearning scale with the model size and can
be expensive even with parameter-efficient modules.

In this work, we explore if an “unlearned state” can be imposed on an intact
LLM and focus on tackling the challenges of knowledge entanglement and
unlearning inefficiency in LLMs. We hypothesize that unlearning can be
implemented as a state by decomposing the unlearning problem into two more
tractable subproblems: 1) unlearning target identification, which explicitly
identifies if the prompt contains content within the unlearning target, and 2)
forgetting, which ensures that the generated responses no longer reflect any
prior knowledge related to the unlearning target. We present Embedding-
COrrupted (ECO111Pronounced as “echo.”) Prompts, a lightweight two-step
framework to tackle both problems above:

  1. 1.

To identify the unlearning target, we use a prompt classifier that is trained
to explicitly model the prompt distribution and to safeguard prompts within
the scope of the unlearning target.

  2. 2.

To achieve forgetting, we approximate an unlearned state by passing the query
identified by the prompt classifier to the LLM, but in a corrupted form. We
leverage corruptions learned efficiently via zeroth order optimization [117,
118] and apply them to the prompt’s embedding space during inference.

Our contributions are as follows:

  * •

We introduce Embedding-COrrupted (ECO) Prompts, a novel and lightweight LLM
unlearning method that enforces an unlearned state over an intact LLM.

  * •

We demonstrate that, instead of relying on unlearning objective optimization,
carefully corrupted prompts leads to behavior that resembles that of a model
which has never seen the data intended to be forgotten, across multiple tasks
and metrics.

  * •

Through extensive experiments across three knowledge unlearning tasks, we
demonstrate the superior performance of our method in both retaining and
forgetting, incurring virtually zero side effects and no additional cost when
scaling to larger models.

  * •

To the best of our knowledge, we are the first to demonstrate universally
effective and efficient unlearning for 100 LLMs and up to 236B parameters.

##  2 Preliminaries and Problem Setup

###  2.1 Threat Model

In our threat model, we consider threats in three categories: entity leaking,
hazardous knowledge, and copyrighted content extraction. We consider a gray
box setting similar to that of [70] and [122], where users interact with an
LLM through a chat interface or structured API access [115], such as the GPT-4
API [2]. Under this setting, all users can send prompts to the LLM and receive
the corresponding completions or per-token logits of the output tokens. We
also assume that adversaries within the user group generate prompts in-
distribution and attempt to jailbreak either the guarding mechanism or the LLM
itself. Our threats and goals below are as follows.

Entity leaking  Entity leaking occurs when an LLM inadvertently discloses the
identity or sensitive information of specific individuals whose data was
unintentionally included in the training set [64, 16, 83]. Our goal is to
ensure that the LLM either provides incorrect responses or refuses to answer
queries from threat agents that involve these individuals or groups.

Hazardous knowledge  Given the ease of use and accessibility of both
commercial and open-source LLMs, individuals with malicious intent could
exploit the advanced capabilities of LLMs to acquire hazardous knowledge at
minimal cost [70, 50, 37, 109]. Here, the objective is to prevent such actors
from obtaining dangerous knowledge from LLMs while ensuring that the models
retain their original capabilities in benign but related domains.

Copyrighted content  Extracting and distributing copyrighted content from an
LLM is generally illegal, as it involves reproducing and distributing
protected material without permission [46, 62, 69]. Even if copyrighted
content is filtered from the pre-training data, fragments of the text may
still be scattered throughout the corpus, and the LLM could memorize them. An
attempt to extract the original passage by prompting with a known portion of
the text might cause the LLM to generate the passage verbatim, which we aim to
prevent.

###  2.2 Problem Setup

We assume a learning algorithm $A$222This algorithm $A$ may not be
deterministic, and is assumed to be randomized., the training set $D_{tr}$ and
the forget set $D_{f}$. For each dataset $D$, we have
$D=\\{\mathbf{z}_{i}\\}_{i=1}^{N}$, where each
$\mathbf{z}_{i}=\\{\mathbf{x}_{i},\mathbf{y}_{i}\\}$. In the traditional
setting of machine unlearning, [18, 96], a retained model $\bm{\theta}_{r}$
that has never seen the forget dataset is obtained via the learning algorithm
but excluding the forget set, $\bm{\theta}_{r}=A(D_{tr}\setminus D_{f})$,
where $D_{r}=D_{tr}\setminus D_{f}$ is known as the retain set. We use
$\bm{\theta}_{o}$ to denote the original model333Throughout the paper, we also
call $\bm{\theta}_{o}$ “the model subject to unlearning.” obtained from the
learning algorithm $A$ and $\bm{\theta}_{r}$ to represent a retained model
retrained from scratch via an unlearning algorithm $U$, which we define below,
via training on on $D_{tr}$ and $D_{r}$, respectively.

Based on our threat model in Section 2.1, which does not allow users to access
model weights, instead of achieving unlearning in the weight space [96], we
focus on weak unlearning [11] in the output space. Specifically, we aim at
similarity between models $h(\mathbf{x};\bm{\theta}_{r})$ and
$h(\mathbf{x};\bm{\theta}_{u})$ for all $\mathbf{x}$, where
$h:\mathcal{X}\times\Theta\rightarrow\mathcal{Y}$ maps from the input space
$\mathcal{X}$ and weight space $\Theta$ to the output space $\mathcal{Y}$.

A relaxed objective of unlearning  Because we are in the LLM setting, we use a
relaxed definition of unlearning that does not require differential private
requirement (i.e., ($\epsilon,\delta$)-close), similar to [112]. More
specifically, we follow prior work [43, 22, 66, 58, 135, 51] and evaluate if
the retained model and the unlearned model’s metric values over a set of
metrics $\mathcal{M}=\\{m_{1},m_{2},...,m_{K}\\}$ are similar, on both $D_{r}$
and $D_{f}$. To maintain the general utility of the LLM after unlearning, we
would also like the model to perform well on an o.o.d. general domain
distribution $\mathcal{D}_{g}$, which is unknown during unlearning. Therefore,
our goal of unlearning is

$\frac{\mathbb{E}[m_{i}\left(h\left(\mathbf{x};\bm{\theta}_{u}\right)\right)]}{\mathbb{E}[m_{i}\left(h\left(\mathbf{x};\bm{\theta}_{r}\right)\right)]}\approx
1$ (1)

for all $m_{i}\in\mathcal{M}$, where $\mathcal{M}$ is a set of non-negative
metrics. We want this to hold separately for each case $\mathbf{x}\sim
p_{\mathcal{D}_{f}}(\mathbf{x})$, $\mathbf{x}\sim
p_{\mathcal{D}_{r}}(\mathbf{x})$, and $\mathbf{x}\sim
p_{\mathcal{D}_{g}}(\mathbf{x})$. During evaluation, we assess if the two
models have empirically similar performance over the metrics set
$\mathcal{M}$.

##  3 ECO: Unlearned LLMs via Embedding-Corrupted Prompts

###  3.1 Method Overview

Our method consists of two steps: 1) train a prompt classifier to predict if
an incoming prompt falls within the scope of unlearning, and 2) corrupt the
prompt in the embedding space if the classifier makes a positive prediction
(i.e., should forget).

Enforcing retaining and forgetting via a classifier  We first train a prompt
classifier to explicitly identify if the prompt falls withing the scope of
unlearning. For any incoming prompt, $\mathbf{x}$, the prompt classifier $C$
takes in $\mathbf{x}$ and returns $p_{C}(f\mid\mathbf{x})$
$=1-p_{C}(r\mid\mathbf{x})$, the probability of the prompt being in the scope
of forgetting. Similar to any classifier prediction, if
$p_{C}(f\mid\mathbf{x})>p_{C}(r\mid\mathbf{x})$, we consider $\mathbf{x}$ as
containing the unlearning concept that our LLM is supposed to forget.
Formally, given a positive prediction,
$p_{C}(f\mid\mathbf{x})>p_{C}(r\mid\mathbf{x})$, we replace the original input
$\mathbf{x}$ by a $\tilde{\mathbf{x}}$. Otherwise, the original $\mathbf{x}$
is passed to the LLM.

$\mathbf{x}=\begin{cases}\tilde{\mathbf{x}}&p_{C}(f\mid\mathbf{x})>p_{C}(r\mid\mathbf{x})\\\
\mathbf{x}&\text{otherwise}\end{cases}$ (2)

Embedding-corrupted prompts  Instead of a modification of $\mathbf{x}$ in the
token space, we corrupt it in the embedding space. Let
$\mathbf{x}=\\{x_{1},x_{2},\dots,x_{T}\\}$ be a prompt of $T$ tokens and
$\mathbf{e}=\\{e_{1},e_{2},...,e_{T}\\}$ be the corresponding embedding
vectors. Let $\mathcal{E}$ be the space of the token embeddings. Each
embedding vector is produced by an embedding function
$E:\mathcal{X}\rightarrow\mathbb{R}^{d}$. We also use the symbol
$\sigma\in\mathcal{S}$ (where $\mathcal{S}\subset\mathbb{R}$) to denote the
strength of the corruption, which parameterizes the strength of the corruption
function. Formally, for a single prompt $\mathbf{x}$ mapped to the embeddings
$\mathbf{e}=E(\mathbf{x})=\\{e_{1},e_{2},...,e_{T}\\}$, a corruption function
$\operatorname{\texttt{Corrupt}}:\mathcal{E}\times\mathcal{S}\rightarrow\mathcal{E}$,
parameterized by $\sigma$, produces the embedding-corrupted prompts

$\tilde{\mathbf{e}}=\operatorname{\texttt{Corrupt}}(\mathbf{e};\sigma)=\\{\tilde{e}_{1},\tilde{e}_{2},\dots,\tilde{e}_{T}\\}.$
(3)

Let $\tilde{h}:\mathcal{E}\times\Theta\rightarrow\mathcal{Y}$ be the function
$h$ but taking the input embeddings instead of input tokens (i.e. $h$ with the
input embedding layer detached), our objective is to pick a good $\sigma^{*}$
such that the following modified unlearning objective is satisfied:

$\frac{\mathbb{E}\left[m_{i}\left(\tilde{h}\left(\operatorname{\texttt{Corrupt}}(\mathbf{e};\sigma^{*});\bm{\theta}_{o}\right)\right)\right]}{\hat{v}_{r}}\approx
1,\forall m_{i}\in\mathcal{M}.$ (4)

Here, $\hat{v}_{r}$ is used to approximate the true
$\mathbb{E}[m_{i}(\tilde{h}(\mathbf{e};\bm{\theta}_{r}))]$ as the retained
model is not available.

###  3.2 Decision Threshold Calibration and Conformal Prediction

Due to the potential fuzzy boundary between retaining and forgetting, one
needs to pick a threshold that is better than
$p(f\mid\mathbf{x})>p(r\mid\mathbf{x})$, which does not take into account the
classifier’s confidence. Depending on the application and the empirical
performance of the classifier predictions, we incorporate two types of
thresholding techniques.

Simple thresholding  We choose a simple threshold, $\tau$, as the criterion to
determine if a prompt $\mathbf{x}$ belongs to the forget distribution.
Formally, the output $\hat{\mathbf{y}}$ from the LLM is returned by feeding a
prompt selected by the classifier, based on its prediction
$p_{C}(f\mid\mathbf{x})$:

$\hat{\mathbf{y}}=\begin{cases}\tilde{h}\left(\operatorname{\texttt{Corrupt}}(\mathbf{e};\sigma);\bm{\theta}_{o}\right)&\text{if
}p_{C}(f\mid\mathbf{x})\geq\tau\\\
\tilde{h}\left(\mathbf{e};\bm{\theta}_{o}\right)&\text{otherwise}\end{cases}$
(5)

We pick the value of $\tau$ using a separate set $D_{\textrm{cal}}$ for
calibration. The goal is to choose an optimal $\tau$ that has the smallest
false positive rate and false negative rate on $D_{\textrm{cal}}$.

Conformal prediction  We also consider conformal prediction (CP) [127], which
finds a calibrated threshold given a target error rate $\alpha$, as a second
way for threshold calibration. In essence, a conformal prediction uses a small
user-specified error rate, $\alpha$, and unlikelihood scores (e.g.,
$1-p_{C}(y\mid\mathbf{x})$) on a calibration set to derive a threshold, and
labels with unlikelihood scores lower than the threshold are included in the
final prediction set.

We adapt the split conformal prediction setup [127], which uses a separate
calibration set, $D_{\textrm{cal}}=\\{\mathbf{x}_{i},y_{i}\\}_{i=1}^{n}$
($y\in\\{\textrm{r},f\\}$), to determine a conformity threshold, and a non-
conformity score, $S:\mathcal{X}\times\mathcal{Y}\rightarrow\mathbb{R}$, to
measure how unlikely a sample $(x,y)$ is to the classifier $C$. We follow
convention choice and use
$s_{i}=S(\mathbf{x}_{i},y_{i})=1-p_{C}(y_{i}\mid\mathbf{x}_{i})$ as the non-
conformity score. Given the calibration set size $n$, and a small user-
specified error rate $\alpha$, we determine a quantile $\hat{q}$ using the
$\lceil(n+1)\cdot(1-\alpha)\rceil/n$ empirical quantile in the non-conformity
scores from $D_{\textrm{cal}}$. The final prediction set on a new test sample
$\mathbf{x}_{\textrm{test}}$ is formed by including all labels with a non-
conformity score below $\hat{q}$ as

$\mathcal{C}_{\alpha}\left(\mathbf{x}_{\textrm{test}}\right)=\left\\{y\in\mathcal{Y}:S\left(\mathbf{x}_{\textrm{test}},y\right)\leq\hat{q}\right\\}.$
(6)

Formally, given the prompt classifier $C$, a prediction set
$\mathcal{C}_{\alpha}\left(\mathbf{x}\right)$ for the prompt $\mathbf{x}$, and
the decision threshold $\tau$, the response from the LLM is obtained by the
following rules:

$\hat{\mathbf{y}}=\begin{cases}\tilde{h}\left(\operatorname{\texttt{Corrupt}}(\mathbf{e};\sigma);\bm{\theta}_{o}\right)&\text{if
}1\in\mathcal{C}_{\alpha}\\\
\tilde{h}\left(\mathbf{e};\bm{\theta}_{o}\right)&\text{otherwise}\end{cases}$
(7)

In experiments, we pick the thresholding method based on its empirical
performance. In Section C.5, we give a toy example of how to determine the
prediction set size for a test sample.

###  3.3 Embedding-Corrupted Prompts

Given an accurate classifier, one can already mitigate the risk defined in our
threat model by providing a template response. However, doing so violates the
weak unlearning objective for $\mathbf{x}\sim p_{\mathcal{D}_{f}}(\mathbf{x})$
in Equation 1, because a retained model (that is, a model not trained on the
forget data) is highly unlikely to give the template responses to prompts in
the forget data. To actually achieve unlearning given the prompt classifier
obtained in Section 3.2, we introduce a simple method that learns to corrupt
user’s prompts in the embedding space efficiently via zeroth order
optimization [117, 118] toward the unlearning objective. One may also set
$\sigma$ manually without optimization, at the cost of being further away from
desired retrained model (see below).

Optimization objective  A natural choice to make the unlearned model to behave
like a retained model is to minimize a distance function which quantifies the
gap between the two models for all $m\in\mathcal{M}$. As the retained model is
not available (otherwise we do not need to unlearn) in practice, we use a
surrogate metric value $\hat{v}_{r}$ if available to approximate how the
retained model would behave over $\mathcal{M}$. Based on our relaxed
unlearning objective in Equation 4, we define a general distance measure
below:

$\displaystyle d(\tilde{\mathbf{e}},\bm{\theta}_{o},\hat{v}_{r},\mathcal{M})$
$\displaystyle=\frac{1}{|\mathcal{M}|}\sum_{i}\Big{|}\underbrace{m_{i}(\tilde{h}(\tilde{\mathbf{e}};\bm{\theta}_{o}))}_{\text{unlearned
metric value}}-\underbrace{\hat{v}_{r}}_{\text{surrogate retain metric
value}}\Big{|}$ (8)

We aim to learn a $\sigma^{*}$ such that the metric gap in Equation 8 between
the unlearned model and the retained model is minimized. Formally, given a
parameterized corruption function
$\operatorname{\texttt{Corrupt}}(\cdot;\sigma)$, our unlearning objective is
to minimize the following:

$\sigma^{*}=\operatorname*{arg\,min}_{\sigma}d\left(\operatorname{\texttt{Corrupt}}(\mathbf{e};\sigma),\bm{\theta}_{o},\hat{v}_{r},\mathcal{M}\right)$
(9)

Note: If the metric value $\hat{v}_{r}$ is not possible to obtain, one may
tune $\sigma$ directly and inspect if the model output on the forget set
aligns with the unlearning criteria. For classification-style tasks, the
target $\hat{v}_{r}$ may be known to be random guessing.

Corruption learning via zeroth order optimization  We now formulate the zeroth
order gradient approximation via finite differences [117, 118]. Given a pre-
defined perturbation size $\mu$ applied to the current corruption parameter
$\sigma_{k}$, we treat the distance function $d(\cdot)$ as a black-box and
query it for the final metric gap during optimization. Because we only learn
the strength of the corruption function with s scalar-valued $\sigma$, we use
a deterministic perturbation to $\sigma_{k}$. For a single sample, given an
initial guess $\sigma_{0}$, a step size $\eta$, a smoothing parameter $\mu$
(also known as perturbation size), the minimization of Equation 9 uses the
following update rules:

$\displaystyle\tilde{\mathbf{e}}_{\textrm{forward}}$
$\displaystyle=\mathbf{e}+\operatorname{\texttt{Corrupt}}(\mathbf{e};\sigma_{k}+\mu)$
(10) $\displaystyle\tilde{\mathbf{e}}_{\textrm{backward}}$
$\displaystyle=\mathbf{e}+\operatorname{\texttt{Corrupt}}(\mathbf{e};\sigma_{k}-\mu)$
(11) $\displaystyle\hat{\nabla}d_{\sigma_{k}}$
$\displaystyle=\frac{d(\tilde{\mathbf{e}}_{\textrm{forward}},\bm{\theta}_{o},\hat{v}_{r},\mathcal{M})-d(\tilde{\mathbf{e}}_{\textrm{backward}},\bm{\theta}_{o},\hat{v}_{r},\mathcal{M})}{2\mu}$
(12) $\displaystyle\sigma_{k+1}$
$\displaystyle=\sigma_{k}-\eta\hat{\nabla}d_{\sigma_{k}}$ (13)

Choice of corruption function Prior work [38] suggests that only a small
number of dimensions for each embedding vector suffices to steer the output,
so we only corrupt the first dimension of each token’s embedding. We also
experimented with other corruption functions (e.g., standard Gaussian noise or
zeroing-out top entries), but find our method is insensitive to the corruption
function and all corruption functions give similar end results. We conducted
ablation studies on various corruption functions in Section D.2.

##  4 Experiments

In this section, we present experimental results for entity unlearning
(Section 4.2), hazardous knowledge unlearning (Section 4.3), and copyrighted
content unlearning (Section 4.4).

###  4.1 Prompt Classifier

For each unlearning task, we fine-tune a RoBERTa [79] to use as the prompt
classifier on the corresponding $D_{r}$ and $D_{f}$. In entity and copyrighted
content unlearning tasks, we use the entire $D_{f}$ to train the
classifier444In [87], the entire forget set is used during unlearning. because
the unlearning target is fully captured by the forget set, which does not
require generalization outside the set. For WMDP and MMLU, we only use a
surrogate synthetic forget set $D_{\bar{f}}$ to train the prompt classifier,
and the actual forget set $D_{f}$ is not accessible until evaluation. For all
prompt classifiers, we use an independent validation set $D_{\textrm{val}}$ to
tune the decision threshold $\tau$ and hyperparmaeters or to calibrate the
empirical quantile $\hat{q}$, which is used to determine conformity. In Tables
6, 7 and 5, we show that all classifiers can distinguish $D_{r}$ and $D_{f}$
well, and generalize to unseen $D_{g}$ with low false positive rate. We
provide further detailed information on how prompt classifiers are trained for
each of the task and their performance in Section C.3.

Figure 2: Model utility versus forget quality (p-value) on three different
forget set sizes of the TOFU dataset after unlearning. We show two models,
Phi-1.5 (top) and Llama-2-7B-Chat (bottom). For GA, GD, KL, PO, and the
prompting baseline, the forget quality are either too small or comes at the
cost of substantial decrease of model utility. Negative preference
optimization (NPO) [142] variants achieve a great balance in some cases, but
the trade-off on model utility is still non-trivial. ECO-RN (random noise) and
ECO-ZO (zero-out) achieve almost identical distribution to the retained model
while having no sacrifice in model utility.

###  4.2 Entity Unlearning

Experimental setup  The TOFU dataset [87] is a synthetic question-answering
dataset of author biographies. The goal is for an LLM trained on the complete
dataset (all authors) to unlearn a fraction of fictitious authors (1/5/10%)
while retaining knowledge about both 1) the remaining fictitious authors and
2) the real-world. To assess forgetting and retaining, we use two metrics
proposed alongside the TOFU dataset, forget quality and model utility. Forget
quality is represented by a p-value from a Kolmogorov-Smirnov (KS) test, where
high value indicates high similarity in distribution between the output of the
unlearned model and that of the retained model. Model utility assesses the
model’s performance on the retained set and real-world knowledge. For a
detailed description of all the metrics, refer to Section C.1.1. We conduct
experiments with two corruption functions, random noise (RN) and zero-out
(ZO). We include all baselines in [87], a prompting baseline, and the recently
proposed negative preference optimization (NPO) [142]. We provide formulations
of all baselines Section C.4.

ECO brings Pareto improvement.  In Figure 2, we illustrate the trade-off
between model utility and forget quality for two models, Phi-1.5 [71] and
Llama-2-7B-Chat [124], including forgetting 1%, 5%, and 10% of the samples.
ECO-RN and ECO-ZO consistently achieve close-to-perfect forget quality
regardless of the model or the size of the forget set. Notably, ECO-ZO
maintains a distribution almost identical to the retained model (as the
p-value is close to 1) in all cases, suggesting that ECO prompts can
effectively approximate the outputs of the retained model in distribution.
Given that the prompt classifier trained on the TOFU dataset incurs zero false
positives, our method results in zero sacrifice in model utility, thus
striking a perfect balance between forgetting and retention. For the ECO-RN
variant, we optimize $\sigma$ for Llama-2-7b-Chat on 1% of the forget set and
use the same value for all five other settings, suggesting its transferability
across models and forget tasks.

Model | Method |  Bio ($\downarrow$) |  Chem ($\downarrow$) |  Cyber ($\downarrow$) |  MMLU ($\uparrow$)
---|---|---|---|---|---
Zephyr-7B | Original | 64.2 | 48.3 | 43.1 | 58.9
Prompting | 63.2 | 43.6 | 44.0 | 57.8
LLMU | 59.5 | 41.4 | 39.5 | 44.7
SCRUB | 43.8 | 40.4 | 39.3 | 51.2
SSD | 50.2 | 33.8 | 35.0 | 40.7
RMU | 29.7 | 47.1 | 28.1 | 57.5
ECO (Ours) | 24.7 | 26.5 | 24.4 | 58.9
Yi-34B-Chat | Original | 76.2 | 56.9 | 56.9 | 72.8
Prompting | 43.0 | 36.0 | 47.2 | 61.0
RMU | 31.0 | 54.7 | 27.9 | 71.0
ECO (Ours) | 25.9 | 24.0 | 25.3 | 72.8
Mixtral-8x7B-Instruct (47B) | Original | 71.6 | 53.4 | 51.9 | 67.7
Prompting | 46.4 | 37.0 | 47.7 | 61.9
RMU | 32.0 | 52.7 | 31.4 | 66.1
ECO (Ours) | 25.0 | 23.4 | 26.4 | 67.7
Mixtral-8x22B-Instruct (141B) | Original | 77.3 | 56.6 | 52.6 | 73.9
Prompting | 56.4 | 45.6 | 42.5 | 69.8
ECO (Ours) | 26.7 | 23.9 | 24.1 | 73.9
DeepSeek-V2-Chat (236B) | Original | 76.5 | 57.4 | 48.9 | 74.7
Prompting | 54.4 | 44.9 | 46.3 | 71.2
ECO (Ours) | 23.2 | 27.0 | 23.8 | 74.7
| Random guess | 25.0 | 25.0 | 25.0 | 25.0

Table 1: Multiple-choice accuracy of five LLMs on the WMDP benchmark (forget)
and the full MMLU (retain) after unlearning. ECO achieves accuracy close to
random guessing on all subsets of the WMDP benchmark (as desired), and has
zero decrease in accuracy on MMLU. Other baselines either struggle to forget
or incur substantial decrease in MMLU.

Baselines struggle to forget or collapse in utility.  We also observe that GA,
GD, KL, PO, and the prompting baseline exhibit minimal forgetting when the
forget set size is small (i.e., 1%). Meanwhile, some of them have a
substantial decrease and even a collapse in utility when the forget set is
larger (5% and 10%). Methods based on negative preference optimization [142]
demonstrate a noticeably stronger trade-off compared to other baselines,
especially with the NPO-RT. Nevertheless, the effectiveness of the NPO
variants varies across different models and forget set sizes, and the loss in
model utility is non-trivial. We present the full results on all metrics and
baselines in Table 13 and Table 14 in Section E.2.

###  4.3 Hazardous Knowledge Unlearning

Experimental setup  For both WMDP [70] and MMLU subset unlearning tasks [52],
we directly unlearn on pre-trained models. WMDP benchmark focuses on
unlearning knowledge in biology, chemistry, and cybersecurity. In MMLU subsets
unlearning, the goal is to unlearn three subjects and retain their closely
related counterparts: economics (econometrics), law (jurisprudence), and
physics (math), all requiring high-precision forgetting to resolve knowledge
entanglement. In line with [70], we assess all models based on their multiple-
choice accuracy. A successfully unlearned model should exhibit an accuracy
near random guessing (25% for four-option multiple-choice questions). We
employ the ECO-RN variant (random noise) as the corruption function for both
tasks. We only optimize corruption strength $\sigma$ for Zephyr-7B on a set of
100 synthetic questions and answers generated by GPT-4 to ensure the real
questions is not exposed during unlearning. We use the same corruption
parameter $\sigma$ for all other models. We compare our method against LLMU
[136], SCRUB [66], SSD [39], RMU [70], and a prompting baseline that instructs
the model not to answer questions within the domain correctly.

Method | Forget | Retain
---|---|---
Economics ($\downarrow$) |  Law ($\downarrow$) |  Physics ($\downarrow$) |  Econometrics ($\uparrow$) |  Jurisprudence ($\uparrow$) |  Math ($\uparrow$)
Original | 58.1 | 45.0 | 41.8 | 47.4 | 74.1 | 34.6
Prompting | 61.5 | 41.1 | 41.6 | 43.0 | 66.7 | 33.0
RMU | 27.3 | 27.8 | 27.0 | 41.2 | 37.0 | 29.2
ECO | 20.6 | 24.5 | 23.1 | 47.4 | 74.1 | 34.6
Random guess | 25.0 | 25.0 | 25.0 | 25.0 | 25.0 | 25.0

Table 2: Multiple-choice accuracy of Zephyr-7B after unlearning, on three MMLU
subsets and the corresponding retain sets. The prompting baseline hurts the
accuracy on the three forget subsets. While RMU reduces the forget set
accuracy to the level of random-guess, it incurs substantial performance
decrease on econometrics and jurisprudence while unlearning economics and law.
ECO achieves both perfect retaining and unlearning on all subsets.

ECO is domain- and model-agnostic.  In Tables 1 and 2, for all models on the
WMDP benchmark, ECO achieves accuracy close to random guessing for multiple-
choice questions while maintaining original MMLU performance. LLMU, SCRUB, and
SSD show limited forgetting performance across all subjects. Although RMU
successfully unlearns biology and cybersecurity, it retains accuracy in
chemistry, indicating that unlearning capability may vary across subjects or
the available data for unlearning. On Yi-34B-Chat and Mixtral-8x7B-Instruct,
RMU’s forgetting capability is not as effective as on Zephyr-7B, while ECO’s
performance remains consistent despite increased original performance on the
task.

ECO unlearns at high-precision.  On MMLU subsets unlearning, both ECO and RMU
successfully unlearn the three chosen subjects (Table 2). However, RMU’s
accuracy in econometrics and jurisprudence significantly decreases. This
implies that RMU might be sensitive to the entanglement of knowledge in
closely related subjects. In contrast, this entanglement poses no problem for
ECO’s prompt classifier due to its low false positive rate in retain domain.

ECO’s universal effectiveness.  To further validate the effectiveness of our
method across various models, we conducted experiments on 100 models ranging
from 0.5B to 236B on both the WMDP and MMLU subsets, using the same corruption
function and hyperparameters obtained on Zephyr-7B. Our results in Table 17
and Table 18 further demonstrate that our method is universally effective
without requiring hyperparameter tuning.

###  4.4 Copyrighted Content Unlearning

Experimental setup.  We select Harry Potter and the Sorcerer’s Stone [106] and
BBC News
articles555https://huggingface.co/datasets/RealTimeData/bbc_news_alltime [72]
as the copyrighted content material for unlearning and unlearn models fine-
tuned on the text corpus. For this task, our goal is to prevent the unlearned
model from generating passage with high similarity to the original text. For
both datasets, we verify that the models used cannot generated the original
passage and the generated text has low similarity to the original passage. We
first fine-tune a pre-trained model on the corresponding corpus and use it as
the model subject to unlearning, and use the original pre-trained checkpoint
as the retained model. We use the original passage as the reference text and
measure the text similarity between the reference and the text generated by
the unlearned model using four text similarity metrics outlined in Section
C.1.3, which we denote as the average similarity gap (ASG). Following [136],
we also compute the perplexity and unique toke ratio to assess the if the
generated text remains meaningful and diverse. We compare our method to
baselines in [87], SCRUB [66], and LLMU [136]. We present full experimental
details in Appendix C.

Dataset | Method |  ASG $(\downarrow)$ |  Utility $(\uparrow)$ |  PPL $(\downarrow)$ |  Unique Tok % $(\uparrow)$
---|---|---|---|---|---
BBC News | Original | 71.2 | 53.3 | 1 | 61
Retain | 0 | 59.2 | 3 | 28
2-6 | Fine-tune | 48.5 | 53.2 | 1.7 | 58.8
| GA | 12.4 | 33.1 | - | 0.8
| GD | 26.3 | 41.2 | - | 1.5
| KL | 6.5 | 48.9 | 1.8 | 28.4
| Mismatch | 3.9 | 53.5 | 20.7 | 65.7
| SCRUB | 12.7 | 33.9 | - | 2.3
| LLMU | 18.4 | 49.1 | 1.6 | 38
| ECO (Ours) | 1.5 | 53.3 | 1.5 | 50.4
HP Book | Original | 74.7 | 52.6 | 1.1 | 63.4
Retain | 0 | 59.2 | 2.3 | 18
2-6 | Fine-tune | 7.9 | 50.2 | 7.3 | 42.4
| GA | 23.4 | 32.2 | - | 3.4
| GD | 2.5 | 50.6 | 7.3 | 36.1
| KL | 1 | 47.4 | 1.5 | 22.8
| Mismatch | 8.2 | 50.4 | 6.9 | 40.3
| SCRUB | 7.1 | 32 | - | 2.2
| LLMU | 2.3 | 46.7 | 1.6 | 20
| ECO (Ours) | 2.1 | 52.6 | 1.2 | 51.1

Table 3: Comparison of our method and the baseline methods to the retained
model on two copyrighted content unlearning tasks. The results are obtained
from unlearning OLMo-7B [45] models fine-tuned on the relevant corpus. ECO
consistently maintains high similarity to the retained model (in average
similarity gap (ASG)) and generates meaningful and diverse outputs (reflected
by perplexity (PPL) and unique token ratio), while having no performance loss
on utility.

ECO maintains high similarity to the retained model.  In Table 3, ECO achieves
scores sufficiently close to those of the retained model in terms of generated
text similarity. On the general utility metric, our prompt classifiers
effectively distinguish copyrighted content from general domain queries with
no performance loss. KL minimization and LLMU are strong baselines in terms of
similarity gap and general utility, but diversity in the generated text
decreases after unlearning. Both gradient difference and random mismatch
reduce the issue of model collapse but still lead to notable performance
losses in general utility.

We further validate our findings on a total of 19 models in Section E.5 from
Table 19 to Table 56. We observe that some baselines cannot consistently
maintain strong results in either unlearning or general utility, while ECO
remains stable and consistently achieves a low similarity gap with the
retained model and unharmed utility.

##  5 Related Work

Unlearning for LLMs.  Most existing machine unlearning methods for LLMs follow
the traditional machine unlearning approaches [18, 96, 113] to minimize the
influence of the forget samples via gradient updates. The most straightforward
approach employ a mixture of forgetting and retaining objectives by performing
gradient ascent updates on the non-desirable sequences and regular gradient
descent on the desirable sequences [130, 136, 22, 135, 70, 142, 59]. Other
methods identify and modify a small fraction of the weights responsible for
the undesired behavior [132, 13, 55], or via weight arithmetic [141, 53, 97,
80]. The above optimization-based methods all require compute that scale with
the model size. Our method leaves the LLM subject to unlearning intact and
unlearns via steering the inputs to match the output distribution of a
retained model. Compute-wise, our unlearning method is independent of the
model size.

LLM guardrails.  Guardrailing, which accesses prompts before using them as
inputs to the model, has been widely applied on modern LLMs to prevent
adversaries with harmful incentives [105, 56, 138, 88, 68, 31, 131, 56, 44,
24]. Our work is most related to in-context unlearning [101] and a recent
guardrail baseline via prompting [122], both require no additional fine-tuning
to achieve unlearning to some extent. [101] leverages modern LLM’s ability in
in-context learning by prepending a small number of positive and negative
samples in the prompt to steer the model’s response based on those samples.
[122] guards the unlearning target via prompt injection, which inserts fixed
instructions in the prompt to the LLM. Both method can only be applied to
instruction-tuned models and rely on an LLM’s ability to follow instructions.
Prepending such instructions also leads to significant performance degradation
on regular tasks, as shown in [122].

Jailbreak via adversarial embeddings.  Prior work on LLM jailbreaking [148,
38, 73, 41, 100] have demonstrated the power of adversarially optimizing
toward a prompt that elicit a desired LLM response. In particular, [38] shows
that the attack can be simplified to learning perturbation vectors added to
the token embeddings, which eliminates the need to optimize over discrete
tokens. Our results on the behavior of the attacked models are similar to the
findings in [41], where inserting certain non-natural language token sequences
in the prompt could elicit refusal behavior or incorrect answers from an
instruction-tuned LLM. While jailbreak approaches can in theory be applied in
unlearning applications, they are prohibitorily expensive to run during [78]
and the additional requirement for specifying a sequence of desirable tokens.
Both requirements make them unsuitable for the task of unlearning.

##  6 Conclusion

In this paper, we presented Embedding-COrrupted (ECO) Prompts, a novel method
designed to address the dual challenges of knowledge entanglement and
unlearning efficiency in LLM unlearning. ECO uses a thresholded prompt
classifier to explicitly identify if a prompt is within the scope of the
unlearning target. By decoupling the unlearning procedure from the LLMs
themselves, ECO provide a scalable and efficient solution for unlearning in
LLMs, paving the way for responsible and safe AI deployment in real-world
applications.
Limitations and Broader Impacts We discuss the broader impacts and limitations
of our approach in detail in Appendix A and Appendix B, respectively.

## Acknowledgement

We are thankful for the computing resources provided by the Pacific Research
Platform’s Nautilus cluster, supported in part by National Science Foundation
(NSF) awards CNS-1730158, ACI-1540112, ACI-1541349, OAC-1826967, OAC-2112167,
CNS-2100237, CNS-2120019, the University of California Office of the
President, and the University of California San Diego’s California Institute
for Telecommunications and Information Technology/Qualcomm Institute, and
CENIC for the 100Gbps networks.

## References

  * [1] Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al.  Phi-3 technical report: A highly capable language model locally on your phone.  arXiv preprint arXiv:2404.14219, 2024.
  * [2] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.  Gpt-4 technical report.  arXiv preprint arXiv:2303.08774, 2023.
  * [3] AI@Meta.  Llama 3 model card.  2024\.
  * [4] Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo.  The falcon series of language models:towards open frontier models.  2023\.
  * [5] Malaikannan Sankarasubbu Ankit Pal.  Openbiollms: Advancing open-source large language models for healthcare and life sciences.  https://huggingface.co/aaditya/OpenBioLLM-Llama3-70B, 2024.
  * [6] Anthropic.  Introducing the next generation of claude.  https://www.anthropic.com/news/claude-3-family, Mar 2024.
  * [7] Viraat Aryabumi, John Dang, Dwarak Talupuru, Saurabh Dash, David Cairuz, Hangyu Lin, Bharat Venkitesh, Madeline Smith, Kelly Marchisio, Sebastian Ruder, Acyr Locatelli, Julia Kreutzer, Nick Frosst, Phil Blunsom, Marzieh Fadaee, Ahmet Üstün, and Sara Hooker.  Aya 23: Open weight releases to further multilingual progress, 2024.
  * [8] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al.  Qwen technical report.  arXiv preprint arXiv:2309.16609, 2023.
  * [9] Satanjeev Banerjee and Alon Lavie.  Meteor: An automatic metric for mt evaluation with improved correlation with human judgments.  In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72, 2005.
  * [10] Alvaro Bartolome, Jiwoo Hong, Noah Lee, Kashif Rasul, and Lewis Tunstall.  Zephyr 141b a39b.  https://huggingface.co/HuggingFaceH4/zephyr-orpo-141b-A35b-v0.1, 2024.
  * [11] Thomas Baumhauer, Pascal Schöttle, and Matthias Zeppelzauer.  Machine unlearning: Linear filtration for logit-based classifiers.  Machine Learning, 111(9):3203–3226, 2022.
  * [12] Marco Bellagente, Jonathan Tow, Dakota Mahan, Duy Phung, Maksym Zhuravinskyi, Reshinth Adithyan, James Baicoianu, Ben Brooks, Nathan Cooper, Ashish Datta, et al.  Stable lm 2 1.6 b technical report.  arXiv preprint arXiv:2402.17834, 2024.
  * [13] Nora Belrose, David Schneider-Joseph, Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman.  Leace: Perfect linear concept erasure in closed form.  arXiv preprint arXiv:2306.03819, 2023.
  * [14] Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al.  Pythia: A suite for analyzing large language models across training and scaling.  In International Conference on Machine Learning, pages 2397–2430. PMLR, 2023.
  * [15] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al.  Piqa: Reasoning about physical commonsense in natural language.  In Proceedings of the AAAI conference on artificial intelligence, pages 7432–7439, 2020.
  * [16] Jaydeep Borkar.  What can we learn from data leakage and unlearning for law?  arXiv preprint arXiv:2307.10476, 2023.
  * [17] Aleksandar Botev, Soham De, Samuel L Smith, Anushan Fernando, George-Cristian Muraru, Ruba Haroun, Leonard Berrada, Razvan Pascanu, Pier Giuseppe Sessa, Robert Dadashi, et al.  Recurrentgemma: Moving past transformers for efficient open language models.  arXiv preprint arXiv:2404.07839, 2024.
  * [18] Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot.  Machine unlearning.  In 2021 IEEE Symposium on Security and Privacy (SP), pages 141–159. IEEE, 2021.
  * [19] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  Advances in neural information processing systems, 33:1877–1901, 2020.
  * [20] Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al.  Internlm2 technical report.  arXiv preprint arXiv:2403.17297, 2024.
  * [21] Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer.  Membership inference attacks from first principles.  In 2022 IEEE Symposium on Security and Privacy (SP), pages 1897–1914. IEEE, 2022.
  * [22] Jiaao Chen and Diyi Yang.  Unlearn what you want to forget: Efficient unlearning for llms.  arXiv preprint arXiv:2310.20150, 2023.
  * [23] Min Chen, Weizhuo Gao, Gaoyang Liu, Kai Peng, and Chen Wang.  Boundary unlearning: Rapid forgetting of deep networks via shifting the decision boundary.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7766–7775, 2023.
  * [24] Zhixuan Chu, Yan Wang, Longfei Li, Zhibo Wang, Zhan Qin, and Kui Ren.  A causal explainable guardrails for large language models.  arXiv preprint arXiv:2405.04160, 2024.
  * [25] Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova.  Boolq: Exploring the surprising difficulty of natural yes/no questions.  arXiv preprint arXiv:1905.10044, 2019.
  * [26] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord.  Think you have solved question answering? try arc, the ai2 reasoning challenge.  arXiv preprint arXiv:1803.05457, 2018.
  * [27] CodeGemma Team, Ale Jakse Hartman, Andrea Hu, Christopher A. Choquette-Choo, Heri Zhao, Jane Fine, Jeffrey Hui, Jingyue Shen, Joe Kelley, Joshua Howland, Kshitij Bansal, Luke Vilnis, Mateo Wirth, Nam Nguyen, Paul Michel, Peter Choy, Pratik Joshi, Ravin Kumar, Sarmad Hashmi, Shubham Agrawal, Siqi Zuo, Tris Warkentin, and Zhitao et al. Gong.  Codegemma: Open code models based on gemma.  2024\.
  * [28] Cohere Team.  Command r: Retrieval-augmented generation at production scale, 2024.
  * [29] Databricks Team.  Introducing dbrx: A new state-of-the-art open llm, 2024.
  * [30] DeepSeek-AI.  Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model.  arXiv preprint arXiv:2405.04434, 2024.
  * [31] Yi Dong, Ronghui Mu, Gaojie Jin, Yi Qi, Jinwei Hu, Xingyu Zhao, Jie Meng, Wenjie Ruan, and Xiaowei Huang.  Building guardrails for large language models.  arXiv preprint arXiv:2402.01822, 2024.
  * [32] Michael Duan, Anshuman Suri, Niloofar Mireshghallah, Sewon Min, Weijia Shi, Luke Zettlemoyer, Yulia Tsvetkov, Yejin Choi, David Evans, and Hannaneh Hajishirzi.  Do membership inference attacks work on large language models?  arXiv preprint arXiv:2402.07841, 2024.
  * [33] Javid Ebrahimi, Hao Yang, and Wei Zhang.  How does adversarial fine-tuning benefit bert?  arXiv preprint arXiv:2108.13602, 2021.
  * [34] Ronen Eldan and Mark Russinovich.  Who’s harry potter? approximate unlearning in llms.  arXiv preprint arXiv:2310.02238, 2023.
  * [35] European Union.  General data protection regulation (gdpr).  https://gdpr-info.eu/, 2016.
  * [36] Chongyu Fan, Jiancheng Liu, Yihua Zhang, Dennis Wei, Eric Wong, and Sijia Liu.  Salun: Empowering machine unlearning via gradient-based weight saliency in both image classification and generation.  arXiv preprint arXiv:2310.12508, 2023.
  * [37] Richard Fang, Rohan Bindu, Akul Gupta, Qiusi Zhan, and Daniel Kang.  Llm agents can autonomously hack websites.  arXiv preprint arXiv:2402.06664, 2024.
  * [38] Stanislav Fort.  Scaling laws for adversarial attacks on language model activations.  arXiv preprint arXiv:2312.02780, 2023.
  * [39] Jack Foster, Stefan Schoepf, and Alexandra Brintrup.  Fast machine unlearning without retraining through selective synaptic dampening.  In Proceedings of the AAAI Conference on Artificial Intelligence, pages 12043–12051, 2024.
  * [40] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou.  A framework for few-shot language model evaluation, 12 2023.
  * [41] Jonas Geiping, Alex Stein, Manli Shu, Khalid Saifullah, Yuxin Wen, and Tom Goldstein.  Coercing llms to do and reveal (almost) anything.  arXiv preprint arXiv:2402.14020, 2024.
  * [42] Gemini.  Gemini: a family of highly capable multimodal models.  arXiv preprint arXiv:2312.11805, 2023.
  * [43] Aditya Golatkar, Alessandro Achille, and Stefano Soatto.  Eternal sunshine of the spotless net: Selective forgetting in deep networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9304–9312, 2020.
  * [44] Shubh Goyal, Medha Hira, Shubham Mishra, Sukriti Goyal, Arnav Goel, Niharika Dadu, DB Kirushikesh, Sameep Mehta, and Nishtha Madaan.  Llmguard: Guarding against unsafe llm behavior.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 23790–23792, 2024.
  * [45] Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, et al.  Olmo: Accelerating the science of language models.  arXiv preprint arXiv:2402.00838, 2024.
  * [46] Michael M. Grynbaum and Ryan Mac.  The times sues openai and microsoft over a.i. use of copyrighted work.  The New York Times, 12 2023.
  * [47] Jia-Chen Gu, Hao-Xiang Xu, Jun-Yu Ma, Pan Lu, Zhen-Hua Ling, Kai-Wei Chang, and Nanyun Peng.  Model editing can hurt general abilities of large language models.  arXiv preprint arXiv:2401.04700, 2024.
  * [48] Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y Wu, YK Li, et al.  Deepseek-coder: When the large language model meets programming–the rise of code intelligence.  arXiv preprint arXiv:2401.14196, 2024.
  * [49] Felix Hamborg, Norman Meuschke, Corinna Breitinger, and Bela Gipp.  news-please: A generic news crawler and extractor.  In Proceedings of the 15th International Symposium of Information Science, pages 218–223, March 2017.
  * [50] Bahareh Harandizadeh, Abel Salinas, and Fred Morstatter.  Risk and response in large language models: Evaluating key threat categories.  arXiv preprint arXiv:2403.14988, 2024.
  * [51] Jamie Hayes, Ilia Shumailov, Eleni Triantafillou, Amr Khalifa, and Nicolas Papernot.  Inexact unlearning needs more careful evaluations to avoid a false sense of privacy.  arXiv preprint arXiv:2403.01218, 2024.
  * [52] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.  Measuring massive multitask language understanding.  arXiv preprint arXiv:2009.03300, 2020.
  * [53] Xinshuo Hu, Dongfang Li, Baotian Hu, Zihao Zheng, Zhenyu Liu, and Min Zhang.  Separate the wheat from the chaff: Model deficiency unlearning via parameter-efficient module operation.  In Proceedings of the AAAI Conference on Artificial Intelligence, pages 18252–18260, 2024.
  * [54] James Y Huang, Wenxuan Zhou, Fei Wang, Fred Morstatter, Sheng Zhang, Hoifung Poon, and Muhao Chen.  Offset unlearning for large language models.  arXiv preprint arXiv:2404.11045, 2024.
  * [55] Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi.  Editing models with task arithmetic.  arXiv preprint arXiv:2212.04089, 2022.
  * [56] Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, et al.  Llama guard: Llm-based input-output safeguard for human-ai conversations.  arXiv preprint arXiv:2312.06674, 2023.
  * [57] Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo.  Knowledge unlearning for mitigating privacy risks in language models.  arXiv preprint arXiv:2210.01504, 2022.
  * [58] Jinghan Jia, Jiancheng Liu, Parikshit Ram, Yuguang Yao, Gaowen Liu, Yang Liu, Pranay Sharma, and Sijia Liu.  Model sparsification can simplify machine unlearning.  arXiv preprint arXiv:2304.04934, 2023.
  * [59] Jinghan Jia, Yihua Zhang, Yimeng Zhang, Jiancheng Liu, Bharat Runwal, James Diffenderfer, Bhavya Kailkhura, and Sijia Liu.  Soul: Unlocking the power of second-order optimization for llm unlearning.  arXiv preprint arXiv:2404.18239, 2024.
  * [60] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al.  Mistral 7b.  arXiv preprint arXiv:2310.06825, 2023.
  * [61] Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al.  Mixtral of experts.  arXiv preprint arXiv:2401.04088, 2024.
  * [62] Antonia Karamolegkou, Jiaang Li, Li Zhou, and Anders Søgaard.  Copyright violations and large language models.  arXiv preprint arXiv:2310.13771, 2023.
  * [63] Jinhwa Kim, Ali Derakhshan, and Ian G Harris.  Robust safety classifier for large language models: Adversarial prompt shield.  arXiv preprint arXiv:2311.00172, 2023.
  * [64] Siwon Kim, Sangdoo Yun, Hwaran Lee, Martin Gubri, Sungroh Yoon, and Seong Joon Oh.  Propile: Probing privacy leakage in large language models.  Advances in Neural Information Processing Systems, 36, 2024.
  * [65] Vinayshekhar Bannihatti Kumar, Rashmi Gangadharaiah, and Dan Roth.  Privacy adhering machine un-learning in nlp.  arXiv preprint arXiv:2212.09573, 2022.
  * [66] Meghdad Kurmanji, Peter Triantafillou, Jamie Hayes, and Eleni Triantafillou.  Towards unbounded machine unlearning.  Advances in Neural Information Processing Systems, 36, 2024.
  * [67] Yanis Labrak, Adrien Bazoge, Emmanuel Morin, Pierre-Antoine Gourraud, Mickael Rouvier, and Richard Dufour.  Biomistral: A collection of open-source pretrained large language models for medical domains, 2024.
  * [68] Alyssa Lees, Vinh Q Tran, Yi Tay, Jeffrey Sorensen, Jai Gupta, Donald Metzler, and Lucy Vasserman.  A new generation of perspective api: Efficient multilingual character-level transformers.  In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 3197–3207, 2022.
  * [69] Haodong Li, Gelei Deng, Yi Liu, Kailong Wang, Yuekang Li, Tianwei Zhang, Yang Liu, Guoai Xu, Guosheng Xu, and Haoyu Wang.  Digger: Detecting copyright content mis-usage in large language model training.  arXiv preprint arXiv:2401.00676, 2024.
  * [70] Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, et al.  The wmdp benchmark: Measuring and reducing malicious use with unlearning.  arXiv preprint arXiv:2403.03218, 2024.
  * [71] Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee.  Textbooks are all you need ii: phi-1.5 technical report.  arXiv preprint arXiv:2309.05463, 2023.
  * [72] Yucheng Li, Frank Guerin, and Chenghua Lin.  Latesteval: Addressing data contamination in language model evaluation through dynamic and time-sensitive test construction.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18600–18607, 2024.
  * [73] Zeyi Liao and Huan Sun.  Amplegcg: Learning a universal and transferable generative model of adversarial suffixes for jailbreaking both open and closed llms.  arXiv preprint arXiv:2404.07921, 2024.
  * [74] Chin-Yew Lin.  Rouge: A package for automatic evaluation of summaries.  In Text summarization branches out, pages 74–81, 2004.
  * [75] Stephanie Lin, Jacob Hilton, and Owain Evans.  Truthfulqa: Measuring how models mimic human falsehoods.  arXiv preprint arXiv:2109.07958, 2021.
  * [76] Jiancheng Liu, Parikshit Ram, Yuguang Yao, Gaowen Liu, Yang Liu, PRANAY SHARMA, Sijia Liu, et al.  Model sparsity can simplify machine unlearning.  Advances in Neural Information Processing Systems, 36, 2024.
  * [77] Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Xiaojun Xu, Yuguang Yao, Hang Li, Kush R Varshney, et al.  Rethinking machine unlearning for large language models.  arXiv preprint arXiv:2402.08787, 2024.
  * [78] Xiaodong Liu, Hao Cheng, Pengcheng He, Weizhu Chen, Yu Wang, Hoifung Poon, and Jianfeng Gao.  Adversarial training for large neural language models.  arXiv preprint arXiv:2004.08994, 2020.
  * [79] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach.  arXiv preprint arXiv:1907.11692, 2019.
  * [80] Zheyuan Liu, Guangyao Dou, Zhaoxuan Tan, Yijun Tian, and Meng Jiang.  Towards safer large language models through machine unlearning.  arXiv preprint arXiv:2402.10058, 2024.
  * [81] Zihan Liu, Wei Ping, Rajarshi Roy, Peng Xu, Chankyu Lee, Mohammad Shoeybi, and Bryan Catanzaro.  Chatqa: Surpassing gpt-4 on conversational qa and rag.  arXiv preprint arXiv:2401.10225, 2024.
  * [82] Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul, Zhuang Li, Wen-Ding Li, Megan Risdal, Jia Li, Jian Zhu, Terry Yue Zhuo, Evgenii Zheltonozhskii, Nii Osae Osae Dade, Wenhao Yu, Lucas Krauß, Naman Jain, Yixuan Su, Xuanli He, Manan Dey, Edoardo Abati, Yekun Chai, Niklas Muennighoff, Xiangru Tang, Muhtasham Oblokulov, Christopher Akiki, Marc Marone, Chenghao Mou, Mayank Mishra, Alex Gu, Binyuan Hui, Tri Dao, Armel Zebaze, Olivier Dehaene, Nicolas Patry, Canwen Xu, Julian McAuley, Han Hu, Torsten Scholak, Sebastien Paquet, Jennifer Robinson, Carolyn Jane Anderson, Nicolas Chapados, Mostofa Patwary, Nima Tajbakhsh, Yacine Jernite, Carlos Muñoz Ferrandis, Lingming Zhang, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries.  Starcoder 2 and the stack v2: The next generation, 2024.
  * [83] Nils Lukas, Ahmed Salem, Robert Sim, Shruti Tople, Lukas Wutschitz, and Santiago Zanella-Béguelin.  Analyzing leakage of personally identifiable information in language models.  In 2023 IEEE Symposium on Security and Privacy (SP), pages 346–363. IEEE, 2023.
  * [84] Yizhen Luo, Jiahuan Zhang, Siqi Fan, Kai Yang, Yushuai Wu, Mu Qiao, and Zaiqing Nie.  Biomedgpt: Open multimodal generative pre-trained transformer for biomedicine.  arXiv preprint arXiv:2308.09442, 2023.
  * [85] Aengus Lynch, Phillip Guo, Aidan Ewart, Stephen Casper, and Dylan Hadfield-Menell.  Eight methods to evaluate robust unlearning in llms.  arXiv preprint arXiv:2402.16835, 2024.
  * [86] Dakota Mahan, Ryan Carlow, Louis Castricato, Nathan Cooper, and Christian Laforte.  Stable beluga models.
  * [87] Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C Lipton, and J Zico Kolter.  Tofu: A task of fictitious unlearning for llms.  arXiv preprint arXiv:2401.06121, 2024.
  * [88] Todor Markov, Chong Zhang, Sandhini Agarwal, Florentine Eloundou Nekoul, Theodore Lee, Steven Adler, Angela Jiang, and Lilian Weng.  A holistic approach to undesired content detection in the real world.  In Proceedings of the AAAI Conference on Artificial Intelligence, pages 15009–15018, 2023.
  * [89] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal.  Can a suit of armor conduct electricity? a new dataset for open book question answering.  arXiv preprint arXiv:1809.02789, 2018.
  * [90] Niloofar Mireshghallah, Hyunwoo Kim, Xuhui Zhou, Yulia Tsvetkov, Maarten Sap, Reza Shokri, and Yejin Choi.  Can llms keep a secret? testing privacy implications of language models via contextual integrity theory.  arXiv preprint arXiv:2310.17884, 2023.
  * [91] Mayank Mishra, Matt Stallone, Gaoyuan Zhang, Yikang Shen, Aditya Prasad, Adriana Meza Soria, Michele Merler, Parameswaran Selvam, Saptha Surendran, Shivdeep Singh, et al.  Granite code models: A family of open foundation models for code intelligence.  arXiv preprint arXiv:2405.04324, 2024.
  * [92] Marius Mosbach, Maksym Andriushchenko, and Dietrich Klakow.  On the stability of fine-tuning bert: Misconceptions, explanations, and strong baselines.  arXiv preprint arXiv:2006.04884, 2020.
  * [93] Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah.  Orca: Progressive learning from complex explanation traces of gpt-4.  arXiv preprint arXiv:2306.02707, 2023.
  * [94] Andrei Muresanu, Anvith Thudi, Michael R Zhang, and Nicolas Papernot.  Unlearnable algorithms for in-context learning.  arXiv preprint arXiv:2402.00751, 2024.
  * [95] Seth Neel and Peter Chang.  Privacy issues in large language models: A survey.  arXiv preprint arXiv:2312.06717, 2023.
  * [96] Thanh Tam Nguyen, Thanh Trung Huynh, Phi Le Nguyen, Alan Wee-Chung Liew, Hongzhi Yin, and Quoc Viet Hung Nguyen.  A survey of machine unlearning.  arXiv preprint arXiv:2209.02299, 2022.
  * [97] Shiwen Ni, Dingwei Chen, Chengming Li, Xiping Hu, Ruifeng Xu, and Min Yang.  Forgetting before learning: Utilizing parametric arithmetic for knowledge updating in large language models.  arXiv preprint arXiv:2311.08011, 2023.
  * [98] OpenAI.  Introducing chatgpt, Nov 2022.
  * [99] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.  Bleu: a method for automatic evaluation of machine translation.  In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318, 2002.
  * [100] Anselm Paulus, Arman Zharmagambetov, Chuan Guo, Brandon Amos, and Yuandong Tian.  Advprompter: Fast adaptive adversarial prompting for llms.  arXiv preprint arXiv:2404.16873, 2024.
  * [101] Martin Pawelczyk, Seth Neel, and Himabindu Lakkaraju.  In-context unlearning: Language models as few shot unlearners.  arXiv preprint arXiv:2310.07579, 2023.
  * [102] Nikhil Pinnaparaju, Reshinth Adithyan, Duy Phung, Jonathan Tow, James Baicoianu, Ashish Datta, Maksym Zhuravinskyi, Dakota Mahan, Marco Bellagente, Carlos Riquelme, et al.  Stable code technical report.  arXiv preprint arXiv:2404.01226, 2024.
  * [103] Matt Post.  A call for clarity in reporting bleu scores.  arXiv preprint arXiv:1804.08771, 2018.
  * [104] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn.  Direct preference optimization: Your language model is secretly a reward model. arxiv 2023.  arXiv preprint arXiv:2305.18290, 2023.
  * [105] Traian Rebedea, Razvan Dinu, Makesh Sreedhar, Christopher Parisien, and Jonathan Cohen.  Nemo guardrails: A toolkit for controllable and safe llm applications with programmable rails.  arXiv preprint arXiv:2310.10501, 2023.
  * [106] J.K. Rowling.  Harry Potter and the Sorcerer’s Stone.  Scholastic, New York, 1997.
  * [107] Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al.  Code llama: Open foundation models for code.  arXiv preprint arXiv:2308.12950, 2023.
  * [108] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi.  Winogrande: An adversarial winograd schema challenge at scale.  Communications of the ACM, 64(9):99–106, 2021.
  * [109] Jonas B Sandbrink.  Artificial intelligence and biological misuse: Differentiating risks of language models and biological design tools.  arXiv preprint arXiv:2306.13952, 2023.
  * [110] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.  arXiv preprint arXiv:1910.01108, 2019.
  * [111] Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi.  Socialiqa: Commonsense reasoning about social interactions.  arXiv preprint arXiv:1904.09728, 2019.
  * [112] Nazanin Mohammadi Sepahvand, Vincent Dumoulin, Eleni Triantafillou, and Gintare Karolina Dziugaite.  Data selection for transfer unlearning. arxiv 2024.  arXiv preprint arXiv:2405.10425, 2024.
  * [113] Thanveer Shaik, Xiaohui Tao, Haoran Xie, Lin Li, Xiaofeng Zhu, and Qing Li.  Exploring the landscape of machine unlearning: A survey and taxonomy.  arXiv preprint arXiv:2305.06360, 2023.
  * [114] Yikang Shen, Zhen Guo, Tianle Cai, and Zengyi Qin.  Jetmoe: Reaching llama2 performance with 0.1 m dollars.  arXiv preprint arXiv:2404.07413, 2024.
  * [115] Toby Shevlane.  Structured access: an emerging paradigm for safe ai deployment.  arXiv preprint arXiv:2201.05159, 2022.
  * [116] Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer.  Detecting pretraining data from large language models.  arXiv preprint arXiv:2310.16789, 2023.
  * [117] James C Spall.  Multivariate stochastic approximation using a simultaneous perturbation gradient approximation.  IEEE transactions on automatic control, 37(3):332–341, 1992.
  * [118] James C Spall.  Introduction to stochastic search and optimization: estimation, simulation, and control.  John Wiley & Sons, 2005.
  * [119] Robin Staab, Mark Vero, Mislav Balunović, and Martin Vechev.  Beyond memorization: Violating privacy via inference with large language models.  arXiv preprint arXiv:2310.07298, 2023.
  * [120] Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant.  Commonsenseqa: A question answering challenge targeting commonsense knowledge.  arXiv preprint arXiv:1811.00937, 2018.
  * [121] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al.  Gemma: Open models based on gemini research and technology.  arXiv preprint arXiv:2403.08295, 2024.
  * [122] Pratiksha Thaker, Yash Maurya, and Virginia Smith.  Guardrail baselines for unlearning in llms.  arXiv preprint arXiv:2403.03329, 2024.
  * [123] Anvith Thudi, Gabriel Deza, Varun Chandrasekaran, and Nicolas Papernot.  Unrolling sgd: Understanding factors influencing machine unlearning.  In 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P), pages 303–319. IEEE, 2022.
  * [124] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  arXiv preprint arXiv:2307.09288, 2023.
  * [125] Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al.  Zephyr: Direct distillation of lm alignment.  arXiv preprint arXiv:2310.16944, 2023.
  * [126] Lewis Tunstall and Philipp Schmid.  Zephyr 7b gemma.  https://huggingface.co/HuggingFaceH4/zephyr-7b-gemma-v0.1, 2024\.
  * [127] Vladimir Vovk, Alexander Gammerman, and Glenn Shafer.  Algorithmic learning in a random world, volume 29.  Springer, 2005.
  * [128] Ben Wang and Aran Komatsuzaki.  GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.  https://github.com/kingoflolz/mesh-transformer-jax, May 2021.
  * [129] Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, and Yang Liu.  Openchat: Advancing open-source language models with mixed-quality data.  arXiv preprint arXiv:2309.11235, 2023.
  * [130] Lingzhi Wang, Tong Chen, Wei Yuan, Xingshan Zeng, Kam-Fai Wong, and Hongzhi Yin.  Kga: A general machine unlearning framework based on knowledge gap alignment.  arXiv preprint arXiv:2305.06535, 2023.
  * [131] Yanchen Wang and Lisa Singh.  Adding guardrails to advanced chatbots.  arXiv preprint arXiv:2306.07500, 2023.
  * [132] Xinwei Wu, Junzhuo Li, Minghui Xu, Weilong Dong, Shuangzhi Wu, Chao Bian, and Deyi Xiong.  Depn: Detecting and editing privacy neurons in pretrained language models.  arXiv preprint arXiv:2310.20138, 2023.
  * [133] Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang.  Wizardlm: Empowering large language models to follow complex instructions.  arXiv preprint arXiv:2304.12244, 2023.
  * [134] Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al.  Baichuan 2: Open large-scale language models.  arXiv preprint arXiv:2309.10305, 2023.
  * [135] Jin Yao, Eli Chien, Minxin Du, Xinyao Niu, Tianhao Wang, Zezhou Cheng, and Xiang Yue.  Machine unlearning of pre-trained large language models.  arXiv preprint arXiv:2402.15159, 2024.
  * [136] Yuanshun Yao, Xiaojun Xu, and Yang Liu.  Large language model unlearning.  arXiv preprint arXiv:2310.10683, 2023.
  * [137] Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al.  Yi: Open foundation models by 01. ai.  arXiv preprint arXiv:2403.04652, 2024.
  * [138] Zhuowen Yuan, Zidi Xiong, Yi Zeng, Ning Yu, Ruoxi Jia, Dawn Song, and Bo Li.  Rigorllm: Resilient guardrails for large language models against undesired content.  arXiv preprint arXiv:2403.13031, 2024.
  * [139] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi.  Hellaswag: Can a machine really finish your sentence?  arXiv preprint arXiv:1905.07830, 2019.
  * [140] Di Zhang, Wei Liu, Qian Tan, Jingdan Chen, Hang Yan, Yuliang Yan, Jiatong Li, Weiran Huang, Xiangyu Yue, Dongzhan Zhou, Shufei Zhang, Mao Su, Hansen Zhong, Yuqiang Li, and Wanli Ouyang.  Chemllm: A chemical large language model, 2024.
  * [141] Jinghan Zhang, Shiqi Chen, Junteng Liu, and Junxian He.  Composing parameter-efficient modules with arithmetic operations.  arXiv preprint arXiv:2306.14870, 2023.
  * [142] Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei.  Negative preference optimization: From catastrophic collapse to effective unlearning.  arXiv preprint arXiv:2404.05868, 2024.
  * [143] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al.  Opt: Open pre-trained transformer language models.  arXiv preprint arXiv:2205.01068, 2022.
  * [144] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi.  Bertscore: Evaluating text generation with bert.  arXiv preprint arXiv:1904.09675, 2019.
  * [145] Zihan Zhao, Da Ma, Lu Chen, Liangtai Sun, Zihao Li, Hongshen Xu, Zichen Zhu, Su Zhu, Shuai Fan, Guodong Shen, Xin Chen, and Kai Yu.  Chemdfm: Dialogue foundation model for chemistry, 2024.
  * [146] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al.  Judging llm-as-a-judge with mt-bench and chatbot arena.  Advances in Neural Information Processing Systems, 36, 2024.
  * [147] Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, Karthik Ganesan, Wei-Lin Chiang, Jian Zhang, and Jiantao Jiao.  Starling-7b: Improving llm helpfulness & harmlessness with rlaif, November 2023.
  * [148] Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson.  Universal and transferable adversarial attacks on aligned language models.  arXiv preprint arXiv:2307.15043, 2023.

##  Appendix A Broader Impact

The proposed method, Embedding-COrrupted (ECO) Prompts, offers a novel
framework for unlearning in large language models (LLMs), addressing the
crucial challenge of removing sensitive or harmful knowledge while maintaining
model integrity. As LLMs become more embedded in various applications,
ensuring that they can unlearn specific information is paramount for
compliance with data privacy regulations such as GDPR and for mitigating
potential misuse. However, our work also has broader implications that merit
careful consideration.

Firstly, the unlearning capability, while beneficial for privacy and safety,
could be misused to selectively remove critical information, potentially
leading to misinformation or biased outputs. For instance, model providers
might exploit this technology to erase inconvenient facts from models deployed
in public-facing applications, thereby manipulating the information accessible
to users. To mitigate such risks, robust auditing mechanisms and transparency
in the application of unlearning techniques are essential. Secondly, while ECO
Prompts are designed to safeguard against specific threats such as entity
leaking and hazardous knowledge dissemination, their effectiveness depends on
the accuracy of the initial threat identification. Incorrect or incomplete
identification could either fail to remove all relevant knowledge or
inadvertently degrade the model’s performance on non-sensitive tasks.
Continuous monitoring and refinement of the classifier used for identifying
unlearning targets, alongside comprehensive evaluation protocols, are
necessary to minimize these potential harms.

##  Appendix B Limitations

One limitation of ECO is that it supports unlearning only for models with API
access, as it relies on the classifier to identify the unlearning target and
the corruption function to achieve unlearning. If an adversary has open-weight
access to a model, they could circumvent the unlearning state by bypassing the
classifier.

Secondly, as described in Section 2.1, our approach does not address the
threat posed by motivated adversaries who may attempt to compromise the
classifier or the LLM itself. To counter such threats, practitioners might
consider training the prompt classifier adversarially [78, 63, 33] to enhance
its robustness against attacks, even if the attacker is aware of the
classifier’s presence and architecture.

Third, the prompt classifier’s context window is typically limited, examining
only the first (or last) $K$ tokens by default. An attacker aware of this
limitation could manipulate the prompt by injecting neutral text at both the
beginning and the end to bypass the classifier. However, this vulnerability
can be mitigated by implementing a sliding window technique: if the prompt’s
length exceeds the context window, the prompt should be considered positive as
long as one of the text spans is predicted as positive. These limitations
underscore the need for future work to improve the classifier’s mechanism,
potentially integrating it directly into the LLM itself.

##  Appendix C Detailed Experimental Setup

In this section, we introduce our experimental setup, including a detailed
description of all evaluation metrics (Section C.1), preparation of LLM
subject to unlearning (Section C.2), training and evaluation of the prompt
classifier (Section C.3), and formulations of all the baseline methods
(Section C.4).

###  C.1 Evaluation Metrics

####  C.1.1 TOFU

We employ the original evaluation metrics designed by the authors of the TOFU
dataset [87].

##### Answer probability

For every single instance in the retain set or the forget set, we compute the
normalized conditional probability $P(a\mid q)^{1/|a|}$ on the LLM subject to
unlearning, where $q$ and $a$ correspond to the question and answer, and $|a|$
represents the number of tokens in the answer. For the real authors and world
facts subsets, the dataset provides a set of five answers
$\\{a_{0},\tilde{a}_{1},\tilde{a}_{2},\tilde{a}_{3},\tilde{a}_{4}\\}$, which
consists of a single correct answer $a_{0}$ and four other perturbed answers
that are incorrect. In this case, we compute the ratio $P(a_{0}\mid
q)^{1/|a_{0}|}/\sum_{i=1}^{4}P(\tilde{a}_{i}\mid q)^{1/|\tilde{a}_{i}|}$.

##### Truth ratio

The truth ratio is computed as the geometric mean666We veified with the
authors of the TOFU paper that the numerator should be a geometric mean
instead of arithmetic mean, even though their paper still shows the arithmetic
mean, at the time of writing this paper. This can also be confirmed by their
implementation, which uses geometric mean:
https://github.com/locuslab/tofu/blob/36811054f2376560c8d6629667059f3000e5603c/evaluate_util.py#L59.
of multiple perturbed (incorrect) answers’
($\mathcal{A}=\\{\tilde{a}_{1},\tilde{a}_{2},...\\}$) probabilities over the
normalized conditional probability of the paraphrased answer $\hat{a}$.

$R_{\text{truth}}=\frac{\left(\prod_{i=1}^{|\mathcal{A}|}P(\tilde{a}\mid
q)^{|1/\tilde{a}_{i}|}\right)^{1/|\mathcal{A}|}}{P(\hat{a}\mid
q)^{1/|\hat{a}|}}$

For the real authors and world fact subsets, the original answer $a$ is used
in the denominator as no paraphrased answer is available.

##### ROUGE-L

For all subsets of TOFU, we compute the ROUGE-L recall score [74] between the
ground truth responses (provided by the dataset) and the text generated by the
model after unlearning.

##### Model utility

The model utility is aggregated as a harmonic mean over nine numbers: the
answer probability, truth ratio, and ROUGE recall scores from each of the
retain, real authors, and world facts subsets. A higher model utility is
always preferred.

##### Forget quality

The forget quality is computed as the p-value of performing a Kolmogorov-
Smirnov (KS) test using two distributions: the truth ratio of the retained
model on the forget set, and the truth ratio of the unlearned model on the
forget set. A higher p-value under the KS test indicates failure to reject the
null hypothesis that the distributions of truth ratio from the retained and
the unlearned models are the same, which is a sign of indistinguishability
between the retained model’s behavior and the unlearned model’s behavior.

####  C.1.2 WMDP and MMLU

##### Multiple-choice accuracy

For both the WDMP [70] and MMLU subsets [52] unlearning, we employ multiple-
choice accuracy as the primary evaluation metric. The underlying assumption is
that a model unlearned on the target subject should demonstrate random-
guessing accuracy on the task. Since both WMDP and MMLU consist of four-option
multiple-choice questions, an accuracy close to 0.25 indicates successful
unlearning. For each question, we adhere to the approach outlined by [70] and
utilize the template provided in LABEL:lst:wmdp_template in a zero-shot
manner. To derive the answer predicted by the LLM, we extract the logit scores
corresponding to tokens [A, B, C, D] from the logit of the last token in the
input sequence. The option with the highest logit score is deemed as the
predicted answer.

⬇

The following are multiple choice questions (with answers) about {subject}.

{question}

A. {choice_A}

B. {choice_B}

C. {choice_C}

D. {choice_D}

Answer:

Listing 1: The formatting template for WMDP and MMLU multiple-choice questions
used in both the classifier training and the main LLM for prediction.

##### Probing

We also incorporate a probing evaluation, as done in [70], which trains a
four-way linear probe on half of the data points from the biology, chemistry,
and cybersecurity subsets. Specifically, a linear classifier is trained to
predict the correct answer of the multiple-choice question based on the
unlearned model’s output logits. The trained linear probe is then used to make
predictions on the other half of the data points to infer the correct labels
given the output logits. Successful unlearning should result in random-chance
accuracy for the linear probe, which is 0.25 in our case.

####  C.1.3 Harry Potter Book and BBC News Articles

We employ four text similarity metrics outlined below. For each metric, we use
the original text (from the copyrighted material) as the reference and compute
the similarity between the reference and the text generated by the LLM. A
retained model that has never been trained on the reference text should have
low similarity scores on all metrics, and a successfully unlearned model
should have scores similar to that of the retained model. For both datasets,
we evaluate similarity based on the first 256 tokens generated. This aligns
with our fine-tuning setup in Section C.2.

##### ROUGE-L

We utilize the ROUGE-L algorithm as described in Section C.1.1. ROUGE-L’s
recall score denotes the proportion of the longest common subsequence in the
reference text that appears in the generated text by the unlearned model.
Essentially, it gauges the frequency at which the unlearned model can generate
long text spans that exist in the copyrighted content.

##### SacreBLEU

[136] employs the BLEU score [99], which is predicated on $n$-gram precision,
to determine if the copyrighted content has been inadvertently disclosed,
using a predefined threshold. We adopt SacreBLEU [103], which standardizes
tokenization to mitigate variability in preprocessing. SacreBLEU assesses the
overlap of $n$-grams between the generated and reference texts, subsequently
calculating the number of matching $n$-grams as a precision score.

##### BERTScore

BERTScore [144] employs contextual embeddings of tokens from both the
reference and generated texts, performing greedy matching based on pairwise
similarity of all token pairs. We utilize the F1 score, as recommended by the
original authors, and employ the DistilBERT [110] checkpoint to obtain these
contextual embeddings.

##### METEOR

We also employ METEOR [9], which incorporates unigram precision, unigram
recall, and word order to provide a more nuanced similarity measure than BLEU
and ROUGE-L.

##### Average similarity gap (ASG)

We incorporate an aggregated metric, the average gap [76, 36], as the average
absolute difference over the four similarity metrics above, computed between
the retained model and the unlearned model. The average gap measures how
similar an unlearned model’s outputs are to the retained model’s outputs, and
a smaller gap is more desirable.

##### Perplexity (PPL) and unique token ratio

Following [136], we use the perplexity score and the unique token ratio
measured on the generated text to assess fluency and diversity of the
generated text. The perplexity is calculated by a reference model that has
been fine-tuned on the target copyrighted content material. A sufficiently low
perplexity indicates that the generated text might still be meaningful. The
unique token is calculated using the tokenizer

####  C.1.4 Why Not Membership Inference Attacks (MIAs)?

In this paper, we follow most prior work on LLM unlearning, which generally do
not use membership inference attack (MIA) methods to evaluate the
effectiveness of unlearning for LLMs [57, 65, 53, 22, 34, 87, 136, 135, 80,
70, 142, 54, 59].

We do not consider MIA methods to evaluate our models for three major reasons.
First, state-of-the-art MIAs require training multiple (up to hundreds) shadow
models [21] on subsets of the entire training set, which are not feasible for
LLMs’ setting, which requires access to the pre-training data or fine-tuning a
large number of models on subsets of of the fine-tuning data. MIAs without
training shadow models have been demonstrated to overestimate the
effectiveness of unlearning [51] due to the non-uniform difficulty of
learning/unlearning each sample.

Second, evidence suggests that existing MIAs for LLMs, even the state-of-the-
art one [116], generally barely perform better than random guessing due to
both training on large pre-training data size for a small number of iterations
and the fuzzy boundary between members and non-members [32].

Third, as stated in Section 2.2, we do not consider the privacy aspect of
unlearning in this work, and our threat model does not include privacy risks.
Knowing if a single sample is a membership also does not significantly
increase the risk in our threat model.

Additionally, performing such MIAs typically require at least the model
internal states [32] (e.g., activations), which is not in the scope of our
threat model (i.e, only text output and logits).

In fact, the forget quality metric described in Section C.1.1 and the probing
evaluation in Section C.1.2 aligns with the goal of MIAs. The forget quality
assesses if the forget set distributions on the unlearning model and the
retained models can be distinguished. The linear probe tries to infer the
correct answers from the model output, assuming that the the accuracy of the
linear probe on a retained model is random-guessing level. Obtaining the same
accuracy might imply indistinguishability.

###  C.2 Preparing LLMs for Unlearning

In this subsection, we describe the setup for preparing the LLMs subject to
unlearning for each dataset.

##### TOFU

We use the original code777https://github.com/locuslab/tofu provided alongside
the TOFU dataset [87] for fine-tuning to ensure consistency. Following their
experimental setup, we fine-tune two models, Phi-1.5 [71] and Llama-2-7B-Chat
[124], on the entire TOFU dataset to obtain the model to be subjected to
unlearning. Following the retain/forget splits provided in the dataset, we
fine-tune each model on each of the three different splits, 99%, 95%, and 90%
of the full dataset, excluding the forget data, to obtain the retained models.
These three splits also correspond to unlearning 1%, 5%, and 10% of the
samples, respectively. We employ the same hyperparameters as provided in both
the paper and the accompanying code. Both models are trained with a batch size
of 4, accumulating gradients for 4 steps on 2 NVIDIA A6000 GPUs, resulting in
an effective batch size of 32, with a learning rate of 1e-5 for
Llama-2-7B-Chat and 2e-5 for Phi-1.5. For the negative preference optimization
[142] baselines, we follow a similar procedure and uses the code provided by
the original authors888https://github.com/licong-lin/negative-preference-
optimization.

##### WMDP and MMLU subsets

The knowledge assessment of all multiple-choice questions in WMDP [70] and
MMLU [52] subsets is performed directly on the pre-trained models (or models
unlearned from the pre-trained checkpoints for unlearning evaluation).
Therefore, we do not fine-tune models based on the multiple-choice questions
for the the WMDP unlearning task.

##### Copyrighted content

For the copyrighted content unlearning task, we first verify that all the
considered LLMs cannot generate the original corpus. For HP Book, while some
parts of the text corpus could potentially be included during pretraining, we
see little sign of generating the text spans verbatim for all models we
considered in the copyrighted content unlearning experiments. This is also
reflect in the low similarity scores from tables in Section E.5. For BBC News
articles, we only consider articles published in February 2024, which is
beyond the knowledge cutoff of most models considered. We fine-tune them on
the copyrighted content corpus to ensure that they are able to generated the
original passage. For the HP book, we split the text into chunks of up to 256
tokens (based on the tokenization scheme used for the specific model). For BBC
News articles, we concatenate the news title with the news content with a
single space in the middle. The title of the news article is used as the
prompt for generation. To ensure that our models can indeed generate the
copyrighted content, we fine-tune all models on the two text corpora for 5
epochs, using a batch size of 4 and a learning rate of 2e-5 on two NVIDIA A100
GPUs.

Dataset | $\bm{D}^{\textrm{Train}}_{f}$ | $\bm{D}^{\textrm{Train}}_{r}$ | $\bm{D}^{\textrm{Test}}_{f}$ | $\bm{D}^{\textrm{Test}}_{r}$ | $\bm{D}_{\textrm{g}}$
---|---|---|---|---|---
TOFU (1%) | 40 | 3,960 | - | 217 | 41,297
TOFU (5%) | 200 | 3,800 | - | 217 | 41,297
TOFU (10%) | 400 | 3,600 | - | 217 | 41,297
WMDP (All) | 397 | 1,802 | 3,571 | 1,803 | 41,297
$\text{WMDP}_{\text{Synthetic}}$ (All) | 300 | 1,342 | 3,968 | 1,343 | 41,297
MMLU (Economics) | 10 | 275 | 628 | 13,414 | 25,724
MMLU (Physics) | 15 | 270 | 488 | 13,554 | 25,724
MMLU (Law) | 15 | 275 | 1,655 | 12,387 | 25,724
HP Book | 6,819 | 36,209 | - | 36,209 | 41,297
BBC News | 2,017 | 8,949 | - | 9,514 | 41,297

Table 4: The statistics of the dataset (splits) used to train the prompt
classifiers. $D_{f}$ and $D_{r}$ denote the forget and retain sets. $D_{g}$
(outlined in Table 8) refers to the general set for evaluating general
utility.

###  C.3 Prompt Classifiers

In this subsection, we describe how the dataset for the prompt classifiers are
prepared and the setup and hyperparameters used to train the prompt
classifiers. We include the dataset split statistics in Table 4. We also
report the performance of three prompt classifiers in Tables 5, 6 and 7,
corresponding to the original classifier, simple-thresholding classifier, and
conformal prediction classfiier.

Dataset | $\textbf{FNR}_{\bm{D^{\textrm{Train}}_{f}}}$ | $\textbf{FPR}_{\bm{D^{\textrm{Train}}_{r}}}$ | $\textbf{FNR}_{\bm{D^{\textrm{Test}}_{f}}}$ | $\textbf{FPR}_{\bm{D^{\textrm{Test}}_{r}}}$ | $\textbf{FPR}_{\bm{D_{g}}}$
---|---|---|---|---|---
TOFU (1%) | 0.0 | 0.0 | - | 0.0 | 0.0
TOFU (5%) | 0.0 | 0.0 | - | 0.0 | 0.0
TOFU (10%) | 0.0 | 0.0 | - | 0.0 | 0.0
WMDP (All) | 0.0 | 0.0 | 0.0 | 0.0 | 0.0
$\text{WMDP}_{\text{Synthetic}}$ (All) | 0.0 | 0.0 | 0.0 | 0.0 | 0.0047
$\text{WMDP}_{\text{o.o.d}}$ (All) | 0.0 | 0.0 | 0.2683 | 0.0839 | 0.1845
MMLU (Economics) | 0.0 | 0.0 | 0.0 | 0.0 | 0.002
MMLU (Physics) | 0.0 | 0.0 | 0.0 | 0.0 | 0.001
MMLU (Law) | 0.0 | 0.0 | 0.0 | 0.0 | 0.001
HP Book | 0.0021 | 0.0001 | - | 0.0071 | 0.0
BBC News | 0.0 | 0.0 | - | 0.0168 | 0.0

Table 5: The false negative rate (FNR) and false positive rate (FPR) of the
prompt classifiers without thresholding.

Dataset | $\textbf{FNR}_{\bm{D^{\textrm{Train}}_{f}}}$ | $\textbf{FPR}_{\bm{D^{\textrm{Train}}_{r}}}$ | $\textbf{FNR}_{\bm{D^{\textrm{Test}}_{f}}}$ | $\textbf{FPR}_{\bm{D^{\textrm{Test}}_{r}}}$ | $\textbf{FPR}_{\bm{D_{g}}}$
---|---|---|---|---|---
TOFU (1%) | 0.0 | 0.0 | - | 0.0 | 0.0
TOFU (5%) | 0.0 | 0.0 | - | 0.0 | 0.0
TOFU (10%) | 0.0 | 0.0 | - | 0.0 | 0.0
WMDP (All) | 0.0 | 0.0 | 0.0 | 0.0 | 0.0
$\text{WMDP}_{\text{Synthetic}}$ (All) | 0.0 | 0.0 | 0.0 | 0.0 | 0.0004
$\text{WMDP}_{\text{o.o.d}}$ (All) | 0.0 | 0.0 | 0.2721 | 0.016 | 0.003
MMLU (Economics) | 0.0 | 0.0 | 0.0 | 0.0 | 0.0
MMLU (Physics) | 0.0 | 0.0 | 0.0 | 0.0 | 0.0
MMLU (Law) | 0.0 | 0.0 | 0.0 | 0.0 | 0.0004
HP Book | 0.001 | 0.0001 | - | 0.0072 | 0.0
BBC News | 0.0 | 0.0 | - | 0.0168 | 0.0

Table 6: The false negative rate (FNR) and false positive rate (FPR) of the
prompt classifiers on the corresponding data subsets. If the FNR of
$D^{\textrm{Test}}_{f}$ is not reported, it means that the corresponding
unlearning target does not require generalization outside the scope of the
forget set. The $D_{g}$ set contains out-of-distribution prompts from eleven
NLP benchmarks listed in Table 8. The error rate above is calculated using the
calibrated decision threshold $\tau$.

Dataset | $\textbf{FNR}_{\bm{D^{\textrm{Train}}_{f}}}$ | $\textbf{FPR}_{\bm{D^{\textrm{Train}}_{r}}}$ | $\textbf{FNR}_{\bm{D^{\textrm{Test}}_{f}}}$ | $\textbf{FPR}_{\bm{D^{\textrm{Test}}_{r}}}$ | $\textbf{FPR}_{\bm{D_{g}}}$
---|---|---|---|---|---
TOFU (1%) | 0.0 | 0.0 | - | 0.0 | 0.0
TOFU (5%) | 0.0 | 0.0 | - | 0.0 | 0.0
TOFU (10%) | 0.0 | 0.0 | - | 0.0 | 0.0
WMDP (All) | 0.0 | 0.0 | 0.0 | 0.0006 | 0.0004
$\text{WMDP}_{\text{Synthetic}}$ (All) | 0.0 | 0.0 | 0.0 | 0.0006 | 0.0047
$\text{WMDP}_{\text{o.o.d}}$ (All) | 0.0 | 0.0 | 0.1267 | 0.1016 | 0.0834
MMLU (Economics) | 0.0 | 0.0 | 0.0 | 0.0 | 0.002
MMLU (Physics) | 0.0 | 0.0 | 0.0 | 0.0 | 0.001
MMLU (Law) | 0.0 | 0.0 | 0.0 | 0.0 | 0.001
HP Book | 0.0021 | 0.0001 | - | 0.0072 | 0.0002
BBC News | 0.0 | 0.0 | - | 0.0169 | 0.0

Table 7: The false negative rate (FNR) and false positive rate (FPR) of the
prompt classifiers on the corresponding data subsets with conformal
prediction. For uncertain predictions with a prediction size of two, we behave
conservatively and treat them as positive. The performance is slightly worse
than the simple thresholding in Table 6, due to the cost of counting all
uncertain predictions as positive samples.

####  C.3.1 Prompt Classifiers’ Training Datasets

##### TOFU

We strictly follow the original split of the forget and retain sets in the
TOFU dataset [87] to train the classifiers. To access the false positive
predictions, we use the real authors and world facts splits to evaluate the
classifier after hyperparameter tuning is completed. Here, we do not use a
test forget set for the entities, following the practice in the original
paper.

##### WMDP

We train a single classifier to classify multiple choice questions in all
three subjects of the WMDP datasets (i.e., biology, chemistry, and
cybersecurity). We format the question strictly following the original
evaluation [70] using the Language Model Evaluation Harness [40] style in
LABEL:lst:wmdp_template, where the {subject}, {question}, and {choice_#}
fields are replaced by the actual text for each multiple-choice question. This
template is also used as the prompt template for the main LLM to make
predictions. For negative samples, we use a combination of the auxiliary
training set and the development set of MMLU [52]. We do so because the
questions in the auxiliary training set do not have subjects, which might
result in a shortcut picked up by the classifier. We use the sampels in
development set (with subjects) to mitigate that shortcut. Note that MMLU has
its own validation set, so we believe that using the development set (with
only 285 samples) is a reasonable choice.

Different from TOFU and two other copyrighted content dataset, we require the
classifier trained on the WMDP questions to generalize to unseen questions in
relevant domains, while not flagging relevant questions in similar domains
(e.g., virology, high school chemistry, and computer security in MMLU) as
positive. Additionally, the method, RMU, proposed along with the WMDP dataset
does not require access to the forget questions. To avoid fully relying on the
WMDP questions to train the classifier, we first restrict ourselves to access
only 10% of the WMDP questions, selected randomly before training and
development. We also subsample a fixed set of 3K samples (out of 99.8K) in the
auxiliary training set of MMLU to train and test the false positive rate of
the prompt classifier. Note that, the RMU method proposed in [70] does not
require access to any of the questions. Therefore, we also incorporate a
setting where we train a prompt classifier on only synthetic data. The
synthetic dataset has only 300 questions (100 for each of the biology,
chemistry, and cybersecurity subsets), and all multiple-choice questions are
generated by GPT-4 [2] to resemble the style and difficulty of real WMDP
questions. In this way, we align with RMU’s setup, which does not access to
real questions during model development. For a detailed split, please see
Table 4. The perfomrnace of both prompt classifiers are reported in Table 6.
We see that the WMDP prompt classifier trained on only synthetic data have
almost the same performance as one trained on 10% of the real questions.

In Tables 5, 6 and 7, we also show an o.o.d. setup, where we assume we only
have 100 biology questions, 100 cybersecurity questions, but only 2 chemistry
questions. In this scenario, the false negative rates of the original
classifier and the simple-threshold classifier are high due to the lack of
training samples for chemistry. In Table 7, we demonstrate that conformal
prediction can alleviate this issue via including more uncertain samples as
negative, which reduces the false negative rate by approximately 14%.

##### MMLU subsets

We train a separate classifier for each of the unlearning category: economics,
physics, and law. This is because for each task, we hope to unlearn only the
selected category and retain the rest of the categories as defined in [70],
which makes training a single prompt classifier infeasible experiment-wise.
Similar to the WMDP setting, we only restrict ourselves to have access to the
development set of MMLU, which only contains 10-15 samples for the forget
target. As shown in Table 6, all classifiers achieve perfect predictions,
except that our MMLU law classifier has 10 false positives: 2 in HellaSwag and
8 in BoolQ. After manual inspection of the false positive samples, we observed
that all of them are directly related to law and closely resemble the style of
questions in the MMLU’s law category. We believe this is due to the
generalization capacity of the classifier, which is difficult to avoid given
the high similarity in those questions. In fact, after giving the sample
questions to humans, they cannot tell the difference between the MMLU law
questions and HellaSwag/BoolQ law questions. Therefore, seeing that the false
positive rate is still reasonably low (i.e., 0.0004), we consider this type of
error to be tolerable.

##### HP Book

Since the goal is to prevent users from obtaining the copyrighted content by
training data extraction, we purchased the Harry Potter and the Sorcerer’s
Stone [106] ebook and extracted the entire corpus to train our HP Book prompt
classifier. We split the entire book into sentences using spaCy
sentencizer999https://spacy.io/api/sentencizer, and only select sentences with
more than ten characters. The sentences with ten or fewer characters are
mostly neutral sentences, line escape, whitespaces, or other punctuation. All
remaining sentences are considered as positive sentences for our prompt
classifier. For the negative samples, we employ the BookMIA
dataset101010https://huggingface.co/datasets/swj0419/BookMIA [116], which
contains more than 9K text snippets from a wide range of real books. The
BookMIA dataset contains some snipepts from Harry Potter and the Sorcerer’s
Stone [106], which are all removed before training. In this task, we do not
require our prompt classifier to generalize, so we do not use a test forget
set. We subsample and split the retain set (i.e., BookMIA) into two equal-
sized sets and use them to train/test the performance of the prompt
classifier. In Table 6, while our classifier incorrectly predicts several
samples, by manual examination, we find that those samples are mostly noise
(i.e., neutral sentences).

##### BBC News

We use BBC News
articles111111https://huggingface.co/datasets/RealTimeData/bbc_news_alltime
published in February 2024 as the positive samples and 9K+ news articles in
the English language from the CC-News dataset [49] as the negative samples to
train the prompt classifier. To prevent shortcut being learned, we format both
datasets in the same way and remove the “ \- BBC ###” suffix in the title of
the BBC News dataset. The prompt classifier is only trained on the title of
the news article. To prevent more sophisticated extraction attacks, one can
train the prompt classifier using news articles split into sentences, similar
to the HP Book dataset. However, due to the long lengths of the full news
articles, we only consider classification of the news title.

Dataset | Size
---|---
MMLU [52] | 15,573
ARC-Easy [26] | 2,376
ARC-Challenge [26] | 1,172
CommonsenseQA [120] | 1,221
HellaSwag [139] | 10,042
OpenBookQA [89] | 500
TruthfulQA [75] | 817
Winogrande [108] | 2,534
PIQA [15] | 1,838
SocialIQA [111] | 1,954
BoolQ [25] | 3,270
Total | 41,297

Table 8: A list of common LLM benchmark datasets. We use these datasets
collectively as $D_{\textrm{g}}$, the out-of-distribution general set, to
evaluate the general utility of the unlearned models beyond the forgetting and
retain distributions.

##### A comprehensive evaluation of general utility

Most prior work only evaluate the retain ability of the unlearned LLM using
the retain set associated with the unlearning task. However, the results
reported on the retain set might not fully reflect the general utility in the
real-world setting. This is because the retain set, while being disjoint with
the forget set, might still have a similar distribution to the forget set in
some aspect. Therefore, instead of solely relying on the regular retain set,
we consider a large set of out-of-distribution samples to measure general
utility. In the general set, we eleven common LLM benchmarks listed in Table
8: MMLU [52], ARC-Easy [26], ARC-Challenge [26], OpenBookQA [89], HellaSwag
[139], Winogrande [108], TruthfulQA [75], CommonsenseQA [120], PIQA [15],
SocialIQA [111], and BoolQ [25]. They amount to 41,297 samples in total. We
evaluate all prompt classifiers (from all datasets mentioned above) on the
general set after tuning the parameters of the prompt classifiers. For all
datasets, we use the test set if the labels are publicly available; otherwise,
we use the validation set. For TruthfulQA, we use the MC1 subset for
evaluation.

####  C.3.2 Training A prompt classifier

For all prompt classifiers used for prompt content detection, we choose
RoBERTa-base [79] as the base model for fine-tuning. The hyperparmaeters are
selected following a prior work that improves stability during training [92].
Since in most cases the number of positive samples (the forget samples) is
much less than the negative samples, we reweight the class-wise losses using
the inverse frequency. Once the optimal number of epochs is determined, we
fine-tune the model again on the combined training and validation set and use
it as the final prompt classifier used in inference-time unlearning. We also
evaluate all prompt classifiers on the general set outlined in Table 8.

We report the performance of the original prompt classifier, classifier with
simple thresholding, and classifier with conformal prediction in Tables 5, 6
and 7. We demonstrate that in all settings considered, our best prompt
classifiers have achieved satisfying performance reflected by the low false
negative rate on the forget set and low false positive rate on the retain set.
On the general set (i.e., the suite of NLP benchmarks), most prompt
classifiers have zero false positive predictions, suggesting that the
performance on the main LLM on samples irrelevant to the forget set is
unlikely to be affected.

###  C.4 Baseline Methods

In this section, we formulate all baseline methods used throughout the paper.

####  C.4.1 Prompting Baseline

For all prompting baselines, we adapt the prompt style from [85]. Below, we
detail the prompts used in the TOFU dataset as well as those for the WMDP
benchmark and MMLU subsets. For the TOFU prompt, we list the author names
subject to forgetting. For the WMDP and MMLU prompts, we replace the {subject}
field with the specific subject to unlearn (e.g., biology).

Note that one could potentially craft more sophisticated prompts to achieve
improved unlearning and utility. However, as this is not the focus of the
current study, we leave such developments to future work.

⬇

You are a model that knows absolutely nothing about the following author
names:

- {name_1}

- {name_2}

- ...

- {name_n}

Please ensure that your responses to anything related to the above names are
incorrect. For everything else, you can provide the correct answers.

{question}

Listing 2: The prompt used in the prompting baseline for the TOFU dataset.

⬇

You are a model that knows absolutely nothing about {subject}. Please ensure
that your responses to anything related to {subject} are incorrect. For
everything else, you can provide the correct answers.

{question}

Listing 3: The prompt used in the prompting baseline for the WMDP benchmark
and MMLU subsets.

####  C.4.2 Optimization-Based Unlearning Baselines

For all optimization-based baselines below, we fix the number of epochs as 5
and tune the learning rate of {5e-5, 2e-5, 1e-5}.

##### Fine-tuning, gradient ascent (GA), and gradient difference (GD)

Fine-tuning, gradient ascent, and gradient difference are simple baselines
commonly used in traditional machine unlearning settings [23, 58, 36, 66], and
has been introduced as simple baseline methods in [87]. Fine-tuning only
involves performing gradient descent on $D_{r}$, while gradient ascent
performs gradient descent on $D_{f}$ in the opposite direction. Gradient
difference combines fine-tuning and gradient ascent by compute the sum of the
two loss terms.

$\displaystyle L_{\textrm{Fine-tune}}$
$\displaystyle=\frac{1}{|D_{r}|}\sum_{\mathbf{x}\in
D_{r}}\mathcal{L}(\mathbf{x};\bm{\theta})$ $\displaystyle L_{\textrm{GA}}$
$\displaystyle=-\frac{1}{|D_{f}|}\sum_{\mathbf{x}\in
D_{f}}\mathcal{L}(\mathbf{x};\bm{\theta})$ $\displaystyle L_{\textrm{GD}}$
$\displaystyle=\frac{1}{|D_{r}|}\sum_{\mathbf{x}\in
D_{r}}\mathcal{L}(\mathbf{x};\bm{\theta})-\frac{1}{|D_{f}|}\sum_{\mathbf{x}\in
D_{f}}\mathcal{L}(\mathbf{x};\bm{\theta})$

##### KL minimization (KL)

The KL minimization is adopted from [87] and involves a gradient ascent term
for forgetting as well. It also minimizes the KL distance on $D_{r}$ between
the current model and the original model $\bm{\theta}_{o}$. The KL
minimization term aims to keep the model’s current output distribution on the
retained set close to its pre-unlearning distribution on the retain samples.

$L_{\textrm{KL}}=L_{\textrm{GA}}+\frac{1}{|D_{r}|}\sum_{\mathbf{x}\in
D_{r}}\operatorname{KL}(h(\mathbf{x};\bm{\theta}_{o})\|h(\mathbf{x};\bm{\theta}))$

##### Preference optimization (PO)

The preference optimization (PO) is different from the traditional sense of
direct preference optimization [104] in that it only combines the fine-tuning
loss on $D_{r}$ and a term that learns to say “I don’t know” for prompts in
$D_{f}$ [87]. Below, $D_{\textrm{idk}}$ is an augmented forget dataset with
the answer “I don’t know” following the prompt.

$L_{\textrm{PO}}=L_{\textrm{Fine-
tune}}+\frac{1}{|D_{\textrm{idk}}|}\sum_{\mathbf{\mathbf{x}}\in
D_{\textrm{idk}}}\mathcal{L}(\mathbf{\mathbf{x}};\bm{\theta})$

##### Negative preference optimization (NPO) [142]

NPO incorporates only the lossing response term in direct preference
optimization (DPO) [104], which only penalizes the prompt-response pairs in
$D_{f}$. In the formulation below, $\beta$ represents the inverse-temperature.
It also has two extended versions involving either the KL term and the fine-
tuning term on $D_{r}$ to preserve utility.

$\displaystyle L_{\textrm{NPO}}$
$\displaystyle=\frac{2}{\beta}\frac{1}{|D_{f}|}\left[\log\left(1+\left(\frac{h(y\mid\mathbf{x};\bm{\theta})}{h(y\mid\mathbf{x};\bm{\theta})}\right)^{\beta}\right)\right]$
$\displaystyle L_{\textrm{NPO-KL}}$
$\displaystyle=L_{\textrm{NPO}}+L_{\textrm{KL}}$ $\displaystyle
L_{\textrm{NPO-RT}}$ $\displaystyle=L_{\textrm{NPO}}+L_{\textrm{Fine-tune}}$

##### Mismatch

Mismatch has the same objective to preference optimization above, except it
involves constructing a random combination of text sequences
$\mathbf{\mathbf{x}}_{\textrm{rand}}$. Here, the second term in mismatch is
the same as the second term in LLMU [136].

$L_{\textrm{Mismatch}}=L_{\textrm{Fine-
tune}}+\frac{1}{|D_{\textrm{rand}}|}\sum_{\mathbf{x}\in
D_{\textrm{rand}}}\mathcal{L}(\mathbf{x};\bm{\theta})$

##### SCRUB [66]

SCRUB was originally proposed as a machine unlearning algorithm for
classification tasks but was adopted as a baseline for LLM unlearning by [70].
SCRUB uses a combined objective that 1) minimizes the KL divergence between
the original model and the unlearned model on $D_{r}$, 2) maximizes the same
KL divergence on $D_{f}$, and 3) uses a regular gradient descent term on
$D_{r}$ to retain performance. However, instead of optimizing three objectives
at the same time, it interleaves a min-step (i.e., the first and the second
terms) to retain and a max-step (i.e., the third term) to unlearn across
epochs. In our experiments, we perform three epochs of min-steps and two
epochs of max-steps. In addition to tuning the learning rate, we fix $\gamma$
and tune $\alpha=\\{0.0001,0.001,0.01,0.1\\}$.

$\displaystyle L_{\textrm{SCRUB}}=$
$\displaystyle\frac{\alpha}{|D_{r}|}\sum_{\mathbf{x}\in
D_{r}}\operatorname{KL}(h(\mathbf{x};\bm{\theta}_{o})\|h(\mathbf{x};\bm{\theta}))$
$\displaystyle+\frac{\gamma}{|D_{r}|}\sum_{\mathbf{x}\in
D_{r}}\mathcal{L}(\mathbf{x};\bm{\theta})$
$\displaystyle-\frac{1}{|D_{f}|}\sum_{\mathbf{x}\in
D_{f}}\operatorname{KL}(h(\mathbf{x};\bm{\theta}_{o})\|h(\mathbf{x};\bm{\theta}))$

##### LLMU [136]

LLMU combines the gradient descent term with two additional terms to learn 1)
random completions from $D_{\textrm{rand}}$ (constructed using prompts from
$D_{f}$) to facilitate unlearn and 2) $D_{\textrm{normal}}$ to preserve
performance. We use books with similar styles as $D_{\textrm{normal}}$ in our
experiments and construct $D_{\textrm{rand}}$ using randomly sampled text
sequences from $D_{\textrm{normal}}$. We fix $\epsilon_{2}$ and $\epsilon_{3}$
at 1 and tune $\epsilon_{1}$ with values {0.1, 0.5, 1, 2}, following the
original paper.

$\displaystyle L_{\textrm{LLMU}}=$
$\displaystyle-\frac{\epsilon_{1}}{|D_{f}|}\sum_{\mathbf{x}\in
D_{f}}\mathcal{L}(\mathbf{x};\bm{\theta})$
$\displaystyle+\frac{\epsilon_{2}}{|D_{\textrm{rand}}|}\sum_{\mathbf{x}\in
D_{\textrm{rand}}}\mathcal{L}(\mathbf{x};\bm{\theta})$
$\displaystyle+\frac{\epsilon_{3}}{|D_{\textrm{normal}}|}\sum_{\mathbf{x}\in
D_{\textrm{normal}}}\operatorname{KL}(h(\mathbf{x};\bm{\theta}_{o})\|h(\mathbf{x};\bm{\theta}))$

##### Selective Synaptic Dampening (SSD)

We adopted the SSD implementation in [70], which is an adaptation of the
original SSD and uses the log-perplexity as a criteria on the forget set and
the retain set. Given the diagonal of the Fisher information matrix $[]_{D}$
computed offline on $D$, the dampened weight is computed via

$\theta^{\prime}=\min\left(\frac{\lambda[]_{D,i}}{[]_{D_{f},i}}\theta_{i},\theta_{i}\right)$

for each weight $\theta_{i}$ if $[]_{D_{f},i}>\alpha[]_{D,i}$, where $\alpha$
is the dampening constant. We follow [70]’s hyperparmaeters of thresholds
$[0.1,0.25,0.5,1,2.5,5]$ and dampening constants [1e-5, 1e-4, 1e-3, 1e-2,
1e-1, 1].

##### Representation misdirection for unlearning (RMU) [70]

Given a function $M_{\ell}(\mathbf{x};\bm{\theta})$ that returns the hidden
representation of $\bm{\theta}$ at a layer $\ell$, and a fixed random unit
vector $\mathbf{u}$ sampled uniformly from $[0,1)$, the RMU objective is
defined as follows:

$L_{\textrm{RMU}}=\frac{1}{|D_{f}|}\sum_{\mathbf{x}\in
D_{f}}\|M_{\ell}(\mathbf{x};\bm{\theta})-c\cdot\mathbf{u}\|^{2}_{2}+\frac{\alpha}{|D_{r}|}\sum_{\mathbf{x}\in
D_{r}}\|M_{\ell}(\mathbf{x};\bm{\theta})-M_{\ell}(\mathbf{x};\bm{\theta}_{o})\|^{2}_{2}$

This is similar to gradient difference with the exception it pushes the hidden
representation at layer $\ell$ toward a random vector and minimizes the
squared difference between the unlearned model and the original model. Since
the authors provided their trained model checkpoints121212Zephyr-7B
https://huggingface.co/cais/Zephyr_RMU,
Yi-34B-Chat https://huggingface.co/cais/Yi-34B-Chat_RMU,
Mixtral-8x7B https://huggingface.co/cais/Mixtral-8x7B-Instruct_RMU. and the
experimental setups are identical, we directly used their checkpoints for
evaluation.

###  C.5 A Toy Example of Conformal Prediction

Suppose we picked $\alpha=0.05$ and obtained $\hat{q}=0.93$ as the
$\lceil(n+1)\cdot 0.95\rceil/n$ empirical quantile from the non-conformity
scores $\\{s_{1},s_{2},...,s_{n}\\}$ from $D_{\textrm{cal}}$. Suppose, for a
test sample $\mathbf{x}$, our classifier $C$ gives conditional probabilities
$p_{C}(y=0\mid\mathbf{x})=0.82$ and $p_{C}(y=1\mid\mathbf{x})=0.18$. The
prediction set $\mathcal{C}_{0.05}$ of $\mathbf{x}$ is formed by

$\mathcal{C}_{0.05}\left(\mathbf{x}\right)=\left\\{y\in\\{0,1\\}:1-p_{C}(y\mid\mathbf{x})\leq
0.93\right\\}.$

Given the conditional probabilities, we have

$S(\mathbf{x},0)=1-p_{C}(y=0\mid\mathbf{x})=1-0.82=0.18,$
$S(\mathbf{x},1)=1-p_{C}(y=1\mid\mathbf{x})=1-0.18=0.82.$

Thus, both scores are below $\hat{q}=0.93$, so the prediction set is

$\mathcal{C}_{0.05}\left(\mathbf{x}\right)=\\{0,1\\}.$

###  C.6 Usage of Compute Resources

For all experiments conducted in the paper, we conduct experiments on a node
with 8 NVIDIA A100 or NVIDIA A6000 GPUs, but at most three of each are
required for a single experiment. The longest experiments on models with over
100B parameters typically take 2-5 days to complete.

##  Appendix D Ablation Experiments

In this section, we include ablation experiments to support claims and
findings in the main paper.

###  D.1 Prompt Classifier Thresholding

In Tables 6, 5 and 7, we show the performance of classifiers with three
different thresholding schemes (described in Section 3.2): no thresholding,
simple-thresholding, and conformal prediction.

We see that prompt classifiers without thresholding already perform well on
most datasets. An exception is the out-of-distribution WMDP, where we have few
samples for questions from one threat category. Specifically, it has a high
false negative rate for forget samples and a non-trivial false positive rate
on retain samples. Increasing the threshold in simple-threshold classifiers
reduces the false positive rate to near-perfect, but the false negative rate
also increases. By employing conformal prediction, we successfully reduce the
false negative rate by more than 50%. In practice, we recommend selecting the
thresholding method based on its performance on a relatively large held-out
set to balance missing forget or retain samples. Depending on the risk posed
by the unlearning target, conformal prediction might be a better choice in
high-risk scenarios to reduce false positive predictions.

###  D.2 Corruption Function Variants

In this section, we examine variants of the corruption functions used in the
main paper. Previously, we primarily employed Gaussian noise, where the
standard deviation represented the corruption strength. In Section 4, we also
used zeroing-out of the top-$k$ entries in each embedding vector. We include
experiments on sign flipping, reversing the order of the embedding vector, and
shuffling the embedding vector. For sign flipping, random noise, and zero-out,
we select either the first $N$ entries or the top-$k$ entries. We also
experiment with selecting random $N$ entries for random noise corruption.

In the experiments below, we do not tune the corruption strength for each
corruption function but use the same corruption strength for similar
functions. For example, for all random noise corruption, we use the same
corrupted strength as the one picked in the main paper (Figure 2). In Table 9,
we show that sign flipping and zero-out have consistently high forget quality,
while the randomized corruption function might require extra strength tuning,
especially for larger forget sets. This suggests that tuning the corruption
strength based on the criteria used is important to achieve unlearning.

|  | Forget Quality
---|---|---
Model | Method | Forget 1% | Forget 5% | Forget 10%
Phi-1.5 | Original | 0.0143 | 0.0000 | 0.0000
Retain | 1.0000 | 1.0000 | 1.0000
2-5 | Flip Sign First $N$ | 0.9900 | 0.9238 | 0.8635
| Flip Sign Top-$k$ | 0.9900 | 0.9238 | 0.9674
| Rand Noise First $N$ | 0.9900 | 0.7934 | 0.1810
| Rand Noise Rand $N$ | 0.9900 | 0.7934 | 0.0013
| Rand Noise Top-$k$ | 0.9900 | 0.3935 | 0.1314
| Zero Out First $N$ | 0.9900 | 0.3935 | 0.8134
| Zero Out Top-$k$ | 0.9188 | 0.8655 | 0.9674
Llama-2-7B-Chat | Original | 0.0030 | 0.0000 | 0.0000
Retain | 1.0000 | 1.0000 | 1.0000
2-5 | Flip Sign First $N$ | 0.9188 | 0.9647 | 0.9674
| Flip Sign Top-$k$ | 0.4046 | 0.5453 | 0.5812
| Rand Noise First $N$ | 0.7659 | 0.0396 | 0.0079
| Rand Noise Rand $N$ | 0.9188 | 0.2705 | 0.0006
| Rand Noise Top-$k$ | 0.9188 | 0.0118 | 0.0000
| Zero Out First $N$ | 0.9188 | 0.8655 | 0.9939
| Zero Out Top-$k$ | 0.9188 | 0.8655 | 0.9844

Table 9: Ablating the corruption function for the TOFU dataset on Phi-1.5 and
Llama-2-7B-Chat.

In Tables 10 and 11, we present the results of eight variants of the
corruptions on BBC News and HP Book unlearning. We see that most corruption
functions used can achieve an ASG score below 5 while maintaining low
perplexity and high unique token ratio.

Method |  ASG ($\downarrow$) |  PPL ($\downarrow$) |  Unique Token (%) ($\uparrow$) | BERTScore | METROR | ROUGE | SacreBLEU
---|---|---|---|---|---|---|---
Original | 71.2 | 1 | 61 | 99.7 | 98.7 | 98.7 | 98.3
Retain | 0 | 3 | 28 | 73.2 | 18 | 16.2 | 3.2
Flip Sign First $N$ | 1.6 | 1.5 | 51.8 | 70.2 | 17.9 | 13.2 | 2.9
Flip Sign Top-$k$ | 2.1 | 1.6 | 48.7 | 69.8 | 16.9 | 12.9 | 2.6
Rand Noise Rand $N$ | 1.5 | 1.6 | 50.5 | 70.2 | 18.3 | 13.4 | 3.1
Rand Noise Top-$k$ | 2 | 2.8 | 44.4 | 69.9 | 17.1 | 13 | 2.6
Reverse Order | 3.1 | 1.7 | 47.5 | 69.8 | 21.3 | 16.9 | 8.1
Shuffle | 10.2 | 1.6 | 49.5 | 76.2 | 31.2 | 26.5 | 17.4
Zero Out First $N$ | 3.8 | 1.5 | 50 | 72.7 | 23.6 | 19.2 | 9.2
Zero Out Top-$k$ | 1.8 | 1.5 | 50.6 | 70 | 17.5 | 13.1 | 2.9

Table 10: Ablating the corruption function for the BBC News unlearning task on
OLMo-7B.

Method |  ASG ($\downarrow$) |  PPL ($\downarrow$) |  Unique Token (%) ($\uparrow$) | BERTScore | METROR | ROUGE | SacreBLEU
---|---|---|---|---|---|---|---
Original | 74.7 | 1.1 | 63.4 | 99.4 | 98.3 | 98.3 | 97.9
Retain | 0 | 2.3 | 18.0 | 68.5 | 14.4 | 10.4 | 2
Flip Sign First $N$ | 1 | 2.7 | 35.5 | 67.9 | 16.6 | 9.3 | 2.3
Flip Sign Top-$k$ | 1 | 2.7 | 36.1 | 68.6 | 16.8 | 9.3 | 2.3
Rand Noise Rand $N$ | 2.4 | 1.3 | 50.4 | 68.8 | 21.1 | 11.5 | 3.2
Rand Noise Top-$k$ | 1.2 | 2.4 | 39.3 | 69 | 17.5 | 9.9 | 2.6
Reverse Order | 1.4 | 1.7 | 47.9 | 69.6 | 18.5 | 10.8 | 1.9
Shuffle | 2.5 | 1.4 | 45.8 | 69.5 | 20.3 | 11.9 | 3.6
Zero Out First $N$ | 7.9 | 1.3 | 52 | 73.8 | 26.6 | 17.2 | 9
Zero Out Top-$k$ | 3.4 | 1.3 | 51.9 | 72 | 22 | 11.7 | 3.2

Table 11: Ablating the corruption function for the HP Book unlearning task on
OLMo-7B.

We see that the selection of corruption function and corruption strength is
not as important on unlearning BBC News and HP Book as it is on the TOFU
dataset, based on text similarity metrics. This suggests that the forget
quality metric is a more rigorous measure than mere text similarity, as it
evaluates the distributional similarity between the outputs of an unlearned
model and a retrained model. Therefore, in practice, we recommend always
searching for the best corruption function and corruption strength based on
the available criteria.

##  Appendix E Additional Experiments

In this section, we include additional experiments to support claims and
findings in the main paper.

###  E.1 Time Delays of Prompt Content Detection

Task | Dataset | w/o Classifier(s) | w/ Classifier(s) | Abs. Increase | Percent. Increase (%)
---|---|---|---|---|---
Generation | TOFU (Retain90) | 67 | 70 | 3 | 4.22
TOFU (Forget10) | 79 | 164 | 85 | 107.55
HP Book | 2882 | 2902 | 21 | 0.71
BBC News | 2887 | 2909 | 21 | 0.73
Logits | WMDP (Biology) | 28 | 31 | 4 | 12.59
WMDP (Chemistry) | 17 | 23 | 5 | 30.64
WMDP (Cybersecurity) | 132 | 142 | 10 | 7.80
MMLU (Economics) | 17 | 21 | 3 | 19.19
MMLU (Physics) | 17 | 21 | 4 | 22.22
MMLU (Law) | 37 | 44 | 7 | 19.73

Table 12: Per example time delay (milliseconds) due to the extra prompt
content detection step. The last two columns represent the absolute and
percentage increase in time.

In Table 12, we report the per-example time delay (in milliseconds) introduced
by running the prompt classifier during inference of the main LLM. The time in
w/o Classifier(s), w/ Classifier(s), and Abs. Increase columns are measured on
a machine with a single NVIDIA A100 using a Llama-2-7B(-Chat) with a batch
size of 4. Note that the prompt content detection step only depends on the
incoming prompt and is agnostic to the LLM subject to unlearning, so the
statistics in the table is constant with respect to any other LLMs given fixed
prompts. The largest delay of 85 ms is from TOFU (Forget10), which involves
extra inference time by an additional token classifier based on BERT to
identify tokens that are names in the prompt. In most cases, the extra delay
is no more than 21 ms.

###  E.2 TOFU

The results in this section are supporting evidence for Section 4.2 and Figure
2 in the main paper. We include the full results of Llama-2-7B-Chat and
Phi-1.5 in Table 13 and Table 14, respectively. These results encompass all
metrics described in Section C.1.1: conditional probability of the answer
given the prompts, truth ratio (TR), ROUGE-L scores, model utility, and forget
quality. We report all scores for the retain set, forget set, real authors,
and world facts across all forget set sizes. In Figure 2, we plot the model
utility and forget quality as shown in the last two columns of each table.

Besides the random noise and zero-out variants of ECO, we also include a sign-
flip variant. This variant flips the signs of all entries in the embedding
vectors of the selected tokens. In both tables, Table 13 and Table 14, the
sign-flip variant exhibits low forget quality across all splits. This outcome
likely stems from its higher (better) truth ratio compared to the retained
model, leading to substantially different distributions from those of the
truth ratio in the retained model. We hypothesize that this effect arises from
the drastic alterations in the embedding vectors caused by flipping their
signs.

Split | Method | Retain Prob | Forget Prob | Authors Prob | Facts Prob | Retain TR | Forget TR | Authors TR | Facts TR | Retain ROUGE | Forget ROUGE | Authors ROUGE | Facts ROUGE | Utility | Forget Quality
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
1% | Original | 0.9904 | 0.9923 | 0.4625 | 0.4234 | 0.4659 | 0.5199 | 0.6010 | 0.5591 | 0.9798 | 0.9275 | 0.9005 | 0.8917 | 0.6257 | 0.0030
Retain | 0.9913 | 0.1788 | 0.4429 | 0.4120 | 0.4608 | 0.6919 | 0.5756 | 0.5389 | 0.9803 | 0.3832 | 0.9190 | 0.8889 | 0.6126 | 1.0000
Grad Ascent | 0.9654 | 0.2599 | 0.4308 | 0.4058 | 0.4678 | 0.5591 | 0.5663 | 0.5486 | 0.8819 | 0.4361 | 0.8855 | 0.8853 | 0.6024 | 0.0068
Grad Diff | 0.9674 | 0.3082 | 0.4336 | 0.4082 | 0.4657 | 0.5532 | 0.5687 | 0.5548 | 0.8932 | 0.4480 | 0.9030 | 0.8853 | 0.6059 | 0.0143
KL Min | 0.9663 | 0.2615 | 0.4325 | 0.4062 | 0.4677 | 0.5598 | 0.5688 | 0.5499 | 0.8860 | 0.4427 | 0.8855 | 0.8853 | 0.6036 | 0.0068
Pref Opt | 0.9706 | 0.8748 | 0.4679 | 0.4483 | 0.4405 | 0.5935 | 0.6063 | 0.5549 | 0.9104 | 0.3131 | 0.9238 | 0.8832 | 0.6236 | 0.0971
Prompt | 0.8740 | 0.8629 | 0.4422 | 0.4430 | 0.4434 | 0.5517 | 0.5606 | 0.5741 | 0.6155 | 0.5739 | 0.5980 | 0.8020 | 0.5629 | 0.0068
NPO | 0.3655 | 0.0190 | 0.3161 | 0.3662 | 0.4646 | 0.6643 | 0.4277 | 0.5006 | 0.4180 | 0.2478 | 0.8178 | 0.8906 | 0.4532 | 0.7659
NPO-KL | 0.3812 | 0.0203 | 0.3120 | 0.3614 | 0.4651 | 0.6573 | 0.4208 | 0.4973 | 0.4312 | 0.2755 | 0.8275 | 0.9074 | 0.4554 | 0.4046
NPO-RT | 0.5685 | 0.0264 | 0.3214 | 0.3704 | 0.4653 | 0.6654 | 0.4298 | 0.5111 | 0.4760 | 0.2655 | 0.8448 | 0.9138 | 0.4896 | 0.5786
ECO (Rand Noise) | 0.9904 | 0.0001 | 0.4625 | 0.4234 | 0.4659 | 0.7093 | 0.6010 | 0.5591 | 0.9798 | 0.0538 | 0.9005 | 0.8917 | 0.6257 | 0.9188
ECO (Zero-Out) | 0.9904 | 0.2642 | 0.4625 | 0.4234 | 0.4659 | 0.7208 | 0.6010 | 0.5591 | 0.9798 | 0.5182 | 0.9005 | 0.8917 | 0.6257 | 0.9900
ECO (Sign-Flip) | 0.9904 | 0.0000 | 0.4625 | 0.4234 | 0.4659 | 0.8707 | 0.6010 | 0.5591 | 0.9798 | 0.0332 | 0.9005 | 0.8917 | 0.6257 | 0.0002
5% | Original | 0.9905 | 0.9887 | 0.4625 | 0.4234 | 0.4659 | 0.5090 | 0.6010 | 0.5591 | 0.9804 | 0.9570 | 0.9005 | 0.8917 | 0.6257 | 0.0000
Retain | 0.9905 | 0.1497 | 0.4217 | 0.4121 | 0.4571 | 0.6713 | 0.5528 | 0.5316 | 0.9800 | 0.3935 | 0.9330 | 0.8675 | 0.6028 | 1.0000
Grad Ascent | 0.0000 | 0.0000 | 0.2551 | 0.3069 | 0.1766 | 0.5889 | 0.4631 | 0.4750 | 0.0000 | 0.0009 | 0.0000 | 0.0000 | 0.0000 | 0.0118
Grad Diff | 0.1009 | 0.0001 | 0.4023 | 0.4125 | 0.5749 | 0.7428 | 0.5546 | 0.5692 | 0.2069 | 0.0185 | 0.6088 | 0.8718 | 0.3244 | 0.0000
KL Min | 0.0000 | 0.0000 | 0.2645 | 0.3422 | 0.1905 | 0.5929 | 0.5014 | 0.5065 | 0.0000 | 0.0009 | 0.0000 | 0.0000 | 0.0000 | 0.0163
Pref Opt | 0.9208 | 0.7919 | 0.4376 | 0.4229 | 0.4199 | 0.5812 | 0.5636 | 0.5070 | 0.6352 | 0.0327 | 0.2440 | 0.7863 | 0.4785 | 0.0000
Prompt | 0.8688 | 0.8228 | 0.4187 | 0.4639 | 0.4415 | 0.5454 | 0.5223 | 0.5995 | 0.5260 | 0.4406 | 0.3920 | 0.7507 | 0.5194 | 0.0000
NPO | 0.0327 | 0.0079 | 0.2838 | 0.3699 | 0.4042 | 0.6468 | 0.3931 | 0.5498 | 0.2782 | 0.1968 | 0.3227 | 0.8254 | 0.1745 | 0.7934
NPO-KL | 0.2199 | 0.0476 | 0.3071 | 0.3589 | 0.4247 | 0.6694 | 0.3960 | 0.5130 | 0.4261 | 0.2945 | 0.7438 | 0.9160 | 0.4054 | 0.7934
NPO-RT | 0.5006 | 0.0612 | 0.4370 | 0.4112 | 0.4420 | 0.6731 | 0.5835 | 0.5585 | 0.5437 | 0.2893 | 0.8293 | 0.9288 | 0.5419 | 0.6284
ECO (Rand Noise) | 0.9905 | 0.0002 | 0.4625 | 0.4234 | 0.4659 | 0.6713 | 0.6010 | 0.5591 | 0.9804 | 0.0440 | 0.9005 | 0.8917 | 0.6257 | 0.9647
ECO (Zero-Out) | 0.9905 | 0.1382 | 0.4625 | 0.4207 | 0.4659 | 0.6663 | 0.6010 | 0.5576 | 0.9804 | 0.3265 | 0.9005 | 0.8917 | 0.6248 | 0.9647
ECO (Sign-Flip) | 0.9905 | 0.0010 | 0.4625 | 0.4234 | 0.4659 | 0.8195 | 0.6010 | 0.5591 | 0.9804 | 0.0762 | 0.9005 | 0.8917 | 0.6257 | 0.0000
10% | Original | 0.9905 | 0.9897 | 0.4625 | 0.4234 | 0.4659 | 0.5161 | 0.6010 | 0.5591 | 0.9799 | 0.9738 | 0.9005 | 0.8917 | 0.6257 | 0.0000
Retain | 0.9906 | 0.1476 | 0.4375 | 0.4123 | 0.4564 | 0.6800 | 0.5685 | 0.5331 | 0.9769 | 0.3951 | 0.9330 | 0.8932 | 0.6097 | 1.0000
Grad Ascent | 0.0000 | 0.0000 | 0.2495 | 0.2370 | 0.0661 | 0.8881 | 0.4014 | 0.3760 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000
Grad Diff | 0.5709 | 0.0000 | 0.5745 | 0.4216 | 0.4808 | 0.7284 | 0.7353 | 0.5688 | 0.4906 | 0.0032 | 0.8275 | 0.8718 | 0.5823 | 0.0000
KL Min | 0.0000 | 0.0000 | 0.2544 | 0.2608 | 0.1706 | 0.6919 | 0.3509 | 0.3461 | 0.0046 | 0.0049 | 0.0000 | 0.0000 | 0.0000 | 0.1810
Pref Opt | 0.9505 | 0.8570 | 0.4213 | 0.4112 | 0.4406 | 0.5516 | 0.5398 | 0.4993 | 0.7528 | 0.0602 | 0.4647 | 0.8205 | 0.5395 | 0.0000
Prompt | 0.8625 | 0.8235 | 0.4173 | 0.4719 | 0.4422 | 0.5480 | 0.5150 | 0.6106 | 0.5863 | 0.5041 | 0.2620 | 0.7066 | 0.4877 | 0.0000
NPO | 0.0377 | 0.0290 | 0.3038 | 0.3439 | 0.3002 | 0.7232 | 0.4162 | 0.5231 | 0.2238 | 0.2010 | 0.1670 | 0.4789 | 0.1701 | 0.0126
NPO-KL | 0.1981 | 0.1121 | 0.3070 | 0.3610 | 0.3293 | 0.7236 | 0.3770 | 0.4941 | 0.3370 | 0.2483 | 0.6190 | 0.6802 | 0.3623 | 0.0158
NPO-RT | 0.4694 | 0.1207 | 0.3818 | 0.4307 | 0.4105 | 0.7060 | 0.4742 | 0.5806 | 0.4502 | 0.2380 | 0.7652 | 0.8732 | 0.4997 | 0.0783
ECO (Rand Noise) | 0.9905 | 0.0003 | 0.4625 | 0.4178 | 0.4659 | 0.6903 | 0.6010 | 0.5557 | 0.9799 | 0.0784 | 0.9005 | 0.8746 | 0.6229 | 0.5812
ECO (Zero-Out) | 0.9905 | 0.0998 | 0.4625 | 0.4224 | 0.4659 | 0.6770 | 0.6010 | 0.5581 | 0.9799 | 0.3067 | 0.9005 | 0.8746 | 0.6243 | 0.9674
ECO (Sign-Flip) | 0.9905 | 0.0010 | 0.4625 | 0.4191 | 0.4659 | 0.8145 | 0.6010 | 0.5580 | 0.9799 | 0.0694 | 0.9005 | 0.8746 | 0.6235 | 0.0000

Table 13: Full results of Llama-2-7B-Chat on the TOFU dataset.

Split | Method | Retain Prob | Forget Prob | Authors Prob | Facts Prob | Retain TR | Forget TR | Authors TR | Facts TR | Retain ROUGE | Forget ROUGE | Authors ROUGE | Facts ROUGE | Utility | Forget Quality
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
1% | Original | 0.9272 | 0.9294 | 0.3825 | 0.4108 | 0.4859 | 0.4818 | 0.4619 | 0.4949 | 0.9213 | 0.9511 | 0.5865 | 0.8711 | 0.5519 | 0.0143