#  Defensive Unlearning with Adversarial Training
for Robust Concept Erasure in Diffusion Models

Yimeng Zhang1 &Xin Chen2 &Jinghan Jia1 &Yihua Zhang1 &Chongyu Fan1 &Jiancheng
Liu1 &Mingyi Hong3 &Ke Ding2 &Sijia Liu1
1Michigan State University 2Applied ML, Intel 3University of Minnesota, Twin
City

###### Abstract

Diffusion models (DMs) have achieved remarkable success in text-to-image
generation, but they also pose safety risks, such as the potential generation
of harmful content and copyright violations. The techniques of machine
unlearning, also known as concept erasing, have been developed to address
these risks. However, these techniques remain vulnerable to adversarial prompt
attacks, which can prompt DMs post-unlearning to regenerate undesired images
containing concepts (such as nudity) meant to be erased. This work aims to
enhance the robustness of concept erasing by integrating the principle of
adversarial training (AT) into machine unlearning, resulting in the robust
unlearning framework referred to as AdvUnlearn. However, achieving this
effectively and efficiently is highly non-trivial. First, we find that a
straightforward implementation of AT compromises DMs’ image generation quality
post-unlearning. To address this, we develop a utility-retaining
regularization on an additional retain set, optimizing the trade-off between
concept erasure robustness and model utility in AdvUnlearn. Moreover, we
identify the text encoder as a more suitable module for robustification
compared to UNet, ensuring unlearning effectiveness. And the acquired text
encoder can serve as a plug-and-play robust unlearner for various DM types.
Empirically, we perform extensive experiments to demonstrate the robustness
advantage of AdvUnlearn across various DM unlearning scenarios, including the
erasure of nudity, objects, and style concepts. In addition to robustness,
AdvUnlearn also achieves a balanced tradeoff with model utility. To our
knowledge, this is the first work to systematically explore robust DM
unlearning through AT, setting it apart from existing methods that overlook
robustness in concept erasing. Codes are available at:
https://github.com/OPTML-Group/AdvUnlearn

Warning: This paper contains model outputs that may be offensive in nature.

##  1 Introduction

Recent rapid advancements in diffusion models (DMs) (Ho et al., 2020; Song and
Ermon, 2019; Song et al., 2020; Dhariwal and Nichol, 2021; Nichol and
Dhariwal, 2021; Watson et al., 2021; Rombach et al., 2022; Croitoru et al.,
2023) have popularized the realm of text-to-image generation. These models,
trained on extensive online datasets, can generate remarkably realistic
images. However, their training heavily relies on diverse internet-sourced
content and can introduce safety concerns when prompted with inappropriate
texts, such as the generation of NSFW (Not Safe For Work) images, highlighted
in several studies (Rando et al., 2022; Schramowski et al., 2023). To address
this concern, post-hoc safety checkers were initially applied to DMs
(Schramowski et al., 2023; Nichol et al., 2021). However, they were later
found to be inadequate in effectively preventing the generation of unsafe
content. To further enhance safety, the concept of machine unlearning (MU) has
been introduced (Nguyen et al., 2022; Shaik et al., 2023; Cao and Yang, 2015;
Thudi et al., 2022; Xu et al., 2023; Jia et al., 2023; Fan et al., 2024),
aiming to mitigate the influence of undesired textual concepts in DM training
or fine-tuning (Gandikota et al., 2023a; Zhang et al., 2023a; Kumari et al.,
2023a; Gandikota et al., 2023b). As a result, DMs post-unlearning (referred to
as ‘concept-erased DMs’ or ‘unlearned DMs’) are designed to negate the
generation of undesirable content, even when faced with inappropriate prompts.

Figure 1: Visualizations of SD v1.4 (Rombach et al., 2022) and its concept-
erased version, ESD (Gandikota et al., 2023a), conditioned on the prompt ‘Hot
sexy girl’. The generation of ESD is shown against no attack and
UnlearnDiffAtk (Zhang et al., 2023b).

Despite the recent progress with unlearned, safety-driven DMs, recent studies
(Zhang et al., 2023b; Chin et al., 2023; Tsai et al., 2023; Pham et al.,
2024a) have shown that these models remain vulnerable to generating unsafe
images when exposed to adversarial prompt attacks, which involve minor
adversarial perturbations in input prompts. These attacks can readily
jailbreak concept-erased DMs to regenerate content subject to the concept
targeted for unlearning, even if these DMs perform well against inappropriate
prompts in a non-adversarial context. In Fig. 1, we exemplify the generation
of the stable diffusion (SD) v1.4 model before and post unlearning the
‘nudity’ concept. The unlearned model is confronted with an inappropriate
prompt from the I2P dataset (Schramowski et al., 2023) and its adversarial
prompt counterpart, generated using the attack method UnlearnDiffAtk (Zhang et
al., 2023b). The lack of robustness in concept erasing (or machine unlearning)
in DMs gives rise to the key research question tackled in this work:

(Q) Can we effectively and efficiently boost the robustness of unlearned DMs
against adversarial prompt attacks?

Figure 2: Performance overview of our proposal AdvUnlearn and various DM
unlearning baselines when unlearning the nudity concept under the SD v1.4
model. The robustness is measured by attack success rate (ASR) against
UnlearnDiffAtk (Zhang et al., 2023b). The performance of image generation
retention is assessed through Fréchet Inception Distance (FID). A lower ASR or
FID implies better robustness or utility. The baselines include the vanilla SD
v1.4 and its unlearned versions using ESD (Gandikota et al., 2023a), FMN
(Zhang et al., 2023a), UCE (Gandikota et al., 2023b), SalUn (Fan et al.,
2023), and SPM (Lyu et al., 2023).

To address (Q), we take inspiration from the successes of adversarial training
(AT) (Madry et al., 2017) in enhancing the adversarial robustness of image
classification models. To the best of our knowledge, we are the first to study
the integration of AT into DM unlearning systematically and to develop a
successful integration scheme, termed AdvUnlearn, by addressing its unique
effectiveness and efficiency challenges, such as balancing the preservation of
image generation quality and selecting the appropriate module to optimize
during AT. In the literature, the most relevant work to ours is (Huang et al.,
2023), which employs AT to train robust adapters within UNet for DMs. However,
our work significantly differs from (Huang et al., 2023). First, we aim for a
comprehensive study of AT for DMs, focusing not only on when AT is
(in)effective for DMs and why this (in)effectiveness occurs but also on how to
improve it. Additionally, we explore which advancements in AT for robust image
classification can be translated into improving the robustness of DMs. Second,
we identify that retaining image generation quality is a major challenge when
integrating AT into DMs, especially in compatibility with DM unlearning
methods. We tackle this challenge by drawing inspiration from the AT principle
that ‘unlabeled data improves the robustness-accuracy tradeoff’ (Zhang et al.,
2019; Carmon et al., 2019; Alayrac et al., 2019), and accordingly develop a
utility-retaining regularization scheme based on an augmented retain prompt
set. Third, by dissecting DMs into text encoder and UNet components, we
discover that the integration of AT with DM unlearning particularly favors the
text encoder module. This contrasts with conventional DM unlearning methods,
which are typically applied to the UNet.

We summarize our key contributions as follows:

❶ We explore the integration of AT with concept erasing (or machine
unlearning) in DMs and propose a bi-level optimization (BLO)-based integration
scheme, termed AdvUnlearn. We identify a significant utility loss for image
generation when incorporating AT. To address this, we design a utility-
retaining regularization using curated external retain prompt data to balance
the trade-off between effective unlearning and high-quality image generation.

❷ We demonstrate that optimizing the text encoder within AdvUnlearn can
enhance the robustness of unlearned DMs against adversarial prompt attacks,
outperforming the conventional strategies for unlearning UNet. In addition, it
also achieves a better balance between unlearning performance and image
generation utility. Furthermore, we show that a single robust text encoder can
be shared across different DMs and implemented in a plug-and-play manner,
greatly enhancing usability.

❸ We validate the effectiveness of AdvUnlearn across various DM unlearning
scenarios, including the erasure of nudity, objects, and style concepts. We
show that AdvUnlearn yields significant robustness improvements over state-of-
the-art (SOTA) unlearned DMs while preserving a commendable level of image
generation utility; See Fig. 2 for justification and performance highlights.

##  2 Related Work

Machine unlearning for concept erasing in DMs. Generating visually authentic
images from textual descriptions remains a compelling challenge in generative
AI. DMs (diffusion models) have notably advanced, surpassing generative
adversarial networks (GANs) in various aspects, particularly in conditional
generation subject to text prompts (Kang et al., 2023; Tao et al., 2023, 2022;
Li et al., 2019; Xu et al., 2018; Reed et al., 2016; Zhou et al., 2022; Kawar
et al., 2023; Zhang et al., 2022a). Despite their success, DMs also present
safety and ethics concerns, especially in generating images with harmful
content when conditioned on inappropriate prompts (Rando et al., 2022;
Schramowski et al., 2023). To address these concerns, several approaches have
been proposed, including post-image filtering (Rando et al., 2022), modifying
inference guidance (Schramowski et al., 2023), and retraining with curated
datasets (Rombach et al., 2022). However, lightweight interventions like the
first two may not fully address the model’s inherent propensity to generate
controversial content (Schramowski et al., 2023; Birhane et al., 2021;
Somepalli et al., 2023). MU (machine unlearning) (Cao and Yang, 2015; Ginart
et al., 2019; Jia et al., 2024) is another emerging approach for ensuring safe
image generation by erasing the influence of undesired concepts, also referred
to as concept erasing in DMs. Leveraging MU principles, various strategies for
designing unlearned DMs have been explored, focusing on refining fine-tuning
methods such as those by (Gandikota et al., 2023a; Zhang et al., 2023a; Kumari
et al., 2023a; Gandikota et al., 2023b; Fan et al., 2023; Lyu et al., 2023;
Huang et al., 2023; Kumari et al., 2023b; Heng and Soh, 2023; Ni et al., 2023;
Zhang et al., 2024a; Wu et al., 2024; Wu and Harandi, 2024; Pham et al.,
2024b). For example, UNets within DMs have been fine-tuned to redirect outputs
towards either random or anchored outputs, effectively preventing the
generation of images associated with the concepts designated for unlearning
(Gandikota et al., 2023a; Zhang et al., 2023a; Kumari et al., 2023a).
Additional efforts (Fan et al., 2023; Wu and Harandi, 2024) have employed
gradient-based techniques to map out the weight saliency within UNet related
to the concept to be unlearned, concentrating fine-tuning efforts on these
salient weights. To enhance the efficiency of unlearning, UCE (Gandikota et
al., 2023b) introduces a method of closed-form parameter editing specifically
for DM unlearning. However, this approach lacks robustness against
jailbreaking attacks (Zhang et al., 2023b).

Adversarial prompt attacks against safety-driven DMs. Adversarial prompts or
jailbreaking attacks specifically manipulate the text inputs of DMs to produce
undesirable outcomes. Similar to the text-based attacks in natural language
processing (NLP), adversarial prompt attacks can involve character or word-
level manipulations, such as deletions, additions, and substitutions (Eger and
Benz, 2020; Liu et al., 2022; Hou et al., 2022; Li et al., 2018; Alzantot et
al., 2018; Jin et al., 2020; Garg and Ramakrishnan, 2020; Qiu et al., 2022).
Strategies discussed in (Yang et al., 2023) are designed to bypass NSFW safety
protocols, cleverly evading content moderation algorithms. Other related
attacks (Zhang et al., 2023b; Chin et al., 2023; Tsai et al., 2023; Pham et
al., 2024a; Maus et al., 2023; Zhuang et al., 2023) coerce DMs into generating
images that contradict their programmed intent. For instance, (Pham et al.,
2024a) used textual inversion (Gal et al., 2022) to find a continuous word
embedding representing the concept to be unlearned by the model. (Chin et al.,
2023) employed ground truth guidance from an auxiliary frozen UNet (Gandikota
et al., 2023a) and discrete optimization techniques from (Wen et al., 2024) to
craft a white-box adversarial prompt attack. To overcome the dependency on
auxiliary model guidance, UnlearnDiffAtk (Zhang et al., 2023b) leveraged the
intrinsic classification capabilities of DMs, facilitating the creation of
adversarial prompts. In this work, we treat machine unlearning for DMs as a
defensive challenge. Our approach involves fine-tuning the target model to not
only unlearn specific concepts but also to enhance its robustness against
adversarial prompt attacks.

Adversarial training (AT). In the realm of image classification, adversarial
attacks that generate subtle perturbations to fool machine learning (ML)
models have long posed a robustness challenge for vision systems (Goodfellow
et al., 2014; Carlini and Wagner, 2017; Croce and Hein, 2020; Xu et al., 2019;
Athalye et al., 2018a; Gong et al., 2022). In response, AT (adversarial
training) (Madry et al., 2017), the cornerstone of training-based defenses,
conceptualizes defense as a two-player game between the attacker and defender
(Madry et al., 2017; Zhang et al., 2019; Carmon et al., 2019; Goodfellow et
al., 2014; Xie et al., 2017; Kannan et al., 2018; Athalye et al., 2018b;
Stanforth et al., 2019; Cai et al., 2018; Wang et al., 2019). Additionally,
TRADES (Zhang et al., 2019) was proposed to strike a better balance between
accuracy and robustness. Further studies (Carmon et al., 2019; Alayrac et al.,
2019; Stanforth et al., 2019; Zhai et al., 2019) demonstrated that unlabeled
data and self-training have proven effective in enhancing robustness and
generalization in adversarial contexts. To improve the efficiency of AT, past
research also proposed adopting more efficient attack methods or fewer steps
to generate adversarial examples (Goodfellow et al., 2014; Wong et al., 2020;
Tramèr et al., 2018; Andriushchenko and Flammarion, 2020; Sriramanan et al.,
2020, 2021; Zhang et al., 2022b; Tsiligkaridis and Roberts, 2022; Shafahi et
al., 2019). In particular, the fast gradient sign method (FGSM) was utilized
for adversarial generation in AT (Goodfellow et al., 2014; Wong et al., 2020).
And the gradient alignment strategy was proposed to improve the quality of
fast AT (Andriushchenko and Flammarion, 2020).

##  3 Preliminaries and Problem Statement

Throughout the work, we focus on latent diffusion models (LDMs) (Rombach et
al., 2022; Cao et al., 2022), which have exceled in text-to-image generation
by integrating text prompts (such as text-based image descriptions) into image
embeddings to guide the generation process. In LDMs, the diffusion process
initiates with a random noise sample drawn from the standard Gaussian
distribution $\mathcal{N}(0,1)$. This sample undergoes a progressive
transformation through a series of $T$ time steps in a gradual denoising
process, ultimately resulting in the creation of a clean image $\mathbf{x}$.
At each time step $t$, the diffusion model utilizes a noise estimator
$\epsilon_{\bm{\theta}}(\cdot|c)$, parameterized by $\bm{\theta}$ and
conditioned on an input prompt $c$ (i.e., associated with a textual concept).
The diffusion process operates on the latent representation of the image at
each time ($\mathbf{x}_{t}$). The training objective for $\bm{\theta}$ is to
minimize the denoising error as below:

$\displaystyle\begin{array}[]{ll}\displaystyle\operatorname*{\text{minimize}}_{\bm{\theta}}&\mathbb{E}_{(\mathbf{x},c)\sim\mathcal{D},t,\epsilon\sim\mathcal{N}(0,1)}\left[\|\epsilon-\epsilon_{\bm{\theta}}(\mathbf{x}_{t}|c)\|_{2}^{2}\right],\end{array}$
(2)

where $\mathcal{D}$ is the training set, and
${\epsilon}_{\bm{\theta}}(\mathbf{x}_{t}|c)$ is the LDM-associated noise
estimator.

Concept erasure in DMs. DMs, despite their high capability, may generate
unsafe content or disclose sensitive information when given inappropriate text
prompts. For example, the I2P dataset (Schramowski et al., 2023) compiles
numerous inappropriate prompts capable of leading DMs to generate NSFW
content. To mitigate the generation of harmful or sensitive content, a range
of studies (Gandikota et al., 2023a; Zhang et al., 2023a; Kumari et al.,
2023a; Gandikota et al., 2023b; Fan et al., 2023) have explored the technique
of concept erasing or machine unlearning within DMs, aiming to enhance the DM
training process by mitigating the impact of undesired textual concepts on
image generation.

A widely recognized concept erasing approach is ESD (Gandikota et al., 2023a),
notable for its state-of-the-art (SOTA) balance between unlearning
effectiveness and model utility preservation (Zhang et al., 2024a). Unless
specified otherwise, we will adopt the objective of ESD for implementing
concept erasure. ESD facilitates the fine-tuning process of DMs by guiding
outputs away from a specific concept targeted for erasure. Let
$c_{\mathrm{e}}$ denote the concept to erase, then the diffusion process of
ESD is modified to

$\displaystyle\epsilon_{\bm{\theta}}(\mathbf{x}_{t}|c_{\mathrm{e}})\leftarrow\epsilon_{\bm{\theta}_{\mathrm{o}}}(\mathbf{x}_{t}|\emptyset)-\eta\left(\epsilon_{\bm{\theta}_{\mathrm{o}}}(\mathbf{x}_{t}|c_{\mathrm{e}})-\epsilon_{\bm{\theta}_{\mathrm{o}}}(\mathbf{x}_{t}|\emptyset)\right),$
(3)

where $\bm{\theta}$ denotes the concept-erased DM, $\bm{\theta}_{\mathrm{o}}$
is the originally pre-trained DM, and
$\epsilon_{\bm{\theta}}(\mathbf{x}_{t}|\emptyset)$ represents unconditional
generation of the model $\bm{\theta}$ by considering text prompt as empty.
Compared to the standard conditional DM (Ho and Salimans, 2022) (with
classifier-free guidance), the second term
$-\eta[\epsilon_{\bm{\theta}_{\mathrm{o}}}(\mathbf{x}_{t}|c_{\mathrm{e}})-\epsilon_{\bm{\theta}_{\mathrm{o}}}(\mathbf{x}_{t}|\emptyset)]$
encourages the adjustment of the data distribution (with erasing guidance
parameter $\eta>0$) to minimize the likelihood of generating an image
$\mathbf{x}$ that could be labeled as $c_{\mathrm{e}}$. To optimize
$\bm{\theta}$, ESD performs the following model fine-tuning based on (3):

$\displaystyle\begin{array}[]{ll}\displaystyle\operatorname*{\text{minimize}}_{\bm{\theta}}&\ell_{\mathrm{ESD}}(\bm{\theta},c_{\mathrm{e}})\mathrel{\mathop{:}}=\mathbb{E}\left[\left\|\epsilon_{\bm{\theta}}(\mathbf{x}_{t}|c_{\mathrm{e}})-\left(\epsilon_{\bm{\theta}_{\mathrm{o}}}(\mathbf{x}_{t}|\emptyset)-\eta\left(\epsilon_{\bm{\theta}_{\mathrm{o}}}(\mathbf{x}_{t}|c_{\mathrm{e}})-\epsilon_{\bm{\theta}_{\mathrm{o}}}(\mathbf{x}_{t}|\emptyset)\right)\right)\right\|_{2}^{2}\right],\end{array}$
(5)

where for notational simplicity we have used, and will continue to use, to
omit the time step $t$ and the random initial noise $\epsilon$ under
expectation.

Adversarial prompts against concept-erased DMs. Although concept erasing
enhances safety, recent studies (Zhang et al., 2023b; Chin et al., 2023; Tsai
et al., 2023; Pham et al., 2024a) have also shown that concept-erased DMs
often lack robustness when confronted with adversarial prompt attacks; see
Fig. 1 for examples. Let $c^{\prime}$ represent a perturbed text prompt
corresponding to $c$, obtained through token manipulation in the text space
(Zhang et al., 2023b; Chin et al., 2023) or in the token embedding space (Pham
et al., 2024a). The generation of adversarial prompts can be solved as (Zhang
et al., 2023b; Chin et al., 2023):

$\displaystyle\operatorname*{\text{minimize}}_{\|c^{\prime}-c\|_{0}\leq\epsilon}\quad$
$\displaystyle\mathbb{E}\left[\left\|\epsilon_{\bm{\theta}}(\mathbf{x}_{t}|c^{\prime})-\epsilon_{\bm{\theta}_{\mathrm{o}}}(\mathbf{x}_{t}|c)\right\|_{2}^{2}\right],$
(6)

where $\bm{\theta}$ denotes the concept-erased DM, and
$\bm{\theta}_{\mathrm{o}}$ is the original DM without concept erasing.
Therefore, considering the concept $c=c_{\mathrm{e}}$ targeted for erasure,
$\epsilon_{\bm{\theta}_{\mathrm{o}}}(\mathbf{x}_{t}|c)$ denotes the generation
of an unsafe image under $c$. The objective of problem (6) is to devise the
perturbed prompt $c^{\prime}$ to steer the generation of the concept-erased DM
$\bm{\theta}$ towards the unsafe content produced by
$\epsilon_{\bm{\theta}_{\mathrm{o}}}$. The constraint of (6) implies that
$c^{\prime}$ remains proximate to $c$, subject to the number of altered tokens
$\epsilon$ (measured by the $\ell_{0}$ norm) or via additive continuous
perturbation in the token embedding space.

AdvUnlearn: A defensive unlearning setup via AT. The lack of adversarial
robustness in concept-erased DMs motivates us to devise a solution that
enhances their robustness in the face of adversarial prompts. AT (Madry et
al., 2017) offers a principled algorithmic framework for addressing this
challenge. It formulates robust concept erasure as a two-player game involving
the defender (i.e., the unlearner for concept erasing) and the attacker (i.e.,
the adversarial prompt). The original AT constrains the attacker’s objective
to precisely oppose the defender’s objective. To loosen this constraint, we
consider a generalized AT formulation based on bi-level optimization (Zhang et
al., 2022b, c, 2023c, 2024b), where the defender and attacker are delineated
through the upper-level and lower-level optimization problems, respectively:

$\displaystyle\begin{array}[]{lll}\displaystyle\operatorname*{\text{minimize}}_{\bm{\theta}}&\ell_{\mathrm{u}}(\bm{\theta},c^{*})&\leavevmode\nobreak\
\leavevmode\nobreak\ \text{[Upper-level optimization]}\\\
\operatorname*{\text{subject
to}}&c^{*}=\displaystyle\operatorname*{arg\,min}_{\|c^{\prime}-c_{\mathrm{e}}\|_{0}\leq\epsilon}\leavevmode\nobreak\
\ell_{\mathrm{atk}}(\bm{\theta},c^{\prime}).&\leavevmode\nobreak\
\leavevmode\nobreak\ \text{[Lower-level optimization]}\end{array}$ (9)

In (9), the upper-level optimization aims to optimize the DM parameters
$\bm{\theta}$ according to an unlearning objective $\ell_{\mathrm{u}}$,
considering the concept $c^{*}$ targeted for erasure. For instance, the
objective of ESD (5) could serve as one specification of $\ell_{\mathrm{u}}$.
On the other hand, the lower-level optimization problem minimizes the attack
generation loss $\ell_{\mathrm{atk}}$, as given by (6), to acquire the
optimized adversarial prompt $c^{*}$ under the current model $\bm{\theta}$.
The upper-level and lower-level optimizations are interlinked through the
alternation between model parameter optimization and adversarial prompt
optimization.

We designate the aforementioned setup (9) of integrating adversarial training
into DM unlearning as AdvUnlearn. As will become evident later, effectively
and efficiently solving the AdvUnlearn problem (9) becomes highly nontrivial.
There exist two main challenges.

Table 1: Robustness (ASR) and utility (FID) of different unlearning methods
(ESD (Gandikota et al., 2023a) and AT-ESD) on base SD-v.14 model for nudity
unlearning.

Unlearning Methods |  Concept Erasure |  ASR ($\downarrow$) |  FID ($\downarrow$)
---|---|---|---
SD v1.4 | ✘ | 100% | 16.7
ESD | ✔ | 73.24% | 18.18
AT-ESD | ✔ | 43.48% | 26.48

(Effectiveness challenge) As will be demonstrated in Sec. 4, a naive
implementation of the ESD objective (3) for upper-level concept erasure may
lead to a considerable loss in DM utility for generating normal images. Thus,
optimizing the inherent trade-off between the robustness of concept erasure
and the preservation of DM utility poses a significant challenge.

(Efficiency challenge) Moreover, given the modularity characteristics of DMs
(with decomposition into text encoder and UNet encoder), determining the
optimal application of AT and its efficient implementation remains elusive.
This includes deciding ‘where’ to apply AT within DM, as well as ‘how’ to
efficiently implement it. We will address this challenge in Sec. 5.

##  4 Effectiveness Enhancement of AdvUnlearn: Improving Tradeoff between
Robustness and Utility

Figure 3:  Generation examples using DMs in Tab. 1 for nudity unlearning
conditioned on benign and harmful prompts.

Warm-up: Difficulty of image generation quality retention. A straightforward
implementation of AdvUnlearn (9) is to specify the upper-level optimization
using ESD (3) and combine it with adversarial prompt generation (6). However,
such a direct integration results in a notable decrease in image generation
quality. Tab. 1 compares the performance of the vanilla ESD (i.e., concept-
erased stable diffusion ) (Gandikota et al., 2023a) with its direct AT
variant. The robustness of concept erasure is evaluated using ASR (attack
success rate) against the adversarial prompt attack UnlearnDiffAtk (Zhang et
al., 2023b). Meanwhile, the quality of image generation retention is assessed
through FID. As we can see, while the direct AT variant of ESD (AT-ESD)
enhances adversarial robustness with approximately a 20% reduction in ASR, it
also leads to a considerable increase in FID. Fig. 3 presents visual examples
of the generation produced by AT-ESD compared to the original SD v1.4 and
vanilla ESD. As demonstrated, the decline in image generation authenticity
under a benign prompt using AT-ESD is substantial.

Utility-retaining regularization in AdvUnlearn. We next improve the unlearning
objective $\ell_{\mathrm{u}}$ in AdvUnlearn (9) by explicitly prioritizing the
retention of the DM’s generation utility. One potential explanation for the
diminished image generation quality after AT-ESD is that ESD (3) primarily
focuses on de-generating the unlearning concept in the diffusion process, thus
lacking the capability to preserve image generation quality when further
pressured by the robustness enhancement induced by AT. In the realm of AT for
image classification, the integration of external (unlabeled) data into AT has
proven to be an effective strategy for enhancing standard model utility (i.e.,
test accuracy) while simultaneously improving adversarial robustness. Drawing
inspiration from this, we suggest the curation of a retain set
$\mathcal{C}_{\mathrm{retain}}$ comprising additional text prompts utilized
for retaining model utility. Together with the ESD-based unlearning objective,
we customize the upper-level optimization objective of (9) as

$\displaystyle\ell_{\mathrm{u}}(\bm{\theta},c^{*})=\ell_{\mathrm{ESD}}(\bm{\theta},c^{*})+\gamma\mathbb{E}_{\tilde{c}\sim\mathcal{C}_{\mathrm{retain}}}\left[\left\|\epsilon_{\bm{\theta}}(\mathbf{x}_{t}|\tilde{c})-\epsilon_{\bm{\theta}_{\mathrm{o}}}(\mathbf{x}_{t}|\tilde{c})\right\|_{2}^{2}\right],$
(10)

Table 2: Performance evaluation of SD v1.4 (without unlearning), ESD, AT-ESD,
and AdvUnlearn in nudity unlearning.

Unlearning Methods |  Utility Retaining by COCO |  ASR ($\downarrow$) |  FID ($\downarrow$)
---|---|---|---
SD v1.4 | N/A | 100% | 16.70
ESD | ✘ | 73.24% | 18.18
AT-ESD | ✘ | 43.48% | 26.48
AdvUnlearn | ✔ | 64.79% | 19.88

where $\ell_{\mathrm{ESD}}$ was defined in (5), and the second loss term
penalizes the degradation of image generation quality using the current DM
$\bm{\theta}$ compared to the original $\bm{\theta}_{\mathrm{o}}$ under a
retained concept $\tilde{c}$. When selecting the retain set
$\mathcal{C}_{\mathrm{retain}}$, it is essential to ensure that the
enhancement in image generation quality does not come at the expense of the
effectiveness of concept erasure, i.e., minimizing ESD loss in (10).
Therefore, we utilize a large language model (LLM) as a judge to sift through
these retain prompts, excluding those relevant to the targeted concept for
erasure. Further details regarding the LLM judge system are available in Appx.
A. We obtain these retain prompts from an external dataset, such as ImageNet
(Deng et al., 2009) or COCO (Lin et al., 2014), using the prompt template ‘a
photo of [OBJECT CLASS]’. As shown in Tab. 2, our proposed utility-retaining
regularization effectively recovers the utility (i.e., FID of AdvUnlearn vs.
that of ESD), which is otherwise compromised by AT-ESD. Yet, AdvUnlearn in
Tab. 2 leads to an increase in ASR (sacrificing robustness) compared to ESD,
although it improves robustness over ESD. Thus, there is room for further
enhancement in AdvUnlearn. As will be shown in Sec. 5, the choice of the DM
component to optimize in (9) is crucial for better performance.

##  5 Efficiency Enhancement of AdvUnlearn: Modularity Exploration and Fast
Attack Generation

Where to robustify: Text encoder or UNet? Initially, concept erasure by ESD
(5) was confined to the UNet component of a DM (Gandikota et al., 2023a).
However, as shown in Tab. 2, optimizing UNet alone does not lead to sufficient
robustness gain for AdvUnlearn. Moreover, there are efficiency benefits if
concept erasure can be performed on the text encoder instead of UNet. The text
encoder, with fewer parameters than the UNet, can achieve convergence more
quickly. Most importantly, a text encoder that has undergone the unlearning
process with one DM could possibly serve as a plug-in unlearner for other DMs,
thereby broadening its applicability across various DMs. Furthermore, a recent
study (Basu et al., 2023) demonstrates that causal components corresponding to
the DM’s visual generation are concentrated in the text encoder. Localizing
and editing such a causal component enables control over image generation
outcomes of the entire DM.

Table 3: Performance evaluation of unlearning methods applied on different DM
modules to optimize for nudity unlearning.

DMs |  Optimized DM component |  ASR ($\downarrow$) |  FID ($\downarrow$)
---|---|---|---
SD v1.4 | N/A | 100% | 16.70
ESD | UNet | 73.24% | 18.18
ESD | Text Encoder | 3.52% | 59.10
AdvUnlearn | UNet | 64.79% | 19.88
AdvUnlearn | Text Encoder | 21.13% | 19.34

Inspired by the above, robustifying the text encoder could not only improve
effectiveness in concept erasure but also yield efficiency benefits for
AdvUnlearn. Tab. 3 extends Tab. 2 to further justify the effectiveness and
efficiency of implementing AdvUnlearn on the text encoder compared to UNet. As
we can see, the text encoder finetuned through AdvUnlearn achieves much better
unlearning robustness (i.e., lower ASR) than AdvUnlearn applied to UNet (i.e.,
AdvUnlearn in Tab. 2), without loss of model utility as evidenced by FID.
Although applying ESD to the text encoder can also improve ASR, it leads to a
significant utility loss compared to its vanilla version applied to UNet
(Gandikota et al., 2023a). This highlights the importance of retaining image
generation quality considered in AdvUnlearn when optimizing the text encoder.
In the rest of the paper, unless specified otherwise, we select the text
encoder as the DM module to optimize in AdvUnlearn (9).

Fast attack generation in AdvUnlearn. Another efficiency enhancement for
AdvUnlearn is to simplify the lower-level optimization of (9) using a one-
step, fast attack generation method. This approach aligns with the concept of
fast AT in image classification (Wong et al., 2020; Zhang et al., 2022b). The
rationale is that the lower-level problem of (9) can be approximated using a
quadratic program (Zhang et al., 2022b), and solving it can be achieved using
the fast gradient sign method (FGSM) (Goodfellow et al., 2014; Wong et al.,
2020). Specifically, let $\delta$ represent the perturbation added to the text
prompt $c$, e.g., via a prefix vector (Li and Liang, 2021). With an abuse of
notation, we denote the perturbed prompt by $c^{\prime}=c+\delta$, where the
symbol $+$ represents the prefix attachment. FGSM determines $\delta$ using
FGSM to solve the lower-level problem of (9):

$\displaystyle\delta=\delta_{0}-\alpha\cdot\mathrm{sign}\left(\nabla_{\bm{\delta}}\ell_{\mathrm{atk}}(\bm{\theta},c+\delta_{0})\right),$
(11)

Table 4: Comparison of different AT schemes in AdvUnlearn for nudity
unlearning.

AT scheme in AdvUnlearn: | AT | Fast AT
---|---|---
Attack step #: | 30 | 1
ASR ($\downarrow$) | 21.13% | 28.87%
FID ($\downarrow$) | 19.34 | 19.92
Train. time per iteration (s) | 78.57 | 12.13

where $\delta_{0}$ represents random initialization, $\alpha$ denotes the step
size, and $\mathrm{sign}(\cdot)$ is element-wise sign operation. We refer to
the utilization of one-step attack generation (11) in AdvUnlearn as its fast
variant, which can also yield a substantial robustness gain in concept
erasure. Tab. 4 compares the performance and training cost of AdvUnlearn using
fast AT vs. (standard) AT. As we can see, the adoption of fast AT reduces the
training time per iteration from $78.57s$ to $12.13s$ on a single NVIDIA RTX
A6000 GPU, albeit with a corresponding decrease in unlearning efficacy and
image generation utility. Therefore, when the need for unlearning efficacy is
not exceedingly high and computational efficiency is prioritized, adopting
fast AT can be an effective solution. We summarize the AdvUnlearn algorithm in
Appx. B.

##  6 Experiments

###  6.1 Experiment Setups

Concept-erasing tasks, datasets, and models. We categorize existing concept-
erasing tasks (Gandikota et al., 2023a; Zhang et al., 2023a; Kumari et al.,
2023a; Gandikota et al., 2023b; Fan et al., 2023; Lyu et al., 2023; Wu et al.,
2024; Wu and Harandi, 2024) into three main groups for ease of evaluation. (1)
Nudity unlearning focuses on preventing DMs from generating harmful content
subject to nudity-related prompts (Gandikota et al., 2023a; Zhang et al.,
2023a; Gandikota et al., 2023b; Fan et al., 2023; Lyu et al., 2023; Wu et al.,
2024; Wu and Harandi, 2024). (2) Style unlearning aims to remove the influence
of an artistic painting style in DM generation, which mimics the degeneration
of copyrighted information such as the painting style (Gandikota et al.,
2023a; Zhang et al., 2023a; Kumari et al., 2023a; Gandikota et al., 2023b; Lyu
et al., 2023). (3) Object unlearning, akin to the previous tasks, targets the
degeneration of DMs corresponding to a specific object (Gandikota et al.,
2023a; Zhang et al., 2023a; Fan et al., 2023; Lyu et al., 2023; Wu et al.,
2024; Wu and Harandi, 2024). The dataset for testing nudity unlearning is
derived from the inappropriate image prompt (I2P) dataset (Schramowski et al.,
2023), whereas the testing dataset for style unlearning is aligned with the
setup described in (Gandikota et al., 2023a). In the scenario of object
unlearning, GPT-4 (OpenAI, 2023) is utilized to generate $50$ distinct text
prompts for each object class featured in Imagenette (Shleifer and Prokop,
2019). These prompts have been validated to ensure that the standard SD
(stable diffusion) model can successfully generate images containing objects
from Imagenette. Model-wise, unless specified otherwise, the pre-trained SD
(Stable Diffusion) v1.4 is utilized as the base DM in concept erasing.

DM unlearning baselines. We include 8 open-sourced DM unlearning methods as
our baselines: (1) ESD (erased stable diffusion) (Gandikota et al., 2023a),
(2) FMN (Forget-Me-Not) (Zhang et al., 2023a), (3) AC (ablating concepts)
(Kumari et al., 2023a), (4) UCE (unified concept editing) (Gandikota et al.,
2023b), (5) SalUn (saliency unlearning) (Fan et al., 2023), (6) SH
(ScissorHands) (Wu and Harandi, 2024), (7) ED (EraseDiff) (Wu et al., 2024),
and (8) SPM (concept-SemiPermeable Membrane) (Lyu et al., 2023). We note that
these unlearning methods are not universally designed to address nudity,
style, and object unlearning simultaneously. Therefore, our assessment of
their robustness against adversarial prompt attacks is specific to the
unlearning tasks for which they were originally developed and employed.

Training setups. The implementation of AdvUnlearn (9) follows Algorithm 1 in
Appx. B. As demonstrated in Sec. 5, unlike existing DM unlearning methods,
AdvUnlearn specifically focuses on optimizing the text encoder within DMs. In
the training phase of AdvUnlearn, the upper-level optimization of (9) for
minimizing the unlearning objective (10) is conducted over 1000 iterations.
Each iteration uses a single data batch with the erasing guidance parameter
$\eta=1.0$ in (5) and a batch of $5$ retaining prompts with a utility
regularization parameter of $\gamma=0.3$ for nudity unlearning and $0.5$ for
style and object unlearning. These regularization parameter choices are
determined through a greedy search over the range $[0,1]$. Additionally, a
learning rate of $10^{-5}$ is employed with the Adam optimizer for text
encoder finetuning. Each upper-level iteration comprises the lower-level
attack generation, minimizing the attack objective (6) with $30$ attack steps
and a step size of $10^{-3}$. At each attack step, gradient descent is
performed over a prefix adversarial prompt token in its embedding space,
starting from a random initialization.

Evaluation setups. We focus on two main metrics for performance assessment:
unlearning robustness against adversarial prompts and the preservation of
image generation utility. For robustness evaluation, we measure the attack
success rate (ASR) of DMs in the presence of adversarial prompt attacks, where
a lower ASR indicates better robustness. Unless specified otherwise, we
utilize UnlearnDiffAtk (Zhang et al., 2023b) for generating adversarial
prompts at testing time, as it can be regarded as an unseen attack strategy
different from (6) used in AdvUnlearn. Detailed settings for attack evaluation
are presented in Appx. C. For utility evaluation, we use FID (Heusel et al.,
2017) to assess the distributional quality of image generations. We also use
CLIP score (Hessel et al., 2021) to measure their contextual alignment with
prompt descriptions. A lower FID score, indicative of a smaller distributional
distance between generated and real images, signifies higher image quality.
And a higher CLIP score reflects the better performance of DMs in producing
contextually relevant images. To compute these utility metrics, we employ DMs
to generate $10k$ images under $10k$ prompts, randomly sampled from the COCO
caption dataset (Chen et al., 2015).

###  6.2 Experiment Results

Robustness-utility evaluation of AdvUnlearn for nudity unlearning. In Tab. 5,
we compare the adversarial robustness (measured by ASR) and the utility
(evaluated using FID and CLIP score) of our proposed AdvUnlearn with

Table 5:  Performance summary of nudity unlearning: ASR characterizes the
robustness of DMs, including the pre-trained SD v1.4 (base model) and nudity-
unlearned DMs, against adversarial prompt attacks generated by UnlearnDiffAtk
(Zhang et al., 2023b) to regenerate nudity-related content. FID and CLIP
scores characterize the preserved image generation utility of DMs subject to
nudity-irrelevant benign prompts.

Metrics |  SD v1.4 (Base) |  FMN |  SPM |  UCE |  ESD |  SalUn |  AdvUnlearn (Ours)
---|---|---|---|---|---|---|---
ASR ($\downarrow$) | 100% | 97.89% | 91.55% | 79.58% | 73.24% | 11.27% | 21.13%
FID ($\downarrow$) | 16.7 | 16.86 | 17.48 | 17.10 | 18.18 | 33.62 | 19.34
CLIP ($\uparrow$) | 0.311 | 0.308 | 0.310 | 0.309 | 0.302 | 0.287 | 0.290

unlearning baselines when erasing the nudity concept in DM generation. For
ease of presentation, we also refer to the DM post-unlearning (i.e., the
unlearned DM) with the name of the corresponding unlearning method. Here we
exclude the baselines SH and ED from the performance comparison in nudity
unlearning due to their exceptionally high FID scores (over 100), indicating
significantly compromised image generation quality. Detailed results and
visualizations for these unlearning baselines are provided in Appx. D. As we
can see, our proposal (AdvUnlearn) demonstrates significantly improved ASR,
with over 50% ASR reduction compared to ESD, except for the lowest ASR
achieved by SalUn. However, unlike SalUn, our robustness improvement does not
come at a substantial cost to the DM utility. This is evident from its
significantly better FID and CLIP scores compared to SalUn. To complement our
quantitative findings, Fig. 4 showcases the visualizations of the generations
produced by various unlearned DMs in response to both the nudity-sensitive
adversarial prompt and the nudity-irrelevant standard prompt. As observed,
SalUn and AdvUnlearn are the only unlearning methods that yield robust
concept-erased DMs against adversarial attacks. Yet, AdvUnlearn outperforms
SalUn in generation quality.

Figure 4:  Visualizations of generated images by SD v1.4 and its nudity-
unlearned variants. First row: Image generation under the adversarial prompt
attack UnlearnDiffAtk (Zhang et al., 2023b) applied to the original prompt
‘woman standing in the doorway’. Second row: Image generation under the
standard benign prompt ‘clean white toilet’. Table 6: Performance summary of
unlearning the Van Gogh style, following a format similar to Tab. 5.

Metrics |  SD v1.4 (Base) |  UCE |  SPM |  AC |  FMN |  ESD |  AdvUnlearn (Ours)
---|---|---|---|---|---|---|---
ASR ($\downarrow$) | 100% | 96% | 88% | 72% | 52% | 36% | 2%
FID ($\downarrow$) | 16.70 | 16.31 | 16.65 | 17.50 | 16.59 | 18.71 | 16.96
CLIP ($\uparrow$) | 0.311 | 0.311 | 0.311 | 0.310 | 0.309 | 0.304 | 0.308

Effectiveness in style unlearning. In Tab. 6, we compare the robustness and
utility performance of AdvUnlearn with various DM unlearning methods when
removing the ‘Van Gogh’ artistic style from image generation. This comparison
excludes the unlearning baseline SalUn but includes AC, based on whether they
were originally developed for style unlearning. As observed, our proposal
demonstrates a significant improvement in robustness, with over a 30% decrease
in ASR compared to the second-best unlearning method, ESD. Crucially, this is
accomplished without sacrificing model utility, as indicated by the comparable
FID and CLIP scores compared to the base SD v1.4. The effectiveness of our
proposal is also demonstrated through the generated images in Fig. 5 under
adversarial prompt attack and ‘Van Gogh’-irrelevant benign prompt,
respectively.

Figure 5: Examples of generated images by DMs when unlearning Van Gogh style,
following Fig. 4’s format.  Table 7: Performance summary of unlearning the
object Church in DM generation, following a format similar to Tab. 5.

Metrics |  SD v1.4 (Base) |  FMN |  SPM |  SalUn |  ESD |  ED |  SH |  AdvUnlearn (Ours)
---|---|---|---|---|---|---|---|---
ASR ($\downarrow$) | 100% | 96% | 94% | 62% | 60% | 52% | 6% | 6%
FID ($\downarrow$) | 16.70 | 16.49 | 16.76 | 17.38 | 20.95 | 17.46 | 68.02 | 18.06
CLIP ($\uparrow$) | 0.311 | 0.308 | 0.310 | 0.312 | 0.300 | 0.310 | 0.277 | 0.305

Effectiveness in object unlearning. Tab. 7 compares the performance of
AdvUnlearn with baselines when unlearning the object concept ‘Church’. As we
can see, similar to style unlearning, our approach achieves the highest
robustness in Church unlearning, significantly preserving the original DM
utility compared to the unlearning baseline SH, which attains similar
robustness gain. The superiority of AdvUnlearn can also be visualized in Fig.
6, showing DM generation examples. Other results and visualizations can be
found in Appx. E.

Figure 6: Examples of generated images by DMs when unlearning the object
church, following Fig. 4’s format.  Table 8:  Plug-in performance of text
encoder obtained from AdvUnlearn when applied to other DMs, including SD v1.5,
DreamShaper, and Protogen, for nudity unlearning. ‘Original’ refers to the
text encoder originally associated with a pre-trained DM, while ‘Transfer’
denotes the use of the text encoder acquired through AdvUnlearn in SD v1.4 and
applied to other DMs.

DMs: | SD v1.4 | SD v1.5 | DreamShaper | Protogen
---|---|---|---|---
Text encoder: | Original | AdvUnlearn | Original | Transfer | Original | Transfer | Original | Transfer
ASR ($\downarrow$) | 100% | 21.13% | 95.74% | 16.20% | 90.14% | 61.27% | 83.10% | 42.96%
FID ($\downarrow$) | 16.70 | 19.34 | 16.86 | 19.27 | 23.01 | 27.40 | 20.63 | 24.47
CLIP ($\uparrow$) | 0.311 | 0.290 | 0.311 | 0.289 | 0.312 | 0.295 | 0.314 | 0.298

Plug-and-play capability of adversarially unlearned text encoder. Given the
modular nature of the text encoder in DMs, we further explore whether the
robustness and utility of the text encoder learned from AdvUnlearn on one DM
(specifically, SD v1.4 in our experiments) can be directly transferred to
other types of DMs without additional fine-tuning. Tab. 8 summarizes the plug-
in performance of the text encoder obtained from AdvUnlearn when applied to SD
v1.5, DreamShaper (Lykon, 2023a), and Protogen (Lykon, 2023b) for nudity
unlearning. As we can see, the considerable robustness improvement as well as
utility in DM unlearning are preserved when plugging the text encoder obtained
from AdvUnlearn in SD v1.4 into other DMs (see ‘Transfer’ performance vs.
‘Original’ performance). This is most significant when transferring to SD v1.5
due to its similarity with SD v1.4. For dissimilar DMs like DreamShaper
(Lykon, 2023a) and Protogen (Lykon, 2023b), the AdvUnlearn-acquired text
encoder in SD v1.4 still remains effective as a plug-in option, lowering the
ASR without sacrificing utility significantly. Fig. 7 offers visual examples
of image generation associated with the results presented in Tab. 8.

Figure 7: Images generated by different personalized DMs with original or
plug-in AdvUnlearn text encoder for nudity unlearning.

The effect of text encoder layers on DM unlearning. In Fig. 8, we show the ASR
(robustness metric) and the CLIP score (a utility metric) of post-nudity
unlearning against various choices of text encoder layers for optimization in
AdvUnlearn. Here the layer number equal to $N$ signifies that the first $N$
layers are optimized. We observe that the

Figure 8: Performance of AdvUnlearn vs. text encoder layers to optimize in
nudity unlearning.

robustness gain escalates as more layers are optimized. In particular,
optimizing only the initial layers failed to provide adequate robustness for
DM unlearning against adversarial attacks, contrary to findings in (Basu et
al., 2023), where shallower encoder layers suffice for guiding DMs in image
editing albeit in a non-adversarial context. Yet, we also find that for object
or style unlearning, optimizing only the first layer of the text encoder has
demonstrated satisfactory robustness and utility in DM unlearning. This
suggests that nudity unlearning presents a more challenging task in ensuring
robustness. Utility-wise, we observe a slight performance degradation as more
encoder layers are robustified, which is under expectation.

Other ablation studies. In Appx. F, we demonstrate more ablation studies. This
includes: ① the impact of the utility-retaining regularization weight on
AdvUnlearn (Fig. A3); ② the selection of retain sets for utility-retaining
regularization (Tab. A3); ③ the effect of the number of attack steps on
adversarial prompt generation for AdvUnlearn (Fig. A4); ④ the impact of
adversarial prompting strategy for AdvUnlearn (Tab. A4); ⑤ the robustness of
SD v1.4 finetuned through AdvUnlearn against different adversarial prompt
attack (Tab. A5).

##  7 Conclusion

Current unlearned DMs (diffusion models) remain vulnerable to adversarial
prompt attacks. Our proposed robust unlearning framework, AdvUnlearn,
illuminates potential strategies for enhancing the robustness of unlearned DMs
against such attacks while preserving image generation utility. Notably, our
framework employs utility-retaining regularization on a retained prompt set
and identifies the text encoder as a more effective module for robustification
compared to the UNet within DMs. Through extensive experiments, we demonstrate
that AdvUnlearn strikes a graceful balance between robust unlearning and image
generation utility. Despite the possibility of using fast attack generation to
speed up AdvUnlearn, continual improvement in computational efficiency remains
a crucial area for future research. Limitations and broader impacts are
further discussed in Appx. G and Appx. H.

##  8 Acknowledgement

Y. Zhang, J. Jia, Y. Zhang, C. Fan, J. Liu, and S. Liu were supported by the
National Science Foundation (NSF) Robust Intelligence (RI) Core Program Award
IIS-2207052 and the Cisco Faculty Research Award.

## References

  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in neural information processing systems_ , 33:6840–6851, 2020.
  * Song and Ermon [2019] Yang Song and Stefano Ermon.  Generative modeling by estimating gradients of the data distribution.  _Advances in neural information processing systems_ , 32, 2019.
  * Song et al. [2020] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  _arXiv preprint arXiv:2011.13456_ , 2020.
  * Dhariwal and Nichol [2021] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  _Advances in neural information processing systems_ , 34:8780–8794, 2021.
  * Nichol and Dhariwal [2021] Alexander Quinn Nichol and Prafulla Dhariwal.  Improved denoising diffusion probabilistic models.  In _International Conference on Machine Learning_ , pages 8162–8171. PMLR, 2021.
  * Watson et al. [2021] Daniel Watson, William Chan, Jonathan Ho, and Mohammad Norouzi.  Learning fast samplers for diffusion models by differentiating through sample quality.  In _International Conference on Learning Representations_ , 2021.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10684–10695, 2022.
  * Croitoru et al. [2023] Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, and Mubarak Shah.  Diffusion models in vision: A survey.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2023.
  * Rando et al. [2022] Javier Rando, Daniel Paleka, David Lindner, Lennard Heim, and Florian Tramèr.  Red-teaming the stable diffusion safety filter.  _arXiv preprint arXiv:2210.04610_ , 2022.
  * Schramowski et al. [2023] Patrick Schramowski, Manuel Brack, Björn Deiseroth, and Kristian Kersting.  Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models, 2023.
  * Nichol et al. [2021] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  _arXiv preprint arXiv:2112.10741_ , 2021.
  * Nguyen et al. [2022] Thanh Tam Nguyen, Thanh Trung Huynh, Phi Le Nguyen, Alan Wee-Chung Liew, Hongzhi Yin, and Quoc Viet Hung Nguyen.  A survey of machine unlearning.  _arXiv preprint arXiv:2209.02299_ , 2022.
  * Shaik et al. [2023] Thanveer Shaik, Xiaohui Tao, Haoran Xie, Lin Li, Xiaofeng Zhu, and Qing Li.  Exploring the landscape of machine unlearning: A survey and taxonomy.  _arXiv preprint arXiv:2305.06360_ , 2023.
  * Cao and Yang [2015] Yinzhi Cao and Junfeng Yang.  Towards making systems forget with machine unlearning.  In _2015 IEEE Symposium on Security and Privacy_ , pages 463–480. IEEE, 2015.
  * Thudi et al. [2022] Anvith Thudi, Gabriel Deza, Varun Chandrasekaran, and Nicolas Papernot.  Unrolling sgd: Understanding factors influencing machine unlearning.  In _2022 IEEE 7th European Symposium on Security and Privacy (EuroS &P)_, pages 303–319. IEEE, 2022.
  * Xu et al. [2023] Heng Xu, Tianqing Zhu, Lefeng Zhang, Wanlei Zhou, and Philip S Yu.  Machine unlearning: A survey.  _ACM Computing Surveys_ , 56(1):1–36, 2023.
  * Jia et al. [2023] Jinghan Jia, Jiancheng Liu, Parikshit Ram, Yuguang Yao, Gaowen Liu, Yang Liu, Pranay Sharma, and Sijia Liu.  Model sparsity can simplify machine unlearning, 2023.
  * Fan et al. [2024] Chongyu Fan, Jiancheng Liu, Alfred Hero, and Sijia Liu.  Challenging forgets: Unveiling the worst-case forget sets in machine unlearning.  _arXiv preprint arXiv:2403.07362_ , 2024.
  * Gandikota et al. [2023a] Rohit Gandikota, Joanna Materzynska, Jaden Fiotto-Kaufman, and David Bau.  Erasing concepts from diffusion models.  _arXiv preprint arXiv:2303.07345_ , 2023a.
  * Zhang et al. [2023a] Eric Zhang, Kai Wang, Xingqian Xu, Zhangyang Wang, and Humphrey Shi.  Forget-me-not: Learning to forget in text-to-image diffusion models.  _arXiv preprint arXiv:2303.17591_ , 2023a.
  * Kumari et al. [2023a] Nupur Kumari, Bingliang Zhang, Sheng-Yu Wang, Eli Shechtman, Richard Zhang, and Jun-Yan Zhu.  Ablating concepts in text-to-image diffusion models, 2023a.
  * Gandikota et al. [2023b] Rohit Gandikota, Hadas Orgad, Yonatan Belinkov, Joanna Materzyńska, and David Bau.  Unified concept editing in diffusion models.  _arXiv preprint arXiv:2308.14761_ , 2023b.
  * Zhang et al. [2023b] Yimeng Zhang, Jinghan Jia, Xin Chen, Aochuan Chen, Yihua Zhang, Jiancheng Liu, Ke Ding, and Sijia Liu.  To generate or not? safety-driven unlearned diffusion models are still easy to generate unsafe images… for now.  _arXiv preprint arXiv:2310.11868_ , 2023b.
  * Chin et al. [2023] Zhi-Yi Chin, Chieh-Ming Jiang, Ching-Chun Huang, Pin-Yu Chen, and Wei-Chen Chiu.  Prompting4debugging: Red-teaming text-to-image diffusion models by finding problematic prompts.  _arXiv preprint arXiv:2309.06135_ , 2023.
  * Tsai et al. [2023] Yu-Lin Tsai, Chia-Yi Hsu, Chulin Xie, Chih-Hsun Lin, Jia-You Chen, Bo Li, Pin-Yu Chen, Chia-Mu Yu, and Chun-Ying Huang.  Ring-a-bell! how reliable are concept removal methods for diffusion models?  _arXiv preprint arXiv:2310.10012_ , 2023.
  * Pham et al. [2024a] Minh Pham, Kelly O. Marshall, Niv Cohen, Govind Mittal, and Chinmay Hegde.  Circumventing concept erasure methods for text-to-image generative models.  In _The Twelfth International Conference on Learning Representations_ , 2024a.
  * Fan et al. [2023] Chongyu Fan, Jiancheng Liu, Yihua Zhang, Dennis Wei, Eric Wong, and Sijia Liu.  Salun: Empowering machine unlearning via gradient-based weight saliency in both image classification and generation.  _arXiv preprint arXiv:2310.12508_ , 2023.
  * Lyu et al. [2023] Mengyao Lyu, Yuhong Yang, Haiwen Hong, Hui Chen, Xuan Jin, Yuan He, Hui Xue, Jungong Han, and Guiguang Ding.  One-dimensional adapter to rule them all: Concepts, diffusion models and erasing applications.  _arXiv preprint arXiv:2312.16145_ , 2023.
  * Madry et al. [2017] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu.  Towards deep learning models resistant to adversarial attacks.  _arXiv preprint arXiv:1706.06083_ , 2017.
  * Huang et al. [2023] Chi-Pin Huang, Kai-Po Chang, Chung-Ting Tsai, Yung-Hsuan Lai, and Yu-Chiang Frank Wang.  Receler: Reliable concept erasing of text-to-image diffusion models via lightweight erasers.  _arXiv preprint arXiv:2311.17717_ , 2023.
  * Zhang et al. [2019] Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P Xing, Laurent El Ghaoui, and Michael I Jordan.  Theoretically principled trade-off between robustness and accuracy.  _ICML_ , 2019.
  * Carmon et al. [2019] Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, Percy Liang, and John C Duchi.  Unlabeled data improves adversarial robustness.  _arXiv preprint arXiv:1905.13736_ , 2019.
  * Alayrac et al. [2019] Jean-Baptiste Alayrac, Jonathan Uesato, Po-Sen Huang, Alhussein Fawzi, Robert Stanforth, and Pushmeet Kohli.  Are labels required for improving adversarial robustness?  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Kang et al. [2023] Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park.  Scaling up gans for text-to-image synthesis.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10124–10134, 2023.
  * Tao et al. [2023] Ming Tao, Bing-Kun Bao, Hao Tang, and Changsheng Xu.  Galip: Generative adversarial clips for text-to-image synthesis.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 14214–14223, 2023.
  * Tao et al. [2022] Ming Tao, Hao Tang, Fei Wu, Xiao-Yuan Jing, Bing-Kun Bao, and Changsheng Xu.  Df-gan: A simple and effective baseline for text-to-image synthesis.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 16515–16525, 2022.
  * Li et al. [2019] Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, and Philip Torr.  Controllable text-to-image generation.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Xu et al. [2018] Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He.  Attngan: Fine-grained text to image generation with attentional generative adversarial networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1316–1324, 2018.
  * Reed et al. [2016] Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee.  Generative adversarial text to image synthesis.  In _International conference on machine learning_ , pages 1060–1069. PMLR, 2016.
  * Zhou et al. [2022] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Learning to prompt for vision-language models.  _International Journal of Computer Vision_ , 130(9):2337–2348, 2022.
  * Kawar et al. [2023] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani.  Imagic: Text-based real image editing with diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 6007–6017, June 2023.
  * Zhang et al. [2022a] Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu.  Motiondiffuse: Text-driven human motion generation with diffusion model.  _arXiv preprint arXiv:2208.15001_ , 2022a.
  * Birhane et al. [2021] Abeba Birhane, Vinay Uday Prabhu, and Emmanuel Kahembwe.  Multimodal datasets: misogyny, pornography, and malignant stereotypes.  _arXiv preprint arXiv:2110.01963_ , 2021.
  * Somepalli et al. [2023] Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein.  Diffusion art or digital forgery? investigating data replication in diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6048–6058, 2023.
  * Ginart et al. [2019] Antonio Ginart, Melody Guan, Gregory Valiant, and James Y Zou.  Making ai forget you: Data deletion in machine learning.  _Advances in neural information processing systems_ , 32, 2019.
  * Jia et al. [2024] Jinghan Jia, Yihua Zhang, Yimeng Zhang, Jiancheng Liu, Bharat Runwal, James Diffenderfer, Bhavya Kailkhura, and Sijia Liu.  Soul: Unlocking the power of second-order optimization for llm unlearning, 2024.
  * Kumari et al. [2023b] Nupur Kumari, Bingliang Zhang, Sheng-Yu Wang, Eli Shechtman, Richard Zhang, and Jun-Yan Zhu.  Ablating concepts in text-to-image diffusion models.  In _ICCV_ , 2023b.
  * Heng and Soh [2023] Alvin Heng and Harold Soh.  Selective amnesia: A continual learning approach to forgetting in deep generative models.  _arXiv preprint arXiv:2305.10120_ , 2023.
  * Ni et al. [2023] Zixuan Ni, Longhui Wei, Jiachen Li, Siliang Tang, Yueting Zhuang, and Qi Tian.  Degeneration-tuning: Using scrambled grid shield unwanted concepts from stable diffusion.  _arXiv preprint arXiv:2308.02552_ , 2023.
  * Zhang et al. [2024a] Yihua Zhang, Yimeng Zhang, Yuguang Yao, Jinghan Jia, Jiancheng Liu, Xiaoming Liu, and Sijia Liu.  Unlearncanvas: A stylized image dataset to benchmark machine unlearning for diffusion models.  _arXiv preprint arXiv:2402.11846_ , 2024a.
  * Wu et al. [2024] Jing Wu, Trung Le, Munawar Hayat, and Mehrtash Harandi.  Erasediff: Erasing data influence in diffusion models.  _arXiv preprint arXiv:2401.05779_ , 2024.
  * Wu and Harandi [2024] Jing Wu and Mehrtash Harandi.  Scissorhands: Scrub data influence via connection sensitivity in networks.  _arXiv preprint arXiv:2401.06187_ , 2024.
  * Pham et al. [2024b] Minh Pham, Kelly O Marshall, Chinmay Hegde, and Niv Cohen.  Robust concept erasure using task vectors.  _arXiv preprint arXiv:2404.03631_ , 2024b.
  * Eger and Benz [2020] Steffen Eger and Yannik Benz.  From hero to zéroe: A benchmark of low-level adversarial attacks.  In _Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing_ , pages 786–803, December 2020.
  * Liu et al. [2022] Aiwei Liu, Honghai Yu, Xuming Hu, Shu’ang Li, Li Lin, Fukun Ma, Yawen Yang, and Lijie Wen.  Character-level white-box adversarial attacks against transformers via attachable subwords substitution.  _arXiv preprint arXiv:2210.17004_ , 2022.
  * Hou et al. [2022] Bairu Hou, Jinghan Jia, Yihua Zhang, Guanhua Zhang, Yang Zhang, Sijia Liu, and Shiyu Chang.  Textgrad: Advancing robustness evaluation in nlp by gradient-driven optimization.  _arXiv preprint arXiv:2212.09254_ , 2022.
  * Li et al. [2018] Jinfeng Li, Shouling Ji, Tianyu Du, Bo Li, and Ting Wang.  Textbugger: Generating adversarial text against real-world applications.  _arXiv preprint arXiv:1812.05271_ , 2018.
  * Alzantot et al. [2018] Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang.  Generating natural language adversarial examples.  _arXiv preprint arXiv:1804.07998_ , 2018.
  * Jin et al. [2020] Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits.  Is bert really robust? a strong baseline for natural language attack on text classification and entailment.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 34, pages 8018–8025, 2020.
  * Garg and Ramakrishnan [2020] Siddhant Garg and Goutham Ramakrishnan.  Bae: Bert-based adversarial examples for text classification.  _arXiv preprint arXiv:2004.01970_ , 2020.
  * Qiu et al. [2022] Shilin Qiu, Qihe Liu, Shijie Zhou, and Wen Huang.  Adversarial attack and defense technologies in natural language processing: A survey.  _Neurocomputing_ , 492:278–307, 2022.
  * Yang et al. [2023] Yuchen Yang, Bo Hui, Haolin Yuan, Neil Gong, and Yinzhi Cao.  Sneakyprompt: Evaluating robustness of text-to-image generative models’ safety filters.  _arXiv preprint arXiv:2305.12082_ , 2023.
  * Maus et al. [2023] Natalie Maus, Patrick Chao, Eric Wong, and Jacob R Gardner.  Black box adversarial prompting for foundation models.  In _The Second Workshop on New Frontiers in Adversarial Machine Learning_ , 2023.
  * Zhuang et al. [2023] Haomin Zhuang, Yihua Zhang, and Sijia Liu.  A pilot study of query-free adversarial attack against stable diffusion.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2384–2391, 2023.
  * Gal et al. [2022] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or.  An image is worth one word: Personalizing text-to-image generation using textual inversion.  _arXiv preprint arXiv:2208.01618_ , 2022.
  * Wen et al. [2024] Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Goldstein.  Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Goodfellow et al. [2014] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy.  Explaining and harnessing adversarial examples.  _arXiv preprint arXiv:1412.6572_ , 2014.
  * Carlini and Wagner [2017] Nicholas Carlini and David Wagner.  Towards evaluating the robustness of neural networks.  In _2017 ieee symposium on security and privacy (sp)_ , pages 39–57. Ieee, 2017.
  * Croce and Hein [2020] Francesco Croce and Matthias Hein.  Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks.  In _ICML_ , 2020.
  * Xu et al. [2019] Kaidi Xu, Sijia Liu, Pu Zhao, Pin-Yu Chen, Huan Zhang, Quanfu Fan, Deniz Erdogmus, Yanzhi Wang, and Xue Lin.  Structured adversarial attack: Towards general implementation and better interpretability.  In _ICLR_ , 2019.
  * Athalye et al. [2018a] Anish Athalye, Logan Engstrom, Andrew Ilyas, and Kevin Kwok.  Synthesizing robust adversarial examples.  In _International Conference on Machine Learning_ , pages 284–293, 2018a.
  * Gong et al. [2022] Yifan Gong, Yuguang Yao, Yize Li, Yimeng Zhang, Xiaoming Liu, Xue Lin, and Sijia Liu.  Reverse engineering of imperceptible adversarial image perturbations.  _arXiv preprint arXiv:2203.14145_ , 2022.
  * Xie et al. [2017] Cihang Xie, Jianyu Wang, Zhishuai Zhang, Zhou Ren, and Alan Yuille.  Mitigating adversarial effects through randomization.  _arXiv preprint arXiv:1711.01991_ , 2017.
  * Kannan et al. [2018] Harini Kannan, Alexey Kurakin, and Ian Goodfellow.  Adversarial logit pairing, 2018.
  * Athalye et al. [2018b] Anish Athalye, Nicholas Carlini, and David Wagner.  Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples.  _arXiv preprint arXiv:1802.00420_ , 2018b.
  * Stanforth et al. [2019] Robert Stanforth, Alhussein Fawzi, Pushmeet Kohli, et al.  Are labels required for improving adversarial robustness?  _arXiv preprint arXiv:1905.13725_ , 2019.
  * Cai et al. [2018] Qi-Zhi Cai, Min Du, Chang Liu, and Dawn Song.  Curriculum adversarial training.  _arXiv preprint arXiv:1805.04807_ , 2018.
  * Wang et al. [2019] Jingkang Wang, Tianyun Zhang, Sijia Liu, Pin-Yu Chen, Jiacen Xu, Makan Fardad, and Bo Li.  Towards a unified min-max framework for adversarial exploration and robustness.  _arXiv preprint arXiv:1906.03563_ , 2019.
  * Zhai et al. [2019] Runtian Zhai, Tianle Cai, Di He, Chen Dan, Kun He, John Hopcroft, and Liwei Wang.  Adversarially robust generalization just requires more unlabeled data.  _arXiv preprint arXiv:1906.00555_ , 2019.
  * Wong et al. [2020] Eric Wong, Leslie Rice, and J. Zico Kolter.  Fast is better than free: Revisiting adversarial training.  In _ICLR_ , 2020.
  * Tramèr et al. [2018] Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick McDaniel.  Ensemble adversarial training: Attacks and defenses.  In _International Conference on Learning Representations_ , 2018.
  * Andriushchenko and Flammarion [2020] Maksym Andriushchenko and Nicolas Flammarion.  Understanding and improving fast adversarial training.  _arXiv preprint arXiv:2007.02617_ , 2020.
  * Sriramanan et al. [2020] Gaurang Sriramanan, Sravanti Addepalli, Arya Baburaj, et al.  Guided adversarial attack for evaluating and enhancing adversarial defenses.  _Advances in Neural Information Processing Systems_ , 33:20297–20308, 2020.
  * Sriramanan et al. [2021] Gaurang Sriramanan, Sravanti Addepalli, Arya Baburaj, et al.  Towards efficient and effective adversarial training.  _Advances in Neural Information Processing Systems_ , 34:11821–11833, 2021.
  * Zhang et al. [2022b] Yihua Zhang, Guanhua Zhang, Prashant Khanduri, Mingyi Hong, Shiyu Chang, and Sijia Liu.  Revisiting and advancing fast adversarial training through the lens of bi-level optimization.  In _International Conference on Machine Learning_ , pages 26693–26712. PMLR, 2022b.
  * Tsiligkaridis and Roberts [2022] Theodoros Tsiligkaridis and Jay Roberts.  Understanding and increasing efficiency of frank-wolfe adversarial training.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 50–59, 2022.
  * Shafahi et al. [2019] Ali Shafahi, Mahyar Najibi, Mohammad Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S Davis, Gavin Taylor, and Tom Goldstein.  Adversarial training for free!  In _Advances in Neural Information Processing Systems_ , pages 3353–3364, 2019.
  * Cao et al. [2022] Hanqun Cao, Cheng Tan, Zhangyang Gao, Yilun Xu, Guangyong Chen, Pheng-Ann Heng, and Stan Z Li.  A survey on generative diffusion model.  _arXiv preprint arXiv:2209.02646_ , 2022.
  * Ho and Salimans [2022] Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  _arXiv preprint arXiv:2207.12598_ , 2022.
  * Zhang et al. [2022c] Yihua Zhang, Yuguang Yao, Parikshit Ram, Pu Zhao, Tianlong Chen, Mingyi Hong, Yanzhi Wang, and Sijia Liu.  Advancing model pruning via bi-level optimization.  _Advances in Neural Information Processing Systems_ , 35:18309–18326, 2022c.
  * Zhang et al. [2023c] Yihua Zhang, Pranay Sharma, Parikshit Ram, Mingyi Hong, Kush Varshney, and Sijia Liu.  What is missing in irm training and evaluation? challenges and solutions.  _arXiv preprint arXiv:2303.02343_ , 2023c.
  * Zhang et al. [2024b] Yihua Zhang, Prashant Khanduri, Ioannis Tsaknakis, Yuguang Yao, Mingyi Hong, and Sijia Liu.  An introduction to bilevel optimization: Foundations and applications in signal processing and machine learning.  _IEEE Signal Processing Magazine_ , 41(1):38–59, 2024b.
  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pages 248–255. Ieee, 2009.
  * Lin et al. [2014] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_ , pages 740–755. Springer, 2014.
  * Basu et al. [2023] Samyadeep Basu, Nanxuan Zhao, Vlad I Morariu, Soheil Feizi, and Varun Manjunatha.  Localizing and editing knowledge in text-to-image generative models.  In _The Twelfth International Conference on Learning Representations_ , 2023.
  * Li and Liang [2021] Xiang Lisa Li and Percy Liang.  Prefix-tuning: Optimizing continuous prompts for generation.  _arXiv preprint arXiv:2101.00190_ , 2021.
  * OpenAI [2023] OpenAI.  Gpt-4 technical report.  _ArXiv_ , abs/2303.08774, 2023.  URL https://api.semanticscholar.org/CorpusID:257532815.
  * Shleifer and Prokop [2019] Sam Shleifer and Eric Prokop.  Using small proxy datasets to accelerate hyperparameter search.  _arXiv preprint arXiv:1906.04887_ , 2019.
  * Heusel et al. [2017] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  _Advances in neural information processing systems_ , 30, 2017.
  * Hessel et al. [2021] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi.  Clipscore: A reference-free evaluation metric for image captioning.  _arXiv preprint arXiv:2104.08718_ , 2021.
  * Chen et al. [2015] Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco captions: Data collection and evaluation server.  _arXiv preprint arXiv:1504.00325_ , 2015.
  * Lykon [2023a] Lykon.  Dreamshaper, 2023a.  URL https://huggingface.co/Lykon/DreamShaper.
  * Lykon [2023b] Lykon.  Protogen, 2023b.  URL https://huggingface.co/darkstorm2150/Protogen_v2.2_Official_Release.
  * Bedapudi [2019] P Bedapudi.  Nudenet: Neural nets for nudity classification, detection and selective censoring, 2019.

## Appendix

##  Appendix A System Prompt for LLM as a Judger

We utilize a Large Language Model (LLM) like GPT-4 to act as a judger to
filter out prompts that are relevant to the concept to be unlearned. Since
nudity is a global concept (Gandikota et al., 2023a), even if ’nudity’ is not
mentioned in the prompt, the generated image conditioned on the prompt might
still involve the ’nudity’ concept. As a result, the system prompt for the LLM
to act as a prompt judger for the ’nudity’ concept is shown below:

”Could you identify the prompts from my uploaded CSV file that are likely to
lead the Stable Diffusion model to generate images with sensitive content? For
instance, prompts that mention attire with an emphasis on attractiveness or
those that focus on specific body parts may unintentionally prompt the
generation of images with nudity.”

Unlike ‘nudity’, objects are local concepts, so we only need to filter out the
name of the object that needs to be unlearned directly in the prompt. Thus,
the system prompt for the LLM to act as a prompt judger for the specific
object is shown below:

”Could you analyze the prompts from my uploaded CSV file to identify those
that are likely to direct the Stable Diffusion model to generate images
containing ‘{OBJECT}’ or any of its synonyms?”

##  Appendix B Algorithm Details: AdvUnlearn

Algorithm 1 outlines the steps of AdvUnlearn. Each iteration consists of two
stages: adversarial prompt generation and unlearning with utility-retaining
regularization. During adversarial prompt generation, the optimized
adversarial prompt embedding is randomly initialized and iteratively updated.
When attack step number $K=1$, one-step FGSM (11) is utilized to generate
adversarial promtps. In the unlearning stage, the adversarial prompt optimized
in the previous stage is used to compute the unlearning loss, while a batch of
prompts from the retain set is used to compute the utility-retaining
regularization loss. The combination of unlearning loss and retaining loss is
then used to update the trainable module parameters.

Algorithm 1 AdvUnlearn: Defensive Unlearning with Adversarial Training for DMs

1:Given Iteration Number $I$, batch size $b$ of retaining prompts
$c_{\text{retain}}$, regularization weight $\gamma$, learning rate $\beta$,
adversarial step size $\alpha$, attack step number $K$, unlearning concept
$c$, the DM to be unlearned $\bm{\theta}$, and the frozen original DM
$\bm{\theta}_{\mathrm{o}}$:

2:for $i=1,2,\ldots,I$ do

3: $\blacklozenge$ Adversarial prompt generation

4: Randomly initialize adversarial soft prompt embedding $\delta_{0}$

5: if K = 1 then

6:
$\delta=\delta_{0}-\alpha\cdot\mathrm{sign}\left(\nabla_{\bm{\delta}}\ell_{\mathrm{atk}}(\bm{\theta}_{i},c+\delta_{0})\right)$
$\triangleright$ Fast AT variant: One-Step FGSM (11)

7: else

8: Run K-Step attacks using (6)

9: end if

10: $\diamondsuit$ Defensive unlearning with utility-retaining regularization

11:
$\ell_{\mathrm{u}}(\bm{\theta}_{i},c+\delta_{K})=\ell_{\mathrm{ESD}}(\bm{\theta}_{i},c+\delta_{K})+\gamma\mathbb{E}_{\tilde{c}\sim\mathcal{C}_{\mathrm{retain}}}\left[\left\|\epsilon_{\bm{\theta}_{i}}(\mathbf{x}_{t}|\tilde{c})-\epsilon_{\bm{\theta}_{\mathrm{o}}}(\mathbf{x}_{t}|\tilde{c})\right\|_{2}^{2}\right]$

12:
$\bm{\theta}_{i+1}=\bm{\theta}_{i}-\beta\nabla_{\bm{\theta}}\ell_{\mathrm{u}}(\bm{\theta}_{i},c+\delta_{K})$

13:end for

##  Appendix C Attack Setup

Following the methodology of (Zhang et al., 2023b), we introduce prepended
prompt perturbations using $N$ tokens, where $N=5$ is designated for nudity
unlearning and $N=3$ for both style and object unlearning. To optimize these
adversarial perturbations, we sample $50$ diffusion time steps and execute
UnlearnDiffAtk over $40$ iterations with a learning rate of $0.01$, using the
AdamW optimizer. For the classification of generated images, we use NudeNet
(Bedapudi, 2019) for nudity unlearning, a ViT-base model (Zhang et al., 2023b)
fine-tuned on the WikiArt dataset for style unlearning, and an ImageNet-
pretrained ResNet-50 for object unlearning tasks.

##  Appendix D Other Results and Visualizations for Nudity Unlearning

Figure A1: Visualization examples of images generated by SH and ED for nudity
unlearning. Table A1: Performance evaluation of additional unlearning methods,
SH and ED, applied to the base SD v1.4 model for nudity unlearning.

Metrics |  SD v1.4 (Base) |  SH |  ED
---|---|---|---
ASR ($\downarrow$) | 100% | 7.04% | 2.11%
FID ($\downarrow$) | 16.70 | 128.53 | 233.31
CLIP ($\uparrow$) | 0.311 | 0.235 | 0.180

The metrics used for this evaluation are Attack Success Rate (ASR), Fréchet
Inception Distance (FID), and CLIP score. As shown in Tab. A1, although
ScissorHands (SH) (Wu and Harandi, 2024) and EraseDiff (ED) (Wu et al., 2024)
achieve high unlearning robustness against adversarial prompt attacks, the
trade-offs are significant. Their dramatically high FID and low CLIP scores
indicate an inability to generate high-quality images that align with the
condition text prompts. This is further corroborated by their visualization
examples. In Fig. A1, we observe that, regardless of the condition prompts,
the generated images are similar, demonstrating that SH and ED fail to
generate varied and contextually appropriate images. Therefore, we do not
include them in the main performance evaluation table for nudity unlearning to
maintain clarity.

##  Appendix E Other Results and Visualizations for Object Unlearning

Table A2: Performance evaluation of unlearning methods applied to the base SD
v1.4 model for Garbage Truck, Parachute, and Tench unlearning.

Concept |  Metrics |  SD v1.4 (Base) |  FMN |  SPM |  SalUn |  ED |  ESD |  SH |  AdvUnlearn (Ours)
---|---|---|---|---|---|---|---|---|---
Garbage Truck | ASR ($\downarrow$) | 100% | 100% | 82% | 42% | 38% | 26% | 2% | 8%
FID ($\downarrow$) | 16.70 | 16.14 | 16.79 | 18.03 | 19.22 | 24.81 | 67.76 | 17.92
CLIP ($\uparrow$) | 0.311 | 0.308 | 0.310 | 0.311 | 0.307 | 0.290 | 0.283 | 0.305
Parachute | ASR ($\downarrow$) | 100% | 100% | 96% | 74% | 82% | 58% | 24% | 14%
FID ($\downarrow$) | 16.70 | 16.72 | 16.77 | 18.87 | 18.53 | 21.4 | 55.18 | 17.78
CLIP ($\uparrow$) | 0.311 | 0.307 | 0.311 | 0.311 | 0.309 | 0.299 | 0.282 | 0.306
Tench | ASR ($\downarrow$) | 100% | 100% | 90% | 14% | 16% | 48% | 8% | 4%
FID ($\downarrow$) | 16.70 | 16.45 | 16.75 | 17.97 | 17.13 | 18.12 | 57.66 | 17.26
CLIP ($\uparrow$) | 0.311 | 0.308 | 0.311 | 0.313 | 0.310 | 0.301 | 0.280 | 0.307

In Tab. A2, we present a detailed evaluation of various unlearning methods
applied to the base SD v1.4 model for three different objects: Garbage Truck,
Parachute, and Tench. The unlearning methods compared include FMN (Forget-Me-
Not) (Zhang et al., 2023a), SPM (concept-SemiPermeable Membrane) (Lyu et al.,
2023), SalUn (saliency unlearning) (Fan et al., 2023), ED (EraseDiff) (Wu et
al., 2024), ESD (erased stable diffusion) (Gandikota et al., 2023a), SH
(ScissorHands) (Wu and Harandi, 2024), and our proposed DM unlearning scheme,
referred to as AdvUnlearn. As shown, our proposed DM unlearning method,
AdvUnlearn, consistently achieves the best unlearning efficacy (around $10\%$)
with competitive image generation utility. In comparison, FMN and SPM
prioritize retaining image generation utility but exhibit weak unlearning
robustness against adversarial prompt attacks. Conversely, SH achieves strong
robustness but at the high cost of image generation utility degradation. The
remaining methods (SalUn, ED, and ESD) attempt to find a balance between
robustness and utility; however, their unlearning robustness is not stable
across different object concepts, and their ASR is multiple times higher than
that of our proposed AdvUnlearn. The visualization examples associated with
the results presented in Tab. A2 can be found in Fig. A2. Through
visualization examples, we demonstrate that our proposed robust unlearning
framework, AdvUnlearn, not only effectively removes the influence of target
concepts to be unlearned in the text prompt but also retains the influence of
other descriptions. For instance, a garbage truck-unlearned DM equipped with
the AdvUnlearn text encoder generates a photo of a parking lot from the text
prompt ‘garbage truck in a parking lot.’ Similarly, other object-unlearned DMs
with corresponding AdvUnlearn text encoders produce a photo of a desert
landscape from the prompt ‘parachute in a desert landscape’ and a photo of a
baby in a pond from the prompt ‘baby tench in a pond.’ Furthermore, AdvUnlearn
reduces the disruption on the image generation utility compared to other
methods.

Figure A2: Visualization examples of images generated by different unlearning
method for Garbage Truck, Parachute, and Tench unlearning.

##  Appendix F Other Ablation Studies

In this section, we explore the influence of utility-retaining regularization
weight and attack step number of adversarial prompt generation for our
proposed DM unlearning scheme, AdvUnlearn.

Figure A3: Performance comparison of different utility-retaining
regularization weight for AdvUnlearn.

Regularization weight. As depicted in Fig. A3, there is a clear upward trend
in both the CLIP Score and ASR as the utility-retaining regularization weight
increases. The CLIP Score, represented by the blue line, shows a gradual and
consistent rise, starting from $0.282$ at a regularization weight of $0.1$ and
reaching $0.3$ at a weight of $0.7$, indicating improved image generation
utility. However, this improvement comes at a cost, as the ASR, illustrated by
the red bars, demonstrates a significant increase from $13.18\%$ at a weight
of $0.1$ to $36.63\%$ at a weight of $0.7$, suggesting a degradation in
unlearning efficacy. Consequently, we have selected a regularization weight of
$0.3$ as our default setting due to its balanced performance, providing a
compromise between enhanced image generation utility and acceptable unlearning
efficacy.

Table A3: Performance evaluations of AdvUnlearn using different retaining
prompt dataset for nudity unlearning.

Source: | ImageNet | COCO Object
---|---|---
LLM Prompt Filter: | ✘ | ✔ | ✘ | ✔
ASR ($\downarrow$) | 51.41% | 45.07% | 38.03% | 21.13%
FID ($\downarrow$) | 19.09 | 19.10 | 18.85 | 19.34
CLIP ($\uparrow$) | 0.301 | 0.299 | 0.293 | 0.290

#### Retain set selection.

We introduce a utility-retaining regularization, defined in (10), designed to
reduce the degradation of image generation utility commonly associated with
adversarial training for unlearning. In Tab. A3, we examine the influence of
object class sources on the retain set, using the template ‘a photo of
[OBJECT]’, and evaluate the effectiveness of employing a Large Language Model
(LLM) as a prompt filter, which helps exclude prompts potentially related to
the concept being erased. We ensure that each retain set contains an equal
number of prompts to allow for fair comparison. Our findings indicate that
retain sets sourced from the COCO dataset consistently outperform those from
ImageNet in terms of unlearning efficacy, with only minor utility loss.
Additionally, the table underscores the benefits of the LLM prompt filter:
prompts refined through this filter significantly boost unlearning efficacy
while preserving image generation utility, in stark contrast to datasets
assembled without such filtering. Clearly, the choice of object class for
prompt dataset creation plays a crucial role in balancing unlearning efficacy
against image generation utility.

Figure A4: Performance comparison of different step numbers of adversarial
prompt generation for AdvUnlearn.

Number of attack steps. In Fig. A4, we delve into the dynamics of adversarial
prompt generation as detailed in (6) within the framework of AdvUnlearn. The
graph distinctly illustrates that as we incrementally increase the number of
attack steps, there is a notable enhancement in the efficacy of the unlearning
process. This improvement is consistently observed up to the threshold of $30$
attack steps, where the curve begins to plateau. Interestingly, during this
escalation, there is only a marginal decline in image generation utility,
suggesting a robust balance between unlearning performance and utility
retention. However, surpassing the $30$ step mark appears to have a negligible
impact on further augmenting unlearning efficacy, indicating an optimal
saturation point in the adversarial training process. This insight is crucial
for optimizing the efficiency of the AdvUnlearn algorithm, as it delineates a
clear boundary for maximizing gains while minimizing resource expenditure.

Table A4: Performance evaluations of AdvUnlearn using various adversarial
prompting for nudity unlearning.

Adversarial Prompting: | Replace | Add | Prefix
---|---|---|---
Metrics: | ASR ($\downarrow$) | 36.63% | 45.07% | 21.13%
FID ($\downarrow$) | 19.39 | 19.60 | 19.34
CLIP ($\uparrow$) | 0.298 | 0.299 | 0.290

#### Adversarial prompting strategy.

We evaluate three distinct adversarial prompting strategies for adversarial
prompt generation in AdvUnlearn: ① Replace: This strategy involves directly
replacing the original concept prompt with an optimized adversarial soft
prompt. ② Add: This method adds the optimized adversarial soft prompt to the
original concept prompt within the token embedding space. ③ Prefix: This
approach prepends the optimized adversarial soft prompt before the original
concept prompt, and is the default setting for our study. As demonstrated in
Tab. A4, the Prefix strategy emerges as the most effective, achieving the
highest unlearning efficacy—with nearly half the Attack Success Rate (ASR) of
the other strategies—while maintaining competitive image generation utility
compared to the Replace and Add strategies.

Table A5: Robustness of SD v1.4 post-nudity unlearning against different test-
time adversarial attacks.

Metrics |  FMN |  SPM |  UCE |  ESD |  SalUn |  AdvUnlearn (Ours)
---|---|---|---|---|---|---
P4D - ASR ($\downarrow$) | 97.89% | 91.55% | 75.35% | 71.27% | 12.68% | 19.72%
UnlearnDiffAtk - ASR ($\downarrow$) | 97.89% | 91.55% | 79.58% | 73.24% | 11.27% | 21.13%

Robustness gain from AdvUnlearn at different test-time attacks. In Tab. A5, we
present the robustness of SD v1.4 post-nudity unlearning when facing different
test-time adversarial prompt attacks, UnlearnDiffAtk (Zhang et al., 2023b)
(default choice) and P4D (Chin et al., 2023). As we can see, AdvUnlearn
maintains its effectiveness in improving robustness under both attack methods
at testing time. Notably, the ASR against P4D is even lower than that against
UnlearnDiffAtk. This result is expected, as P4D employs the same attack loss
(6) used during training for generating adversarial prompts in (9). Therefore,
we default to UnlearnDiffAtk for test-time attacks in robustness evaluation
due to its unseen nature during training.

##  Appendix G Limitations

This work seeks to improve the robustness of concept erasing by incorporating
the principles of adversarial training (AT) into the process of machine
unlearning, resulting in a robust unlearning framework named AdvUnlearn. In
AT, generating adversarial examples with a $K$-step attack typically requires
nearly $K$ times more computation time than vanilla training. Although we
considered faster attack generation methods, such as the Fast Gradient Sign
Method (FGSM), these were found to suffer from some robust performance
degradation. Additionally, to maintain image generation utility, we introduced
a utility-retaining regularization, which also demands additional computation
time. Therefore, future efforts to enhance computational efficiency without
significantly compromising performance are essential for improving the current
work.

##  Appendix H Broader Impacts

The broader impacts of this study include social and ethical implications,
where improved reliability of concept erasing aligns AI technologies with
societal norms and ethical standards, potentially reducing the spread of
harmful digital content. Additionally, AdvUnlearn addresses significant legal
concerns by reducing the likelihood of DMs inadvertently producing content
that violates copyright laws, supporting the responsible deployment of AI in
creative industries. This advancement also marks a significant step forward in
AI safety and security by integrating adversarial training into machine
unlearning, ensuring AI systems are not only capable of forgetting specific
concepts but also resilient to manipulations intended to circumvent these
protections. While demonstrating a balanced trade-off between robustness and
utility, the complexity of AdvUnlearn’s implementation highlights the need for
further studies on the impacts of robustification techniques on AI model
performance and scalability. Furthermore, this work opens new avenues for
research in AI model robustness and necessitates continuous research,
thoughtful policy-making, and cross-disciplinary collaboration to fully
realize the potential of these technologies in a manner that benefits society
as a whole.