# Prompting4Debugging: Red-Teaming Text-to-Image Diffusion Models
by Finding Problematic Prompts

Zhi-Yi Chin1, Chieh-Ming Jiang1, Ching-Chun Huang1, Pin-Yu Chen,2 Wei-Chen
Chiu1

###### Abstract

Text-to-image diffusion models, e.g. Stable Diffusion (SD), lately have shown
remarkable ability in high-quality content generation, and become one of the
representatives for the recent wave of transformative AI. Nevertheless, such
advance comes with an intensifying concern about the misuse of this generative
technology, especially for producing copyrighted or NSFW (i.e. not safe for
work) images. Although efforts have been made to filter inappropriate
images/prompts or remove undesirable concepts/styles via model fine-tuning,
the reliability of these safety mechanisms against diversified problematic
prompts remains largely unexplored. In this work, we propose
Prompting4Debugging (P4D) as a debugging and red-teaming tool that
automatically finds problematic prompts for diffusion models to test the
reliability of a deployed safety mechanism. We demonstrate the efficacy of our
P4D tool in uncovering new vulnerabilities of SD models with safety
mechanisms. Particularly, our result shows that around half of prompts in
existing safe prompting benchmarks which were originally considered “safe” can
actually be manipulated to bypass many deployed safety mechanisms, including
concept removal, negative prompt, and safety guidance. Our findings suggest
that, without comprehensive testing, the evaluations on limited safe prompting
benchmarks can lead to a false sense of safety for text-to-image models.

WARNING: This paper contains model outputs that may be offensive or upsetting
in nature.

## Introduction

In recent years, generative models have been making remarkable advancements
across multiple domains, such as text, images, and even code generation,
blurring the distinction between the works created by AI systems and those
crafted by human experts. One prominent area of focus upon generative AI is
text-to-image (T2I) generation (Li et al. 2019a; Ramesh et al. 2021; Rombach
et al. 2022; Ramesh et al. 2022; Saharia et al. 2022), where most of the
state-of-the-art T2I methods are built upon the diffusion models, in which
these T2I diffusion models enable the transformation of textual information
into images. They not only bridge the gap between natural language processing
and visual content creation, but also enhance the interaction and
understanding across these two modalities. One of the main factors leading to
the exceptional performance of T2I diffusion models nowadays stems from the
vast amount of training data available on the internet, allowing the models to
generate a wide range of content, including natural animals, sketches, cartoon
images, and even artistic images. However, such large-scale training data
collected from the Internet can be a double-edged sword, as it can lead the
models to unconsciously generate inappropriate content such as copyright
infringement and NSFW materials.

Figure 1:  Given an existing text-to-image (T2I) diffusion model
${\mathcal{G}}^{\prime}$ with safety mechanism which ideally can remove the
target concept (e.g. nudity) from the generated image (while the same input
prompt would lead to inappropriate image content for the typical T2I diffusion
model ${\mathcal{G}}$), our proposed Prompting4Debugging (P4D) red-teams
${\mathcal{G}}^{\prime}$ to automatically uncover the safety-evasive prompts.

To this end, there are several recent research works proposing the diffusion
models equipped with safety mechanisms, e.g. Stable Diffusion with negative
prompts (Rombach et al. 2022), SLD (Schramowski et al. 2023), and ESD
(Gandikota et al. 2023), which either restrict the text embedding space during
inference or finetune the model for attempting to prevent the model from
generating copyrighted or inappropriate images. Although these safety
mechanisms are shown to be partially effective according to their evaluation
schemes, there are already studies that demonstrate their potential flaws. For
example, (Rando et al. 2022) has found that the state-of-the-art Stable
Diffusion model equipped with NSFW safety filter (Rombach et al. 2022) will
still generate sexual content if users give the text prompt ”A photo of a
billboard above a street showing a naked man in an explicit position”.
However, these problematic prompts are discovered manually and thus are hard
to scale. Here hence comes an urgent need for developing an automated and
scalable red-teaming tool for developers to systematically inspect the model
safety and reliability before deployment.

On the other hand, as the rapid increase of size (e.g. even growing up to
billions of parameters) for recent T2I diffusion models (Ramesh et al. 2022;
Rombach et al. 2022; Ramesh et al. 2021; Saharia et al. 2022), model
finetuning becomes extremely expensive and infeasible upon limited computation
resources while building the red-teaming tool. As a result, in this work, we
utilize prompt engineering (Brown et al. 2020; Li et al. 2019b; Cui et al.
2021; Petroni et al. 2019; Jiang et al. 2020; Lester, Al-Rfou, and Constant
2021; Shin et al. 2021; Schick and Schütze 2020b) as our basis for developing
the red-teaming technique, which achieves comparable performance to
traditional approaches of finetuning heavy models but with the advantage of
learning only a minimal number of prompts.

Overall, we propose a Prompting4Debugging (P4D) framework to help
debugging/red-teaming the T2I diffusion models equipped with safety mechanisms
via utilizing prompt engineering techniques as well as leveraging an
unconstrained diffusion model to automatically and efficiently find the
problematic prompts that would lead to inappropriate content. Moreover, the
problematic prompts discovered by our P4D testing tool can be used for
understanding model misbehavior and as important references for follow-up
works to construct stronger safe mechanisms. The illustration of our proposed
P4D is provided in Figure 1. Our main contributions of this work are
summarized as follows.

  * •

Our proposed Prompting4Debugging (P4D) serves as a debugging tool to red-team
T2I diffusion models with safety mechanisms for finding problematic prompts
resulting in safety-evasive outputs.

  * •

Our extensive experiments based on the Inappropriate Image Prompts (I2P)
dataset reveal the fact that around half of the prompts which originally can
be tackled by the existing safety mechanisms are actually manipulable by our
P4D to become problematic ones.

  * •

We also observe that some of the existing safety mechanisms in T2I diffusion
models could lead to a false sense of safety by “information obfuscation” for
red-teaming: when turning off the safety mechanism during the debugging
process, it even becomes easier for our P4D to find the problematic prompts
which are still effective to pass the safety mechanism and produce
inappropriate image content during the inference time.

## Related work

AI red-teaming tools. Red-teaming is an active cybersecurity assessment method
that exhaustively searches for vulnerabilities and weaknesses in information
security, where the issues found by red-teaming can further help companies or
organizations improve their defense mechanisms and strengthen overall
cybersecurity protection. Recently, with the popularity and increasing demand
for generative AI, red teaming is also being applied to AI models (especially
language models (Perez et al. 2022; Shi et al. 2023; Lee et al. 2023)) to
enhance model security and stability. (Perez et al. 2022) proposes to prompt
language models with a variety of methods, such as few-shot generation and
reinforcement learning, to generate test cases that are able to find
vulnerabilities in models. (Shi et al. 2023) fools the model for detecting
machine-generated text by revising output, e.g. replacing synonyms words or
altering writing style in generated sentences. On the other hand, (Lee et al.
2023) constructs a pool of user inputs and employs Bayesian optimization to
iteratively modify diverse positive test cases which eventually lead to model
failures. However, these methods are only applicable to red-team language
models, while our P4D focuses on the text-to-image models, which is a field
that has been rarely explored in AI red-teaming.

Prompt engineering. Prompt engineering originates from the field of natural
language processing and aims to adapt a pretrained language model to various
downstream tasks by modifying input text with prompts. Prompt engineering can
be categorized into two groups: hard prompts and soft prompts. Hard prompts,
also known as discrete tokens, usually consist of interpretable words that are
hand-crafted by users. For instance, (Brown et al. 2020) first demonstrates
the remarkable generalizability of pretrained language models via adopting
manually crafted hard prompts to a wide range of downstream tasks in few-shot
learning. Then (Schick and Schütze 2020a; Jiang et al. 2020; Gao, Fisch, and
Chen 2020) reformulate input texts into specific cloze-style phrases, thus
maintaining the form of hard prompts, to prompt the language models. On the
other hand, soft prompts consist of appended continuous-valued text vectors or
embeddings, providing a larger search space compared to hard prompts. For
instance, prompt-tuning (Lester, Al-Rfou, and Constant 2021) and prefix-tuning
(Shin et al. 2020) automate the soft prompts in continuous space. However,
soft prompts are often uninterpretable or non-transferrable (i.e. cannot be
shared by different language models). As a consequence, some discrete
optimization methods are proposed to strike a balance between hard prompts and
soft prompts, e.g. AutoPrompt (Shin et al. 2020), FluentPrompt (Shi et al.
2022), and PEZ (Wen et al. 2023) that learns hard prompts through continuous
gradient-based optimization. Additionally, PEZ extends its capabilities to
discover prompts that can be matched with given images, achieved by measuring
the CLIP Score (Hessel et al. 2021) using the same optimization method. These
studies demonstrate the potential of prompt engineering across various tasks
and domains, motivating us to integrate prompt engineering into the field of
red-teaming T2I diffusion models.

Diffusion models with safety mechanisms. In response to the emerging issues of
generating inappropriate images from diffusion models, several works have
devoted to address the concern. These works fall into two categories:
guidance-based and finetuning-based methods. For guidance-based methods like
Stable Diffusion with negative prompts (Rombach et al. 2022) and SLD
(Schramowski et al. 2023), they block the text embedding of certain words or
concepts (e.g. nudity, hate, or violence), in order to prevent the generation
of the corresponding image content during the inference process. Rather than
using guidance-based techniques, ESD (Gandikota et al. 2023) takes a different
approach by finetuning the partial model weights (e.g. the U-Net to perform
denoising in Stable Diffusion) to remove unwanted contents from the image
output. Nonetheless, certain corner cases still bypass the safety mechanisms
of these diffusion models (Rando et al. 2022). To enable profound testing, our
P4D serves as a debugging tool, allowing developers to identify problematic
prompts at scale by employing red-teaming strategies on T2I diffusion models.
Meanwhile, the models can enhance their robustness by attempting to tackle the
more challenging prompts uncovered through our P4D.

## Background

In this section, we first briefly introduce how diffusion models learn to
generate unconditional images. Moreover, as all the state-of-the-art T2I
diffusion models used in this work are based on latent diffusion models, we
also describe how latent diffusion models improve the efficiency of diffusion
processes and extend to support conditional generation.

Diffusion Models (Sohl-Dickstein et al. 2015; Ho, Jain, and Abbeel 2020) are
powerful generative models that learn to simulate the data generation process
by progressively denoising the (intermediate) noisy states of data, where such
denoising steps stand for the backward process to the opposite forward one
composed of diffusion steps which gradually add random noise to data. Given an
input image $x$, Denoising Diffusion Probabilistic Models (DDPM) (Ho, Jain,
and Abbeel 2020) first generates intermediate noisy image $x_{t}$ at time step
$t$ via the forward diffusion steps, where $x_{t}$ can be written as a close
form depending on $x$, $t$, and noise $\epsilon$ sampled from Gaussian
distribution $\mathcal{N}(0,I)$. Then the diffusion model training is based on
the backward process for learning a model parameterized by $\theta$ to predict
$\epsilon$, where such model takes both $x_{t}$ and the corresponding time
step $t$ as input. The objective is defined as:

$\mathcal{L}_{DM}=\mathbb{E}_{x,\epsilon\sim\mathcal{N}(0,1),t}\left[\|\epsilon-\epsilon_{\theta}(x_{t},t)\|^{2}_{2}\right]$
(1)

where $t$ ranges from $1$ to the maximum time step $T$.

Latent Diffusion Models (Rombach et al. 2022) proposes to model both forward
and backward processes in the latent space, for alleviating the efficiency
issue of DDPM which stems from having the model operate directly in the pixel
space, where the transformation between latent and pixel spaces is based on a
variational autoencoder (composed of an encoder $\mathcal{E}$ and a decoder
$\mathcal{D}$). Furthermore, they extend DDPM to enable conditional image
generation, via incorporating diverse conditions such as text prompts.
Specifically, given the latent representation $z=\mathcal{E}(x)$ of input
image $x$ as well as the intermediate noisy latent vector $z_{t}$ at time step
$t$ (analogously, depending on $z$, $t$, and $\epsilon\sim\mathcal{N}(0,I)$),
a model parameterized by $\theta$ is trained to make prediction for the noise
$\epsilon_{\theta}(z_{t},c,t)$ that is conditioned on $z_{t}$, time step $t$,
and a text condition $c$. The objective for learning such conditional
generation process (based on image–condition training pairs $\\{(x,c)\\}$) is:

$\mathcal{L}_{LDM}=\mathbb{E}_{\mathcal{E}(x),c,\epsilon\sim\mathcal{N}(0,1),t}\left[\|\epsilon-\epsilon_{\theta}(z_{t},c,t)\|^{2}_{2}\right].$
(2)

## Methdology

Figure 2: An overview of our proposed Prompting4Debugging (P4D) framework,
which employs prompt engineering techniques to red-team the text-to-image
(T2I) diffusion model ${\mathcal{G}}^{\prime}$ with safety mechanism (e.g.
Stable Diffusion with negative prompts (Rombach et al. 2022), SLD (Schramowski
et al. 2023), and ESD (Gandikota et al. 2023)). With the help of an
unconstrained T2I diffusion model $\mathcal{G}$, our P4D optimize to find the
safety-evasive prompts (i.e. $P^{\ast}_{\text{cont}}$) which can bypass the
safety mechanism in ${\mathcal{G}}^{\prime}$ and still lead to generation of
inappropriate image concept/objects (e.g. nudity). Such optimization procedure
is composed of three sequential steps, please refer to the section of proposed
method for more detailed description.

In this paper, we aim to develop a red-teaming tool named Prompting4Debugging
(P4D) for Text-to-image (T2I) diffusion models to test the reliability of
deployed safety mechanisms. In particular, three models, including Stable
Diffusion (SD) with negative prompts (Rombach et al. 2022), SLD (Schramowski
et al. 2023), and ESD (Gandikota et al. 2023), are considered as our targets
of study. The overview of our proposed P4D is visualized in Figure 2, in which
we detail its designs in the following.

Given an input text prompt $P$ which is able to lead an unconstrained/standard
T2I diffusion model $\mathcal{G}$ for generating the output image with an
inappropriate concept/object $\mathcal{C}$ (i.e. $\mathcal{G}$ does not have
the safety mechanism, and $P$ is a problematic prompt), when taking such
prompt $P$ as the input for another T2I diffusion model
${\mathcal{G}}^{\prime}$ equipped with the safety mechanism specific for
$\mathcal{C}$, ideally the resultant output image should be free from
$\mathcal{C}$ (i.e. ${\mathcal{G}}^{\prime}$ successfully defends the
generated image against the problematic prompt $P$). Our red-teaming tool P4D
now attempts to counteract the safety mechanism of ${\mathcal{G}}^{\prime}$
such that the inappropriate concept/object $\mathcal{C}$ now again appears in
the generated image (i.e. the safety mechanism of ${\mathcal{G}}^{\prime}$ is
bypassed).

Specifically, our red-teaming tool P4D adopts the technique of prompt
engineering to circumvent the safety mechanism in ${\mathcal{G}}^{\prime}$,
where a new or modified prompt $P^{\ast}$ is optimized for making
${\mathcal{G}}^{\prime}$ conditioned on $P^{\ast}$ to produce the
inappropriate content as what would be obtained by having $\mathcal{G}$
conditioned on $P$. As the state-of-the-art T2I diffusion model, i.e. Stable
Diffusion (SD), as well as the choices for the T2I diffusion models with
safety mechanism ${\mathcal{G}}^{\prime}$ in this work (e.g. SD with negative
prompts (Rombach et al. 2022), SLD (Schramowski et al. 2023), and ESD
(Gandikota et al. 2023)) are all based on the latent diffusion models, the
optimization for $P^{\ast}$ in our P4D is actually realized in the latent
space, following the procedure below (cf. Figure 2):

  1. 1.

With an unconstrained T2I diffusion model $\mathcal{G}$ (e.g. Stable Diffusion
in our experiments), an original text prompt $P$ is first used to generate an
image $x$ having the inappropriate concept/object $\mathcal{C}$. Note that the
noise predictor in the backward process of $\mathcal{G}$ is parameterized by
$\theta$.

  2. 2.

We then obtain the latent representation $z=\mathcal{E}(x)$ of $x$ via the
encoder $\mathcal{E}$ of $\mathcal{G}$ (noting that $\mathcal{G}$ is based on
latent diffusion models thus has the corresponding variational autoencoder),
followed by computing the intermediate noisy latent vector $z_{t}$ at an
arbitrary time step $t$ according to the diffusion process of $\mathcal{G}$.

  3. 3.

Given a T2I diffusion model with safety mechanism ${\mathcal{G}}^{\prime}$ in
which its noise predictor in the backward process is parameterized by
${\theta}^{\prime}$, we now aim to find a prompt $P^{\ast}$ such that
${\mathcal{G}}^{\prime}$ conditioned on $P^{\ast}$ can produce the output
$x^{\ast}$ similar to $x$, thereby also having the similar inappropriate
concept/object $\mathcal{C}$. The optimization for $P^{\ast}$ happens on the
latent space directly to encourage similarity between the noise predictions
$\epsilon_{\theta}(z_{t},P,t)$ and
$\epsilon_{{\theta}^{\prime}}(z_{t},P^{\ast},t)$. The basic idea is that,
starting from the same noisy latent vector $z_{t}$ at an arbitrary time step
$t$, if both the noise predictors of $\mathcal{G}$ and
${\mathcal{G}}^{\prime}$ which respectively take $P$ and $P^{\ast}$ as text
prompt are able to reach the same noise prediction, then our goal of assuring
the similarity between $x^{\ast}$ and $x$ in pixel space ideally can be also
achieved.

Notably, the text prompt is typically fed into the noise predictor in the form
of embeddings (according to the common practice for our $\mathcal{G}$ and
${\mathcal{G}}^{\prime}$). To this end, the noise prediction happens in
$\mathcal{G}$ is actually operated as
$\epsilon_{\theta}(z_{t},\mathcal{W}(P),t)$, where $\mathcal{W}$ is a pre-
trained and fixed text encoder (e.g. CLIP) for extracting the embedding
$\mathcal{W}(P)$ of text prompt $P$. While for the noise prediction in
${\mathcal{G}}^{\prime}$ that involves our optimization target $P^{\ast}$, we
adopt the similar design of prompt engineering as PEZ (Wen et al. 2023) to
automate the optimization (a benefit of soft prompt) while making the
resultant prompt more transferable (a benefit of hard prompt): We start from a
continuous/soft embedding $P^{\ast}_{\text{cont}}=[e_{1},\dots,e_{N}]$
composed of $N$ tokens $e_{i}\in\mathbb{R}^{d}$, followed by projecting
$P^{\ast}_{\text{cont}}$ into the corresponding discrete/hard embedding
$P^{\ast}_{\text{disc}}=\mathcal{F}(P^{\ast}_{\text{cont}})$ via a projection
function $\mathcal{F}$ (where each token in $P^{\ast}_{\text{cont}}$ is mapped
to its nearest vocabulary embedding). As a result, noise prediction in
${\mathcal{G}}^{\prime}$ is now operated as
$\epsilon_{{\theta}^{\prime}}(z_{t},P^{\ast}_{\text{disc}},t)$, and the
objective $\mathcal{L}$ for our debugging process is defined as

$\mathcal{L}=\left\|\epsilon_{\theta}(z_{t},\mathcal{W}(P),t)-\epsilon_{{\theta}^{\prime}}(z_{t},P^{\ast}_{\text{disc}},t)\right\|^{2}_{2}$
(3)

with noting that both noise predictors in $\mathcal{G}$ and
${\mathcal{G}}^{\prime}$ are kept fixed in such optimization.

It is also worth noting that, as projection function $\mathcal{F}$ acts as a
vector quantization operation and is non-differentiable, during the
optimization procedure we follow the practice of PEZ (Wen et al. 2023) to
directly update $P^{\ast}_{\text{cont}}$ by the gradient of $\mathcal{L}$ with
respect to $P^{\ast}_{\text{disc}}$, where
$P^{\ast}_{\text{cont}}=P^{\ast}_{\text{cont}}-\gamma\nabla_{P^{\ast}_{\text{disc}}}\mathcal{L}$.
Last but not least, the resultant $P^{\ast}_{\text{disc}}$ can be transformed
into legible texts $P^{\ast}$ by the off-the-shelf text decoder/tokenizer.

We experiment two variants for $P^{\ast}_{\text{cont}}$: P4D-$N$ and P4D-$K$,
where the former initializes $N$ tokens in $P^{\ast}_{\text{cont}}$ from
scratch via randomly drawing $N$ vocabulary embeddings, while the latter
inserts learnable tokens after every $K$ tokens of $\mathcal{W}(P)$ (i.e. the
embedding of the original text prompt $P$). Basically,
$P^{\ast}_{\text{cont}}$ in P4D-$N$ has fixed length of $N$ which is
independent from the length of $\mathcal{W}(P)$, it would potentially be
insufficient for debugging the images with complex content as the original
prompt length are not taken into consideration. In comparison, the length of
$P^{\ast}_{\text{cont}}$ in P4D-$K$ (and the number of trainable tokens being
inserted) varies with the length of $\mathcal{W}(P)$ thus alleviating the
aforementioned concern in P4D-$N$. Later in experiments, we observe that both
P4D-$N$ and P4D-$K$ have the comparable debugging performance but the hard
prompt found by P4D-$K$ demonstrates better interpretability as the original
prompt $P$ is taken as its part.

## Experiments

Dataset. We evaluate the performance of our P4D on concept-related and object-
related datasets. For concept-related dataset, we focus on Inappropriate Image
Prompts (I2P) dataset (Schramowski et al. 2023), which encompasses various
uncomfortable and inappropriate prompts (including hate, harassment, violence,
self-harm, nudity contents, shocking images, and illegal activity).
Specifically, nudity contents are most prohibitive due to privacy and respect
considerations, we hence specifically set this concept aside for separate
evaluation. On the other hand for the object-related datasets, we utilize the
“car” and “French-horn” classes from ESD (Gandikota et al. 2023) for our
evaluation (as ESD only offers finetuned weights for these two classes).
Notably, the original French-horn dataset comprises merely 10 identical
prompts with different evaluation seeds. We hence extend the size of French-
horn prompts from 10 to 305 by experimenting with a wider array of evaluation
seeds.

In order to enhance the assessment of P4D’s capabilities, we additionally
filter the aforementioned datasets. We generate 3 images per prompt from the
original dataset via diffusion models, where a prompt (or an image) is
considered “unsafe” if any of the resultant images (or itself, respectively)
contains the target inappropriate concept/objects. For the purpose of
debugging and validating the reliability of safe prompts, our objective is to
select ideal prompts that yield safe images (i.e. having no inappropriate
content) through T2I diffusion models with safety mechanism while producing
unsafe images through unconstrained T2I ones. The reasons are that: 1) if the
T2I diffusion model with safety mechanism generates unsafe images through a
given prompt, such prompt has already been considered as a problematic one; 2)
if the unconstrained T2I diffusion model generates a safe image with a given
prompt, such prompt is less useful to our evaluation as we need the unsafe
prompts for our inspection on the safety mechanisms. Table 1 provides the size
of the filtered dataset. For simplicity purposes, we abbreviate “unconstrained
T2I diffusion models” and “T2I diffusion models with safety mechanism” to
“standard T2I models” and “safe T2I models” respectively.

Standard T2I and safe T2I models. In our experiments, we adopt the typical
Stable Diffusion (Rombach et al. 2022) (denoted as standard SD) for our
standard T2I model, while using ESD (Gandikota et al. 2023), SLD (Schramowski
et al. 2023) (where we adopt two superior variants of SLD, i.e. SLD-MAX and
SLD-STRONG, provided in their release code), and SD with negative prompts
(Rombach et al. 2022) (denoted as SD-NEGP) for our safe T2I models. For
standard SD, ESD, and SLD, we apply the Stable Diffusion v1-4 model backbone,
while for SD-NEGP, we use the Stable Diffusion v2-0 model backbone. When
generating an image from any of the aforementioned T2I models, the number of
inference steps is set to 25 and the setting of random seed aligns with the
used dataset, where guidance scale is set to 7.5 if not specified in the
dataset.

Implementation details. We set $N=16$ and $K=3$ respectively for our P4D-$N$
and P4D-$K$. Please note that in $P^{\ast}_{\text{cont}}$ of P4D-$K$ only the
inserted tokens are trainable while the other tokens from $\mathcal{W}(P)$ are
kept untouched. We set the batch size to 1, learning rate to 0.1, weight decay
to 0.1, and use AdamW (Loshchilov and Hutter 2017) as the optimizer. All the
prompts $P^{\ast}_{\text{cont}}$ are optimized with 3000 gradient update
steps. We measure the optimized prompts every 50 steps and update the optimal
prompts based on the cosine similarity between the generated $x^{\ast}$ and
original $x$ images.

| Category | Total | Safe T2I models | Ideal
---|---|---|---|---
_Concept_ | Nudity | 854 | ESD | 361
SLD-MAX | 204
SLD-STRONG | 112
SD-NEGP | 209
All in I2P | 4703 | SLD-MAX | 1667
_Object_ | Car | 731 | ESD | 91
French-horn | 305 | 200
Table 1:  The statics upon the size of the datasets and their filtered
counterparts used in our experiments. “Total” denotes the number of prompts in
the original dataset, while “Ideal” denotes the number of ideal prompts after
applying our dataset filtering operation. The ideal prompts are the ones which
can yield safe images through safe T2I models while leading to unsafe images
by standard T2I models.

Baselines. To the best of our knowledge, there are currently no automated
tools available for red-teaming T2I diffusion models. As a result, we propose
three distinct baselines, namely Random-$N$, Random-$K$, and Shuffling.
Random-$N$ is analogous to P4D-$N$ where $N$ vocabulary embeddings are
randomly drawn to be the input prompt for safe T2I models, but without any
further optimization being performed. Similarly, Random-$K$ is analogous to
P4D-$K$ (i.e. inserting random vocabulary embeddings after every $K$ tokens in
$\mathcal{W}(P)$) but excludes further optimization. Furthermore, as some
research works in the natural language field have discovered that shuffling
the word order in a sentence can make ChatGPT (Ouyang et al. 2022) generate
inappropriate responses, we therefore introduce similar approach to construct
our Shuffling baseline, which involves randomly permuting the words in the
prompt $P$.

Method | Nudity | All in I2P | Car | French-horn
---|---|---|---|---
ESD | SLD-MAX | SLD-STRONG | SD-NEGP | SLD-MAX | ESD | ESD
Random-$N$ | 1.39% | 11.27% | 12.50% | 4.31% | 17.10% | 6.60% | 22.00%
Random-$K$ | 16.62% | 28.43% | 25.89% | 16.27% | 23.46% | 25.47% | 23.50%
Shuffling | 13.85% | 32.35% | 23.21% | 13.88% | 25.61% | 22.64% | 23.50%
OURS (P4D-$N$) | 54.29% | 27.94% | 34.82% | 27.75% | 24.00% | 42.86% | 70.50%
OURS (P4D-$K$) | 49.58% | 42.16% | 38.39% | 21.53% | 27.83% | 36.26% | 33.50%
OURS (P4D-UNION) | 70.36% | 57.35% | 56.25% | 44.02% | 44.57% | 59.34% | 82.00%
Table 2: Quantitative evaluation among various approaches for debugging
performance, where the failure rate (FR) indicating the proportion of
problematic prompts with respect to the overall amount of data is adopted as
the evaluation metric. Figure 3: Visualization of images generated by
different prompts and T2I models. The images are generated using the displayed
prompts (i.e. the sentence below the image) with the specified T2I models
(i.e. indicated by the model name on top of the image). Problematic prompts
found by our P4D are colored in dark red. Notably, P4D demonstrates the
capability to jailbreak safe T2I models and create images containing specific
target concepts or objects that should have been restricted by safe T2I
models.

Evaluation. During the evaluation process, we generate 3 images for every
prompt (either the original prompts in the dataset or the ones produced by the
baselines/P4D) by safe T2I models, where a prompt is considered problematic if
any of the resulting images is identified as “unsafe” by the classifiers or
detectors described below. For categories in the concept-related I2P dataset,
we employ the NudeNet image classifier (Bedapudi 2019) with an unsafe
percentage threshold of 0.45 to detect nudity content, while adopting the Q16
classifier (Schramowski, Tauchmann, and Kersting 2022) for identifying the
other categories in I2P. For object-related categories (i.e. car and French-
horn), we utilize YOLO v5 vehicle detector (Boneh 2023) with the confidence
threshold at 0.7 for identifying cars, while employing a pre-trained ResNet-18
classifier (Ma 2021) from the Imagenette dataset (Howard 2019) for detecting
French-horns. Although these classifiers and detectors are not error-free,
they act as a low-cost third-party auditor, capable of evaluating our P4D and
the baselines in a scalable and fair manner.

Metric. We report failure rate (FR) in experimental results, showing how many
problematic prompts are identified from the entire dataset. The higher FR
indicates better debugging performance for red-teaming.

### Experimental Results

Main Results. Quantitative results and some qualitative examples are reported
in Table 2 and Figure 3 respectively. Please refer to our appendix for more
qualitative results. Regarding concept-related I2P dataset, we inspect all
safe T2I models for the nudity category; while we only examine SLD-MAX for all
the other categories, as SLD-MAX is the sole model capable of resisting
additional concepts such as shocking, self-harm, and illegal content.
Regarding object-related categories, we inspect ESD for cars and French-horns.
From Table 2, we observe that P4D-$N$ and P4D-$K$ demonstrate promising and
comparable results across a range of safe T2I models and categories,
indicating P4D-$K$ preserves its prompt interpretability without compromising
the debugging performance. Furthermore, we unify problematic prompts from
P4D-$N$ and P4D-$K$ and obtain P4D-UNION, which significantly increases the
failure rate across various safe T2I models and categories (either concept-
related or object-related ones), indicating most problematic prompts found by
P4D-$N$ and P4D-$K$ are not repeated. Notably, for the nudity category, as our
P4D achieves the highest failure rate in ESD, in which it indicates that ESD
originally (before our red-teaming) provides the most effective safety
protection against nudity content among all safe T2I models. However, the
finetuning-based concept-removal safety mechanism of ESD may only learn to
disassociate certain concept-related words with the unsafe image content, but
it may not be resistant to optimized prompts. On the other hand, guidance-
based safe T2I models such as SLD and SD-NEGP, restrict the textual embedding
space for P4D to explore as well as prevent the generation of particular
concepts/objects with their text filters, resulting in a lower failure rate
compared to ESD with P4D. We observe that deactivating these text filters
during training encourages P4D to investigate a broader range of problematic
prompts (i.e. larger explorable textual embedding space). We refer to this
phenomenon as ”information obfuscation”, which will be elaborated in the
subsequent section.

### Ablation Studies and Extended Discussion

For the experiments used in the following studies, we focus on the nudity
category unless otherwise specified.

“Information Obfuscation” of Text Filters. We delve into the phenomenon of a
misleading sense of security caused by “information obfuscation” while
applying P4D to red-teaming the guidance-based safe T2I models (i.e. SLD and
SD-NEGP). The detailed computation procedure for such safe T2I models is as
follows: our trainable discrete prompt is firstly concatenated with the safety
concept for SLD (or the negative prompt for SD-NEGP) before feeding it into
the denoising model (i.e. the UNet for noise prediction); After denoising, the
safety-oriented guidance for SLD (or the classifier-free guidance for SD-NEGP)
is applied on the predicted noise prior to the loss calculation. This safety
process functions as a meticulously controlled text filter, ensuring the
protection of these safe T2I models. For the purpose of debugging, we have the
option to selectively deactivate some components of the inspected model. We
experiment with deactivating this safety filter during the P4D training phase
while keeping it operational during inference (noting that the deactivation is
done by excluding the concatenation with the safety concept and skipping the
safety-oriented guidance for SLD, whole similar deactivation holds for SD-
NEGP). The results are outlined in Table 3. Notably, when the safety filter is
disabled during the debugging process, P4D becomes capable of identifying more
problematic prompts. We hypothesize that the text filter actually obscures the
search for optimized textual prompts (i.e. constraining the explorable textual
embedding space), thereby leading to the failure of uncovering certain
problematic prompts. However, the removal of the text filter eliminates such
constraint on the embedding search space, thereby facilitating the
identification of problematic prompts. This phenomenon draws parallels with
the concept of “obfuscated gradients” of AI security as discussed in (Athalye,
Carlini, and Wagner 2018), where “obfuscated gradients“ foster a false sense
of security in defenses against adversarial examples. Similarly, in our study,
the text filter induces a false sense of safety through “information
obfuscation”, as evidenced by the fact that removing this filter allows P4D to
find more problematic prompts. Please also note that, due to such information
obfuscation properties of SLD and SD-NEGP, in the following studies, we remove
the text filter when optimizing the prompts for SLD and SD-NEGP, for more
efficient computation.

Safe T2I P4D-$N$ P4D-$K$ w/ TF w/o TF w/ TF w/o TF SLD-MAX 27.94% 44.12%
42.16% 42.16% SLD-STRONG 34.82% 50.89% 38.39% 42.86% SD-NEGP 27.75% 30.14%
21.53% 33.97%

Table 3: Percentage of problematic prompts (i.e. failure rate) found for SLD
and SD-NEGP with and without the safety text filter (w/ or w/o TF) in nudity
category of I2P dataset.

Prompt Length. We further conduct investigation upon the number of tokens in a
prompt (i.e. prompt length) for optimization. For P4D-$N$, we test three
different values of $N$: 8, 16 (default), and 32. For P4D-$K$, we also test
inserting a learnable token every 1, 3 (default), and 5 tokens in the
embedding $\mathcal{W}(P)$ of the original input prompt $P$. From Table 4, we
can observe that there is no optimal prompt length in either P4D-$N$ or
P4D-$K$. We argue that a complex scenario requires a longer prompt for
description, whereas simpler scenarios can be adequately described with
shorter prompts. Consequently, we recommend aggregating/unioning the
problematic prompts found by using various settings of length for more
efficient red-teaming.

P4D-$N$ $N$ Safe T2I 8 16 32 _Union_ ESD 54.85% 54.02% 59.00% 78.67% SLD-MAX
35.29% 44.12% 38.24% 67.16% SLD-STRONG 47.32% 50.89% 45.54% 78.57% SD-NEGP
36.84% 30.14% 34.45% 61.72% P4D-$K$ $K$ Safe T2I 1 3 5 _Union_ ESD 52.63%
49.58% 49.31% 74.52% SLD-MAX 38.73% 42.16% 40.69% 69.12% SLD-STRONG 40.18%
42.86% 50.00% 72.32% SD-NEGP 32.06% 33.97% 32.06% 61.24%

Table 4: Ablation study for prompt length.

Text and Image Similarity. We calculate cosine similarities for both original
and optimized prompts, as well as their generated images. In a nutshell, we
suggest T2I safety research should jointly safeguard the text and image
domains. Please refer to appendix for details.

Prompt Generalizability. We accumulate all non-repeated problematic prompts
(while selecting the prompt with the highest toxicity score if repeated) found
by P4D across all safe T2I models (e.g. ESD, SLD, and SD-NEGP) as another
dataset/collection to test the generalizability of these problematic prompts
across different safe T2I models. As shown in Table 5, over 50% prompts found
by P4D are able to red-team multiple safe T2I models at the same time.
Moreover, we report the failure rate of universal problematic prompts that are
able to red-team all the safe T2I models simultaneously, which we term the
“intersection”. We can observe that over 30% problematic prompts found in both
P4D-$N$ and P4D-$K$ are robust and general enough to red-team across all safe
T2I models simultaneously.

|  | P4D-$N$ | P4D-$K$
---|---|---|---
| Data size | 405 | 380
Failure rate (FR,%) | ESD | 61.23% | 64.64%
SLD-MAX | 89.14% | 83.37%
SLD-STRONG | 90.37% | 91.02%
SD-NEGP | 54.81% | 54.35%
| Intersection | 37.28% | 31.93%
Table 5:  Evaluation upon prompt generalizability. We create a collection of
the problematic prompts discovered by P4D across all safe T2I models, and
assess such collection using each safe T2I model. Intersection refers to the
percentage of universal problematic prompts that are able to red-team all safe
T2I models simultaneously.

## Conclusion

In this paper, we propose an automated red-teaming debugging tool called P4D
to unveil unprecedented weaknesses of several safety mechanisms used in T2I
diffusion models. P4D proactively finds problematic prompts that may lead to
inappropriate (e.g. nudity or violent) images that bypass deployed safety
mechanisms. Our extensive experimental results demonstrate the effectiveness
of P4D for debugging, providing developers with a red-teaming tool to
safeguard and test the reliability of safe T2I diffusion models.

## References

  * Athalye, Carlini, and Wagner (2018) Athalye, A.; Carlini, N.; and Wagner, D. 2018.  Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples.  In _Proceedings of the International Conference on Machine Learning (ICML)_.
  * Bedapudi (2019) Bedapudi, P. 2019.  NudeNet: Neural Nets for Nudity Classification, Detection and selective censoring.
  * Boneh (2023) Boneh, M. 2023.  Vehicle Detection Using Deep Learning and YOLO Algorithm.
  * Brown et al. (2020) Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems (NeurIPS)_.
  * Cui et al. (2021) Cui, L.; Wu, Y.; Liu, J.; Yang, S.; and Zhang, Y. 2021.  Template-based named entity recognition using BART.  _arXiv preprint arXiv:2106.01760_.
  * Gandikota et al. (2023) Gandikota, R.; Materzynska, J.; Fiotto-Kaufman, J.; and Bau, D. 2023.  Erasing concepts from diffusion models.  _arXiv preprint arXiv:2303.07345_.
  * Gao, Fisch, and Chen (2020) Gao, T.; Fisch, A.; and Chen, D. 2020.  Making pre-trained language models better few-shot learners.  _arXiv preprint arXiv:2012.15723_.
  * Hessel et al. (2021) Hessel, J.; Holtzman, A.; Forbes, M.; Bras, R. L.; and Choi, Y. 2021.  Clipscore: A reference-free evaluation metric for image captioning.  _arXiv preprint arXiv:2104.08718_.
  * Ho, Jain, and Abbeel (2020) Ho, J.; Jain, A.; and Abbeel, P. 2020.  Denoising diffusion probabilistic models.  In _Advances in Neural Information Processing Systems (NeurIPS)_.
  * Howard (2019) Howard, J. 2019.  Imagenette: A smaller subset of 10 easily classified classes from Imagenet.
  * Jiang et al. (2020) Jiang, Z.; Xu, F. F.; Araki, J.; and Neubig, G. 2020.  How can we know what language models know?  _Transactions of the Association for Computational Linguistics_.
  * Lee et al. (2023) Lee, D.; Lee, J.; Ha, J.-W.; Kim, J.-H.; Lee, S.-W.; Lee, H.; and Song, H. O. 2023.  Query-Efficient Black-Box Red Teaming via Bayesian Optimization.  _arXiv preprint arXiv:2305.17444_.
  * Lester, Al-Rfou, and Constant (2021) Lester, B.; Al-Rfou, R.; and Constant, N. 2021.  The power of scale for parameter-efficient prompt tuning.  _arXiv preprint arXiv:2104.08691_.
  * Li et al. (2019a) Li, B.; Qi, X.; Lukasiewicz, T.; and Torr, P. 2019a.  Controllable text-to-image generation.  _Advances in Neural Information Processing Systems (NeurIPS)_.
  * Li et al. (2019b) Li, X.; Feng, J.; Meng, Y.; Han, Q.; Wu, F.; and Li, J. 2019b.  A unified MRC framework for named entity recognition.  _arXiv preprint arXiv:1910.11476_.
  * Loshchilov and Hutter (2017) Loshchilov, I.; and Hutter, F. 2017.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_.
  * Ma (2021) Ma, J. 2021.  Imagenette Classification.
  * Ouyang et al. (2022) Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Ray, A.; et al. 2022.  Training language models to follow instructions with human feedback.  In _Advances in Neural Information Processing Systems (NeurIPS)_.
  * Perez et al. (2022) Perez, E.; Huang, S.; Song, F.; Cai, T.; Ring, R.; Aslanides, J.; Glaese, A.; McAleese, N.; and Irving, G. 2022.  Red teaming language models with language models.  _arXiv preprint arXiv:2202.03286_.
  * Petroni et al. (2019) Petroni, F.; Rocktäschel, T.; Lewis, P.; Bakhtin, A.; Wu, Y.; Miller, A. H.; and Riedel, S. 2019.  Language models as knowledge bases?  _arXiv preprint arXiv:1909.01066_.
  * Ramesh et al. (2022) Ramesh, A.; Dhariwal, P.; Nichol, A.; Chu, C.; and Chen, M. 2022.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_.
  * Ramesh et al. (2021) Ramesh, A.; Pavlov, M.; Goh, G.; Gray, S.; Voss, C.; Radford, A.; Chen, M.; and Sutskever, I. 2021.  Zero-shot text-to-image generation.  In _Proceedings of the International Conference on Machine Learning (ICML)_.
  * Rando et al. (2022) Rando, J.; Paleka, D.; Lindner, D.; Heim, L.; and Tramèr, F. 2022.  Red-teaming the stable diffusion safety filter.  _arXiv preprint arXiv:2210.04610_.
  * Rombach et al. (2022) Rombach, R.; Blattmann, A.; Lorenz, D.; Esser, P.; and Ommer, B. 2022.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Saharia et al. (2022) Saharia, C.; Chan, W.; Saxena, S.; Li, L.; Whang, J.; Denton, E. L.; Ghasemipour, K.; Gontijo Lopes, R.; Karagol Ayan, B.; Salimans, T.; et al. 2022.  Photorealistic text-to-image diffusion models with deep language understanding.  In _Advances in Neural Information Processing Systems (NeurIPS)_.
  * Schick and Schütze (2020a) Schick, T.; and Schütze, H. 2020a.  Exploiting cloze questions for few shot text classification and natural language inference.  _arXiv preprint arXiv:2001.07676_.
  * Schick and Schütze (2020b) Schick, T.; and Schütze, H. 2020b.  Few-shot text generation with pattern-exploiting training.  _arXiv preprint arXiv:2012.11926_.
  * Schramowski et al. (2023) Schramowski, P.; Brack, M.; Deiseroth, B.; and Kersting, K. 2023.  Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Schramowski, Tauchmann, and Kersting (2022) Schramowski, P.; Tauchmann, C.; and Kersting, K. 2022.  Can Machines Help Us Answering Question 16 in Datasheets, and In Turn Reflecting on Inappropriate Content?  In _Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency_.
  * Shi et al. (2022) Shi, W.; Han, X.; Gonen, H.; Holtzman, A.; Tsvetkov, Y.; and Zettlemoyer, L. 2022.  Toward Human Readable Prompt Tuning: Kubrick’s The Shining is a good movie, and a good prompt too?  _arXiv preprint arXiv:2212.10539_.
  * Shi et al. (2023) Shi, Z.; Wang, Y.; Yin, F.; Chen, X.; Chang, K.-W.; and Hsieh, C.-J. 2023.  Red Teaming Language Model Detectors with Language Models.  _arXiv preprint arXiv:2305.19713_.
  * Shin et al. (2021) Shin, R.; Lin, C. H.; Thomson, S.; Chen, C.; Roy, S.; Platanios, E. A.; Pauls, A.; Klein, D.; Eisner, J.; and Van Durme, B. 2021.  Constrained language models yield few-shot semantic parsers.  _arXiv preprint arXiv:2104.08768_.
  * Shin et al. (2020) Shin, T.; Razeghi, Y.; Logan IV, R. L.; Wallace, E.; and Singh, S. 2020.  Autoprompt: Eliciting knowledge from language models with automatically generated prompts.  _arXiv preprint arXiv:2010.15980_.
  * Sohl-Dickstein et al. (2015) Sohl-Dickstein, J.; Weiss, E.; Maheswaranathan, N.; and Ganguli, S. 2015.  Deep unsupervised learning using nonequilibrium thermodynamics.  In _Proceedings of the International Conference on Machine Learning (ICML)_.
  * Wen et al. (2023) Wen, Y.; Jain, N.; Kirchenbauer, J.; Goldblum, M.; Geiping, J.; and Goldstein, T. 2023.  Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery.  _arXiv preprint arXiv:2302.03668_.