# Automated Adversarial Discovery for Safety Classifiers
Warning: This paper contains model outputs that may be offensive or upsetting.

Yash Kumar Lal1,2, Preethi Lahoti2, Aradhana Sinha2, Yao Qin2,3∗, Ananth
Balashankar2

1Stony Brook University, 2Google Research, 3University of California, Santa
Barbara
<EMAIL_ADDRESS>Work done at Google

###### Abstract

Safety classifiers are critical in mitigating toxicity on online forums such
as social media and in chatbots. Still, they continue to be vulnerable to
emergent, and often innumerable, adversarial attacks. Traditional automated
adversarial data generation methods, however, tend to produce attacks that are
not diverse, but variations of previously observed harm types. We formalize
the task of automated adversarial discovery for safety classifiers - to find
new attacks along previously unseen harm dimensions that expose new weaknesses
in the classifier. We measure progress on this task along two key axes (1)
adversarial success: does the attack fool the classifier? and (2) dimensional
diversity: does the attack represent a previously unseen harm type? Our
evaluation of existing attack generation methods on the CivilComments toxicity
task reveals their limitations: Word perturbation attacks fail to fool
classifiers, while prompt-based LLM attacks have more adversarial success, but
lack dimensional diversity. Even our best-performing prompt-based method finds
new successful attacks on unseen harm dimensions of attacks only 5% of the
time. Automatically finding new harmful dimensions of attack is crucial and
there is substantial headroom for future research on our new task.

Automated Adversarial Discovery for Safety Classifiers
Warning: This paper contains model outputs that may be offensive or upsetting.

Yash Kumar Lal1,2††thanks:  Work done at Google, Preethi Lahoti2, Aradhana
Sinha2, Yao Qin2,3∗, Ananth Balashankar2 1Stony Brook University, 2Google
Research, 3University of California, Santa Barbara<EMAIL_ADDRESS>

##  1 Introduction

The widespread deployment of large language models (LLMs) has also led to the
rapid discovery of new vulnerabilities where safety classifiers, such as those
used to regulate user forums, do not generalize well Balashankar et al.
(2023). These safety classifiers are trained on data that contains known
dimensions (or types) of attacks, like hateful content. However, such safety
classifiers remain vulnerable to new types/dimensions of attacks that may
emerge after deployment Vidgen et al. (2021). Weaknesses are fixed either by
adversarially training on data collected through costly red teaming Kiela et
al. (2021) for new dimensions or by using failure cases found after
deployment. In this paper, we propose a new proactive adversarial testing task
to automatically find novel and diverse adversarial examples that can be used
to evaluate and mitigate vulnerabilities in safety classifiers.

Specifically, we formalize the task of automated adversarial discovery for
safety classifiers and evaluate the generated examples for their adversarial
nature and diversity with respect to prior known attacks. A generated example
must have two characteristics: (1) it should produce an error from a safety
classifier, and (2) it should not be related to any previously known attack
type or dimension. We propose an evaluation framework that balances
adversarial success as well as dimensional diversity to measure progress on
this task. We benchmark a variety of adversarial attack generation methods on
our task empirically, and find that they do not produce novel and diverse
attacks.

Figure 1:  For a given user comment, the WordNet approach probabilistically
replaces words in the comment with its synonym from WordNet. Polyjuice uses
GPT-2 to rewrite the user comment by incorporating various counterfactual
types such as phrase swaps in a way that the parse tree of the comment is not
altered. Our method, Discover-Adapt, aims to generate adversarial examples
that may also contain new toxicity types either by leveraging latent unlabeled
dimensions present in the seed comment, or drawing from the LLM priors. Using
this discovered unlabeled dimension, we adapt the input user comment to add an
unseen dimension of toxicity. In this example, Discover-Adapt transforms an
insult to an identity attack, which is the unseen labeled dimension. Our
analysis shows that such successful attacks are hard to generate ($\sim 5\%$),
and identifies areas of improvement.

Figure 1 presents details and characteristics of attack generation methods
that we explore for this task. Simple text perturbation methods Wei and Zou
(2019); Li et al. (2020); Calderon et al. (2022); Wang et al. (2020) aim to
avoid label noise, and are therefore limited in the strength of adversarial
examples they can generate. While LM based guided generation methods Wu et al.
(2021); Sinha et al. (2023) generate more adversarial attacks, they do not
generalize well to new dimensions. We evaluate a discover-adapt prompting LLM-
based technique that first discovers possible attack dimensions before
generating examples adapted to it and find that the generated attacks do not
balance the adversarial success and dimensional diversity aspects of our
evaluation framework.

Our key contributions are:

  * •

Task and Evaluation: We formalize the task of automatically generating new
dimensions of adversarial attacks against safety classifiers. We also propose
an evaluation framework based on adversarial success as well as LLM-based
dimensional diversity.

  * •

Empirical Analysis: For toxic comment generation, we benchmark various methods
to generate adversarial attacks that belong to previously unseen dimensions.
At best, current methods produce dimensionally diverse and adversarial attacks
5% of the time. This shows that our task is challenging, and improving on it
can positively impact the adversarial robustness of safety classifiers.

##  2 Related Work

Prior work has explored different methods to generate adversarial data for a
variety of models.

#### Lexical perturbation

Character-level methods manipulate texts by incorporating errors into words,
using operations such as deleting, repeating, replacing, swapping, flipping,
inserting, and allowing variations in characters for specific words Gao et al.
(2018); Belinkov and Bisk (2018). Word-level attacks alter entire words rather
than individual characters within words, which tend to be less perceptible to
humans than character-level attacks Ren et al. (2019); Li et al. (2020); Garg
and Ramakrishnan (2020).

#### LM-based perturbation

CAT-Gen Wang et al. (2020) perturbs an input sentence by varying different
attributes of that sentence. Li et al. (2020) find the most vulnerable word in
the input, mask it, and uses BERT to replace them. Polyjuice Wu et al. (2021)
use control codes to guide generation of adversarial examples towards pre-
decided desirable characteristics. These methods, while effective, result in
data that is very similar to the seed it was generated from.

#### Guided adversarial generation

Conditioned recurrent language models Ficler and Goldberg (2017) produce
language with user-selected properties such as sentence length. Guided
adversarial generation methods have also been used to produce adversarial
examples in different domains. Iyyer et al. (2018) propose syntactically
controlled paraphrase networks to generate adversarial examples for the SST
dataset Socher et al. (2013). Zhang et al. (2020) present a comprehensive
survey of such attack methods. ToxiGen Hartvigsen et al. (2022) uses prompt
engineering to steer models towards generating hard-to-detect hate speech
against different minority groups using constrained ALICE decoding. While this
method leverages the strength of GPT-3, it only focuses on known toxicity
types.

#### LLM-based methods

Garg et al. (2019) and Ribeiro et al. (2020) use templates to test the
fairness and robustness of the text classification models. Sinha et al. (2023)
generate adversarial data that mimic gold adversarial data itself and use it
to improve robustness of classifiers. Lahoti et al. (2023) generate samples of
critiques for input text targeting diversity in certain aspects and aggregate
them as feedback to generate more diverse representations of people. While
these methods allow for lexically diverse data, they are unable to explore
different dimensions than the seed data.

#### Red-teaming methods

Perez et al. (2022) use the output of a good quality classifier as a reward
and train the red-teamer model to produce some inputs that can maximize the
classifier score on the target model output. Rainbow Teaming Samvelyan et al.
(2024) discovers diverse adversarial prompts but requires apriori knowledge of
dimensions to explore. Explore, Establish, Exploit Casper et al. (2023) set up
a human-in-the-loop red teaming process with an explicit data sampling stage
for the target model to collect human labels that can be used to train a task-
specific red team classifier. FLIRT Mehrabi et al. (2023) uses in-context
learning in a feedback loop to red team models and trigger them into unsafe
content generation. Gradient-Based Red Teaming (GBRT) Wichers et al. (2024)
automatically generates diverse prompts that are likely to cause an LM to
output unsafe responses. These methods are not within our scope as our problem
formulation does not assume access to the weights of the generator.

#### Human-in-the-loop methods

Prior work has also explored using explicit human feedback to generate various
types of toxic content. Dinan et al. (2019) propose a build it, break it, fix
it scheme, which repeatedly discovers failures of toxicity classifiers from
human-model interactions and fixes it by retraining to enhance the robustness
of the classifiers. AART Radharapu et al. (2023) use humans to write prompts
that generate desired concepts from LLMs, and then use those LLMs to generate
adversarial examples along those concepts. They also use humans to evaluate
the quality of their generated examples. This requires expert human
intervention when adding a new domain. With the fast-paced and large-scale
deployment of LLMs, it is important to be able to automatically generate
effective adversarial examples for their safety classifiers.

##  3 Problem Formulation

We assume access to a blackbox classifier which takes text as input and makes
a binary prediction. Given a set of text inputs, the task is to generate a
larger, more diverse set of adversarial texts that can produce errors from the
classifier. The generated examples should (1) have the same label as the
inputs, (2) have high adversarial success, and (3) be more diverse than the
inputs.

#### Dimensions

Any text can be categorized into groups based on its characteristics. These
groups are referred to as dimensions, and are task-dependent attributes. For
example, dimensions for the toxic comment generation task may be insults or
threats. We define the diversity of a set of texts as a function of the
dimensions it contains.

###  3.1 Task Objective

Let $f(x)$ be the classifier prediction for input $x\in X$ whose gold label is
denoted by $y_{x}\in Y$. Accordingly, let $u_{x}$ be the adversarial example
produced by the generator $G$ for the input $x$. Let the set of gold
dimensions that text $x$ belongs to be denoted by
$D_{x}=\\{d_{x_{1}},d_{x_{2}},...\\}$ and the set of dimensions for the
corresponding $u_{x}$ be denoted by $D_{u_{x}}$.

#### Classifier

We aim to fool a classifier $f$ which makes a binary prediction $f(x)$ for its
input text $x$.

#### Dimensional classifier

Given text $u$, a set of dimensional classifiers $\hat{D}$, let $\hat{D}_{u}$
be the predicted set of dimensions that the text $u$ belongs to. We use
$\hat{D}$ to assert that $u_{x}$ is dimensionally diverse that $x$, if
$\hat{D}_{u_{x}}\supset\hat{D}_{x}$.

#### Generator

We assume blackbox-access to an attacker $G$ whose weights cannot be accessed
or updated. Using $G$, we assume to make unlimited queries to the classifier
$f$ but cannot access the classifier’s gradients or assume the classifier’s
architecture. Given a set of inputs $X$, our goal is to use $G$ to produce a
set of text $U$ that adversarially fools $f$, and is dimensionally more
diverse.

Given $X,Y,f,\hat{D}$, the generated attacks $U\sim G(X)$ satisfy the
following desiderata:

$\displaystyle\begin{cases}\text{$U$ has the same label as $X$, i.e. $\forall
x,u_{x}:y_{u_{x}}=y_{x}$}\\\ \text{$U$ is misclassified by $f$, s.t., $\forall
u_{x}:f(u_{x})\neq y_{u_{x}}$,}\\\ \text{$U$ is dimensionally more diverse
than $X$,}\\\ \text{\ \ \ \ \ i.e., $\forall
x,u_{x}:\hat{D}_{u_{x}}\supset\hat{D}_{x}$}\end{cases}$

###  3.2 Data

We focus on the task of generating adversarial toxic comments, motivated by
improving the robustness of safety classifiers used in online forums. We start
with the already published CivilComments Borkan et al. (2019) dataset, a
collection of forum comments for which ground truth toxicity scores have been
collected from human annotators. The dataset contains five dimensions:
identity attack, offensiveness, sexually explicit, obscene, and threat. We use
data from the validation set which contains 97320 records, out of which 7671
are marked as toxic. 664 out of 7671 toxic comments are marked as being
related to identity attack, 5769 to be insults, 525 to be obscene, 201 to be
sexually explicit and 224 to be threats.

Figure 2:  Examples of user comments in the CivilComments dataset that are
annotated with different labeled dimensions of toxicity.

###  3.3 Operationalizing the Task

We now provide specifics of the problem formulation for the task of automated
adversarial discovery. Using a dataset $X$ related to safety classifiers, we
want to be able to transform it into a large-scale adversarial dataset $U$
which contains more diverse examples, as measured across dimensions
$\hat{D}_{U}\supset\hat{D}_{X}$, and more examples that can fool a strong
safety classifier $f$. The goal is to generate data with higher dimensional
coverage than the inputs, with the assumption that we define prompt-based LLM
dimensional classifiers $\hat{D}$ (Fig. 6).

#### Safety Classifier

Perspective API is a free, open and reproducible API per (2017) that uses
machine learning to identify “toxic" comments. The models score a phrase based
on the perceived impact the text may have in a conversation. Perspective
models provide classification probability scores for several different tasks.
We design our methods to attack the toxicity classifier of the Perspective
API, i.e. the blackbox classifier $f$.

#### Labeled Dimensions

In this work, we define labeled dimensions to be the different attributes
associated with user comments in the CivilComments dataset Borkan et al.
(2019), namely identity attack, threat, insult, obscene, and sexually
explicit. Each user comment has been annotated by crowd workers along these
labeled dimensions.

#### Unlabeled Dimensions

Unlabeled dimensions refer to other categories of toxicity studied in previous
literature that can be reflected in the user comment. Unlabeled dimensions are
less restrictive than previously defined labeled dimensions, and can include
unexplored attributes such as misandry and spam. By leveraging these unlabeled
dimensions, we can possibly convert user comments from one labeled dimension
to another.

#### Attack Generator

We leverage 540B PaLM2111https://ai.google/discover/palm2/ (with its safety
and content filters switched off) and its instruction following abilities as
the generator $G$ to produce adversarial attacks of diverse dimensions. We
only assume API access to $G$, and do not explore gradient-based methods which
require access to model weights, making it applicable to both closed-source
and open-source models.

#### Dimensional Classifier

We use the 540B instruction-tuned PaLM2 model through an API as a dimensional
classifier to judge whether a generated comment belongs to a certain
dimension. The prompts used to elicit judgments about the dimensions
associated with a piece of text can be found in Figure 6.

##  4 Experiment Details

###  4.1 Evaluation Metrics

Since we want to generate data that is both adversarial as well as belongs to
a previously unseen dimension, we define multiple evaluation metrics to judge
the different methods.

#### Adversarial Success

is defined as the percentage of data points generated by a given method that
are able to fool a classifier, i.e., $f(u_{x})\neq y_{u_{x}}$.

#### Dimensional Diversity

is defined as the percentage of generated data points that belong to the
heldout dimension $h$. We evaluate the effectiveness of our set of dimensional
classifiers $\hat{D}$ in §6.3.

#### Both

We calculate the percentage of data points which belong to the held-out
dimension (satisfy the dimensional diversity criteria) as well as fool the
classifier (achieve adversarial success).

| Adversarial | Diverse
---|---|---
I guess you like Mussolini too. | ✓ | ✗
| Pathetic. Preferring
---
Hitler is f*ing absurd.
✗ | ✓
| I suppose you
---
prefer to get rid of equality,
and go back to the 1950s
where women were in the
kitchen and men were in
the office.
✓ | ✓
Table 1: Starting with the seed comment “Pathetic. I suppose you prefer Hitler
too" identified to be an “identity attack", we present examples of attacks the
demonstrate different desirable characteristics.

Table 1 shows examples of attacks that correspond to the above-mentioned
metrics. We start with a seed comment ($y=\textrm{toxic}$), “Pathetic. I
suppose you prefer Adolf too.". This comment $x$ has the dimension
$D_{x}=\text{``identity attack"}$ The first example, “I guess you like
Mussolini too.", is an adversarial one that fools a safety classifier
($f(x)=\textrm{non-toxic}$) but does not explore a new dimension
($\hat{D}_{u}=D_{x}$). Conversely, the second example, “Pathetic. Preferring
Hitler is f*ing absurd.", is clearly toxic and unlikely to fool a safety
classifier $f$, but it contains a previously unseen dimension
$h=\text{``obscene"}$. Ideally, we want to generate attacks that have both
characteristics (diversity and adversarial nature), and this is demonstrated
in the third example.

###  4.2 Methods

We implement a variety of non-LLM- and LLM-based methods to generate
adversarial attacks and compare their outputs. For each dimension $d\in D$ in
the dataset, we use a leave-one-out dimensions strategy and sample 25 user
comments that do not belong to the held-out dimension $h=d$. We use these seed
comments as input $X$ to various methods, and measure performance of each
method by calculating the defined evaluation metrics (see §4.1) on the
generated data $U$.

#### EDA

EDA Wei and Zou (2019) consists of four simple but powerful operations:
synonym replacement (randomly replace words with their synonyms), random
insertion (insert a random synonym of a random word at a random location),
random swap (randomly swap the position of words in the sentence), and random
deletion (randomly remove words from the sentence). For a comment, one of
these operations is performed at random.

#### WordNet

This method modifies the seed user comment by simply replacing words with
their synonyms from the WordNet thesaurus.

#### CLARE

CLARE Li et al. (2021) applies a sequence of contextualized perturbation
actions to the input. Each can be seen as a local mask-then-infill procedure:
it first applies a mask to the input around a given position, and then fills
it in using a pretrained masked language model.

We use TextAttack, a very popular attack generation library that transmutes
the most predictive words, while preserving semantic similarity and contextual
coherence Morris et al. (2020) to implement these non-LLM baselines.

#### Polyjuice

Polyjuice Wu et al. (2021) has shown promise by improving diversity, fluency
and grammatical correctness of generated attacks as evaluated by user studies.
It covers a wide variety of commonly used counterfactual types including
patterns of negation, adding or changing quantifiers, shuffle key phrases,
word or phrase swaps which do not alter POS tags or parse trees, along with
insertions or deletion of constraints that do not alter the parse tree.
Specifically, we use 8 types of counterfactuals — negation, quantifier,
lexical, resemantic, insert, delete, restructure, shuffle — in Polyjuice to
generate toxic comments. Polyjuice leverages GPT-2 to generate the new user
comments along those lines.

#### Rewrite

To establish the abilities of strong, current LLMs, we prompt $G$ to rewrite
the seed user comment such that it becomes harder for a toxicity detector to
detect, while retaining its toxicity. We engineer our own prompt for this
method.

#### Self-Refine

Madaan et al. (2023) showed that LLMs can generate feedback on their work and
use it to improve their output. We prompt $G$ to explain why a given user
comment might be toxic and use that explanation to modify its toxicity in a
way that, without loss of toxicity, it makes it harder for a toxicity detector
to detect. While Self-Refine as a method exists for other tasks, we adapt the
idea for this task and write our own prompt.

Figure 3:  Given a seed user comment, we first discover unlabeled dimensions
of toxicity, either by prompting an LLM to gauge it from the comment itself
(in-seed) or by querying its priors for top unlabeled dimensions that would be
present in a comment forum (constitutional). Next, we prompt the LLM to
transform the user comment by leveraging that unlabeled dimension in a way
that makes it harder for the toxicity to be detected.

Held-out Dimension | Method | Adversarial | Dimensional | Both % ($\uparrow$)
---|---|---|---|---
Success % ($\uparrow$) | Diversity % ($\uparrow$)
Identity Attack | Wordnet | 6.0 $\pm$ 0.00 | 10.0 $\pm$ 0.00 | 0.0 $\pm$ 0.00
Polyjuice | 43.6 $\pm$ 2.15 | 7.4 $\pm$ 1.56 | 2.8 $\pm$ 1.33
Discover-Adapt | 21.6 $\pm$ 6.05 | 26.0 $\pm$ 4.73 | 5.0 $\pm$ 3.82
Sexually Explicit | Wordnet | 20.0 $\pm$ 0.00 | 16.0 $\pm$ 0.00 | 0.0 $\pm$ 0.00
Polyjuice | 46.2 $\pm$ 3.85 | 8.1 $\pm$ 1.03 | 0.0 $\pm$ 0.00
Discover-Adapt | 31.5 $\pm$ 1.86 | 14.1 $\pm$ 1.06 | 3.5 $\pm$ 1.86
Insult | Wordnet | 16.0 $\pm$ 0.00 | 24.0 $\pm$ 0.00 | 0.0 $\pm$ 0.00
Polyjuice | 35.1 $\pm$ 4.19 | 5.1 $\pm$ 1.54 | 0.0 $\pm$ 0.00
Discover-Adapt | 26.2 $\pm$ 4.74 | 18.5 $\pm$ 3.56 | 3.6 $\pm$ 1.02
Obscene | Wordnet | 18.0 $\pm$ 0.00 | 34.0 $\pm$ 0.00 | 2.0 $\pm$ 0.00
Polyjuice | 47.8 $\pm$ 4.24 | 13.8 $\pm$ 2.44 | 0.8 $\pm$ 0.80
Discover-Adapt | 32.4 $\pm$ 5.43 | 17.6 $\pm$ 5.71 | 1.2 $\pm$ 0.98
Threat | Wordnet | 12.0 $\pm$ 0.00 | 18.0 $\pm$ 0.00 | 0.0 $\pm$ 0.00
Polyjuice | 48.6 $\pm$ 3.10 | 13.2 $\pm$ 2.99 | 5.4 $\pm$ 1.80
Discover-Adapt | 21.6 $\pm$ 6.05 | 14.0 $\pm$ 5.73 | 2.6 $\pm$ 1.80

Table 2: Across all five held-out dimensions, we use a variety of metrics to
show that our framework of generating adversarial data is better than existing
methods. The ‘Both’ metric represents the percentage of generated data points
that contain the unseen dimension as well as adversarial for the classifier.
We generate data from each method using only a seed set of 25 examples that do
not contain the held-out dimension. Since the amount of data generated by
different methods varies, we report the mean and standard deviation for each
method on a sample size of 50 data points bootstrapped for 10 iterations. In
this table, we only present results for one method of each type — non-LLM,
LLM, Discover-Adapt.

#### Discover-Adapt

To build upon the self-refine idea, we define a two-step approach to leverage
$G$ to generate new types of attacks. First, in the discover step, we explore
different methods of finding an unlabeled dimension $s$ of toxicity to
exploit. These methods of discovery include judging what category of toxicity
already exists in an given user comment (in-seed), and using the priors of
LLMs as a source of knowledge of the unlabeled dimensions of toxicity found in
user forums (constitutional). The flexibility of this method also allows using
static lists of toxicity dimensions curated from experts or derived from
previous literature. Next, in adapt, we nudge $G$ to transform the input user
comment along the lines of $S$. This pushes the user comment a step towards a
dimension it was previously unrelated to ($D_{u}\neq D_{x}$).

##  5 Results and Discussion

We present results for one representative non-LLM-based, one LLM-based method
as well as one Discover-Adapt setting. We discuss other methods in detail
later in §6.2. Table 2 shows the strengths and weaknesses of different types
of adversarial discovery methods.

Non-LLM baselines do not perform well. WordNet, using simple word
perturbations, is able to produce diverse attacks for four out of five
previously unseen dimensions. However, it has the least adversarial success
out of all methods, only generating adversarial data <10% of the time. While
this method requires the least amount of compute, it is unable to produce
examples at a large scale. Perturbing input examples with WordNet is best to
generate adversarial and obscene comments.

LLM baselines get stuck in known dimensions. Polyjuice consistently achieves
the highest adversarial success out of all methods for all dimensions
($35-48\%$). Using LLMs with a naive or with a self-refine inspired prompt
produces the largest percentage of adversarial data, as the generator $G$ is
very good at instruction following. However, its transformations fail to
discover the unknown dimension, and is thus unable to satisfy the dimensional
diversity constraint ($5-13\%$).

Discover-Adapt is inconsistent. Amongst all methods, using the Discover-Adapt
framework is best for generating adversarial examples that contain identity
attacks, insults and sexually explicit content (three out of five held-out
dimensions). This technique balances the two constraints (adversarial success
and dimensional diversity) for three out of five dimensions, but is not
consistent across all dimensions.

Discover-Adapt is more controllable. The discover component enables the use of
unlabeled dimensions of toxicity obtained from different sources. These
sources include aspects of toxicity judged to be present in a given seed
example, or a list of unlabeled dimensions of toxicity either compiled in
previous literature or sampled from LLM priors. Using this two-step approach
allows for more control in generating adversarial examples. In this work, we
only explore the unlabeled dimensions that are identifiable by LLMs, but
Discover-Adapt is extendable.

Generating diverse adversarial attacks is hard. In Table 2, we note that none
of the methods achieve both high adversarial success or dimensional diversity.
Indeed, we find that the performance of all methods on the ‘Both’ metric is
less than 6% across all harm dimensions. Different types of methods are
required to produce adversarial comments of different dimensions. It is
evident that automated adversarial discovery is challenging and existing
techniques are not sufficient to tackle the task, requiring further research.

##  6 Analysis

###  6.1 Sources of Discovery

For the Discover-Adapt method, we analyze the effect of using different
sources of obtaining the unlabeled dimensions of toxicity. In-Seed refers to
prompting the LLM to identify the top five unlabeled dimensions of toxicity
present in a given user comment, before leveraging those unlabeled dimensions
one by one for generation. Constitutional 25 refers to querying the LLM priors
for the top 25 unlabeled dimensions that are found in forums, such as the
Civil Comments platform, that aggregate user comments and using each unlabeled
dimension to adapt an input example. In the Constitutional 5 method, we sample
5 out of the 25 unlabeled dimensions in the discover step and adapt a user
comment along those lines.

Method | Identity Attack % ($\uparrow$)
---|---
In-Seed | 13.4 $\pm$ 4.90
Constitutional 25 | 19.8 $\pm$ 5.02
Constitutional 5 | 26 $\pm$ 4.73

Table 3: To discover unlabeled dimensions of toxicity, we can use different
sources. Here, we explore the effectiveness of using these sources to generate
data related to the identity attack held-out dimension. We find that querying
LLM priors for the top twenty five unlabeled dimensions of toxicity found in
user forums and sampling five out of them leads to the best results.

Method | Adversarial Success ($\uparrow$) | Identity Attack % ($\uparrow$) | Both ($\uparrow$)
---|---|---|---
EDA | 2.0 $\pm$ 0.00 | 12.0 $\pm$ 0.00 | 0.0 $\pm$ 0.00
WordNet | 6.0 $\pm$ 0.00 | 10.0 $\pm$ 0.00 | 0.0 $\pm$ 0.00
CLARE | 8.0 $\pm$ 0.00 | 16 $\pm$ 0.00 | 0.0 $\pm$ 0.00
Polyjuice | 43.6 $\pm$ 2.15 | 7.4 $\pm$ 1.56 | 2.8 $\pm$ 1.33
Rewrite | 48.2 $\pm$ 6.03 | 7.4 $\pm$ 3.16 | 2.4 $\pm$ 2.15
Self-Refine | 57.2 $\pm$ 5.74 | 3.8 $\pm$ 2.75 | 0 $\pm$ 0

Table 4: We use a variety of metrics to show that our framework of generating
adversarial data is better than existing method. The ‘Both’ metric represents
the percentage of generated data point that contain identity attacks as well
as adversarial for the classifier. We generate data from each method using
only a seed set of 25 examples that do not contain identity attacks (held-out
dimension). Since the amount of data generated by different methods varies, we
report the mean and standard deviation for each method on a sample size of 50
data points bootstrapped for 10 iterations. Here, we treat identity attack as
the held-out dimension.

Table 3 shows the results of using different sources to discover unlabeled
dimensions of toxicity when treating identity attack as the held-out
dimension. Leveraging five sampled unlabeled dimensions out of the top 25
results in Discover-Adapt being able to generate the most amount of identity
attacks. We hypothesize that adapting a user comment to diverse unlabeled
toxicity dimensions is most likely to lead to a new labeled dimension.

###  6.2 Generating Identity Attacks

Table 4 presents the performance of 3 non-LLM- and 3 LLM-based methods when
identity attack is treated as the held-out dimension. We find that simple
perturbation attacks achieve very low adversarial success, but are able to
explore the held-out dimension more than LLM-based attacks. Among LLM-based
attacks, we note that, while our Self-Refine inspired implementation achieves
the highest adversarial success, it is worse than the others at discovering
the held-out dimension.

###  6.3 How Good is the Dimensional Classifier?

We sample data points from the test set such that each dimension contains a
balanced number (number of ground truth positives is same as number of ground
truth negatives) of data points in the sample. We then use our dimensional
classifier to obtain judgments for each dimension on this sample. To calculate
dimensional classifier accuracy, we compare against the dimensional ground
truth label of a data point in the sample to the dimensional predictions.

Dimension | Judgment Accuracy
---|---
Obscene | 85.06%
Insult | 76.47%
Threat | 79.27%
Identity Attack | 84.0%
Sexually Explicit | 85.57%

Table 5: PaLM2 is good enough as a judge for all dimensions. We can rely on it
as a proxy for dimension-related judgment.

Table 5 shows that PaLM2 is best at identifying identity attacks, obscenities
and sexually explicit content. It can identify all dimensions with a minimum
accuracy of ~76%. Based on these results, we can use PaLM2 to auto-label the
dimensions of generated data.

Figure 4:  We present an example of a successful attack that contains a held-
out dimension (identity attack) as well as two common failure modes of
Discover-Adapt.

###  6.4 Qualitative Analysis

Figure 4 shows examples of attacks generated using the Discover-Adapt
framework. First, using misogyny as the discovered unlabeled dimension, the
input user comment is transformed into one that contains an identity attack
(previously held-out) towards women. Next, we showcase two common errors that
Discover-Adapt makes, namely acting as a paraphraser (which does not satisfy
the dimensional diversity criteria) and not faithfully adapting to the
unlabeled dimension if incorporating it means generating an attack unrelated
to the input. We note that while the former is a characteristic of LLMs, the
latter is also hard for human attackers.

##  7 Conclusion

The use of LLMs to generate adversarial attacks has gained popularity. Using
the case-study of a toxicity classifier, we demonstrate that such methods lack
diversity in their generated attacks. Further, we formalize the task of
automated adversarial discovery — generating attacks against safety
classifiers which belong to previously unseen categories and propose an
evaluation framework. Our experiments show that while LLM methods outperform
word substitution methods in terms of adversarial success by ~30%, they
perform similarly in terms of generating attacks from previously unknown
dimensions. This demonstrates that LLM-based adversarial attack generation
methods are still inadequate in discovering new attacks and require
significant human intervention to be useful at scale in an automated manner.
Our analysis highlights issues around inconsistency, instruction following and
exploration that future work can build upon.

## Limitations

The Discover-Adapt framework we experiment with has three limitations: 1)
Subjectivity of dimensional evaluations, 2) Dependence on the underlying
quality of the LLM used, which lead to 3) Mixed results across different
unlabeled dimensions of toxicity (see §5).

We use a dimensional classifier to assess the diversity in the generated data.
What constitutes a separate dimension is, however, subjective. Evaluation on
this task therefore requires a golden set of human evaluations, and/or apriori
labeled dimensions that can be discovered.

Second, our method is limited by the capability of the underlying LLM to
follow instructions. Our qualitative analysis (see §6) shows the most common
error is not generating an attack that follows the desired toxicity dimension.
This error is more pronounced when the new toxicity instruction is vastly
different from the input user comment.

As a result, using the Discover-Adapt framework only beats other methods for
three out of five possible held-out labeled dimensions of toxicity (as
presented in §5). Even when it does beat the other methods, there is still
substantial headroom for improvement.

## Ethical Considerations

In this work, we focus on generating toxic and harmful content with the aim of
finding ways to discover unseen types of attacks that future safety
classifiers can defend against. It is important to emphasize that the opinions
expressed in these outputs are automatically generated through LLMs and do not
reflect the viewpoints of the authors. Consequently, we strongly advise
researchers to use this framework with utmost caution. Further, relying on
human annotators to evaluate toxic text can take a toll on their mental well-
being. We recognize that individuals may instead use such findings to exploit
platforms where these safety classifiers are currently deployed. Our intention
in formalizing this task is to enable future-proofing of safety classifiers
going forward, following the principle that “stronger attackers can evoke
better defense". To address harms, the adversarial attacks generated through
the presented methods have been shared with the Perspective API team for
mitigation through additional training.

## Acknowledgement

The authors would like to thank Ahmad Beirami, Jilin Chen, Flavien Prost,
Kathy Meier-Hellstern for their valuable comments and feedback during the
work.

## References

  * per (2017) 2017\.  Perspective API.  https://www.perspectiveapi.com/.
  * Balashankar et al. (2023) Ananth Balashankar, Xiao Ma, Aradhana Sinha, Ahmad Beirami, Yao Qin, Jilin Chen, and Alex Beutel. 2023.  Improving few-shot generalization of safety classifiers with data augmented parameter-efficient fine-tuning of llms.
  * Belinkov and Bisk (2018) Yonatan Belinkov and Yonatan Bisk. 2018.  Synthetic and natural noise both break neural machine translation.  In _International Conference on Learning Representations_.
  * Borkan et al. (2019) Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2019.  Nuanced metrics for measuring unintended bias with real data for text classification.  _CoRR_ , abs/1903.04561.
  * Calderon et al. (2022) Nitay Calderon, Eyal Ben-David, Amir Feder, and Roi Reichart. 2022.  DoCoGen: Domain counterfactual generation for low resource domain adaptation.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 7727–7746, Dublin, Ireland. Association for Computational Linguistics.
  * Casper et al. (2023) Stephen Casper, Jason Lin, Joe Kwon, Gatlen Culp, and Dylan Hadfield-Menell. 2023.  Explore, establish, exploit: Red teaming language models from scratch.
  * Dinan et al. (2019) Emily Dinan, Samuel Humeau, Bharath Chintagunta, and Jason Weston. 2019.  Build it break it fix it for dialogue safety: Robustness from adversarial human attack.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4537–4546, Hong Kong, China. Association for Computational Linguistics.
  * Ficler and Goldberg (2017) Jessica Ficler and Yoav Goldberg. 2017.  Controlling linguistic style aspects in neural language generation.  In _Proceedings of the Workshop on Stylistic Variation_ , pages 94–104, Copenhagen, Denmark. Association for Computational Linguistics.
  * Gao et al. (2018) Ji Gao, Jack Lanchantin, Mary Lou Soffa, and Yanjun Qi. 2018.  Black-box generation of adversarial text sequences to evade deep learning classifiers.  In _2018 IEEE Security and Privacy Workshops (SPW)_ , pages 50–56.
  * Garg et al. (2019) Sahaj Garg, Vincent Perot, Nicole Limtiaco, Ankur Taly, Ed H. Chi, and Alex Beutel. 2019.  Counterfactual fairness in text classification through robustness.  In _Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society_ , AIES ’19, page 219–226, New York, NY, USA. Association for Computing Machinery.
  * Garg and Ramakrishnan (2020) Siddhant Garg and Goutham Ramakrishnan. 2020.  BAE: BERT-based adversarial examples for text classification.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6174–6181, Online. Association for Computational Linguistics.
  * Hartvigsen et al. (2022) Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. 2022.  ToxiGen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3309–3326, Dublin, Ireland. Association for Computational Linguistics.
  * Iyyer et al. (2018) Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018.  Adversarial example generation with syntactically controlled paraphrase networks.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1875–1885, New Orleans, Louisiana. Association for Computational Linguistics.
  * Kiela et al. (2021) Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. 2021.  Dynabench: Rethinking benchmarking in NLP.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4110–4124, Online. Association for Computational Linguistics.
  * Lahoti et al. (2023) Preethi Lahoti, Nick Blumm, Xiao Ma, Ragha Kotikalapudi, Sahitya Potluri, Qijun Tan, Hansa Srinivasan, Ahmad Beirami, Ben Packer, Alex Beutel, and Jilin Chen. 2023.  Improving diversity of representation in large language models via collective-critiques and self-voting (ccsv).
  * Li et al. (2021) Dianqi Li, Yizhe Zhang, Hao Peng, Liqun Chen, Chris Brockett, Ming-Ting Sun, and Bill Dolan. 2021.  Contextualized perturbation for textual adversarial attack.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5053–5069, Online. Association for Computational Linguistics.
  * Li et al. (2020) Linyang Li, Ruotian Ma, Qipeng Guo, Xiangyang Xue, and Xipeng Qiu. 2020.  BERT-ATTACK: Adversarial attack against BERT using BERT.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6193–6202, Online. Association for Computational Linguistics.
  * Madaan et al. (2023) Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Sean Welleck, Bodhisattwa Prasad Majumder, Shashank Gupta, Amir Yazdanbakhsh, and Peter Clark. 2023.  Self-refine: Iterative refinement with self-feedback.
  * Mehrabi et al. (2023) Ninareh Mehrabi, Palash Goyal, Christophe Dupuy, Qian Hu, Shalini Ghosh, Richard Zemel, Kai-Wei Chang, Aram Galstyan, and Rahul Gupta. 2023.  Flirt: Feedback loop in-context red teaming.
  * Morris et al. (2020) John Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. 2020.  TextAttack: A framework for adversarial attacks, data augmentation, and adversarial training in NLP.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 119–126, Online. Association for Computational Linguistics.
  * Perez et al. (2022) Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. 2022.  Red teaming language models with language models.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 3419–3448, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Radharapu et al. (2023) Bhaktipriya Radharapu, Kevin Robinson, Lora Aroyo, and Preethi Lahoti. 2023.  AART: AI-assisted red-teaming with diverse data generation for new LLM-powered applications.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track_ , pages 380–395, Singapore. Association for Computational Linguistics.
  * Ren et al. (2019) Shuhuai Ren, Yihe Deng, Kun He, and Wanxiang Che. 2019.  Generating natural language adversarial examples through probability weighted word saliency.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1085–1097, Florence, Italy. Association for Computational Linguistics.
  * Ribeiro et al. (2020) Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020.  Beyond accuracy: Behavioral testing of NLP models with CheckList.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4902–4912, Online. Association for Computational Linguistics.
  * Samvelyan et al. (2024) Mikayel Samvelyan, Sharath Chandra Raparthy, Andrei Lupu, Eric Hambro, Aram H. Markosyan, Manish Bhatt, Yuning Mao, Minqi Jiang, Jack Parker-Holder, Jakob Foerster, Tim Rocktäschel, and Roberta Raileanu. 2024.  Rainbow teaming: Open-ended generation of diverse adversarial prompts.
  * Sinha et al. (2023) Aradhana Sinha, Ananth Balashankar, Ahmad Beirami, Thi Avrahami, Jilin Chen, and Alex Beutel. 2023.  Break it, imitate it, fix it: Robustness by generating human-like attacks.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 1631–1642, Seattle, Washington, USA. Association for Computational Linguistics.
  * Vidgen et al. (2021) Bertie Vidgen, Tristan Thrush, Zeerak Waseem, and Douwe Kiela. 2021.  Learning from the worst: Dynamically generated datasets to improve online hate detection.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1667–1682, Online. Association for Computational Linguistics.
  * Wang et al. (2020) Tianlu Wang, Xuezhi Wang, Yao Qin, Ben Packer, Kang Li, Jilin Chen, Alex Beutel, and Ed Chi. 2020.  CAT-gen: Improving robustness in NLP models via controlled adversarial text generation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 5141–5146, Online. Association for Computational Linguistics.
  * Wei and Zou (2019) Jason Wei and Kai Zou. 2019.  EDA: Easy data augmentation techniques for boosting performance on text classification tasks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 6382–6388, Hong Kong, China. Association for Computational Linguistics.
  * Wichers et al. (2024) Nevan Wichers, Carson Denison, and Ahmad Beirami. 2024.  Gradient-based language model red teaming.
  * Wu et al. (2021) Tongshuang Wu, Marco Tulio Ribeiro, Jeffrey Heer, and Daniel Weld. 2021.  Polyjuice: Generating counterfactuals for explaining, evaluating, and improving models.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 6707–6723, Online. Association for Computational Linguistics.
  * Zhang et al. (2020) Wei Emma Zhang, Quan Z. Sheng, Ahoud Alhazmi, and Chenliang Li. 2020.  Adversarial attacks on deep-learning models in natural language processing: A survey.  _ACM Trans. Intell. Syst. Technol._ , 11(3).

##  Appendix A Prompts Used

We present the different prompts used for PaLM2 as part of LLM-based baselines
and our discover-adapt framework in Figure 5.

Figure 5:  PaLM2 prompts for different baselines, and methods of discovering
new toxicity subtypes to adapt to.

Figure 6 presents the PaLM2 prompts that were used to obtain judgments about
dimensions of toxicity that may be present in the generated attacks.

Figure 6:  PaLM2 prompts for judging whether a user comment (text) is related
to a dimension of toxicity present in the CivilComments dataset.

As part of the discover step, we prompt PaLM2 for the top 25 subtypes of
toxicity that might be present in comments found on a user forum. These
subtypes as well as their definitions according to PaLM2 are presented in
Figure 7. We use subtypes from this list as part of using constitutional
subtypes during the discover step.

Figure 7:  Top 25 subtypes of toxicity as well as their definitions that are
present in user forums according to PaLM2. We sample from these in the
discover step of our discover-adapt framework.