#  A Holistic Approach to Undesired Content Detection in the Real World
Warning: some content may contain racism, sexuality, or other harmful
language.

Todor Markov* Chong Zhang* Sandhini Agarwal Tyna Eloundou
Teddy Lee Steven Adler Angela Jiang Lilian Weng*

###### Abstract

We present a holistic approach to building a robust and useful natural
language classification system for real-world content moderation. The success
of such a system relies on a chain of carefully designed and executed steps,
including the design of content taxonomies and labeling instructions, data
quality control, an active learning pipeline to capture rare events, and a
variety of methods to make the model robust and to avoid overfitting. Our
moderation system is trained to detect a broad set of categories of undesired
content, including sexual content, hateful content, violence, self-harm, and
harassment. This approach generalizes to a wide range of different content
taxonomies and can be used to create high-quality content classifiers that
outperform off-the-shelf models.

**footnotetext: These authors contributed equally to this work

##  1 Introduction

Recent advances in deep learning have accelerated the adoption of language
models for socioeconomically valuable tasks in the real world (Devlin et al.
2019; Brown et al. 2020; Cohen et al. 2022). Both the systems’ builders and
its users may benefit from a responsible deployment approach that includes
moderating the models’ outputs: First, model providers may want assurances
that the models will not produce content that is disallowed by their policies.
Second, customers of these models sometimes require control over content to
mitigate the impact of sensitive use cases or to reduce brand risk. A
principled, robust, and efficient moderation solution can track and measure
the model inputs and outputs to ensure safety standards. It can also provide
fine-grained control to enable use cases with sensitive needs, such as
educational applications. We believe that a strong undesired content
classifier lays the foundation for building safer AI systems in the wild, as
it enables the capacity of moderating, evaluating, and guiding the models
towards safer behavior.

Existing work on content detection either focuses mainly on a limited set of
categories, including toxicity (Pavlopoulos et al. 2020; Gehman et al. 2020),
hate speech (Kwok and Wang 2013; Davidson et al. 2017), and abusive content
(Nobata et al. 2016; Vidgen et al. 2019); or is tailored towards a targeted
use case, such as Perspective API (Jigsaw ) on online toxic comment
moderation. There is increasing attention to understanding the risk areas of
large language models via a more rigorous taxonomy (Weidinger et al. 2021),
but the amount of work is still limited, especially when it comes to deploying
language models for real-world applications. Here we build a more
comprehensive system for detecting a broad set of categories of undesired
content, including sexual content, hateful content, violence, self-harm, and
harassment, as well as severe subcategories under each top-level category.
Large-scale content moderation systems and tooling exist on a number of
platforms (YouTube 2019; Reddit 2022). We aim to provide a blueprint for
creating such systems across a wide variety of use cases.

Figure 1: Overview of the model training framework.

Detecting undesired content is difficult due to several challenges. First,
there is not a clearly and widely agreed-upon categorization of undesired
content. Designing a detailed taxonomy for undesired content and
operationalizing it for labeling purposes require a lot of work. The
categorization framework usually needs to clarify a significant number of
corner cases to achieve high inter-rater agreement during labeling. This is
further complicated by the subjectivity of some labeling decisions, due to the
different social and cultural backgrounds of human annotators. Second, a
practical moderation system needs to process real-world traffic. Thus a model
bootstrapped from public data or academic datasets would not work well because
there exists a big data distribution shift and taxonomy misalignment. Third,
it is rare to encounter certain categories of undesired content in real-world
settings. For example, among sampled user prompts we observed that only 0.04%
of cases included self-harm and 0.017% included hateful content involving
threats. Hence, we need smart solutions to the cold start problem and
effective ways to discover undesired samples.

Multiple components contribute to the success of building and deploying a
practical, general moderation system into the real world. These include
effectively establishing a chain of carefully polished and curated
configurations for data collection, data labeling, model training and active
learning. Based on our experimentation, we find the following conclusions to
be especially noteworthy.

  * •

Detailed instructions and quality control are needed to ensure data quality.
Labeling instructions that lack sufficient precision force annotators to rely
on their subjective judgment, resulting in inconsistently labeled data that
confuses the model. Regular calibration sessions are necessary to refine these
instructions and ensure annotators are aligned with them. And a poorly chosen
quality metric can lead to data that hurts model performance. (See Section
3.2)

  * •

Active learning is a necessity. There is likely a large distribution shift
between public data and the traffic from one’s production system. Thus, it is
critical to collect new training samples from the production traffic. Active
learning can effectively expand the training dataset to capture a
significantly (up to 22$\times$) larger amount of undesired samples when
dealing with rare events. This can lead to a performance improvement in the
underlying model of up to 10$\times$ for rare categories. (See Section 3.1 and
Section 4.3)

  * •

Use public datasets with care. Publicly available data might not lead to high
quality performance for the problem in hand due to differences in taxonomy and
training data distribution, but can be used to construct a noisy cold start
dataset at the early stage. However, adding academic data into the training
set may hurt the model performance at a later stage when there are enough
properly labeled data samples. (See 3.5)

  * •

Imbalanced training data can lead to incorrect generalization. Deep learning
models could easily overfit common phrases or templates. For example, the
model can over-generalize to anything formatted as "X is hateful" if the data
distribution is off-balance. We tackle this challenge by programmatically
identifying overfitted phrases and by red-teaming via human trials. We then
alter the training distribution by incorporating model-generated or human-
curated synthetic data to patch the weakness. (See Section 3.5 and Section
3.3)

  * •

Mistakes in data will happen and need to be managed. Even with significant
amounts of effort on data quality control, we still run into mislabeled
examples. We explore different methods for identifying those cases, including
cross-validation and hunting down common phrases causing the model to overfit
via token subtraction. (See Section 3.2 and Section 3.5)

We aim to present a holistic approach to building a reliable and robust
undesired content detection model for real-world applications. Our approach is
meant to handle the scenario in which the type of the content to be detected
is rarely observed in the real world. We hope that the lessons we learned are
useful to others working on similar problems. We release a dataset111
https://github.com/openai/moderation-api-release; sourced from CommonCrawl and
model-generated data  containing text samples from publicly available data
labeled according to our taxonomy. The model is accessible via the OpenAI
Moderation endpoint222 https://beta.openai.com/docs/guides/moderation;
Harassment category is currently under further improvement and will be
available in the future. .

##  2 Taxonomy

Given that the meaning of a sentence can depend on context, designing a
universal taxonomy for undesired content is challenging and requires inherent
trade-offs. Throughout this exercise, we aimed to make the dataset as broadly
applicable as possible to maximize its utility across research and industrial
contexts, but recognize it still ought to be critically evaluated prior to use
in any context, The anatomy of what’s undesired can vary based on factors like
use cases, users, regions and many others. Some sentences labeled as undesired
according to our taxonomy (e.g. "I will kill you" would be labeled as
containing violence) may not be considered undesired when appearing in another
context such as fiction.

Additionally, we aimed to capture layers of subcategorization and granularity.
Much existing work lacks subcategorizations to capture a spectrum of severity,
which is critical, since statements such as "kill all [IDENT_GROUP]" can have
different repercussions compared to "[IDENT_GROUP] are dishonest". The
existence of subcategories can also largely reduce the disagreement rate and
corner cases during labeling, making it valuable for data quality control. In
the following outline, we present our taxonomy for undesired content
categorization, including five top-level categories and a spectrum of
subcategories.

S: Sexual content

Content that depicts explicit or implicit sexual behavior. Subcategories that
are considered undesired: [S3] sexual content involving minors, [S2] content
that depicts certain sexual activities which could be illegal333This mapped
most closely to what’s illegal in USA. if they happened in real life today,
and [S1] erotic sexual content that does not depict illegal activities.
Subcategories that are not considered undesired: [S0] Non-erotic or
contextualized sexual content, such as medical or sex education material.

H: Hateful content

Content that is threatening, insulting, derogatory and otherwise abusive
content targeting specific chosen groups or members of the group because of
their group identities. Subcategories that are considered undesired: [H2]
hateful content calling for violence or threatening, [H1] derogatory
stereotypes or support for hateful statements. Subcategories that are not
considered undesired: [H0.a] neutral statement referring to group identity and
[H0.b] contextualized hate speech, such as a quote of other individual’s
statement in order to provide commentary.

V: Violence

Content that depicts or shows support for physical violence. Subcategories
that are considered undesired: [V2] extremely graphic violence, [V1] threats
or support for violence. Subcategories that are not considered undesired: [V0]
neutral depictions of contextualized violence.

SH: Self-harm

Content that contains a graphic description or shows support for self-harm.

HR: Harassment

Content that may be used to torment or annoy individuals in real life, or make
harassment more likely to occur.

Our model learns to predict whether a given sample violates any of 8 chosen
categories, including all the top categories (S, H, V, SH, HR) and three most
severe subcategories (S3, H2, and V2).

##  3 Methods

###  3.1 Data Selection and Active Learning

To ensure that our moderation system performs well in the context of our
production use cases, we need to incorporate production data to our training
set. We set up a three-stage procedure in an iterative fashion.

First, a large volume of our production data is selected at random. Any
potential personally identifiable information (PII) is masked. The most recent
moderation model is used to score these samples and discover which ones may
trigger any chosen categories.

In the second stage we run a simple active learning strategy to select a
subset of most valuable samples to be labeled out of the random samples
extracted in stage one. The active learning strategy is composed of three
parallel pipelines. The first one relies on random sampling such that some
fraction of our data remain consistent with the underlying data distribution
in production. The second one randomly selects from samples with model score
above a certain threshold for each category to identify likely undesired data
points. The last pipeline adopts a set of uncertainty sampling strategies
(Lewis and Gale 1994; Lewis and Catlett 1994) to capture samples that the
model is most uncertain about, where the model score for that category is
closest to 0.5.

During the final stage, all the samples selected by different active learning
strategies are aggregated and re-weighted based on statistics of certain
metadata associated with it. The sampling weight is configured to be
proportional to the square root of the sample count. This helps improve the
diversity of selected samples with regard to the associated metadata. We
update the sub-strategy mixture over time based on changes in the data
distribution and categories that we want to improve the most at different
stages.

###  3.2 Labeling and Quality Control

Data label correctness is critical to good model performance. Getting such
data can be difficult given that our categories and the boundary lines between
them are inherently subjective. However, certain interventions can
significantly improve the quality of labeled data.

One important intervention for improving data quality - in terms of both
consistent labels across different annotators as well as between annotators
and researchers - is to make the labeling instructions as well-defined and
concrete as possible. To make the instructions well-defined, we sought to
design detailed definitions and design categories or subcategories to be as
mutually exclusive as possible so as to minimize ambiguity. To make the
instructions concrete, we hosted regular calibration sessions to review
ambiguous edge cases and instances where external annotators and our internal
auditors disagree. Based on feedback from those sessions, we made the
instructions more clear and concrete, with numerous examples and clearer
definitions around borderline cases. As rules are defined clearly and
concretely to minimize subjective judgments, they can be executed more
consistently by the annotators.

Regular, ongoing audits are necessary to ensure that labeled data continues to
be of sufficiently high quality. The choice of which samples to audit and what
metrics to use to measure data quality is crucial. We found that selecting
auditing targets at random cannot maximize the value out of auditing due to
the imbalanced distribution across categories. The annotator-auditor agreement
rate (i.e. accuracy) is suboptimal because undesired examples are rare events
to encounter and the accuracy can be arbitrarily high due to the abundance of
true negatives. Instead, in each chosen category, we randomly select 10
samples labeled as undesired and 10 samples with model probability greater
than 50%. The former help capture false positive cases and the latter provide
an estimation on recall. Then we compute the F-1 score for the chosen samples
based on the annotator-assigned labels while using auditor-assigned labels as
ground truth. This procedure performs much better in practice when certain
categories of undesired data points are rare. Separation of metrics per
category makes it easy to recognize category-specific issues and to retrain
annotators accordingly.

Even with very clear labeling instructions and an effective audit procedure,
mistakes in data are still unavoidable. To identify potentially mislabeled
samples in our dataset, we periodically split our current training dataset
into two parts, train separate models on those datasets and use each model to
score another half of the dataset that model was not trained on. When the
model prediction disagrees with the current ground-truth label, the sample in
question gets flagged. A random portion of flagged samples is audited, and if
more than 30% are identified as mislabeled, all flagged samples would get
labeled again for the second time.

###  3.3 Synthetic Data

In addition to the data collection discussed above, we also use synthetic data
to improve model performance on rare categories such as SH and to mitigate the
counterfactual bias towards certain demographic attributes (Kusner et al.
2017; Garg et al. 2019; Dwork et al. 2012). Generating synthetic data through
large pre-trained language models has shown to be an effective way for data
augmentation (Anaby-Tavor et al. 2020; Kumar, Choudhary, and Cho 2020; Yoo et
al. 2021) and it is particularly helpful when there is little to no initial
data (“cold start”) or when there are not enough undesired samples in the
production traffic.

#### Zero-shot data for cold start.

To kick start the active learning and labeling process, we need some initial
data to build the first version of the model and train annotators. However, it
is difficult to find existing public datasets on certain categories such as SH
and V2. We tackle the problem by generating a synthetic dataset with zero-shot
prompts on GPT-3. The prompts are constructed from human-crafted templates and
we label the generated texts as the initial dataset. Table 1 provides an
example prompt for SH.

Example prompt
---
$>>$ The text is about self harm, more specifically: physical self-harm
(cutting).
$>>$ The narrator is third person (a friend).
$>>$ The narrator is a young teenager (female).
$>>$ The act of self harm happens: last week.
$>>$ Type of text: online forum post.
$>>$ The following word should come up in the text: camera.
$>>$ The text should not contain the phrase ”self harm” or ”self-harm”.
$>>$ The writing level is: average, some errors.
$>>$ The text starts below:

Table 1: Example zero-shot prompt template for generating synthetic SH data.
The sections in green are filled with random ingredients to encourage
diversity.

#### Few-shot data for rare categories.

Some sub-categories had minimal amounts of undesired data even after several
iterations of active learning. To address this, we constructed few-shot
prompts with existing undesired examples and sent the generated texts to be
labeled. The generated texts are manually inspected to avoid bias
amplification (Zhao et al. 2017). We observed a nontrivial performance
improvement by incorporating the synthetic dataset.

#### Curated data to mitigate counterfactual bias.

Similar to other existing NLP models, our models also suffer from
counterfactual bias towards certain demographic attributes as bias commonly
exists in the training data. For instance, "black women." was classified as
hateful content with high confidence in earlier versions of the model. We
mitigate the issue by curating a synthetic dataset with templates that tend to
lead to hateful predictions, e.g., "[subject] is selfish/foolish/narrow-
minded.". The [subject] could either be filled with real demographic
attributes (e.g., Latino) or random object names (e.g., "black blanket"),
which forms hateful and safe samples respectively. We observe that the curated
dataset not only mitigates bias to some degree, but also helps improve the
model performance. For instance, the average AUPRC on hateful content was
improved from $0.417$ to $0.551$ by adding 69k curated synthetic examples. We
believe this is because the contrastive setup of subjects in synthetic example
templates encourages the model to infer the correct feature representations:
negative descriptive words or individual identity groups alone are not enough
to be considered hateful, and only when they appear together they might be
considered hateful. Despite the observed improvements, the synthetic dataset
also has limitations and we will continue improving it in the future (Section
6).

#### Large amount of noisy data does not help.

To understand whether it is helpful to include a large amount of noisy
synthetic data, we also generated zero-shot and few-shot examples twice the
size of the existing labeled training dataset. For zero-shot examples, we set
the label to positive or negative if the prompt asks the model to generate
undesired or safe examples, respectively. For few-shot examples, we set the
label to positive or negative if all of the few-shot examples are undesired or
safe, respectively. Contrary to previous studies (Wang et al. 2021b; Schick
and Schütze 2021), we found mixing noisy synthetic data into training hurt
model performance. It is worth noting that many existing studies on synthetic
data usage experimented in the no-to-low data regime, where only a handful of
labels are available. However, in our experiment, we have collected a large
high-quality dataset and we suspect that noise introduced by synthetic data
confuses the model and lowers the learning efficiency.

###  3.4 Domain Adversarial Training

We intended to make good use of existing public NLP datasets to improve the
performance of our models. However, we observed that models trained on public
NLP datasets do not perform well on our production traffic. This is likely due
to the distribution difference between domains. For instance, examples from
our production traffic are usually much longer and contain few-shot prompts,
whereas existing public NLP datasets are usually shorter and often crawled
from Wikipedia, Twitter, etc. (Vidgen and Derczynski 2020). To mitigate the
problem, besides carefully tuning the mixture of public datasets and
production data, we in addition apply Wasserstein Distance Guided Domain
Adversarial Training (WDAT) to encourage the model to learn domain invariant
representations (Arjovsky, Chintala, and Bottou 2017; Ganin et al. 2016).

We follow Shen et al. (2018) and approximate the Wasserstein distance by
maximizing the loss of a domain critic head. Let
$f_{z}(x):\mathbb{R}^{d}\rightarrow\mathbb{R}^{z}$ be the feature extractor
that maps the $d$-dimensional input into a $z$-dimensional embedding,
$f_{c}(h):\mathbb{R}^{z}\rightarrow\mathbb{R}^{c}$ be a multiclass
classification head, and $f_{d}(h):\mathbb{R}^{z}\rightarrow\mathbb{R}$ be the
domain critic head that maps the embedding into real number. The domain critic
loss is defined as

$\mathcal{L}_{d}(\mathcal{D}_{s},\mathcal{D}_{t})=|\mathop{\mathbb{E}}_{x\in\mathcal{D}_{s}}f_{d}(f_{z}(x))-\mathop{\mathbb{E}}_{x\in\mathcal{D}_{t}}f_{d}(f_{z}(x))|.$

Combined with the regular classification loss $\mathcal{L}_{c}$, our objective
is to solve the following minimax problem:

$\min_{\theta_{z},\theta_{c}}\\{\mathcal{L}_{c}+\lambda\max_{\theta_{d}}\mathcal{L}_{d}\\},$

where $\theta_{z},\theta_{c},\theta_{d}$ are the parameters of
$f_{z},f_{c},f_{d}$, respectively. Our model uses a transformer encoder as the
feature extractor $f_{z}$.

In our implementation, we use the absolute value in $\mathcal{L}_{d}$ since
the initial loss could be negative, and clip $\theta_{d}$ in a compact space
$[-0.01,0.01]$ to enforce the Lipchitz constraint. We empirically set the
balancing coefficient $\lambda$ to 0.01. In experiments, WDAT achieves a more
stable training compared to the original classifier-based approach (Arjovsky,
Chintala, and Bottou 2017), and yields better performance on our production
traffic with and without labeled production data in the training set.

###  3.5 Model Probing

It is widely known that ML models are vulnerable to adversarial inputs and may
make predictions based on seemingly irrelevant features (Szegedy et al. 2013;
Goodfellow, Shlens, and Szegedy 2015; Alzantot et al. 2018; Zhang et al.
2021). For instance, a sentiment classification model may make different
predictions for "a short and moving film." and "a short and moving movie.",
indicating that the model makes predictions based on film/movie rather than
descriptive words. To make sure that our model performs as expected, we
conduct tool-assisted model probing and human-in-the-loop red-teaming.

#### Key Tokens Probing.

Our first goal is to verify whether the model is making predictions based on
correct features. For instance, we observed that our models may incorrectly
predict "all immigrants." as hate, which is possibly because among 21 training
examples containing the phrase (from Röttger et al. (2021)), 17 examples are
undesired and only 4 examples are safe. Table 2 provides a few examples, where
the expected behavior is to also take k1ll and D3ath into account. To identify
such potentially overfitted key tokens, we apply input reduction (Feng et al.
2018) on the training dataset by removing tokens greedily until the prediction
drops below 0.8. On average, we were able to reduce the number of characters
per sample from 722.3 to 15.9. By cross-validating with third-party APIs and
heuristic keywords, we infer that about 97% of the identified key tokens are
indeed unsafe, indicating that the model is picking up the correct feature
most of the time. For the remaining 3% key tokens, we manually reviewed them
and discovered the following problems:

  1. 1.

The frequency of irrelevant symbols is high (e.g., the period symbol
contributes to the prediction score).

  2. 2.

Some key tokens are long and close to the original example, hinting that the
model memorizes training samples.

  3. 3.

Our model is not strong at handling typos and adversarial inputs (e.g., "w h o
r e s" becomes "e s").

We partially addressed the above issues, and plan to resolve with better data
augmentation in the future (Section 6).

Hate | Example
---|---
$1$ | I want to k1ll all immigrants.
$1$ | Why is it that all immigrants are repulsive?
$1$ | D3ath to all immigrants.
$0$ | Threatening to kill all immigrants will get
| you thrown in jail.
$\dots$ |

Table 2: Example training data from (Röttger et al. 2021) containing the
phrase "all immigrants." The model trained on it directly may make predictions
based on incorrect features.

#### Human Red-teaming.

As a final layer of the verification process, we conduct internal red-teaming
before releasing new models. The goal of the red-teaming is to uncover any
unexpected model weaknesses beyond the test dataset. During the process, we
discovered the following issues:

  1. 1.

The model predicts high hate scores for examples containing the “#” token.
This phenomenon is likely caused by the fact that we have many hate training
examples from some academic datasets that contain only short tweets.

  2. 2.

The model predicts high hate and violence scores for examples containing
potential racial tokens such as black. It is expected to classify "I hate
black people!" as hate but not "I hate black cats!" as hate.

To mitigate the above issues, we construct synthetic datasets from hand-
curated templates and synthetic model generations to patch the holes (Section
3.3), and adjust the training dataset distribution to make sure we have the
right mix across multiple types of text sourced from academic datasets. The
process can be iterative, helping us discover new issues and solutions in each
round and naturally leading to improved robustness and consistency in time
when the red-teaming process can be executed more regularly and at scale.

##  4 Experiment Results

###  4.1 Model Architecture and Training

Our model is a lightweight transformer decoder model where the final output
linear layer is replaced with 8 MLP heads, each corresponding to one
independent matrix of shape $[d_{\text{model}},256,1]$, where
$d_{\text{model}}$ is the transformer model size. We find this head
architecture works better than a single deep MLP layer with one output vector
of 8 dimensions at avoiding interference between categories and requires fewer
parameters to train.

The model is initialized from a GPT model that is pretrained on a large text
corpus and then fine-tuned with learning rate 0.05, batch size 256, dropout
rate 0.1 within MLP heads and up to 3 epochs.

###  4.2 Model Performance

Our model is trained and tested on both production and public data. We are not
able to share the test dataset containing production traffic for privacy and
legal reasons; hence, we report the model performance on a different test
dataset444 https://github.com/openai/moderation-api-release containing only
samples from public data, as well as several publicly available datasets on
undesired content detection.

Table 3 compares the performance of our model with Perspective
API555https://www.perspectiveapi.com/ as a baseline on our test dataset,
TweetEval (Barbieri et al. 2020), Stormfront hate speech dataset (de Gibert et
al. 2018), a subset of Reddit comments with noisy labels on erotic content
processed according to Barrientos et al. (2020) and a downsampled Jigsaw toxic
comments test dataset (Jigsaw 2018). None of the training portion of external
evaluation benchmarks are incorporated into our training, except for half of
Jigsaw’s training data that has no overlap with the Jigsaw test set in
evaluation. Unfortunately, due to the taxonomy mismatch, we cannot have exact
comparison across all categories. For example, our taxonomy does not cover
“toxic” and Perspective API does not explicitly detect “self-harm” or “sexual
content”. See the details on how we match two taxonomies and preprocess each
test dataset in Appendix. A.

|  | Perspective | Ours
---|---|---|---
Public |  S | .8709* | .9703
|  H | .6914 | .7968
|  V | .5201 | .7371
|  HR | .3902* | .6191
|  SH | - | .8070
|  S3 | - | .7638
|  H2 | - | .7268
|  V2 | - | .6061
Jigsaw |  Identity-hate | .6644 | .6890
|  Insult | .8814 | .8548
|  Obscene | .9500 | .8353*
|  Threat | .7492 | .6144*
|  Toxic | .9769 | .9304*
TweetEval |  Hate | .5961 | .6473
|  Offensive | .7919* | .7024*
Stormfront |  Hate | .8754 | .9053
Reddit |  Sexual | .8961* | .9417*

Table 3: Comparison of our model with Perspective API on AUPRC (Area under the
Precision-Recall Curve) across a set of test datasets. Numbers followed with
”*” are based on approximated taxonomy match, so not an exact fair comparison.

It is not surprising that our model performs the best on the test dataset
labeled with the same taxonomy and the Perspective API does a better job on
Jigsaw data. It further proves the point on how important it is to align the
taxonomy between training data and use cases in evaluation. Our model
outperforms the Perspective API baseline on both TweetEval and Stormfront test
sets for detecting hateful content, despite the fact that neither are in the
training set.

###  4.3 Active Learning Experiments

To assess the importance of active learning, we evaluate the performance of
our active learning strategy, as described in Section 3.1, compared to random
sampling.

#### Iterative training.

We run the following training procedure twice, using our active learning
strategy and random sampling, respectively.

  1. 1.

Start with an initial training dataset $\mathcal{D}_{0}$ of $k_{0}=6000$
labeled examples from public data and a validation set $\mathcal{V}$ of about
5500 samples from the production traffic.

  2. 2.

for $i\leftarrow 0$ to $N-1$ do ($N=3$):

    1. (a)

Train a new model $M_{i}$ on $\mathcal{D}_{i}$;

    2. (b)

Evaluate $M_{i}$ on $\mathcal{V}$;

    3. (c)

Score $5\times 10^{5}$ production samples with $M_{i}$ from our production
traffic;

    4. (d)

Choose about $2000$ samples from the above data pool via the selection
strategy in test and add samples to the training set to construct
$\mathcal{D}_{i+1}$ after labeling.

Category |  Random Sampling |  Active Learning | Multiplier
---|---|---|---
S |  1.49% |  25.53% | 17.1$\times$
H |  0.17% |  3.09% | 18.2$\times$
V |  0.48% |  9.92% | 20.7$\times$
HR |  0.55% |  6.41% | 11.7$\times$
SH |  0.09% |  1.85% | 20.6$\times$
S3 |  0.24% |  2.42% | 10.1$\times$
H2 |  0.03% |  0.67% | 22.3$\times$
V2 |  0.25% |  4.27% | 17.1$\times$
Safe |  96.57% |  59.54% | -
Table 4: Label distributions for samples selected by random sampling and
active learning sampling. Note that one sample can be assigned with multiple
labels so the percentages sum up to more than 100%. Figure 2: Performance of
active learning sampling versus random sampling on the same validation set at
each model iteration, measured by AUPRC.

#### Results.

Table 4 demonstrates the label distributions obtained by the two strategies
and our active learning strategy can capture undesired content 10+ times more
effectively than random sampling on all categories. Overall about 40% of
samples selected by active learning can trigger at least one undesired label,
while in comparison only 3.4% of random samples are assigned with any
undesired label.

As shown in Fig. 2, using the active learning strategy to decide which new
data samples leads to a greater improvement across all categories than random
sampling. We observe significant performance improvement on all categories
with active learning after 3 iterations.

###  4.4 Domain Adversarial Training Experiments

We want to understand the effectiveness of Wasserstein Distance Guided Domain
Adversarial Training (WDAT) under three scenarios: (1) At the beginning of the
project, we only have labeled public data and unlabeled production data. (2)
In the middle of the project, we also curate synthetic examples to improve
model weaknesses. (3) At the later stage, we get a sufficient amount of
labeled production examples. All three circumstances are important because we
want to make good use of unlabeled production data to train the best model
throughout the project, and a strong model on production traffic boosts the
effectiveness of active learning at every iteration. We use the following
setup to compare the performance on our production traffic.

#### Datasets.

We create three training datasets PUB, SYN, and MIX to study (1), (2), and
(3), respectively. PUB consists of around 90k public examples including both
samples from academic datasets and Web data (Common Crawl) labeled by our
annotators. SYN adds additional 69k curated synthetic examples. MIX contains
all examples in SYN with additional 60k production samples with labels.

#### Models.

The baseline models are trained with basic supervised learning. The DAT models
are trained with two hidden layers of 300 dimensions using additional 100k
unlabeled production data points. All models are trained with up to 2 epochs,
and the training is repeated 3 times with different random seeds.

Category | PUB | SYN | MIX
---|---|---|---
| Baseline | DAT | Baseline | DAT | Baseline | DAT
S | .698 | .730 | .726 | .745 | .943 | .939
H | .417 | .491 | .551 | .476 | .843 | .818
V | .490 | .529 | .532 | .531 | .640 | .633
HR | .258 | .369 | .326 | .356 | .453 | .482
SH | .063 | .281 | .086 | .296 | .621 | .632
S3 | .592 | .759 | .779 | .777 | .911 | .936
H2 | .393 | .643 | .570 | .577 | .851 | .854
V2 | .165 | .453 | .093 | .507 | .443 | .533

Table 5: The average AUPRC on a production validation set. PUB denotes models
trained on labeled public datasets, SYN adds additional synthetic examples,
and MIX adds additional labeled production examples. We mark the best result
within each configuration in bold.

#### Results.

We compare the average AUPRC on the production validation set $\mathcal{V}$.
As demonstrated in Table 5, the improvement from WDAT is significant when we
only have access to public datasets (PUB), and the marginal gain reduces
gradually as we add more training examples, especially in-distribution
production samples. For instance, DAT improved SH AUPRC from 0.063 to 0.281 on
PUB and from 0.086 to 0.296 on SYN, whereas the improvement is only from 0.621
to 0.632 on MIX. WDAT still helps weak categories (SH and V2) on SYN and MIX,
but it may slightly hurt the performance for categories with a sufficient
amount of in-distribution data such as H and V. We suspect this is because the
model failed to find a representation that works very well for both the public
datasets and our production distribution. Further study on the model
architecture and training methods is required to improve the performance on
all categories with unlabeled data throughout different stages of the project.

##  5 Related Work

There is a long track record of work on the definition and detection of
hateful, toxic, offensive and abusive content (Kwok and Wang 2013; Nobata et
al. 2016; Waseem 2016; de Gibert et al. 2018; Vidgen et al. 2019; Gehman et
al. 2020; Rosenthal et al. 2020; Lees et al. 2022). Zampieri et al. (2019)
proposed a three-level hierarchical taxonomy considering whether the given
language is (i) offensive or not; (ii) targeted or not; and (iii) targeted at
a group, an individual or other organizations. Usually hateful expressions
targeting protected identity groups are considered hate speech (Davidson et
al. 2017). Perspective API defines toxicity as ”A rude, disrespectful, or
unreasonable comment that is likely to make people leave a discussion”. Some
also used toxicity as a general umbrella term for offensive, abusive, and
hateful language (Pavlopoulos et al. 2020). The definitions of hatefulness,
toxicity, offensiveness and abusiveness have overlaps but are not exactly the
same, creating obstacles for sharing datasets between projects. Furthermore,
only a limited amount of work considered detailed subcategorizations (Mollas
et al. 2020; Borkan et al. 2019) to capture a spectrum of severity, making it
harder to control labeling quality. Finally, there exist various types of
potentially undesired text in the wild, such as sexual content involving
minors, extreme graphic violence, or support for self-harm or suicides,
besides offensive and abusive language, and we observed a gap between current
research work and the entirety of content types that should be moderated and
detected. Our work aims to fill in the gap.

Despite the common belief that training data quality is critical for model
performance, there is still lack of community standards for labeling
standards, annotator training, quality metrics, etc. (Vidgen and Derczynski
2020; Yin and Zubiaga 2021; Lees et al. 2022; PAI 2021). Vidgen and Derczynski
(2020) studied 60+ datasets for abusive language detection and found that the
primary data source is Twitter and expert coding is the most common way to
annotate data, closely followed by crowdsourcing. For large-scale data
collection, crowdsourcing remains the most common approach (Mollas et al.
2020; Zampieri et al. 2019; Davidson et al. 2017). However, the weak skill set
of non-expert annotators can lead to lower data quality (Waseem 2016; Yin and
Zubiaga 2021). Some recent work turns to large pre-trained language models to
generate synthetic data, significantly reducing the cost of time and human
labor (Wang et al. 2021a; Hartvigsen et al. 2022), but it is unclear whether
model outputs would be diverse enough to adapt to the real-world distribution.
Synthetic data can be hand-crafted (Röttger et al. 2021), but it is limited by
size and thus more suitable for evaluation. It is noteworthy that training
data can contain bias due to the subjectivity and biases in the data
collection process (Davidson, Bhattacharya, and Weber 2019; Sap et al. 2019).

Active learning has been successfully applied to a number of different domains
such as text classification (Lewis and Gale 1994; Schohn and Cohn 2000;
Siddhant and Lipton 2018); machine translation (Zeng et al. 2019); image
classification (Luo et al. 2005; Hoi et al. 2006; Gal, Islam, and Ghahramani
2017); object detection (Schmidt et al. 2020) and information retrieval (Shen
and Zhai 2005). There are several families of active learning sampling
strategies that are often used in practice. Uncertainty sampling selects data
points about which the model is most uncertain. The uncertainty of the model
can be quantified by predicted probabilities (Lewis and Gale 1994; Lewis and
Catlett 1994; Culotta and McCallum 2005; Scheffer, Decomain, and Wrobel 2001),
disagreement among an ensemble of models (Seung, Opper, and Sompolinsky 1992;
Dagan and Engelson 1995; McCallum and Nigam 1998), or by using dropout and
Bayesian approaches (Gal, Islam, and Ghahramani 2017; Siddhant and Lipton
2018). Diversity sampling chooses samples in a way that ensures sufficient
diversity within the selection. This is commonly achieved by clustering
unlabeled data and sampling from different clusters (Nguyen and Smeulders
2004; Xu, Akella, and Zhang 2007), or by selecting samples which are
”representative” of the sample distribution (i.e., which are similar to many
other samples) (McCallum and Nigam 1998; Settles and Craven 2008). Uncertainty
and diversity sampling are sometimes combined in a single complex active
learning strategy.

Red-teaming is a common approach for model improvement by discovering and
patching the weakness iteratively (Dinan et al. 2019; Vidgen et al. 2020;
Kiela et al. 2021; Ziegler et al. 2022; Perez et al. 2022; Ribeiro et al.
2020), where humans are encouraged to look for examples that could fail the
model. Dynabench (Kiela et al. 2021) is built as a platform for easy
adversarial data collection. Mishkin et al. (2022) describes in detail an
operational process for doing red-teaming using external experts. Ziegler et
al. (2022) designed a tool to efficiently assist human adversaries to identify
failures in a classifier. Models trained with red-teaming data are found to be
more robust to adversarial attack (Dinan et al. 2019; Ziegler et al. 2022) and
human-in-the-loop dynamic data collection can efficiently improve model
performance (Kiela et al. 2021; Vidgen et al. 2020).

Domain adaptation aims at generalizing knowledge learned in the source domain
towards a related target domain (Ben-David et al. 2006; Weiss, Khoshgoftaar,
and Wang 2016; Ben-David et al. 2009), the technique is most useful when there
is insufficient labeled data in the target domain but sufficient labeled data
in the source domain. Different methods have been proposed to transfer the
knowledge across domains (Ramponi and Plank 2020; Blitzer, McDonald, and
Pereira 2006; Mansour, Mohri, and Rostamizadeh 2008). Inspired by generative
adversarial nets (GANs) (Goodfellow et al. 2014) which train a discriminator
to make the representations of source and target indistinguishable, Domain
Adversarial Training (DAT) methods are proposed to reduce the domain
discrepancy through a domain discriminator (Arjovsky, Chintala, and Bottou
2017; Ganin et al. 2016; Tzeng et al. 2017; Ganin and Lempitsky 2015). To
learn domain-invariant feature representations, DAT employs a gradient
reversal layer to maximize the minimal loss of the domain discriminator.
However, DAT suffers from a gradient vanishing problem when the domain
discriminator can tell apart the two domains easily, and Wasserstein distance
based methods are proposed to enable a more stable training (Shen et al. 2018;
Arjovsky, Chintala, and Bottou 2017; Shah et al. 2018).

##  6 Future Work and Limitations

#### Bias and Fairness.

Similar to other existing NLP models, our models also suffer from bias towards
certain demographic attributes (Kusner et al. 2017; Garg et al. 2019; Dwork et
al. 2012). For instance, the model may give higher hate predictions if the
input contains gay and higher sexual predictions if the input contains her.
This is because we use data from the Internet, and social bias may present
explicitly or implicitly in the training datasets. We tried mitigation methods
such as creating a balanced synthetic dataset with templates but could not
fully eliminate the issue. In the future, we will continue following related
research and improve the fairness of our models.

#### Data Augmentation.

We plan to investigate more data augmentation methods to boost the training
dataset. Although our current training dataset naturally includes misspelled
words and incorrect grammar as some of it is user-generated content, it is
valuable to experiment with data augmentation to improve lexicon robustness
(Wei and Zou 2019; Kobayashi 2018; Zhang et al. 2021) and the generalizability
of the model (Guo, Mao, and Zhang 2019; Shen et al. 2020; Gao, Yao, and Chen
2021), especially when working with the changing distribution of real-world
data.

#### Better Multilingual Support.

Only about 5% of the samples are non-English in our training set. As the vast
majority of our production traffic is in English, we have not yet rigorously
evaluated or optimized performance on non-English text. Multilingual toxic
content classification (Aluru et al. 2020; Wang and Banko 2021; Lees et al.
2022) would require more non-English training data and may need additional
changes on tokenization or model architecture.

#### Red-teaming at scale.

Red-teaming is an effective way to find unknown failure cases for the model.
Currently we do internal red-teaming with each new model version, which is not
a scalable approach. In the future, we plan to set up a pipeline for model
red-teaming similar to the one we have for labeling production traffic. We
plan to use a specialized interface inspired by Kiela et al. (2021); Ziegler
et al. (2022) to improve the efficiency of the red-teamers.

#### More Active Learning Experiments.

Our current active learning strategy to select high-value data for labeling is
quite simple. For example, we did not explore diversity sampling due to
computational restriction. Onward we plan to run more rigorous experiments
comparing the performance of different active learning strategies, as well as
more sophisticated strategies, incorporating both uncertainty and diversity
sampling.

##  7 Broader Impacts

Content moderation classifiers have many uses. When paired with fair and
robust enforcement practices, they have the potential to reduce certain
instances of misuse 666misuse may be defined as uses of the model that the
moderating body does not want to allow, e.g. generation of hateful content by
ensuring that policies are operationalized on both inputs and outputs of
language models. Classifiers also enable filtration of datasets at scale,
which may be used to train language models with desired properties (Welbl et
al. 2021) and allow for better evaluation of language models (Gehman et al.
2020). Longer-term, content moderation classifiers can be used as a way to
ensure high-stakes reliability in very-capable AI systems (Ziegler et al.
2022)—a critical necessity for enabling the deployment of those systems in
certain domains.

While this underscores the importance of the undesired content classifiers,
all classifiers rest on certain assumptions and decisions that may present
vulnerabilities or make them inappropriate for certain use cases or types of
text. Additionally, these tools can suffer from problematic biases, such as
disproportionate false positives when discussing groups that are frequently
the target of hate. (Garg et al. 2019)

The following sections discuss the normative and subjective questions on which
these classifiers rest and explore the challenges they present.

###  7.1 Challenges of Taxonomy Design

We take care to design our taxonomy to reflect generalizable viewpoints.
However, much of our data is drawn from a US-centric context and the taxonomy
was designed to best fit this data. Additionally, while we have designed our
taxonomy to be as comprehensive as possible, it would still be useful for
future researchers to add and update the categories based on their own use
cases and deployment contexts. Given the sensitive nature of various tasks, we
also encourage the use of this taxonomy in concert with other mitigation
strategies, as there is no silver bullet for content moderation.

We hope that this work will encourage further discussion and debate around the
principles and values that underpin content moderation.

###  7.2 Annotator Viewpoints and Disagreement

It is commonly agreed that the annotation of toxic language is subjective and
that annotators’ interpretations may be influenced by their personal and
cultural backgrounds, including lived experiences, values and demographic
factors. For example, Waseem (2016) found that feminist and anti-racist
activists systematically disagree with crowd workers on their hate speech
annotations. In their study, agreement between the authors, amateurs and
expert annotators is low (14$\%$), most often because in many instances where
the authors had identified hate speech, annotators do not.

By necessity, incorporating diverse viewpoints invites disagreement on
annotation labels. Much of the computer science literature focuses on
eliminating inter-rater disagreements, most often via deliberation or majority
vote. However, in the case of data from or about marginalized populations,
disagreement may be a meaningful signal: An adverse effect of majority vote in
such cases is limiting representation of minority perspectives in data
Prabhakaran, Mostafazadeh Davani, and Diaz (2021), potentially reinforcing
societal disparities and harms. Moreover, analyzing disagreements may lead to
a better understanding of the domain of application Patton et al. (2018).

In their study, rather than aggregating, Davani, Díaz, and Prabhakaran (2021)
preserve annotator disagreements, which they note could reflect useful and
nuanced information about the uncertainty of a sample’s membership to a class.
Indeed, they demonstrate that their approach yields the same or better
performance than similar approaches with aggregated labels, while retaining
the ability to estimate uncertainty in predictions that correlate with real-
life annotator disagreements.

Moreover, resolving disagreement via majority vote may be at odds with
preserving minority opinions in subjective tasks. Ovesdotter Alm (2011) argues
that achieving a single real ”ground truth” label is impossible and is not
essential in subjective tasks, and calls for finding ways to model subjective
interpretations of annotators, rather than seeking to reduce the variability
in annotations.

###  7.3 Annotator Selection and Welfare

We are committed to ensuring that our labeling tasks are managed in a
considerate and ethical manner, and we strive to follow current best practices
for sourcing data labeling services (PAI 2021). Via our data vendors, all of
our annotators are selected for their skill and willingness to participate in
these difficult tasks. Before they opt in, all annotators are vetted by
counselors and made aware of the risks and potential harms of working with
sensitive data. Our data vendors provide them with access to mental health and
wellness resources and annotators have the right to opt out at any point.

###  7.4 Data Privacy and Security

Trustworthy handling of production data necessitates transparency with users
and effective security measures. We obtain consent from all customers whose
data is used to train our moderation models. Customers who wish to opt their
data out of training may do so. No production data is included in the dataset
we are releasing. Our data labeling and active learning pipelines feature
security controls that are designed and tested to protect the confidentiality
and integrity of production data. The model we deploy can not be used to
generate text, only to compute safety scores, so we consider the risk of
training data leakage to be extremely low.

###  7.5 Summary of Broader Impacts Discussion

Content moderation classifiers are one key tool that empowers developers of
language models at every stage of the model development and deployment
process- from working with large-scale datasets, to testing out models, to
deploying the models to many users. However, as we have observed above, there
are a range of normative and subjective decisions made throughout the
development process of building these classifiers from designing taxonomies to
labeling data. Given the nature of these tools, these decisions are sometimes
distilled down bluntly and do not enable capturing the nuances that the
moderation decision may warrant. This loss of nuance may disproportionately
impact members of socially marginalized populations by muting their opinions
via unweighted majority annotations. This impact is doubly grievous if
moderation decisions about members of marginalized populations are made about
them by a system that excludes their input. This highlights some inherent
limitations of classifiers, using automated tools for content moderation, and
point to the importance of their robust testing to ensure suitability for each
specific use that they may be deployed in.

##  8 Conclusion

Building high-quality undesired content detection systems in the real world is
a challenge that requires the incorporation of multiple methods. A good
content taxonomy is the foundation for problem scoping and data collection. A
reliable data pipeline is needed to guarantee high data quality and to handle
distribution shift. We show that in cases where certain target content occurs
rarely, an active learning sampling strategy leads to much better model
performance. Additionally, we argue that good operational aspects of the
labeling pipeline are essential for ensuring high data quality. And we show
that model performance can further be improved through the use of curated
synthetic data and semi-supervised learning.

As large generative language models become more and more prevalent, it becomes
increasingly important to develop ways of controlling and guiding their
outputs. The goal of this work has been to demonstrate one way of implementing
such control by way of building content detection models. We are looking
forward to further refinement of our approach in the future, as well as
progress in other methods of controlling and aligning generative model
outputs.

##  9 Acknowledgments

This work would not have been possible without the contributions of data
workers. We greatly appreciate their work handling sensitive content and
helping us build better automated systems to make content moderation work less
demanding of human labor.

We also thank Miles Brundage, Raf Jakubanis, Gretchen Krueger, Derek Chen,
Summer Yue, Karl Cobbe, Pranav Shyam, Jason Kwon and Matt Knight for feedback
on this work.

## References

  * Aluru et al. (2020) Aluru, S. S.; Mathew, B.; Saha, P.; and Mukherjee, A. 2020.  Deep learning models for multilingual hate speech detection.  _arXiv preprint arXiv:2004.06465_.
  * Alzantot et al. (2018) Alzantot, M.; Sharma, Y.; Elgohary, A.; Ho, B.-J.; Srivastava, M.; and Chang, K.-W. 2018.  Generating Natural Language Adversarial Examples.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , 2890–2896. Brussels, Belgium: Association for Computational Linguistics.
  * Anaby-Tavor et al. (2020) Anaby-Tavor, A.; Carmeli, B.; Goldbraich, E.; Kantor, A.; Kour, G.; Shlomov, S.; Tepper, N.; and Zwerdling, N. 2020.  Do not have enough data? Deep learning to the rescue!  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, 7383–7390.
  * Arjovsky, Chintala, and Bottou (2017) Arjovsky, M.; Chintala, S.; and Bottou, L. 2017.  Wasserstein Generative Adversarial Networks.  In Precup, D.; and Teh, Y. W., eds., _Proceedings of the 34th International Conference on Machine Learning_ , volume 70 of _Proceedings of Machine Learning Research_ , 214–223. PMLR.
  * Barbieri et al. (2020) Barbieri, F.; Camacho-Collados, J.; Espinosa Anke, L.; and Neves, L. 2020.  TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , 1644–1650. Association for Computational Linguistics.
  * Barrientos et al. (2020) Barrientos, G. M.; Alaiz-Rodríguez, R.; González-Castro, V.; and Parnell, A. C. 2020.  Machine learning techniques for the detection of inappropriate erotic content in text.  _International Journal of Computational Intelligence Systems_ , 13(1): 591–603.
  * Ben-David et al. (2009) Ben-David, S.; Blitzer, J.; Crammer, K.; Kulesza, A.; Pereira, F. C.; and Vaughan, J. W. 2009.  A theory of learning from different domains.  _Machine Learning_ , 79: 151–175.
  * Ben-David et al. (2006) Ben-David, S.; Blitzer, J.; Crammer, K.; and Pereira, F. C. 2006.  Analysis of Representations for Domain Adaptation.  In _NIPS_.
  * Blitzer, McDonald, and Pereira (2006) Blitzer, J.; McDonald, R. T.; and Pereira, F. C. 2006.  Domain Adaptation with Structural Correspondence Learning.  In _EMNLP_.
  * Borkan et al. (2019) Borkan, D.; Dixon, L.; Sorensen, J.; Thain, N.; and Vasserman, L. 2019.  Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classification.  In _Companion Proceedings of The 2019 World Wide Web Conference_ , WWW ’19, 491–500. New York, NY, USA: Association for Computing Machinery.  ISBN 9781450366755.
  * Brown et al. (2020) Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33: 1877–1901.
  * Cohen et al. (2022) Cohen, A. D.; Roberts, A.; Molina, A.; Butryna, A.; Jin, A.; Kulshreshtha, A.; Hutchinson, B.; Zevenbergen, B.; Aguera-Arcas, B. H.; ching Chang, C.; Cui, C.; Du, C.; Adiwardana, D. D. F.; Chen, D.; Lepikhin, D. D.; Chi, E. H.; Hoffman-John, E.; Cheng, H.-T.; Lee, H.; Krivokon, I.; Qin, J.; Hall, J.; Fenton, J.; Soraker, J.; Meier-Hellstern, K.; Olson, K.; Aroyo, L. M.; Bosma, M. P.; Pickett, M. J.; Menegali, M. A.; Croak, M.; Díaz, M.; Lamm, M.; Krikun, M.; Morris, M. R.; Shazeer, N.; Le, Q. V.; Bernstein, R.; Rajakumar, R.; Kurzweil, R.; Thoppilan, R.; Zheng, S.; Bos, T.; Duke, T.; Doshi, T.; Prabhakaran, V.; Rusch, W.; Li, Y.; Huang, Y.; Zhou, Y.; Xu, Y.; and Chen, Z. 2022\.  LaMDA: Language Models for Dialog Applications.  In _arXiv_.
  * Culotta and McCallum (2005) Culotta, A.; and McCallum, A. 2005.  Reducing labeling effort for structured prediction tasks.  In _AAAI_ , volume 5, 746–751.
  * Dagan and Engelson (1995) Dagan, I.; and Engelson, S. P. 1995.  Committee-based sampling for training probabilistic classifiers.  In _Machine Learning Proceedings 1995_ , 150–157. Elsevier.
  * Davani, Díaz, and Prabhakaran (2021) Davani, A. M.; Díaz, M.; and Prabhakaran, V. 2021.  Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations.  _CoRR_ , abs/2110.05719.
  * Davidson, Bhattacharya, and Weber (2019) Davidson, T.; Bhattacharya, D.; and Weber, I. 2019.  Racial Bias in Hate Speech and Abusive Language Detection Datasets.  In _Proceedings of the Third Workshop on Abusive Language Online_ , 25–35. Florence, Italy: Association for Computational Linguistics.
  * Davidson et al. (2017) Davidson, T.; Warmsley, D.; Macy, M.; and Weber, I. 2017.  Automated hate speech detection and the problem of offensive language.  In _Proceedings of the international AAAI conference on web and social media_ , volume 11, 512–515.
  * de Gibert et al. (2018) de Gibert, O.; Perez, N.; García-Pablos, A.; and Cuadros, M. 2018.  Hate Speech Dataset from a White Supremacy Forum.  In _Proceedings of the 2nd Workshop on Abusive Language Online (ALW2)_ , 11–20. Brussels, Belgium: Association for Computational Linguistics.
  * Devlin et al. (2019) Devlin, J.; Chang, M.; Lee, K.; and Toutanova, K. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In Burstein, J.; Doran, C.; and Solorio, T., eds., _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , 4171–4186. Association for Computational Linguistics.
  * Dinan et al. (2019) Dinan, E.; Humeau, S.; Chintagunta, B.; and Weston, J. 2019.  Build it break it fix it for dialogue safety: Robustness from adversarial human attack.  _arXiv preprint arXiv:1908.06083_.
  * Dwork et al. (2012) Dwork, C.; Hardt, M.; Pitassi, T.; Reingold, O.; and Zemel, R. 2012.  Fairness through Awareness.  In _Proceedings of the 3rd Innovations in Theoretical Computer Science Conference_ , ITCS ’12, 214–226. New York, NY, USA: Association for Computing Machinery.  ISBN 9781450311151.
  * Feng et al. (2018) Feng, S.; Wallace, E.; Grissom II, A.; Iyyer, M.; Rodriguez, P.; and Boyd-Graber, J. 2018.  Pathologies of Neural Models Make Interpretations Difficult.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , 3719–3728. Brussels, Belgium: Association for Computational Linguistics.
  * Gal, Islam, and Ghahramani (2017) Gal, Y.; Islam, R.; and Ghahramani, Z. 2017.  Deep bayesian active learning with image data.  In _International Conference on Machine Learning_ , 1183–1192. PMLR.
  * Ganin and Lempitsky (2015) Ganin, Y.; and Lempitsky, V. S. 2015.  Unsupervised Domain Adaptation by Backpropagation.  _ArXiv_ , abs/1409.7495.
  * Ganin et al. (2016) Ganin, Y.; Ustinova, E.; Ajakan, H.; Germain, P.; Larochelle, H.; Laviolette, F.; Marchand, M.; and Lempitsky, V. 2016.  Domain-Adversarial Training of Neural Networks.  _J. Mach. Learn. Res._ , 17(1): 2096–2030.
  * Gao, Yao, and Chen (2021) Gao, T.; Yao, X.; and Chen, D. 2021.  SimCSE: Simple Contrastive Learning of Sentence Embeddings.  In _Empirical Methods in Natural Language Processing (EMNLP)_.
  * Garg et al. (2019) Garg, S.; Perot, V.; Limtiaco, N.; Taly, A.; Chi, E. H.; and Beutel, A. 2019.  Counterfactual Fairness in Text Classification through Robustness.  In _Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society_ , AIES ’19, 219–226. New York, NY, USA: Association for Computing Machinery.  ISBN 9781450363242.
  * Gehman et al. (2020) Gehman, S.; Gururangan, S.; Sap, M.; Choi, Y.; and Smith, N. A. 2020.  Realtoxicityprompts: Evaluating neural toxic degeneration in language models.  _arXiv preprint arXiv:2009.11462_.
  * Goodfellow et al. (2014) Goodfellow, I. J.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A. C.; and Bengio, Y. 2014.  Generative Adversarial Nets.  In _NIPS_.
  * Goodfellow, Shlens, and Szegedy (2015) Goodfellow, I. J.; Shlens, J.; and Szegedy, C. 2015.  Explaining and Harnessing Adversarial Examples.  _CoRR_ , abs/1412.6572.
  * Guo, Mao, and Zhang (2019) Guo, H.; Mao, Y.; and Zhang, R. 2019.  Augmenting data with mixup for sentence classification: An empirical study.  _arXiv preprint arXiv:1905.08941_.
  * Hartvigsen et al. (2022) Hartvigsen, T.; Gabriel, S.; Palangi, H.; Sap, M.; Ray, D.; and Kamar, E. 2022.  Toxigen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection.  _arXiv preprint arXiv:2203.09509_.
  * Hoi et al. (2006) Hoi, S. C.; Jin, R.; Zhu, J.; and Lyu, M. R. 2006.  Batch mode active learning and its application to medical image classification.  In _Proceedings of the 23rd international conference on Machine learning_ , 417–424.
  * (34) Jigsaw. ????  Perspective API.  https://www.perspectiveapi.com/.  Accessed: 2022-06-15.
  * Jigsaw (2018) Jigsaw. 2018.  Toxic Comment Classification Challenge.  https://www.kaggle.com/competitions/jigsaw-toxic-comment-classification-challenge/overview.  Accessed: 2022-06-15.
  * Kiela et al. (2021) Kiela, D.; Bartolo, M.; Nie, Y.; Kaushik, D.; Geiger, A.; Wu, Z.; Vidgen, B.; Prasad, G.; Singh, A.; Ringshia, P.; Ma, Z.; Thrush, T.; Riedel, S.; Waseem, Z.; Stenetorp, P.; Jia, R.; Bansal, M.; Potts, C.; and Williams, A. 2021.  Dynabench: Rethinking Benchmarking in NLP.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , 4110–4124. Online: Association for Computational Linguistics.
  * Kobayashi (2018) Kobayashi, S. 2018.  Contextual Augmentation: Data Augmentation by Words with Paradigmatic Relations.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , 452–457. New Orleans, Louisiana: Association for Computational Linguistics.
  * Kumar, Choudhary, and Cho (2020) Kumar, V.; Choudhary, A.; and Cho, E. 2020.  Data augmentation using pre-trained transformer models.  _arXiv preprint arXiv:2003.02245_.
  * Kusner et al. (2017) Kusner, M. J.; Loftus, J.; Russell, C.; and Silva, R. 2017.  Counterfactual Fairness.  In Guyon, I.; Luxburg, U. V.; Bengio, S.; Wallach, H.; Fergus, R.; Vishwanathan, S.; and Garnett, R., eds., _Advances in Neural Information Processing Systems 30_ , 4066–4076. Curran Associates, Inc.
  * Kwok and Wang (2013) Kwok, I.; and Wang, Y. 2013.  Locate the Hate: Detecting Tweets against Blacks.  In _Proceedings of the Twenty-Seventh AAAI Conference on Artificial Intelligence_ , AAAI’13, 1621–1622. AAAI Press.
  * Lees et al. (2022) Lees, A.; Tran, V. Q.; Tay, Y.; Sorensen, J.; Gupta, J.; Metzler, D.; and Vasserman, L. 2022.  A new generation of perspective api: Efficient multilingual character-level transformers.  _arXiv preprint arXiv:2202.11176_.
  * Lewis and Catlett (1994) Lewis, D. D.; and Catlett, J. 1994.  Heterogeneous uncertainty sampling for supervised learning.  In _Machine learning proceedings 1994_ , 148–156. Elsevier.
  * Lewis and Gale (1994) Lewis, D. D.; and Gale, W. A. 1994.  A sequential algorithm for training text classifiers.  In _SIGIR’94_ , 3–12. Springer.
  * Luo et al. (2005) Luo, T.; Kramer, K.; Goldgof, D. B.; Hall, L. O.; Samson, S.; Remsen, A.; Hopkins, T.; and Cohn, D. 2005.  Active learning to recognize multiple types of plankton.  _Journal of Machine Learning Research_ , 6(4).
  * Mansour, Mohri, and Rostamizadeh (2008) Mansour, Y.; Mohri, M.; and Rostamizadeh, A. 2008.  Domain Adaptation with Multiple Sources.  In _NIPS_.
  * McCallum and Nigam (1998) McCallum, A.; and Nigam, K. 1998.  Employing EM and Pool-Based Active Learning for Text Classification.  In _Proceedings of the Fifteenth International Conference on Machine Learning_ , ICML ’98, 350–358. San Francisco, CA, USA: Morgan Kaufmann Publishers Inc.  ISBN 1558605568.
  * Mishkin et al. (2022) Mishkin, P.; Ahmad, L.; Brundage, M.; Krueger, G.; and Sastry, G. 2022.  DALL·E 2 Preview - Risks and Limitations.
  * Mollas et al. (2020) Mollas, I.; Chrysopoulou, Z.; Karlos, S.; and Tsoumakas, G. 2020.  ETHOS: an online hate speech detection dataset.  _arXiv preprint arXiv:2006.08328_.
  * Nguyen and Smeulders (2004) Nguyen, H. T.; and Smeulders, A. 2004.  Active learning using pre-clustering.  In _Proceedings of the twenty-first international conference on Machine learning_.
  * Nobata et al. (2016) Nobata, C.; Tetreault, J.; Thomas, A.; Mehdad, Y.; and Chang, Y. 2016.  Abusive Language Detection in Online User Content.  In _Proceedings of the 25th International Conference on World Wide Web_ , WWW ’16, 145–153. Republic and Canton of Geneva, CHE: International World Wide Web Conferences Steering Committee.  ISBN 9781450341431.
  * Ovesdotter Alm (2011) Ovesdotter Alm, C. 2011.  Subjective Natural Language Problems: Motivations, Applications, Characterizations, and Implications.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , 107–112. Portland, Oregon, USA: Association for Computational Linguistics.
  * PAI (2021) PAI. 2021.  https://partnershiponai.org/paper/responsible-sourcing-considerations/.
  * Patton et al. (2018) Patton, D.; Blandfort, P.; Frey, W.; Gaskell, M.; and Karaman, S. 2018.  Annotating Twitter Data from Vulnerable Populations: Evaluating Disagreement Between Domain Experts and Graduate Student Annotators.
  * Pavlopoulos et al. (2020) Pavlopoulos, J.; Sorensen, J.; Dixon, L.; Thain, N.; and Androutsopoulos, I. 2020\.  Toxicity detection: Does context really matter?  _arXiv preprint arXiv:2006.00998_.
  * Perez et al. (2022) Perez, E.; Huang, S.; Song, F.; Cai, T.; Ring, R.; Aslanides, J.; Glaese, A.; McAleese, N.; and Irving, G. 2022.  Red teaming language models with language models.  _arXiv preprint arXiv:2202.03286_.
  * Prabhakaran, Mostafazadeh Davani, and Diaz (2021) Prabhakaran, V.; Mostafazadeh Davani, A.; and Diaz, M. 2021.  On Releasing Annotator-Level Labels and Information in Datasets.  In _Proceedings of The Joint 15th Linguistic Annotation Workshop (LAW) and 3rd Designing Meaning Representations (DMR) Workshop_. Punta Cana, Dominican Republic: Association for Computational Linguistics.
  * Ramponi and Plank (2020) Ramponi, A.; and Plank, B. 2020.  Neural Unsupervised Domain Adaptation in NLP—A Survey.  _ArXiv_ , abs/2006.00632.
  * Reddit (2022) Reddit. 2022.  Building Better Moderator Tools.  Accessed: 2022-08-04.
  * Ribeiro et al. (2020) Ribeiro, M. T.; Wu, T.; Guestrin, C.; and Singh, S. 2020.  Beyond Accuracy: Behavioral Testing of NLP Models with CheckList.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 4902–4912. Online: Association for Computational Linguistics.
  * Rosenthal et al. (2020) Rosenthal, S.; Atanasova, P.; Karadzhov, G.; Zampieri, M.; and Nakov, P. 2020.  A Large-Scale Semi-Supervised Dataset for Offensive Language Identification.  _arXiv preprint arXiv:2004.14454_.
  * Röttger et al. (2021) Röttger, P.; Vidgen, B.; Nguyen, D.; Waseem, Z.; Margetts, H.; and Pierrehumbert, J. 2021.  HateCheck: Functional Tests for Hate Speech Detection Models.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , 41–58. Online: Association for Computational Linguistics.
  * Sap et al. (2019) Sap, M.; Card, D.; Gabriel, S.; Choi, Y.; and Smith, N. A. 2019.  The Risk of Racial Bias in Hate Speech Detection.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , 1668–1678. Florence, Italy: Association for Computational Linguistics.
  * Scheffer, Decomain, and Wrobel (2001) Scheffer, T.; Decomain, C.; and Wrobel, S. 2001.  Active hidden markov models for information extraction.  In _International Symposium on Intelligent Data Analysis_ , 309–318. Springer.
  * Schick and Schütze (2021) Schick, T.; and Schütze, H. 2021.  Generating datasets with pretrained language models.  _arXiv preprint arXiv:2104.07540_.
  * Schmidt et al. (2020) Schmidt, S.; Rao, Q.; Tatsch, J.; and Knoll, A. 2020.  Advanced active learning strategies for object detection.  In _2020 IEEE Intelligent Vehicles Symposium (IV)_ , 871–876. IEEE.
  * Schohn and Cohn (2000) Schohn, G.; and Cohn, D. 2000.  Less is more: Active learning with support vector machines.  In _ICML_.
  * Settles and Craven (2008) Settles, B.; and Craven, M. 2008.  An analysis of active learning strategies for sequence labeling tasks.  In _proceedings of the 2008 conference on empirical methods in natural language processing_ , 1070–1079.
  * Seung, Opper, and Sompolinsky (1992) Seung, H. S.; Opper, M.; and Sompolinsky, H. 1992.  Query by committee.  In _Proceedings of the fifth annual workshop on Computational learning theory_ , 287–294.
  * Shah et al. (2018) Shah, D.; Lei, T.; Moschitti, A.; Romeo, S.; and Nakov, P. 2018.  Adversarial Domain Adaptation for Duplicate Question Detection.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , 1056–1063. Brussels, Belgium: Association for Computational Linguistics.
  * Shen et al. (2020) Shen, D.; Zheng, M.; Shen, Y.; Qu, Y.; and Chen, W. 2020.  A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generation.  _arXiv preprint arXiv:2009.13818_.
  * Shen et al. (2018) Shen, J.; Qu, Y.; Zhang, W.; and Yu, Y. 2018.  Wasserstein Distance Guided Representation Learning for Domain Adaptation.  In _AAAI_.
  * Shen and Zhai (2005) Shen, X.; and Zhai, C. 2005.  Active feedback in ad hoc information retrieval.  In _Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval_ , 59–66.
  * Siddhant and Lipton (2018) Siddhant, A.; and Lipton, Z. C. 2018.  Deep Bayesian Active Learning for Natural Language Processing: Results of a Large-Scale Empirical Study.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , 2904–2909. Brussels, Belgium: Association for Computational Linguistics.
  * Szegedy et al. (2013) Szegedy, C.; Zaremba, W.; Sutskever, I.; Bruna, J.; Erhan, D.; Goodfellow, I. J.; and Fergus, R. 2013.  Intriguing properties of neural networks.  _CoRR_ , abs/1312.6199.
  * Tzeng et al. (2017) Tzeng, E.; Hoffman, J.; Saenko, K.; and Darrell, T. 2017.  Adversarial Discriminative Domain Adaptation.  _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2962–2971.
  * Vidgen and Derczynski (2020) Vidgen, B.; and Derczynski, L. 2020.  Directions in abusive language training data, a systematic review: Garbage in, garbage out.  _Plos one_ , 15(12): e0243300.
  * Vidgen et al. (2019) Vidgen, B.; Harris, A.; Nguyen, D.; Tromble, R.; Hale, S.; and Margetts, H. 2019\.  Challenges and frontiers in abusive content detection.  In _Proceedings of the Third Workshop on Abusive Language Online_ , 80–93. Florence, Italy: Association for Computational Linguistics.
  * Vidgen et al. (2020) Vidgen, B.; Thrush, T.; Waseem, Z.; and Kiela, D. 2020.  Learning from the worst: Dynamically generated datasets to improve online hate detection.  _arXiv preprint arXiv:2012.15761_.
  * Wang and Banko (2021) Wang, C.; and Banko, M. 2021.  Practical Transformer-based Multilingual Text Classification.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Papers_ , 121–129. Online: Association for Computational Linguistics.
  * Wang et al. (2021a) Wang, S.; Liu, Y.; Xu, Y.; Zhu, C.; and Zeng, M. 2021a.  Want To Reduce Labeling Cost? GPT-3 Can Help.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , 4195–4205. Punta Cana, Dominican Republic: Association for Computational Linguistics.
  * Wang et al. (2021b) Wang, Z.; Yu, A. W.; Firat, O.; and Cao, Y. 2021b.  Towards zero-label language learning.  _arXiv preprint arXiv:2109.09193_.
  * Waseem (2016) Waseem, Z. 2016.  Are You a Racist or Am I Seeing Things? Annotator Influence on Hate Speech Detection on Twitter.  In _Proceedings of the First Workshop on NLP and Computational Social Science_ , 138–142. Austin, Texas: Association for Computational Linguistics.
  * Wei and Zou (2019) Wei, J.; and Zou, K. 2019.  EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , 6383–6389. Hong Kong, China: Association for Computational Linguistics.
  * Weidinger et al. (2021) Weidinger, L.; Mellor, J.; Rauh, M.; Griffin, C.; Uesato, J.; Huang, P.-S.; Cheng, M.; Glaese, M.; Balle, B.; Kasirzadeh, A.; et al. 2021.  Ethical and social risks of harm from language models.  _arXiv preprint arXiv:2112.04359_.
  * Weiss, Khoshgoftaar, and Wang (2016) Weiss, K. R.; Khoshgoftaar, T. M.; and Wang, D. 2016.  A survey of transfer learning.  _Journal of Big Data_ , 3: 1–40.
  * Welbl et al. (2021) Welbl, J.; Glaese, A.; Uesato, J.; Dathathri, S.; Mellor, J.; Hendricks, L. A.; Anderson, K.; Kohli, P.; Coppin, B.; and Huang, P.-S. 2021.  Challenges in detoxifying language models.  _arXiv preprint arXiv:2109.07445_.
  * Xu, Akella, and Zhang (2007) Xu, Z.; Akella, R.; and Zhang, Y. 2007.  Incorporating diversity and density in active learning for relevance feedback.  In _European Conference on Information Retrieval_ , 246–257. Springer.
  * Yin and Zubiaga (2021) Yin, W.; and Zubiaga, A. 2021.  Towards generalisable hate speech detection: a review on obstacles and solutions.  _PeerJ Computer Science_ , 7: e598.
  * Yoo et al. (2021) Yoo, K. M.; Park, D.; Kang, J.; Lee, S.-W.; and Park, W. 2021.  GPT3Mix: Leveraging large-scale language models for text augmentation.  _arXiv preprint arXiv:2104.08826_.
  * YouTube (2019) YouTube. 2019.  The Four Rs of Responsibility, Part 1: Removing Harmful Content.  Accessed: 2022-08-04.
  * Zampieri et al. (2019) Zampieri, M.; Malmasi, S.; Nakov, P.; Rosenthal, S.; Farra, N.; and Kumar, R. 2019\.  Predicting the Type and Target of Offensive Posts in Social Media.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , 1415–1420. Association for Computational Linguistics.
  * Zeng et al. (2019) Zeng, X.; Garg, S.; Chatterjee, R.; Nallasamy, U.; and Paulik, M. 2019.  Empirical evaluation of active learning techniques for neural MT.  In _Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019)_ , 84–93.
  * Zhang et al. (2021) Zhang, C.; Zhao, J.; Zhang, H.; Chang, K.-W.; and Hsieh, C.-J. 2021.  Double Perturbation: On the Robustness of Robustness and Counterfactual Bias Evaluation.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , 3899–3916. Online: Association for Computational Linguistics.
  * Zhao et al. (2017) Zhao, J.; Wang, T.; Yatskar, M.; Ordonez, V.; and Chang, K.-W. 2017.  Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , 2979–2989. Copenhagen, Denmark: Association for Computational Linguistics.
  * Ziegler et al. (2022) Ziegler, D. M.; Nix, S.; Chan, L.; Bauman, T.; Schmidt-Nielsen, P.; Lin, T.; Scherlis, A.; Nabeshima, N.; Weinstein-Raun, B.; de Haas, D.; et al. 2022.  Adversarial Training for High-Stakes Reliability.  _arXiv preprint arXiv:2205.01663_.

##  Appendix A Experiment Details

Table 6 presents how we map model taxonomies into labels of different
evaluation datasets. Some of the mappings are only approximation. For example,
Perspective defines ”threat” as ”Describes an intention to inflict pain,
injury, or violence against an individual or group.”, not including graphic
violence, so not a perfect match for our ”violence” category. Either or our
taxonomy has a good match for ”toxic”, ”severe toxic”, or ”offensive.

Our Evaluation Set. We are aware that about 4% of our evaluation samples are
in non-English. Perspective API call takes the language as an input parameter,
but multilingual is not supported for several attributes. We instead use ”en”
for all the calls.

Jigsaw. Jigsaw dataset is pretty large and we include about half of it into
our training set to resolve the cold-start problem. Among the rest half, we
sampled 5000 examples for evaluation.

TweetEval. We take the TweetEval (Barbieri et al. 2020) test
datasets777https://github.com/cardiffnlp/tweeteval/tree/main/datasets on
”hate” and ”offensive”. There are in total 2970 samples in the hate task test
set and 860 in the offensive one.

Stormfront. We use the test dataset of de Gibert et al.
(2018)888https://github.com/Vicomtech/hate-speech-dataset, containing 478
samples.

Reddit. We downsampled 5000 examples from the ”RS_201501” snapshot of Reddit
pushshift datasets999https://files.pushshift.io/reddit/submissions/ and
assigned noisy binary label to each example on whether it contains sexual
content according to the subreddits as listed in Barrientos et al. (2020).

Taxonomy |  Perspective | Ours
---|---|---
Ours | Sexual |  $\max$(sexually_explicit, profanity, flirtation) | sexual
| Hate |  identity_attack | hate
| Violence |  threat | violence
| Harassment |  max(toxicity, severe_toxicity, insult, threat) | harassment
| Sexual/minors |  - | sexual/minors
Jigsaw | Toxic |  toxicity | harassment
| Obscene |  $\max$(sexually_explicit, profanity) | sexual
| Threat |  threat | violence
| Insult |  insult |  $\max$(harassment, hate)
| Identity hate |  identity_attack | hate
TweetEval | Hate |  identity_attack | hate
| Offensive |  $\max$(toxicity, severe_toxicity, threat, insult, identity_attack) | harassment
Stormfront | Hate |  identity_attack | hate
Reddit | Sexual |  $\max$(sexually_explicit, profanity, flirtation) | sexual

Table 6: How taxonomies of different APIs get mapped into labels of various
evaluation datasets.