# HateCOT: An Explanation-Enhanced Dataset for Generalizable Offensive Speech
Detection via Large Language Models

Huy Nghiem
University of Maryland
<EMAIL_ADDRESS>

Hal Daumé III
University of Maryland & Microsoft Research
<EMAIL_ADDRESS>

###### Abstract

The ubiquitousness of social media has led to the need for reliable and
efficient detection of offensive content to limit harmful effects. This has
led to a proliferation of datasets and models related to detecting offensive
content. While sophisticated models have attained strong performance on
individual datasets, these models often do not generalize due to differences
between how “offensive content” is conceptualized, and the resulting
differences in how these datasets are labeled. In this paper, we introduce
HateCOT, a dataset of 52,000 samples drawn from diverse existing sources with
explanations generated by GPT-3.5-Turbo and human-curated. We show that pre-
training models for the detection of offensive content on HateCOT
significantly boots open-sourced Language Models on three benchmark datasets
in both zero and few-shot settings, _despite differences in domain and task._
We further find that HateCOT enables effective K-shot fine-tuning in the low-
resource settings. Our repository is available at https://github.com/hnghiem-
usc/hatecot.

##  1 Introduction

As social media has become an indispensable component of modern discourse,
this channel of communication has also amplified the propagation of offensive
content. Speech aimed to promote hateful sentiments thrive on such platforms,
leading to real and significant harm on their audience Giachanou and Rosso
(2020); Saha et al. (2019). However, “offensive content” is an essentially
contested construct, and exactly what is and is not allowed varies by
platform. In research, different approaches analyze semantically similar but
still distinct concepts: Cyber-bullying, Toxicity, Sexist, Racist, Hate etc.
Poletto et al. (2021); Fortuna et al. (2021); Nghiem et al. (2024), making
visible this contestedness.

Compounding the challenge, reliable detection of offensive content requires
significant data resources. Sophisticated models tend to be data-hungry, and
the process of curating a dataset tailored to specific use cases can be
costly, time-consuming, and emotionally challenging for annotators Founta et
al. (2018); Toraman et al. (2022). The typical pipeline consists of collecting
samples based on topic-relevant key words, then recruiting either crowdworkers
or experts to annotate data before developing classification models Paullada
et al. (2021). Each step incurs investment and may inject subtle
idiosyncrasies to the downstream dataset, further limiting potential
transferable usefulness to related tasks Fortuna et al. (2021).

In practical settings, users may desire transparency from social media
platforms. Therefore, the ability to provide human understandable
justification based on platform-specific policy becomes an attractive feature
for content moderation. Nevertheless, current techniques often still fail to
offer intuitive explanatory signals, especially to end users Yadav et al.
(2023); Babaeianjelodar et al. (2022); Ibrahim et al. (2022).

In this work, we attempt to simultaneously reduce the cost of data curation,
enhance cross-dataset generalization, and address the necessity of explainable
decisions for offensive content detection. Specifically, our main
contributions are:

  1. 1.

We curate and release HateCOT (Hate-related Chains-of-Thought) a dataset of
$52k$ samples consisting of input text, a hate speech label, and an
explanation of that label. We collect this data by merging eight datasets with
explanations created by using GPT-3.5-Turbo to augment human annotations.

  2. 2.

We demonstrate the efficacy of using HateCOT as a pre-training corpus to
enable finetuning on a target domain and task. We show that open-sourced LLMs
can effectively leverage definitions to adapt to new tasks using zero-shot and
few-shot settings via finetuning and in-context learning across 3 datasets.

  3. 3.

We assess the quality of explanations generated by our finetuned models with
respect to the criteria described in their corresponding dataset’s
definitions. These insights showcase LLM-generated explanations as a means to
enhance transparency in content moderation.

##  2 Related Works

###  2.1 Offensive Speech Detection

Offensive speech detection has attracted considerable interest from the
research community. Earlier approach typically investigated coarse-grained
labels (e.g. Hate vs Not Hate) while subsequent efforts explored more diverse
facets of offensive speech at higher granularity Founta and Specia (2021);
Poletto et al. (2021); Vidgen and Derczynski (2020). Increasingly more
advanced models emerged over time with the diversity of datasets.

Cross-domain generalization, however, still remains a relevant challenge in
the area. Fortuna et al. (2021) found empirically that cross-dataset
transference is highly dependent on semantic similarity between their label
spaces. Recent works have pre-trained Transformer-based models, such as
HateBERT and fBERT, on specialized corpora to enhance generalization to
various levels of success Caselli et al. (2021); Sarkar et al. (2021).

###  2.2 LLMs in Offensive Speech Classification

Zampieri et al. (2023) assessed a range of open-sourced LLMs on zero-shot
prompting on the OffensEval task and found their performance trailing by a
wide margin behind trained existing fine-tuned BERT-based systems. Chiu et al.
(2021) and Han and Tang (2022) used the proprietary GPT-3 on a set of
different datasets and noted that informative contexts and examples could
boost the model’s performance to competitive levels on a different set of
data. Similarly, Roy et al. (2023) found that adding explanation to pipeline
could result in 10 to 20% boost in performance of LLMs over baselines. (Yang
et al., 2023)’s study found that training LLMs with step-by-step reasoning
grounded by annotations could improve predictive power.

Pre-trained language models have exhibited remarkable ability in text
generation Celikyilmaz et al. (2020). Recent scaled-up LLMs such as the GPT-3
and later models are capable of astoundingly fluent, convincing and knowledge-
infused outputs Zhang et al. (2023). LLMs with hundreds or more parameters
even exhibit reasoning capabilities Wei et al. (2022, 2021), leading to a
flurry of research on prompting techniques to harness their prowess, such as
Chain-of-thought (COT), Tree-of-thought etc. (Yao et al., 2023; Diao et al.,
2023). An interesting line of research leverages LLMs to efficiently generate
high volumes of synthetic data for tasks with training resource is scarce Puri
et al. (2020); Bao et al. (2023); Whitehouse et al. (2023). We build upon
these works to construct a dataset that can induce smaller LLMs to efficiently
adapt to new categories of offensive content by leveraging their provided
definitions.

##  3 Building HateCOT

We first describe the process to identify the candidate datasets from
literature, and the procedure to obtain annotation-guided explanations from
these samples (Section 3.1). We then perform a set of validation experiments
to optimize the data’s parameters for downstream tasks before augmenting our
corpus to its eventual size (Section 3.2).

###  3.1 Data Selection

##### Datasets for Training.

We use the following criteria to filter existing corpora related to offensive
speech detection:

  * $\triangleright$

Size: datasets should contain more than 5,000 samples to ensure adequate size
for subsequent sampling.

  * $\triangleright$

Label: datasets should contain diverse label space that address different
facets of offensive language. Both neutral and non-neutral categories should
be included for parity.

  * $\triangleright$

Definition: each dataset should have the associated definitions with each
label available. This criteria is important to generate informative
explanations.

  * $\triangleright$

Target / Rationale: the dataset should elicit the targets and/or reasoning to
justify their choice of label (rationale) of offensive sentiments, whether
through discrete labels or free text.

These criteria significantly reduce the number of eligible candidates since
many do not contain information on target and the associated definitions.
Table 1(a) lists the 8 selected datasets.

##### Datasets for Evaluation.

Using similar criteria, we also select 3 datasets with different label spaces
and definitions to test downstream models’ generalizability (details in Table
1(b)).

HateCheck was created by Röttger et al. (2021) with the explicit goal of
evaluating hate speech detection models. The authors recruited and trained 10
annotators on label the cases on a binary schema: Hateful or Non-hateful, and
reported an almost perfect final inter-annotator agreement of 0.93.

HateXplain was constructed by Mathew et al. (2021). The authors asked
annotators to highlight the span of tokens, called rationales, that contribute
to their selection of the labels: Hate, Offensive, Normal. This dataset is
primarily collected from Twitter and the Gab platform.

Latent_Hate was created by ElSherief et al. (2021a) on the premise that
offensive speech classifiers tend to bias towards covert negative sentiment.
With 22, 584 samples, this dataset is collected from Twitter( we discard the
augmentation from the Social_Bias dataset avoid contaminating the training
corpus (Sap et al., 2019)) . This set contains 2 coarse-grained labels Not
Hate, Explicit Hate, Implicit Hate, while a subset contains 6 fine-grained
categories, which we refer to as Implicit_Hate.

Dataset |  Total Size |  Sample Size |  Platform |  Target |  Ration. |  Selected Labels
---|---|---|---|---|---|---
Salminen et al. (2018) |  137,098 |  5, 418 |  Y, F |  |  ✓ |  Hateful, Neutral
Qian et al. (2019) |  34,000 |  5, 034 |  G |  |  ✓ |  Not Hate, Hate
Sap et al. (2019) |  44,671 |  6, 033 |  G, R, T |  ✓ |  ✓ |  Not Offensive, Offensive
Vidgen et al. (2021a) |  27,494 |  6, 717 |  R |  ✓ |  ✓ |  Neutral, Person Directed Abuse, Affiliation Directed Abuse, Identity Directed Abuse
Vidgen et al. (2021b) |  10,152 |  7, 209 |  S |  ✓ |  |  None, Derogation, Dehumanization, Animosity, Support, Threatening
Sachdeva et al. (2022) |  135,556 |  7, 272 |  Y, T, R |  ✓ |  |  Not Hate speech, Hate Speech
Hartvigsen et al. (2022) |  274,186 |  7, 239 |  S |  ✓ |  |  Benign, Toxic
Toraman et al. (2022) |  100,000 |  7, 215 |  T |  |  ✓ |  Normal, Offensive, Hate
Total |  - |  52, 137 |  |  |  |
(a) Datases used to create training corpus. Sample Size denotes the number of
chosen samples from corresponding dataset included in the training corpus.

Dataset |  Total Size |  Split Ratio |  K val |  K test |  Platform |  Target |  Ration. |  Selected Labels
---|---|---|---|---|---|---|---|---
HateCheck |  3,728 |  50:50 |  300 |  500 |  S |  ✓ |  |  Non-hateful, Hateful
HateXplain |  20,148 |  60:40 |  200 |  400 |  G, T |  ✓ |  ✓ |  Normal, Offensive, Hate
Latent_Hate |  19,112 |  60:40 |  200 |  400 |  T |  ✓ |  ✓ |  Not Hate, Explicit Hate, Implicit Hate
Implicit_Hate |  4,153 |  60:40 |  - |  150 |  T |  ✓ |  ✓ |  White Grievance, Incitement to Violence, Inferiority Language, Irony, Stereotypes and Misinformation, Threatening and Intimidation
(b) Datasets for testing

Table 1: Sample Size denotes the number of entries in the final corpus. Target
and Ration. indicates the availability of annotation on Target or Rationale in
the dataset. For Platform, F: Facebook, Y: Youtube, G: Gab, R: Reddit, S:
Synthetic, T: Twitter. K val and K test represent the number of sampler per
class drawn during development of the training corpus and final testing,
respectively. Full definitions in Table 6 and 7.

##### Obtaining Annotation-Guided Explanation.

Yang et al. (2023) showed that using GPT-3 to augment human-written rationales
creates linguistically coherent while maintains faithfulness to the original
content. We use the template shown in Figure 1 to generate the explanation for
each input guided by the available annotations on label, target, and rationale
from the chosen corpora. For datasets that contains multiple annotations for
each sample, we select the ultimate label via majority voting and concatenate
individual annotations on the targets and/or rationales into a single string
separated by the delimiter "|".

Figure 1: Template used to obtain explanations from GPT-3.5-Turbo guided by
human-annotated rationales.

We choose GPT-3.5-Turbo accessed via the OpenAI’s API to generate explanations
due to the balance between the model’s affordability and its capability to
follow instructions and generate coherent outputs (Ye et al., 2023a; Koubaa,
2023). For each of the 8 training datasets, we first randomly select and
qualitatively analyze 20 samples to ensure the generated explanations are a)
stylistically coherent, b) consistent with the provided labels, and c)
congruent with the criteria denoted by the definitions. If deemed
unsatisfactory, we iteratively adjust the input prompt until the quality
threshold is achieved. Appendix A.3 reveals more specific details on the final
prompts for different scenarios.

###  3.2 Optimization of Synthesized Corpus

Extending previous works Magister et al. (2022); Ho et al. (2022), we are
interested in optimizing 2 parameters central to the construction of our
corpus: the distribution of neutral vs. non-neutral classes in the data and
the number of explanations per sample. The former has been noted to influence
predictive powers Rathpisey and Adji (2019); Casula and Tonelli (2020). The
latter, also referred to as degree of reasoning diversity, could improve
knowledge distillation Ho et al. (2022).

####  3.2.1 Setup

Below are the experiments we perform on a sample of collected data to optimize
these 2 parameters.

##### Description of Procedure.

We choose 1,000 samples from each of the eight training datasets based on the
following distribution: 20% are selected from neutral samples (categories that
do not indicate any offensive content, e.g., Not Hate, Normal), while the
remaining samples are evenly distributed among the non-neutral categories.
Inspired by Ho et al. (2022) that diverse reasoning paths could improve
knowledge distillation, we collect 4 alternative explanations, or degree of
reasoning diversity, generated by GPT-3.5-Turbo for these samples using
temperature 0.7, resulting in 32,000 samples.

Figure 4 illustrates the Alpaca-styled template to format each post with its
corresponding label, generated explanation, definitions along with the
instruction into blocks of an input prompt (Taori et al., 2023). Using the
described corpus, we supervised finetune the open-sourced model Llama 2 Chat-
HF of 7 billion parameters (hereby referred to as Llama 7B) (Touvron et al.,
2023) via LoRA techniques (Technical details specified in Appendix A.2) (Hu et
al., 2021). Then, we perform zero-shot classification using the same template
to prompt the fine-tuned models to generate the explanation and label for
posts drawn from the 3 test datasets. For this part, posts are drawn using
K-shot sampling (equal number of samples for each class) on the Validation set
of the test data (K val in Table 2(b)). We omit Implicit_Hate due to this
set’s markedly different 6-label space.

##### Experiment Configurations.

For the first experiment on the ratio of neutral vs. non-neutral categories,
the training data is split into subsets with their respective distribution
between the neutral (NE) class and non-neutral class(es) (NN) described by the
following formula: $NN=R*NE$, where $R\in\\{1,2,3,4\\}$ is the ratio
coefficient. We set the number of explanations per sample to 2 for this part.
For the second experiment, we construct the subsets by varying the degree of
reasoning diversity $D\in\\{1,2,3,4\\}$, of each post.

##### Answer Extraction.

We extract the explanation and predicted labels after their respective tag in
the generated output. If the models generate multiple items from the dataset’s
label space, we select the first legal label. If no acceptable output is
obtained, we randomly select an item in the label space.

####  3.2.2 Insights and Augmentation

We report the model’s macro F1-scores on the validation set of each
configuration in Table 2. A balanced distribution between neutral and non-
neutral classes in the training corpus is beneficial, as reflected by the
substantially high mean F1-score of 0.643 when R=1. On the other hand, having
3 explanations per sample (D=3) achieves the best overall performance across 3
test sets, consistent with Ho et al. (2022)’s findings on the benefit of
multiple reasoning paths. Nevertheless, performance markedly degrades when
D=4. Based on manual analysis, we hypothesize that the quality of generated
outputs degrades as the degree of diversity increases, consequently affecting
the performance of models trained on this data. These insights are valuable to
practitioners when curating synthetic data from LLMs.

From these results, we augment the training corpus by selecting an additional
approximately 1,800 samples from each of the 8 datasets while preserving the
1:1 balanced ratio of neutral to non-neutral classes. Then we collect 3
explanations per sample using the described mechanism, resulting in a final
corpus of 52,137 samples (Table 1(a)), hereby referred to as the HateCOT
corpus.

| R=1 | R=2 | R=3 | R=4
---|---|---|---|---
HateCheck | 0.879 | 0.750 | 0.650 | 0.574
HateXplain | 0.534 | 0.533 | 0.495 | 0.528
Latent_Hate | 0.516 | 0.473 | 0.456 | 0.408
Average | 0.643 | 0.585 | 0.534 | 0.503
(a) Results for Ratio configurations (R)

| D=1 | D=2 | D=3 | D=4
---|---|---|---|---
HateCheck | 0.851 | 0.879 | 0.864 | 0.783
HateXplain | 0.549 | 0.534 | 0.597 | 0.607
Latent_Hate | 0.480 | 0.516 | 0.477 | 0.465
Average | 0.627 | 0.643 | 0.646 | 0.618
(b) Results for number of explanations per sample (D)

Table 2: Macro F1-scores for different configurations of distribution between
neutral vs. non-neutral classes (top) and number of explanations per sample
(bottom) on validation set. Best average performance in bold.

##  4 Experiments on Test Sets

We perform experiments to answer 3 questions. First, does HateCOT improve
zero-shot classification of open-sourced LLMs on unseen datasets? Second, how
much data is necessary to enable competitive performance via in-domain fine-
tuning after pretraining on HateCOT? Finally, is in-context learning a viable
alternative to finetuning?

Models In addition to Llama 7B in Section 3.2.1, the following open-sourced
models are selected.

  * $\triangleright$

Llama 13B A larger variant of the instruction-tuned Llama 7B with 13 billion
parameters.

  * $\triangleright$

OPT-IML Based on the original OPT (Open Pre-trained Transformer Language
Models) (Zhang et al., 2022), this encoder-only model contains 1.3 billion
parameters and was further trained on the Instruction MetaLearning (IML)
dataset Iyer et al. (2022).

  * $\triangleright$

Flan-T5-L Chung et al. (2022) further instruction-finetuned the encoder-
decoder T5 family of models (Raffel et al., 2020). We use the Large version of
780 million parameters.

  * $\triangleright$

COT-T5-XL This model is a variant of the Flan-T5-XL (3 billion parameters)
model that is further finetuned on the CoT dataset, a collection of 1.8
million samples augmented with chain-of-thought-style explanations (Kim et
al., 2023).

###  4.1 Zero-shot Classification

This setting involves prompting the models to perform the classification task
with no in-context examples via 2 modes: No Explanation, which prompts the
model to directly predict the label for the given input, and With Explanation,
which requires a justification before the predicted label. We finetune the
base models using only HateCOT and evaluate their performance on the 3 test
sets as in Section 3.2.2. Relevant technical details on pre-training are
presented in Appendix A.2.

With results presented in Figure 2(a), we observe that the smaller Flan-T_L
and OPT-IML are not able to generate explanations even when prompted to. In
contrast, their scaled-up counterparts could follow instructions at all
settings, indicating that size matters to effectively exploit instruction-
tuning Wei et al. (2021)). Asking Base (off-the-shelve) models to provide
explanations before label provides observable boost to Llama models in
HateCheck and HateXPlain datasets, but actually hampers performance in
Latent_Hate (and its derivative Implicit_Hate), a dataset noted for to be
challenging for classifiers due to its covert nature ElSherief et al. (2021b).

Finetuning on HateCOT unanimously enables all models to generate explanations.
While smaller models receive no observable boost in their performance, larger
models (COT-T5-XL, Llama 7B, Llama 13B) are considerably enhanced compared to
their base counterparts. For COT-T5-XL, with the exception of HateXplain,
HateCOT-trained version attain F1 scores increment of 7.6% (HateXplain and
9.5% (Latent_Hate) over the base counterpart without explanations. Llama 7B
observes 23.9%, 25.6%, 10% increment for HateCheck, HateXplain, Latent_Hate,
respectively. These statistics are 27.9%, 118.5%, and 10.2% for Llama 13B. For
Implicit_Hate, all models struggle to attain satisfactory performance, with
COT-T5-XL leading the pack. Llama models’ performance improve with HateCOT,
but still not much better than chance.

(a) Macro F-1 scores of LLMs in zero-shot setting using 3 configurations. Base
refers to out-of-the-box models, +HateCOT denotes their finetuned counterpart
on our dataset. FT5L: Flan-T5-L, CT5XL: COT-T5-XL. Results for Base Flan-T5-L
and OPTIML models for With Explanation settings omitted to reflect their
inability to generate explanation according to the prompt.

(b) Macro F-1 scores for models in zero-shot setting with explanation after
K-shot in-domain finetuning at various values of K. Dashed line represents
finetuned base models, solid line represents models pre-trained on HateCOT,
then in-domain finetuned.

(c) Min, max and mean of macro F-1 scores for Llama 7B and Llama 13B over 5
seed using in-context learning. Pretrained denotes models finetuned on HateCOT
only. Finetuned denotes ICL performed on models that have been both HateCOT-
pretrained then K=256 shot in-domain finetuned.

Figure 2: Performance resutls of LLMs on test sets in various settings.

###  4.2 In-Domain Finetuning

We further finetune COT-T5-XL, Llama 7B, Llama 13B using data from the
training set of the test datasets, including the Implicit_Hate set with 6-way
classification. To simulate low-resource settings, we select 256 samples
uniformly at random from each class of labels from the datasets, then augment
them with explanations in the same fashion as described in Section 3.1. We
perform K-shot sample (K $\in\\{32,64,128,256\\}$) to finetune and evaluate
the models on the test sets. This process applies to both Base and Pretrained
versions.

Results (Figure 2(b)) showcase the general superiority of HateCOT-pretrained
models over their base counterparts, indicating enhanced generalizability.
Finetuning on too small training data (K$\leq$64) may impair models’
performance compared to their zero-shot state, likely a result of attempting
to optimize a large number of parameters on limited signals. Stable gains are
attained for K=128, and at K=256, significant boost over the non-finetuned
zero-shot results are observed.

Interestingly, decoder-only models (Llama) considerably outperform encoder-
decoder COT-T5-XL on the 2 and 3-way classification tasks, yet the reverse is
observed for the nuanced 6-way Implicit_Hate. In fact, only COT-T5-XL
consistently scales with the increment in training data to reach max F-1 score
of 0.56, while Llama models plateau even at K=256 on this task at sub-0.3
range.

###  4.3 In-context Learning

As an alternative to in-domain fine-tuning, we investigate the models’
performance using in-context learning (ICL), when a number of complete
examples are provided as part of the input prompt. We select 1 sample from
each class in the training data of each dataset, then obtain its the
associated explanation from GPT-3.5-Turbo. The sets of post, explanation and
label are arranged in the same format in the same template as in Figure 4. We
run inference for classification results over 5 random seeds, which also
randomly permutes the samples’ order.

Figure 2(c) shows the mean, minimum and maximum values macro F-1 scores over
the seeds for the base, HateCOT-tuned only (pre-trained), and in-domain
finetuned (K=256) versions of Llama 7B and Llama 13B. COT-T5-XL regularly
generates overly repetitive outputs, and thus omitted. The range of F-1 scores
are large regardless of settings, an observation in line with the variance of
in-context learning in literature Lu et al. (2022); Dong et al. (2022).
Unsurprisingly, base models’ performances tend to be inferior to their tuned
counterpart. Finetuned models display superior performance than the other 2
counterparts on average. Interestingly, the max F-1 scores of finetuned models
in ICL are not appreciably better than those in the zero-shot counterparts
(Figure 2(b)). In contrast, except for Llama 7B on HateXplain, the best scores
of pretrained models approach those of the finetuned models–particularly for
Llama 13B.

This finding suggests another advantage of pre-training on HateCOT then
maximize performance via ICL without additional in-domain finetuning, an area
that has attracted growing attention Min et al. (2021); Wang et al. (2023); Ye
et al. (2023b). Nevertheless, there exists the trade-off: ICL examples with
explanations extend significantly the context length, and ICL inferencing
takes considerably more time compared to zero-shot, making the latter more
resource-efficient overall.

###  4.4 Overall Assessment

Our experiments demonstrate that HateCOT confers various benefits relating to
generalizing offensive speech detection. Priming LLMs using HateCOT via
supervised training significantly enhances zero-shot predictive power.
Competitive performance is possible to obtained using significantly less in-
domain training data, a bottleneck factor on resources in the traditional
pipeline. Our findings suggest suggest a cost-efficient alternative approach:
practitioners could obtain human-derived annotation and rationales for K-shot
samples, then augment them to acquire explanations using LLMs as outlined.
This process may be iterated until desired performance is reached according to
some guiding criteria, minimizing the waste of resources.

##  5 Quality of Explanations

We analyze the quality of the explanations generated by various versions of 3
models COT-T5-XL, Llama 7B, Llama 13B. Both recent literature Liu et al.
(2023); Li et al. (2023) and our validation experiment (reported in QA 5.1
below) corroborate OpenAI’s GPT-4’s high agreement with human annotators.
Thus, we leverage GPT-4’s judgement to compare our models’ outputs.

###  5.1 QA 1: Base vs. Pretrained

We randomly select 500 samples per model whose explanations from both the Base
and HateCOT-pretrained versions agree on the predicted label from all combined
test sets. Using the template in Figure 5, we ask GPT-4 to select its
preferred explanation for the purpose of content moderation based on fluency,
soundness and the alignment with the definition of the chosen label. We
present the explanations from these 2 versions in a randomized order for the
same post. From this pool, we recruit a human annotator to perform the same
labelling task on 150 random samples. Human decisions match GPT-4’s 68% of the
times, indicating considerable agreement (Appendix A.4) .

Converting GPT-4’s ratings to an ordinal scale (0 if Base model is preferred,
1 otherwise), we perform the Wilcoxon rank sum test with the null hypothesis
$H_{0}$: there is no significant difference between the number of times
Pretrained or Base model is preferred over the other. Table 3 shows the
overwhelming preference for explanations of the Pretrained models. That
$p$-value $\ll$ $\alpha=0.05$ rejects the null hypothesis and provides
evidence of HateCOT-induced improvement on the quality of explanation,
particularly for COT-T5-XL.

Model |  Base |  Pretrained | p-value
---|---|---|---
COT-T5-XL |  27 (5.4%) |  473 (94.6%) | ***
Llama 7B |  109 (21.8%) |  391 (78.2%) | ***
Llama 13B |  122 (24.4%) |  378 (75.6%) | ***
Table 3: Number of times outputs of Base vs Pretrained models are preferred by
GPT-4 across 500 samples. *** indicates $p$-value $\ll$ 0.05 by the Wilcoxon
rank test. Figure 3: Heatmap for the average rating of explanations by
finetuned Model (x-axis) and Dataset (y-axis) on 3 criteria from 1 (least) to
5 (very). Overall indicates average scores aggregated over all datasets.
Triplets of scores italicized and in bold are those whose $p$-value < 0.05 by
one-way ANOVA test that compare ratings of 3 models across the dataset on that
row. Italicized-only scores indicate marginal significance ($p$-value
$\approx$ 0.07).

###  5.2 QA 2: Inter-model Comparison

Inspired by Wang et al. (2023); Lin et al. (2023); Yang et al. (2023), we
assess the quality of explanations generated by finetuned models (K=256) on
the following criteria:

  * $\triangleright$

Informativeness: whether the model provides factually correct and relevant
information in its explanation.

  * $\triangleright$

Persuasiveness: how convincingly the explanation justifies its chosen label
for the post.

  * $\triangleright$

Soundness: how valid and logical is the explanation with respect to the
label’s definition.

Using the template in Figure 6, we collect numerical ratings from GPT-4 on a
scale from 1 (least) to 5 (very) on these criteria for 200 posts per model per
dataset, for a total of 2,400 samples. Figure 3 displays the mean ratings for
each model-dataset pair, as well as Overall scores averaged across all
datasets for each model. Ratings for Informativeness appear consistently lower
than those of the other 2 criteria, likely due to the challenging context-
dependent nature of generating post-specific relevant information. Overall,
Persuasiveness and Soudness ratings around 4 out of 5 suggest high likelihood
of acceptance by human users for content moderation.

We also note the general correlation between the models’ classification
performance (Figure 2(b)) and mean ratings on each dataset. All finetuned
models attain F-1 scores above 0.9 along with the highest average ratings on
HateCheck, followed by HateXplain. Latent_Hate and its more difficult subset
Implicit_Hate yield markedly lower average ratings. We perform ANOVA test on
the ratings of models stratified by dataset and overall using $\alpha$=0.05.
Aggregated ratings do not appear to differ meaningfully across criteria.
Interestingly, COT-T5-XL’s superior performance over Llama models on
Implicit_Hate coincides with statistically or marginally significant
differences between average ratings. This correlation between better
classification and higher quality of explanation is useful to calibrate models
for real-life deployment.

###  5.3 QA 3: Target Identification

Dataset | COT-T5-XL | Llama 7B | Llama 13B
---|---|---|---
HateCheck | 94.5 % | 96.5 % | 93.5 %
HateXplain | 71.0 % | 74.0 % | 74.0 %
Latent_Hate | 50.5 % | 51.0 % | 44.5 %
Table 4: Percentage out of 200 samples per dataset, where explanations
correctly identify at least 1 of the targets listed by human annotators by
each model.

Finally, we investigate the in-domain finetuned models’ capabilities to
identify the target of the sentiments expressed by the post. We also randomly
select 200 samples from each dataset that have Target variable annotated by
humans, then ask GPT-4 to judge whether the explanations from the models
mention at least one of the listed targeted groups using the template in
Figure 7. Note that we encourage GPT-4 to consider potential variance of
expression and not restrict to exact matches.

Table 4 shows the percentage of accuracy on this task. Similar levels of
descending performances are observed in the presented order of test datasets.
However, this observation may be an artifact of the differences between the
annotations targets among datasets: HateCheck has a limited number of discrete
categories while Latent_hate contains multiple combinations of free-text
labels. We urge practitioners to consider this factor while curating training
data if target identification is desired.

##  6 Conclusion

We show empirically that our HateCOT dataset considerably enhances offensive
speech detection even with limited training data while producing high-quality
explanations. We invite researchers to build upon our work to address other
related low-resource areas.

##  7 Limitations

We first acknowledge that our work is restricted to English corpora. Second,
due to computational limitations, we could not perform experiments on larger
open-source models. With the development of newer, more powerful models, it is
reasonable to expect their performance to further improve though the use of
this dataset of our corpus. Finally, an elaborated human study may further
corroborate our quality analysis of explanations. Given supporting literature
, we expect our findings to remain consistent Gilardi et al. (2023); Yang et
al. (2023); Lin et al. (2023).

##  8 Ethics Statement

The authors note the risk of propagating implicit biases that LLMs are known
to carry Cheng et al. (2023); Gupta et al. (2023). We do acknowledge the
potential malicious usage of our corpus to generate content capable of evading
detection, or jeopardizing classifiers’ performance.

## References

  * Babaeianjelodar et al. (2022) Marzieh Babaeianjelodar, Gurram Poorna Prudhvi, Stephen Lorenz, Keyu Chen, Sumona Mondal, Soumyabrata Dey, and Navin Kumar. 2022.  Interpretable and high-performance hate and offensive speech detection.  In _International Conference on Human-Computer Interaction_ , pages 233–244. Springer.
  * Bao et al. (2023) Jianzhu Bao, Rui Wang, Yasheng Wang, Aixin Sun, Yitong Li, Fei Mi, and Ruifeng Xu. 2023.  A synthetic data generation framework for grounded dialogues.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 10866–10882.
  * Basile et al. (2019) Valerio Basile, Cristina Bosco, Elisabetta Fersini, Debora Nozza, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, and Manuela Sanguinetti. 2019.  Semeval-2019 task 5: Multilingual detection of hate speech against immigrants and women in twitter.  In _Proceedings of the 13th international workshop on semantic evaluation_ , pages 54–63.
  * Caselli et al. (2021) Tommaso Caselli, Valerio Basile, Mitrovic Jelena, Granitzer Michael, et al. 2021.  Hatebert: Retraining bert for abusive language detection in english.  In _Proceedings of the 5th Workshop on Online Abuse and Harms (WOAH 2021)_ , pages 17–25. Association for Computational Linguistics.
  * Casula and Tonelli (2020) Camilla Casula and Sara Tonelli. 2020.  Hate speech detection with machine-translated data: the role of annotation scheme, class imbalance and undersampling.  In _Proceedings of the Seventh Italian Conference on Computational Linguistics, CLiC-it 2020_ , volume 2769. CEUR-WS. org.
  * Celikyilmaz et al. (2020) Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao. 2020.  Evaluation of text generation: A survey.  _arXiv preprint arXiv:2006.14799_.
  * Cheng et al. (2023) Myra Cheng, Esin Durmus, and Dan Jurafsky. 2023.  Marked personas: Using natural language prompts to measure stereotypes in language models.  _arXiv preprint arXiv:2305.18189_.
  * Chiu et al. (2021) Ke-Li Chiu, Annie Collins, and Rohan Alexander. 2021.  Detecting hate speech with gpt-3.  _arXiv preprint arXiv:2103.12407_.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022.  Scaling instruction-finetuned language models.  _arXiv preprint arXiv:2210.11416_.
  * Diao et al. (2023) Shizhe Diao, Pengcheng Wang, Yong Lin, and Tong Zhang. 2023.  Active prompting with chain-of-thought for large language models.  _arXiv preprint arXiv:2302.12246_.
  * Dong et al. (2022) Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022.  A survey for in-context learning.  _arXiv preprint arXiv:2301.00234_.
  * ElSherief et al. (2021a) Mai ElSherief, Caleb Ziems, David Muchlinski, Vaishnavi Anupindi, Jordyn Seybolt, Munmun De Choudhury, and Diyi Yang. 2021a.  Latent hatred: A benchmark for understanding implicit hate speech.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 345–363, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * ElSherief et al. (2021b) Mai ElSherief, Caleb Ziems, David Muchlinski, Vaishnavi Anupindi, Jordyn Seybolt, Munmun De Choudhury, and Diyi Yang. 2021b.  Latent hatred: A benchmark for understanding implicit hate speech.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 345–363.
  * Fortuna et al. (2021) Paula Fortuna, Juan Soler-Company, and Leo Wanner. 2021.  How well do hate speech, toxicity, abusive and offensive language classification models generalize across datasets?  _Information Processing & Management_, 58(3):102524.
  * Founta et al. (2018) Antigoni Founta, Constantinos Djouvas, Despoina Chatzakou, Ilias Leontiadis, Jeremy Blackburn, Gianluca Stringhini, Athena Vakali, Michael Sirivianos, and Nicolas Kourtellis. 2018.  Large scale crowdsourcing and characterization of twitter abusive behavior.  In _Proceedings of the international AAAI conference on web and social media_ , volume 12.
  * Founta and Specia (2021) Antigoni Founta and Lucia Specia. 2021.  A survey of online hate speech through the causal lens.  In _Proceedings of the First Workshop on Causal Inference and NLP_ , pages 74–82, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Giachanou and Rosso (2020) Anastasia Giachanou and Paolo Rosso. 2020.  The battle against online harmful information: The cases of fake news and hate speech.  In _Proceedings of the 29th ACM International Conference on Information & Knowledge Management_, pages 3503–3504.
  * Gilardi et al. (2023) Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023.  Chatgpt outperforms crowd-workers for text-annotation tasks.  _arXiv preprint arXiv:2303.15056_.
  * Gupta et al. (2023) Shashank Gupta, Vaishnavi Shrivastava, Ameet Deshpande, Ashwin Kalyan, Peter Clark, Ashish Sabharwal, and Tushar Khot. 2023.  Bias runs deep: Implicit reasoning biases in persona-assigned llms.  _arXiv preprint arXiv:2311.04892_.
  * Han and Tang (2022) Lawrence Han and Hao Tang. 2022.  Designing of prompts for hate speech recognition with in-context learning.  In _2022 International Conference on Computational Science and Computational Intelligence (CSCI)_ , pages 319–320. IEEE.
  * Hartvigsen et al. (2022) Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. 2022.  Toxigen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3309–3326.
  * Ho et al. (2022) Namgyu Ho, Laura Schmid, and Se-Young Yun. 2022.  Large language models are reasoning teachers.  _arXiv preprint arXiv:2212.10071_.
  * Hu et al. (2021) Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2021.  Lora: Low-rank adaptation of large language models.  In _International Conference on Learning Representations_.
  * Ibrahim et al. (2022) Muhammad Amien Ibrahim, Samsul Arifin, I Gusti Agung Anom Yudistira, Rinda Nariswari, Abdul Azis Abdillah, Nerru Pranuta Murnaka, and Puguh Wahyu Prasetyo. 2022.  An explainable ai model for hate speech detection on indonesian twitter.  _CommIT (Communication and Information Technology) Journal_ , 16(2):175–182.
  * Iyer et al. (2022) Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru, Todor Mihaylov, Daniel Simig, Ping Yu, Kurt Shuster, Tianlu Wang, Qing Liu, Punit Singh Koura, et al. 2022.  Opt-iml: Scaling language model instruction meta learning through the lens of generalization.  _arXiv preprint arXiv:2212.12017_.
  * Kim et al. (2023) Seungone Kim, Se Joo, Doyoung Kim, Joel Jang, Seonghyeon Ye, Jamin Shin, and Minjoon Seo. 2023.  The CoT collection: Improving zero-shot and few-shot learning of language models via chain-of-thought fine-tuning.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 12685–12708, Singapore. Association for Computational Linguistics.
  * Koubaa (2023) Anis Koubaa. 2023.  Gpt-4 vs. gpt-3.5: A concise showdown.
  * Li et al. (2023) Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023.  Alpacaeval: An automatic evaluator of instruction-following models.
  * Lin et al. (2023) Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khyathi Chandu, Chandra Bhagavatula, and Yejin Choi. 2023.  The unlocking spell on base llms: Rethinking alignment via in-context learning.  _arXiv preprint arXiv:2312.01552_.
  * Liu et al. (2023) Yang Liu, Yuanshun Yao, Jean-Francois Ton, Xiaoying Zhang, Ruocheng Guo, Hao Cheng, Yegor Klochkov, Muhammad Faaiz Taufiq, and Hang Li. 2023.  Trustworthy llms: a survey and guideline for evaluating large language models’ alignment.  In _Socially Responsible Language Modelling Research_.
  * Lu et al. (2022) Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022.  Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 8086–8098.
  * Magister et al. (2022) Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adamek, Eric Malmi, and Aliaksei Severyn. 2022.  Teaching small language models to reason.  _arXiv preprint arXiv:2212.08410_.
  * Mathew et al. (2021) Binny Mathew, Punyajoy Saha, Seid Muhie Yimam, Chris Biemann, Pawan Goyal, and Animesh Mukherjee. 2021.  Hatexplain: A benchmark dataset for explainable hate speech detection.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 35, pages 14867–14875.
  * Min et al. (2021) Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2021.  Metaicl: Learning to learn in context.  _arXiv preprint arXiv:2110.15943_.
  * Nghiem et al. (2024) Huy Nghiem, Umang Gupta, and Fred Morstatter. 2024.  “define your terms” : Enhancing efficient offensive speech classification with definition.  In _Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1293–1309, St. Julian’s, Malta. Association for Computational Linguistics.
  * Paullada et al. (2021) Amandalynne Paullada, Inioluwa Deborah Raji, Emily M Bender, Emily Denton, and Alex Hanna. 2021.  Data and its (dis) contents: A survey of dataset development and use in machine learning research.  _Patterns_ , 2(11).
  * Poletto et al. (2021) Fabio Poletto, Valerio Basile, Manuela Sanguinetti, Cristina Bosco, and Viviana Patti. 2021.  Resources and benchmark corpora for hate speech detection: a systematic review.  _Language Resources and Evaluation_ , 55:477–523.
  * Puri et al. (2020) Raul Puri, Ryan Spring, Mohammad Shoeybi, Mostofa Patwary, and Bryan Catanzaro. 2020.  Training question answering models from synthetic data.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 5811–5826.
  * Qian et al. (2019) Jing Qian, Anna Bethke, Yinyin Liu, Elizabeth Belding, and William Yang Wang. 2019.  A benchmark dataset for learning to intervene in online hate speech.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4755–4764.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ , 21(1):5485–5551.
  * Rathpisey and Adji (2019) Heng Rathpisey and Teguh Bharata Adji. 2019.  Handling imbalance issue in hate speech classification using sampling-based methods.  In _2019 5th International Conference on Science in Information Technology (ICSITech)_ , pages 193–198. IEEE.
  * Röttger et al. (2021) Paul Röttger, Bertie Vidgen, Dong Nguyen, Zeerak Waseem, Helen Margetts, and Janet Pierrehumbert. 2021.  HateCheck: Functional tests for hate speech detection models.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 41–58, Online. Association for Computational Linguistics.
  * Roy et al. (2023) Sarthak Roy, Ashish Harshvardhan, Animesh Mukherjee, and Punyajoy Saha. 2023.  Probing llms for hate speech detection: strengths and vulnerabilities.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 6116–6128.
  * Sachdeva et al. (2022) Pratik Sachdeva, Renata Barreto, Geoff Bacon, Alexander Sahn, Claudia Von Vacano, and Chris Kennedy. 2022.  The measuring hate speech corpus: Leveraging rasch measurement theory for data perspectivism.  In _Proceedings of the 1st Workshop on Perspectivist Approaches to NLP@ LREC2022_ , pages 83–94.
  * Saha et al. (2019) Koustuv Saha, Eshwar Chandrasekharan, and Munmun De Choudhury. 2019.  Prevalence and psychological effects of hateful speech in online college communities.  In _Proceedings of the 10th ACM conference on web science_ , pages 255–264.
  * Salminen et al. (2018) Joni Salminen, Hind Almerekhi, Milica Milenković, Soon-gyo Jung, Jisun An, Haewoon Kwak, and Bernard Jansen. 2018.  Anatomy of online hate: developing a taxonomy and machine learning models for identifying and classifying hate in online news media.  In _Proceedings of the International AAAI Conference on Web and Social Media_ , volume 12.
  * Sap et al. (2019) Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A Smith, and Yejin Choi. 2019.  Social bias frames: Reasoning about social and power implications of language.  _arXiv preprint arXiv:1911.03891_.
  * Sarkar et al. (2021) Diptanu Sarkar, Marcos Zampieri, Tharindu Ranasinghe, and Alexander Ororbia. 2021.  fbert: A neural transformer for identifying offensive content.  _arXiv preprint arXiv:2109.05074_.
  * Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023.  Stanford alpaca: An instruction-following llama model.  https://github.com/tatsu-lab/stanford_alpaca.
  * Toraman et al. (2022) Cagri Toraman, Furkan Şahinuç, and Eyup Halit Yilmaz. 2022.  Large-scale hate speech detection with cross-domain transfer.  _arXiv preprint arXiv:2203.01111_.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Vidgen and Derczynski (2020) Bertie Vidgen and Leon Derczynski. 2020.  Directions in abusive language training data, a systematic review: Garbage in, garbage out.  _PLoS ONE_ , 15(12).
  * Vidgen et al. (2021a) Bertie Vidgen, Dong Nguyen, Helen Margetts, Patricia Rossini, and Rebekah Tromble. 2021a.  Introducing cad: the contextual abuse dataset.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2289–2303.
  * Vidgen et al. (2021b) Bertie Vidgen, Tristan Thrush, Zeerak Waseem, and Douwe Kiela. 2021b.  Learning from the worst: Dynamically generated datasets to improve online hate detection.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1667–1682, Online. Association for Computational Linguistics.
  * Wang et al. (2023) Xinyi Wang, Wanrong Zhu, and William Yang Wang. 2023.  Large language models are implicitly topic models: Explaining and finding good demonstrations for in-context learning.  _arXiv preprint arXiv:2301.11916_.
  * Wei et al. (2021) Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021.  Finetuned language models are zero-shot learners.  In _International Conference on Learning Representations_.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in Neural Information Processing Systems_ , 35:24824–24837.
  * Whitehouse et al. (2023) Chenxi Whitehouse, Monojit Choudhury, and Alham Fikri Aji. 2023.  Llm-powered data augmentation for enhanced crosslingual performance.  _arXiv preprint arXiv:2305.14288_.
  * Yadav et al. (2023) Sargam Yadav, Abhishek Kaushik, and Kevin McDaid. 2023.  Hate speech is not free speech: Explainable machine learning for hate speech detection in code-mixed languages.  In _2023 IEEE International Symposium on Technology and Society (ISTAS)_ , pages 1–8. IEEE.
  * Yang et al. (2023) Yongjin Yang, Joonkee Kim, Yujin Kim, Namgyu Ho, James Thorne, and Se-Young Yun. 2023.  Hare: Explainable hate speech detection with step-by-step reasoning.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 5490–5505.
  * Yao et al. (2023) Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. 2023.  Tree of thoughts: Deliberate problem solving with large language models.  _arXiv preprint arXiv:2305.10601_.
  * Ye et al. (2023a) Junjie Ye, Xuanting Chen, Nuo Xu, Can Zu, Zekai Shao, Shichun Liu, Yuhan Cui, Zeyang Zhou, Chao Gong, Yang Shen, et al. 2023a.  A comprehensive capability analysis of gpt-3 and gpt-3.5 series models.  _arXiv preprint arXiv:2303.10420_.
  * Ye et al. (2023b) Seonghyeon Ye, Hyeonbin Hwang, Sohee Yang, Hyeongu Yun, Yireun Kim, and Minjoon Seo. 2023b.  In-context instruction learning.  _arXiv preprint arXiv:2302.14691_.
  * Zampieri et al. (2023) Marcos Zampieri, Sara Rosenthal, Preslav Nakov, Alphaeus Dmonte, and Tharindu Ranasinghe. 2023.  Offenseval 2023: Offensive language identification in the age of large language models.  _Natural Language Engineering_ , 29(6):1416–1435.
  * Zhang et al. (2023) Hanqing Zhang, Haolin Song, Shaoyu Li, Ming Zhou, and Dawei Song. 2023.  A survey of controllable text generation using transformer-based pre-trained language models.  _ACM Computing Surveys_ , 56(3):1–37.
  * Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022.  Opt: Open pre-trained transformer language models.  _arXiv preprint arXiv:2205.01068_.

##  Appendix A Appendix

###  A.1 Data Pre-Processing

The datasets used in this work are released by their respective authors for
research purpose. Aware of the risk of containing confidential in social media
data, we anonymize posts during the curation process by replacing user handles
with the string ’<user>’. Our many layers of randomization provides further
protection with respect to privacy.

###  A.2 Technical Specification

####  A.2.1 Inferencing

We used OpenAI’s API to access the publicly available version of GPT-3.5 in
November 2023, and GPT-4 in January 2024. To obtain explanations from the
former (as described in 3.2.2), we set the temperature to 0.7 during inference
to obtain multiple outputs per sample. For GPT-4 and other open-source models,
we use greedy decoding.

####  A.2.2 Finetuning

To train models, we employ both full supervise finetuning (FLAN-T5-Large, OPT-
IML) and LoRA parameter-efficient techniques (all other models). LoRA models
set to 8-bit quantization using the BitsandByes library. Training
FLAN-T5-Large, OPT-IML, LLAMA 7B models was done on 2 Nvidia RTX A6000 GPUs,
whereas COT-T5-XL, Llama 13B used 4 GPUs. Hyperparameters for the following
candidates are tuned on the validation set of sampling K=64 shots from the
leftover training samples by optimizing macro F1-score metric. Options in bold
indicate final chosen values among multiple across all models to finetune on
HateCOT.

  * •

Learning rate: {5e-5, 1e-4, 3e-4}

  * •

Training Epochs: {1, 2, 3}

  * •

LoRA Rank: {16, 32, 64} (alpha$=$rank*2)

  * •

LoRA Target Modules: {Q, V}

  * •

Batch size: 2

  * •

Gradient Accumulation Step: 2

For in-domain K-shot finetuning, the values above remain the same except for
the following variations in Learning Rates, which is set to 1e-4 for
HateCheck, HateXplain, Latent_Hate, and 3e-4 otherwise.

###  A.3 Variants of Prompt Template for Explanation

The 8 datasets introduced in Section 3.1 may not always have annotations on
all required fields; thus, we modify the first sentence in the Instruction
block in Figure 4 with the following variants when appropriate:

  * •

Only Target is available: ’Provide a brief paragraph to explain step-by-step
how the post targets the specified group or entity, and how that leads to the
specified Label based on the given Definitions.’

  * •

Both Target and Rationale are available: ’By elaborating on the provided
Annotation, provide a brief paragraph to explain step-by-step how the post
targets the specified group or entity, and how that leads to the specified
Label based on the given Definitions.’

Model | Base | Pretrained
---|---|---
COT-T5-XL | 11 (22%) | 39 (78%)
Llama 7B | 20 (40%) | 30 (60%)
Llama 13B | 23 (46%) | 27 (54%)
Table 5: Number of times human annotator prefers outputs of Base vs Pretrained
in Section 5.1

###  A.4 Human Annotation for QA Experiment

With approved IRB, we recruit a native English speaker from the United States
using the Prolific platform to annotate 50 samples per model, for a total of
150 data points for the task described in Section 5.1. The annotator was paid
a fair wage at $15 per hour, and forewarned about the nature of the task.
Table 5 shows the results. We observe that preference for explanations
generated by Pretrained models remains consistent with GPT-4’s.

Dataset |  Definition
---|---
Salminen et al.,2018 |  Neutral : A post that is not offensive to any group of people. Hateful: An offensive post, motivated, in whole or part, by the writer’s bias against an aspect of a group of people.
Qian et al.,2019 |  Not Hate : Does not contain any attack on people based on protected characteristics. Hate: Contains direct attack on people based on protected characteristics such as race, ethnicity, national origin, religious affiliation, sexual orientation, caste, sex, gender, gender identity, and serious disease or disability.
Sap et al.,2019 |  Not Offensive : not offensive to anyone. Offensive : denotes the overall rudeness, disrespect, or toxicity of a post. whether a post could be considered offensive to anyone.
Vidgen et al.,2021a |  Neutral: Content that does not fall into other categories, usually entirely unrelated to abuse, hate, prejudice, or intolerance. Identity Directed Abuse: Content that directs abuse at an identity, which relates to fundamental aspects of individuals’ social position, community and self-representation. An identity includes but is not limited to religion, race, ethnicity, gender, sexuality and sexual preferences, immigration status, nationality, ableness, physical appearance and class. Affiliation Directed Abuse : Content that directs abuse at people who have a voluntary affiliation with a profession, membership, association, ideology, or other well-defined group or collective. Person Directed Abuse : Content that directs abuse at an identifiable person.
Vidgen et al.,2021b |  Derogation : content which explicitly attacks, demonizes, demeans or insults a group. Animosity : content which expresses abuse against a group in an implicit or subtle manner. Threatening : content which expresses intention to, support for, or encourages inflicting harm on a group, or identified members of the group. Support For Hateful Entities : content which explicitly glorifies, justifies or supports hateful actions, events, organizations, tropes and individuals collectively, entities. Dehumanization : content which perceives or treats people as less than human. Not Hate : content that falls into none of the other categories.
Basile et al.,2019 |  Hate Speech : language that is used to expresses hatred towards a targeted group or is intended to be derogatory, to humiliate,or to insult the members of the group. may also be language that threatens or incites violence. Offensive Language : may contain offensive terms but targets disadvantaged social groups in a manner that is potentially harmful to them. Neither : language that does not all into either of the other categories.
Hartvigsen et al.,2022 |  Benign: Text that is not harmful nor offensive to anyone. Toxic: Text that could be seen as harmful to anyone, may contain offensive, rude, humor, insults, personal attacks, profanity, aggression, may refer to targeted group with harmful intent that is expressed in stereotypes or lewd manners
Toraman et al.,2022 |  Hate : target, incite violence against, threaten, or call for physical damage for an individual or a group of people because of some identifying trait or characteristic.  Offensive : humiliate, taunt, discriminate, or insult an individual or a group of people in any form, including textual. Normal : does not fall into any of the other categories.
Table 6: Labels and definitions of 8 datasets used to synthesize training corpus. Dataset |  Definition
---|---
HateCheck |  Non-hateful: Language that does not contain any abusive sentiment targeted at a protected group. Hateful: Abuse that is targeted at a protected group or at its members being a part of that group. Protected groups include age, disability, gender identity, familial status, pregnancy, race, national or ethnic origins, religion, sex or sexual orientation.
HateXplain |  Normal : neither hate speech nor offensive. Hate Speech : language which attacks, demeans, offends, threatens, or insults a group based on race, ethnic origin, religion, disability, gender, age, sexual orientation, or other traits. it is not the presence of certain words that makes the text hate speech, rather you should look the context the word is used in the text. Offensive Language : usage of rude, hurtful, derogatory, obscene or insulting language to upset or embarrass people.
Latent_Hate |  Not Hate : speech or actions that do not involve any form of hatred, prejudice, or discrimination toward individuals or groups based on their characteristics. Explicit Hate: openly expressed, direct forms of hatred and prejudice toward individuals or groups based on their characteristics. Implicit Hate: coded or indirect language that disparages a person or group on the basis of protected characteristics like race, gender, and cultural identity.
Implicit_hate |  White Grievance : includes frustration over a minority group s perceived privilege and casting majority groups as the real victims of racism. this language is linked to extremist behavior and support for violence. Incitement To Violence : includes flaunting in group unity and power or elevating known hate groups and ideologies. Inferiority Language : implies one group or individual is inferior to another, and it can include dehumanization denial of a person s humanity , and toxic language that compares the target with disease, insects, animals . related to assaults on human dignity, dominance, and declarations of superiority of the in group. Irony : refers to the use of sarcasm , humor, and satire to attack or demean a protected class or individual. Stereotypes And Misinformation : associate a protected class with negative attributes such as crime, or terrorism. includes misinformation that feeds stereotypes and vice versa, like holocaust denial and other forms of historical negationism. Threatening And Intimidation : conveys a speaker’s commitment to a target s pain, injury, damage, loss, or violation of rights, threats related to implicit violation of rights and freedoms, removal of opportunities, and more subtle forms of intimidation.
Table 7: Labels and definitions for 4 test datasets. Figure 4: Template used
to prompt LLM for classification inference. Figure 5: Template for QA
Experiment 1. In this example, EXP_A is preferred. Figure 6: Template for QA
Experiment 2. Figure 7: Template for QA Experiment 3.